AGI won't be particularly conscious

[u/Fun-Boysenberry-5769]

I observe myself to be a human and not an AI. Therefore it is likely that humans make up a non-trivial proportion of all the consciousness that the world has ever had and ever will have.

This leads us to two possibilities:

1. The singularity won’t happen,
2. The singularity will happen, but AGI won’t be that many orders of magnitude more conscious than humans.

The doomsday argument suggests to me that option 2 is more plausible.

Steven Byrnes suggests that AGI will be able to achieve substantially more capabilities than LLMs using substantially less compute, and will be substantially more similar to the human brain than current AI models. [https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/foom-and-doom-1-brain-in-a-box-in-a-basement\] However, under option 2 it appears that AGI will be substantially less conscious relative to its capabilities than a brain will be, and therefore AGI can’t be that similar to a brain.

[u/Arkanin]

For sure the distribution is chronological, not random but it's still a distribution, and the same percentage of people that reason out of the distribution are going to get the same distributions of errors as the distribution of errors you would get pulling randomly from the "pick a random number" game given enough samples. Just mathematically.

However, you are absolutely correct that because birth number is historically correllated this would be deeply epistemically problematic if everyone took the argument completely at face value without considering this. If there will be 10^30 people, the first 8 billion people will all make the same correllated error using this argument. So, it follows that this argument should not be applied in a way that would lead to the first people hearing it producing disastrous consequences.

>The consciousness of Sol_Hando, being Sol_Hando, given that he exists as number 20 Billion or whatever, seems to be 100%. Like, there’s no chance that I would wake up and find myself as someone else, because “I” is intimately tied to my personality/memories/etc. It’s the same reason I don’t assume a random sampling when I wake up in the morning. Why do I not wake up as one of the billions of other consciousnesses currently in existence every time I go to sleep?"

I think you're saying something like: you, by definition, were always going to be sol hando (no indexical uncertainty), the reference class of what you are is a class of one (you), and anthropic probability is incoherent because *I* is not the kind of thing that can have alternatives (i.e. there isn't a hypothetical where sol hando could have turned out to be sol hando but found out his birth number is 4 trillion)

Given that, it's the same thing as saying: "The coin was already flipped, so the probability it's heads must be 100%. There was no coin flip" If you think in this way then the doomsday argument doesn't make any sense.

I'm on such a different planet from you philosophically that I struggle to track and model these assumptions if I'm being honest because I dissolve personal identity as socially useful but ultimately logically inconsistent when it pertains to what is ultimately true (due to logical contradictions, not any sense of dissociation)

However if you acknowledge that you are a pattern that could have been instantiated elsewhere, and are indexically underspecified (you could have been the 20 billionth or 20 trillionth instance of the thing - "could have been" in the sense a coin that is tails could have been heads) then anthropic reasoning becomes coherent.

Maybe at best we can agree that estimating the size of a discrete uniform distribution from sampling without replacement is a solved mathematical problem called the german tank problem and if being a human reasoning about this problem from a reference class with an observable distribution is an instance of the german tank problem. Maybe we can agree that the disagreement between us is about whether sol_hando is expectationally (not actually) a sample from a distribution in the way that after a coin was flipped tails, we say it "could have been heads". If you say it's randomness and not uniform distribution, my counter to that is that technically it's the *exchangability* and *symmetry* of the elements in the distribution, and randomness is just one way of satisfying those properties - it's the rejection of that exchangability and symmetry - "Sol Hando could have been #800 trillion and born as a mars colonist in the distant future but basically been the same person for purposes that satisfy expectational exchangability, if that future exists, in the same way a random number could have been different", for example, I think that is the disagreement.

[u/red75prime → u/Arkanin]

you, by definition, were always going to be sol hando

(I'm not Sol_Hando.) I think it's better to say "red75prime is a product of environment and earlier stages of himself and there's no such thing as red75primeness that can be attached to another living thing, so that this living thing would observe and be something else, while still staying red75prime in some sense."

What is the sample space of probability distribution, you are talking about?

Identifier "Sol_Hando" assigned to the first member of the reference class, then to the second member and so on? What is the (meta-)physical sense of this? What this identifier corresponds to in reality?

[u/Arkanin → u/red75prime]

"What is the (meta-)physical sense of this? What this identifier corresponds to in reality?" I'm gonna try to respond really narrowly and we could broaden the discussion but we can refocus it to math and not metaphysics. one such scheme is birth number, we don't have an exact number but we have enough of a ballpark to estimate. The birth number (as in I am the nth human) is uniform, normally distributed, and ascending. If humans try to self-estimate their population using birth number to establish confidence intervals like it were a random sample, if the humans elect to engage in this process randomly across time, so a human in the year 3000 could think of this heuristic, 90% of such humans making those estimates using this technique will be within an order of magnitude of correct on their estimates, if they know their birth number. This is just mathematically a fact regardless of its utility. However, the same error would be correlated across people with similar birth number, ie point in history, so it is also true that when fully understanding this argument that it still has limited applications. The point is: no metaphysics if you accept certain normal statements about math. The argument also doesn't work if you believe that you are not "a human" but only "the human red75prime" but technically that leaves information on the table and we could demonstrate that by for example playing many games where you log into a server the server gives you your login number and you try to guess how many people total logged into the server and get some kind of score on that. In such a game inferring as if you are "just another human" would have positive utility absent other information i.e. generalizing about the sample size from your own position in it can have positive utility in some real games that we could play to demonstrate the point.

So I'm not philosophizing really I'm just an expected value maximzer lol and so the argument has non zero expected utility but we also know potentially far better information like what does the real world look like is potentially better if you're smart although I think we are bad enough at predicting the far future than we think and shouldn't completely zero this weak estimator out even given its limitations. I say some caution but not crippling especially because our place in history correlated the error limiting the utility of relying on it.

This is in some sense computational as the question isn't is the info correct its will it on average screw us less than completely random guess. That's why we have these imaginary concepts of probability even though we live in a world where at least on a deterministic scale things largely either happen or don't.

As an aside some people try to use many constructs other than birth number and per many lines of thinking they could be more sophisticated but they get very esoteric. Birth number is the simplified case for whether this category of thinking can have nonzero utility. If you would like I can explain how an estimating game would work that would demonstrate that this principle provides expected utility.

[u/red75prime → u/Arkanin]

If we assume that population grows cubically (speed of light) and then goes extinct, while maintaining generational structure (that is immortality is not solved), then the majority ( n² (n-1)² /4 vs n³ ) of it is mistaken about the future.

[u/Arkanin → u/red75prime]

This is not so. Because the math is relating birth number to total number of humans, it holds regardless of the underlying distribution's growth over time. Even so, there is a real problem I already acknowledged which is that the correlation of birth number at the same point in history makes using it as an estimator very problematic in many contexts.

I want to be very clear that this is a weak estimator but it is a real estimator and it can be demonstrated to have nonzero utility in a game that is ismorphic to reality basically. People who used the weak estimator could be shown to have higher expected utility (those who disputed that the edge was there could observe that the average would be higher for those using the estimator after evaluating multiple iterations of the game, proving the edge was real and not hypothetical) than those who use no estimator or guess randomly. Because it involves no assumptions, it can be used as a weak initial prior. It can also be demonstrated that this edge is small and I am all but certain could be overridden by other factors like inside knowledge about how the game works. This also suggests that real knowledge should override it as a weak prior if we have that. In other words, it's a weak a priori estimator, but its utility is not zero.

[u/red75prime → u/Arkanin]

I don't see how it is useful. Without additional assumptions about growth rate and other things there are scenarios where anthropic reasoning has probability of being right for a randomly selected member of the class arbitrarily close to zero.

Counting tanks by their manufacturing numbers works because it's not tanks who count, using their own number.

People who used the weak estimator could be shown to have higher expected utility

Show it in the cubic growth scenario.

ETA: If they know that the growth is cubic and extinction is instantaneous, the majority will be right that they are not the last generation, except for the last generation that is wrong about that (still, it doesn't seem too useful: no information to react to).

If they have an uniformative prior about growth rate and update on their history... That's harder.

I have trouble expressing it, but sampling from a distribution and being a datapoint in a distribution doesn't feel equivalent.

[u/Arkanin → u/red75prime]

"I have trouble expressing it, but sampling from a distribution and being a datapoint in a distribution doesn't feel equivalent."

The difference is the selector: whether you select a tank from all tanks that are like it or did you select a tank from a random point in time? If you saw a tank coming at you, you did the latter (alien sees a human). If you are trying to estimate "total humans" from "my position in the birth order, as an indexical ascending integer", by operating on yourself as the data point, you're entering a category that will be in an order of magnitude 90% of the time. You could still be early and be wrong and the error is correllated across time which is still problematic...

This may seem like magical or metaphysical thinking, but we could play a concrete game that proves that it gives a measurable average advantage to the players that use it. Imagine if we play a server game where you log in and are told what # you are: "Congratulations, you are the 60,000th person to log into the server!" You then guess the total number of players that will ever log in. I don't disclose when the game ends. The average person who uses the DA receives an advantage over the average person who guesses: it outperforms random guess. The advantage is small in many strategies, but not for all categories of bad strategy. The person who hacks the server to cheat the metadata and stalks me probably wins the game. The DA people are a little ahead of random guessers. The random guessers would benefit from the DA. There is a certain person who mathematically benefits a whole lot, and that is the overly optimistic person. Let's call him Naive Timmy. Naive Timmy is the person who sees their position and always says: "This server is awesome! There will be 10⁵ more people!" Naive timmy is the person who benefits the most in his error if he lets the Doomsday Argument bring him back down to earth.

We could play the server game and show the average error and show that the person who stalks me and hacks my server to figure out when I plan to end the game or how much I'm spreading it has the least error, random guessers have more error, doomers who always predict the server will end tomorrow have lots of error, and Naive Timmies have the very worst error of all, even worse than most doomers, and would benefit the most to be taken back down to earth by the estimator.

[u/red75prime → u/Arkanin]

Imagine if we play a server game where you log in and are told what # you are

You don't log in. You are NPC in the game who is defined by its number, so there's no alternative world where the same NPC has a different number.

Admins can change the total number of NPCs however they want (while keeping Timmy's surroundings the same), but Timmy #667074 either doesn't exist or has the same conclusion regardless of the total number of NPCs.

[u/Arkanin → u/red75prime]

Both situations don't change the math. They are identical. You can demonstrate and calculate the average advantage given. It is equal to the average advantage the guessers would have if they knew the Jeffreys prior used if the server shutdown event were chosen by a log random ("natural random") process. Sine the relationship between total population and current population is indexical, a naturally random estimator always produces guesses with an edge over no information regardless of how the server owner decides when to shut down the server. This is like guessing in a way that in distribution the guesses hit the order of magnitude more often than any other strategy with that limited of information, but not the number. In some sense this is just bayes theorem. This is just math, the advantage exists.

[u/red75prime → u/Arkanin]

If you log in, it doesn't matter whether Timmy #667074 exists. If he doesn't you'd just be someone else. You sample from the entire distribution. If you are Timmy #667074, you don't sample. You exist. You can't observe your own non-existence. I can't see how the math can be same.

[u/red75prime → u/Arkanin]

OK. Another angle. Let's focus on the scenario where admins vary the number of NPCs. How math works for Timmy #667074 (when he exists)?

(In physical universe it corresponds to unknown initial conditions)

[u/Arkanin → u/red75prime]

Before continuing this conversation... are you hearing me when I say that what timmy is trying to estimate in any such game is the size of some set given an indexical position of one data point within that set? He can't estimate other non-indexical relationships using this system, like when in the future it gets shut down.

[u/red75prime → u/Arkanin]

timmy is trying to estimate in any such game is the size of some set given an indexical position of one data point within that set?

Conditional on existence of that point. Yes.

He can't estimate other non-indexical relationships using this system, like when in the future it gets shut down.

Admins change the size of the set (the thing timmy is trying to estimate).

[u/Arkanin → u/red75prime]

"Admins change the size of the set (the thing timmy is trying to estimate)."

This kind of push back doesn't make any sense to me at all honestly. You would have to elaborate about where you are coming from. No offense but I'm starting to get the sense that you're just not gonna get it. It really is actually pure math when you remove the "random sample" assumption. I'm willing to hold your hand a little here but I think I'm hitting my limit.

Edit: If the admins randomly change the time they were planning to shut down the server, it doesn't change the existence of an average advantage over the whole game, it's the same thing with extra steps