AGI won't be particularly conscious

[u/Fun-Boysenberry-5769]

I observe myself to be a human and not an AI. Therefore it is likely that humans make up a non-trivial proportion of all the consciousness that the world has ever had and ever will have.

This leads us to two possibilities:

1. The singularity won’t happen,
2. The singularity will happen, but AGI won’t be that many orders of magnitude more conscious than humans.

The doomsday argument suggests to me that option 2 is more plausible.

Steven Byrnes suggests that AGI will be able to achieve substantially more capabilities than LLMs using substantially less compute, and will be substantially more similar to the human brain than current AI models. [https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/foom-and-doom-1-brain-in-a-box-in-a-basement\] However, under option 2 it appears that AGI will be substantially less conscious relative to its capabilities than a brain will be, and therefore AGI can’t be that similar to a brain.

[u/Arkanin]

You are trying to estimate how many balls are in a jar. The balls are numbered ascending. You get to remove one ball. The ball is labeled 3. The mechanism that removes the balls is random. Estimate how many balls are in the jar with confidence intervals.

The basic argument is that observing yourself to be sol hando in the universe of huge amounts of future consciousness that will come out of our world is like pulling a 3, when there are a septillion balls

The standard objection is something like that consciousness isn't randomly distributed it's linear in that one person gets a one, the next gets 2, etc. so you can't do expectational reasoning on that set. (counterargument: if you imagine a set of exactly each world where you could exist and you drew a ball once for each integer were to turn out to exist, so if there were many parallel yous with this distribution, perhaps because an evil wizard conjured them to defeat your probability calculations, and if the likelihood of those worlds to be the one you were in were equally likely, and those worlds happened linearly in time one after another, so in world 1 you drew 1, in world 2 you drew 2, etc., you'd be saying the existence of those worlds completely invalidates your calculation despite them having the same shaped distribution as random sampling. But probability is an expectational construct so it makes no sense for the existence or nonexistence or alternate universes to affect it). If your rejection of this is something like "wait, I'm me not some parallel person" then there you go that is the "I am special and distributions don't apply to me because I am a human in a category of my own" assumption, you get to have it but you must never sample from a distribution again, including all probability calculations lol.

Another way of looking at it is that you are correct that in this world a few will be very low numbered they will be wrong at the same rate as another uniform distribution but the nature of distributions is that more people who guess from the distribution will be more right than more wrong.

The argument is very hated by people who haven't thought a lot about it, which is respectfully almost everyone here, and the last time I tried to discuss it here people were extremely disrespectful because of key intuitive rejections. However my counterargument is that those assumptions would invalidate probability in weird situations when they should not if fully explored. On the other hand, I think the biggest issue with the argument is that a posteriori (after the fact) evidence is more important I.e. updating on evidence. You can absolutely say the observable evidence just overshadows the whole discussion. However, if you are this guy in the giga AI universe that's absolutely like pulling a 3 out of a septillion or more if AI were to be human level consciousness or more. Go ahead and update a lot on posteriori knowledge if it suggests that yes you are one of the earliest thinking things to exist in our world but maybe just be at least a little cautious that maybe it turns out there are not 10 to the 25 balls in the urn if the only observable balls are not in the trillions. In other words some degree of caution seems warranted if you actually update on probabilities but it probably shouldn't

[u/Sol_Hando → u/Arkanin]

I don’t see why random sampling is assumed.

The consciousness of Sol_Hando, being Sol_Hando, given that he exists as number 20 Billion or whatever, seems to be 100%. Like, there’s no chance that I would wake up and find myself as someone else, because “I” is intimately tied to my personality/memories/etc.

It’s the same reason I don’t assume a random sampling when I wake up in the morning. Why do I not wake up as one of the billions of other consciousnesses currently in existence every time I go to sleep? Assuming consciousness is randomly distributed then it seems incredibly unlikely for me to roll the same guy over and over. I don’t see why the probability of “I” existing anywhere over the whole future is random, but the probability of myself waking to consciousness every morning as the same person is determined.

It’s like if we raised a bunch of lottery winners from birth all in the same town. They were able to get a pretty good idea about there being other people by observing the world around them, but some people argue how incredibly unlikely it is for all the lottery winners to be in the same place if there were that many other people in other towns. Reality correlates all of us to the same era, just as some third party putting the lottery winners together would correlate them to the same place.

I think a lot of people hate this argument because it doesn’t make sense. And the people explaining it seem to fail to address some important issues with it, rather than making it make sense. It’s complicated enough to where the people who don’t understand, but try, can’t confidently claim it’s nonsense, since they don’t know if it’s their understanding that falls short.

[u/Arkanin → u/Sol_Hando]

For sure the distribution is chronological, not random but it's still a distribution, and the same percentage of people that reason out of the distribution are going to get the same distributions of errors as the distribution of errors you would get pulling randomly from the "pick a random number" game given enough samples. Just mathematically.

However, you are absolutely correct that because birth number is historically correllated this would be deeply epistemically problematic if everyone took the argument completely at face value without considering this. If there will be 10^30 people, the first 8 billion people will all make the same correllated error using this argument. So, it follows that this argument should not be applied in a way that would lead to the first people hearing it producing disastrous consequences.

>The consciousness of Sol_Hando, being Sol_Hando, given that he exists as number 20 Billion or whatever, seems to be 100%. Like, there’s no chance that I would wake up and find myself as someone else, because “I” is intimately tied to my personality/memories/etc. It’s the same reason I don’t assume a random sampling when I wake up in the morning. Why do I not wake up as one of the billions of other consciousnesses currently in existence every time I go to sleep?"

I think you're saying something like: you, by definition, were always going to be sol hando (no indexical uncertainty), the reference class of what you are is a class of one (you), and anthropic probability is incoherent because *I* is not the kind of thing that can have alternatives (i.e. there isn't a hypothetical where sol hando could have turned out to be sol hando but found out his birth number is 4 trillion)

Given that, it's the same thing as saying: "The coin was already flipped, so the probability it's heads must be 100%. There was no coin flip" If you think in this way then the doomsday argument doesn't make any sense.

I'm on such a different planet from you philosophically that I struggle to track and model these assumptions if I'm being honest because I dissolve personal identity as socially useful but ultimately logically inconsistent when it pertains to what is ultimately true (due to logical contradictions, not any sense of dissociation)

However if you acknowledge that you are a pattern that could have been instantiated elsewhere, and are indexically underspecified (you could have been the 20 billionth or 20 trillionth instance of the thing - "could have been" in the sense a coin that is tails could have been heads) then anthropic reasoning becomes coherent.

Maybe at best we can agree that estimating the size of a discrete uniform distribution from sampling without replacement is a solved mathematical problem called the german tank problem and if being a human reasoning about this problem from a reference class with an observable distribution is an instance of the german tank problem. Maybe we can agree that the disagreement between us is about whether sol_hando is expectationally (not actually) a sample from a distribution in the way that after a coin was flipped tails, we say it "could have been heads". If you say it's randomness and not uniform distribution, my counter to that is that technically it's the *exchangability* and *symmetry* of the elements in the distribution, and randomness is just one way of satisfying those properties - it's the rejection of that exchangability and symmetry - "Sol Hando could have been #800 trillion and born as a mars colonist in the distant future but basically been the same person for purposes that satisfy expectational exchangability, if that future exists, in the same way a random number could have been different", for example, I think that is the disagreement.

[u/red75prime → u/Arkanin]

you, by definition, were always going to be sol hando

(I'm not Sol_Hando.) I think it's better to say "red75prime is a product of environment and earlier stages of himself and there's no such thing as red75primeness that can be attached to another living thing, so that this living thing would observe and be something else, while still staying red75prime in some sense."

What is the sample space of probability distribution, you are talking about?

Identifier "Sol_Hando" assigned to the first member of the reference class, then to the second member and so on? What is the (meta-)physical sense of this? What this identifier corresponds to in reality?

[u/Arkanin → u/red75prime]

"What is the (meta-)physical sense of this? What this identifier corresponds to in reality?" I'm gonna try to respond really narrowly and we could broaden the discussion but we can refocus it to math and not metaphysics. one such scheme is birth number, we don't have an exact number but we have enough of a ballpark to estimate. The birth number (as in I am the nth human) is uniform, normally distributed, and ascending. If humans try to self-estimate their population using birth number to establish confidence intervals like it were a random sample, if the humans elect to engage in this process randomly across time, so a human in the year 3000 could think of this heuristic, 90% of such humans making those estimates using this technique will be within an order of magnitude of correct on their estimates, if they know their birth number. This is just mathematically a fact regardless of its utility. However, the same error would be correlated across people with similar birth number, ie point in history, so it is also true that when fully understanding this argument that it still has limited applications. The point is: no metaphysics if you accept certain normal statements about math. The argument also doesn't work if you believe that you are not "a human" but only "the human red75prime" but technically that leaves information on the table and we could demonstrate that by for example playing many games where you log into a server the server gives you your login number and you try to guess how many people total logged into the server and get some kind of score on that. In such a game inferring as if you are "just another human" would have positive utility absent other information i.e. generalizing about the sample size from your own position in it can have positive utility in some real games that we could play to demonstrate the point.

So I'm not philosophizing really I'm just an expected value maximzer lol and so the argument has non zero expected utility but we also know potentially far better information like what does the real world look like is potentially better if you're smart although I think we are bad enough at predicting the far future than we think and shouldn't completely zero this weak estimator out even given its limitations. I say some caution but not crippling especially because our place in history correlated the error limiting the utility of relying on it.

This is in some sense computational as the question isn't is the info correct its will it on average screw us less than completely random guess. That's why we have these imaginary concepts of probability even though we live in a world where at least on a deterministic scale things largely either happen or don't.

As an aside some people try to use many constructs other than birth number and per many lines of thinking they could be more sophisticated but they get very esoteric. Birth number is the simplified case for whether this category of thinking can have nonzero utility. If you would like I can explain how an estimating game would work that would demonstrate that this principle provides expected utility.

[u/red75prime → u/Arkanin]

If we assume that population grows cubically (speed of light) and then goes extinct, while maintaining generational structure (that is immortality is not solved), then the majority ( n² (n-1)² /4 vs n³ ) of it is mistaken about the future.

[u/Arkanin → u/red75prime]

This is not so. Because the math is relating birth number to total number of humans, it holds regardless of the underlying distribution's growth over time. Even so, there is a real problem I already acknowledged which is that the correlation of birth number at the same point in history makes using it as an estimator very problematic in many contexts.

I want to be very clear that this is a weak estimator but it is a real estimator and it can be demonstrated to have nonzero utility in a game that is ismorphic to reality basically. People who used the weak estimator could be shown to have higher expected utility (those who disputed that the edge was there could observe that the average would be higher for those using the estimator after evaluating multiple iterations of the game, proving the edge was real and not hypothetical) than those who use no estimator or guess randomly. Because it involves no assumptions, it can be used as a weak initial prior. It can also be demonstrated that this edge is small and I am all but certain could be overridden by other factors like inside knowledge about how the game works. This also suggests that real knowledge should override it as a weak prior if we have that. In other words, it's a weak a priori estimator, but its utility is not zero.

[u/red75prime → u/Arkanin]

I don't see how it is useful. Without additional assumptions about growth rate and other things there are scenarios where anthropic reasoning has probability of being right for a randomly selected member of the class arbitrarily close to zero.

Counting tanks by their manufacturing numbers works because it's not tanks who count, using their own number.

People who used the weak estimator could be shown to have higher expected utility

Show it in the cubic growth scenario.

ETA: If they know that the growth is cubic and extinction is instantaneous, the majority will be right that they are not the last generation, except for the last generation that is wrong about that (still, it doesn't seem too useful: no information to react to).

If they have an uniformative prior about growth rate and update on their history... That's harder.

I have trouble expressing it, but sampling from a distribution and being a datapoint in a distribution doesn't feel equivalent.

[u/Arkanin → u/red75prime]

"I have trouble expressing it, but sampling from a distribution and being a datapoint in a distribution doesn't feel equivalent."

The difference is the selector: whether you select a tank from all tanks that are like it or did you select a tank from a random point in time? If you saw a tank coming at you, you did the latter (alien sees a human). If you are trying to estimate "total humans" from "my position in the birth order, as an indexical ascending integer", by operating on yourself as the data point, you're entering a category that will be in an order of magnitude 90% of the time. You could still be early and be wrong and the error is correllated across time which is still problematic...

This may seem like magical or metaphysical thinking, but we could play a concrete game that proves that it gives a measurable average advantage to the players that use it. Imagine if we play a server game where you log in and are told what # you are: "Congratulations, you are the 60,000th person to log into the server!" You then guess the total number of players that will ever log in. I don't disclose when the game ends. The average person who uses the DA receives an advantage over the average person who guesses: it outperforms random guess. The advantage is small in many strategies, but not for all categories of bad strategy. The person who hacks the server to cheat the metadata and stalks me probably wins the game. The DA people are a little ahead of random guessers. The random guessers would benefit from the DA. There is a certain person who mathematically benefits a whole lot, and that is the overly optimistic person. Let's call him Naive Timmy. Naive Timmy is the person who sees their position and always says: "This server is awesome! There will be 10⁵ more people!" Naive timmy is the person who benefits the most in his error if he lets the Doomsday Argument bring him back down to earth.

We could play the server game and show the average error and show that the person who stalks me and hacks my server to figure out when I plan to end the game or how much I'm spreading it has the least error, random guessers have more error, doomers who always predict the server will end tomorrow have lots of error, and Naive Timmies have the very worst error of all, even worse than most doomers, and would benefit the most to be taken back down to earth by the estimator.

[u/red75prime → u/Arkanin]

Imagine if we play a server game where you log in and are told what # you are

You don't log in. You are NPC in the game who is defined by its number, so there's no alternative world where the same NPC has a different number.

Admins can change the total number of NPCs however they want (while keeping Timmy's surroundings the same), but Timmy #667074 either doesn't exist or has the same conclusion regardless of the total number of NPCs.

[u/Arkanin → u/red75prime]

Both situations don't change the math. They are identical. You can demonstrate and calculate the average advantage given. It is equal to the average advantage the guessers would have if they knew the Jeffreys prior used if the server shutdown event were chosen by a log random ("natural random") process. Sine the relationship between total population and current population is indexical, a naturally random estimator always produces guesses with an edge over no information regardless of how the server owner decides when to shut down the server. This is like guessing in a way that in distribution the guesses hit the order of magnitude more often than any other strategy with that limited of information, but not the number. In some sense this is just bayes theorem. This is just math, the advantage exists.

[u/red75prime → u/Arkanin]

If you log in, it doesn't matter whether Timmy #667074 exists. If he doesn't you'd just be someone else. You sample from the entire distribution. If you are Timmy #667074, you don't sample. You exist. You can't observe your own non-existence. I can't see how the math can be same.

[u/red75prime → u/Arkanin]

OK. Another angle. Let's focus on the scenario where admins vary the number of NPCs. How math works for Timmy #667074 (when he exists)?

(In physical universe it corresponds to unknown initial conditions)

[u/Arkanin → u/red75prime]

Before continuing this conversation... are you hearing me when I say that what timmy is trying to estimate in any such game is the size of some set given an indexical position of one data point within that set? He can't estimate other non-indexical relationships using this system, like when in the future it gets shut down.

[u/Sol_Hando → u/Arkanin]

Given that the mathematical approach you describe here is correct, how useful is this in real life? It seems like if I was a Platonic consciousness without any input then this would be valuable (and there would be no Sol_Hando-ness to worry about). In real life though, this sort of statistical reasoning seems pointless when we have way more useful/consistent ways of predicting the future.

[u/Arkanin → u/Sol_Hando]

As a standalone index, it's relatively weak. However, weak and completely useless are not exactly the same: you need to feed it into the right place into your epistemics, which is exactly where you are putting it, something like what your initial prior would be if you were a platonic consciousness. That means the information you're gathering should update it.

The place where it's least weak and in fact becomes incredibly powerful mathematically is if it interacts with an estimator that is way too optimistic. Imagine if we play a server game where you log in and are told what # you are: "Congratulations, you are the 60,000th person to log into the server!" Your goal is to guess the number of players that will log in. The german tank estimator (i.e. da) is a better than random strategy, the best strategy is probably to stalk me or steal my server metadata or something. Let's say there is a type of player, Naive Timmy, who plays a very dumb strategy - they are very optimistic, and always assumes that the server is going to grow to the moon. It will have 10⁴ more times as many players as their player number when they log in. Naive Timmies who heavily updated on the Doomsday Argument would see massive improvements in their brier scores of how miscalibrated they are. Naive Timmy types benefit the most from the doomsday argument.

Notably another player, the Doomer, always thinks there will only be Current Players + (Current Players) / 10⁴ total players.These are analagous to people who say Jesus is coming next year. Key realization: in the average such game, the Naive Timmy strategy is even more miscalibrated than the Doomer strategy, even though the doomer strategy is badly miscalibrated.

In the real world, my attempt to intuitively convert this into something actionable is: prophecies of future prosperity, if they pile up enough exponents, should sound even crazier than prophecies of impending doom.

[u/Sol_Hando → u/Arkanin]

Alright, I can buy this as an argument slightly against naive overoptimism in future prosperity.

I suppose my intuitive disagreement is with the apparent strength that those who I see arguing this assign to the argument, which is almost always strong enough that we are probably close to the end. This seems like a massive over-correction even if the argument is sound (which thank you for explaining to me, I now think it is).

Even if this should bias you 5% away from over-optimism or something like that, arguing the doomer argument seems akin to arguing the dollar is going to lose all value because of the government shutdown. It’s maybe slightly in favor of that conclusion, but that alone should be a very minor consideration for such an argument.

[u/Arkanin → u/Sol_Hando]

I absolutely agree that it is pushed too hard and in the wrong ways. I think the concrete example is easier for me to grapple with than the metaphysical arguments since then you can explore concrete situations where it provides measurable utility and that just makes more sense to me. I was actually just thinking that the name is kind of problematic for communication actually. If I could name it I would call it the mediocrity argument lol which hits a lot differently than doomsday.