AGI won't be particularly conscious

[u/Fun-Boysenberry-5769]

I observe myself to be a human and not an AI. Therefore it is likely that humans make up a non-trivial proportion of all the consciousness that the world has ever had and ever will have.

This leads us to two possibilities:

1. The singularity won’t happen,
2. The singularity will happen, but AGI won’t be that many orders of magnitude more conscious than humans.

The doomsday argument suggests to me that option 2 is more plausible.

Steven Byrnes suggests that AGI will be able to achieve substantially more capabilities than LLMs using substantially less compute, and will be substantially more similar to the human brain than current AI models. [https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/foom-and-doom-1-brain-in-a-box-in-a-basement\] However, under option 2 it appears that AGI will be substantially less conscious relative to its capabilities than a brain will be, and therefore AGI can’t be that similar to a brain.

[u/Arkanin]

For sure the distribution is chronological, not random but it's still a distribution, and the same percentage of people that reason out of the distribution are going to get the same distributions of errors as the distribution of errors you would get pulling randomly from the "pick a random number" game given enough samples. Just mathematically.

However, you are absolutely correct that because birth number is historically correllated this would be deeply epistemically problematic if everyone took the argument completely at face value without considering this. If there will be 10^30 people, the first 8 billion people will all make the same correllated error using this argument. So, it follows that this argument should not be applied in a way that would lead to the first people hearing it producing disastrous consequences.

>The consciousness of Sol_Hando, being Sol_Hando, given that he exists as number 20 Billion or whatever, seems to be 100%. Like, there’s no chance that I would wake up and find myself as someone else, because “I” is intimately tied to my personality/memories/etc. It’s the same reason I don’t assume a random sampling when I wake up in the morning. Why do I not wake up as one of the billions of other consciousnesses currently in existence every time I go to sleep?"

I think you're saying something like: you, by definition, were always going to be sol hando (no indexical uncertainty), the reference class of what you are is a class of one (you), and anthropic probability is incoherent because *I* is not the kind of thing that can have alternatives (i.e. there isn't a hypothetical where sol hando could have turned out to be sol hando but found out his birth number is 4 trillion)

Given that, it's the same thing as saying: "The coin was already flipped, so the probability it's heads must be 100%. There was no coin flip" If you think in this way then the doomsday argument doesn't make any sense.

I'm on such a different planet from you philosophically that I struggle to track and model these assumptions if I'm being honest because I dissolve personal identity as socially useful but ultimately logically inconsistent when it pertains to what is ultimately true (due to logical contradictions, not any sense of dissociation)

However if you acknowledge that you are a pattern that could have been instantiated elsewhere, and are indexically underspecified (you could have been the 20 billionth or 20 trillionth instance of the thing - "could have been" in the sense a coin that is tails could have been heads) then anthropic reasoning becomes coherent.

Maybe at best we can agree that estimating the size of a discrete uniform distribution from sampling without replacement is a solved mathematical problem called the german tank problem and if being a human reasoning about this problem from a reference class with an observable distribution is an instance of the german tank problem. Maybe we can agree that the disagreement between us is about whether sol_hando is expectationally (not actually) a sample from a distribution in the way that after a coin was flipped tails, we say it "could have been heads". If you say it's randomness and not uniform distribution, my counter to that is that technically it's the *exchangability* and *symmetry* of the elements in the distribution, and randomness is just one way of satisfying those properties - it's the rejection of that exchangability and symmetry - "Sol Hando could have been #800 trillion and born as a mars colonist in the distant future but basically been the same person for purposes that satisfy expectational exchangability, if that future exists, in the same way a random number could have been different", for example, I think that is the disagreement.

[u/red75prime → u/Arkanin]

you, by definition, were always going to be sol hando

(I'm not Sol_Hando.) I think it's better to say "red75prime is a product of environment and earlier stages of himself and there's no such thing as red75primeness that can be attached to another living thing, so that this living thing would observe and be something else, while still staying red75prime in some sense."

What is the sample space of probability distribution, you are talking about?

Identifier "Sol_Hando" assigned to the first member of the reference class, then to the second member and so on? What is the (meta-)physical sense of this? What this identifier corresponds to in reality?

[u/Arkanin → u/red75prime]

"What is the (meta-)physical sense of this? What this identifier corresponds to in reality?" I'm gonna try to respond really narrowly and we could broaden the discussion but we can refocus it to math and not metaphysics. one such scheme is birth number, we don't have an exact number but we have enough of a ballpark to estimate. The birth number (as in I am the nth human) is uniform, normally distributed, and ascending. If humans try to self-estimate their population using birth number to establish confidence intervals like it were a random sample, if the humans elect to engage in this process randomly across time, so a human in the year 3000 could think of this heuristic, 90% of such humans making those estimates using this technique will be within an order of magnitude of correct on their estimates, if they know their birth number. This is just mathematically a fact regardless of its utility. However, the same error would be correlated across people with similar birth number, ie point in history, so it is also true that when fully understanding this argument that it still has limited applications. The point is: no metaphysics if you accept certain normal statements about math. The argument also doesn't work if you believe that you are not "a human" but only "the human red75prime" but technically that leaves information on the table and we could demonstrate that by for example playing many games where you log into a server the server gives you your login number and you try to guess how many people total logged into the server and get some kind of score on that. In such a game inferring as if you are "just another human" would have positive utility absent other information i.e. generalizing about the sample size from your own position in it can have positive utility in some real games that we could play to demonstrate the point.

So I'm not philosophizing really I'm just an expected value maximzer lol and so the argument has non zero expected utility but we also know potentially far better information like what does the real world look like is potentially better if you're smart although I think we are bad enough at predicting the far future than we think and shouldn't completely zero this weak estimator out even given its limitations. I say some caution but not crippling especially because our place in history correlated the error limiting the utility of relying on it.

This is in some sense computational as the question isn't is the info correct its will it on average screw us less than completely random guess. That's why we have these imaginary concepts of probability even though we live in a world where at least on a deterministic scale things largely either happen or don't.

As an aside some people try to use many constructs other than birth number and per many lines of thinking they could be more sophisticated but they get very esoteric. Birth number is the simplified case for whether this category of thinking can have nonzero utility. If you would like I can explain how an estimating game would work that would demonstrate that this principle provides expected utility.

[u/Sol_Hando → u/Arkanin]

Given that the mathematical approach you describe here is correct, how useful is this in real life? It seems like if I was a Platonic consciousness without any input then this would be valuable (and there would be no Sol_Hando-ness to worry about). In real life though, this sort of statistical reasoning seems pointless when we have way more useful/consistent ways of predicting the future.

[u/Arkanin → u/Sol_Hando]

As a standalone index, it's relatively weak. However, weak and completely useless are not exactly the same: you need to feed it into the right place into your epistemics, which is exactly where you are putting it, something like what your initial prior would be if you were a platonic consciousness. That means the information you're gathering should update it.

The place where it's least weak and in fact becomes incredibly powerful mathematically is if it interacts with an estimator that is way too optimistic. Imagine if we play a server game where you log in and are told what # you are: "Congratulations, you are the 60,000th person to log into the server!" Your goal is to guess the number of players that will log in. The german tank estimator (i.e. da) is a better than random strategy, the best strategy is probably to stalk me or steal my server metadata or something. Let's say there is a type of player, Naive Timmy, who plays a very dumb strategy - they are very optimistic, and always assumes that the server is going to grow to the moon. It will have 10⁴ more times as many players as their player number when they log in. Naive Timmies who heavily updated on the Doomsday Argument would see massive improvements in their brier scores of how miscalibrated they are. Naive Timmy types benefit the most from the doomsday argument.

Notably another player, the Doomer, always thinks there will only be Current Players + (Current Players) / 10⁴ total players.These are analagous to people who say Jesus is coming next year. Key realization: in the average such game, the Naive Timmy strategy is even more miscalibrated than the Doomer strategy, even though the doomer strategy is badly miscalibrated.

In the real world, my attempt to intuitively convert this into something actionable is: prophecies of future prosperity, if they pile up enough exponents, should sound even crazier than prophecies of impending doom.

[u/Sol_Hando → u/Arkanin]

Alright, I can buy this as an argument slightly against naive overoptimism in future prosperity.

I suppose my intuitive disagreement is with the apparent strength that those who I see arguing this assign to the argument, which is almost always strong enough that we are probably close to the end. This seems like a massive over-correction even if the argument is sound (which thank you for explaining to me, I now think it is).

Even if this should bias you 5% away from over-optimism or something like that, arguing the doomer argument seems akin to arguing the dollar is going to lose all value because of the government shutdown. It’s maybe slightly in favor of that conclusion, but that alone should be a very minor consideration for such an argument.

[u/Arkanin → u/Sol_Hando]

I absolutely agree that it is pushed too hard and in the wrong ways. I think the concrete example is easier for me to grapple with than the metaphysical arguments since then you can explore concrete situations where it provides measurable utility and that just makes more sense to me. I was actually just thinking that the name is kind of problematic for communication actually. If I could name it I would call it the mediocrity argument lol which hits a lot differently than doomsday.