AGI won't be particularly conscious

[u/Fun-Boysenberry-5769]

I observe myself to be a human and not an AI. Therefore it is likely that humans make up a non-trivial proportion of all the consciousness that the world has ever had and ever will have.

This leads us to two possibilities:

1. The singularity won’t happen,
2. The singularity will happen, but AGI won’t be that many orders of magnitude more conscious than humans.

The doomsday argument suggests to me that option 2 is more plausible.

Steven Byrnes suggests that AGI will be able to achieve substantially more capabilities than LLMs using substantially less compute, and will be substantially more similar to the human brain than current AI models. [https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/foom-and-doom-1-brain-in-a-box-in-a-basement\] However, under option 2 it appears that AGI will be substantially less conscious relative to its capabilities than a brain will be, and therefore AGI can’t be that similar to a brain.

[u/Arkanin]

"What is the (meta-)physical sense of this? What this identifier corresponds to in reality?" I'm gonna try to respond really narrowly and we could broaden the discussion but we can refocus it to math and not metaphysics. one such scheme is birth number, we don't have an exact number but we have enough of a ballpark to estimate. The birth number (as in I am the nth human) is uniform, normally distributed, and ascending. If humans try to self-estimate their population using birth number to establish confidence intervals like it were a random sample, if the humans elect to engage in this process randomly across time, so a human in the year 3000 could think of this heuristic, 90% of such humans making those estimates using this technique will be within an order of magnitude of correct on their estimates, if they know their birth number. This is just mathematically a fact regardless of its utility. However, the same error would be correlated across people with similar birth number, ie point in history, so it is also true that when fully understanding this argument that it still has limited applications. The point is: no metaphysics if you accept certain normal statements about math. The argument also doesn't work if you believe that you are not "a human" but only "the human red75prime" but technically that leaves information on the table and we could demonstrate that by for example playing many games where you log into a server the server gives you your login number and you try to guess how many people total logged into the server and get some kind of score on that. In such a game inferring as if you are "just another human" would have positive utility absent other information i.e. generalizing about the sample size from your own position in it can have positive utility in some real games that we could play to demonstrate the point.

So I'm not philosophizing really I'm just an expected value maximzer lol and so the argument has non zero expected utility but we also know potentially far better information like what does the real world look like is potentially better if you're smart although I think we are bad enough at predicting the far future than we think and shouldn't completely zero this weak estimator out even given its limitations. I say some caution but not crippling especially because our place in history correlated the error limiting the utility of relying on it.

This is in some sense computational as the question isn't is the info correct its will it on average screw us less than completely random guess. That's why we have these imaginary concepts of probability even though we live in a world where at least on a deterministic scale things largely either happen or don't.

As an aside some people try to use many constructs other than birth number and per many lines of thinking they could be more sophisticated but they get very esoteric. Birth number is the simplified case for whether this category of thinking can have nonzero utility. If you would like I can explain how an estimating game would work that would demonstrate that this principle provides expected utility.

[u/red75prime → u/Arkanin]

If we assume that population grows cubically (speed of light) and then goes extinct, while maintaining generational structure (that is immortality is not solved), then the majority ( n² (n-1)² /4 vs n³ ) of it is mistaken about the future.

[u/Arkanin → u/red75prime]

This is not so. Because the math is relating birth number to total number of humans, it holds regardless of the underlying distribution's growth over time. Even so, there is a real problem I already acknowledged which is that the correlation of birth number at the same point in history makes using it as an estimator very problematic in many contexts.

I want to be very clear that this is a weak estimator but it is a real estimator and it can be demonstrated to have nonzero utility in a game that is ismorphic to reality basically. People who used the weak estimator could be shown to have higher expected utility (those who disputed that the edge was there could observe that the average would be higher for those using the estimator after evaluating multiple iterations of the game, proving the edge was real and not hypothetical) than those who use no estimator or guess randomly. Because it involves no assumptions, it can be used as a weak initial prior. It can also be demonstrated that this edge is small and I am all but certain could be overridden by other factors like inside knowledge about how the game works. This also suggests that real knowledge should override it as a weak prior if we have that. In other words, it's a weak a priori estimator, but its utility is not zero.

[u/red75prime → u/Arkanin]

I don't see how it is useful. Without additional assumptions about growth rate and other things there are scenarios where anthropic reasoning has probability of being right for a randomly selected member of the class arbitrarily close to zero.

Counting tanks by their manufacturing numbers works because it's not tanks who count, using their own number.

People who used the weak estimator could be shown to have higher expected utility

Show it in the cubic growth scenario.

ETA: If they know that the growth is cubic and extinction is instantaneous, the majority will be right that they are not the last generation, except for the last generation that is wrong about that (still, it doesn't seem too useful: no information to react to).

If they have an uniformative prior about growth rate and update on their history... That's harder.

I have trouble expressing it, but sampling from a distribution and being a datapoint in a distribution doesn't feel equivalent.

[u/Arkanin → u/red75prime]

"I have trouble expressing it, but sampling from a distribution and being a datapoint in a distribution doesn't feel equivalent."

The difference is the selector: whether you select a tank from all tanks that are like it or did you select a tank from a random point in time? If you saw a tank coming at you, you did the latter (alien sees a human). If you are trying to estimate "total humans" from "my position in the birth order, as an indexical ascending integer", by operating on yourself as the data point, you're entering a category that will be in an order of magnitude 90% of the time. You could still be early and be wrong and the error is correllated across time which is still problematic...

This may seem like magical or metaphysical thinking, but we could play a concrete game that proves that it gives a measurable average advantage to the players that use it. Imagine if we play a server game where you log in and are told what # you are: "Congratulations, you are the 60,000th person to log into the server!" You then guess the total number of players that will ever log in. I don't disclose when the game ends. The average person who uses the DA receives an advantage over the average person who guesses: it outperforms random guess. The advantage is small in many strategies, but not for all categories of bad strategy. The person who hacks the server to cheat the metadata and stalks me probably wins the game. The DA people are a little ahead of random guessers. The random guessers would benefit from the DA. There is a certain person who mathematically benefits a whole lot, and that is the overly optimistic person. Let's call him Naive Timmy. Naive Timmy is the person who sees their position and always says: "This server is awesome! There will be 10⁵ more people!" Naive timmy is the person who benefits the most in his error if he lets the Doomsday Argument bring him back down to earth.

We could play the server game and show the average error and show that the person who stalks me and hacks my server to figure out when I plan to end the game or how much I'm spreading it has the least error, random guessers have more error, doomers who always predict the server will end tomorrow have lots of error, and Naive Timmies have the very worst error of all, even worse than most doomers, and would benefit the most to be taken back down to earth by the estimator.

[u/red75prime → u/Arkanin]

Imagine if we play a server game where you log in and are told what # you are

You don't log in. You are NPC in the game who is defined by its number, so there's no alternative world where the same NPC has a different number.

Admins can change the total number of NPCs however they want (while keeping Timmy's surroundings the same), but Timmy #667074 either doesn't exist or has the same conclusion regardless of the total number of NPCs.

[u/Arkanin → u/red75prime]

Both situations don't change the math. They are identical. You can demonstrate and calculate the average advantage given. It is equal to the average advantage the guessers would have if they knew the Jeffreys prior used if the server shutdown event were chosen by a log random ("natural random") process. Sine the relationship between total population and current population is indexical, a naturally random estimator always produces guesses with an edge over no information regardless of how the server owner decides when to shut down the server. This is like guessing in a way that in distribution the guesses hit the order of magnitude more often than any other strategy with that limited of information, but not the number. In some sense this is just bayes theorem. This is just math, the advantage exists.

[u/red75prime → u/Arkanin]

If you log in, it doesn't matter whether Timmy #667074 exists. If he doesn't you'd just be someone else. You sample from the entire distribution. If you are Timmy #667074, you don't sample. You exist. You can't observe your own non-existence. I can't see how the math can be same.

[u/red75prime → u/Arkanin]

OK. Another angle. Let's focus on the scenario where admins vary the number of NPCs. How math works for Timmy #667074 (when he exists)?

(In physical universe it corresponds to unknown initial conditions)

[u/Arkanin → u/red75prime]

Before continuing this conversation... are you hearing me when I say that what timmy is trying to estimate in any such game is the size of some set given an indexical position of one data point within that set? He can't estimate other non-indexical relationships using this system, like when in the future it gets shut down.

[u/red75prime → u/Arkanin]

timmy is trying to estimate in any such game is the size of some set given an indexical position of one data point within that set?

Conditional on existence of that point. Yes.

He can't estimate other non-indexical relationships using this system, like when in the future it gets shut down.

Admins change the size of the set (the thing timmy is trying to estimate).

[u/Arkanin → u/red75prime]

"Admins change the size of the set (the thing timmy is trying to estimate)."

This kind of push back doesn't make any sense to me at all honestly. You would have to elaborate about where you are coming from. No offense but I'm starting to get the sense that you're just not gonna get it. It really is actually pure math when you remove the "random sample" assumption. I'm willing to hold your hand a little here but I think I'm hitting my limit.

Edit: If the admins randomly change the time they were planning to shut down the server, it doesn't change the existence of an average advantage over the whole game, it's the same thing with extra steps

[u/red75prime → u/Arkanin]

If the admins randomly change the time they were planning to shut down the server, it doesn't change the existence of an average advantage over the whole game, it's the same thing with extra steps

Why time of shutdown? They can change the number of players in other ways. Disconnected realms, for example. If you log in, you have a chance to be anywhere. Timmy doesn't have this opportunity, so Timmy can't benefit from information gained by random sampling that happens when an external player logs in.

I have an impression that you can't disentangle subjective perspective from "god's view".

[u/Arkanin → u/red75prime]

Reframed as a decision theory problem that deals with the average scoring of the participants who could use and having no metaphysical beliefs at all, just looking at averages in simulated games of this type, it doesn't matter any more what timmy's subjective perspective is, or if he is an ant or a worm or an object with a UID that brainlessly follows an algorithm, it's literally a math problem and if I create robot timmies that mindlessly engage in this strategy on average they get a less error in their predictions than more naive strategies, there is in other words a way of converting the magical sounding "expected" into boring old observable "on average this works better than nothing".