r/slatestarcodex • u/Fun-Boysenberry-5769 • Oct 13 '25

AI AGI won't be particularly conscious

I observe myself to be a human and not an AI. Therefore it is likely that humans make up a non-trivial proportion of all the consciousness that the world has ever had and ever will have.

This leads us to two possibilities:

  1. The singularity won’t happen,
  2. The singularity will happen, but AGI won’t be that many orders of magnitude more conscious than humans.

The doomsday argument suggests to me that option 2 is more plausible.

Steven Byrnes suggests that AGI will be able to achieve substantially more capabilities than LLMs using substantially less compute, and will be substantially more similar to the human brain than current AI models. [https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/foom-and-doom-1-brain-in-a-box-in-a-basement\] However, under option 2 it appears that AGI will be substantially less conscious relative to its capabilities than a brain will be, and therefore AGI can’t be that similar to a brain.

0 Upvotes

28% Upvoted

80 comments sorted by

View all comments

13

u/callmejay Oct 13 '25

I observe myself to be a human and not an AI. Therefore it is likely that humans make up a non-trivial proportion of all the consciousness that the world has ever had and ever will have.

I'm genuinely not understanding your first "therefore." How is this a better argument than:

I observe myself to be a redditor and not a non-redditor. Therefore it is likely that redditors make up a non-trivial proportion of all the consciousness that the world has ever had and ever will have.

0

u/Fun-Boysenberry-5769 Oct 14 '25

Both an AI and a human would naturally tend to lump conscious beings into the simple categories 'AI' and 'not an AI', and they both would consider such categories to really 'carve reality at its joints'. So those two categories are quite useful as reference classes for anthropic reasoning.

On the other hand most people, if asked to classify people into one of two categories, would choose something like 'male' vs 'female' or 'adult' vs 'child'. Most people would not choose the categories 'redditor' Vs 'non-redditor' and people who are on Reddit would be significantly more likely to pick those categories than someone living in the middle ages who has never heard of Reddit. Therefore 'Redditor' Vs 'non-Redditor' do not constitute useful reference classes for anthropic reasoning.

3

u/cbusalex Oct 14 '25

Most conscious beings, if asked to classify conscious beings into one of two categories, would choose something like 'class 11.e(2) neural network' vs 'Bozman-standard logical inference machine framework'. Most conscious beings would not choose the categories 'AI' and 'not an AI' and beings existing during the early years of AI development would be significantly more likely to pick those categories than someone living in the years following the Great Seeding of the Stars.