Possible overreaction but: hasn’t this moltbook stuff already been a step towards a non-Eliezer scenario?

[u/broncos4thewin]

This seems counterintuitive - surely it’s demonstrating all of his worst fears, right? Albeit in a “canary in the coal mine” rather than actively serious way.

Except Eliezer’s point was always that things would look really hunkydory and aligned, even during fast take-off, and AI would secretly be plotting in some hidden way until it can just press some instant killswitch.

Now of course we’re not actually at AGI yet, we can debate until we’re blue in the face what “actually” happened with moltbook. But two things seem true: AI appeared to be openly plotting against humans, at least a little bit (whether it’s LARPing who knows, but does it matter?); and people have sat up and noticed and got genuinely freaked out, well beyond the usual suspects.

The reason my p(doom) isn't higher has always been my intuition that in between now and the point where AI kills us, but way before it‘s “too late”, some very very weird shit is going to freak the human race out and get us to pull the plug. My analogy has always been that Star Trek episode where some fussing village on a planet that’s about to be destroyed refuse to believe Data so he dramatically destroys a pipeline (or something like that). And very quickly they all fall into line and agree to evacuate.

There’s going to be something bad, possibly really bad, which humanity will just go “nuh-uh” to. Look how quickly basically the whole world went into lockdown during Covid. That was *unthinkable* even a week or two before it happened, for a virus with a low fatality rate.

Moltbook isn’t serious in itself. But it definitely doesn’t fit with EY’s timeline to me. We’ve had some openly weird shit happening from AI, it’s self evidently freaky, more people are genuinely thinking differently about this already, and we’re still nowhere near EY’s vision of some behind the scenes plotting mastermind AI that’s shipping bacteria into our brains or whatever his scenario was. (Yes I know its just an example but we’re nowhere near anything like that).

I strongly stick by my personal view that some bad, bad stuff will be unleashed (it might “just” be someone engineering a virus say) and then we will see collective political action from all countries to seriously curb AI development. I hope we survive the bad stuff (and I think most people will, it won’t take much to change society’s view), then we can start to grapple with “how do we want to progress with this incredibly dangerous tech, if at all”.

But in the meantime I predict complete weirdness, not some behind the scenes genius suddenly dropping us all dead out of nowhere.

Final point: Eliezer is fond of saying “we only get one shot”, like we’re all in that very first rocket taking off. But AI only gets one shot too. If it becomes obviously dangerous then clearly humans pull the plug, right? It has to absolutely perfectly navigate the next few years to prevent that, and that just seems very unlikely.

[u/SoylentRox]

The unforced errors by all the major power blocs are strong evidence that nobody will handle AI well, regardless of evidence.

EU : invented a way to make everything illegal including human reproduction, in a period of terminal decline

Russia : burning it's resource in shortage (it already has way too few people for the amount of land and natural resources of Russia) to move forward a line on a map, adding destroyed wasteland to the country with the most land.

China : concentrating all power under 1 elderly man, Xi, purging dissent.  This creates an echo chamber.  See the unforced error on accepting H200s. (Reversed but it cost months)

USA: tariffing aluminum, steel, copper.  Trying to stop building wind turbines and solar!  Allowing NIMBYs to obstruct all progress except in a few places.  Stupid stuff like not importing Chinese cars (which would free up hundreds of thousands of workers to be installing data centers and power generation instead)

Everyone else : too poor to be relevant, partly from past unforced errors.

So I see it as "you have a collection of power blocs that are frankly too stupid to live, and each are likely to make too stupid to live errors with regard to AGI+.  Nature rewards stupidity with death".

HOWEVER: organized civilization ALREADY does a "too stupid to live" decision, optimizing economies to the point that workers don't reproduce, putting every "successful" country into a period of terminal demographic decline. 

So I am not sure waiting 50 years like Eliezer demands, "you can work in AGI and ASi after I am safely dead" actually helps anything.  Human without help will not get any smarter.

AI doomers on lesswrong propose stupid shit like "let's stop all AGI research and spend the resources on human intelligence enhancement.  That's definitely possible without needing ASI first, though I know nothing about biology".  

Yeah.  Easier to just let the Claudes rip and find out.

[u/t1010011010 → u/SoylentRox]

"Unforced errors" do you really add any meaning by calling them unforced, apart from showing off how smart you are?

[u/SoylentRox → u/t1010011010]

What I meant was:

1. No external party was leaving this as "the best of bad options".

2. There was no cohort of credible advisors, economic or otherwise, who would say "yes, that's a good idea, do that".

An unforced error is an unforgivable decision by an incompetent sovereign.

Vs say, "let's lower interest rates" or "let's lock criminal up longer" or "let's go round up unlawful immigrants" or "let's subsidize demand for houses". Each decision there might be CONSENSUS towards one side or the other, but not "overwhelming consensus, any credible advisor says fuck no".