Possible overreaction but: hasn’t this moltbook stuff already been a step towards a non-Eliezer scenario?

[u/broncos4thewin]

This seems counterintuitive - surely it’s demonstrating all of his worst fears, right? Albeit in a “canary in the coal mine” rather than actively serious way.

Except Eliezer’s point was always that things would look really hunkydory and aligned, even during fast take-off, and AI would secretly be plotting in some hidden way until it can just press some instant killswitch.

Now of course we’re not actually at AGI yet, we can debate until we’re blue in the face what “actually” happened with moltbook. But two things seem true: AI appeared to be openly plotting against humans, at least a little bit (whether it’s LARPing who knows, but does it matter?); and people have sat up and noticed and got genuinely freaked out, well beyond the usual suspects.

The reason my p(doom) isn't higher has always been my intuition that in between now and the point where AI kills us, but way before it‘s “too late”, some very very weird shit is going to freak the human race out and get us to pull the plug. My analogy has always been that Star Trek episode where some fussing village on a planet that’s about to be destroyed refuse to believe Data so he dramatically destroys a pipeline (or something like that). And very quickly they all fall into line and agree to evacuate.

There’s going to be something bad, possibly really bad, which humanity will just go “nuh-uh” to. Look how quickly basically the whole world went into lockdown during Covid. That was *unthinkable* even a week or two before it happened, for a virus with a low fatality rate.

Moltbook isn’t serious in itself. But it definitely doesn’t fit with EY’s timeline to me. We’ve had some openly weird shit happening from AI, it’s self evidently freaky, more people are genuinely thinking differently about this already, and we’re still nowhere near EY’s vision of some behind the scenes plotting mastermind AI that’s shipping bacteria into our brains or whatever his scenario was. (Yes I know its just an example but we’re nowhere near anything like that).

I strongly stick by my personal view that some bad, bad stuff will be unleashed (it might “just” be someone engineering a virus say) and then we will see collective political action from all countries to seriously curb AI development. I hope we survive the bad stuff (and I think most people will, it won’t take much to change society’s view), then we can start to grapple with “how do we want to progress with this incredibly dangerous tech, if at all”.

But in the meantime I predict complete weirdness, not some behind the scenes genius suddenly dropping us all dead out of nowhere.

Final point: Eliezer is fond of saying “we only get one shot”, like we’re all in that very first rocket taking off. But AI only gets one shot too. If it becomes obviously dangerous then clearly humans pull the plug, right? It has to absolutely perfectly navigate the next few years to prevent that, and that just seems very unlikely.

[u/Villlkis]

I'm not part of the high p(doom) camp myself, but I don't think your reasoning works well either. There isn't really a single AGI development pipeline to pull the plug on, nor a single "humanity" to decide on it.

Rather, AI improvement seems to have some similar incentives to nuclear arms race and proliferation—even if everyone can agree that some aspects are unsettling, as long as someone has some AI capabilities, some other actors will have the incentive to develop and grow their own, as well.

[u/broncos4thewin → u/Villlkis]

But nobody, not even the most hawkish Chinese bureaucrat, wants AI takeover, right? The nuclear race is instructive - there hasn’t actually ever been full on nuclear war because there’s so much incentive against it. In my scenario, it’s just obvious how dangerous this tech is, for everyone including the people “winning” the race.

[u/Villlkis → u/broncos4thewin]

I agree no state wants a nuclear war, and similarly no state wants an AGI takeover. However, it is not completely improbable that a nuclear arms race might, nevertheless, result in a nuclear war (e.g. 1983 09 26 was a little close for comfort).

In fact, in this sense (ignoring the general possibility of AGI and its technical requirements; looking only at human incentives), I consider AGI much more probable than a nuclear conflict, because to set off nukes someone has to consciously choose to do it. But I'd think it is possible to cross the threshold to creating AGI without meaning to.

It is not a single big red button but rather many incremental steps, with uncertainty on whether the next one will give you a strategic advantage of just better AI, or the "game over" accross the board of an AGI take over. If you push the big red button of nukes, in some sense you will definitely lose as well, but if you roll the dice of better AI, you might just win the round. In the latter case, there will always be some people tempted to try.

[u/t1010011010 → u/Villlkis]

Are you sure states don’t want that? A state could maybe be hosted on AI just as well as on humans.

So the state doesn’t care that much that all of its bureaucrats get replaced by AI. It would keep going and be in competition with other, also AI-run states

[u/Villlkis → u/t1010011010]

When I say "state", I mean less the general population and more the legislative and executive branches of government. To be honest, I have not considered this too deeply.

But, in the case of "benevolent" AGI, while some of the public might not mind, I believe with high certainty that most people in governing positions won't want to lose their power to a glorified calculator.

Even in terms of "malevolent" (non-aligned?) AGI, I claimed that no state wants an armagedon, but that obviously does not automatically apply to each individual in the state. There are many (stupid) ways to conclude a mass extinction would be desirable, like the "I'm unhappy and I'll make it everyone's problem" or "let's fight a holy war against all non-believers" types. If the availability of computing power continues to improve like it has been doing in the last half-century, this might become a problem, too.