You have no idea what you're talking about. AI is a black box right now, we understand how it works but we can't properly control it and it still does a lot of unintentional behavior, like how chatbots can sometimes be aggressive or insult you. Chatbots like GPT try getting around this by having a million filters but the point is that the underlying AI doesn't behave properly. Mix that with superintelligence and you can have an AI that does random things based on what it felt like doing. This is dangerous. We're not asking to stop AI development, we're asking to do it more responsibly and follow proper AI ethics which a lot of companies seem to start ignoring in favor of pushing out products faster.

Author is a simple brainless student doing some part-time job to write about bullshit and make a living.

I have a pretty solid opinion of Eliezer Yudkowsky. I've read material that he's written in the past, and he's not bullshitting in that; it's well-thought through.

Or just let’s do nothing until Iran creates AI powered soldiers?

I haven't watched the current video, but from what I've read from him in the past, Yudkowsky isn't an opponent of developing AI. He's pointing out that there are serious risks that need addressing.

It's not as if there are two camps regarding AI, one "everything is perfect" utopian and the other Luddite and "we should avoid AI".

EDIT: Okay, I went through the video. That's certainly a lot blunter than he normally is. He's advocating for a global ban on developing specifically superintelligent AI until we do have consensus on dealing with it and monitoring AI development in the meantime; he's talking about countries being willing to go to war with countries that are developing them, so his answer would be "if Iran is working on a superintelligent AI, you bomb them preemptively".

EDIT2:

Humans create AI - humans design how AI is created and what AI can do.

The major point that Yudkowsky has raised in his past work is that it is likely quite difficult to constrain what AI can do.

Just because we developed an AI does not mean that it is trivial for us to place constraints on it that will hold as it evolves, as we will not be able to understand the systems that we will be trying to constrain.

Last week, lemmy had a serious security exploit involving cross-site scripting. The authors of that software wrote (or at least committed) the code in question. Sure, in theory, if they had perfect understanding of all of the implications of every action that they took, they would not have introduced that security hole -- but they didn't. Just being the author doesn't mean that the software necessarily does what they intend, because even today, translating intent to functionality is not easy.

A self-improving AI is going to be something that we will be very far-removed from in terms of how it ultimately winds up operating; it will be much more-complex than a human is.

Programmers do create, say, software that has bugs. An infinite loop, or software that allocates all memory on a computer today. The systems today (mostly) operated in constrained environments, where they are easy to kill off. If you look at, say, DARPA's autonomous vehicles challenges, where that is not the case, the robots are required to have an emergency stop button that permits them to be killed remotely in case they start doing something dangerous.

But a superintelligent AI would likely not be something that is easy to contain or constrain. If it decides that an emergency stop button is in conflict with its own goals and understands that emergency stop button, it is not at all clear that we have the ability to keep it from defeating such a mechanism -- or to keep it from manipulating us into doing so. And the damage that a self-replicating/self-improving AI could potentially do is at least potentially much greater than what an DARPA-style out-of-control autonomous armored vehicle could do. The vehicle might run over a few dozen people before it runs out of fuel, but its nature limits degree to which it can go wrong.

We didn't have an easy time purging the Morris Internet Worm back in 1988, because our immediate responses, cutting the links that sites had to the Internet to block more instances of the worm from hitting their systems from the Internet, crippled our own infrastructure. That took mailing lists offline and took down Usenet and finger -- which was used by sysadmins to communicate network status and to find out how to contact other people via the phone system -- and that was a simple worm in an era much-less dependent on the Internet. It wasn't self-improving or intelligent, and its author even tried -- without much success, as we'd already had a lot of infrastructure go down -- to tell people how to disable it some hours after it started taking the Internet out.

I am not terribly sanguine on our ability to effectively deal with a system that is that plus a whole lot more.

Lmao do we have an equivalent of /r/confidentlyincorrect on Lemmy?

Yudkowsky has a background in this, purely aside from likely being smarter than any five of us put together. Do let us all know how you're qualified to call him a student, let alone a brainless one.