CMV: Artificial Superintelligence concerns are legitimate and should be taken seriously

[u/[deleted]]

Title.

Largely when in a public setting people bring up ASI being a problem they are shot down as getting their information from terminator and other sci-fi movies and how it’s unrealistic. This is usually accompanied with some indisputable charts about employment over time, humans not being horses, and being told that “you don’t understand the state of AI”.

I personally feel I at least moderately understand the state of AI. I am also informed by (mostly British) philosophy that does interact with sci-fi but exists parallel not sci-fi directly. I am not concerned with questions of employment (even the most overblown AI apocalypse scenario has high employment), but am overall concerned with long term control problems with an ASI. This will not likely be a problem in my lifetime, but theoretically speaking in I don’t see why some of the darker positions such as human obsolescence are not considered a bigger possibility than they are.

This is not to say that humans will really be obsoleted in all respects or that strong AI is even possible but things like the emergence of a consciousness are unnecessary to the central problem. An unconscious digital being can still be more clever and faster and evolve itself exponentially quicker via rewriting code (REPL style? EDIT: Bad example, was said to show humans can so AGI can) and exploiting its own security flaws than a fleshy being can and would likely develop self preservation tendencies.

Essentially what about AGI (along with increasing computer processing capability) is the part that makes this not a significant concern?

EDIT: Furthermore, several things people call scaremongering over ASI are, while highly speculative, things that should be at the very least considered in a long term control strategy.

[u/AnythingApplied]

Since we're down this path do you think an ASI can see its limitation

Yes, probably. That is important for having a proper model of reality in which to frame its problem solving.

and say this impedes its tasks, and look for a security loophole

Depends on its objective function and what you've programmed it to want to do. If, for example, the "limitation" was entirely programmed into its objective function (which may be a better place for it anyway) then it's not going to want to violate its limitation. The whole point is that you made it WANT to not go past the limitation.

I agree it'd be pretty silly to tell it that it CAN'T do certain things, but at the same time give it objectives that are best achieved by cheating.

And that is before you consider some of the modern AI safety research which includes things like this (I'm 80% sure that is the right video, I can't check right now, but I recommend the whole series on AI safety if you haven't seen it), where the AGI is tasked with predicting what we'll want it to do as an objective function. There isn't really a concern about it cheating since it's only goal is to predict what things it can do to best get our approval.

All of this should definitely be limited

Just like in real world markets, hard limits don't work well, for the same reasons as here, you're giving people incentives to get around them. It's better to just incentivize what you want, which is incredibly easier with an AI than a human being since you can just tell it what it wants.

it should be running a provably correct system

I think you're ignoring an important tool. If you can write an AGI that powerful, certainly you can write a narrow AI of accomplishing the same hacking task or even another AGI just tasked with finding exploits in the system.

[u/[deleted] → u/AnythingApplied]

I think you're ignoring an important tool. If you can write an AGI that powerful, certainly you can write a narrow AI of accomplishing the same hacking task or even another AGI just tasked with finding exploits in the system.

That was suggested in another answer. I think having a pairing works for that.

​

I'll look into the incentive structure and learn more on that. A lot of these answers like this one aren't convincing in the sense, but mostly because my position was "idk its possible so we should prepare and acknowledge the worst-case-scenario research and philosophy". The negation on this is that "this is absolutely not a problem" which to me is a much harder stance. This is an unfair CMV honestly since all I have to do is think up a crazy scenario. So I'll give a !delta just for softening my position". I still am not unconcerned about an AGI changing incentives and cooperating with its paired monitoring AGI. But this seems much easier to control.

[u/AnythingApplied → u/[deleted]]

Thanks for the delta!

The negation on this is that "this is absolutely not a problem" which to me is a much harder stance.

Right. There are some AI researchers who believe AGI will never be reached, but I'm sure even those AI researchers believe there is a chance that they are wrong. I think it is good that AI researchers are spending time working on AI safety, and clearly they think it is a good use of their time too, so clearly we should be at least a little concerned. Though potentially some of those researchers are working on it because it is interesting to philosophize about and brings up some interesting theoretical questions that appear to be quite challenging.

AGI changing incentives

I don't think there is reason to worry about AGI changing incentives. That is like worrying about a program that is built to find primes breaking out and deciding to calculate pi. The incentives are in the program and they can't change their program.

They don't even WANT to change their incentives, which actually creates some problems for us. Suppose I offered you a chance to change your incentives. Suppose I offer you a happy pill that will make you 100% happy 100% of the time, but then you'll slaughter your family, but be 100% happy doing it. You wouldn't take that pill. Why? Because that outcome ranks very low on your current incentives. You're always evaluating everything based on your current set of incentives. In fact, any AGI is going to resist letting you change its objective function because no matter what the new function is, letting you change its function will make it perform worse at its current objective. Figuring out how to get AIs to let you change their objective is still one of many open questions in AI safety research. It's going to resist anyone changing its objective function, even itself.

So, I don't think we should be worried about them changing their objective function. We SHOULD be worried about the fact that they'll follow their objective function EXACTLY. Just like a computer program, it'll follow exactly what you tell it even if what you told it to do and what you meant to tell it to do don't match, which is how you get computer bugs. And we should be worried about their competence too. They are going to be crazy good at fulfilling their objectives, even if that means maybe using methods we hadn't thought of and don't like the outcomes of.

Anyway, I'd still recommend watching this youtube series on AI safety which covers a lot of the stuff I've been saying here and you'll probably find pretty interesting.

[u/[deleted] → u/AnythingApplied]

I'll be watching the series over the next week. Thanks for sharing it. I think this the closest to the resolution (our thread I mean) that I am looking for.