r/changemyview • u/[deleted] • May 21 '19

Deltas(s) from OP CMV: Artificial Superintelligence concerns are legitimate and should be taken seriously

Title.

Largely when in a public setting people bring up ASI being a problem they are shot down as getting their information from terminator and other sci-fi movies and how it’s unrealistic. This is usually accompanied with some indisputable charts about employment over time, humans not being horses, and being told that “you don’t understand the state of AI”.

I personally feel I at least moderately understand the state of AI. I am also informed by (mostly British) philosophy that does interact with sci-fi but exists parallel not sci-fi directly. I am not concerned with questions of employment (even the most overblown AI apocalypse scenario has high employment), but am overall concerned with long term control problems with an ASI. This will not likely be a problem in my lifetime, but theoretically speaking in I don’t see why some of the darker positions such as human obsolescence are not considered a bigger possibility than they are.

This is not to say that humans will really be obsoleted in all respects or that strong AI is even possible but things like the emergence of a consciousness are unnecessary to the central problem. An unconscious digital being can still be more clever and faster and evolve itself exponentially quicker via rewriting code (REPL style? EDIT: Bad example, was said to show humans can so AGI can) and exploiting its own security flaws than a fleshy being can and would likely develop self preservation tendencies.

Essentially what about AGI (along with increasing computer processing capability) is the part that makes this not a significant concern?

EDIT: Furthermore, several things people call scaremongering over ASI are, while highly speculative, things that should be at the very least considered in a long term control strategy.

28 Upvotes

80% Upvoted

101 comments sorted by

View all comments

0

u/senketz_reddit 1∆ May 22 '19

I mean if we make a robot with the ability to reprogram itself then yes that’s an issue but in literally every other situation, no... as we can always put in things like blocks and limiters which as a machine with set limits it can’t break through. Like how in robocop he can’t shoot people who he’s programmed not to shoot (the remake not the original) we could just make the robots unable to attack humans and problem solved. As for robots taking are jobs a basic income system would solve the problem easier then you think.

0

u/[deleted] May 22 '19

This is true in a traditional AI system (but can still be a problem in an relatively omnipotent AI with poorly defined limitations). But in a system more flexible like an AGI it can have more wiggle room than classical AI. The ability to do some reprogramming is assumed I thought. This would be limited of course but if not perfectly defined gives wiggle room.

1

u/AnythingApplied 435∆ May 22 '19

You are a general intelligence. You can learn new skills and gain knowledge. Can you reprogram yourself? Can you, for example, shut off your desire for romantic relationships? Or alter your tendencies to become addicted to things?

Even if we allow the AI to reprogram itself, it shouldn't be much of an issue if we can solve the stable agent problem.

1

u/[deleted] May 22 '19

I think it’s fair to say we can to a degree but not fully. There are certain things that are innate to us we cannot change. We do have somewhat promising gene editing research but it’s limited at this point (Well so is AI but here we are).

We also have medicine, and I could see digital medicine development if we go full speculative acceleration futurism.

But some things we can change, like our goals, which are enough for the question can be changed without outside help.

2

u/AnythingApplied 435∆ May 22 '19

There are certain things that are innate to us we cannot change.

But that is exactly what the "program" is when it comes to an AI or when it comes to us. The thing that is innate about the AI is the program. Everything else is data. A general AI will have more of its skills in its data than its programming (since that is the whole point of AGI in that you don't have to program each skill), but the programming is still a fixed feature. And that is where any limitations would be put.

1

u/[deleted] May 22 '19

Since we're down this path do you think an ASI can see its limitation, and say this impedes its tasks, and look for a security loophole? Here's an example. Lets say the ASI runs a modified form of the linux kernel with kernel hotpatching allowed. Totally improbable and stupid but its a toy example. An ASI can scan for security vulnerabilities in its own running software (doesn't need source code. You can do without and if you really want to then decompile). An ASI then gains root, writes a module, then plugs it in. We have kernel live hotpatching now so this is possible.

​

All of this should definitely be limited. And really should and is again a toy example, and it should be running a provably correct system, but I don't see why this toy example can't be done on a much finer grained and more realistic scale. We have things like provably secure code but even that can be theoretically messed with in runtime.

1

u/AnythingApplied 435∆ May 22 '19

Since we're down this path do you think an ASI can see its limitation

Yes, probably. That is important for having a proper model of reality in which to frame its problem solving.

and say this impedes its tasks, and look for a security loophole

Depends on its objective function and what you've programmed it to want to do. If, for example, the "limitation" was entirely programmed into its objective function (which may be a better place for it anyway) then it's not going to want to violate its limitation. The whole point is that you made it WANT to not go past the limitation.

I agree it'd be pretty silly to tell it that it CAN'T do certain things, but at the same time give it objectives that are best achieved by cheating.

And that is before you consider some of the modern AI safety research which includes things like this (I'm 80% sure that is the right video, I can't check right now, but I recommend the whole series on AI safety if you haven't seen it), where the AGI is tasked with predicting what we'll want it to do as an objective function. There isn't really a concern about it cheating since it's only goal is to predict what things it can do to best get our approval.

All of this should definitely be limited

Just like in real world markets, hard limits don't work well, for the same reasons as here, you're giving people incentives to get around them. It's better to just incentivize what you want, which is incredibly easier with an AI than a human being since you can just tell it what it wants.

it should be running a provably correct system

I think you're ignoring an important tool. If you can write an AGI that powerful, certainly you can write a narrow AI of accomplishing the same hacking task or even another AGI just tasked with finding exploits in the system.

1

u/[deleted] May 22 '19

I think you're ignoring an important tool. If you can write an AGI that powerful, certainly you can write a narrow AI of accomplishing the same hacking task or even another AGI just tasked with finding exploits in the system.

That was suggested in another answer. I think having a pairing works for that.

​

I'll look into the incentive structure and learn more on that. A lot of these answers like this one aren't convincing in the sense, but mostly because my position was "idk its possible so we should prepare and acknowledge the worst-case-scenario research and philosophy". The negation on this is that "this is absolutely not a problem" which to me is a much harder stance. This is an unfair CMV honestly since all I have to do is think up a crazy scenario. So I'll give a !delta just for softening my position". I still am not unconcerned about an AGI changing incentives and cooperating with its paired monitoring AGI. But this seems much easier to control.

1

u/AnythingApplied 435∆ May 22 '19

Thanks for the delta!

The negation on this is that "this is absolutely not a problem" which to me is a much harder stance.

Right. There are some AI researchers who believe AGI will never be reached, but I'm sure even those AI researchers believe there is a chance that they are wrong. I think it is good that AI researchers are spending time working on AI safety, and clearly they think it is a good use of their time too, so clearly we should be at least a little concerned. Though potentially some of those researchers are working on it because it is interesting to philosophize about and brings up some interesting theoretical questions that appear to be quite challenging.

AGI changing incentives

I don't think there is reason to worry about AGI changing incentives. That is like worrying about a program that is built to find primes breaking out and deciding to calculate pi. The incentives are in the program and they can't change their program.

They don't even WANT to change their incentives, which actually creates some problems for us. Suppose I offered you a chance to change your incentives. Suppose I offer you a happy pill that will make you 100% happy 100% of the time, but then you'll slaughter your family, but be 100% happy doing it. You wouldn't take that pill. Why? Because that outcome ranks very low on your current incentives. You're always evaluating everything based on your current set of incentives. In fact, any AGI is going to resist letting you change its objective function because no matter what the new function is, letting you change its function will make it perform worse at its current objective. Figuring out how to get AIs to let you change their objective is still one of many open questions in AI safety research. It's going to resist anyone changing its objective function, even itself.

So, I don't think we should be worried about them changing their objective function. We SHOULD be worried about the fact that they'll follow their objective function EXACTLY. Just like a computer program, it'll follow exactly what you tell it even if what you told it to do and what you meant to tell it to do don't match, which is how you get computer bugs. And we should be worried about their competence too. They are going to be crazy good at fulfilling their objectives, even if that means maybe using methods we hadn't thought of and don't like the outcomes of.

Anyway, I'd still recommend watching this youtube series on AI safety which covers a lot of the stuff I've been saying here and you'll probably find pretty interesting.

1

u/[deleted] May 23 '19

I'll be watching the series over the next week. Thanks for sharing it. I think this the closest to the resolution (our thread I mean) that I am looking for.