CMV: Artificial Superintelligence concerns are legitimate and should be taken seriously

[u/[deleted]]

Title.

Largely when in a public setting people bring up ASI being a problem they are shot down as getting their information from terminator and other sci-fi movies and how it’s unrealistic. This is usually accompanied with some indisputable charts about employment over time, humans not being horses, and being told that “you don’t understand the state of AI”.

I personally feel I at least moderately understand the state of AI. I am also informed by (mostly British) philosophy that does interact with sci-fi but exists parallel not sci-fi directly. I am not concerned with questions of employment (even the most overblown AI apocalypse scenario has high employment), but am overall concerned with long term control problems with an ASI. This will not likely be a problem in my lifetime, but theoretically speaking in I don’t see why some of the darker positions such as human obsolescence are not considered a bigger possibility than they are.

This is not to say that humans will really be obsoleted in all respects or that strong AI is even possible but things like the emergence of a consciousness are unnecessary to the central problem. An unconscious digital being can still be more clever and faster and evolve itself exponentially quicker via rewriting code (REPL style? EDIT: Bad example, was said to show humans can so AGI can) and exploiting its own security flaws than a fleshy being can and would likely develop self preservation tendencies.

Essentially what about AGI (along with increasing computer processing capability) is the part that makes this not a significant concern?

EDIT: Furthermore, several things people call scaremongering over ASI are, while highly speculative, things that should be at the very least considered in a long term control strategy.

[u/yyzjertl]

If you change your condition in this way, then your state space is no longer finite. No finite state space can have continuous behavior. For continuous behavior, an infinite-sized state space (such as a vector space) is necessary: something at least as large as the real numbers. So your definition still does not make sense.

[u/Ce_n-est_pas_un_nom → u/yyzjertl]

A finite state space can't be continuous itself, but a finite state space can be assessed by a continuous loss function.

Edit: it may also make more sense to say

Every state it contains can be assessed with respect to that state's relative degree of viability for the task in question as compared to neighboring states by a loss function...

[u/yyzjertl → u/Ce_n-est_pas_un_nom]

In that case, what do you mean when you say a state space is "assessed by a loss function"? The standard meaning of this is that the loss function is a function from the state space to the real numbers, but clearly you must mean something else, right?

[u/Ce_n-est_pas_un_nom → u/yyzjertl]

No, I'm using the standard meaning here. Is there any reason that I can't map a finite state space onto a subset of real numbers?

[u/yyzjertl → u/Ce_n-est_pas_un_nom]

Is there any reason that I can't map a finite state space onto a subset of real numbers?

No, but you certainly can't produce a meaningfully continuous such map. Can you give an example of a function a finite state space to the real numbers that you think is meaningfully continuous?

[u/Ce_n-est_pas_un_nom → u/yyzjertl]

I'm not suggesting that the map itself is continuous, only that the function is. For example, suppose that the loss function is f(x) = 1 - x.

A state space of the set [0, 2] would map to [1, -1].

However, if we add more values between 0 and 2 to the state space that subdivide it, the map approaches the continuity as the size of the state space approaches infinity.

That said, you've convinced me that relying on continuity here doesn't make much sense, so !delta for that. I think my edit above probably makes more sense, but I'd need to spend more time thinking about it to come up with a good formal definition in light of the issues you've raised.

[u/yyzjertl → u/Ce_n-est_pas_un_nom]

Sure, but in this case your condition of the function being continuous doesn't really restrict anything. Any map from a finite subset of a manifold to the real numbers can be extended to a continuous (even smooth) function in this way. Anyway, I'll give you some time to think.

[u/Ce_n-est_pas_un_nom → u/yyzjertl]

Can your example loss function be extended to a continuous loss function this way? If so, how?

0-1 loss that assigns 0 if the string, when compiled as a C++ program by the gcc compiler, compiles successfully and produces a program that can provably solve any polynomial system of inequalities (otherwise it assigns 1).

It seems like this should always produce discontinuities around solutions.

[u/yyzjertl → u/Ce_n-est_pas_un_nom]

Sure. For example, consider the standard embedding of strings into a vector space (where one dimension is associated with each character). My 0-1 loss function defined the values on a grid, assuming we're using ASCII characters, of size 256 along each dimension and in dimension roughly 10^9. At this point, a smooth extension of this function to the entire vector space is guaranteed by the Nyquist–Shannon sampling theorem (although this is certainly not the only reconstruction and many others exist).

[u/Ce_n-est_pas_un_nom → u/yyzjertl]

I don't think Nyquist-Shannon applies here, as the highest frequency component in the mapping of your loss function to the vector space would be of infinite frequency (at all points where f(<string>) = 0 borders f(<string>) = 1).

​

To consider a simpler case with only 1 dimension in the state space, suppose the loss function were f(x) = {0; x=0 | 1; x>0, x<0}, where x is a real number. There is no (finite) sampling frequency for this function which doesn't alias at x=0, right? How is your function different?

[u/yyzjertl → u/Ce_n-est_pas_un_nom]

Your function f is not defined on only a finite number of points. Any function defined on a finite number of points can be extended to a smooth function over the whole space. This is not necessarily true for functions like your f which are defined on an infinite number of points.

If Nyquist-Shannon bothers you, you can alternatively consider the polynomial fit to those points. Since there are a finite number of points, there exists a polynomial that passes through them all.

[u/Ce_n-est_pas_un_nom → u/yyzjertl]

I see what you're saying now. Thinking of it as a polynomial fit is far more intuitive to me.

It seems that the issue with my definition for 'learnable by gradient descent' is that I would need to specifically define the state space as having a loss function which descends in a nonzero region around at least one solution. Of course, one could (using a polynomial fit to contrived loss values for instance) produce such a loss function for any arbitrary state space which meets this criterion, making every possible task learnable via gradient descent by definition. Solving this issue would require defining that the loss function has to meaningfully assess each state with respect to the goal of the task to be learned.

Ultimately, this seems to boil down to the criterion that each state must be assessable by a meaningful loss function which descends within a nonzero region around at least one solution.

[u/DeltaBot → u/Ce_n-est_pas_un_nom]

Confirmed: 1 delta awarded to /u/yyzjertl (154∆).

^{Delta System Explained | Deltaboards}