Ben Goertzel dismisses Singularity Institute's "Scary Idea"

AI theorist Ben Goertzel has posted an article in which he declares his rejection of the Singularity Institute for Artificial Intelligence's claim that "progressing toward advanced AGI without a design for "provably non-dangerous AGI"...is highly likely to lead to an involuntary end for the human race." Goertzel calls this their "Scary Idea" and attempts to show that the fear is largely overstated.

He breaks the SIAI argument down to four primary points:

1. If one pulled a random mind from the space of all possible minds, the odds of it being friendly to humans (as opposed to, e.g., utterly ignoring us, and being willing to repurpose our molecules for its own ends) are very low
2. Human value is fragile as well as complex, so if you create an AGI with a roughly-human-like value system, then this may not be good enough, and it is likely to rapidly diverge into something with little or no respect for human values
3. "Hard takeoffs" (in which AGIs recursively self-improve and massively increase their intelligence) are fairly likely once AGI reaches a certain level of intelligence; and humans will have little hope of stopping these events
4. A hard takeoff, unless it starts from an AGI designed in a "provably Friendly" way, is highly likely to lead to an AGI system that doesn't respect the rights of humans to exist

Taking these points into consideration, Goertzel pieces together what he feels is the SIAI's argument:

If someone builds an advanced AGI without a provably Friendly architecture, probably it will have a hard takeoff, and then probably this will lead to a superhuman AGI system with an architecture drawn from the vast majority of mind-architectures that are not sufficiently harmonious with the complex, fragile human value system to make humans happy and keep humans around.

Goertzel then expresses his particular concerns with this argument, including SIAI's Eliezer Yudkowsky's suggestion that we can get human values into an AGI system, what he calls Coherent Extrapolated Volition:

...I think this is a very science-fictional and incredibly infeasible idea (though a great SF notion). I've discussed it and proposed some possibly more realistic alternatives in a previous blog post (e.g. a notion called Coherent Aggregated Volition). But my proposed alternatives aren't guaranteed-to-succeed nor neatly formalized.

But setting those worries aside, is the computation-theoretic version of provably safe AI even possible? Could one design an AGI system and prove in advance that, given certain reasonable assumptions about physics and its environment, it would never veer too far from its initial goal (e.g. a formalized version of the goal of treating humans safely, or whatever)?

I very much doubt one can do so, except via designing a fictitious AGI that can't really be implemented because it uses infeasibly much computational resources. My GOLEM design, sketched in this article, seems to me a possible path to a provably safe AGI -- but it's too computationally wasteful to be practically feasible.

Oooh, it looks like we have the makings of a great debate, here. I'll be interested to see if the SIAI retorts and how they address Goertzel's concerns.

Goertzel on psi

Much to the chagrin of his readers, AI theorist Ben Goertzel has dared to explore the topic of parapsychology on his blog (or just 'psi'). He argued that the scientific evidence for psi is stronger than we think.

When his readers complained in the comments section, Goertzel came back by saying,

These emails basically present arguments of two forms:

1. You're nuts, don't you know all the psi experiments are fraud and experimental error, everyone knows that...
2. Look, even if there's a tiny chance that some psi phenomena are real, you're a fool to damage your reputation by aligning yourself with the kooks who believe in it

What shocks me (though it shouldn't, as I've been around 41 years and seen a lot of human nature already) about arguments of the first form is the irrational degree of skepticism toward this subject, displayed by otherwise highly rational and reflective individuals...

...What shocks me about arguments of the second form is how often they come from individuals who are publicly aligned with other extremely radical ideas. For instance a few Singularitarians have emailed me and warned me that me talking about psi is bad, because then people will think Singularitarians are kooks.

I'm with Ben on this one. We shouldn't rule anything out at this point or worry about damaging reputations. Science is about falsification. But before you can falsify something you have to throw the theory out there. And as Goertzel noted, there's a ton of research being done in this area with some rather bizarre observations being made.

I've written some articles along these lines in the past:

Does the brain tap into the future?
The Global Consciousness Project
Paul Smith: Reading the Enemy's Mind