Can artificial superintelligence convince you that you are virtual? Imagine an incredibly smart artificial superintelligence locked in a virtual world - say, just in a snuff box. You don't know if he will be angry, friendly or neutral. All you know is that he wants to get out of the snuff box and that you can interact with him through a text interface. If the AI is truly superintelligent, will you be able to talk to it for five hours and not succumb to its persuasion and manipulation - not to open the snuff box? This thought experiment was proposed by Eliezer Yudkowsky, a researcher at the Mechanical Engineering Research Institute (MIRI). MIRI consists of many scientists who research the risks of developing artificial superintelligence; although it hasn't even appeared yet, it is already attracting attention and sparking debate. Yudkowsky argues that artificial superintelligence can say anything it can to persuade you: careful reasoning, threats, deception, rapport-building, subliminal suggestion, and so on. At the speed of light, the AI builds a story, probes for weaknesses, and determines how to most easily convince you. As existential threat theorist Nick Bostrom puts it, “We have to believe that superintelligence can achieve anything it sets its mind to.” The snuff-box AI experiment raises doubts about our ability to control what we could create. It also forces us to contemplate the rather bizarre possibilities of what we don't know about our own reality. Simulation may be the ideal breeding ground for artificial intelligence. Some of the most plausible ways to create powerful AI involve simulated worlds—by imposing constraints on an artificially simulated world and selecting desirable traits, scientists can attempt to roughly duplicate a person's own model of development and conscious sentience. Constructing AI in a simulated environment can also prevent it from leaking out into the world until its intent and threat level become clearer. Recent advances hint that it may be impossibly difficult to develop cognition from scratch. No one can build a human-scale mind by writing one line of code at a time. There are other promising approaches, of course, and as computing power increases, it may well be possible to develop a process that will eventually create an AI as smart and capable as a human. Will you free the AI or risk dooming yourself to never-ending torture? One idea to "create" AI is to scan, map and emulate a virtual model of the human brain. The human brain contains billions of neurons and trillions of connections between these neurons. If we virtually simulated someone's complete structure of neurons and their connections, would this be an adequate replica of human consciousness? Other plans for virtual intelligence development are much less straightforward. One of the key concepts is the creation of a learning machine - a weak intelligence that can learn, perhaps thanks to an artificial neural network programmed to endlessly expand and improve its own algorithms. The scenarios described above involve a troubling problem. The emulated brain can operate 2, 10, or 100 times faster than a normal biological brain. AI can modify its own code or start creating more intelligent versions of itself. The explosion in speed and intelligence of self-improving computers was discussed back in 1965 by statistician I. J. Goode, who wrote that “the first superintelligent machine will be the last invention that humanity will need to create.” If we build an incredibly intelligent machine, we'll have to be careful where we place it. Engineers are unlikely to be able to create a model of the world that will be completely protected from leaks. Scientists must create a simulation world that allows them to peer into this universe and collect information. It may seem harmless, but it is difficult to imagine whether such a superintelligence would find a way to harm us through simple observations. Returning to Yudkowsky boxes, there is a particularly creepy method the AI can use to force you to release it. It inv yt3.ggpht