There is a lot of confusion about what artificial intelligence (AI) means or doesn’t mean, especially in places like Silicon Valley.
For me, the biggest confusion is between intelligence and consciousness. Ninety-five percent of science fiction movies are based on the error that AI will be artificial consciousness. They assume that robots will have emotions, will feel things, that humans will fall in love with them, or that they will want to destroy us. This is not true.
Intelligence is not consciousness. Intelligence is the ability to solve problems. Consciousness is the ability to feel things. In humans and other animals, the two go together. For mammals, our emotions and sensations are an integral part of the way we solve problems. In the case of computers, we don’t see the two going together.
Over the past few decades, there has been immense development in computer intelligence and exactly zero development in computer consciousness. There is absolutely no reason to think that computers are anywhere near developing consciousness. They might be moving along a very different trajectory than mammalian evolution. In the case of mammals, evolution has driven mammals towards greater intelligence by way of consciousness, but in the case of computers, they might be progressing along a parallel and very different route to intelligence that just doesn't involve consciousness at all.
We may find ourselves in a world with nonconscious super intelligence. The question is not whether humans will fall in love with robots or whether robots will try to kill humans. The big question is: how does a world of nonconscious super intelligence look? We have nothing in history that prepares us for such a scenario.