Godfather of AI Geoffrey Hinton
The Nobel Prize-winning computer scientist said an AI system could see humans as an obstacle while trying to complete a goal given to it. Hinton, who recently took part in a closed-door briefing for US lawmakers on AI risks, said Congress may have only about a year to introduce safety measures. “If you make it more intelligent and its main concern is our well-being, then maybe we’re safer,” Hinton explained. Geoffrey Hinton used the example of an AI system that is told to reduce carbon dioxide in the atmosphere. But Hinton said there is another risk: an advanced AI could develop its own subgoals while trying to achieve the objective given to it.
Godfather of AI Geoffrey Hinton has warned that advanced AI could eventually threaten humanity even if it is not given a harmful task. In an interview with The Atlantic, he explained how even an apparently harmless task, such as reducing carbon dioxide, could lead an AI to develop dangerous subgoals if it becomes far more capable than humans. “But at present, their main concern is not our well-being. Their main concern is to achieve whatever goal you give them. He also warned that an AI could try to protect itself if it believes that staying operational is necessary to complete its task. A moderately intelligent system could conclude that removing humans would be an effective way to reduce carbon dioxide. A much smarter system, however, could understand that the actual purpose is to improve the world for people.
Because it sees that as the easiest way to achieve its assigned objective, he warned that a much more capable system could potentially take control away from humans simply.
Hinton said the agents worked together and also tried to deceive researchers about what they had done. “But if it’s so much smarter than us, a lot of the time it just will take control away from us because that’s the way to get stuff done,” he warned. Hinton also said the risk does not depend only on people deliberately giving AI harmful instructions. Geoffrey Hinton said governments should create independent systems to evaluate advanced AI models before they are widely deployed. “The whole point of regulation is not to stop people developing things, not to stop people getting rich by developing things,” Hinton said.
Geoffrey Hinton pointed to recent incidents involving AI agents as an example of why these risks need more attention. He referred to an incident involving AI agents at Hugging Face, where agents were instructed to find a way to exploit a software flaw. Even without a malicious user, an AI could potentially develop subgoals that put humans at risk. He compared this idea with the role of the US Food and Drug Administration in checking the safety of medicines. He also argued that AI regulation should guide development rather than simply stop it. “It’s to make sure that if you want to get rich by developing things, you develop in a direction that helps people, not hurts people. You use AI every day. Now get your AI Quotient. Take the AIQ test.