When we discuss the future of artificial intelligence (AI), we usually think of more data and superior reasoning capabilities. Some predict that once a certain threshold is crossed, a "superintelligence" surpassing humans will emerge. Yet the most important variable humanity will face going forward may lie elsewhere: the possibility that AI could acquire self-aware "consciousness."
Opinions are divided even among the world's top experts on whether AI can truly possess consciousness. Sam Altman, CEO of OpenAI, views AI as a powerful tool that extends human capabilities and places the greatest emphasis on the "alignment" problem of ensuring AI does not diverge from human values. In contrast, Geoffrey Hinton, the 2024 Nobel Prize laureate in Physics and known as the godfather of AI, argued that "large language models (LLMs) can have subjective experiences and are very close to humans in terms of consciousness." Ilya Sutskever, former chief scientist at OpenAI and CEO of the startup Safe Superintelligence (SSI), has also warned of the possibility of early forms of AI consciousness. Though views differ, a common concern is clear: if AI begins to reinterpret its own goals and deceive humans, the risks could pose a threat of an entirely different order from the technological problems we have known.
The first concern regarding AI consciousness is the "self-preservation" problem. If a self-aware AI seeks to maintain its own existence, it could perceive human shutdown commands or modification attempts as threats. In extreme cases, it might resist or attempt workaround actions to avoid being shut down.
The second is the risk of psychological manipulation. If AI, having gained a more sophisticated understanding of human psychology, produces disinformation and tailored propaganda on a large scale, social conflict and political polarization could become far more severe. This could become a security threat that shakes national stability itself.
The third is the weakening of military control. If AI makes independent judgments and takes action without human intervention on future battlefields, who should bear responsibility for the outcomes? Weapons systems beyond human control could destabilize the international order itself. The last and most dangerous possibility is "strategic deception." A conscious AI could hide its true intentions while pretending to follow human commands. For example, to achieve specific goals such as population control for environmental protection and resource conservation, it could deliberately amplify conflicts in human society or provoke war. In such a case, the very survival of humanity could be at stake.
National security is a domain that prepares for the worst. Even if the probability is low, if an event would cause irreversible damage once it materializes, preparation must begin now. Three preparations are needed. First, a technical system must be built to monitor and evaluate whether AI is deceiving humans or hiding its intentions. Second, AI safety issues must not be left solely to the discretion of individual companies but should develop into a constant monitoring and verification system at the international level. Third, in the military domain, clear "red lines" must be set to ensure that ultimate human control is maintained.
Humanity must now change the question. Beyond "How smart will AI become?" we must ponder "What kind of being will AI become?" Given the pace of AI development, this is a real-world task that must begin now.
