About the author Xiuquan Li is Deputy Director of Center for Artifcial Intelligence at the Institute of Scientific and Technical Information of China. He is also Deputy Director of the Ministry of Science and Technology’s Next Generation AI Development Research Center. His main research areas include: AI technology trends, technology roadmaps for intelligent industries, and AI development strategies and innovation policies. Projects he has led include “Preparation of China’s Next Generation AI Development Annual Report,” and “Research on Frontier Trends and Policies of AI at Home and Abroad.” He has published over 40 academic papers and contributed to more than 10 published books.
The following excerpt is a translation of Section 11.2.3 of Li’s The Intelligent Revolution: The Evolution and Value Creation of AI Technology (智能化变革：人工智能技术进化与价值创造) (2021), entitled ‘Concerns about whether the possible emergence of superintelligence is controllable’.
Sections 11.2.1 (Safety/security risks brought by imperfections in the technology itself), 11.2.2 (Safety/security challenges caused by the abuse of technology) and 11.3 (Towards AI that humans can understand) also contain some material related to AI safety risks.
▶ Cite This Work Li, Xiuquan. The Intelligent Transformation: The Evolution and Value Creation of AI Technology. 2021. Translated by Concordia AI, May 2023.
People’s most long-term concerns about AI safety risks stem from the fear of superintelligence becoming uncontrollable. Assume that AI develops to the stage of superintelligence, can evolve autonomously, has human-like self-awareness, and has its own worldview and values. At this time, the machine’s own needs may take precedence over human instructions, and it can change the behavioral goals set by designers or users, thereby posing a threat to human dominance and controllability. In other words, we are not afraid of AI being “smart,” but rather that it “develops a mind [of its own].”1
According to Roland Berger Consulting, there are currently two main types of AI: weak AI and strong AI. But when technology surpasses the limits of human beings, super AI may also appear. If humans cannot predict and control it, they may even be destroyed by super AI that is smarter and more powerful than them. This is the main aspect of the public’s fear. British physicist Stephen Hawking worried that human beings are limited by slow biological evolution and completely lack competitiveness; AI might catch up with or even surpass them one day in the future. Bill Gates, Elon Musk, etc. are also worried that the unconstrained development of AI technology will allow machines to acquire intelligence beyond that of humans and will cause some safety hazards that are hard to control.
Few people question whether strong AI can be realized; this is the founding intention of AI and the direction that the field is working hard towards. Knowledge representation, cross-domain learning, common sense and semantic understanding, logical reasoning, etc. are constantly strengthening the ability of AI to imitate human thinking. Machines can even properly understand a book — it is obviously inappropriate to call this weak AI. However, there has been continued controversy over recent decades as to whether artificial objects such as intelligent machines can form self-identity and self-awareness (e.g., a sense of existence, selfishness, and beliefs), whether they should have consciousness, and whether human beings can control them if they do have consciousness.
American philosopher John Searle published a paper in 1980 which proposed the famous Chinese room experiment and contained many arguments and discussions relating to whether AI based on designing programs can have the ability to understand, or in other words a real mind2. Searle believed that silicon-based electronic systems also have their physical and chemical limitations and capability boundaries. Many advanced intelligent behaviors of the human brain (such as consciousness) require the participation of physical and chemical processes, and the difference [between the human brain] and using silicon and metal to simulate the brain to generate intelligent systems could be huge.
At that time, in Searle’s research, the ability to understand and ‘mind’ were still considered equivalent. In fact, with the development of cognitive intelligence in recent decades, it is already possible to use knowledge modeling and reasoning techniques to extract part of the knowledge in a book and continue to accumulate it into existing machine-based knowledge systems. We are getting closer and closer to strong AI that can understand books. As envisioned in the book Homo Deus: A Brief History of Tomorrow, a species with a high level of intelligence and no consciousness may appear in the future.
Roger Penrose claimed in his book “Shadows of the Mind” that the current AI research paradigm is unlikely to produce consciousness. He believes that human consciousness may be related to some kind of overall quantum state that occurs in large areas of the brain and that these collective quantum effects may be produced in the smallest unit of human brain tissue structure — microtubules.
As for whether AI can gain consciousness in the future, there is still no school of thought holding any point of view that can produce strong enough evidence for everyone to recognize. Perhaps discussion on this issue will continue for quite some time in fields such as AI academia and social science. How to give full play to the advantages of subjective human consciousness, combine it with the powerful ability of machine intelligence to solve problems, and move towards an integration of humans and machines in which the advantages of both complement each other has become the main development trend of AI.
As far as the development stage is concerned, AI has gone through the stage of computational intelligence, is currently at a stage of great development of perceptual intelligence and is beginning to move towards cognitive intelligence. At present, the narrow abilities of many AI systems have far surpassed those of humans, and the gap is widening at an extremely fast pace. For example, in most games with clear rules, humans are no longer worthy opponents [for AI]. But AI surpasses humans only in specific skills; semantic understanding, logical reasoning, creativity, etc. are still relatively weak, and the flexibility and generality to solve problems across fields is far from being achieved.
From the perspective of technological paths, AI based on silicon-based material and binary digital signal calculations is ultimately a productivity-transforming technology with great potential, and the intelligent transformation that it is leading will be an advanced stage of the information technology revolution. New computing architectures such as biological computing, neuromorphic computing, and quantum computing that scientists are exploring will have a profound impact on the long-term development of AI. Some experts even believe that they may lead AI to develop to a stage beyond the information technology revolution. However, these cutting-edge computing technologies are still in the stage of striving towards the computation power of the first transistor computer. A way to use new architectures to realize advanced human-like intelligence, such as the capacity for autonomous learning, generalization and deduction, has not yet been found. In addition, human beings are still far from fully understanding themselves, especially the working principles of the 100 billion-neuron human brain and have not seen a substantial scientific breakthrough in the understanding of consciousness.
All in all, it is unclear whether superintelligence or a “singularity” will arrive, and there are no technical signs of development in that direction. Mr. Kai-Fu Lee believes that “at least at present, human beings are still quite far away from the threat of super AI.” We don't need to worry too much about this, let alone feel inexplicable fear.
Of course, due to uncertainty over the emergence of superintelligence and the disastrous consequences that may be caused by uncontrollability, it is still necessary to be vigilant and pay attention to the issue of controllability in the process of AI technology R&D. As early as the 1940s, Asimov had thought deeply about the possible threats brought by the future development of AI and proposed the “Three Laws of Robotics” to constrain the research of robots, so as to ensure safety and controllability in a future where robots and humans coexist. Academician Bo Zhang of Tsinghua University also said that the unique consciousness of human beings is our last line of defense. Mr. Kai-Fu Lee believes that when it comes to superintelligence, we should not make the leap unless we first clearly address all control and security issues. In future, the issue of preventing the loss of control of superintelligence will still be the bottom line in the scientific exploration of AI3.
The consensus view of most experts is that, judging from the response to governance questions concerning high-risk technologies such as nuclear energy and cloning, human beings have strong self-discipline and self-adjustment ability. We still have enough time to observe and prepare. Through proactive precautions and a prescient response, the development of AI will eventually be controllable by humans.
Therefore, the three types of safety/security and control risks of AI must be treated differently and dealt with separately4. Those bringing the risks must be the ones to address them; the safety of intelligent models and methods should be the key direction of the current new generation of AI technology R&D, and an active response5 is required for the safety risks caused by shortcomings with the technology itself. To respond to safety/security challenges triggered by abuse and malicious use of the technology, governance must move in step with technological development. Through the construction of laws and regulations and strict and effective oversight and accountability, we must ensure that intelligent new technologies are used to improve people’s livelihood and well-being, and we must prevent technology from doing evil. Whether the superintelligence that might emerge would be controllable should mainly be left to the scientific community to discuss. We, the public, don’t have to worry yet, but in terms of technology R&D, we must also adhere to the original intention of designing AI to assist humans, and ensure that humans have the ability to effectively intervene in circumstances where machines endanger human interests.
1. ”心眼“, here translated as mind, combines the characters for “heart/mind“ and “eye”, and conveys a capacity for cognition, feeling and intention.
2. “心灵”, which can also be translated as “soul”, is here translated as “mind”, which is the term used in Searle’s paper.
3. “底线思维” (bottom line thinking) has been commonly used by President Xi and other political/policy figures since the 18th CCP Party Congress in 2012. According to a CCP news website, it means “preparing for the bad points and working at the same time to obtain the best possible results. It’s about being prepared, not panicking, and firmly grasping the initiative.”
4. Earlier in the book, these are defined as risks from: (1) the AI technology itself being immature and systems being imperfectly designed; (2) the subjective tendency of those designing or manipulating the technology to abuse it; (3) superintelligence with self-awareness.
5. Here translated as, “Those bringing the risks must be the ones to address them”, “解铃还须系铃人” is a Chinese idiom that literally means “Untying the bell [from the tiger’s neck] requires the person who fastened the bell”.