Xiuquan LI
李修全
About the author Xiuquan Li is Deputy Director of Center for Artifcial Intelligence at the Institute of Scientific and Technical Information of China. He is also Deputy Director of the Ministry of Science and Technology’s Next Generation AI Development Research Center. His main research areas include: AI technology trends, technology roadmaps for intelligent industries, and AI development strategies and innovation policies. Projects he has led include “Preparation of China’s Next Generation AI Development Annual Report,” and “Research on Frontier Trends and Policies of AI at Home and Abroad.” He has published over 40 academic papers and contributed to more than 10 published books.
关于作者
李修全，工学博士，中国科学技术信息研究所人工智能中心副主任，科技部新一代人工智能发展研究中心副主任。主要研究领域包括：人工智能前沿技术趋势、智能化产业技术路线图、人工智能发展战略及创新政策等。主持《中国新一代人工智能发展年度报告编制》《国内外人工智能前沿趋势与政策研究》等课题10余项，发表学术论文40余篇，参与出版著作10余部。
The following excerpt is a translation of Section 11.2.3 of Li’s The Intelligent Revolution: The Evolution and Value Creation of AI Technology (智能化变革：人工智能技术进化与价值创造) (2021), entitled ‘Concerns about whether the possible emergence of superintelligence is controllable’.
Sections 11.2.1 (Safety/security risks brought by imperfections in the technology itself), 11.2.2 (Safety/security challenges caused by the abuse of technology) and 11.3 (Towards AI that humans can understand) also contain some material related to AI safety risks.
▶ Cite Our Translation
Concordia AI. “Xiuquan Li — Chinese Perspectives on AI Safety.” Chineseperspectives.ai, 29 Mar. 2024, chineseperspectives.ai/Xiuquan-Li.
▶ Cite This Work 李修全(2021). “智能化变革：人工智能技术进化与价值创造”. 得到. https://www.dedao.cn/ebook/detail?id=rEQKv6PKN7rEo2Gxg96ZjApyMvQVlw5GmjWXb14PJzDkYaReqd8n5LOmB8d7egBx
Translation
People’s most long-term concerns about AI safety risks stem from the fear of superintelligence becoming uncontrollable. Assume that AI develops to the stage of superintelligence, can evolve autonomously, has human-like self-awareness, and has its own worldview and values. At this time, the machine’s own needs may take precedence over human instructions, and it can change the behavioral goals set by designers or users, thereby posing a threat to human dominance and controllability. In other words, we are not afraid of AI being “smart,” but rather that it “develops a mind [of its own].”1
原文
“人们对人工智能安全风险最远期的担忧，源于对超级智能出现不可控风险的恐惧。假设人工智能发展到超级智能阶段，能够自主进化，具有类人的自我意识，拥有自己的世界观、价值观，这时机器的自我诉求有可能优先于人类的指示，并且能够自行变更设计人员或使用者设定的行为目标，从而对人类的主导性和可控性造成威胁。也就是说，我们其实不是怕人工智能“聪明”，而是怕它“长了心眼”。
“人们对人工智能安全风险最远期的担忧，源于对超级智能出现不可控风险的恐惧。假设人工智能发展到超级智能阶段，能够自主进化，具有类人的自我意识，拥有自己的世界观、价值观，这时机器的自我诉求有可能优先于人类的指示，并且能够自行变更设计人员或使用者设定的行为目标，从而对人类的主导性和可控性造成威胁。也就是说，我们其实不是怕人工智能“聪明”，而是怕它“长了心眼”。
According to Roland Berger Consulting, there are currently two main types of AI: weak AI and strong AI. But when technology surpasses the limits of human beings, super AI may also appear. If humans cannot predict and control it, they may even be destroyed by super AI that is smarter and more powerful than them. This is the main aspect of the public’s fear. British physicist Stephen Hawking worried that human beings are limited by slow biological evolution and completely lack competitiveness; AI might catch up with or even surpass them one day in the future. Bill Gates, Elon Musk, etc. are also worried that the unconstrained development of AI technology will allow machines to acquire intelligence beyond that of humans and will cause some safety hazards that are hard to control.
根据罗兰贝格咨询的观点，当前主要存在两种类型的人工智能：弱人工智能和强人工智能。但当技术突破了人类的极限，可能还会出现超人工智能。如果人类无法预测和控制超人工智能，甚至很有可能会被比自己更聪明、更强大的超人工智能所毁灭，这是公众产生恐惧感的主要方面。英国物理学家斯蒂芬·霍金担心，人类要受到缓慢的生物进化的限制，根本没有竞争力，人工智能也许会在将来的某一天赶上甚至超过人类；比尔·盖茨、埃隆·马斯克等也担忧，对人工智能技术不加约束的开发，会让机器获得超越人类智力水平的智能，并引发一些难以控制的安全隐患。
Few people question whether strong AI can be realized; this is the founding intention of AI and the direction that the field is working hard towards. Knowledge representation, cross-domain learning, common sense and semantic understanding, logical reasoning, etc. are constantly strengthening the ability of AI to imitate human thinking. Machines can even properly understand a book — it is obviously inappropriate to call this weak AI. However, there has been continued controversy over recent decades as to whether artificial objects such as intelligent machines can form self-identity and self-awareness (e.g., a sense of existence, selfishness, and beliefs), whether they should have consciousness, and whether human beings can control them if they do have consciousness.
对于能否实现强人工智能，很少有人质疑，这也是人工智能创立的初衷和努力方向，知识表示、跨领域学习、常识和语义理解、逻辑推理等都在不断强化人工智能模仿人类思维的能力，机器都能真正读懂一本书了，称它为弱人工智能显然不适合。但对于智能机器这样的人工物能不能形成自我认同和存在感、自私、信仰等自我意识，是否应该具有意识，如果智能机器具有意识人类能否掌控等问题，近几十年来在科学界、产业界持续争论。
American philosopher John Searle published a paper in 1980 which proposed the famous Chinese room experiment and contained many arguments and discussions relating to whether AI based on designing programs can have the ability to understand, or in other words a real mind2. Searle believed that silicon-based electronic systems also have their physical and chemical limitations and capability boundaries. Many advanced intelligent behaviors of the human brain (such as consciousness) require the participation of physical and chemical processes, and the difference [between the human brain] and using silicon and metal to simulate the brain to generate intelligent systems could be huge.
美国哲学家约翰·塞尔在1980年发表论文提出了著名的中文屋实验，对基于设计程序的人工智能是否能够具有理解能力或者说真正的心灵进行了大量论证和讨论。塞尔认为，硅基电子系统也有其物理化学局限性和能力边界，人脑很多高级智能行为（如意识），是需要物理化学过程参与的，用硅和金属作为材质，模拟脑生成智能系统，其差异可能是巨大的。
At that time, in Searle’s research, the ability to understand and ‘mind’ were still considered equivalent. In fact, with the development of cognitive intelligence in recent decades, it is already possible to use knowledge modeling and reasoning techniques to extract part of the knowledge in a book and continue to accumulate it into existing machine-based knowledge systems. We are getting closer and closer to strong AI that can understand books. As envisioned in the book Homo Deus: A Brief History of Tomorrow, a species with a high level of intelligence and no consciousness may appear in the future.
当时塞尔的研究中，还是把理解能力与心灵等同起来考虑，实际上随着近几十年来认知智能的发展，通过知识建模和推理技术，已能够实现抽取一本书中的部分知识，并不断累积到现有机器化知识体系中，我们已经距离机器能读懂书的强人工智能越来越近了。就像《未来简史》一书中展望的，未来可能出现一种高智能水平而没有意识的物种。
Roger Penrose claimed in his book “Shadows of the Mind” that the current AI research paradigm is unlikely to produce consciousness. He believes that human consciousness may be related to some kind of overall quantum state that occurs in large areas of the brain and that these collective quantum effects may be produced in the smallest unit of human brain tissue structure — microtubules.
罗杰·彭罗斯（Roger Penrose）在其出版的《心灵的阴影》一书中认为，当前人工智能研究范式不太可能产生意识，人脑意识可能与发生在大脑的大面积区域的某种全局量子态有关。而这些集体量子效应有可能产生于人脑组织结构最小单元—微管中。
As for whether AI can gain consciousness in the future, there is still no school of thought holding any point of view that can produce strong enough evidence for everyone to recognize. Perhaps discussion on this issue will continue for quite some time in fields such as AI academia and social science. How to give full play to the advantages of subjective human consciousness, combine it with the powerful ability of machine intelligence to solve problems, and move towards an integration of humans and machines in which the advantages of both complement each other has become the main development trend of AI.
对于人工智能在未来能否获得意识，目前尚看不到持哪一观点的学派能拿出足以令所有人认可的强有力证据。或许关于这一问题的讨论还将在人工智能学术界和社会科学等领域持续相当一段时间。而如何发挥人类意识的主观性优势，结合机器智能解决问题的强大能力，优势互补走向人机共融则成为目前人工智能主要发展趋势。
As far as the development stage is concerned, AI has gone through the stage of computational intelligence, is currently at a stage of great development of perceptual intelligence and is beginning to move towards cognitive intelligence. At present, the narrow abilities of many AI systems have far surpassed those of humans, and the gap is widening at an extremely fast pace. For example, in most games with clear rules, humans are no longer worthy opponents [for AI]. But AI surpasses humans only in specific skills; semantic understanding, logical reasoning, creativity, etc. are still relatively weak, and the flexibility and generality to solve problems across fields is far from being achieved.
就发展阶段来看，目前人工智能已经走过了计算智能阶段，正处于感知智能大发展阶段，并向认知智能开始迈进。现在很多人工智能系统的专项能力已经远远超越人类，并在以极快的速度拉大这种差距。比如，绝大部分具有明确规则的游戏人类都已不是对手。但目前人工智能超越人类只是在特定技能方面，语义理解、逻辑推理、创造力等还较弱，能够跨领域解决问题的灵活性和通用性还远未具备。
From the perspective of technological paths, AI based on silicon-based material and binary digital signal calculations is ultimately a productivity-transforming technology with great potential, and the intelligent transformation that it is leading will be an advanced stage of the information technology revolution. New computing architectures such as biological computing, neuromorphic computing, and quantum computing that scientists are exploring will have a profound impact on the long-term development of AI. Some experts even believe that they may lead AI to develop to a stage beyond the information technology revolution. However, these cutting-edge computing technologies are still in the stage of striving towards the computation power of the first transistor computer. A way to use new architectures to realize advanced human-like intelligence, such as the capacity for autonomous learning, generalization and deduction, has not yet been found. In addition, human beings are still far from fully understanding themselves, especially the working principles of the 100 billion-neuron human brain and have not seen a substantial scientific breakthrough in the understanding of consciousness.
从技术路线上看，以硅基材料0-1数字信号计算为基础的人工智能终究是一项潜力巨大的生产力变革技术，其引领智能化变革也将是信息科技变革的高级阶段。科学家们在探索的生物计算、神经形态计算、量子计算等新的计算架构会给人工智能远期发展带来深刻影响，甚至也有专家认为有可能引领人工智能向超越信息科技变革的新阶段发展。但目前这些前沿计算技术仍处在向第一台晶体管计算机的算力水平努力阶段，如何基于新的架构实现类似人脑的自主学习、泛化推演等高级智能还没有找到实现途径。另外，现在人类离完全了解自身，尤其是了解拥有1000亿神经元的人类脑神经系统的工作原理还差得很远，还没看到在关于意识的认识上实现科学层面实质性突破。
All in all, it is unclear whether superintelligence or a “singularity” will arrive, and there are no technical signs of development in that direction. Mr. Kai-Fu Lee believes that “at least at present, human beings are still quite far away from the threat of super AI.” We don't need to worry too much about this, let alone feel inexplicable fear.
综合来看，我们还不清楚超级智能或“奇点”是否会到来，从技术上也还看不到向那个方向发展的迹象。李开复先生认为，“至少在目前，人类离超人工智能的威胁还相当遥远”。对此我们完全不需要过于忧虑，更没必要产生莫名其妙的恐惧感。
Of course, due to uncertainty over the emergence of superintelligence and the disastrous consequences that may be caused by uncontrollability, it is still necessary to be vigilant and pay attention to the issue of controllability in the process of AI technology R&D. As early as the 1940s, Asimov had thought deeply about the possible threats brought by the future development of AI and proposed the “Three Laws of Robotics” to constrain the research of robots, so as to ensure safety and controllability in a future where robots and humans coexist. Academician Bo Zhang of Tsinghua University also said that the unique consciousness of human beings is our last line of defense. Mr. Kai-Fu Lee believes that when it comes to superintelligence, we should not make the leap unless we first clearly address all control and security issues. In future, the issue of preventing the loss of control of superintelligence will still be the bottom line in the scientific exploration of AI3.
当然，由于超级智能出现的不确定性和不可控可能导致的灾难性后果，在人工智能技术研发过程中，仍需对可控性问题予以警惕和重视。早在20世纪40年代，阿西莫夫就已经对人工智能未来发展可能带来的威胁进行了深入思考，并提出用于约束机器人研究的“机器人三定律”，以确保机器人与人共存之后的安全可控。清华大学张钹院士也曾表示，人类特有的意识是我们最后一道防线。李开复先生认为，对于超级智能，除非我们首先明确解决了所有控制和安全问题，否则我们不应该跨越。未来，防范超级智能失去掌控的问题仍将是人工智能科学探索过程中的底线思维。
The consensus view of most experts is that, judging from the response to governance questions concerning high-risk technologies such as nuclear energy and cloning, human beings have strong self-discipline and self-adjustment ability. We still have enough time to observe and prepare. Through proactive precautions and a prescient response, the development of AI will eventually be controllable by humans.
大部分专家的共识性观点认为，从对核能、生物克隆等高风险技术的治理问题的应对情况来看，人类拥有强大的自我约束、自我调整能力。我们还有足够的时间观察和准备，通过主动防范和前瞻应对，人工智能发展终将是人类可控的。
Therefore, the three types of safety/security and control risks of AI must be treated differently and dealt with separately4. Those bringing the risks must be the ones to address them; the safety of intelligent models and methods should be the key direction of the current new generation of AI technology R&D, and an active response5 is required for the safety risks caused by shortcomings with the technology itself. To respond to safety/security challenges triggered by abuse and malicious use of the technology, governance must move in step with technological development. Through the construction of laws and regulations and strict and effective oversight and accountability, we must ensure that intelligent new technologies are used to improve people’s livelihood and well-being, and we must prevent technology from doing evil. Whether the superintelligence that might emerge would be controllable should mainly be left to the scientific community to discuss. We, the public, don’t have to worry yet, but in terms of technology R&D, we must also adhere to the original intention of designing AI to assist humans, and ensure that humans have the ability to effectively intervene in circumstances where machines endanger human interests.
因此，对于人工智能的三类安全可控风险，还须区别对待和分别应对。解铃还须系铃人，智能模型与方法的安全性问题应是当前新一代人工智能技术研发的重点方向，积极应对技术自身不完善带来的安全风险；应对技术滥用、恶用引发的安全挑战更多需要技术治理同步跟进，通过法律法规等制度建设和严厉有效的监管问责，保障智能化新技术用于改善民生福祉，杜绝技术做恶；对可能出现的超级智能是否可控问题，主要还应留给科学界探讨，我们大众尚不必担忧，但技术研发上也需坚持设计AI用来协助人类的初衷，并确保在机器危害人类利益的情况下，人类有能力进行有效干预。
Translator’s notes
1. ”心眼“, here translated as mind, combines the characters for “heart/mind“ and “eye”, and conveys a capacity for cognition, feeling and intention.
2. “心灵”, which can also be translated as “soul”, is here translated as “mind”, which is the term used in Searle’s paper.
3. “底线思维” (bottom line thinking) has been commonly used by President Xi and other political/policy figures since the 18th CCP Party Congress in 2012. According to a CCP news website, it means “preparing for the bad points and working at the same time to obtain the best possible results. It’s about being prepared, not panicking, and firmly grasping the initiative.”
4. Earlier in the book, these are defined as risks from: (1) the AI technology itself being immature and systems being imperfectly designed; (2) the subjective tendency of those designing or manipulating the technology to abuse it; (3) superintelligence with self-awareness.
5. Here translated as, “Those bringing the risks must be the ones to address them”, “解铃还须系铃人” is a Chinese idiom that literally means “Untying the bell [from the tiger’s neck] requires the person who fastened the bell”.
Other Authors
