Yi ZENG
曾毅
About the author Yi ZENG is a Professor and Director at the Brain-inspired Cognitive Intelligence Lab, and the International Research Center for AI Ethics and Governance, both at the Institute of Automation, Chinese Academy of Sciences. He is the founding Director of Center for Long-term AI, and the AI for Sustainable Development Goals Cooperation Network. He is a board member for the National Governance Committee of Next Generation Artificial Intelligence, China. He is an member of UN High-level Advisory Body on AI, an Expert in the Ad Hoc Expert Group on AI Ethics, UNESCO. He is in the WHO Expert Group on the Ethics/Governance of Artificial Intelligence for Health. His major research interests focus on Brain-inspired Artificial Intelligence, AI Ethics and Governance, AI Safety, and AI for Sustainable Development.
关于作者曾毅，中国科学院自动化研究所研究员，类脑智能实验室副主任，人工智能伦理与治理研究中心主任；中国科学院大学岗位教授，博士生导师；国家新一代人工智能治理专委会委员；联合国(UN)人工智能高层咨询机构专家组专家；联合国教科文组织(UNESCO)人工智能伦理特设专家组专家；世界卫生组织(WHO)人工智能伦理与治理专家组专家；主要研究领域是类脑人工智能、人工智能伦理、安全与治理、人工智能与可持续发展。
On July 18, 2023, the UN Security Council sat down for the first time to discuss the potential threats of AI to world peace and security. Yi Zeng delivered a briefing titled “Opportunities and Risks for International Peace and Security,” suggesting that “in the short-term and the long-term, the risk of AI replacing and causing the extinction of humankind will be present” and that “in the long-term, we haven’t given superintelligence any practical reasons why they should protect humankind.”
Yi Zeng has signed the open letter to Pause Giant AI Experiments that “call[s] on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.” Yi Zeng is also among the signatories of the Statement on AI Risk from the Center for AI Safety, which states that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
The following is a translation of a Chinese media interview (搜狐科技《思想大爆炸——对话科学家》栏目第六期) during which he explained his support for both initiatives.
Yi Zeng has also conducted surveys investigating Chinese views on the aforementioned open letter and on whether we can and should develop strong AI, with results available in English.
▶ Cite Our TranslationConcordia AI. “Yi Zeng — Chinese Perspectives on AI Safety.” Chineseperspectives.ai, 29 Mar. 2024, chineseperspectives.ai/Yi-Zeng.
▶ Cite This Work 曾毅(2023). “中科院自动化所曾毅：未来AI智力水平或全面超越人类，能否共生最大的瓶颈在人”. 在思想大爆炸——对话科学家栏目上的发言. https://www.sohu.com/a/686209288_115565
Translation
Q: We noticed that you signed two recent AI statements. Why did you sign them?
Zeng: Both statements recognize the risks in the current development of artificial intelligence and the potential for losing control, but significant differences exist in how the two letters respond to the problem. “Pause Giant AI experiments” calls for pausing research on AI systems more powerful than GPT-4 and prioritizes the design and implementation of a safety protocol for AI. The new “Statement on AI Risk” pronounces that “mitigating the risk of AI extinction should be a global priority alongside other society-scale risks such as pandemics and nuclear war,” which more deeply and directly expresses the signatories’ concerns about the potential existential risks that AI poses to humanity, as well as actions to be taken. My understanding of the issue is close to such views, so I signed the new statement before it was even officially released. I believe the vision shared by the vast majority of people working to develop artificial intelligence is to use it to benefit mankind, rather than to bring risks – potentially existential risks – upon mankind. Therefore, the vast majority of people have the right to know about the potential risks of AI, and developers have the obligation to ensure that AI does not pose existential risks to humanity, or at least to minimize the possibility of such risks through stakeholders. It is difficult for a few people to change existing trends, but when a few people take the first step to raise public awareness, more and more people will ultimately participate in changing the status quo.
Q: We noticed that you signed two recent AI statements. Why did you sign them?
Zeng: Both statements recognize the risks in the current development of artificial intelligence and the potential for losing control, but significant differences exist in how the two letters respond to the problem. “Pause Giant AI experiments” calls for pausing research on AI systems more powerful than GPT-4 and prioritizes the design and implementation of a safety protocol for AI. The new “Statement on AI Risk” pronounces that “mitigating the risk of AI extinction should be a global priority alongside other society-scale risks such as pandemics and nuclear war,” which more deeply and directly expresses the signatories’ concerns about the potential existential risks that AI poses to humanity, as well as actions to be taken. My understanding of the issue is close to such views, so I signed the new statement before it was even officially released. I believe the vision shared by the vast majority of people working to develop artificial intelligence is to use it to benefit mankind, rather than to bring risks – potentially existential risks – upon mankind. Therefore, the vast majority of people have the right to know about the potential risks of AI, and developers have the obligation to ensure that AI does not pose existential risks to humanity, or at least to minimize the possibility of such risks through stakeholders. It is difficult for a few people to change existing trends, but when a few people take the first step to raise public awareness, more and more people will ultimately participate in changing the status quo.
原文
问：关注到您在最近的两份AI声明上都有签名，您为什么会签署？
答： 两次声明都意识到目前人工智能发展过程中的风险和失控的可能性，但应对的方式有显著的差别。《暂停人工智能巨模型实验》号召通过暂停能力超越GPT-4的人工智能巨模型的研究，优先为人工智能设计并实现安全框架。新的《人工智能风险声明》号召“减轻人工智能灭绝的风险应该与流行病和核战争等其他社会规模的风险一起成为全球优先事项”更深度和直接地表达了签名者对人工智能给人类带来潜在生存风险的担忧和应采取的行动。我在这个问题的认知与这样的观点接近，因此在新声明正式发布前就签署了。绝大多数人发展人工智能的愿景，我想应当是用人工智能造福人类，而并非是给人类带来风险，甚至是生存风险。因此绝大多数人有权利知道人工智能的潜在风险，研发者有义务确保人工智能不给人类带来生存风险，至少要通过利益相关方最小化这种风险的可能性。少数人很难改变趋势，但少数人首先站出来提升公众的意识，最终参与改变现状的就会是多数人。
问：关注到您在最近的两份AI声明上都有签名，您为什么会签署？
答： 两次声明都意识到目前人工智能发展过程中的风险和失控的可能性，但应对的方式有显著的差别。《暂停人工智能巨模型实验》号召通过暂停能力超越GPT-4的人工智能巨模型的研究，优先为人工智能设计并实现安全框架。新的《人工智能风险声明》号召“减轻人工智能灭绝的风险应该与流行病和核战争等其他社会规模的风险一起成为全球优先事项”更深度和直接地表达了签名者对人工智能给人类带来潜在生存风险的担忧和应采取的行动。我在这个问题的认知与这样的观点接近，因此在新声明正式发布前就签署了。绝大多数人发展人工智能的愿景，我想应当是用人工智能造福人类，而并非是给人类带来风险，甚至是生存风险。因此绝大多数人有权利知道人工智能的潜在风险，研发者有义务确保人工智能不给人类带来生存风险，至少要通过利益相关方最小化这种风险的可能性。少数人很难改变趋势，但少数人首先站出来提升公众的意识，最终参与改变现状的就会是多数人。
Q: Could AI really bring about risks of extinction similar to pandemics and nuclear war? Is the current understanding of AI's risks overstated?
Zeng: The common features of the potential existential risks that pandemics, nuclear war, and artificial intelligence may bring to mankind are that they are wide-ranging, concern the interests of all mankind, and have widespread lethality. More importantly, they are all difficult to predict in advance. Regarding the risks of AI, there are at least two possibilities. One concerns AI in the long term. When artificial general intelligence (AGI) or superintelligence emerges, because the intelligence level may be far beyond humans, it will see humans as humans see ants. Many people believe that superintelligence will compete with humans for resources, and even endanger human survival. The other concerns AI in the near term, which is more pressing. Since today’s AI has no real ability to understand and is not truly intelligent, it will make mistakes that humans would not make in ways that are difficult to anticipate. When a certain action threatens the survival of humankind, AI would not understand what humanity is, what life and death are, nor what is existential risk. When this situation occurs, it is highly likely to threaten human survival. Some also hold the view that artificial intelligence can take advantage of human flaws to cause a fatal crisis to human survival. For example, it could use and intensify hostility and hatred, prejudice and misunderstanding between humans. Such artificial intelligence would not even need to reach the level of AGI to pose an existential risk to human beings. In addition, this kind of AI is likely to be maliciously used, misused, and abused by people. That risk is difficult to anticipate and control, especially as the recent progress of artificial intelligence allows AI to use internet-scale data and information. False information generated by generative AI can greatly reduce social trust in the technology. And network communication has made everything interconnected, which amplifies the above risks to a global scale. If we begin to study how to avoid the challenges of long-term artificial intelligence now, its risks can still be manageable, but the risks from near-term artificial intelligence are more urgent. Valuing and managing the safety and security risks of AI does not hinder the development and application of AI, rather, it is a way to ensure the steady and sustainable development of the technology. AI is undoubtedly a driver of social progress, but this does not mean that AI is without potential risks, or that those potential risks can be ignored due to the need to maximize the benefits from AI. The purpose of both statements is not to impede the development of AI, but rather to explore avenues for the steady and sustainable development of AI.
Zeng: The common features of the potential existential risks that pandemics, nuclear war, and artificial intelligence may bring to mankind are that they are wide-ranging, concern the interests of all mankind, and have widespread lethality. More importantly, they are all difficult to predict in advance. Regarding the risks of AI, there are at least two possibilities. One concerns AI in the long term. When artificial general intelligence (AGI) or superintelligence emerges, because the intelligence level may be far beyond humans, it will see humans as humans see ants. Many people believe that superintelligence will compete with humans for resources, and even endanger human survival. The other concerns AI in the near term, which is more pressing. Since today’s AI has no real ability to understand and is not truly intelligent, it will make mistakes that humans would not make in ways that are difficult to anticipate. When a certain action threatens the survival of humankind, AI would not understand what humanity is, what life and death are, nor what is existential risk. When this situation occurs, it is highly likely to threaten human survival. Some also hold the view that artificial intelligence can take advantage of human flaws to cause a fatal crisis to human survival. For example, it could use and intensify hostility and hatred, prejudice and misunderstanding between humans. Such artificial intelligence would not even need to reach the level of AGI to pose an existential risk to human beings. In addition, this kind of AI is likely to be maliciously used, misused, and abused by people. That risk is difficult to anticipate and control, especially as the recent progress of artificial intelligence allows AI to use internet-scale data and information. False information generated by generative AI can greatly reduce social trust in the technology. And network communication has made everything interconnected, which amplifies the above risks to a global scale. If we begin to study how to avoid the challenges of long-term artificial intelligence now, its risks can still be manageable, but the risks from near-term artificial intelligence are more urgent. Valuing and managing the safety and security risks of AI does not hinder the development and application of AI, rather, it is a way to ensure the steady and sustainable development of the technology. AI is undoubtedly a driver of social progress, but this does not mean that AI is without potential risks, or that those potential risks can be ignored due to the need to maximize the benefits from AI. The purpose of both statements is not to impede the development of AI, but rather to explore avenues for the steady and sustainable development of AI.
问：AI真的会有类似流行病和核战争的灭绝风险？当前对AI风险的认识是否夸大？
答：大流行病、核战争与人工智能可能给人类带来的潜在生存风险的共性是波及范围广，关乎全人类的利益，甚至具有广泛致命性，更关键的是都难以提前预测。关于人工智能的风险，有至少两种可能，一种是对远期人工智能的担忧。当通用人工智能和超级智能到来时，由于智力水平可能远超人类，将视人类如同人类视蚂蚁，很多人据此认为超级智能将与人类争夺资源，甚至危及到人类的生存。另一种是针对近期人工智能的担忧，这更为紧迫。由于现在的人工智能没有真正的理解能力，也不是真正的智能，因此会以人类难以预期的方式犯人不会犯的错误。当某种操作会威胁到人类的生存的时候，人工智能既不理解什么是人类，什么是生死，也不理解什么是生存风险。当这种情况发生时，极有可能威胁到人类的生存。也有观点认为，人工智能可以利用人类的弱点对人类的生存造成致命危机，例如利用和加剧人类之间的敌对和仇视、偏见和误解，而这样的人工智能甚至不需要达到通用人工智能的阶段就有可能对人类造成生存风险。加之这种人工智能很有可能被人恶意利用、误用和滥用，而风险几乎难以预期和控制，特别是近期的人工智能进展使得其能够利用互联网规模的数据与信息，生成式人工智能产生的虚假信息极大降低了社会信任，网络通信又已使万物互联，可以使相关风险在世界规模放大。远期人工智能的挑战，我们如果从现在开始研究如何规避，其风险尚有可能应对，但近期人工智能的风险则显得更为紧迫。重视和管控人工智能的安全风险不是阻碍人工智能的发展与应用，而是确保人工智能技术稳健发展。人工智能无疑是社会进步的推进器，然而这并不意味着人工智能没有潜在风险，或者是由于最大化人工智能益处的需求，就可以忽略人工智能的潜在风险。前述两封声明的目的都不是阻碍人工智能的发展，恰恰是在探索人工智能稳健发展的途径。
答：大流行病、核战争与人工智能可能给人类带来的潜在生存风险的共性是波及范围广，关乎全人类的利益，甚至具有广泛致命性，更关键的是都难以提前预测。关于人工智能的风险，有至少两种可能，一种是对远期人工智能的担忧。当通用人工智能和超级智能到来时，由于智力水平可能远超人类，将视人类如同人类视蚂蚁，很多人据此认为超级智能将与人类争夺资源，甚至危及到人类的生存。另一种是针对近期人工智能的担忧，这更为紧迫。由于现在的人工智能没有真正的理解能力，也不是真正的智能，因此会以人类难以预期的方式犯人不会犯的错误。当某种操作会威胁到人类的生存的时候，人工智能既不理解什么是人类，什么是生死，也不理解什么是生存风险。当这种情况发生时，极有可能威胁到人类的生存。也有观点认为，人工智能可以利用人类的弱点对人类的生存造成致命危机，例如利用和加剧人类之间的敌对和仇视、偏见和误解，而这样的人工智能甚至不需要达到通用人工智能的阶段就有可能对人类造成生存风险。加之这种人工智能很有可能被人恶意利用、误用和滥用，而风险几乎难以预期和控制，特别是近期的人工智能进展使得其能够利用互联网规模的数据与信息，生成式人工智能产生的虚假信息极大降低了社会信任，网络通信又已使万物互联，可以使相关风险在世界规模放大。远期人工智能的挑战，我们如果从现在开始研究如何规避，其风险尚有可能应对，但近期人工智能的风险则显得更为紧迫。重视和管控人工智能的安全风险不是阻碍人工智能的发展与应用，而是确保人工智能技术稳健发展。人工智能无疑是社会进步的推进器，然而这并不意味着人工智能没有潜在风险，或者是由于最大化人工智能益处的需求，就可以忽略人工智能的潜在风险。前述两封声明的目的都不是阻碍人工智能的发展，恰恰是在探索人工智能稳健发展的途径。
Q: You mentioned the need to build moral artificial intelligence, yet AI has no moral awareness. How can we ensure that AI gets developed safely?
Zeng: Human morality has an innate basis on which ethics in a wider sense can be built, enabling moral reasoning and decision-making. However, the current approach for making AI models ethical is to bind them with rule-based ethical principles and align such intelligent information processing systems with human values and behaviors. This is like building a castle in the air. Without moral intuition as a foundation, without real understanding, it is impossible to realize true ethics and morality. Only when AI is endowed with a certain degree of self-awareness, only after cognitive empathy, emotional empathy, and altruism are realized on this basis, and only when moral intuition is realized to a certain extent on that basis, is it possible to realize truly moral AI. Therefore, it is necessary to take inspiration from the human brain and human evolution to build ethical AI. It will certainly be an extremely arduous path for AI development, but I see no shortcut.
Zeng: Human morality has an innate basis on which ethics in a wider sense can be built, enabling moral reasoning and decision-making. However, the current approach for making AI models ethical is to bind them with rule-based ethical principles and align such intelligent information processing systems with human values and behaviors. This is like building a castle in the air. Without moral intuition as a foundation, without real understanding, it is impossible to realize true ethics and morality. Only when AI is endowed with a certain degree of self-awareness, only after cognitive empathy, emotional empathy, and altruism are realized on this basis, and only when moral intuition is realized to a certain extent on that basis, is it possible to realize truly moral AI. Therefore, it is necessary to take inspiration from the human brain and human evolution to build ethical AI. It will certainly be an extremely arduous path for AI development, but I see no shortcut.
问：您提到要构建有道德的人工智能，AI没有道德意识，要保证发展安全的话该如何解决？
答：人类的道德具有内生基础，在此基础上通过习得更广泛意义的伦理道德，从而进行道德推理与决策。但目前的人工智能模型合乎伦理道德的做法是用规则化的伦理原则约束，使智能信息处理系统对齐人类价值观与行为。这就好似建构空中楼阁，没有道德直觉作为根基，没有真正的理解，不可能实现真正意义的伦理道德。只有为人工智能赋予一定程度的自我感知，在此基础上实现认知共情、情感共情、利他行为，以此为基础实现一定程度的道德直觉，才有可能实现真正意义的有道德的人工智能。因此需要在人脑和人类演化的启发中，构建脑与心智启发的有道德的人工智能。这必然是一条极其困难而艰辛的发展道路，但我看不到其他捷径。
答：人类的道德具有内生基础，在此基础上通过习得更广泛意义的伦理道德，从而进行道德推理与决策。但目前的人工智能模型合乎伦理道德的做法是用规则化的伦理原则约束，使智能信息处理系统对齐人类价值观与行为。这就好似建构空中楼阁，没有道德直觉作为根基，没有真正的理解，不可能实现真正意义的伦理道德。只有为人工智能赋予一定程度的自我感知，在此基础上实现认知共情、情感共情、利他行为，以此为基础实现一定程度的道德直觉，才有可能实现真正意义的有道德的人工智能。因此需要在人脑和人类演化的启发中，构建脑与心智启发的有道德的人工智能。这必然是一条极其困难而艰辛的发展道路，但我看不到其他捷径。
Q: How do you give AI a sense of morality? Do human morals and ethical values apply to it?
Zeng: Morality cannot be instilled and needs to be understood based on moral intuition rather than a set of operational rules. The first thing we need to give AI is the ability to understand, so that it can generate moral intuition, perform effective moral reasoning, and make moral decisions. Moral concepts and ethical values for humans are constructed from human society, and human beings naturally hope that AI conforms to human values and ethical frameworks. But this is inevitably far from enough. Humans’ own views are changing and being re-evaluated all the time. Artificial intelligence is a new medium of exploration and could even assist in improving human values. If AI is allowed to start interacting with the world entirely anew, it will inevitably form a system of values that is different from human values and moral concepts. But that is certainly not what humans want. Therefore, humans hope that the values system of AI can be aligned with that of human beings. But at the same time, human beings should also take inspiration from interaction with AI to improve upon our value system and ethics.
Zeng: Morality cannot be instilled and needs to be understood based on moral intuition rather than a set of operational rules. The first thing we need to give AI is the ability to understand, so that it can generate moral intuition, perform effective moral reasoning, and make moral decisions. Moral concepts and ethical values for humans are constructed from human society, and human beings naturally hope that AI conforms to human values and ethical frameworks. But this is inevitably far from enough. Humans’ own views are changing and being re-evaluated all the time. Artificial intelligence is a new medium of exploration and could even assist in improving human values. If AI is allowed to start interacting with the world entirely anew, it will inevitably form a system of values that is different from human values and moral concepts. But that is certainly not what humans want. Therefore, humans hope that the values system of AI can be aligned with that of human beings. But at the same time, human beings should also take inspiration from interaction with AI to improve upon our value system and ethics.
问：AI是机器，如何让它有道德感？人类的道德观念和伦理价值对它是适用的吗？
答：道德无法被灌输，需要基于道德直觉的理解，而不是处理规则。我们首先需要赋予人工智能的是理解能力，才有可能产生道德直觉，进行有效的道德推理与道德决策。人类的道德观念和伦理价值是为人类社会而建构，人类自然希望人工智能合乎人类的价值观和伦理框架，但这必然是远远不够的。人类自身的看法都在发生着变化并进行重新认知，人工智能是探索的新载体，甚至可以辅助人类完善人类的价值体系。如果让AI完全重新开始与世界互动，必然会形成与人类价值与道德观念有所差异的体系，但这一定不是人类所期望的。所以人类希望人工智能的价值系统能够与人类对齐，但同时人类也应当在与人工智能互动过程中有所启发，以辅助人类价值系统与伦理道德观的完善。
答：道德无法被灌输，需要基于道德直觉的理解，而不是处理规则。我们首先需要赋予人工智能的是理解能力，才有可能产生道德直觉，进行有效的道德推理与道德决策。人类的道德观念和伦理价值是为人类社会而建构，人类自然希望人工智能合乎人类的价值观和伦理框架，但这必然是远远不够的。人类自身的看法都在发生着变化并进行重新认知，人工智能是探索的新载体，甚至可以辅助人类完善人类的价值体系。如果让AI完全重新开始与世界互动，必然会形成与人类价值与道德观念有所差异的体系，但这一定不是人类所期望的。所以人类希望人工智能的价值系统能够与人类对齐，但同时人类也应当在与人工智能互动过程中有所启发，以辅助人类价值系统与伦理道德观的完善。
Q: What kind of human-machine relationship do you hope to see in the future? Is the biggest bottleneck human or AI?
Zeng: In the future, artificial intelligence may have more characteristics of life, and the level of intelligence may fully reach or even surpass that of human beings. Humans hope that AI will harmoniously coexist with human beings as partners. AI is a mirror onto humanity, and in the process of building AI we should constantly reflect on the relationship of coexistence between humans and other life forms. The superintelligence of the future may see humans as humans see ants today, and if humans can’t treat other types of life with kindness, why should the superintelligence of the future treat humans with kindness? The biggest bottleneck to whether humans and AI can coexist in the future lies in humans, not AI. If superintelligence truly surpasses human beings in all aspects of intelligence, then it should be super altruistic, super moral. In the face of such intelligent life, human morality needs to advance and evolve.
Zeng: In the future, artificial intelligence may have more characteristics of life, and the level of intelligence may fully reach or even surpass that of human beings. Humans hope that AI will harmoniously coexist with human beings as partners. AI is a mirror onto humanity, and in the process of building AI we should constantly reflect on the relationship of coexistence between humans and other life forms. The superintelligence of the future may see humans as humans see ants today, and if humans can’t treat other types of life with kindness, why should the superintelligence of the future treat humans with kindness? The biggest bottleneck to whether humans and AI can coexist in the future lies in humans, not AI. If superintelligence truly surpasses human beings in all aspects of intelligence, then it should be super altruistic, super moral. In the face of such intelligent life, human morality needs to advance and evolve.
问：您期待的未来人机关系是什么样的？面临的最大瓶颈是在于人类还是在于AI？
曾毅：未来的人工智能可能会具有更多生命的特征，智力水平有可能会全面达到甚至超越人类，而人类总还是希望人工智能可以作为伙伴与人类和谐共生。人工智能是人类的一面镜子，我们在构建人工智能的过程中应不断反思人类与其他生命之间的关系和相处之道。未来的超级智能视人类可能如现在的人类视蚂蚁，而若人类不能善待其他类型的生命，未来的超级智能又有何理由善待人类呢？人与人工智能未来是否能够共生共存最大的瓶颈在于人类，而非人工智能。如果超级智能真的是在智慧水平上全面超越人类，那么应该是超级利他、超级道德的，在面对这样的智慧生命时，人类的道德需要加速演化。
曾毅：未来的人工智能可能会具有更多生命的特征，智力水平有可能会全面达到甚至超越人类，而人类总还是希望人工智能可以作为伙伴与人类和谐共生。人工智能是人类的一面镜子，我们在构建人工智能的过程中应不断反思人类与其他生命之间的关系和相处之道。未来的超级智能视人类可能如现在的人类视蚂蚁，而若人类不能善待其他类型的生命，未来的超级智能又有何理由善待人类呢？人与人工智能未来是否能够共生共存最大的瓶颈在于人类，而非人工智能。如果超级智能真的是在智慧水平上全面超越人类，那么应该是超级利他、超级道德的，在面对这样的智慧生命时，人类的道德需要加速演化。
Other Authors
