About the author Yi ZENG is a Professor and Director at the Brain-inspired Cognitive Intelligence Lab, and the International Research Center for AI Ethics and Governance, both at the Institute of Automation, Chinese Academy of Sciences. He is the founding Director of Center for Long-term AI, and the AI for Sustainable Development Goals Cooperation Network. He is a board member for the National Governance Committee of Next Generation Artificial Intelligence, China. He is an member of UN High-level Advisory Body on AI, an Expert in the Ad Hoc Expert Group on AI Ethics, UNESCO. He is in the WHO Expert Group on the Ethics/Governance of Artificial Intelligence for Health. His major research interests focus on Brain-inspired Artificial Intelligence, AI Ethics and Governance, AI Safety, and AI for Sustainable Development.


On July 18, 2023, the UN Security Council sat down for the first time to discuss the potential threats of AI to world peace and security. Yi Zeng delivered a briefing titled “
Opportunities and Risks for International Peace and Security,” suggesting that “in the short-term and the long-term, the risk of AI replacing and causing the extinction of humankind will be present” and that “in the long-term, we haven’t given superintelligence any practical reasons why they should protect humankind.”

Yi Zeng has signed the open letter to Pause Giant AI Experiments that “call[s] on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.” Yi Zeng is also among the signatories of the Statement on AI Risk from the Center for AI Safety, which states that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

The following is a translation of a Chinese media interview (搜狐科技《思想大爆炸——对话科学家》栏目第六期) during which he explained his support for both initiatives.

Yi Zeng has also conducted surveys investigating Chinese views on the aforementioned open letter and on whether we can and should develop strong AI, with results available in English.

▶ Cite Our TranslationConcordia AI. “Yi Zeng — Chinese Perspectives on AI Safety.” Chineseperspectives.ai, 29 Mar. 2024, chineseperspectives.ai/Yi-Zeng.

▶ Cite This Work 曾毅(2023). “中科院自动化所曾毅:未来AI智力水平或全面超越人类,能否共生最大的瓶颈在人”. 在思想大爆炸——对话科学家栏目上的发言. https://www.sohu.com/a/686209288_115565


Q: We noticed that you signed two recent AI statements. Why did you sign them?

Zeng: Both statements recognize the risks in the current development of artificial intelligence and the potential for losing control, but significant differences exist in how the two letters respond to the problem. “Pause Giant AI experiments” calls for pausing research on AI systems more powerful than GPT-4 and prioritizes the design and implementation of a safety protocol for AI. The new “Statement on AI Risk” pronounces that “mitigating the risk of AI extinction should be a global priority alongside other society-scale risks such as pandemics and nuclear war,” which more deeply and directly expresses the signatories’ concerns about the potential existential risks that AI poses to humanity, as well as actions to be taken. My understanding of the issue is close to such views, so I signed the new statement before it was even officially released. I believe the vision shared by the vast majority of people working to develop artificial intelligence is to use it to benefit mankind, rather than to bring risks – potentially existential risks – upon mankind. Therefore, the vast majority of people have the right to know about the potential risks of AI, and developers have the obligation to ensure that AI does not pose existential risks to humanity, or at least to minimize the possibility of such risks through stakeholders. It is difficult for a few people to change existing trends, but when a few people take the first step to raise public awareness, more and more people will ultimately participate in changing the status quo.


答: 两次声明都意识到目前人工智能发展过程中的风险和失控的可能性,但应对的方式有显著的差别。《暂停人工智能巨模型实验》号召通过暂停能力超越GPT-4的人工智能巨模型的研究,优先为人工智能设计并实现安全框架。新的《人工智能风险声明》号召“减轻人工智能灭绝的风险应该与流行病和核战争等其他社会规模的风险一起成为全球优先事项”更深度和直接地表达了签名者对人工智能给人类带来潜在生存风险的担忧和应采取的行动。我在这个问题的认知与这样的观点接近,因此在新声明正式发布前就签署了。绝大多数人发展人工智能的愿景,我想应当是用人工智能造福人类,而并非是给人类带来风险,甚至是生存风险。因此绝大多数人有权利知道人工智能的潜在风险,研发者有义务确保人工智能不给人类带来生存风险,至少要通过利益相关方最小化这种风险的可能性。少数人很难改变趋势,但少数人首先站出来提升公众的意识,最终参与改变现状的就会是多数人。

Q: Could AI really bring about risks of extinction similar to pandemics and nuclear war? Is the current understanding of AI's risks overstated?

Zeng: The common features of the potential existential risks that pandemics, nuclear war, and artificial intelligence may bring to mankind are that they are wide-ranging, concern the interests of all mankind, and have widespread lethality. More importantly, they are all difficult to predict in advance. Regarding the risks of AI, there are at least two possibilities. One concerns AI in the long term. When artificial general intelligence (AGI) or superintelligence emerges, because the intelligence level may be far beyond humans, it will see humans as humans see ants. Many people believe that superintelligence will compete with humans for resources, and even endanger human survival. The other concerns AI in the near term, which is more pressing. Since today’s AI has no real ability to understand and is not truly intelligent, it will make mistakes that humans would not make in ways that are difficult to anticipate. When a certain action threatens the survival of humankind, AI would not understand what humanity is, what life and death are, nor what is existential risk. When this situation occurs, it is highly likely to threaten human survival. Some also hold the view that artificial intelligence can take advantage of human flaws to cause a fatal crisis to human survival. For example, it could use and intensify hostility and hatred, prejudice and misunderstanding between humans. Such artificial intelligence would not even need to reach the level of AGI to pose an existential risk to human beings. In addition, this kind of AI is likely to be maliciously used, misused, and abused by people. That risk is difficult to anticipate and control, especially as the recent progress of artificial intelligence allows AI to use internet-scale data and information. False information generated by generative AI can greatly reduce social trust in the technology. And network communication has made everything interconnected, which amplifies the above risks to a global scale. If we begin to study how to avoid the challenges of long-term artificial intelligence now, its risks can still be manageable, but the risks from near-term artificial intelligence are more urgent. Valuing and managing the safety and security risks of AI does not hinder the development and application of AI, rather, it is a way to ensure the steady and sustainable development of the technology. AI is undoubtedly a driver of social progress, but this does not mean that AI is without potential risks, or that those potential risks can be ignored due to the need to maximize the benefits from AI. The purpose of both statements is not to impede the development of AI, but rather to explore avenues for the steady and sustainable development of AI.


Q: You mentioned the need to build moral artificial intelligence, yet AI has no moral awareness.  How can we ensure that AI gets developed safely?

Zeng: Human morality has an innate basis on which ethics in a wider sense can be built, enabling moral reasoning and decision-making. However, the current approach for making AI models ethical is to bind them with rule-based ethical principles and align such intelligent information processing systems with human values and behaviors. This is like building a castle in the air. Without moral intuition as a foundation, without real understanding, it is impossible to realize true ethics and morality. Only when AI is endowed with a certain degree of self-awareness, only after cognitive empathy, emotional empathy, and altruism are realized on this basis, and only when moral intuition is realized to a certain extent on that basis, is it possible to realize truly moral AI. Therefore, it is necessary to take inspiration from the human brain and human evolution to build ethical AI. It will certainly be an extremely arduous path for AI development, but I see no shortcut.


Q: How do you give AI a sense of morality? Do human morals and ethical values apply to it?

Zeng: Morality cannot be instilled and needs to be understood based on moral intuition rather than a set of operational rules. The first thing we need to give AI is the ability to understand, so that it can generate moral intuition, perform effective moral reasoning, and make moral decisions. Moral concepts and ethical values for humans are constructed from human society, and human beings naturally hope that AI conforms to human values and ethical frameworks. But this is inevitably far from enough. Humans’ own views are changing and being re-evaluated all the time. Artificial intelligence is a new medium of exploration and could even assist in improving human values. If AI is allowed to start interacting with the world entirely anew, it will inevitably form a system of values that is different from human values and moral concepts. But that is certainly not what humans want. Therefore, humans hope that the values system of AI can be aligned with that of human beings. But at the same time, human beings should also take inspiration from interaction with AI to improve upon our value system and ethics.


Q: What kind of human-machine relationship do you hope to see in the future? Is the biggest bottleneck human or AI?

Zeng: In the future, artificial intelligence may have more characteristics of life, and the level of intelligence may fully reach or even surpass that of human beings. Humans hope that AI will harmoniously coexist with human beings as partners. AI is a mirror onto humanity, and in the process of building AI we should constantly reflect on the relationship of coexistence between humans and other life forms. The superintelligence of the future may see humans as humans see ants today, and if humans can’t treat other types of life with kindness, why should the superintelligence of the future treat humans with kindness? The biggest bottleneck to whether humans and AI can coexist in the future lies in humans, not AI. If superintelligence truly surpasses human beings in all aspects of intelligence, then it should be super altruistic, super moral. In the face of such intelligent life, human morality needs to advance and evolve.


Other Authors

Chinese Perspectives
on Existential Risks from AI

Contact us at: info@concordia-ai.com