https://chineseperspectives.ai Chinese Perspectives on AI Safety Tue, 12 Mar 2024 03:59:30 +0000 https://chineseperspectives.ai en https://chineseperspectives.ai/Open-title Sat, 06 May 2023 18:20:11 +0000 Chinese Perspectives on AI Safety https://chineseperspectives.ai/Open-title Chinese Perspectiveson &nbsp;AI Safety Raising awareness of discussions on AI safety and risks in China https://chineseperspectives.ai/Technical-Perspectives Sat, 06 May 2023 16:53:41 +0000 Chinese Perspectives on AI Safety https://chineseperspectives.ai/Technical-Perspectives This section collates excerpts from the works of leading Chinese AI scientists about catastrophic risks from both highly capable general-purpose AI systems and narrow AI models which could cause harm in specific domains, which match or exceed the capabilities present in today’s most advanced AI systems. Key concerns featured in the discussions include loss of human control over AI systems and catastrophic risks from advanced AI systems. Wen GAO︎︎︎Director of Peng Cheng Laboratory Dean of the School of Information Science and Technology at Peking UniversityAcademician of the Chinese Academy of EngineeringMember of China’s Science and Technology Ethics Committee Deputy Chief of the Advisory Group of the Ministry of Education’s AI Technology Innovation Expert Group One of 27 members of the Ministry of Science and Technology’s Next Generation Artificial Intelligence Strategic Advisory CommitteeFormerly Deputy Director of the National Natural Science Foundation of China <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/f1826d016d72655d578dd2ce9192cef78febba10057f69499d01c991bf9dd3d4/c.jpg" data-mid="179709170" border="0" data-scale="60" src="https://freight.cargo.site/w/900/i/f1826d016d72655d578dd2ce9192cef78febba10057f69499d01c991bf9dd3d4/c.jpg" /> Bo ZHANG︎︎︎ Honorary Dean of AI Institute and Professor of Computer Science at Tsinghua University Academician of the Chinese Academy of Sciences&nbsp; Chief Scientist at AI safety/security start-up RealAITechnical consultant for Microsoft Asia Research Institute <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" data-mid="179709378" border="0" data-scale="60" src="https://freight.cargo.site/w/900/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" /> Andrew YAO ︎︎︎ Recepient of the Turing Award in 2000Recepient of the Kyoto Award in Advanced Technology in 2021Academician of Chinese Academy of SciencesMember of U.S. National Academy of SciencesDean of Institute for Interdisciplinary Information Sciences (IIIS) in Tsinghua University <img width="778" height="934" width_o="778" height_o="934" data-src="https://freight.cargo.site/t/original/i/879a6eea0588e12ecdf4808b1ff339da1c4edf5f031deaf45b8d14408916f46d/IMG_6395.JPG" data-mid="188668546" border="0" data-scale="60" src="https://freight.cargo.site/w/778/i/879a6eea0588e12ecdf4808b1ff339da1c4edf5f031deaf45b8d14408916f46d/IMG_6395.JPG" /> Ya-Qin ZHANG︎︎︎Academician of Chinese Academy of Engineering, Tsinghua University Chair Professor of Intelligent ScienceDirector of the Tsinghua Institute for AI Industry Research Fellow of the American Academy of Arts and Sciences Foreign fellow of Australian Academy of Technology Sciences and EngineeringPreviously served as President of Baidu, Corporate Senior Vice President of Microsoft, Chairman of Microsoft Research Asia, Managing Director of Microsoft Research Asia, and Chairman of Microsoft China R&amp;D Group <img width="302" height="362" width_o="302" height_o="362" data-src="https://freight.cargo.site/t/original/i/d9b9f5d5e8b90ea849b5a9b7169520ebc1f2b9a5da94db47fcc26eef8bfe8431/IMG_6393.JPG" data-mid="188669096" border="0" data-scale="60" src="https://freight.cargo.site/w/302/i/d9b9f5d5e8b90ea849b5a9b7169520ebc1f2b9a5da94db47fcc26eef8bfe8431/IMG_6393.JPG" /> Yi ZENG︎︎︎Researcher at the Institute of Automation, Chinese Academy of SciencesDeputy Director of the Brain-Inspired Intelligence Research CenterCo-Director of the China-UK Research Centre for AI Ethics and GovernanceMember of the National Next Generation AI Governance Expert Committee in ChinaExpert at UNESCO Ad Hoc Expert Group for the Recommendation on the Ethics of Artificial Intelligence &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<img width="708" height="849" width_o="708" height_o="849" data-src="https://freight.cargo.site/t/original/i/c8dbaef948a56b22c9b51290ae90586f1ab36f10463858ea91e2729d17f9025b/WechatIMG2235-2.jpeg" data-mid="191536529" border="0" data-scale="60" src="https://freight.cargo.site/w/708/i/c8dbaef948a56b22c9b51290ae90586f1ab36f10463858ea91e2729d17f9025b/WechatIMG2235-2.jpeg" /> Tiejun HUANG︎︎︎Professor at the School of Computer Science, Peking UniversityDirector of Beijing Academy of Artificial Intelligence (BAAI) Fellow of the China Computer Federation, Chinese Association for Artificial Intelligence, and China Society of Image and GraphicsRecipient of the National Science Fund for Distinguished Young Scholars, Changjiang Distinguished Professorship, and National Talent Program for Technological Innovation Leadership <img width="720" height="864" width_o="720" height_o="864" data-src="https://freight.cargo.site/t/original/i/d52175e0957fa3ad717ad15be56f22a71cc3dba04fef40f4835ed9a469f8b428/23a22ce9da08d46b614a1a8c09e15a1f.jpg" data-mid="188669203" border="0" data-scale="60" src="https://freight.cargo.site/w/720/i/d52175e0957fa3ad717ad15be56f22a71cc3dba04fef40f4835ed9a469f8b428/23a22ce9da08d46b614a1a8c09e15a1f.jpg" /> Binxing FANG︎︎︎Cybersecurity expertAcademician of the Chinese Academy of Engineering, Honorary director of the National Computer Network and Information Security Management Center Director of the Network Security Technology National Engineering Laboratory, Guangzhou UniversityHonorary president of the Institute of Advanced Cyberspace Technology <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/d078330bdbdcb3b8b48f904739bb45797248f658c7e2429bb26415707a85ae84/f.jpg" data-mid="188669266" border="0" data-scale="60" src="https://freight.cargo.site/w/900/i/d078330bdbdcb3b8b48f904739bb45797248f658c7e2429bb26415707a85ae84/f.jpg" /> https://chineseperspectives.ai/Humanities-Perspectives Sat, 06 May 2023 17:08:35 +0000 Chinese Perspectives on AI Safety https://chineseperspectives.ai/Humanities-Perspectives This section collates excerpts from the work of influential Chinese philosophers and humanistic thinkers about risks that threaten outright human extinction or leave humanity in a drastically inferior state. All of the curated scholars recognize the development of powerful technologies, especially advanced artificial intelligence, as among the most serious existential risks that humanity faces. Huaihong HE ︎︎︎Professor of the Department of Philosophy and Director of the Ethics Teaching and Research Section, Peking University&nbsp;Author of Social Ethics in a Changing China - Moral Decay or Ethical Awakening? <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/4ffaf416714dab827583c88eaaf4d1a1bdae4298dcd7f9d58d06a36e5b6f510e/a.jpg" data-mid="179709930" border="0" data-scale="60" src="https://freight.cargo.site/w/900/i/4ffaf416714dab827583c88eaaf4d1a1bdae4298dcd7f9d58d06a36e5b6f510e/a.jpg" /> Zhouying JIN ︎︎︎ Senior researcher and professor, Chinese Academy of Social Sciences (CASS)&nbsp; President and founder of the Beijing Academy of Soft Technology&nbsp; Author of Global Technological Change: From Hard Technology to Soft Technology&nbsp; <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/673e2d5821ae25c1a759132e93d83c3c07f95a42969f973f97187ed3738ddf61/b.jpg" data-mid="179710060" border="0" data-scale="60" src="https://freight.cargo.site/w/900/i/673e2d5821ae25c1a759132e93d83c3c07f95a42969f973f97187ed3738ddf61/b.jpg" /> https://chineseperspectives.ai/Policy Tue, 12 Mar 2024 03:59:30 +0000 Chinese Perspectives on AI Safety https://chineseperspectives.ai/Policy This section presents key writings from influential Chinese AI policy advisors. Their work focuses on policy prescriptions for leveraging AI's potential and addressing its risks. Highlighting China's goals and concerns in AI development, this part offers insight into the national policy-making efforts. Lan XUE ︎︎︎Cheung Kong Chair Distinguished Professor and Dean of Schwarzman College at Tsinghua University&nbsp;Director of the China Institute for Science and Technology Policy <img width="602" height="852" width_o="602" height_o="852" data-src="https://freight.cargo.site/t/original/i/a724350613d548cfe523d3107dcc41ccec85bd60335bc75b447a3c1b4912ff12/Screenshot-2024-03-07-at-5.19.34PM.png" data-mid="206571836" border="0" data-scale="51" src="https://freight.cargo.site/w/602/i/a724350613d548cfe523d3107dcc41ccec85bd60335bc75b447a3c1b4912ff12/Screenshot-2024-03-07-at-5.19.34PM.png" /> Linghan ZHANG&nbsp;︎︎︎ Professor at the Institute of Data Law and doctoral supervisor at China University of Political Science and Law&nbsp;Visiting scholar at Cornell University <img width="770" height="1014" width_o="770" height_o="1014" data-src="https://freight.cargo.site/t/original/i/f2ed083b59f847ede0f232f048f9673645ac006e828afd450ce10ceac6969dce/Screenshot-2024-03-12-at-5.46.39PM.png" data-mid="206570809" border="0" data-scale="50" src="https://freight.cargo.site/w/770/i/f2ed083b59f847ede0f232f048f9673645ac006e828afd450ce10ceac6969dce/Screenshot-2024-03-12-at-5.46.39PM.png" /> Qiqi GAO︎︎︎Director of the Artificial Intelligence and Big Data Index Research Institute and the Institute of Political Science at East China University of Political Science and Law <img width="607" height="800" width_o="607" height_o="800" data-src="https://freight.cargo.site/t/original/i/181cd526fc62a46e8fa3bb730366e9cf415ea8bdfd852cb3d52fbcff34ce2320/WechatIMGfce73053bf3ebc761994cdcbe84a1970.jpeg" data-mid="210554336" border="0" data-scale="49" src="https://freight.cargo.site/w/607/i/181cd526fc62a46e8fa3bb730366e9cf415ea8bdfd852cb3d52fbcff34ce2320/WechatIMGfce73053bf3ebc761994cdcbe84a1970.jpeg" /> Xiuquan LI︎︎︎Deputy Director of Center for Artifcial Intelligence at the Institute of Scientific and Technical Information of ChinaDeputy Director of the Ministry of Science and Technology‘s Next Generation AI Development Research Center <img width="1392" height="1660" width_o="1392" height_o="1660" data-src="https://freight.cargo.site/t/original/i/8ae984b47ba387f4e19ff3d35375838364048523e2d093d1e45f09b42d4bb3ed/Screenshot-2024-03-12-at-6.31.49PM.png" data-mid="206572326" border="0" data-scale="54" src="https://freight.cargo.site/w/1000/i/8ae984b47ba387f4e19ff3d35375838364048523e2d093d1e45f09b42d4bb3ed/Screenshot-2024-03-12-at-6.31.49PM.png" /> https://chineseperspectives.ai/Binxing-FANG Fri, 05 May 2023 19:31:52 +0000 Chinese Perspectives on AI Safety https://chineseperspectives.ai/Binxing-FANG Binxing FANG 方滨兴 About the author Binxing Fang is an expert in cybersecurity and an academician of the Chinese Academy of Engineering. He is the honorary director of the National Computer Network and Information Security Management Center, which sits directly under the Ministry of Industry and Information Technology. At Guangzhou University he is the Director of the Network Security Technology National Engineering Laboratory and the honorary president of the Institute of Advanced Cyberspace Technology. He is known as the "father of China's firewall", and is author of Cyberspace Sovereignty: Reflections on building a community of common future in cyberspace. 关于作者 方滨兴，网络空间安全专家，中国工程院院士。2007年至2013年担任北京邮电大学校长，国家计算机网络与信息安全管理中心名誉主任，广州大学网络空间先进技术研究院名誉院长。他的著作《论网络空间主权》在2017年由北京科学出版社出版。 <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/d078330bdbdcb3b8b48f904739bb45797248f658c7e2429bb26415707a85ae84/f.jpg" data-mid="179715071" border="0" data-scale="100" src="https://freight.cargo.site/w/900/i/d078330bdbdcb3b8b48f904739bb45797248f658c7e2429bb26415707a85ae84/f.jpg" /> The following excerpts are translated from Fang’s Artificial Intelligence Safety and Security1 (人工智能安全) (2020). We think the chapters most relevant to existential risks from AI are Chapters 6 (AI-derived safety problems), 7 (AI Actants2), 8 (Safety Hoops for AI Actants), 9 (Safety Evaluation and Detection for AI Actants), and 12 (Looking to the Future of AI Safety). We precede each excerpt with our own bolded summary of the key point and provide section references in brackets after each excerpt. ▶ Cite Our TranslationConcordia AI. “Binxing Fang — Chinese Perspectives on AI Safety.” Chineseperspectives.ai, 29 Mar. 2024, chineseperspectives.ai/Binxing-Fang. ▶ Cite This Work 方滨兴(2020). 人工智能安全. 电子工业出版社. Selected excerpts Intelligent weapons may seriously threaten human survival: "In terms of the possible risks brought by the machine itself, intelligent weapons present the risk of operational errors. In theory, the higher the degree of intelligence of the weapon, the more complex the software controlling its intelligent behavior is, and the higher the probability of failure and errors… Intelligent weapons will be upgraded to autonomous systems, which may endanger human beings. The development of intelligent weapons from intelligence to autonomy is both a gradual process and an inevitable trend. When intelligent weapons cross the threshold of intelligence and upgrade to autonomy, human beings may also lose the ultimate decision-making authority over intelligent weapons. This leapfrog development of intelligent weapons brings mankind not only the joy of achievements in development and the assurance of achieving goals, but also worries and fears that intelligent weapons may seriously threaten human survival. If actors in a military context lose control of intelligent weapons, those weapons are very likely to cease executing their original military tasks and become threatening enemies. Due to their powerful destructive abilities, intelligent weapons may transform from the main force for attacking the enemy into formidable enemies that are difficult to defeat, with the level of intelligence determining the strength of each side. What’s more, [intelligent weapons] increase the uncertainty of the outcome of the war. Smart weapons which were once the magic weapons for defeating the enemy are likely to become "traitors" that hurt one’s own side. Once emotionless, untiring intelligent weapons are given power over life and death and completely replace soldiers in fighting, will they wantonly kill innocent people because of an "excessive" approach to battle? Will they expand the scope of combat targets and become “humanity’s terminator"? These are the major hidden dangers associated with the possibility of loss of control in AI’s militarization."&nbsp;(6.2.2) 精选原文 人工智能武器军备竞赛带来的风险：“从机器自身可能带来的风险看，智能武器存在操作失误的风险。从理论上讲，武器的智能化程度越高，控制其智能化行为的软件组织结构就相应越复杂，出现故障失误的概率也就相对增加。…… 智能武器将升级为自主系统，存在着可能危害人类的隐患。智能武器由智能化向自主化发展既是一个渐进过程，也是一个必然趋势。当智能武器越过智能化的门槛升级为自主化时，人类也就可能失去了对智能武器的终极决定权。这种智能武器跨越式发展带给人类的就不仅仅是成果开发的喜悦和实现目标的把握，还有担心智能化武器或许会严重威胁人类生存的隐忧和恐惧。运用于军事领域的智能武器如果失控，很可能由原来的军事任务执行者走向反面，转变成为具有威胁力的敌人。由于具备强大的杀伤力，可能使智能武器由打击敌人的主力转变成为难以战胜的强敌，并由智能化程度的高低决定着敌我双方能力的强弱。而更为纠结的是，增加了战争胜负的不确定性，智能武器很可能由原来是本方克敌制胜的法宝，变成了伤害本方的“叛徒”。对于毫无感情、不知疲倦的智能武器，一旦被赋予“生杀大权”，完全替代士兵来作战，会不会因“过度”作战而滥杀无辜?会不会扩大作战对象范围成为“人类终结者”?这些都是人工智能军事化可能失控形成危险的重大隐患。” There are three necessary elements for out-of-control AI systems: "When talking about the threat of AI systems to human beings, the first thing to consider is AI systems with the ability to act. We call these “AI actants" (AIA)... Although as of the date of finalizing this book, there have been no publicly reported cases of AIAs "actually becoming out-of-control and causing harm", there has long been consensus that there is a risk of AIAs becoming out-of-control. In the process of treating AIAs with a high degree of vigilance, we also need to know that not all AIAs present risks of loss of control - only when certain conditions are met can they become out-of-control. Academician Binxing Fang, the chief editor of this book, made a report entitled “My Views on AI Safety and Security” at the China International Software Expo held on June 30, 2018. This report put forward for the first time "three necessary elements for out-of-control AIA". These three elements are: AIAs have the ability to act and destroy3, have uninterpretable decision-making ability, and have the ability to evolve and can evolve into autonomous systems.”&nbsp;(6.4) 人工智能行为体失控三要素：&nbsp; “谈起人工智能系统对人类的威胁，首当其冲要考虑的是具有行为能力的人工智能系统。我们将具有行为能力的人工智能系统称为“人工智能行为体“。 在我们高度警惕AIA的过程中，还需要知道，并非所有AIA都存在失控风险——只有满足了特定条件才可能会失控。本书的主编方滨兴院士在2018年6月30日举办的“中国国际软件博览会”上做了题为《人工智能安全之我见》的报告。该报告首次提出了“人工智能行为体失控三要素”。这三要素是指，人工智能行为体具有行为能力以及破坏力、具有不可解释的决策能力、具有进化能力并可进化成自主系统。” AIA could become an autonomous, conscious system that can fight against humans: "Once AIA starts to evolve itself or even autonomously set the objective function, it could be possible for it to break away from the “cage” with which humans limit its activity and escape human control. Obviously, if an out-of-control AIA is just like an out-of-control car with no goal, it only presents a derivative safety problem4; however, if AIA evolves to the point where it needs to defend its "right to survive" and does not hesitate to harm human beings in order to protect itself, it will become an autonomous system that can fight against human beings, which will lead to disasters for humanity. Here, our definition of autonomous system is not simply "not controlled by people". Once an autonomous weapon is on the battlefield, it is indeed not controlled by people, but that [loss of control] is limited to the level of decision-making. What we mean by autonomous systems here is that they have "consciousness". Like humans, they know how to protect themselves, and their objective function is to protect themselves from harm. They can even have the ability to socialize and organize. In this case, they will indeed become oppositional to human beings.” (6.4.3) 人工智能行为体具有进化能力，可进化成自主系统： “一旦AIA开始自我进化甚至自主设置目标函数的时候，将有可能脱离人类限制其活动的“牢笼”，使其失去了人类的控制。显然，失去控制的AIA如果仅仅是像一辆没有目的性的失控汽车一样，那还仅仅是衍生安全的问题；但如果AIA进化到需要捍卫自身的“生存权”，为了自我保护而不惜伤害人类的时候，就会变成可以与人类对抗的自主系统了，这将会导致人类灾难的到来。在此，我们所指的自主系统还不仅仅是“不受人控制”这么简单，自主武器一旦上了战场也不受人控制，但那还仅仅停留在具有决策能力的层面；我们所指的自主系统是具有“意识”，像人类一样懂得如何自我保护，它们的目标函数是保护自己不受伤害；它们甚至可以具备社交、组织能力。在这种情况下，它们确实将会成为人类的对立面。” AIA can learn things humans haven’t taught them:&nbsp; "Autonomy is the third characteristic of AIA. Assuming that harm caused by AIA to humans is not due to loss of control, such as in the case of a driverless car losing control and harming humans due to inertia, then the process of choosing to harm humans will bring about more dangerous situations, and the learning ability of AIA will make it difficult for human beings to mitigate such dangers. Of course, some people think that since AIA is artificial and is a moving object constructed on a closed set, it is impossible for it to do things that human beings have not taught it to do. For example, when operating on the set of binary digits {0,1}, it would never produce an output from the set of alphabetic characters. In particular, researchers in the field of AI sometimes deny that AI systems (AIS) have the ability to create, because they know that everything AIS learn is taught by human beings. If human beings have not taught something, AIS have no reason to be able to do it; at least AIS have no reason to choose a result that has not previously been deemed correct. For example, even though the Boston Dynamics robot introduced in Section 7.2.1 has the ability to walk on two feet, it will not evolve into a master of "Shaolin Kung Fu" on its own. The problem is that people may overlook one thing, that is, the connective ability of AI (关联能力). If the Boston Dynamics robot has grasped some basic abilities, even becoming a Kung Fu master would not constitute jumping out of the closed set, so long as it has mastered the evaluation function through observation and learning. In fact, suppose we have taught the robot how to select a reward function such that, if it falls over, the reward function gives a negative value and, if it does not fall over, the reward function gives a positive value. At the same time, the robot has the ability to generate random actions, so after a long time of training and trying various results, the robot will inevitably choose the most suitable combination of actions, which is enough to enable it to achieve the optimal state - a state that was originally impossible for humans to teach. After all, there are many reasons why these robots may master capabilities that exceed expectations.”&nbsp;(7.6.3) 人工智能行为体的自主特性： “自主性是AIA的第三个特性。假定AIA对人类的危害不是因为失控而造成的，如无人驾驶汽车出现了失控而以惯性来伤害人类，那么自主选择伤害人类的过程就会带来更为危险的局面，AIA的学习能力会变得使人类难以防范。 当然，有人认为，AIA既然是人造的，而且是在一个封闭集上构造出来的运动体，就永远不可能去做人类没有教过的事情，就好比在{0，1}集合上进行操作，永远不会产生字符集中的结果一样。尤其是人工智能专业领域研究者有时否认AIS具有创造能力，因为他们知道AIS所学会的一切都是人类教的，人类没有教的，AIS就没有道理会，至少AIS没有道理去选择一个不曾被认为是正确的结果。例如，就算7.2.1节中所介绍的波士顿动力机器人具有了双足行走的能力，但总不会自己进化成为会“少林武功”的高手。 问题是人们可能忽视了一件事，那就是人工智能的关联能力。如果波士顿动力机器人掌握了一些基本能力，那么通过观察、学习，只要掌握了评价函数，即便是变成武功高手，也不算是跳出封闭集的行为。事实上，从人工智能专业领域的视角来看，假定我们教会了机器人如何选取奖励函数，如摔倒，则奖励函数为负值，如果没有摔倒，则奖励函数为正值，同时机器人具有产生随机动作的能力，那么机器人经过长时间的训练、尝试各种结果，势必会选择出一种最适合的动作组合，从而足以让其获得最优的状态，而这一点原本就是人类无法教授的，毕竟存在多种原因使这些机器人可能掌握超出预想的能力。” AIAs need to be designed with risk mitigation in mind: "In order to prevent AIAs from threatening human beings, people need to consider risk at the outset of designing AIAs. The first consideration is the need to develop design principles that can protect human safety. The design principles of AIA should include but not be limited to: (1) Design to minimize risk: First, eliminate hazards in the design. If the known hazards cannot be eliminated, design choices should reduce risk to an acceptable level; (2) Use safety devices: Use a safety hoop5 or other safety protection device to reduce the risk to an acceptable level; (3) Use warning devices: Use a warning device to announce danger and send appropriate warning signals to nearby personnel by voice, light, etc.” (9.1) 人工智能行为体的安全管理概述： “为了防止AIA威胁人类，人们在设计AIA之初就需要对风险问题加以考虑。为此，首先考虑的就是需要制定能够保护人类安全的设计原则。AIA的设计原则应该包括但不限于： （1）最小风险设计：首先在设计上消除危险，若不能消除已判定的危险，应通过设计方案的选择将其风险降低到可接受的水平； （2）采用安全装置：采用保险箍或其他安全防护装置，使风险减小到可接受的水平； （3）采用告警装置：采用告警装置来告知危险，以语音、光线等方式向接近人员发出适当的警告信号等。” Safety assessments and evaluation of AI are needed: [The author writes that if the aforementioned “safety hoop” becomes a necessary component of AIA, it will be necessary to check that the safety hoop works before an AIA can be placed on the market, just as how a car’s brakes and collision avoidance system must undergo testing] …"More importantly, an evaluation mechanism must be designed to evaluate the level to which an AIA has evolved. For example, can people design an evaluation mechanism to identify what [Go] rank AlphaGo has evolved to? Of course, this should not simply rely on competitions, as is the case currently, but should be a method for fast appraisal. At present, there is no such method because people do not need one. No one worries that AlphaGo will harm mankind, and since for humans Go rank and ability are assessed by competition, the same is true for AlphaGo. However, this does not mean that there is no relevant fast appraisal method. It’s just like how with a classification algorithm, it is relatively easy to quickly evaluate capability as long as the training examples and test examples are separated. So can we extract intermediate processes in Go play to act as evaluation methods? In short, for future technological development and social progress, it is necessary to try to construct corresponding evaluation mechanisms to evaluate how far AIA has evolved.” (12.4) 人工智能的安全评估与检测： 【作者写道，如果上述“保险箍”成为 AIA 的必要组成部分，那么在 AIA 投放市场之前，必须检查保险箍是否有效，就像汽车的刹车和防撞系统必须接受测试一样】……“更为关键的是，必须设计出一种检测机制，使之能检测出一个AIA已经进化到什么程度。例如，人们是否能够设计出一种检测机制，以鉴定AlphaGo已经进化到什么样的段位?当然这不应该像现在这样简单地靠比赛来鉴定，而应该提出一种方法来进行快速鉴定。目前没有这种手段，是因为人们并不需要，没有人担心AlphaGo会危害人类；而对围棋的段位水平的鉴定就是靠比赛，既然对人类如此，对AlphaGo也就没什么变化。但这并不意味着不存在相关的快速检测手段，就像对一个分类算法进行检测一样，只要将训练样例与检测样例分开就比较容易快速地检测一个分类算法的能力，那么是否可以将围棋对弈中的某些中间过程提取出来作为检测方法呢?总之，设法构造相应的检测机制，以便能够检测出AIA的进化程度，对未来的技术发展和社会进步都是十分必要的。” Translator’s notes&nbsp; 1.&nbsp;In English, “safety” and “security” typically refer to protection against unintended and intended harms respectively. In Chinese, the word 安全 (anquan) can encompass the meaning of both “safety” and “security.” Throughout the pieces featured on this website, we select the English translation that we think best fits the author’s meaning. Fang’s book covers both unintended and intended harms, so we translate the title as Artificial Intelligence Safety and Security. In the excerpts on this page, the discussion is mostly of unintended harms/loss of control, so we translate anquan as ‘safety.’ 2. Defined as systems with the ability to act. 3. Elsewhere in the chapter, the author expands: “AIA’s ability to act comprises two kinds of external manifestations: one is that it can move in physical space and has kinetic energy; the second is that it can exchange energy with the outside world and thus change the state of other objects.” 4.&nbsp;AI-derived/derivative safety problems are defined elsewhere as “AI vulnerabilities threatening safety in other fields,” e.g. safety accidents caused by AI system errors, the development of AI weapons leading to an international arms race, and the potential loss of control of future AI systems. This is distinguished from AI’s internal safety problems, which are vulnerabilities in the AI system itself. 5.&nbsp;Elsewhere, the author expands: “A safety hoop can be inserted between the propulsion system and the decision-making system, and the decision-making system of the robot needs to pass through the safety hoop to allow propulsion to occur. Just like a fuse will automatically blow when current is abnormal to ensure the safe operation of an electrical system, the safety hoop will be activated once certain conditions are met, and then begin to restrict the robot. Other Authors <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" data-mid="188596306" border="0" alt="Bo ZHANG" data-caption="Bo ZHANG" src="https://freight.cargo.site/w/900/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" /> <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/f1826d016d72655d578dd2ce9192cef78febba10057f69499d01c991bf9dd3d4/c.jpg" data-mid="188596309" border="0" alt="Wen GAO" data-caption="Wen GAO" src="https://freight.cargo.site/w/900/i/f1826d016d72655d578dd2ce9192cef78febba10057f69499d01c991bf9dd3d4/c.jpg" /> <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/fccde0537f2601b71a2777693204bc4576929944b625638c208a732c0c55e1b1/e.jpg" data-mid="188596310" border="0" alt="Xiuquan LI" data-caption="Xiuquan LI" src="https://freight.cargo.site/w/900/i/fccde0537f2601b71a2777693204bc4576929944b625638c208a732c0c55e1b1/e.jpg" /> https://chineseperspectives.ai/Wen-Gao Tue, 23 May 2023 18:33:57 +0000 Chinese Perspectives on AI Safety https://chineseperspectives.ai/Wen-Gao Wen GAO 高文 About the author Wen Gao is Director of Peng Cheng Laboratory, a research lab in Shenzhen that has contributed to Chinese advances in pre-trained language models, and Dean of the School of Information Science and Technology at Peking University. He is also an academician of the Chinese Academy of Engineering. He has extensive experience advising on state-led S&amp;T initiatives. In 2018 he delivered a presentation at a Politburo study session on AI. He is a member of China’s Science and Technology Ethics Committee; Deputy Chief of the Advisory Group of the Ministry of Education’s AI Technology Innovation Expert Group, which was established in 2018 to advise on topics including talent development and academia-industry collaboration; and is one of 27 members of the Ministry of Science and Technology’s Next Generation Artificial Intelligence Strategic Advisory Committee. He was formerly deputy director of the National Natural Science Foundation of China and from 1996-2000 was chief of the National High-Tech R&amp;D Program (863 Program) Intelligent Computing Expert Group.&nbsp; 关于作者高文，中国工程院院士、北京大学博雅讲席教授，鹏城实验室主任，新一代人工智能产业技术创新战略联盟理事长，全国信息技术标准化技术委员会委员，数字音视频编解码技术标准(AVS)工作组组长，国际电气电子工程师协会会士（IEEE Fellow）、国际计算机协会会士（ACＭ Fellow）。他曾任国家自然科学基金委员会副主任。1996年担任国家863计划信息领域智能计算机主题专家组组长。主要从事人工智能应用和多媒体技术、计算机视觉、模式识别与图像处理、虚拟现实方面的研究。主要著作有《数字视频编码技术原理》、《Advanced Video Coding Systems》等。 <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/f1826d016d72655d578dd2ce9192cef78febba10057f69499d01c991bf9dd3d4/c.jpg" data-mid="179714447" border="0" src="https://freight.cargo.site/w/900/i/f1826d016d72655d578dd2ce9192cef78febba10057f69499d01c991bf9dd3d4/c.jpg" /> The following excerpts are from Technical Countermeasures for Security Risks of Artificial General Intelligence,&nbsp;Strategic Study of Chinese Academy of Engineering, 2021, Volume 23, Issue 3. Gao has mentioned this paper at least three times during 2021-22: at the China Electronics and Information Technology Conference (May 2022); at the AI Cooperation and Governance international conference hosted by Tsinghua University (December 2021 and December 2022). ▶ Cite Our TranslationConcordia AI. “Wen Gao — Chinese Perspectives on AI Safety.” Chineseperspectives.ai, 29 Mar. 2024, chineseperspectives.ai/Wen-Gao. ▶ Cite This Work &nbsp;刘宇擎,张玉槐,段沛奇,施柏鑫,余肇飞,黄铁军 &amp; 高文(2021). 针对强人工智能安全风险的技术应对策略. 中国工程科学(03), 75-81. Selected excerpts Abstract： “Human beings might face significant security risks after entering into the artificial general intelligence (AGI) era. By summarizing the difference between AGI and traditional artificial intelligence, we analyze the sources of the security risks of AGI from the aspects of model uninterpretability, unreliability of algorithms and hardware, and uncontrollability over autonomous consciousness. Moreover, we propose a security risk assessment system for AGI from the aspects of ability, motivation, and behavior. Subsequently, we discuss the defense countermeasures in the research and application stages. In the research stage, theoretical verification should be improved to develop interpretable models, the basic values of AGI should be rigorously constrained, and technologies should be standardized. In the application stage, man-made risks should be prevented, motivations should be selected for AGI, and human values should be given to AGI. Furthermore, it is necessary to strengthen international cooperation and the education of AGI professionals, to well prepare for the unknown coming era of AGI.” 精选原文 摘要：“未来进入强人工智能（AGI）时代，人类可能面临重大安全风险。本文归纳了 AGI 与传统人工智能的区别，从模型的不可解释性、算法及硬件的不可靠性、自主意识的不可控性三方面研判了 AGI 安全风险的来源，从能力、动机、行为3 个维度提出了针对 AGI 的安全风险评估体系。为应对安全风险，从理论及技术研究、应用两个层面分别探讨相应风险的防御策略：在理论技术研究阶段，完善理论基础验证，实现模型可解释性，严格限制 AGI 底层价值取向，促进技术标准化；在应用阶段，预防人为造成的安全问题，对 AGI 进行动机选择，为 AGI 赋予人类价值观。此外，建议加强国际合作，培养强AI 研究人才，为迎接未知的强AI 时代做好充分准备。” Consciousness is core to AGI: “In terms of cognitive theory, the concept of AGI emphasizes the existence of consciousness and highlights systems of values and worldviews.”&nbsp; 意识是强人工智能的核心：&nbsp;“在认知论方面，AGI 强调意识的存在，突出价值观和世界观体系，认为智能体可以拥有生物的本能。” China is behind in research relating to AGI security： “Evaluating and formulating strategies for coping with potential AGI security risks, and finding measures that will ensure that AGI is beneficial to humanity rather than harmful to society, have become research topics worldwide. For instance, in 2016, the U.S. lab OpenAI analyzed potential security problems that might arise in the development of AI1. In 2018, the U.S. government established the National Security Commission on Artificial Intelligence2. In addition, the EU set up the High-Level Expert Group on AI to help it strive for discourse and rule-making power in technological development3. AI has also become a subject of significant attention in the field of national defense. For example, AI is being adopted to improve the capability of defense systems, and AI anomaly detection technology is being developed to prevent malicious tampering of private data. AI theories and technologies, including algorithms integrating multiple disciplines, self-adaptive situational awareness, and human-machine trust, are also being studied4.” 中国在与强人工智能安全相关的研究方面落后： “对 AGI 可能的安全性风险进行评估并制定适宜对策，探讨有效驾驭 AGI 并使之既造福于人类又不对社会造成危害的举措，已经成为世界性的研究议题。例如，美国 OpenAI 团队 2016 年分析了 AI 发展过程中可能遇到的安全问题，随后美国政府成立了人工智能安全委员会；欧盟设立了人工智能高级别专家组，争取技术发展的话语权和规则制定权 。此外，AI 也成为国防领域的重点关注对象，如采用 AI 手段提高防御系统能力，发展 AI 异常检测技术用于防止隐私数据被恶意篡改，研究涉及多学科融合算法、自适应态势感知能力、人机信任等方面的 AI 理论与技术。” “It should be noted that, in terms of research related to AGI security issues, there is a gap between China and the international frontier of progress. Chinese academic and industry circles are paying more attention to the development of AI and less to the value of and need for AGI security.” “也要注意到，针对 AGI 安全问题，我国相比国际前沿进展存在一定差距；国内学术界、产业界较多专注于 AI 的发展，很少关注 AGI 安全性保障的价值和需求。” Uncontrollability of autonomous consciousness is one of three main sources of AGI security risk: “The construction of an initial intelligent agent and effective principles for evolution are key to the design of an AGI system that can conduct self-development and self-iteration. Although human beings can control the initial intelligent agent well, AGI can design rules of evolution autonomously, possibly much more efficiently than human beings. After it undergoes recursive self-improvement, AGI will have a higher development efficiency in the subsequent stages and will surpass the cognition of human beings by a long way through recursive self-improvement. 自主意识是强人工智能安全风险的三大主要来源之一：“构建初始智能体、有效进化准则，是能够自我发展、自我迭代 AGI 系统设计的关键。人类可以很好地控制初始智能，但是 AGI 可以自主设计进化规则，这种设计进化规则的效率可能足以碾压人类。自我发展后的 AGI，在后续阶段的发展效率将会更高，通过递归地自我改进而使其远超人类认知。” “AGI with autonomous consciousness5 carries potential risks. Unlike those of the human brain, the computational and analytical abilities of AGI are theoretically limitless. AGI has efficient data collection, processing, and analysis abilities and can understand all the information it sees, hears, and receives. Once it achieves consciousness, AGI will be able to share and exchange information through communication and significantly improve its understanding of the world and the efficiency through which it can transform reality. Accordingly, AI may gradually conduct various human activities. With the emergence of autonomous consciousness, the legal status of AGI becomes unclear: should it be seen as a subject with consciousness or as personal property? This may lead to disagreements at the legal, ethical, and political levels, and cause unexpected consequences.” “具有自主意识的 AGI 具有潜在风险。不同于人脑，AGI 的计算和分析能力在理论上是没有边界的，具有高效的数据收集、处理、分析能力，可理解看到、听到、接收到的所有信息。AGI 被赋予自主意识后，可通过交流、沟通的方式进行信息的分享与交换，显著提高对世界的认知、理解与改造效率。相应地，人类的各种活动都有可能逐步被 AI 取代。由于自主意识的呈现，AGI 的法律定位出现了模糊：将其视为有意识的主体，还是个人的私有财产？这可能在法律、伦理、政治层面引入分歧，从而引发难以预料的后果。” AGI might undergo a “treacherous turn” after its capabilities grow and it develops consciousness: “There is no need to worry that AI might cause harm to human beings when it is weak and can be controlled by them. However, once AI completely surpasses humans in all abilities and possesses consciousness, it will become difficult to assess whether AI will necessarily continue to obey the orders of human beings. This situation has been called the “treacherous turn”6. Although the questions of whether AI has human consciousness and how it will realize humanlike consciousness remain unanswered, they are worthy of attention and research.” 强人工智能在能力提升、发展出意识之前可能经历一次“背叛转折”：“就 AI 而言，在其能力弱小、可被人类控制的阶段，不必担心对人类造成危害。当 AI 的各方面能力超过人类、和人类一样拥有意识后，就很难判断是否必然继续听从人类命令，这种情况称为“背叛转折”。AI 是否具有人类意识、依靠何种方式实现类人意识，尽管尚属未知，但同样值得关注和研究。” Monitoring the behavior of AGI during testing is not sufficient to provide reliability guarantees: “The supervision and control of AGI behaviors can be regarded as a “principal-agent” problem, where humans are the principals and AGI systems are the agents. However, this differs from the current “principal-agent” problems between people, because AGI can formulate differential strategies and actions based on its analytical capabilities and knowledge reserves. Therefore, the monitoring of AGI behavior during testing at an early stage of research and development cannot support humans in making rational inferences about the future reliability of AGI. This means that behaviorist methods may fail.” 在测试过程中监督强人工智能的行为不足以提供可靠保障： “对 AGI 行为的监督和控制，可视为一类“委托–代理”问题，即人类是委托方， AGI 系统是代理方。这与当前人类实体的“委托–代理”问题性质不同，即 AGI 可根据自己的分析能力、知识储备来自行制定差异化的策略与行动。因此，监测 AGI 在研发初期的测试行为，并不能支持人类合理推测 AGI 未来的可靠性。如此，行为主义方法可能失效。” AGI based on cognitive neuroscience and meta learning may have advantages for verification and interpretability: “Improving the verification of theoretical foundations and exploring the interpretability of models constitute the foundations of AGI accuracy and the formal guarantees of AGI security.” 基于认知神经科学和元学习的强人工智能可能有助于可验证性和可解释性：“完善理论基础验证、探索模型的可解释性，是 AGI 正确性的构建基础，也是 AGI 安全的形式化保障。” “The model design of AGI should be explored based on cognitive neuroscience, the discipline that studies the brain’s structure and investigates the brain’s mode of operation based on its biological structure and the cognitive ability of human beings. A suitable AGI model can be designed based on the structure and mode of operation of the human brain.” “应以认知神经科学为基础，探索 AGI 的模型设计。认知神经科学是基于大脑的生物结构、人类的认知能力，研究脑构造、探索脑运行方式的学科；借鉴人脑结构和运行方式，可设计适当的 AGI 模型。” “The implementation of AGI should be based on meta learning, a method of learning how to learn7 that enables AI to think and reason… As one of the implementation methods of semi-supervised and unsupervised learning, meta learning is an important mathematical implementation for simulating human learning processes. Seeking methods for such simulations can improve the model interpretability, explore ways to enable AGI to “learn to learn,” and develop consciousness similar to that of human beings.” “应以元学习为基础，探索 AGI 的实现方法。元学习是学习“学习方法”的方法，可赋予 AI 思考和推理的能力；……元学习则是经验导向，基于过去的经验去学习新任务的解决办法，可使 AI 掌握更多技能、更好适应复杂的实际环境。元学习作为半监督、无监督学习的实现方式之一，是模拟人类学习过程的重要数学实现；寻求通过数学方法模拟人类学习过程的手段，据此提高模型的可解释性，探索让 AGI “学会学习”，像人类一样“产生自主意识”。” The underlying value orientation of AGI must be constrained and monitored: “Explicit rules should be designed to limit the range of action of AI. In view of the complexity and uninterpretability of AI, it is difficult to constrain and monitor its value orientation using the source code. Constraining the value orientation of AI from a behavioral perspective and limiting the behavioral ability and action permissions of AGI using explicit rules are key research objectives. ” 强人工智能的底层价值取向需要被控制和监督：“应设计明文规则，限制 AI 的行动范围。鉴于 AI 的复杂性、不可解释性，很难从源代码角度对其价值取向进行限制和监控。从行为角度对 AGI 的价值取向进行限制，通过明文规则来限制 AGI 的行为能力和动作权限，是重要的研究目标。” “An underlying value network can be constructed during the process of meta learning to accelerate inference training and guide the action network to take action.8 The algorithm for the underlying value network is complex, and the dataset cannot be controlled, making it extremely difficult to adopt measures to limit the inference process of the network. For the action network, explicit rules can be manually added to ensure that each action is in line with the correct values (i.e., limiting the occurrence of incorrect behaviors for every independent action).” “在元学习的过程中，可构建底层的价值观网络来加速推理，指导行动网络采取行为。关于底层的价值观网络，算法具有复杂性，数据集存在不可控性，很难采取措施对其推理过程进行限制。关于行动网络，可人为加入明文规则，确保在原子行动上符合正确的价值观（即针对每一个独立动作，限制错误行为的出现）。” “Trusted computing technology should be applied to monitor AI actions. Trusted computing is a mechanism for defending against malicious code and attacks and can be regarded as an “immune system” for computers. Additional supervision is introduced to build a complete, trustworthy, and quantifiable evaluation mechanism for various computer behaviors, and then judge whether these behaviors meet the expectations of human beings, thus preventing and handling actions that cannot be trusted.”&nbsp; “要应用可信计算技术，监控 AI 的行动内容。可信计算是一种针对恶意代码、恶意攻击的防御机制，可视为计算机的“免疫系统”；引入额外监督，对计算机的各种行为建立完整、可信、可量化的评价机制，据此判断各种行为是否符合人类的预期、对不可信的行动进行防治。” “The operation process of AGI should be monitored and analyzed. A time series analysis can be used to determine if the current behavior has a reasonable value orientation. If it does not conform to such an orientation, an external intervention method should be adopted to interrupt the current action of the AGI and ensure that the AGI will not act contrary to values.” “应用于 AI 的行动过程监控，即可认为具备正确价值观的行为是合理可信的。监控并分析 AGI 行为的运行过程，通过时间序列来判断当前行为是否具备合理的价值取向；如不符合，采用外部干预的方式干扰或打断 AGI 的当前行动，确保 AGI 不会做出违背价值观的行为。” Motivation selection for AGI should happen in advance so that superintelligence does not wish to harm humans: “At the “treacherous turn” stage, AI has already developed cognitive abilities far exceeding those of human beings in various fields. This can be referred to as superintelligence.9 Given the reasonable assumption that superintelligence may betray human beings, humans should select the motivation of intelligent agents in advance to fully prevent undesirable results and equip superintelligence with the innate wish to not harm human beings.” 应该提前对强人工智能进行动机选择，使超级AI不具有对人类造成危害的自发意愿： “‘背叛转折’阶段的 AI 已经具有在各个领域都远超人类的认知能力，可称为超级 AI 。基于超级 AI 可能会背叛人类的合理猜想，人类应当提前对智能体的动机进行选择，全力制止不良结果的出现；应使超级 AI 具有不对人类造成危害的自发意愿。” Four approaches to motivation selection have been proposed: direct specification, domesticity, augmentation, and indirect normativity.10 [An explanation of these four approaches follows.] 针对动机选择问题，当前研究讨论提出了直接规定、驯化、扩增、间接规范 4 种应对方式。【后续解释了这四种应对方式】 Loading AGI with human values may be a more reliable approach than motivation selection, and requires whole-brain emulation:“Although motivation selection improves the effectiveness of human control over AI, compared to limiting its ability, several problems still exist. For example, AI may face an infinite number of situations, making it impossible to discuss solutions for every situation, and it is infeasible for human beings to continuously monitor the motivation of AI. In this case, one feasible solution is to endow AGI with human values (by loading them into the AGI), thereby allowing it to consciously execute actions that will not pose a threat to human beings. It is impossible to fully represent the motivation systems present under all situations in a table (which would lead to an infinitely large table)... 与动机选择相比，为强人工智能赋予人类价值可能是更可靠的方法，且需要全脑仿真：“相比限制 AI 的能力，动机选择已经在一定程度上提升了人类控制 AI 的有效性，但仍面临一些问题。例如，AI 可能面对无穷多种情况，不可能具体讨论每一种情况下的对策，而人类本身不可能持续监视 AI 的动机。可行的思路之一是将人类的价值观赋予 AI（加载到 AGI 内部），让其自觉地执行那些不对人类构成威胁的事件。无法将各种情况下的动机系统均完整具象为可以查询的表格（导致无穷大的表格）......” “The use of evolutionary algorithms is a feasible route for value loading… However, the accumulation of human values is the result of our genetic mechanism evolving over millions of years and imitating or reproducing this process would be extremely difficult. Because this mechanism has adapted to the neural cognitive architecture of human beings, it can only be realized through whole-brain emulation.11 As the premise of whole-brain emulation, the brain is a computer that can be simulated. However, it faces three challenges: scanning, translation, and simulation.12&nbsp;The required precision can only be achieved using high-throughput microscopy and supercomputing systems.” “进化算法可能是加载价值观的可行途径之一......然而，人类价值观的积累过程是人类相关基因机理经历成千上万年进化的结果，模仿并复现这一过程非常困难；这一机理与人类神经认知体系结构相适应，因而只能应用于全脑仿真。全脑仿真的前提是大脑可被模拟、可以计算，面临着扫描、翻译、模拟 3 类条件的制约，采用高通量显微镜、超级计算系统才能达到所需精确度。” International cooperation for AGI is necessary: “AGI research has become a subject of international attention. Only by concentrating the scientific and technological strengths of the whole of humanity can we ensure that AGI better serves society. The process of AGI research and its gradual application involve many unknown problems. Strengthening international AGI cooperation and promoting the sharing of research will be necessary to improve the ability to respond to emerging situations and guarantee the implementation and expansion of AGI applications….” 强人工智能国际合作是必要的：“AGI 研究已经成为国际性的关注点，集中全人类的科技力量来推进 AGI 的深化研究，才能使 AGI 更好服务人类社会。相关研究和逐步应用的过程，将面临许多未知问题。加强 AGI 国际合作、促进研究成果共享，才能根本性地提高应对突发情况的能力，也才能真正保障 AGI 的应用落地和拓展......” A controlled intelligence explosion and dynamically responding to risks are needed to prevent catastrophic outcomes: “The intelligence and behavior of AGI cannot simply be equated to those of human beings. The motivation for creating AGI is to benefit human society. However, to protect the privacy of individuals and society as a whole, AGI should be controlled such that it only serves human beings passively, rather than learning on its own initiative. If there is an intelligence explosion once AI has evolved to a certain level, the default result will inevitably be catastrophic. In view of such potential threats, humanity should continue to monitor the risks and search for countermeasures to avoid the occurrence of this default ending. Humanity should design a controlled intelligence explosion and set the proper initial foundations, all while achieving humanity’s desired results and ensuring that all consequences remain within an acceptable range. 受控制的智能爆发与对风险的动态应对对于防范灾难性后果是不可或缺的：“AGI 的智慧与行为不能简单地与人类划等号，创造 AGI 的动机是为了更好地造福人类社会。对于人类社会的隐私，应控制 AGI 只能给人类提供被动的服务，而不是主动的学习。如果 AI 进化到一定水平后出现智能爆发，默认后果必然是造成确定性灾难。面对这样的潜在威胁，人类应持续关注并着力寻求应对方法，坚决避免这种默认结局的出现；设计出受控制的智能爆发，设置必要的初始条件，在获得人类想要的特定结果的同时，至少保证结果始终处于人类能接受的范围。” In the future, we recommend paying close attention to the technological evolution of AGI and proposing dynamic strategies for responding to potential security risks. We should examine international discussions and drafting of AGI policies, integrate cutting-edge legal and ethical findings, and explore the elements of China’s AGI policymaking in a deeper and more timely manner.” “着眼未来发展，建议持续关注 AGI 的技术演进路线，对技术伴生的潜在安全风险提出动态的应对策略；参考国际性的 AGI 政策研讨和制定过程，结合法律、伦理方面的前沿成果，更为及时、深刻地探讨我国 AGI 政策的制定要素。” Notes&nbsp; 1.&nbsp;Amodei D, Olah C, Steinhardt J, et al. Concrete problems in AI safety [EB/OL]. (2016-07-25) [2021-02-15]. https://arxiv.org/ abs/1606.06565. 2. Congress of the United States. H.R.5356-National security commission artificial intelligence act of 2018 [EB/OL]. (2018-03-20) [2021-02-15]. https://www.congress.org/bill/115th-congress/housebill/5356. 3. China Academy of Information and Communications Technology. Global AI governance report [EB/OL]. (2020-12-30) [2021-02- 15]. https://pdf.dfcfw.com/pdf/H3_AP202012301445361107_1.pdf?1609356816000.pdf. Chinese. 4.&nbsp;Jin J, Qin H, Dai Z X. Top-level strategy of artificial intelligence security and the research status of key institutions in the United States [J]. Civil-Military Integration on Cyberspace, 2020 (5): 45–48. Chinese. 5.&nbsp;Translator’s note: The meaning of the word ‘consciousness‘ (意识) in this piece is ambiguous. Due to the association with the words ‘uncontrollability’ (不可控性) and ‘autonomous’ (自主), we interpret it as emphasising an ability to control one’s own development. 6.&nbsp;Bostrom N. Superintelligence: Paths, dangers, strategies [M]. Oxford: Oxford University Press, 2015. 7.&nbsp;Vilalta R, Drissi Y. A perspective view and survey of meta-learning [J]. Artificial Intelligence Review, 2002, 18(2): 77–95. 8.&nbsp;Translator’s note: The value network (价值观网络, jiazhiguan wangluo) and action network (行为网络 xingwei wangluo) referred to here are distinct from the value network (数值网络 shuzhi wangluo) and policy network (策略网络 celue wangluo) of reinforcement learning. 9.&nbsp;Bostrom N. Superintelligence: Paths, dangers, strategies [M]. Oxford: Oxford University Press, 2015. 10.&nbsp;Bostrom N. Superintelligence: Paths, dangers, strategies [M]. Oxford: Oxford University Press, 2015. 11.&nbsp;Huang T J. Can human build “super brain”? [N]. China Reading Weekly, 2015-01-07(5). Chinese. 12. Bostrom N. Superintelligence: Paths, dangers, strategies [M]. Oxford: Oxford University Press, 2015. Other Authors <img width="720" height="864" width_o="720" height_o="864" data-src="https://freight.cargo.site/t/original/i/d52175e0957fa3ad717ad15be56f22a71cc3dba04fef40f4835ed9a469f8b428/23a22ce9da08d46b614a1a8c09e15a1f.jpg" data-mid="188596652" border="0" alt="Tiejun HUANG" data-caption="Tiejun HUANG" src="https://freight.cargo.site/w/720/i/d52175e0957fa3ad717ad15be56f22a71cc3dba04fef40f4835ed9a469f8b428/23a22ce9da08d46b614a1a8c09e15a1f.jpg" /> <img width="302" height="362" width_o="302" height_o="362" data-src="https://freight.cargo.site/t/original/i/d9b9f5d5e8b90ea849b5a9b7169520ebc1f2b9a5da94db47fcc26eef8bfe8431/IMG_6393.JPG" data-mid="188596802" border="0" alt="Ya-Qin ZHANG" data-caption="Ya-Qin ZHANG" src="https://freight.cargo.site/w/302/i/d9b9f5d5e8b90ea849b5a9b7169520ebc1f2b9a5da94db47fcc26eef8bfe8431/IMG_6393.JPG" /> <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" data-mid="188596807" border="0" alt="Bo ZHANG" data-caption="Bo ZHANG" src="https://freight.cargo.site/w/900/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" /> https://chineseperspectives.ai/Bo-ZHANG Fri, 26 May 2023 08:49:08 +0000 Chinese Perspectives on AI Safety https://chineseperspectives.ai/Bo-ZHANG Bo ZHANG 张钹 About the author Bo Zhang is the honorary dean of Tsinghua University’s AI institute and a professor in the university’s Department of Computer Science. He is an academician of the Chinese Academy of Sciences, Chief Scientist at AI safety/security start-up RealAI, and technical consultant for Microsoft Asia Research Institute. He is one of the founding figures of China’s AI field. 关于作者张钹，清华大学计算机系教授，中科院院士。现任微软亚洲研究院技术顾问。他参与人工智能、人工神经网络、机器学习等理论研究，以及这些理论应用于模式识别、知识工程与机器人等技术研究。 <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" data-mid="180028265" border="0" src="https://freight.cargo.site/w/900/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" /> This is an English translation of a piece titled “Academician Zhang Bo: Make Responsible AI” (张钹院士：做负责任的人工智能). The piece is a transcription of a speech delivered online by Zhang to the World Internet Conference Wuzhen Summit’s “AI and Digital Ethics Sub-Forum” on November 10, 2022.&nbsp; ▶ Cite Our TranslationConcordia AI. “Bo Zhang — Chinese Perspectives on AI Safety.” Chineseperspectives.ai, 29 Mar. 2024, chineseperspectives.ai/Bo-Zhang. ▶ Cite This Work张钹(2022-11-10). “做负责任的人工智能”. 在世界互联网大会乌镇峰会人工智能与数字伦理分论坛上的演讲. https://mp.weixin.qq.com/s/OgGoqoy6dCzJiEopMpnNXw Translation Make Responsible AI The first person who proposed that there might be ethical risks in the development of robots was Asimov, an American science fiction novelist, in his novel “Runaround.” The year was 1942, long before the birth of AI. He also proposed methods to avoid risks — the well-known “three laws of robotics.” Putting forward these issues at that time showed foresight. Later, the physicist Hawking and others continued to put forward similar warnings, but these warnings weren’t taken seriously by everyone, especially not by the AI community. 原文 做负责任的人工智能 最早提出机器人发展中可能存在伦理风险的是，美国科幻小说家阿西莫夫在他的小说“环舞”（Runaround）中提出的，时间是1942年，早在人工智能诞生之前，为此他还提出规避风险的方法，即大家熟知的“机器人三定律”，应该说这些问题的提出具有前瞻性。后来物理学家霍金等也不断地提出类似的警告，但这些警告并没有引起大家特别是人工智能界的重视。 The reason was that the foundations of the argument were not sufficient. Those sounding the alarm believed that the continuous progress and development of machines would one day lead to their intelligence exceeding that of humans, and especially when machines gained subjective consciousness, that is, when so-called "superintelligence" emerged, humans would lose control over machines and there would be catastrophic consequences1. This "technical logic" was not convincing for most AI researchers because everyone knew that AI research work was still in the exploratory stages, progress was slow, and there were still many difficult problems. It is not easy to build a "superhuman" robot. In addition, it has always been controversial whether the goal of "superintelligence" can be achieved through so-called "artificial general intelligence". Therefore, we believed that these risks were only concerns for the far future, and we were not in a hurry to consider them. 原因在于他们的立论依据不够充分，他们认为机器的不断进步和发展，有朝一日当它的智力超过人类，特别是机器具有主观意识时，即出现所谓的“超级智能”时，人类将会失去对机器的控制，从而带来灾难性的后果。这种“技术逻辑”对于大多数人工智能研究者来讲并不具有说服力。因为大家清楚地知道，人工智能研究工作目前还处于探索的阶段，进展缓慢、还受到很多问题的困扰，难以解决，制造“超人类”的机器人谈何容易。而且能不能通过所谓“通用人工智能”达到“超智能”的目标，也一直存在着争议。因此我们认为这些风险只不过是未来的“远虑”而已，不急于考虑。 However, after the rise of deep learning based on big data at the beginning of this century, people's understanding has changed a lot. They feel deeply that the ethical risks of AI are right in front of us and governance is a pressing problem! Why? As you know, deep learning based on big data has been widely used in various fields to complete tasks such as decision-making, prediction and recommendation, with significant impacts on human society. However, people soon found that deep learning algorithms based on big data had problems such as opacity, uncontrollability and unreliability, making it very easy to unintentionally misuse AI technology2, which could bring serious consequences to human society. 可是，当本世纪初基于大数据的深度学习在人工智能中崛起之后，人们的认识有了很大的变化，深切地感到人工智能的伦理风险就在眼前，治理迫在眉睫！这是为什么？大家知道，本世纪初基于大数据的深度学习被广泛地应用于各个领域，用来完成决策、预测和推荐等任务，给人类社会带来很大的影响。但是，人们很快发现基于大数据的深度学习算法具有不透明、不可控和不可靠等缺陷，导致AI技术很容易被无意误用，可能给人类社会带来严重的后果。 As you know, with current AI technology we can generate high-quality text and images that meet a user's requirements through a generative neural network. However, the same neural network can also generate text and images that are full of prejudice (racial, gender, etc.), partiality, and errors and do not conform with the user's requirements. This happens completely out of the user's control. It is conceivable that making decisions or predictions based on these incorrect generated texts could bring serious consequences that undermine fairness and impartiality. 大家知道，根据目前人工智能的技术，我们可以通过生成式神经网络根据使用者的要求生成符合要求且质量良好的文本和图像。但同样的神经网络也可以违背用户的要求生成充满（种族、性别等）偏见、不公正和错误百出的文本与图像，完全不受使用者的控制。可以设想，如果根据这些生成的错误文本做决策或预测，就可能带来破坏公平性与公正性的严重后果。 We previously thought that only when the intelligence of a robot approached or exceeded that of human beings would we lose control of it. Unexpectedly, despite machine intelligence still being so rudimentary, we have already lost control of it, much faster than anticipated. This is the very serious reality facing us. 我们原以为，只有当机器人的智能接近或超过人类之后，我们才会失去对它的控制。没有想到的是，在机器的智能还是如此低下的时候，我们已经失去对它的控制，时间居然来得这么快，这是摆在我们面前很严峻的现实。 Asimov put forward a plan to avoid the ethical3 crisis in the "Three Laws of Robotics", which are: "First, robots must not harm humans, or cause humans to be injured due to inaction; Second, robots must obey the commands of humans, unless these commands conflict with the first law; Third, robots must protect their own existence, as long as this protection does not violate the first or second law". In a word, human beings should maintain firm control over machines. Let machines be slaves to humans! Can this approach solve the ethical crisis posed by machines? The answer is obviously no! 阿西莫夫在《机器人三定律》中曾提经出规避伦理危机的方案，内容是“一，机器人不得伤害人类，或因不作为而让人类受到伤害；二，机器人必须服从人类的命令，除非这些命令与第一定律相冲突；三，机器人必须保护自己的存在，只要这种保护不违反第一或第二定律”。总之一句话，人类应该牢牢把握机器的控制权。让机器做人类的奴隶！这种办法能否解决机器的伦理危机？答案显然是否定的！ In fact, in the early days of "non-intelligent" machines we made "machines completely obey the commands of humans". But if we want machines to develop in the direction of intelligence, we can't let them be completely at the mercy of human beings. We need to give them a certain degree of freedom and initiative. It is based on this principle that generative neural networks make use of the mathematical tool of "probability" to enable machines to generate rich and diverse texts and images. But it’s also for this reason that there is definitely some probability (possibility) of generating text and images that don’t meet the standard required and are harmful. This is the price we have to pay when giving machines intelligence - it’s difficult to avoid. 实际上，让“机器完全听从人类的指挥”，在早期“无智能”的机器中我们就是这样做的。但是如果我们想让机器向智能化的方向发展，就不能让机器完全听候人类的“摆布”，需要赋予它一定的自由度和主动权。生成式神经网络就是根据这个原理，利用“概率”这一数学工具，使机器能够生成丰富多样的文本和图像。但也因为这个原因，就一定存在生成不合格和有害文本与图像的概率（可能性）。这是我们在赋予机器智能的时候所必须付出的代价，难以避免。 So is it possible for us to limit incorrect behavior from a machine by setting strict ethical principles for it? In fact, this is also very difficult! Not only because it is difficult to accurately describe "ethical" principles, but also because such principles - even if they could be defined - would be difficult to implement. To take a simple example, if a self-driving vehicle is driving along an ordinary road, if we stipulate that the vehicle must strictly abide by traffic rules, this "principle" should be very clear. However, if there are also manned vehicles and pedestrians on the road carrying out "intentional or unintentional violations of traffic rules", the self-driving vehicle cannot drive to complete its own task. For example, the self-driving vehicle might need to merge to the left in order to turn left, but be unable to do so because the vehicles in the left lane are not maintaining the prescribed distance between them. This shows that if a self-driving vehicle must strictly abide by traffic rules while also completing the task of reaching its destination, it is difficult to simultaneously attend to both goals in an uncertain traffic environment. We can see that the development of AI will inevitably disrupt the field of ethics and traditional norms. 那么我们有没有可能通过给机器规定严格的伦理准则来限制它的错误行为？实际上，这也很困难！不仅因为“伦理”的准则很难准确描述，但即便可以定义，也很难执行。举一个简单的例子，比如自动驾驶车（或无人车）行驶在普通的马路上，如果我们规定自动驾驶车必须严格遵守交通规则，这个“准则”应该是很明确的。但如果路上同时还有“有意或无意违反交通规则”的有人车和行人，自动驾驶车则无法行驶去完成自身的任务。比如，自动驾驶车需要向左并线以便左拐，由于左路车道上的车辆之间没有保持规定的车距，自动驾驶车就无法实现向左并线。这恰恰说明，自动驾驶车一方面要严格遵守交通规则，另一方面要完成达到目的地的任务，在不确定的交通环境下，这两项目标是难以兼顾的。可见，人工智能的发展必然带来对伦理和传统规范的冲击。 The insecure, untrustworthy and brittle nature of deep learning algorithms also brings opportunities for intentional abuse. People can maliciously exploit the brittleness (non-robustness) of algorithms to attack them, resulting in the failure of AI systems based on such algorithms, or even destructive actions. Deep learning can also be used to fake things - that is, so-called "deepfaking". Through AI "deepfaking", a large volume of realistic fake news (fake videos), fake speech (fake audio), etc. can be produced, disturbing social order and framing innocent people. 深度学习算法的不安全、不可信与不鲁棒，同时给有意的滥用带来机会。人们可以恶意利用算法的脆弱性（不鲁棒）对算法进行攻击，导致基于该算法的人工智能系统失效，甚至做出相反的破坏行为。深度学习还可以用来造假-即所谓“深度造假”，通过AI的“深度造假”，可以制造出大量逼真的假新闻（假视频）、假演说（假音频）等，扰乱社会的秩序、诬陷无辜的人。 Intentional abuse and unintentional misuse of AI both need governance, but the nature of governance is completely different for these two problems. The former will require legal constraints and oversight by public opinion: a form of governance involving compulsion. The latter is different. It will require formulating corresponding evaluation standards and rules, conducting strict scientific evaluation and end-to-end supervision of the research, development and use of AI, and taking remedial measures after problems occur to help people avoid misuse of AI. 人工智能无论是被有意的滥用还是被无意的误用都需要治理，不过对这两者的治理性质上完全不同。前者要靠法律的约束和社会舆论的监督，是带有强制性的治理。后一种则不同，需要通过制定相应的评估标准和规则，对人工智能的研究、开发和使用过程进行严格的科学评估和全程监管，以及问题出现之后可能采取的补救措施等，帮助大家避免AI被误用。 Fundamentally speaking, the research, development and application of AI need to be people-oriented; we need to make responsible AI, starting from ethical principles of impartiality and fairness. To this end, we need to work hard to establish interpretable and robust AI theory. Only on this basis can we develop safe, trustworthy, controllable, reliable and scalable AI technology, and ultimately promote the fair, impartial and universally beneficial application and industrial development of AI. This is the line of thinking we advocate for in developing the third generation of AI. 从根本上来讲，人工智能的研究、开发与应用都需要以人为本，从公正、公平的伦理原则出发，做负责任的人工智能。为此，我们需要努力去建立可解释、鲁棒的人工智能理论，在此基础上，才能开发出安全、可信、可控、可靠和可扩展的人工智能技术，最终推动人工智能的公平、公正和有益于全人类的应用和产业发展。这就是我们提倡的发展第三代人工智能的思路。 AI research and governance need the participation and cooperation of people in different fields all over the world. In addition to those engaged in AI R&amp;D and use, they also need the participation of people in different fields such as law, morality and ethics. We need to clarify the standards of ethics and morality and what it means to be "moral" and "ethical". Different countries, ethnic and other groups and individuals have different understandings. Therefore, we need global cooperation to jointly develop a set of standards that are in line with the interests of all mankind. Human beings are a community of shared destiny4. We believe that through joint efforts, we will find standards that meet the common interests of mankind. Only when researchers, developers and users of AI abide by jointly formulated principles can AI develop healthily and benefit all mankind.” 人工智能研究和治理都需要全世界不同领域人员的参与与合作，除从事人工智能的研发和使用人员之外，还需要法律、道德、伦理等不同领域人员的参与。我们需要明晰伦理、道德的标准，什么是符合“道德”和“伦理”的，不同的国家、民族、团体和个人都有不尽相同的认识，因此需要全球范围的合作，共同制定出一套符合全人类利益的标准。人类是命运的共同体，我们相信通过共同的努力，一定会找到符合人类共同利益的标准。只有人工智能的研究、开发和使用人员，人人都遵守共同制定的原则，才能让人工智能健康地发展并造福于全人类。 Translator’s notes&nbsp; 1.&nbsp;It is not clear whether ‘subjective consciousness’ (主观意识), here presented as a necessary condition for superintelligence, entails the ability to feel, an awareness of one’s situation in the world, or something else. 2. Misuse (误用) can also be translated as ‘use incorrectly,’ and is distinct from abuse (滥用).&nbsp; 3. Zhang uses the term ‘ethical crisis’ to refer to a problem that AI safety researchers would typically describe as a control/safety problem. 4. This draws on a slogan commonly used in Chinese politics and diplomacy. Other Authors <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/f1826d016d72655d578dd2ce9192cef78febba10057f69499d01c991bf9dd3d4/c.jpg" data-mid="180028240" border="0" alt="Wen GAO" data-caption="Wen GAO" src="https://freight.cargo.site/w/900/i/f1826d016d72655d578dd2ce9192cef78febba10057f69499d01c991bf9dd3d4/c.jpg" /> <img width="778" height="934" width_o="778" height_o="934" data-src="https://freight.cargo.site/t/original/i/879a6eea0588e12ecdf4808b1ff339da1c4edf5f031deaf45b8d14408916f46d/IMG_6395.JPG" data-mid="188597426" border="0" alt="Andrew YAO" data-caption="Andrew YAO" src="https://freight.cargo.site/w/778/i/879a6eea0588e12ecdf4808b1ff339da1c4edf5f031deaf45b8d14408916f46d/IMG_6395.JPG" /> <img width="302" height="362" width_o="302" height_o="362" data-src="https://freight.cargo.site/t/original/i/d9b9f5d5e8b90ea849b5a9b7169520ebc1f2b9a5da94db47fcc26eef8bfe8431/IMG_6393.JPG" data-mid="188597428" border="0" alt="Ya-Qin ZHANG" data-caption="Ya-Qin ZHANG" src="https://freight.cargo.site/w/302/i/d9b9f5d5e8b90ea849b5a9b7169520ebc1f2b9a5da94db47fcc26eef8bfe8431/IMG_6393.JPG" /> https://chineseperspectives.ai/Xiuquan-LI Tue, 23 May 2023 18:34:01 +0000 Chinese Perspectives on AI Safety https://chineseperspectives.ai/Xiuquan-LI Xiuquan LI 李修全 About the author Xiuquan Li is Deputy Director of Center for Artifcial Intelligence at the Institute of Scientific and Technical Information of China. He is also Deputy Director of the Ministry of Science and Technology’s Next Generation AI Development Research Center. His main research areas include: AI technology trends, technology roadmaps for intelligent industries, and AI development strategies and innovation policies.&nbsp;Projects he has led include “Preparation of China’s Next Generation AI Development Annual Report,” and “Research on Frontier Trends and Policies of AI at Home and Abroad.” He has published over 40 academic papers and contributed to more than 10 published books. 关于作者李修全，工学博士，中国科学技术信息研究所人工智能中心副主任，科技部新一代人工智能发展研究中心副主任。主要研究领域包括：人工智能前沿技术趋势、智能化产业技术路线图、人工智能发展战略及创新政策等。主持《中国新一代人工智能发展年度报告编制》《国内外人工智能前沿趋势与政策研究》等课题10余项，发表学术论文40余篇，参与出版著作10余部。 <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/fccde0537f2601b71a2777693204bc4576929944b625638c208a732c0c55e1b1/e.jpg" data-mid="179714837" border="0" src="https://freight.cargo.site/w/900/i/fccde0537f2601b71a2777693204bc4576929944b625638c208a732c0c55e1b1/e.jpg" /> The following excerpt is a translation of Section 11.2.3 of Li’s The Intelligent Revolution: The Evolution and Value Creation of AI Technology (智能化变革：人工智能技术进化与价值创造) (2021), entitled ‘Concerns about whether the possible emergence of superintelligence is controllable’. Sections 11.2.1 (Safety/security risks brought by imperfections in the technology itself), 11.2.2 (Safety/security challenges caused by the abuse of technology) and 11.3 (Towards AI that humans can understand) also contain some material related to AI safety risks.&nbsp;&nbsp; ▶ Cite Our Translation Concordia AI. “Xiuquan Li — Chinese Perspectives on AI Safety.” Chineseperspectives.ai, 29 Mar. 2024, chineseperspectives.ai/Xiuquan-Li.▶ Cite This Work&nbsp;李修全(2021). “智能化变革：人工智能技术进化与价值创造”. 得到. https://www.dedao.cn/ebook/detail?id=rEQKv6PKN7rEo2Gxg96ZjApyMvQVlw5GmjWXb14PJzDkYaReqd8n5LOmB8d7egBx Translation People’s most long-term concerns about AI safety risks stem from the fear of superintelligence becoming uncontrollable. Assume that AI develops to the stage of superintelligence, can evolve autonomously, has human-like self-awareness, and has its own worldview and values. At this time, the machine’s own needs may take precedence over human instructions, and it can change the behavioral goals set by designers or users, thereby posing a threat to human dominance and controllability. In other words, we are not afraid of AI being “smart,” but rather that it “develops a mind [of its own].”1 原文 “人们对人工智能安全风险最远期的担忧，源于对超级智能出现不可控风险的恐惧。假设人工智能发展到超级智能阶段，能够自主进化，具有类人的自我意识，拥有自己的世界观、价值观，这时机器的自我诉求有可能优先于人类的指示，并且能够自行变更设计人员或使用者设定的行为目标，从而对人类的主导性和可控性造成威胁。也就是说，我们其实不是怕人工智能“聪明”，而是怕它“长了心眼”。 According to Roland Berger Consulting, there are currently two main types of AI: weak AI and strong AI. But when technology surpasses the limits of human beings, super AI may also appear. If humans cannot predict and control it, they may even be destroyed by super AI that is smarter and more powerful than them. This is the main aspect of the public’s fear. British physicist Stephen Hawking worried that human beings are limited by slow biological evolution and completely lack competitiveness; AI might catch up with or even surpass them one day in the future. Bill Gates, Elon Musk, etc. are also worried that the unconstrained development of AI technology will allow machines to acquire intelligence beyond that of humans and will cause some safety hazards that are hard to control. 根据罗兰贝格咨询的观点，当前主要存在两种类型的人工智能：弱人工智能和强人工智能。但当技术突破了人类的极限，可能还会出现超人工智能。如果人类无法预测和控制超人工智能，甚至很有可能会被比自己更聪明、更强大的超人工智能所毁灭，这是公众产生恐惧感的主要方面。英国物理学家斯蒂芬·霍金担心，人类要受到缓慢的生物进化的限制，根本没有竞争力，人工智能也许会在将来的某一天赶上甚至超过人类；比尔·盖茨、埃隆·马斯克等也担忧，对人工智能技术不加约束的开发，会让机器获得超越人类智力水平的智能，并引发一些难以控制的安全隐患。 Few people question whether strong AI can be realized; this is the founding intention of AI and the direction that the field is working hard towards. Knowledge representation, cross-domain learning, common sense and semantic understanding, logical reasoning, etc. are constantly strengthening the ability of AI to imitate human thinking. Machines can even properly understand a book — it is obviously inappropriate to call this weak AI. However, there has been continued controversy over recent decades as to whether artificial objects such as intelligent machines can form self-identity and self-awareness (e.g., a sense of existence, selfishness, and beliefs), whether they should have consciousness, and whether human beings can control them if they do have consciousness. 对于能否实现强人工智能，很少有人质疑，这也是人工智能创立的初衷和努力方向，知识表示、跨领域学习、常识和语义理解、逻辑推理等都在不断强化人工智能模仿人类思维的能力，机器都能真正读懂一本书了，称它为弱人工智能显然不适合。但对于智能机器这样的人工物能不能形成自我认同和存在感、自私、信仰等自我意识，是否应该具有意识，如果智能机器具有意识人类能否掌控等问题，近几十年来在科学界、产业界持续争论。 American philosopher John Searle published a paper in 1980 which proposed the famous Chinese room experiment and contained many arguments and discussions relating to whether AI based on designing programs can have the ability to understand, or in other words a real mind2. Searle believed that silicon-based electronic systems also have their physical and chemical limitations and capability boundaries. Many advanced intelligent behaviors of the human brain (such as consciousness) require the participation of physical and chemical processes, and the difference [between the human brain] and using silicon and metal to simulate the brain to generate intelligent systems could be huge. 美国哲学家约翰·塞尔在1980年发表论文提出了著名的中文屋实验，对基于设计程序的人工智能是否能够具有理解能力或者说真正的心灵进行了大量论证和讨论。塞尔认为，硅基电子系统也有其物理化学局限性和能力边界，人脑很多高级智能行为（如意识），是需要物理化学过程参与的，用硅和金属作为材质，模拟脑生成智能系统，其差异可能是巨大的。 At that time, in Searle’s research, the ability to understand and ‘mind’ were still considered equivalent. In fact, with the development of cognitive intelligence in recent decades, it is already possible to use knowledge modeling and reasoning techniques to extract part of the knowledge in a book and continue to accumulate it into existing machine-based knowledge systems. We are getting closer and closer to strong AI that can understand books. As envisioned in the book Homo Deus: A Brief History of Tomorrow, a species with a high level of intelligence and no consciousness may appear in the future. 当时塞尔的研究中，还是把理解能力与心灵等同起来考虑，实际上随着近几十年来认知智能的发展，通过知识建模和推理技术，已能够实现抽取一本书中的部分知识，并不断累积到现有机器化知识体系中，我们已经距离机器能读懂书的强人工智能越来越近了。就像《未来简史》一书中展望的，未来可能出现一种高智能水平而没有意识的物种。 Roger Penrose claimed in his book “Shadows of the Mind” that the current AI research paradigm is unlikely to produce consciousness. He believes that human consciousness may be related to some kind of overall quantum state that occurs in large areas of the brain and that these collective quantum effects may be produced in the smallest unit of human brain tissue structure — microtubules. 罗杰·彭罗斯（Roger Penrose）在其出版的《心灵的阴影》一书中认为，当前人工智能研究范式不太可能产生意识，人脑意识可能与发生在大脑的大面积区域的某种全局量子态有关。而这些集体量子效应有可能产生于人脑组织结构最小单元—微管中。 As for whether AI can gain consciousness in the future, there is still no school of thought holding any point of view that can produce strong enough evidence for everyone to recognize. Perhaps discussion on this issue will continue for quite some time in fields such as AI academia and social science. How to give full play to the advantages of subjective human consciousness, combine it with the powerful ability of machine intelligence to solve problems, and move towards an integration of humans and machines in which the advantages of both complement each other has become the main development trend of AI. 对于人工智能在未来能否获得意识，目前尚看不到持哪一观点的学派能拿出足以令所有人认可的强有力证据。或许关于这一问题的讨论还将在人工智能学术界和社会科学等领域持续相当一段时间。而如何发挥人类意识的主观性优势，结合机器智能解决问题的强大能力，优势互补走向人机共融则成为目前人工智能主要发展趋势。 As far as the development stage is concerned, AI has gone through the stage of computational intelligence, is currently at a stage of great development of perceptual intelligence and is beginning to move towards cognitive intelligence. At present, the narrow abilities of many AI systems have far surpassed those of humans, and the gap is widening at an extremely fast pace. For example, in most games with clear rules, humans are no longer worthy opponents [for AI]. But AI surpasses humans only in specific skills; semantic understanding, logical reasoning, creativity, etc. are still relatively weak, and the flexibility and generality to solve problems across fields is far from being achieved. 就发展阶段来看，目前人工智能已经走过了计算智能阶段，正处于感知智能大发展阶段，并向认知智能开始迈进。现在很多人工智能系统的专项能力已经远远超越人类，并在以极快的速度拉大这种差距。比如，绝大部分具有明确规则的游戏人类都已不是对手。但目前人工智能超越人类只是在特定技能方面，语义理解、逻辑推理、创造力等还较弱，能够跨领域解决问题的灵活性和通用性还远未具备。 From the perspective of technological paths, AI based on silicon-based material and binary digital signal calculations is ultimately a productivity-transforming technology with great potential, and the intelligent transformation that it is leading will be an advanced stage of the information technology revolution. New computing architectures such as biological computing, neuromorphic computing, and quantum computing that scientists are exploring will have a profound impact on the long-term development of AI. Some experts even believe that they may lead AI to develop to a stage beyond the information technology revolution. However, these cutting-edge computing technologies are still in the stage of striving towards the computation power of the first transistor computer. A way to use new architectures to realize advanced human-like intelligence, such as the capacity for autonomous learning, generalization and deduction, has not yet been found. In addition, human beings are still far from fully understanding themselves, especially the working principles of the 100 billion-neuron human brain and have not seen a substantial scientific breakthrough in the understanding of consciousness. 从技术路线上看，以硅基材料0-1数字信号计算为基础的人工智能终究是一项潜力巨大的生产力变革技术，其引领智能化变革也将是信息科技变革的高级阶段。科学家们在探索的生物计算、神经形态计算、量子计算等新的计算架构会给人工智能远期发展带来深刻影响，甚至也有专家认为有可能引领人工智能向超越信息科技变革的新阶段发展。但目前这些前沿计算技术仍处在向第一台晶体管计算机的算力水平努力阶段，如何基于新的架构实现类似人脑的自主学习、泛化推演等高级智能还没有找到实现途径。另外，现在人类离完全了解自身，尤其是了解拥有1000亿神经元的人类脑神经系统的工作原理还差得很远，还没看到在关于意识的认识上实现科学层面实质性突破。 All in all, it is unclear whether superintelligence or a “singularity” will arrive, and there are no technical signs of development in that direction. Mr. Kai-Fu Lee believes that “at least at present, human beings are still quite far away from the threat of super AI.” We don't need to worry too much about this, let alone feel inexplicable fear. 综合来看，我们还不清楚超级智能或“奇点”是否会到来，从技术上也还看不到向那个方向发展的迹象。李开复先生认为，“至少在目前，人类离超人工智能的威胁还相当遥远”。对此我们完全不需要过于忧虑，更没必要产生莫名其妙的恐惧感。 Of course, due to uncertainty over the emergence of superintelligence and the disastrous consequences that may be caused by uncontrollability, it is still necessary to be vigilant and pay attention to the issue of controllability in the process of AI technology R&amp;D. As early as the 1940s, Asimov had thought deeply about the possible threats brought by the future development of AI and proposed the “Three Laws of Robotics” to constrain the research of robots, so as to ensure safety and controllability in a future where robots and humans coexist. Academician Bo Zhang of Tsinghua University also said that the unique consciousness of human beings is our last line of defense. Mr. Kai-Fu Lee believes that when it comes to superintelligence, we should not make the leap unless we first clearly address all control and security issues. In future, the issue of preventing the loss of control of superintelligence will still be the bottom line in the scientific exploration of AI3. 当然，由于超级智能出现的不确定性和不可控可能导致的灾难性后果，在人工智能技术研发过程中，仍需对可控性问题予以警惕和重视。早在20世纪40年代，阿西莫夫就已经对人工智能未来发展可能带来的威胁进行了深入思考，并提出用于约束机器人研究的“机器人三定律”，以确保机器人与人共存之后的安全可控。清华大学张钹院士也曾表示，人类特有的意识是我们最后一道防线。李开复先生认为，对于超级智能，除非我们首先明确解决了所有控制和安全问题，否则我们不应该跨越。未来，防范超级智能失去掌控的问题仍将是人工智能科学探索过程中的底线思维。 The consensus view of most experts is that, judging from the response to governance questions concerning high-risk technologies such as nuclear energy and cloning, human beings have strong self-discipline and self-adjustment ability. We still have enough time to observe and prepare. Through proactive precautions and a prescient response, the development of AI will eventually be controllable by humans. 大部分专家的共识性观点认为，从对核能、生物克隆等高风险技术的治理问题的应对情况来看，人类拥有强大的自我约束、自我调整能力。我们还有足够的时间观察和准备，通过主动防范和前瞻应对，人工智能发展终将是人类可控的。 Therefore, the three types of safety/security and control risks of AI must be treated differently and dealt with separately4. Those bringing the risks must be the ones to address them; the safety of intelligent models and methods should be the key direction of the current new generation of AI technology R&amp;D, and an active response5 is required for the safety risks caused by shortcomings with the technology itself. To respond to safety/security challenges triggered by abuse and malicious use of the technology, governance must move in step with technological development. Through the construction of laws and regulations and strict and effective oversight and accountability, we must ensure that intelligent new technologies are used to improve people’s livelihood and well-being, and we must prevent technology from doing evil. Whether the superintelligence that might emerge would be controllable should mainly be left to the scientific community to discuss. We, the public, don’t have to worry yet, but in terms of technology R&amp;D, we must also adhere to the original intention of designing AI to assist humans, and ensure that humans have the ability to effectively intervene in circumstances where machines endanger human interests. 因此，对于人工智能的三类安全可控风险，还须区别对待和分别应对。解铃还须系铃人，智能模型与方法的安全性问题应是当前新一代人工智能技术研发的重点方向，积极应对技术自身不完善带来的安全风险；应对技术滥用、恶用引发的安全挑战更多需要技术治理同步跟进，通过法律法规等制度建设和严厉有效的监管问责，保障智能化新技术用于改善民生福祉，杜绝技术做恶；对可能出现的超级智能是否可控问题，主要还应留给科学界探讨，我们大众尚不必担忧，但技术研发上也需坚持设计AI用来协助人类的初衷，并确保在机器危害人类利益的情况下，人类有能力进行有效干预。 Translator’s notes&nbsp; 1. ”心眼“, here translated as mind, combines the characters for “heart/mind“ and “eye”, and conveys a capacity for cognition, feeling and intention. 2. “心灵”, which can also be translated as “soul”, is here translated as “mind”, which is the term used in Searle’s paper. 3. “底线思维” (bottom line thinking) has been commonly used by President Xi and other political/policy figures since the 18th CCP Party Congress in 2012. According to a CCP news website, it means “preparing for the bad points and working at the same time to obtain the best possible results. It’s about being prepared, not panicking, and firmly grasping the initiative.” 4.&nbsp;Earlier in the book, these are defined as risks from: (1) the AI technology itself being immature and systems being imperfectly designed; (2) the subjective tendency of those designing or manipulating the technology to abuse it; (3) superintelligence with self-awareness. 5. Here translated as, “Those bringing the risks must be the ones to address them”, “解铃还须系铃人” is a Chinese idiom that literally means “Untying the bell [from the tiger’s neck] requires the person who fastened the bell”. Other Authors <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" data-mid="179714714" border="0" alt="Bo ZHANG" data-caption="Bo ZHANG" src="https://freight.cargo.site/w/900/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" /> <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/d078330bdbdcb3b8b48f904739bb45797248f658c7e2429bb26415707a85ae84/f.jpg" data-mid="188596864" border="0" alt="Binxing FANG" data-caption="Binxing FANG" src="https://freight.cargo.site/w/900/i/d078330bdbdcb3b8b48f904739bb45797248f658c7e2429bb26415707a85ae84/f.jpg" /> <img width="533" height="640" width_o="533" height_o="640" data-src="https://freight.cargo.site/t/original/i/d5a5c7b4ad8bfe31065a49a2b60f019a9bacf79c94a4138978fd3ebaf6c438dd/IMG_6394.JPG" data-mid="188596871" border="0" alt="Yi ZENG" data-caption="Yi ZENG" src="https://freight.cargo.site/w/533/i/d5a5c7b4ad8bfe31065a49a2b60f019a9bacf79c94a4138978fd3ebaf6c438dd/IMG_6394.JPG" /> https://chineseperspectives.ai/Yi-ZENG Mon, 21 Aug 2023 02:15:27 +0000 Chinese Perspectives on AI Safety https://chineseperspectives.ai/Yi-ZENG Yi ZENG 曾毅 About the author Yi ZENG is a Professor and Director at the Brain-inspired Cognitive Intelligence Lab, and the International Research Center for AI Ethics and Governance, both at the Institute of Automation, Chinese Academy of Sciences. He is the founding Director of Center for Long-term AI, and the AI for Sustainable Development Goals Cooperation Network. He is a board member for the National Governance Committee of Next Generation Artificial Intelligence, China. He is an member of UN High-level Advisory Body on AI, an Expert in the Ad Hoc Expert Group on AI Ethics, UNESCO. He is in the WHO Expert Group on the Ethics/Governance of Artificial Intelligence for Health. His major research interests focus on Brain-inspired Artificial Intelligence, AI Ethics and Governance, AI Safety, and AI for Sustainable Development.关于作者曾毅，中国科学院自动化研究所研究员，类脑智能实验室副主任，人工智能伦理与治理研究中心主任；中国科学院大学岗位教授，博士生导师；国家新一代人工智能治理专委会委员；联合国(UN)人工智能高层咨询机构专家组专家；联合国教科文组织(UNESCO)人工智能伦理特设专家组专家；世界卫生组织(WHO)人工智能伦理与治理专家组专家；主要研究领域是类脑人工智能、人工智能伦理、安全与治理、人工智能与可持续发展。 <img width="708" height="849" width_o="708" height_o="849" data-src="https://freight.cargo.site/t/original/i/b48e96fe846ff2d044ac5ee9fbdef84f48d388d830b91633d4ff2eec10256bd9/WechatIMG2235-2.jpeg" data-mid="191536521" border="0" src="https://freight.cargo.site/w/708/i/b48e96fe846ff2d044ac5ee9fbdef84f48d388d830b91633d4ff2eec10256bd9/WechatIMG2235-2.jpeg" /> On July 18, 2023, the UN Security Council sat down for the first time to discuss the potential threats of AI to world peace and security. Yi Zeng delivered a briefing titled “Opportunities and Risks for International Peace and Security,” suggesting that “in the short-term and the long-term, the risk of AI replacing and causing the extinction of humankind will be present” and that “in the long-term, we haven’t given superintelligence any practical reasons why they should protect humankind.” Yi Zeng has signed the open letter to Pause Giant AI Experiments that “call[s] on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.” Yi Zeng is also among the signatories of the Statement on AI Risk from the Center for AI Safety, which states that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” The following is a translation of a Chinese media interview (搜狐科技《思想大爆炸——对话科学家》栏目第六期) during which he explained his support for both initiatives.Yi Zeng has also conducted surveys investigating Chinese views on the aforementioned open letter and on whether we can and should develop strong AI, with results available in English. ▶ Cite Our TranslationConcordia AI. “Yi Zeng — Chinese Perspectives on AI Safety.” Chineseperspectives.ai, 29 Mar. 2024, chineseperspectives.ai/Yi-Zeng. ▶ Cite This Work 曾毅(2023). “中科院自动化所曾毅：未来AI智力水平或全面超越人类，能否共生最大的瓶颈在人”. 在思想大爆炸——对话科学家栏目上的发言. https://www.sohu.com/a/686209288_115565 Translation Q: We noticed that you signed two recent AI statements. Why did you sign them? Zeng: Both statements recognize the risks in the current development of artificial intelligence and the potential for losing control, but significant differences exist in how the two letters respond to the problem. “Pause Giant AI experiments” calls for pausing research on AI systems more powerful than GPT-4 and prioritizes the design and implementation of a safety protocol for AI. The new “Statement on AI Risk” pronounces that “mitigating the risk of AI extinction should be a global priority alongside other society-scale risks such as pandemics and nuclear war,” which more deeply and directly expresses the signatories’ concerns about the potential existential risks that AI poses to humanity, as well as actions to be taken. My understanding of the issue is close to such views, so I signed the new statement before it was even officially released. I believe the vision shared by the vast majority of people working to develop artificial intelligence is to use it to benefit mankind, rather than to bring risks – potentially existential risks – upon mankind. Therefore, the vast majority of people have the right to know about the potential risks of AI, and developers have the obligation to ensure that AI does not pose existential risks to humanity, or at least to minimize the possibility of such risks through stakeholders. It is difficult for a few people to change existing trends, but when a few people take the first step to raise public awareness, more and more people will ultimately participate in changing the status quo. 原文 问：关注到您在最近的两份AI声明上都有签名，您为什么会签署？ 答： 两次声明都意识到目前人工智能发展过程中的风险和失控的可能性，但应对的方式有显著的差别。《暂停人工智能巨模型实验》号召通过暂停能力超越GPT-4的人工智能巨模型的研究，优先为人工智能设计并实现安全框架。新的《人工智能风险声明》号召“减轻人工智能灭绝的风险应该与流行病和核战争等其他社会规模的风险一起成为全球优先事项”更深度和直接地表达了签名者对人工智能给人类带来潜在生存风险的担忧和应采取的行动。我在这个问题的认知与这样的观点接近，因此在新声明正式发布前就签署了。绝大多数人发展人工智能的愿景，我想应当是用人工智能造福人类，而并非是给人类带来风险，甚至是生存风险。因此绝大多数人有权利知道人工智能的潜在风险，研发者有义务确保人工智能不给人类带来生存风险，至少要通过利益相关方最小化这种风险的可能性。少数人很难改变趋势，但少数人首先站出来提升公众的意识，最终参与改变现状的就会是多数人。 Q: Could AI really bring about risks of extinction similar to pandemics and nuclear war? Is the current understanding of AI's risks overstated? Zeng: The common features of the potential existential risks that pandemics, nuclear war, and artificial intelligence may bring to mankind are that they are wide-ranging, concern the interests of all mankind, and have widespread lethality. More importantly, they are all difficult to predict in advance. Regarding the risks of AI, there are at least two possibilities. One concerns AI in the long term. When artificial general intelligence (AGI) or superintelligence emerges, because the intelligence level may be far beyond humans, it will see humans as humans see ants. Many people believe that superintelligence will compete with humans for resources, and even endanger human survival. The other concerns AI in the near term, which is more pressing. Since today’s AI has no real ability to understand and is not truly intelligent, it will make mistakes that humans would not make in ways that are difficult to anticipate.&nbsp;When a certain action threatens the survival of humankind, AI would not understand what humanity is, what life and death are, nor what is existential risk. When this situation occurs, it is highly likely to threaten human survival. Some also hold the view that artificial intelligence can take advantage of human flaws to cause a fatal crisis to human survival. For example, it could use and intensify hostility and hatred, prejudice and misunderstanding between humans. Such artificial intelligence would not even need to reach the level of AGI to pose an existential risk to human beings. In addition, this kind of AI is likely to be maliciously used, misused, and abused by people. That risk is difficult to anticipate and control, especially as the recent progress of artificial intelligence allows AI to use internet-scale data and information. False information generated by generative AI can greatly reduce social trust in the technology. And network communication has made everything interconnected, which amplifies the above risks to a global scale. If we begin to study how to avoid the challenges of long-term artificial intelligence now, its risks can still be manageable, but the risks from near-term artificial intelligence are more urgent. Valuing and managing the safety and security risks of AI does not hinder the development and application of AI, rather, it is a way to ensure the steady and sustainable development of the technology. AI is undoubtedly a driver of social progress, but this does not mean that AI is without potential risks, or that those potential risks can be ignored due to the need to maximize the benefits from AI. The purpose of both statements is not to impede the development of AI, but rather to explore avenues for the steady and sustainable development of AI. 问：AI真的会有类似流行病和核战争的灭绝风险？当前对AI风险的认识是否夸大？ 答：大流行病、核战争与人工智能可能给人类带来的潜在生存风险的共性是波及范围广，关乎全人类的利益，甚至具有广泛致命性，更关键的是都难以提前预测。关于人工智能的风险，有至少两种可能，一种是对远期人工智能的担忧。当通用人工智能和超级智能到来时，由于智力水平可能远超人类，将视人类如同人类视蚂蚁，很多人据此认为超级智能将与人类争夺资源，甚至危及到人类的生存。另一种是针对近期人工智能的担忧，这更为紧迫。由于现在的人工智能没有真正的理解能力，也不是真正的智能，因此会以人类难以预期的方式犯人不会犯的错误。当某种操作会威胁到人类的生存的时候，人工智能既不理解什么是人类，什么是生死，也不理解什么是生存风险。当这种情况发生时，极有可能威胁到人类的生存。也有观点认为，人工智能可以利用人类的弱点对人类的生存造成致命危机，例如利用和加剧人类之间的敌对和仇视、偏见和误解，而这样的人工智能甚至不需要达到通用人工智能的阶段就有可能对人类造成生存风险。加之这种人工智能很有可能被人恶意利用、误用和滥用，而风险几乎难以预期和控制，特别是近期的人工智能进展使得其能够利用互联网规模的数据与信息，生成式人工智能产生的虚假信息极大降低了社会信任，网络通信又已使万物互联，可以使相关风险在世界规模放大。远期人工智能的挑战，我们如果从现在开始研究如何规避，其风险尚有可能应对，但近期人工智能的风险则显得更为紧迫。重视和管控人工智能的安全风险不是阻碍人工智能的发展与应用，而是确保人工智能技术稳健发展。人工智能无疑是社会进步的推进器，然而这并不意味着人工智能没有潜在风险，或者是由于最大化人工智能益处的需求，就可以忽略人工智能的潜在风险。前述两封声明的目的都不是阻碍人工智能的发展，恰恰是在探索人工智能稳健发展的途径。 Q: You mentioned the need to build moral artificial intelligence, yet AI has no moral awareness.&nbsp; How can we ensure that AI gets developed safely? Zeng: Human morality has an innate basis on which ethics in a wider sense can be built, enabling moral reasoning and decision-making. However, the current approach for making AI models ethical is to bind them with rule-based ethical principles and align such intelligent information processing systems with human values and behaviors. This is like building a castle in the air. Without moral intuition as a foundation, without real understanding, it is impossible to realize true ethics and morality. Only when AI is endowed with a certain degree of self-awareness, only after cognitive empathy, emotional empathy, and altruism are realized on this basis, and only when moral intuition is realized to a certain extent on that basis, is it possible to realize truly moral AI. Therefore, it is necessary to take inspiration from the human brain and human evolution to build ethical AI. It will certainly be an extremely arduous path for AI development, but I see no shortcut. 问：您提到要构建有道德的人工智能，AI没有道德意识，要保证发展安全的话该如何解决？ 答：人类的道德具有内生基础，在此基础上通过习得更广泛意义的伦理道德，从而进行道德推理与决策。但目前的人工智能模型合乎伦理道德的做法是用规则化的伦理原则约束，使智能信息处理系统对齐人类价值观与行为。这就好似建构空中楼阁，没有道德直觉作为根基，没有真正的理解，不可能实现真正意义的伦理道德。只有为人工智能赋予一定程度的自我感知，在此基础上实现认知共情、情感共情、利他行为，以此为基础实现一定程度的道德直觉，才有可能实现真正意义的有道德的人工智能。因此需要在人脑和人类演化的启发中，构建脑与心智启发的有道德的人工智能。这必然是一条极其困难而艰辛的发展道路，但我看不到其他捷径。 Q: How do you give AI a sense of morality? Do human morals and ethical values apply to it? Zeng: Morality cannot be instilled and needs to be understood based on moral intuition rather than a set of operational rules. The first thing we need to give AI is the ability to understand, so that it can generate moral intuition, perform effective moral reasoning, and make moral decisions. Moral concepts and ethical values for humans are constructed from human society, and human beings naturally hope that AI conforms to human values and ethical frameworks. But this is inevitably far from enough. Humans’ own views are changing and being re-evaluated all the time. Artificial intelligence is a new medium of exploration and could even assist in improving human values. If AI is allowed to start interacting with the world entirely anew, it will inevitably form a system of values that is different from human values and moral concepts. But that is certainly not what humans want.&nbsp;Therefore, humans hope that the values system of AI can be aligned with that of human beings. But at the same time, human beings should also take inspiration from interaction with AI to improve upon our value system and ethics. 问：AI是机器，如何让它有道德感？人类的道德观念和伦理价值对它是适用的吗？ 答：道德无法被灌输，需要基于道德直觉的理解，而不是处理规则。我们首先需要赋予人工智能的是理解能力，才有可能产生道德直觉，进行有效的道德推理与道德决策。人类的道德观念和伦理价值是为人类社会而建构，人类自然希望人工智能合乎人类的价值观和伦理框架，但这必然是远远不够的。人类自身的看法都在发生着变化并进行重新认知，人工智能是探索的新载体，甚至可以辅助人类完善人类的价值体系。如果让AI完全重新开始与世界互动，必然会形成与人类价值与道德观念有所差异的体系，但这一定不是人类所期望的。所以人类希望人工智能的价值系统能够与人类对齐，但同时人类也应当在与人工智能互动过程中有所启发，以辅助人类价值系统与伦理道德观的完善。 Q: What kind of human-machine relationship do you hope to see in the future? Is the biggest bottleneck human or AI? Zeng: In the future, artificial intelligence may have more characteristics of life, and the level of intelligence may fully reach or even surpass that of human beings. Humans hope that AI will harmoniously coexist with human beings as partners.&nbsp;AI is a mirror onto humanity, and in the process of building AI we should constantly reflect on the relationship of coexistence between humans and other life forms.&nbsp;The superintelligence of the future may see humans as humans see ants today, and if humans can’t treat other types of life with kindness, why should the superintelligence of the future treat humans with kindness? The biggest bottleneck to whether humans and AI can coexist in the future lies in humans, not AI. If superintelligence truly surpasses human beings in all aspects of intelligence, then it should be super altruistic, super moral. In the face of such intelligent life, human morality needs to advance and evolve. 问：您期待的未来人机关系是什么样的？面临的最大瓶颈是在于人类还是在于AI？ 曾毅：未来的人工智能可能会具有更多生命的特征，智力水平有可能会全面达到甚至超越人类，而人类总还是希望人工智能可以作为伙伴与人类和谐共生。人工智能是人类的一面镜子，我们在构建人工智能的过程中应不断反思人类与其他生命之间的关系和相处之道。未来的超级智能视人类可能如现在的人类视蚂蚁，而若人类不能善待其他类型的生命，未来的超级智能又有何理由善待人类呢？人与人工智能未来是否能够共生共存最大的瓶颈在于人类，而非人工智能。如果超级智能真的是在智慧水平上全面超越人类，那么应该是超级利他、超级道德的，在面对这样的智慧生命时，人类的道德需要加速演化。 Other Authors <img width="778" height="934" width_o="778" height_o="934" data-src="https://freight.cargo.site/t/original/i/879a6eea0588e12ecdf4808b1ff339da1c4edf5f031deaf45b8d14408916f46d/IMG_6395.JPG" data-mid="188465809" border="0" alt="Andrew YAO" data-caption="Andrew YAO" src="https://freight.cargo.site/w/778/i/879a6eea0588e12ecdf4808b1ff339da1c4edf5f031deaf45b8d14408916f46d/IMG_6395.JPG" /> <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" data-mid="188448872" border="0" alt="Bo ZHANG" data-caption="Bo ZHANG" src="https://freight.cargo.site/w/900/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" /> <img width="302" height="362" width_o="302" height_o="362" data-src="https://freight.cargo.site/t/original/i/d9b9f5d5e8b90ea849b5a9b7169520ebc1f2b9a5da94db47fcc26eef8bfe8431/IMG_6393.JPG" data-mid="188560537" border="0" alt="Ya-Qin ZHANG" data-caption="Ya-Qin ZHANG" src="https://freight.cargo.site/w/302/i/d9b9f5d5e8b90ea849b5a9b7169520ebc1f2b9a5da94db47fcc26eef8bfe8431/IMG_6393.JPG" /> https://chineseperspectives.ai/Tiejun-Huang Mon, 21 Aug 2023 02:15:37 +0000 Chinese Perspectives on AI Safety https://chineseperspectives.ai/Tiejun-Huang Tiejun HUANG 黄铁军 About the author Dr. Tiejun Huang is a professor at the School of Computer Science, Peking University, and the Dean of Beijing Academy of Artificial Intelligence (BAAI). He is a recipient of the National Science Fund for Distinguished Young Scholars, Changjiang Distinguished Professorship, and National Talent Program for Technological Innovation Leadership. His research focuses on visual information processing and brain-inspired intelligence. He has made important contributions to efficient video coding standards and visual big data analysis and processing frameworks. He proposed the spike vision model and developed ultra-high-speed spike vision chips and systems. He has received one Second Prize of the National Award for Technological Invention and two Second Prizes of the National Award for Progress in Science and Technology. He has published over 300 academic papers and participated in formulating over 20 national, international, and Institute of Electrical and Electronics Engineers (IEEE) standards as the primary drafter. He has authorized over 100 Chinese and international invention patents. He is a fellow of the China Computer Federation, Chinese Association for Artificial Intelligence, and China Society of Image and Graphics.关于作者黄铁军，博士，北京大学计算机学院教授，北京智源人工智能研究院院长，国家杰出青年科学基金获得者，长江特聘教授和万人计划科技创新领军人才。研究方向为视觉信息处理与类脑智能，对高效视频编码标准和视觉大数据分析处理技术体系做出了重要贡献，提出了脉冲视觉模型并开发了超高速脉冲视觉芯片和系统。获国家技术发明二等奖一次、国家科学技术进步二等奖两次。发表学术论文300多篇，作为主要起草人制定国家标准、国际标准和IEEE标准20多项，授权中国和国际发明专利100多项。中国计算机学会、中国人工智能学会和中国图象图形学学会会士。 <img width="720" height="864" width_o="720" height_o="864" data-src="https://freight.cargo.site/t/original/i/d52175e0957fa3ad717ad15be56f22a71cc3dba04fef40f4835ed9a469f8b428/23a22ce9da08d46b614a1a8c09e15a1f.jpg" data-mid="188465960" border="0" src="https://freight.cargo.site/w/720/i/d52175e0957fa3ad717ad15be56f22a71cc3dba04fef40f4835ed9a469f8b428/23a22ce9da08d46b614a1a8c09e15a1f.jpg" /> The following excerpt is a translation of Tiejun Huang’s 2023 BAAI Conference AI Safety and Alignment Forum Closing Keynote (黄铁军：如何构建安全AI，我们知之甚少，讨论无法闭幕).&nbsp;▶ Cite Our TranslationConcordia AI. “Tiejun Huang — Chinese Perspectives on AI Safety.” Chineseperspectives.ai, 29 Mar. 2024, chineseperspectives.ai/Tiejun-Huang. ▶ Cite This Work&nbsp;黄铁军(2023-6-10). “如何构建安全AI，我们知之甚少，讨论无法闭幕”. 在AI 安全与对齐论坛闭幕式上的演讲. https://mp.weixin.qq.com/s/CqEh1T2-Up9jZr7eEardmg Translation This morning, Mr. Sam Altman, the young CEO of OpenAI, kicked off the highly anticipated “AI Safety and Alignment” forum at the BAAI Conference. This fascinating forum concluded with a speech by Professor Geoffrey Hinton, known as the “Godfather of Deep Learning.” Hinton, now approaching his eighties, and Sam, in the prime of his life, both showed us a future without definite answers. 原文 今早，年轻的 OpenAI CEO Sam Altman 先生拉开了本次智源大会最受关注的「AI 安全与对齐」论坛的序幕，这一重磅论坛以「深度学习之父」Geoffrey Hinton教授的演讲结尾。Hinton如今已年近八旬，Sam则是而立之年，他们都给我们展示了一个没有确定答案的未来。 In general, AI is becoming more and more powerful. The risks are evident and growing. That is our reality today. How do we build a safe AI? We still know very little about this. We can draw on historical experiences in managing drugs and nuclear weapons. Academician Andrew Yao discussed quantum computing, a completely unknowable prospect that nevertheless can be regulated to some extent. However, highly complex AI systems generate unpredictable outcomes. Can we effectively explain their mechanisms or attempt to understand their generalization abilities using traditional risk assessment methods? Our explorations have just begun. We face entirely new challenges, and existing experiences and methods may not be able to solve these new problems. In particular, Professor Stuart Russell and Professor Hinton both mentioned the question of whether an AI with its own goals will serve itself or serve humanity. This is an open question that requires careful consideration. 总得来说，AI越来越强大。风险显而易见，与日俱增，这就是我们今天的现实。如何构建一个安全的AI？我们仍知之甚少。我们可以借鉴历史上管理药物、管理核武器等方面的经验。姚期智院士谈到量子计算，那是完全不可知的一个世界，都有办法一定程度上去管控它。但是高度复杂的AI系统产生的难以预测。用传统的风险测试的方法，解释其机制，或试图理解泛化能力，是否有效？所有的探索刚刚开始，我们面临着全新的挑战，原有的经验和方法可能都无法解决些新问题。 特别是，Russell 教授和 Hinton 教授都讲到，如果 AI 有了自己的目标，它到底是服务于自己的目标还是服务于人类？这是一个开放问题，我们需要谨慎思考。 Today, many people believe that General Artificial Intelligence (GAI) refers to an increasingly capable form of AI, and we are excitedly working towards creating it. However, in the field of AI, the accurate term for this is Artificial General Intelligence (AGI), not GAI. 今天，许多人认为通用人工智能指的是通用性越来越强的一种人工智能，我们抱着一种很兴奋的态度去创造它。但是，在 AI 领域中，对其准确的定义应该是 AGI、而非 GAI。 AGI refers to an AI that can match human-level performance in all aspects of human intelligence, adaptively respond to challenges from the external environment, and accomplish any task that a human can do. In short, it is a “superhuman.” Only intelligence that surpasses human capabilities can truly be called AGI. Can we create such an AI? As early as 2015, I believed the answer was affirmative. Hinton pointed out in this forum that we don’t necessarily have to use digital methods and can even use simulated devices to achieve this goal. In my popular science article from 2015, I proposed that with new simulation device materials, we could create such artificial intelligence around 2045. I published that article on January 7, 2015. Almost simultaneously, at a January 2 to 5 AGI conference organized by Professor Max Tegmark in Puerto Rico, experts made predictions about the timeline for achieving AGI. There was significant variation in their opinions. Among the attending experts, half believed AGI could be achieved before 2045, while the other half believed it would come after 2045. Some even thought it would never be achieved. Previously, many people considered this goal to be “too sci-fi.” However, with the emergence of GPT-4, views have changed. AGI 的意思是：在人类的智能所有方面都达到人类水平，能够自适应地应对外界环境挑战，完成人类能完成的所有任务的人工智能。它就是「超人」，一定是比人类强大的智能，才真正叫 AGI。所以，自主智能、超人智能、强人工智能，其实讲的都是一种全面超越人类的智能。我们能否创造出这种人工智能？早在 2015 年，我就认为答案是肯定的。Hinton 在本次论坛中指出，我们不一定用数字的方法，甚至可以用模拟的器件实现这一目标。在2015年的这篇文章中我就提出，用全新的模拟器件材料，在 2045 年前后能够创造出这样的人工智能。我发表那篇科普文章的时间是 2015 年 1 月 7 日。几乎同时，在 2015 年 1 月 2 日- 5 日，在波多黎各举行的，由 Max Tegmark 教授组织的 AGI 的会上，专家们对实现 AGI 的时间进行了预测，不同专家的看法差别很大。 在与会的专家中，有一半的人认为在 2045 年之前能够实现 AGI、当然，也有一半人认为这一时间在 2045 年之后，甚至有人认为永远不能实现。以前，许多人认为这一目标「过于科幻」。但是，随着GPT-4的出现，大家的看法发生了变化。 Should we create this “superhuman” AGI? And what would be the consequences if we do? In fact, the famous Ashby’s Law of Requisite Variety in cybernetics provided us with a conclusion nearly seven decades ago: any effective control system must be as complex as the system it controls. Professor Hinton also pointed out that a simple system cannot control a system that is more complex than itself. As he said, if a frog invents humans, can it control humans? If humans invent an AGI that is more powerful than themselves, theoretically, it is simply impossible for humans to control it. As long as something is more powerful than us, it will become the controller of this world. Currently, our enthusiasm for developing AGI is running high, driven by investment opportunities. However, if our goal is truly to develop an AGI that is more powerful than us and completely under its own control, should we proceed? “To be or not to be?” Tegmark presented various possibilities in his book Life 3.0. His most important point is: it will be a more powerful AGI that determines the fate of the world, not us. Should we create such artificial intelligence? 我们是否应该创造这种「全面超人」的人工智能？如果创造出来了，后果如何？实际上，早在六七十年前，控制论中著名的「阿什比定律」就为我们给出了结论：任何有效的控制系统都必须和它控制的系统一样复杂。Hinton 教授也指出，一个简单的系统是无法控制一个比它更复杂的系统的。也就是他刚才说的，青蛙如果发明了人类，它能控制人类吗？如果人发明了比自己更强大的 AGI，从理论上说根本不可能控制它。只要它比你强大，它就是控制这个世界的控制者。目前，我们对研发通用人工智能的热情高涨，出于投资风口。但是，如果真的将研发一种比我们强大、完全被它控制的 AGI 作为目标，我们是否应该做？「To be or Not to be」？泰格马克在他的《生命3.0》中提出了多种可能性。最重要的观点是：决定这个世界的是更强大的 AGI，而不是我们。我们是否应该创造这样的人工智能？ At the moment, we are in an uncertain phase, I call this “Near AGI.” Everything is controllable as long as it is certain. Uncertainty is what we have to fear. Yet, today we are in a state of uncertainty. Several years ago, the famous AlphaGo displayed better decision-making than any human. Go is an excellent medium for demonstrating decision-making ability.&nbsp;AlphaGo’s decision-making ability, when dealing with complex situations, is stronger than that of 9-dan professionals, meaning it is much better than&nbsp; almost all of us. I invented a spike vision chip called “Electric Eye” that can perceive 1,000 times faster than a human. To robots, humans move as slowly as bugs crawl. It is very difficult for humans to stand a chance against such agents. GPT-4 knows orders of magnitude more than humans. How many books can a person read in a lifetime? It is often said that one can read no more than 10,000 books. On the contrary, the data GPT-4 has is almost complete; if not complete now, it will be complete within three years. Although we do not consider GPT-4 to be a true AGI, its knowledge base and ability to master that knowledge are already strong. Is such a “Near AGI” better than us? Is it more intelligent than we are? None of the guests at today’s forum gave a definite answer in their reports. They did not explicitly say “NO”, “rest assured”, or “today’s AI systems are not as powerful as humans.” And that’s the problem: we don’t know for sure whether AI has already overtaken us, we don’t know when it will, and the problem is in a completely uncontrollable state. If we can deal with risk with the same enthusiasm as for investment, there is at least some possibility of shaping the future. 目前，我们处在一个模糊的阶段。我将其称之为「Near AGI」，任何事情只要确定都是可以把控的，就怕不能确定。而今天，我们就处在一个不能确定的状态。前些年大名鼎鼎的 AlphaGO 的决策能力比任何人都要强。围棋可以充分体现出决策能力。AlphaGO的决策能力，在处理复杂局面的情况下，强于九段高手，它的决策能力几乎比我们所有人都要强得多。我发明了一种脉冲视觉芯片「电眼」，它的感知速度比人快 1000 倍。在机器人眼中，人类的动作像虫子爬行那么慢。人类很难与这样的智能体对抗。GPT-4 所知道的东西，多于人类好几个数量级。每个人一生能读多少书？经常说不会超过1万本书，而它掌握的数据几乎是全量的，如果现在不是全量，3年之内也会全量。虽然我们认为 GPT-4 还不算真正的 AGI，但是其知识储备量和融会贯通的能力已经很强。这样的「Near AGI」比我们强吗？超过我们的智能了吗？今天论坛的所有嘉宾在报告中都没有给大家一个确定的答案。并没有明确说：“NO”，“放心”，“今天的AI系统还不如人类强大呢”。这就是问题所在。我们并不清楚地知道人工智能是不是已经超过我们，不知道它何时会超过我们，该问题处在一个完全无法把控的状态。如果我们能够像投资那么热情一样来应对风险，至少还有一定把握未来的可能。 But, do you believe humans can do it? I don't know. 但是，你相信人类能做到吗？我不知道。 Other Authors <img width="900" height="1080" width_o="900" height_o="1080" data-src="https://freight.cargo.site/t/original/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" data-mid="188448906" border="0" alt="Bo ZHANG" data-caption="Bo ZHANG" src="https://freight.cargo.site/w/900/i/25e206be47242ce777d050d6a3c29fc3346604ee274da768c590186835ae6b8e/d.jpg" /> <img width="533" height="640" width_o="533" height_o="640" data-src="https://freight.cargo.site/t/original/i/d5a5c7b4ad8bfe31065a49a2b60f019a9bacf79c94a4138978fd3ebaf6c438dd/IMG_6394.JPG" data-mid="188568671" border="0" alt="Yi ZENG" data-caption="Yi ZENG" src="https://freight.cargo.site/w/533/i/d5a5c7b4ad8bfe31065a49a2b60f019a9bacf79c94a4138978fd3ebaf6c438dd/IMG_6394.JPG" /> <img width="302" height="362" width_o="302" height_o="362" data-src="https://freight.cargo.site/t/original/i/d9b9f5d5e8b90ea849b5a9b7169520ebc1f2b9a5da94db47fcc26eef8bfe8431/IMG_6393.JPG" data-mid="188568710" border="0" alt="Ya-Qin ZHANG" data-caption="Ya-Qin ZHANG" src="https://freight.cargo.site/w/302/i/d9b9f5d5e8b90ea849b5a9b7169520ebc1f2b9a5da94db47fcc26eef8bfe8431/IMG_6393.JPG" />