Binxing FANG

About the author
Binxing Fang is an expert in cybersecurity and an academician of the Chinese Academy of Engineering. He is the honorary director of the National Computer Network and Information Security Management Center, which sits directly under the Ministry of Industry and Information Technology. At Guangzhou University he is the Director of the Network Security Technology National Engineering Laboratory and the honorary president of the Institute of Advanced Cyberspace Technology. He is known as the "father of China's firewall", and is author of Cyberspace Sovereignty: Reflections on building a community of common future in cyberspace.

关于作者 方滨兴,网络空间安全专家,中国工程院院士。2007年至2013年担任北京邮电大学校长,国家计算机网络与信息安全管理中心名誉主任,广州大学网络空间先进技术研究院名誉院长。他的著作《论网络空间主权》在2017年由北京科学出版社出版。

The following excerpts are translated from Fang’s Artificial Intelligence Safety and Security1 (人工智能安全) (2020). We think the chapters most relevant to existential risks from AI are Chapters 6 (AI-derived safety problems), 7 (AI Actants2), 8 (Safety Hoops for AI Actants), 9 (Safety Evaluation and Detection for AI Actants), and 12 (Looking to the Future of AI Safety). We precede each excerpt with our own bolded summary of the key point and provide section references in brackets after each excerpt.

Cite This Work Fang, Binxing. Artificial Intelligence Safety and Security. Publishing House of Electronics Industry, 2020. Translated by Concordia AI, May 2023.

Selected excerpts

Intelligent weapons may seriously threaten human survival:

"In terms of the possible risks brought by the machine itself, intelligent weapons present the risk of operational errors. In theory, the higher the degree of intelligence of the weapon, the more complex the software controlling its intelligent behavior is, and the higher the probability of failure and errors…

Intelligent weapons will be upgraded to autonomous systems, which may endanger human beings. The development of intelligent weapons from intelligence to autonomy is both a gradual process and an inevitable trend. When intelligent weapons cross the threshold of intelligence and upgrade to autonomy, human beings may also lose the ultimate decision-making authority over intelligent weapons. This leapfrog development of intelligent weapons brings mankind not only the joy of achievements in development and the assurance of achieving goals, but also worries and fears that intelligent weapons may seriously threaten human survival. If actors in a military context lose control of intelligent weapons, those weapons are very likely to cease executing their original military tasks and become threatening enemies. Due to their powerful destructive abilities, intelligent weapons may transform from the main force for attacking the enemy into formidable enemies that are difficult to defeat, with the level of intelligence determining the strength of each side. What’s more, [intelligent weapons] increase the uncertainty of the outcome of the war. Smart weapons which were once the magic weapons for defeating the enemy are likely to become "traitors" that hurt one’s own side. Once emotionless, untiring intelligent weapons are given power over life and death and completely replace soldiers in fighting, will they wantonly kill innocent people because of an "excessive" approach to battle? Will they expand the scope of combat targets and become “humanity’s terminator"? These are the major hidden dangers associated with the possibility of loss of control in AI’s militarization." (6.2.2)




There are three necessary elements for out-of-control AI systems:

"When talking about the threat of AI systems to human beings, the first thing to consider is AI systems with the ability to act. We call these “AI actants" (AIA)...

Although as of the date of finalizing this book, there have been no publicly reported cases of AIAs "actually becoming out-of-control and causing harm", there has long been consensus that there is a risk of AIAs becoming out-of-control. In the process of treating AIAs with a high degree of vigilance, we also need to know that not all AIAs present risks of loss of control - only when certain conditions are met can they become out-of-control. Academician Binxing Fang, the chief editor of this book, made a report entitled “My Views on AI Safety and Security” at the China International Software Expo held on June 30, 2018. This report put forward for the first time "three necessary elements for out-of-control AIA". These three elements are: AIAs have the ability to act and destroy3, have uninterpretable decision-making ability, and have the ability to evolve and can evolve into autonomous systems.” (6.4)




AIA could become an autonomous, conscious system that can fight against humans:

"Once AIA starts to evolve itself or even autonomously set the objective function, it could be possible for it to break away from the “cage” with which humans limit its activity and escape human control. Obviously, if an out-of-control AIA is just like an out-of-control car with no goal, it only presents a derivative safety problem4; however, if AIA evolves to the point where it needs to defend its "right to survive" and does not hesitate to harm human beings in order to protect itself, it will become an autonomous system that can fight against human beings, which will lead to disasters for humanity. Here, our definition of autonomous system is not simply "not controlled by people". Once an autonomous weapon is on the battlefield, it is indeed not controlled by people, but that [loss of control] is limited to the level of decision-making. What we mean by autonomous systems here is that they have "consciousness". Like humans, they know how to protect themselves, and their objective function is to protect themselves from harm. They can even have the ability to socialize and organize. In this case, they will indeed become oppositional to human beings.” (6.4.3)



AIA can learn things humans haven’t taught them: 

"Autonomy is the third characteristic of AIA. Assuming that harm caused by AIA to humans is not due to loss of control, such as in the case of a driverless car losing control and harming humans due to inertia, then the process of choosing to harm humans will bring about more dangerous situations, and the learning ability of AIA will make it difficult for human beings to mitigate such dangers.

Of course, some people think that since AIA is artificial and is a moving object constructed on a closed set, it is impossible for it to do things that human beings have not taught it to do. For example, when operating on the set of binary digits {0,1}, it would never produce an output from the set of alphabetic characters. In particular, researchers in the field of AI sometimes deny that AI systems (AIS) have the ability to create, because they know that everything AIS learn is taught by human beings. If human beings have not taught something, AIS have no reason to be able to do it; at least AIS have no reason to choose a result that has not previously been deemed correct. For example, even though the Boston Dynamics robot introduced in Section 7.2.1 has the ability to walk on two feet, it will not evolve into a master of "Shaolin Kung Fu" on its own.

The problem is that people may overlook one thing, that is, the connective ability of AI (关联能力). If the Boston Dynamics robot has grasped some basic abilities, even becoming a Kung Fu master would not constitute jumping out of the closed set, so long as it has mastered the evaluation function through observation and learning. In fact, suppose we have taught the robot how to select a reward function such that, if it falls over, the reward function gives a negative value and, if it does not fall over, the reward function gives a positive value. At the same time, the robot has the ability to generate random actions, so after a long time of training and trying various results, the robot will inevitably choose the most suitable combination of actions, which is enough to enable it to achieve the optimal state - a state that was originally impossible for humans to teach. After all, there are many reasons why these robots may master capabilities that exceed expectations.” (7.6.3)





AIAs need to be designed with risk mitigation in mind:

"In order to prevent AIAs from threatening human beings, people need to consider risk at the outset of designing AIAs. The first consideration is the need to develop design principles that can protect human safety. The design principles of AIA should include but not be limited to:
(1) Design to minimize risk: First, eliminate hazards in the design. If the known hazards cannot be eliminated, design choices should reduce risk to an acceptable level;
(2) Use safety devices: Use a safety hoop5 or other safety protection device to reduce the risk to an acceptable level;
(3) Use warning devices: Use a warning device to announce danger and send appropriate warning signals to nearby personnel by voice, light, etc.” (9.1)



Safety assessments and evaluation of AI are needed:

[The author writes that if the aforementioned “safety hoop” becomes a necessary component of AIA, it will be necessary to check that the safety hoop works before an AIA can be placed on the market, just as how a car’s brakes and collision avoidance system must undergo testing]

…"More importantly, an evaluation mechanism must be designed to evaluate the level to which an AIA has evolved. For example, can people design an evaluation mechanism to identify what [Go] rank AlphaGo has evolved to? Of course, this should not simply rely on competitions, as is the case currently, but should be a method for fast appraisal. At present, there is no such method because people do not need one. No one worries that AlphaGo will harm mankind, and since for humans Go rank and ability are assessed by competition, the same is true for AlphaGo. However, this does not mean that there is no relevant fast appraisal method. It’s just like how with a classification algorithm, it is relatively easy to quickly evaluate capability as long as the training examples and test examples are separated. So can we extract intermediate processes in Go play to act as evaluation methods? In short, for future technological development and social progress, it is necessary to try to construct corresponding evaluation mechanisms to evaluate how far AIA has evolved.” (12.4)


【作者写道,如果上述“保险箍”成为 AIA 的必要组成部分,那么在 AIA 投放市场之前,必须检查保险箍是否有效,就像汽车的刹车和防撞系统必须接受测试一样】


Translator’s notes 

1. In English, “safety” and “security” typically refer to protection against unintended and intended harms respectively. In Chinese, the word 安全 (anquan) can encompass the meaning of both “safety” and “security.” Throughout the pieces featured on this website, we select the English translation that we think best fits the author’s meaning. Fang’s book covers both unintended and intended harms, so we translate the title as Artificial Intelligence Safety and Security. In the excerpts on this page, the discussion is mostly of unintended harms/loss of control, so we translate anquan as ‘safety.’

2. Defined as systems with the ability to act.

3. Elsewhere in the chapter, the author expands: “AIA’s ability to act comprises two kinds of external manifestations: one is that it can move in physical space and has kinetic energy; the second is that it can exchange energy with the outside world and thus change the state of other objects.”

4. AI-derived/derivative safety problems are defined elsewhere as “AI vulnerabilities threatening safety in other fields,” e.g. safety accidents caused by AI system errors, the development of AI weapons leading to an international arms race, and the potential loss of control of future AI systems. This is distinguished from AI’s internal safety problems, which are vulnerabilities in the AI system itself.

5. Elsewhere, the author expands: “A safety hoop can be inserted between the propulsion system and the decision-making system, and the decision-making system of the robot needs to pass through the safety hoop to allow propulsion to occur. Just like a fuse will automatically blow when current is abnormal to ensure the safe operation of an electrical system, the safety hoop will be activated once certain conditions are met, and then begin to restrict the robot.

Other Authors

Chinese Perspectives
on Existential Risks from AI

Contact us at: