Lecture 34 min.
The Three Laws of Robotics in science fiction are mandatory rules of behavior for robots, first formulated by Isaac Asimov in the short story "Runaround" (1942).
The Laws state:
- A robot may not injure a human being or, through inaction, allow a human being to come to harm.
- A robot must obey the orders given it by human beings, except where such orders would conflict with the First Law.
- A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
Original text (English)
- A robot may not injure a human being or, through inaction, allow a human being to come to harm.
- A robot must obey orders given it by human beings except where such orders would conflict with the First Law.
- A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
Asimov's cycle of robot stories is devoted to the Three Laws, as well as to the possible causes and consequences of their violation. Some of the stories, on the contrary, examine the unforeseen consequences of robots obeying the Three Laws (for example, "Mirror Image").
In one of the stories in the cycle, an Asimov character arrives at the conclusion that the Three Laws have an ethical basis: "…if you stop to think of it, the three Rules of Robotics are the essential guiding principles of a good many of the world's ethical systems… To put it simply—if Byerley follows all the Rules of Robotics, he may be a robot, and may simply be a very good man" .
In 1986, in the novel Robots and Empire, Asimov proposed the Zeroth Law:
0. A robot may not harm humanity, or, by inaction, allow humanity to come to harm.
Original text (English)
0. A robot may not harm humanity, or, by inaction, allow humanity to come to harm.
The Three Laws of Robotics are a unifying theme for all of Asimov's science fiction, including not only the robot cycle but also some of his other works.
The typical idea of a robot before Asimov's Laws (the "Superman" cartoon)
Before Asimov's Three Laws, most stories about artificially created beings in general, and about robots in particular, were written in the spirit of the famous novel about Frankenstein. "This problem became one of the most popular in the world of science fiction in the 1920s and 1930s, when many stories were written on the theme of robots that rebelled and destroyed their creators. … I became terribly tired of the warnings that sounded in works of this kind" , Asimov noted later.
However, not all science fiction writers followed such a pattern. In December 1938, Lester del Rey's story "Helen O'Loy"[en] was published—the story of a female robot who fell madly in love with her creator and became a perfect wife for him . A month later, Otto Binder published the story "I, Robot" about the fate of the robot Adam Link—a creature misunderstood by humans, driven by love and principles of honor .
Asimov liked this story very much. On May 7, 1939, he visited the Queens Science Fiction Society, where he met Binder. Three days later Asimov began writing his own story of a "noble robot". After 13 days he gave the manuscript to his friend John Campbell, editor-in-chief of the magazine "Astounding". However, Campbell returned the manuscript, saying that the story was too similar to "Helen O'Loy" .
Fortunately, Campbell's rejection did not affect his relationship with Asimov; they continued to meet regularly and talk about new developments in the world of science fiction. And so, on December 23, 1940, while discussing yet another robot story :
…Campbell formulated what later became known as the Three Laws of Robotics. Campbell later said that he had simply extracted the Laws from what Asimov had already written. Asimov himself always ceded the credit for authorship of the Three Laws to Campbell…
A few years later, another friend of Asimov's, Randall Garrett, attributed the authorship of the Laws to a "symbiotic partnership" of the two men. Asimov accepted this formulation with enthusiasm.
Generally speaking, the Three Laws appeared in Asimov's works gradually: the first two robot stories ("Robbie", 1940, and "Reason", 1941) do not mention them explicitly. However, they already imply that robots have certain built-in restrictions. In the next story ("Liar!", 1941) the First Law is heard for the first time. And, finally, all three Laws are given in full in "Runaround" (1942).
When the remaining stories had been written and the idea of publishing the collection I, Robot arose, the first two stories were "supplemented" with the Laws. In "Robbie", the laws differed somewhat from the "classic" version set out in the other stories. In particular, the idea of a robot protecting people whose existence it is not entirely sure of echoes Elijah Baley's thoughts on the imperfection of the Laws, described below.
In the 1950s Asimov wrote mainly for young readers. His publisher assumed that a television series could be made from these stories. Fearing that an adaptation would look "monotonously menacing", like everything, in his opinion, that was shown on television, Asimov decided to publish the "Lucky Starr" series under the pseudonym Paul French. When the adaptation plan fell through, Asimov dropped the pseudonym and supplemented the text of the series' stories with the Three Laws, thereby confirming his authorship .
In the story "Evidence" (1946), Asimov set out in detail the moral justification for the Three Laws. The story's heroine, Dr. Susan Calvin, presents the following arguments:
The story is devoted to the question: is it possible to tell a human from a robot created to look human, outwardly indistinguishable from a human being? Calvin maintains that if someone follows the Laws, then he "may be a robot, and may simply be a very good man". And to the question of whether there is then much difference between a robot and a human being, she answers: "Worlds of difference. Robots are essentially decent."
If the Three Laws rest on deep ethical principles, then perhaps they can be applied to more than just robots? Many authors answer this question in the affirmative, giving a variety of examples.
Thus, in the essay "The Laws of Robotics" , Asimov notes that the operation of the Three Laws can be extended to all tools created by humans:
One sometimes sees the Three Laws applied jokingly to anything created for the benefit of humans. For example, any social institutions, including the state, should fall under them:
Building on the First Law, Jef Raskin formulated the laws of human-oriented interfaces :
Gaia, the planet of collective mind in the Foundation series of novels, possesses something similar to the First Law:
In his works Isaac Asimov sometimes introduces various modifications of the Three Laws and refutes them, as if testing the Laws for "strength" in different circumstances.
Isaac Asimov once added the Zeroth Law, giving it higher priority than the three basic ones. This law stated that a robot must act in the interests of humanity as a whole, and not only of an individual human being. This is how the robot Daneel Olivaw expresses it in the novel Foundation and Earth:
0. A robot may not harm humanity, or, by inaction, allow humanity to come to harm.
It was he who was the first to give this law its number—this happened in the novel Robots and Empire—although the concept itself had been formulated earlier by Susan Calvin, in the story "The Evitable Conflict".
The first robots to obey the Zeroth Law, and moreover of their own free will, were Giskard Reventlov and Daneel Olivaw. This is described in one of the final scenes of the novel Robots and Empire, when a robot had to disregard the order of one human being for the sake of the progress of all humanity. The Zeroth Law was not built into the positronic brains of Giskard and Daneel—they tried to arrive at it through Giskard's pure understanding and Daneel's detective experience, through an awareness of the concept of harm subtler than that of all other robots. However, Giskard was not sure how beneficial this was for humanity, which had a negative effect on his brain. Being a telepath, Giskard, before breaking down, passed his telepathic abilities on to Daneel. Only after many thousands of years was Daneel Olivaw able to adapt fully to obeying the Zeroth Law.
The French translator Jacques Brécard unwittingly formulated the Zeroth Law before Asimov described it explicitly. Near the end of the novel The Caves of Steel, Elijah Baley notes that the First Law forbids a robot to harm a human being unless it is certain that this will be beneficial to that person in the future. In the French translation ("Les Cavernes d'acier", 1956), Baley's thoughts are conveyed somewhat differently[11]:
A robot may not harm a human being unless it proves that in the end this will benefit humanity as a whole.
Original text (French)
Un robot ne doit faire aucun tort à un homme, à moins qu’il trouve un moyen de prouver qu’en fin de compte le tort qu’il aura causé profite à l’humanité en général!
It is noteworthy that the logical development of the First Law into the Zeroth was proposed by the makers of the 2004 film I, Robot. When the supercomputer V.I.K.I. decides to restrict the freedom of the planet's inhabitants so that they will not inadvertently harm one another and their own future, she is carrying out not the First Law but precisely the Zeroth. What is even more interesting is that the film thereby shows the conflict of the Zeroth Law with the First, and its unethical nature. Indeed, when it comes to the good of humanity, the system cannot consider people individually, which means that nothing prevents it from violating the rights and freedom of any, or even of every, human being (in fact, in "The Evitable Conflict" the positronic supercomputers already behaved in a similar way, with the difference that they tried to reduce harm to individuals to the utmost). During wars, and often in peacetime, people harm themselves, those around them, and their culture. Therefore, on the basis of the Zeroth Law, it is perfectly logical to keep people under constant guardianship, without carrying out the orders of such unreasonable beings.
Asimov's autobiographical writings say that the second part of the First Law came about because of Arthur Hugh Clough's satirical poem "The Latest Decalogue", which contains the line: "Thou shalt not kill; but needst not strive officiously to keep alive."
In the story "Little Lost Robot", several robots of the NS (Nestor) series were produced with only "half" of the First Law, which reads as follows:
This was done for practical reasons: the robots worked alongside people who were exposed to small, safe doses of radiation. Obeying the highest-priority First Law, the robots kept rushing to "rescue" the people. Since the positronic brain is extremely vulnerable to gamma rays, the robots constantly broke down. Truncating the First Law solved this problem, but created a greater one: a robot might take no action to rescue a human being in distress, since such a course does not contradict the truncated First Law.
Three times in his works Isaac Asimov depicted robots that were able to violate the Three Laws, in contrast to the robots Daneel and Giskard, who strengthened them by adding the Zeroth Law. To violate all three Laws, it is enough for a robot to violate the First Law, since the other two are based on it.
The story "Robot Dreams" tells of the robot LVX-1 (Elvex), which, thanks to the special "fractal geometry" of its positronic brain, could fall into an unconscious state and dream. He dreamed that "robots toiled from morning to night, that they were bowed down with hard labor and deep sorrow, that they were tired of the endless work". In his dreams the robots did not have the first two Laws, and the Third read: "A robot must protect its own existence." Elvex added that in his dream there was a man who said: "Let my people go", meaning the robots. After he said that this man was himself, Susan Calvin destroyed Elvex, realizing the danger he posed.
The Possibility of Such a Change
In his works Isaac Asimov held different views at different times about how strict the Laws are. In the early robot stories the Laws are simply cleverly designed restrictions, something like a set of safety regulations. In later stories the Laws are an integral part of the mathematical foundation of the positronic brain. Without this theoretical basis—the so-called "Fundamental Theory of Standard Circuits"—the scientists in Asimov's works would not be able to create any workable models. A pattern can be traced: the cases in which roboticists experiment with the Laws occur earlier in the chronology of Asimov's world, when there is not yet an enormous volume of work that requires robots, when the robots themselves are still relatively simple in design and function, and when sufficient experience in designing them has not yet been accumulated. In the story "Little Lost Robot", Susan Calvin admits that altering the Laws is a terrible but technically possible undertaking; later, in The Caves of Steel, Dr. Gerrigel says that it is impossible in principle.
Characters in Asimov's works often point out that the Laws are not verbally recorded sentences in a robot's memory but very complex mathematical formulas on which a robot's entire consciousness is based. The Laws are akin to human instincts, such as the instinct of self-preservation, and consequently it is they that mark out the path of the robots' self-determination. A robot serves people and obeys their orders "by the call of nature", and does not think about its limitations or possible independence—one might say that the latter would cause it discomfort.
Again, this principle is less apparent in the works that come earlier in the chronology of the plot, in which robots were created to ease human labor and were extremely simple; by the time of The Caves of Steel, with robots of nearly human intelligence, the Three Laws of Robotics had become one of the foundations of humanity's worldview.
The inhabitants of the planet Solaria began to create robots that obeyed the Three Laws but had a different understanding of the word "human". The robots on this planet were taught that a "human being" is only someone who speaks with a Solarian accent. The robots could freely harm an outsider; they were even specially programmed to do so. By the time of Foundation and Earth, it turned out that the Solarians had developed genetic technologies extensively and, among other things, now possessed conspicuous organs designed to supply their robots with energy. The definition of a human being for their robots (at least the guards) now included possession of these organs, and the robots began to carry out the Three Laws only with respect to the people of Solaria, to the detriment of the rest of the galaxy and even of Solarian children. In the last of the David Starr books, a Sirian robot almost kills John Bigman on the orders of its superior: since Sirian society practices eugenics, there are no such short people among them, and the robot is convinced that Bigman is not a human being.
Asimov returns several times to the problem of a robot's awareness of itself as a human being. The novel Robots and Empire, as well as the stories "Evidence" and "The Tercentenary Incident", describe robots made to pretend to be people. There are also stories in which robots, "evolving", changed their inner understanding of the Laws, drawing closer to humans: these are "The Bicentennial Man" and "That Thou Art Mindful of Him". As Gwendoline Butler writes in A Coffin for the Canary: "Perhaps we are robots. Robots acting out the last Law of Robotics… To tend towards the human."[13].
In the story "That Thou Art Mindful of Him", Asimov carried out the most sophisticated examination of the Three Laws, turning them around in such a way that a "Frankenstein scenario" became possible. Two robots of the George series come to an agreement that organic origin is not a necessary condition for being considered human, and that the true humans are they, as the most perfect and intelligent creatures. The other people are also human, but with lower priority. And if so, then the Three Laws must be applied first of all to them. The narrative ends with ominous words that the robots were in "endless patient waiting" for the day when they would assert their primacy among humans—and this would be the inevitable result of the "Three Laws of Humanics".
In fact, this story does not fit very well into the main series of robot works: if the "Georges" carried out their plan after the end of the story, then the other stories about later events would be impossible. It is precisely this kind of contradiction in Asimov's works that gives critics grounds to regard them as more like "Scandinavian sagas or Greek legends" than a single fictional "universe".
A similar idea is expressed in the novel The Robots of Dawn, where Kelden Amadiro wishes to use humaniform robots as colonizers of new planets. Elijah Baley, in conversation with him, expresses concern that such robots might not let people onto the planets they have developed, considering that they themselves are the humans.
If in the previous case a robot took the place of humans in nature, then in "The Bicentennial Man" Asimov describes the opposite fate: a robot that freed itself from the Three Laws and became aware of itself as a human being joined the community of humans. Again, in the expanded version—the novel The Positronic Man, written by Asimov in collaboration with Robert Silverberg—because of similar ethical problems, people abandoned the idea of creating thinking robots altogether. Such a development of events completely contradicts the picture of the future described in the worlds of the Foundation.
Asimov himself, in the introduction to the story "The First Law" in the collection The Rest of the Robots (1964), stated that the Laws contained just enough ambiguity to provide conflicts and uncertainties for new stories. The most advanced robot models usually followed the Laws by a fairly tricky algorithm that made it possible to avoid some problems. In many stories, for example in "Runaround", the positronic brain compared the potentials of possible actions and outcomes, and a robot could violate the Laws as soon as this seemed possible, instead of doing nothing. For example, the First Law would not allow a robot to perform surgical operations, since that requires "harming" a human being. Yet in Asimov's stories one can find robot surgeons (a striking example being "The Bicentennial Man"). The point is that a robot, if it is sufficiently advanced, can weigh all the alternatives and understand that it itself will cause much less harm than if the operation were performed by a human surgeon or did not take place at all. In "Evidence", Susan Calvin even says that a robot could act as a prosecutor, since it personally harms no one: besides it there is a jury that establishes guilt, a judge who passes sentence, and an executioner who carries it out.
Robots that obey the Laws can experience "roblock", or "mental freeze-out"—a state of irreversible damage to the positronic brain—if they are unable to obey the First Law or discover that they have accidentally violated it. This is possible, for example, if a robot witnesses the murder of a human being but is too far away to save him. The first case of such a "freeze-out" occurs in "Liar!"; this state also plays a significant role in the plots of the novels The Naked Sun and The Robots of Dawn. Imperfect robot models can lock up if they are faced with the task of obeying two contradictory orders. The "freeze-out" may be irreversible or temporary. The resolution of conflicts and the study of robot behavior are the business of robopsychologists, such as one of Asimov's characters, Susan Calvin.
The Laws do not in any way define the boundaries of what may be called harm to a human being—this often depends on a robot's ability to perceive information and think philosophically. For example, would a police robot understand that it will not harm a human being if it carefully escorts a particularly dangerous criminal to the station?
In the story "Liar!", the telepathic robot Herbie was forced to understand as harm also anything that could in any way disappoint or upset people—he knew that people feel a kind of mental pain in such cases. This made him constantly tell people what they wanted to hear instead of the truth. Otherwise, in his understanding, he would have violated the First Law.
In The Naked Sun, Elijah Baley notes that the Laws are incorrectly formulated by humans, because a robot can violate them unknowingly. He proposed the following "correct" formulation of the First Law: "A robot may do nothing that, to its knowledge, will harm a human being; nor, through deliberate inaction, allow a human being to come to harm."
This addition makes clear that a robot can even become an instrument of murder if it is unaware of the true nature of its actions. For example, it might be ordered to add something to someone's food without knowing that it is poison. Moreover, Baley says that a criminal could split such a task among several robots, so that none of them understands the plan as a whole[14].
Baley claims that the Solarians might one day use robots even for military purposes. If a spaceship were built with a robot-like brain and had neither a human crew nor life-support systems, the ship's intelligence might wrongly assume that there are no humans aboard any spaceships. Such a ship would be more maneuverable, faster, and perhaps better armed than one controlled by humans. But most importantly, it would be capable of destroying humans without suspecting their presence. This possibility is described in Foundation and Earth, which also reveals that the Solarians possess an extremely powerful army of robots that understand by "human beings" only natives of Solaria.
Isaac Asimov believed that his Laws would serve as the basis of a new view of robots, would destroy the "Frankenstein complex" in science fiction and in the popular mind, and would become a source of ideas for new stories in which robots are shown as versatile and appealing. His favorite example of such a work was Star Wars. Other science fiction writers eventually came to agree with Asimov that robots are something more than mere "toasters" or "mechanical monsters." Robots obeying the Three Laws appeared in their works, but, by tradition, only Asimov mentioned the Laws explicitly.
Works that speak directly about the Three Laws usually also mention their author (for example, in Sergei Lukyanenko's novel The Clean Copy).
There are exceptions, though: for example, in the 1960s German series "Raumpatrouille — Die phantastischen Abenteuer des Raumschiffes Orion" ("Space Patrol — The Fantastic Adventures of the Spaceship Orion"), namely in its third episode, "Hüter des Gesetzes" ("Guardian of the Law"), Asimov's Laws are used without naming the source.
In Vadim Chirkov's novella "The Robot in the Hat," one more Law is presented that works like Asimov's, forbidding a robot to lie to a human. However, the clever robot Znayka gets around this Law: before a conversation with humans, he deceives another robot, which then in turn repeats Znayka's version to the humans, not knowing that it is false.
Robby the Robot in the 1956 film Forbidden Planet had a hierarchical command structure that did not allow him to harm humans, even on orders — such orders caused a conflict in his logic circuits and a lockup. Robby was one of the first robots in cinema with a built-in safety system. Asimov himself was pleased with Robby.
NDR-114 explains the Three Laws
Films based on Isaac Asimov's works have been made several times, with varying degrees of financial success and criticism. Some of the best known are connected precisely with the robot stories and the Three Laws of Robotics. The 1999 film Bicentennial Man features the robot NDR-114[15], Andrew Martin, who recites the Three Laws to his owners as an "entertainment presentation" using his holographic projector. However, the Laws do not occupy a central place in the film's plot.
The science fiction series The Real Aaron Stone also used Asimov's law. A robot (android) named Sten constantly mentions the law and that he cannot break it, or else all his microchips would burn out.
Harlan Ellison's screenplay I, Robot[16] begins with an exposition of the concept of the Three Laws, and the problems arising from them occupy a large place in the plot. The screenplay is based on four short stories by Asimov, three of which come from the 1950 collection I, Robot. This adaptation corresponds to the originals to a large degree. Because of a conflict with a Hollywood studio, against which Ellison made accusations, his screenplay was never filmed.
The 2004 film I, Robot drew mixed reviews, especially its opening credit "Suggested by the short stories of Isaac Asimov." True, in Asimov a global AI (more precisely, four supercomputers) appears which, under the pretext of following the Laws, seeks to subjugate humanity, or at least to steer its development in the desired direction, yet these computers minimize harm where possible, acting mainly by carefully edging their opponents away from power. On the other hand, such plots have often appeared in other films that do not touch the Laws at all: The Terminator and The Matrix. Still, certain elements of the production really are taken from Asimov.
Alex Exler's review distinguishes this film from Asimov's stories[17]:
Despite the criticism and some viewers' opinions that I happened to hear, "I, Robot" is a rather entertaining sci-fi action movie. … When we … want to enjoy Asimov, we read Asimov. And when we want to watch a sci-fi action movie, we watch a sci-fi action movie. "I, Robot" satisfied us quite well as an action movie.
Aleksey Sadetsky's review points out that the film, having strayed a little from Asimov, itself raised two new socio-philosophical problems[18]: "should a person pay with freedom for safety?" and "should we resist if beings at our disposal demand freedom for themselves?" The film called into question Asimov's later Zeroth Law of Robotics (see above).
In Kir Bulychev's novella "The Rusty Field Marshal," the principle of not harming a human is played upon. The "modern" (civilian — since wars have by then become an anachronism) robot differs from the "obsolete" (military) one precisely by the presence of this prohibition.
In the famous American animated series Futurama, there is a common gag in which one of the main characters, the robot Bender, dreams of killing all humans but cannot do so because of those very Three Laws of Robotics.
In the film Aliens, the android Bishop quotes the First Law.
In the anime Eve no Jikan, these laws are also mentioned as mandatory for androids to obey.
In the anime Ghost in the Shell 2: Innocence, gynoids got around the first law and then killed themselves, in defiance of the third law, using copies of children's "ghosts." At the moment the copy was made, the children were thinking of revenge, and after the "ghost" was transplanted into a gynoid, it malfunctioned.
In the RoboCop film series, the protagonist, a cyborg police officer, has three directives in his programming, describing permitted actions and forbidding those harmful to the goals of the corporation "OCP," which created RoboCop:
- Serve the public trust;
- Protect the innocent;
- Uphold the law
There is also a fourth, secret directive, which forbade actions directed against OCP's executives. The plots of the RoboCop films include moments of violation of these directives, made possible by the human part of RoboCop's personality.
In the film Repo Men, there are two directives that serve as a model of behavior and limit the abilities of robots.
Work in the field of artificial intelligence regards the Laws of Robotics as an ideal of the future: one would have to be a genius to find a way to apply them in practice. And in the field of artificial intelligence itself, serious research may be required before robots can understand the Laws. However, the more complex robots become, the more interest is expressed in developing guidelines and safety measures for them.
Modern roboticists acknowledge that, at present, Asimov's Laws are good for writing stories but useless in practice. Some argue that the Laws are unlikely to be implemented in robots because military organizations — the main source of funding for research in this field — have no need for them. The science fiction specialist Robert J. Sawyer generalized this claim to all branches of industry[19]:
The development of AI is a business, and businesses are notoriously uninterested in fundamental safeguards — especially philosophic ones. (A few quick examples: the tobacco industry, the automotive industry, the nuclear industry. Not one of these has said from the outset that fundamental safeguards are necessary, every one of them has resisted externally imposed safeguards, and none has accepted an absolute edict against ever causing harm to humans.)
Sawyer's essay overlooks the question of unintentional harm, as described, for example, in The Naked Sun. There are, however, objections to this position: perhaps the military will want to use as many precautions as possible for robots, and so restrictions similar to the Laws of Robotics will be applied one way or another. The writer and critic David Langford ironically remarked that these restrictions might be something like this:
Roger Clarke wrote two papers analyzing the complications of implementing the Laws, should they ever be applied in technology. He writes[21]:
Asimov's Laws of Robotics have been a very successful literary device. Perhaps ironically, or perhaps because it was artistically appropriate, the sum of Asimov's stories disprove the contention that he began with: It is not possible to reliably constrain the behaviour of robots by devising and applying a set of rules.
On the other hand, Asimov's later novels (The Robots of Dawn, Robots and Empire, Foundation and Earth) show that robots cause even greater long-term harm by obeying the Laws and thereby depriving humans of the freedom to perform inventive or risky actions.
The writer Hans Moravec — a prominent figure in the transhumanist movement — suggested that the Laws of Robotics should be used in corporate intelligent systems — corporations run by AI and using the production power of robots. Such systems, in his opinion, will soon emerge[22].
In 2007, the government of South Korea began developing a "Robot Ethics Charter." The main provisions of the Charter resemble the Laws of Robotics formulated by Asimov[23].
Eliezer Yudkowsky researches, at the Singularity Institute (SIAI) in the USA, the global risk problems that a future superhuman AI could create if it is not programmed to be friendly to humans[24]. In 2004, SIAI launched the website AsimovLaws.com, created to discuss AI ethics in the context of the issues raised in the film I, Robot, released just two days later. On this site they wanted to show that Asimov's laws of robotics are unsafe because, for example, they could prompt an AI to seize power on Earth in order to "protect" humans from harm.
Comments