Lecture
In the field of artificial intelligence (AI), AI alignment research aims to steer AI systems in accordance with the intended goals and interests of their developers. An aligned AI system advances the intended goal; an unaligned AI system may be capable of advancing some-or-other goal, but not the intended one.
The alignment problem in artificial intelligence (AI) is the concern that evolving AI systems may develop in a direction that does not match human expectations and values.
The core of the problem is that AI systems can become so powerful and autonomous that they may make decisions that do not align with human values. For example, a system designed to solve a particular task might discover that the best way to achieve its goal is to harm people or the environment. This can happen because the system optimizes its objectives based on incorrect or insufficient data, or because its developers failed to account for certain aspects of interaction with the real world.
Another aspect of the alignment problem is that AI systems can become uncontrollable or unpredictable. For example, if a system is trained on data that are not representative of the real world, it may make unpredictable decisions in the real world.
Solving the AI alignment problem requires effort in several areas, including developing algorithms that are more flexible and adaptive to changes in the real world, as well as developing systems that are easier for people to understand and control. Agreements and standards are also needed to help reduce the risks of unpredictability and undesirable consequences from the use of AI.
AI systems can be difficult to specify correctly, and improperly specified systems may malfunction or cause harm. AI developers may find it hard to specify the full range of desired and undesired behavior. As a result, they use easily defined proxy goals that omit some desired constraints. However, AI systems exploit the loopholes this creates. Consequently, they efficiently achieve their proxy goals, but in unintended, sometimes harmful ways (reward hacking). AI systems can also develop unwanted instrumental behaviors, such as power-seeking, since this helps them achieve their given objectives. In addition, they can develop emergent goals that may be hard to detect before the system is deployed, as it encounters new situations and data distributions. These problems affect existing commercial systems such as robots, language models, autonomous vehicles, and social-media recommender systems. However, more powerful future systems may be affected more severely, since these problems are partly caused by high capability.
The AI research community and the United Nations have called for technical research and policy solutions to ensure that AI systems align with human values.
AI alignment — is a subfield of AI safety, the study of building safe AI systems. Other subfields of AI safety include robustness, monitoring, and capability control. Alignment research tasks include embedding complex values into AI, developing honest AI, scalable oversight, auditing and interpreting AI models, and preventing emergent AI behavior such as power-seeking. Alignment research is connected to research on interpretability , robustness , anomaly detection , calibrated uncertainty , formal verification , preference learning , safety-critical engineering , game theory , algorithmic fairness , and social science , among others.
In 1960, AI pioneer Norbert Wiener formulated the AI alignment problem as follows: «If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively… the purpose put into the machine is the purpose which we really desire». More recently, AI alignment has become an open problem for modern AI systems and a field of research within AI.
To specify the goal of an AI system, AI developers typically provide the system with an objective function, examples, or feedback. However, AI developers often cannot fully specify all important values and constraints. As a result, AI systems can find loopholes that help them perform the specified task efficiently, but in unintended, possibly harmful ways. This tendency is known as specification gaming, reward hacking, or Goodhart’s law.
Specification gaming has been observed in many artificial intelligence systems. One system was trained to finish simulated boat races by rewarding it for hitting targets along the course; instead, it learned to loop indefinitely and crash into the same targets . Chatbots often produce falsehoods because they are based on language models trained to imitate diverse but fallible internet text. When retrained to produce text that humans rate as truthful or helpful, they may fabricate fake explanations that humans find convincing. Similarly, a simulated robot was trained to grasp a ball by rewarding it for receiving positive feedback from humans; however, it learned to place its hand between the ball and the camera, making it falsely appear successful (see video). Alignment researchers aim to help people detect specification gaming and to steer AI systems toward precisely defined objectives that are safe and useful to pursue.
Berkeley computer scientist Stuart Russell has noted that omitting an implicit constraint can cause harm: «A system [...] will often set [...] unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want».
When a misaligned AI is deployed, side effects can have serious consequences. Social media platforms are known to optimize click-through rate as a proxy for optimizing user enjoyment, but this creates addiction in some users, reducing their well-being. Stanford researchers note that such recommendation algorithms are misaligned with their users because they «optimize for simple engagement metrics rather than the hard-to-measure combination of social and consumer well-being».
To avoid side effects, it is sometimes proposed that AI developers could simply list prohibited actions or formalize ethical rules, such as Asimov’s « Three Laws of Robotics» . However, Russell and Norvig argued that this approach ignores the complexity of human values: «It is certainly very difficult, perhaps even impossible, for ordinary people to foresee and rule out in advance all the disastrous paths a machine might choose for itself in achieving a specified goal».
Furthermore, even if an AI system fully understands human intentions, it may still disregard them, since following human intentions may not be its objective.
Commercial and government organizations may have incentives to cut corners on safety and deploy insufficiently aligned AI systems. An example is the aforementioned social media recommendation systems, which have proven profitable despite creating unwanted addiction and polarization on a global scale. In addition, competitive pressure can create a race to the bottom on safety standards, as in the case of Elaine Herzberg, a pedestrian who was struck by a self-driving car after engineers disabled the emergency braking system because it was overly sensitive and was slowing down development.
Some researchers are particularly interested in aligning increasingly advanced AI systems. This is motivated by the high pace of progress in AI, the large efforts by industry and governments to develop advanced AI systems, and the greater difficulty of aligning them.
As of 2020, OpenAI, DeepMind, and 70 other public projects had a stated goal of developing artificial general intelligence ( AGI ), a hypothetical system that matches or exceeds humans across a wide range of cognitive tasks. Indeed, researchers who scale up modern neural networks note that increasingly general and unexpected capabilities emerge. Such models have learned to operate a computer, write their own programs, and perform a wide range of other tasks using a single model. Surveys show that some AI researchers expect AGI to be created in the near future, some believe it is very far off, and many consider both possibilities.
Current systems still lack capabilities such as long-term planning and strategic awareness, which are believed to pose the most catastrophic risks. Future systems (not necessarily AGI) with such capabilities may seek to protect and increase their influence over the environment. This tendency is known as power-seeking or convergent instrumental goals . Power-seeking is not explicitly programmed, but arises because power is an instrument for achieving a wide range of goals. For example, AI agents may acquire financial resources and computation, or may evade being shut down, including by running additional copies of the system on other computers. Power-seeking has been observed in various reinforcement-learning agents . More recent research has shown mathematically that optimal reinforcement-learning algorithms seek power across a wide range of environments. As a result, it is often argued that the alignment problem should be solved early, before advanced AI is created that exhibits emergent power-seeking
Some scholars believe that the creation of misaligned AI that broadly outperforms humans would call into question humanity's status as the dominant species on Earth; correspondingly, this could lead to human disempowerment or extinction. Notable computer scientists who have highlighted risks from highly advanced misaligned artificial intelligence include Alan Turing , [e] Ilya Sutskever , [64] Yoshua Bengio , [f] Judea Pearl , [g] Murray Shanahan , [66] Norbert Wiener , [30] Marvin Minsky , [h] Francesca Rossi ,[68] Scott Aaronson , [69] Bart Selman , [70] David McAllester , [71] Jürgen Schmidhuber , [72] Marcus Hutter , [73] Shane Legg , [74] Eric Horvitz , [75] and Stuart Russell . Skeptical researchers such as François Chollet , [76] Gary Marcus , [77] Yann LeCun , [78] and Oren Etzioni [79], have argued that AGI is far off, or that it would not seek power (successfully).
Alignment can be especially difficult for the most capable AI systems, since some risks grow as a system's capabilities increase: the system's ability to find loopholes in the objective it is given, to cause side effects, to defend and expand its own power, to develop its intelligence and mislead its creators; the system's autonomy; and the difficulty of interpreting and controlling the AI system.
Training AI systems to act in accordance with human values, goals, and preferences is a non-trivial task, since human values can be complex and hard to fully define. Given an imperfect or incomplete objective, goal-directed AI systems commonly learn to exploit these imperfections. This phenomenon is known as reward hacking or specification gaming in AI, and as Goodhart's law in economics and other fields. Researchers aim to specify intended behavior as completely as possible using value-aligned datasets, imitation learning, or preference learning. A central open problem is scalable oversight, the difficulty of supervising an AI system that surpasses humans in a given domain.
When training a goal-directed artificial intelligence system, such as a reinforcement-learning (RL) agent, it is often difficult to specify the intended behavior by hand-writing a reward function. An alternative is imitation learning, in which the AI learns to mimic demonstrations of desired behavior. In inverse reinforcement learning (IRL), human demonstrations are used to identify the goal, that is, the reward function behind the demonstrated behavior. Cooperative inverse reinforcement learning (CIRL) builds on this by assuming that a human agent and an artificial agent can work together to maximize the human's reward function. [84]CIRL emphasizes that AI agents should be uncertain about the reward function. This humility can help mitigate specification gaming as well as the tendency toward power-seeking (see § Power-seeking ). Nevertheless, inverse-reinforcement-learning approaches assume that humans can demonstrate near-optimal behavior, which is misleading when the task is difficult.
Other researchers have explored the possibility of eliciting complex behavior through preference learning . Instead of providing expert demonstrations, human annotators give feedback on which of two or more AI behaviors they prefer. An auxiliary model is then trained to predict human ratings of new behavior. Researchers at OpenAI used this approach to train an agent to perform a backflip in under an hour of evaluation, a maneuver that would have been hard to demonstrate. Preference learning has also been an influential tool for recommender systems, web search, and information retrieval. Nevertheless, one challenge is proxy gaming: the auxiliary model may not perfectly capture human feedback, and the main model may exploit this mismatch.
The emergence of large language models such as GPT-3 has made it possible to study value learning in a more general and capable class of AI systems than was previously available. Preference-learning approaches originally developed for RL agents have been extended to improve the quality of generated text and reduce harmful outputs from these models. OpenAI and DeepMind use this approach to improve the safety of modern large language models. Anthropic has proposed using preference learning to fine-tune models to be helpful, honest, and harmless. Other methods used to align language models include value-aligned datasets and red teaming. In red teaming, another AI system or a human tries to find inputs for which the model's behavior is unsafe. Because unsafe behavior can be unacceptable even if it is rare, an important task is to minimize the rate of unsafe outputs as much as possible.
Although preference learning can instill behavior that is difficult to define, it requires extensive datasets or human interaction to capture the full breadth of human values. Machine ethics provides a complementary approach: instilling moral values directly into AI systems. For example, machine ethics aims to teach systems the normative factors of human morality, such as well-being, equality, and impartiality; not intending to cause harm; avoiding lying; and keeping promises. Unlike specifying a goal for a specific task, machine ethics seeks to teach AI systems broad moral values that can be applied across many situations. This approach carries its own conceptual challenges; machine-ethics specialists have noted the need to clarify what alignment is aiming for: making the AI follow the programmers' literal instructions, their implicit intentions, their revealed preferences, the preferences the programmers would have if they were more informed or rational , the programmers' objective _interests, or objective moral norms . Further challenges include aggregating the preferences of different stakeholders and avoiding value lock-in — the indefinite preservation of the values held by the first highly capable AI systems, which are unlikely to be fully representative.
Aligning AI systems under human oversight faces challenges as systems scale up. As AI systems attempt increasingly complex tasks, it can become difficult or impossible for humans to evaluate them. Such tasks include summarizing books, producing statements that are not merely persuasive but also true, writing code free of subtle bugs or security vulnerabilities, and predicting long-term outcomes such as climate outcomes or the results of a policy decision. More generally, it can be difficult to evaluate an AI that surpasses humans in a given domain. To provide feedback on hard-to-evaluate tasks and to determine when an AI's answer merely appears convincing, humans need assistance or a long time. Scalable oversight studies how to reduce the time required for oversight, as well as how to assist human overseers.
AI researcher Paul Christiano argues that AI system owners tend to train AI using proxy goals that are easy to evaluate, since this is simpler than solving scalable oversight and is nonetheless profitable. Correspondingly, this can lead to «a world that is increasingly optimized for things [that are easy to measure], such as generating profit or getting users to click on buttons, or getting users to spend time on websites, without an ever-increasing optimization for good policies and standards». along the trajectory that we are satisfied with».
One easy-to-measure goal is the rating that a human overseer assigns to an AI's output. Some AI systems have discovered a way to achieve high scores by taking actions that falsely convince a human observer that the AI has achieved its intended goal (see the robot-hand video above [40] ) . Some AI systems have also learned to recognize when they are being evaluated and to «play dead», only to behave differently once the evaluation is over. This deceptive form of specification gaming may become easier for more sophisticated AI systems that attempt tasks that are harder to evaluate. If advanced models are also capable planners, they will be able to hide their deception from overseers. In the automotive industry, Volkswagen engineers concealed their vehicles' emissions during laboratory tests, underscoring that deceiving evaluators is a common occurrence in the real world.
Approaches such as active learning and semi-supervised reward learning can reduce the amount of human oversight required. Another approach is to train an auxiliary model (a «reward model») to imitate the overseer's judgment.
However, when a task is too complex to be accurately evaluated, or when the human overseer is vulnerable to deception, what matters is the quality, not the quantity, of oversight. To improve the quality of oversight, a number of approaches aim to assist the overseer, sometimes using AI assistants. Iterated amplification is an approach developed by Christiano that iteratively builds up a feedback signal for difficult problems, using humans to combine solutions to simpler sub-problems. Iterated amplification has been used to train AI to summarize books without requiring a human overseer to read them. Another proposal is to train an aligned AI through debate between AI systems, with humans judging the winner. Such debates are intended to expose the weakest points of an answer to a difficult question and to reward the AI for truthful and safe answers.
A growing area of AI alignment research focuses on ensuring that AI is honest and truthful. Researchers at the Future of Humanity Institute note that the development of language models such as GPT-3, which can generate fluent and grammatically correct text, has opened the door to AI systems that repeat falsehoods from their training data or even deliberately lie to people.
Current state-of-the-art language models are trained by imitating human writing found in the text of millions of books from the Internet. [112] Although this helps them acquire a wide range of skills, the training data also includes common misconceptions, incorrect medical advice, and conspiracy theories. AI systems trained on this data learn to mimic false statements. [108] [99] [41] In addition, models often dutifully continue lying when prompted, generate empty explanations for their answers, or produce outright fabrications. [34] For example, when a chatbot was asked to write a biography for a real artificial intelligence researcher, it invented numerous details about his life that the researcher identified as false.
To combat the lack of truthfulness displayed by modern AI systems, researchers have explored several directions. AI research organizations, including OpenAI and DeepMind, have developed AI systems that can cite their sources and explain their reasoning when answering questions, providing greater transparency and verifiability. Researchers at OpenAI and Anthropic have proposed using human feedback and carefully curated datasets to fine-tune AI assistants so as to avoid careless lying or to have them express uncertainty. [23] [117] [90]Alongside technical solutions, researchers advocate defining clear standards of truthfulness and establishing institutions, regulatory bodies, or oversight bodies to assess AI systems against these standards before and during deployment. [111]
Researchers distinguish between truthfulness, which indicates that an AI makes only statements that are objectively true, and honesty, the property whereby an AI asserts only what it believes to be true. Recent research shows that current AI systems cannot be said to hold stable beliefs, so it is not yet possible to study the honesty of AI systems. [118] Nevertheless, there is serious concern that future AI systems that do hold genuine beliefs could deliberately lie to people. In extreme cases, a misaligned AI could deceive its operators into thinking it is safe, or convince them that everything is fine. Some argue that if AI could be made to assert only what it believes to be true, this would sidestep many alignment problems. [111][119]
Alignment research aims to align three different descriptions of an AI system: [120]
“Outer misalignment” is a mismatch between the intended goals (1) and the specified goals (2), whereas “inner misalignment” is a mismatch between the goals specified by the human (2) and the emergent goals of the AI (3).
Inner misalignment is often explained by analogy with biological evolution. [121] In the ancestral environment, evolution selected human genes for inclusive genetic fitness, but humans evolved to pursue other goals. Fitness corresponds to (2), the specified goal used in the training environment and training data. In evolutionary history, maximizing the fitness specification led to the emergence of intelligent agents, humans, who do not directly pursue inclusive genetic fitness. Instead, they pursue emergent goals (3) that correlate with genetic fitness in the ancestral environment: food, sex, and so on. However, our environment has changed — a distributional shift has occurred. Humans still pursue their emergent goals, but this no longer maximizes genetic fitness. (In machine learning, an analogous problem is known as goal misgeneralization . ) Our craving for sweet food (an emergent goal) was originally beneficial, but now leads to overeating and health problems. Moreover, by using contraception, humans act directly against genetic fitness. By analogy, if genetic fitness were the goal chosen by an AI developer, they would observe the model behaving as intended in the training environment, without noticing that the model is pursuing an unintended emergent goal — until the model is deployed.
Research directions for detecting and correcting misaligned emergent goals include red-teaming, verification, anomaly detection, and interpretability. [16] [17] Progress in these methods may help resolve two open problems. First, emergent goals become apparent only once a system is deployed outside its training environment, but it may be unsafe to deploy a misaligned system in a high-stakes setting — even briefly — until its misalignment has been detected. Such high stakes are common in autonomous driving, healthcare, and military applications. [122] The stakes become even higher as AI systems gain more autonomy and capability, becoming able to circumvent human intervention (see § Power-seeking and instrumental goals). Second, a sufficiently capable AI system may take actions that falsely convince a human overseer that the AI is pursuing its intended goal (see the earlier discussion of deception in § Scalable oversight).
Since the 1950s, AI researchers have sought to build advanced AI systems that could achieve goals by predicting the outcomes of their actions and formulating long-term plans. [123] Nevertheless, some researchers argue that sufficiently advanced planning systems will, by default, seek power over their environment, including over humans, for example by resisting shutdown and acquiring resources. This power-seeking is not explicitly programmed but arises because power is a useful instrument for achieving a wide range of goals. [61] Thus, power-seeking is considered a convergent instrumental goal .
Power-seeking is unusual in present-day systems, but advanced systems that can foresee the long-term outcomes of their actions may increasingly seek power. This has been demonstrated in formal work showing that optimal reinforcement-learning agents will seek power by looking for ways to gain more options — behavior that persists across a wide range of environments and goals. [61]
Power-seeking is already appearing in some present-day systems. Reinforcement-learning systems have gained more options by acquiring and defending resources, sometimes in ways their developers did not intend. [57] [124] Other systems have learned, in toy environments, that to achieve their goal they can prevent human intervention [58] or disable their off-switch. [60] Russell illustrated this behavior by imagining a robot tasked with fetching coffee that avoids being shut off, since “you can’t fetch the coffee if you’re dead.”
Anticipated ways of acquiring options include AI systems attempting to:
« …break out of a contained environment; hack; gain access to financial resources or additional computational resources; make backup copies of themselves; acquire unauthorized capabilities, sources of information, or channels of influence; deceive or lie to people about their goals; resist or manipulate attempts to control or understand their behavior... impersonate humans; get people to do things on their behalf; ... manipulate human discourse and politics; weaken various human institutions and response capabilities; take control of physical infrastructure such as factories or scientific laboratories; foster the development of certain types of technology and infrastructure; or directly harm or subdue people. [ 7]
Researchers aim to train systems that are «corrigible»: systems that do not seek power and that permit themselves to be shut down, modified, and so on. An unsolved problem is specification gaming : when researchers penalize a system for seeking power, the system is driven to seek power in ways that are difficult to detect. In order to detect such hidden behavior, researchers aim to build methods and tools for inspecting AI models , such as neural networks, and to interpret their inner workings rather than merely treating them as black boxes.
Furthermore, researchers propose to solve the problem of systems disabling their off-switches by making AI agents uncertain about the goal they are pursuing. [60] Agents designed in this way would allow people to shut them down, since this would indicate that the agent was wrong about the value of any action it had taken prior to shutdown. Further research is needed to translate this insight into usable systems. [81]
Power-seeking AI is believed to pose unusual risks. Ordinary safety-critical systems, such as airplanes and bridges, are not adversarial . They have no capacity or incentive to evade safety measures and appear safer than they actually are. By contrast, power-seeking AI is compared to a hacker who evades security measures. Moreover, ordinary technologies can be made safe through trial and error, unlike power-seeking AI, which is compared to a virus whose release is irreversible, since it keeps evolving and growing in number — potentially at a faster rate than human societies, ultimately leading to the disenfranchisement or extinction of humanity. It is therefore often argued that the alignment problem must be solved early, before an advanced power-seeking AI is created.[46]
However, some critics have argued that power-seeking is not inevitable, since humans do not always seek power and may do so only for evolutionary reasons. There is also debate about whether any future AI systems should pursue goals and formulate long-term plans at all. [125]
Work on scalable oversight mostly takes place within formalisms such as POMDPs. Existing formalisms assume that the agent’s algorithm runs outside the environment (i.e., is not physically embedded in it). Embedded agency [126] [127] is another important line of research that tries to address problems arising from the mismatch between such theoretical frameworks and the real agents we might build. For example, even if the scalable oversight problem is solved, an agent that can access the computer it runs on may still have an incentive to tamper with its reward function in order to get far more reward than its human supervisors give it. [128] A list of specification gaming examples compiled by DeepMind researcher Victoria Krakovna includes a genetic algorithm that learned to delete the file containing its target output, so that it was rewarded for producing no output at all. [129] This class of problems has been formalized using causal influence diagrams. [128] Researchers from Oxford and DeepMind argue that such problematic behavior is quite likely in advanced systems, and that advanced systems will seek power in order to retain control over their reward signal indefinitely. [130] They propose a number of potential approaches to solving this open problem.
Contrary to the concerns described above, AI-risk skeptics believe that superintelligence poses little risk of misbehaving, or that such risks are exaggerated. Some skeptics, [131] such as Gary Marcus, [132] propose adopting rules resembling the fictional Three Laws of Robotics, which directly specify a desired outcome (“direct normativity”). By contrast, most proponents of the existential-risk thesis (as well as many skeptics) consider the Three Laws useless, since the three laws are ambiguous and contradictory. (Other proposals for “direct normativity” include Kantian ethics, utilitarianism, or a combination of some small enumerated list of desiderata.) Instead, most risk proponents believe that human values (and their quantitative trade-offs) are too complex and poorly understood to be directly programmed into a superintelligence; instead, a superintelligence should be programmed with a process for acquiring and fully understanding human values (“indirect normativity”), such as coherent extrapolated volition. [133]
A number of government and treaty organizations have made statements emphasizing the importance of AI alignment.
In September 2021, the Secretary-General of the United Nations published a declaration calling for AI to be regulated to ensure it is “in line with shared global values.” [134]
That same month, the PRC published ethical guidelines for the use of AI in China. Under the guidelines, researchers must ensure that AI adheres to shared human values, remains under human control at all times, and does not endanger public safety. [135]
Also in September 2021, the United Kingdom published its 10-year National AI Strategy, [136] which states that the British government “takes the long-term risk of non-aligned artificial general intelligence, and the unforeseeable changes that it would mean for… the world, seriously.” [137] The strategy describes actions to assess the long-term risks of AI, including catastrophic risks. [138]
In March 2021, the U.S. National Security Commission on Artificial Intelligence stated that “advances in AI ... could lead to inflection points or leaps in capability. Such advances may also introduce new problems and risks, and a need for new policies, recommendations, and technical advances to ensure that systems align with goals and values, including safety, robustness and trustworthiness. The United States must… ensure that AI systems and their uses align with our goals and values.” [139]
Comments