Towards Scalable Automated Alignment of LLMs: A Survey

Abstract

Alignment is the most critical step in building large language models (LLMs) that meet human needs. With the rapid development of LLMs gradually surpassing human capabilities, traditional alignment methods based on human-annotation are increasingly unable to meet the scalability demands. Therefore, there is an urgent need to explore new sources of automated alignment signals and technical approaches. In this paper, we systematically review the recently emerging methods of automated alignment, attempting to explore how to achieve effective, scalable, automated alignment once the capabilities of LLMs exceed those of humans. Specifically, we categorize existing automated alignment methods into 4 major categories based on the sources of alignment signals and discuss the current status and potential development of each category. Additionally, we explore the underlying mechanisms that enable automated alignment and discuss the essential factors that make automated alignment technologies feasible and effective from the fundamental role of alignment.

Figure 1: Illustrations of four representative paradigms for automated alignment. Figures are generated by DALL·E (Ramesh et al., 2021).

1 Introduction

Recent years have witnessed the rapid advancements of large language models (LLMs), which have dramatically reshaped the landscape of artificial intelligence (Ouyang et al., 2022; Touvron et al., 2023; OpenAI, 2023b). Alignment is at the core of shaping behaviors of LLMs corresponding to human intentions and values (Yao et al., 2023a; Shen et al., 2023b), e.g., teaching LLMs to follow “helpful, harmless and honest (HHH)” principles during responding (Askell et al., 2021). As a result, increasing efforts have been made for aligning LLMs to meet the human requirements, which makes it a hotspot research direction in LLM era (Wang et al., 2023g, 2024g; Ji et al., 2023).

Previous studies of alignment have primarily relied on manually annotated alignment data, which includes human preference information, to perform post-training on pre-trained models to achieve alignment (Stiennon et al., 2020). Specifically, there are two primary forms of alignment data: 1) instruction-response pairs, which typically consist of a query and a human-written golden reference. This form of data is often used for supervised fine-tuning of LLMs to inject human preference information into the model (Taori et al., 2023; Peng et al., 2023; Ding et al., 2023); 2) preference data, which usually includes a query, several potential responses, and human preferences regarding these responses (Cui et al., 2024). Preference data can be applied for direct preference optimization via algorithms such as DPO (Rafailov et al., 2023), IPO (Azar et al., 2024), and PRO (Song et al., 2023). Besides, it can also be used to train a reward model, which aligns the target policy LLM to the preference information in the data by providing feedback on the model’s responses (Stiennon et al., 2020; Bai et al., 2022a; Ouyang et al., 2022). However, the construction process for both instruction-response pairs and preference data requires very expensive, meticulous human annotation with high quality standards, making each step of scaling these methods very costly (Ouyang et al., 2022; Touvron et al., 2023; Zhou et al., 2023a).

Even with such high costs, the scalability of these human annotation-dependent alignment methods is still unsustainable. First, with the rapid development of LLMs, the capabilities of LLMs have gradually approached or even surpassed human in many aspects, making it increasingly challenging for humans to produce alignment data that is meaningful for LLMs (Bowman et al., 2022; Burns et al., 2023). In fact, many studies have found that the quality of data generated by LLMs has already exceeded the quality of data annotated by general human annotators in many perspectives (Zheng et al., 2024b; Chen et al., 2024d; Wei et al., 2024). This phenomenon not only significantly raises the cost of obtaining single meaningful human-annotated data (due to the need for increasingly expensive high-quality annotators), but also substantially reduces the potential benefits of human-annotated data for LLMs. Second, as the capabilities of LLMs gradually surpass human capability boundaries, it becomes increasingly difficult for humans to effectively judge the quality of the responses generated by LLMs. This leads to a significant decline in the quality of the preference signals generated by humans, which can no longer accurately reflect human needs, thereby making it challenging to provide effective guidance for LLMs. Therefore, alignment methods based on human annotation are increasingly unable to cope with the rapid improvement in the capabilities of LLMs, making it difficult to achieve scalable oversight for LLMs.

To address these challenges, automated alignment has drawn great attention very recently (Yuan et al., 2024; Chen et al., 2024g). Unlike previous methods that relied on human annotation to obtain alignment signals, the goal of automated alignment is constructing scalable and high-quality alignment systems with minimal human intervention. Therefore, automated alignment has the potential to address the core challenges posed by the rapid development of LLMs, where human annotation is either infeasible or extremely expensive. For automated alignment, the most crucial part is to find a scalable alignment signal that can replace human manually-created preference signals and remain effective amid the rapid development of LLMs.

To this end, this survey categorizes the rapidly developing automated alignment methods according to the mechanisms used to construct different alignment signals, summarizes the current developments in each direction, and discusses the developmental trajectory and potential future directions. Specifically, this survey explores the following representative directions for constructing alignment signals to achieve automated alignment, including:

• Aligning through inductive bias (§3), which automatically steers the model towards desired behaviors by introducing suitable assumptions and constraints, without the use of additional training signals beyond the model itself.

• Aligning through behavior imitation (§4), which achieves automated alignment by mimicking the behavior of another aligned model. For instance, using a well-aligned model to generate instruction-response pairs, and then train target model with imitation learning.

• Aligning through model feedback (§5), which involves guiding the alignment optimization of the target model by obtaining feedback from other models.

• Aligning through environment feedback (§6), which involves automatically obtaining alignment signals or feedback through interaction with environment to achieve automated alignment of the target model.

Furthermore, this survey also explores the underlying mechanisms (§7) that enable automated alignment and, from the fundamental role of alignment, discuss the essential factors that make automated alignment technologies feasible and effective.

The rest of this survey is organized as follows: Section 2 describes the scope of automated alignment covered in this survey, as well as the our taxonomy. Section 3-6 provide a detailed introduction to the progress and limitations of the four aforementioned representative directions in automated alignment. Section 7 explores the underlying mechanisms of automated alignment. And we include a overall conclusion of this survey in Section 8.

for tree= forked edges, grow’=0, draw, rounded corners, node options=align=center,, text width=2.7cm, s sep=6pt, calign=edge midpoint, , [Scalable
Automated Alignment, fill=gray!45, parent [Inductive
Bias §3, for tree=acquisition [From features of LLMs, [Uncertainty Filtering, acquisition [Self-Consistency (Wang et al., 2023e); Self-Improve (Huang et al., 2023a); West-of-N (Pace et al., 2024); etc., acquisition_work] ] [Self-Judge/Critic/Refine, acquisition [ Constitutional AI (Bai et al., 2022b); Tree-of-Thought (Yao et al., 2023b); Self-Rewarding (Yuan et al., 2024); etc.,acquisition_work ] ] [Context Distillation, acquisition [Lab for Alignment (Askell et al., 2021); Dromedary (Sun et al., 2023d); Llama-2-Chat (Touvron et al., 2023); RLCD (Yang et al., 2024b); etc., acquisition_work] ] ] [From organization of LLMs, [Task Decomposition, acquisition [Least-to-Most (Zhou et al., 2023b); IDA (Christiano et al., 2018); etc., acquisition_work] ] [Self-play, acquisition [SPIN (Chen et al., 2024g); Consensus Game (Jacob et al., 2024); etc., acquisition_work] [Debate (Irving et al., 2018); SPAG (Cheng et al., 2024); etc., acquisition_work] ] ] ] [Behavior
Imitation §4, for tree=representation [ Instruction Construction, [ Unnatural Instructions (Honovich et al., 2023); Self-Instruct (Wang et al., 2023f); Evol-Instruct (Xu et al., 2024a); Humpback (Li et al., 2024e); etc., representation_work_2 ] ] [ Strong-to-Weak Distillation, [Response-guided, representation [ LLaMA-GPT4 (Peng et al., 2023); Stanford Alpaca (Taori et al., 2023); Ultrachat (Ding et al., 2023); etc., representation_work ] ] [Preference-guided, representation [ Zephyr (Tunstall et al., 2023); IterAlign (Chen et al., 2024e); Openchat (Wang et al., 2024b); etc., representation_work ] ] ] [ Weak-to-Strong Alignment, [ Weak2Strong (Burns et al., 2023); IaR (Somerstep et al., 2024); Liu and Alahi (2024); Hase et al. (2024); etc., representation_work_2 ] ] ] [Model
Feedback §5, for tree=probing [Scalar Reward, [RLHF , probing [ InstructGPT (Ouyang et al., 2022); DPRM (Li et al., 2024a); etc., probing_work] ] [RLAIF, probing [RLAIF (Lee et al., 2023); RLCD (Yang et al., 2024b); etc., probing_work] ] [Feedback-guided Decoding, probing [Critic-driven Decoding (Lango and Dusek, 2023); RAD (Deng and Raffel, 2023); etc., probing_work] ] [Filtering SFT Data, probing [Quark (Lu et al., 2022); RRHF (Yuan et al., 2023a); RAFT (Dong et al., 2023); etc., probing_work] ] ] [Binary Verifier, [Outcome Verifier , probing [V-STaR (Hosseini et al., 2024); SORMs (Havrilla et al., 2024); etc., probing_work] ] [Process Verifier, probing [MATH-SHEPHERD (Wang et al., 2024e); MiPS (Wang et al., 2024h) etc., probing_work] ] ] [Text Critic, probing [ILF (Scheurer et al., 2022); LEMA (An et al., 2024); etc., cus_probing_work] ] ] [Environment
Feedback §6, for tree=editing [ Social Interactions, editing [StableAlignment (Liu et al., 2023a); MoralDial (Sun et al., 2023a); SOTOPIA-$\pi$ (Wang et al., 2024f); etc., editing_work] ] [Human
Collective Reason, editing [Constitutional AI (Bai et al., 2022b); Collective Constitutional AI (Anthropic, 2023) etc., editing_work] ] [Tool Execution, editing [Self-Debugging (Qiao et al., 2024); CodeRL (Qiao et al., 2024); SelfEvlove (Jiang et al., 2023); CRITIC (Gou et al., 2024); etc., editing_work ] ] [Embodied Environment, editing [GLAM (Carta et al., 2023); E2WM (Xiang et al., 2023); TWOSOME (Tan et al., 2024); etc., editing_work ] ] ] [Mechanism §7, for tree=application [Alignment
Mechanism, application [LIMA (Zhou et al., 2023a); Rethinking (Ren et al., 2024); URIAL (Lin et al., 2024a); ICL&IT (Duan et al., 2023); Behavior shift (Wu et al., 2023a) etc., application_work] ] [Inner Workings of Self-feedback,application [GV-consistency (Li et al., 2024f); CriticBench (Lin et al., 2024b); Self-Rewarding (Yuan et al., 2024); Humback (Li et al., 2024e); LLM-as-a-Judge (Zheng et al., 2024b) etc., application_work] ] [Feasibility of Weak-to-strong, application [Easy2Hard (Sun et al., 2024b; Hase et al., 2024); Weak2Strong (Burns et al., 2023); Principle2Behavior (Bai et al., 2022b; Sun et al., 2023d) etc., application_work] ] ] ]

Figure 2: This paper reviews the work on scalable automated alignment through the lens of the source of aligment signals.

2 Overview

In this section, we will discuss the scope of automated alignment covered in this survey, followed by a description of our taxonomy.

2.1 Scope of Automated Alignment

In the rapidly evolving field of artificial intelligence, studies of alignment play a critical role in ensuring that machine behaviors align with human values and expectations. As AI systems, particularly LLMs, become more complex and capable, aligning these models with nuanced human standards becomes increasingly challenging and resource-intensive. This necessity has spurred the development of methodologies known as “automated alignment”.

Automated alignment does not imply the complete absence of human involvement. Instead, it aims to minimize human intervention while building scalable, high-quality systems that adhere strictly to desired alignment outcomes. The essence of automated alignment lies in its ability to dynamically adjust and respond to alignment criteria through automated processes, thereby reducing dependence on continuous human oversight. Based on the source of alignment signals, current studies of automated alignment can be categorized into four main categories. First, inductive bias involves enhancing models with assumed generalizations or rules, enabling them to produce better-aligned responses without explicit external guidance. Second, behavior imitation techniques involve training AI systems by mimicking the outputs of already aligned models, leveraging imitation learning to propagate desired behaviors. Third, automated alignment is supported by integrating feedback mechanisms. Model feedback aligns a target model by incorporating insights from other models’ feedback. Fourth, environment feedback automates the acquisition of alignment targets from the operational context itself, enabling the model to adapt based on real-time data and interactions.

The evolution towards automated alignment suggests a paradigm where AI systems can not only self-regulate based on pre-defined alignment protocols but also evolve these protocols autonomously through continuous learning and adaptation. This shift promises significant advancements in AI governance, making it possible to deploy AI solutions that are both effective and trustworthy on a larger scale. However, despite these advancements, the necessity for human oversight remains crucial to ensure that AI systems do not diverge from ethical boundaries or societal norms even as they gain autonomy. Such a blend of automation in alignment with strategic human oversight encapsulates the current trajectory and complexities involved in the field of AI alignment.

2.2 Taxonomy

In this section, we will provide a detailed description of our taxonomy as illustrated in Figure 2.

Aligning through inductive bias (§3)

discusses enhancing the model by introducing additional assumptions, enabling it to leverage self-generated signals for further improvement. Currently, there are two types of inductive bias (Mitchell, 1980) that facilitate the self-improvement of large language models. The first type includes inductive biases derived from the inherent features of LLMs. For instance, Wei et al. (2022); Kojima et al. (2022); Wang et al. (2023e); Wang and Zhou (2024) focus on eliciting better outcomes from LLMs by utilizing patterns within the model’s output probabilities. Additionally, Bai et al. (2022b); Yao et al. (2023b); Saunders et al. (2022); Shinn et al. (2023) exploit the models’ capabilities to critique, judge, and refine their responses, thereby enhancing safety and quality. Another line of works (Ganguli et al., 2022; Lin et al., 2024a) finds that simply providing aligned target signals within the context allows LLMs to use their robust in-context learning abilities for automated alignment. The second type involves inductive biases that arise from the organizational structure of LLMs. For example, based on the assumption of factored cognition, Khot et al. (2023); Zhou et al. (2023b); Wang et al. (2023b) use task decomposition to enable LLMs to solve complex tasks. Furthermore, inspired by the success of AlphaGo Zero (Silver et al., 2018), several studies propose enhancing LLMs by having them play iterative games against themselves (Fu et al., 2023a; Chen et al., 2024g).

Aligning through behavior imitation (§4)

aims to align the behaviors of a target model with those of a teacher model through imitation. Based on the characteristics of the teacher and target models, research on alignment via behavior imitation can be categorized into two main paradigms: strong-to-weak distillation and weak-to-strong alignment. Specifically, strong-to-weak distillation involves using a well-aligned and powerful LLM to generate training data, and then aligning the target model’s behaviors with the responses (Taori et al., 2023; Peng et al., 2023; Xu et al., 2024a) or preferences (Tunstall et al., 2023; Cui et al., 2024) of the teacher model. In contrast, weak-to-strong alignment uses a weaker model as a supervisor to guide the stronger target model towards further alignment (Burns et al., 2023; Zheng et al., 2024a; Hase et al., 2024).

Align through model feedback (§5)

aims to guide the alignment optimization of the target model by introducing feedback from additional models. This feedback generally falls into three categories: 1) scalar signals (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022). These are typically provided by a reward model trained on pairs of preference data. The reward model is expected to learn the alignment signal from preference data and generalize to unseen samples obtained during the reinforcement learning process. Additionally, feedback from the reward model can guide the selection of instruction tuning data (Zhou et al., 2023a; Touvron et al., 2023; Yuan et al., 2023b) and model decoding (Lango and Dusek, 2023; Deng and Raffel, 2023). 2) binary signals. These are widely used in mathematical reasoning tasks to provide binary feedback on the correctness of results. Given that most mathematical tasks require multiple reasoning steps for solving, binary verifiers can be categorized into outcome verifiers, which estimate the correctness of final results (Zelikman et al., 2022; Singh et al., 2024; Havrilla et al., 2024), and process verifiers, which can further provide feedback on intermediate steps (Lightman et al., 2023; Uesato et al., 2022; Ying et al., 2024; Shao et al., 2024). 3) text signals. These are typically generated by LLMs to provide more intuitive feedback for humans (Scheurer et al., 2022; Chen et al., 2024a).

Align through environment feedback (§6)

aims to automatically obtaining alignment signal or feedback from existing environment instead of a trained model, such as social interactions (Liu et al., 2023a; Sun et al., 2023a), public opinion (Anthropic, 2023), external tools (Qiao et al., 2024; Jiang et al., 2023) and embodied environment (Bousmalis et al., 2023; Xu et al., 2024b). Environment feedback serves as an essential supplement for previous origins of alignment signal, which enable the AI system better adapt to real-world application scenarios. However, how to effectively utilize environment feedback is still a research direction that urgently needs further exploration.

Underlying mechanisms for automated alignment (§7)

Apart from reviewing the above representative technique for achieving automated alignment, we also provide an in-depth discussion about the underlying mechanisms for automated alignment. Specifically, we devote to investigate the following three critical questions about automated alignment:

• What is the underlying mechanism of current alignment?

• Why does self-feedback work?

• Why is weak-to-strong feasible?

The explorations of these questions are crucial for achieving scalable automated alignment. For each question, we summarize existing research and perspectives, raise open questions, and discuss their limitations and future directions.

3 Aligning Through Inductive Bias

Self-education is, I firmly believe, the only kind of education there is.

Isaac Asimov

Figure 3: Illustrations of aligning through 3 kinds of representative inductive bias stemming from inherent features of LLMs.

Currently, aligning through inductive bias is one of the most promising directions for achieving automated alignment. Inductive bias (Mitchell, 1980), are essentially assumptions or constraints that guide model learning and decision-making processes. By carefully selecting and implementing suitable inductive biases, we can steer models towards behaviors and decisions that are more likely to meet human standards and expectations, which can then generalize to unseen data distributions.

Compared to other methods of achieving automated alignment, aligning through inductive bias offers two primary advantages:

• It does not require additional supervisory signals beyond the model itself, thus avoiding the high cost of obtaining additional annotated data. This is particularly relevant given the current scenario where training data is becoming scarce or has already been exhausted (Xue et al., 2023).

• It has the potential to address the scalable oversight problem (Bowman et al., 2022). As the potential of LLMs continues to scale, it becomes challenging for humans to provide supervisory signals that surpass their own level of knowledge. However, through inductive bias, models can continuously self-improve, transcending the limitations of human knowledge.

After conducting a thorough review of the relevant literature, we find that current efforts towards self-improvement solely through language model itself can be decomposed into a set $\mathscr{H}$ of five inductive biases. These inductive biases fall into two broad categories: 1) Those stemming from inherent features of LLMs (§3.1), and 2) Those arising from the self-play dynamics of LLMs (§3.2).

For each type of inductive bias, we will begin by introducing its origin. Following this, we will enumerate the works that employ this inductive bias as a single-step policy improvement strategy. Next, we will discuss the works that iteratively train on these biases, aiming for continuous improvement. Finally, we will address the open research problems associated with the given inductive bias.

3.1 Inductive Bias from Features of LLMs

LLMs have intrinsic features that can function as inductive biases. These features primarily originate from the pre-training of deep Transformer networks on large datasets (H1, H3). Additionally, some biases result from initial alignment processes designed to improve the models’ helpfulness (H2). In this section, we summarize three key inductive biases, as depicted in Figure 3. It is important to note that these inductive biases are not entirely independent; instead, they represent three different perspectives on automated alignment.

LLMs possess intrinsic features that can act as inductive biases. These features largely arise from the pre-training of deep Transformer networks on massive datasets (H1, H3), while some also stem from preliminary alignment procedures aimed at enhancing the models’ helpfulness (H2). In this section, we will summarize three key inductive biases, as depicted in Figure 3. It is important to note that these inductive biases are not entirely independent; instead, they represent three different perspectives on automated alignment.

3.1.1 H1: Uncertainty as a indicator of helpfulness

Probability distributions from models can represent uncertainty. As Kadavath et al. (2022) discovered, when prompts are suitably designed, the responses obtained from pre-trained LLMs can be well-calibrated, and the degree of calibration can scale with the number of parameters and the number of exemplars. In other words, the higher the probabilities assigned by an LLM to a given answer, the more likely that answer is to be correct. This hypothesis has also been validated by Wang et al. (2021) and He et al. (2023). Similarly, Manakul et al. (2023) found a correlation between the probabilities output by aligned models and factuality.

In the machine learning literature, early applications of this inductive bias were evident in a series of works using self-training (Scudder, 1965) for semi-supervised learning (Nigam and Ghani, 2000; Amini and Gallinari, 2002). The basic paradigm of these works involves using learners trained on supervised data to continue learning from confidently classified unlabeled data, thereby enhancing supervised learning performance with unlabeled data. This approach has been applied in classification tasks with methods such as Pseudo-labeling (Lee et al., 2013; Ferreira et al., 2023) and Entropy Minimization (Grandvalet and Bengio, 2004). He et al. (2020) extended this approach to sequence generation NLP tasks, highlighting that biased sampling and noise perturbation are key factors for the success of self-training in these tasks. Pace et al. (2024) extended the paradigm of self-training to the alignment problem, improving the robustness of reward models by allowing them to iteratively learn from the highest-scoring and lowest-scoring answers in the candidate pool generated for a query.

The frequency of specific answers also reflects uncertainty. Therefore, synthesizing candidate answers from multiple samplings can lead to better performance than relying on a single sample. This approach is particularly effective when LLMs are used in tasks requiring deliberate thinking (e.g., solving math problems), as a single Chain-of-Thought (CoT) (Wei et al., 2022) reasoning path can sometimes fall into local optima and generate plausible but unfaithful answers. Self-Consistency (Wang et al., 2023e), by aggregating multiple reasoning paths through a weighted sum, can alleviate this problem by marginalizing the model’s likelihood to the reasoning path. Interestingly, it was also found that an unweighted sum (i.e., majority vote) can achieve comparable performance, which is attributed to the “similarly likely” probabilities over all reasoning paths. Wang and Zhou (2024) further discover that the presence or absence of a CoT reasoning path correlates with the probability of the final answer when inference is conducted without any prompting techniques.

To perpetuate such enhancement, Self-Improve (Huang et al., 2023a) views CoT with Self-Consistency as a policy improvement operator, substantially improving the reasoning capabilities and potential of LLMs through iterative learning of the reasoning paths obtained from Self-Consistency. Zhang and Parkes (2023) demonstrate that LMs could self-improve on large number addition problems by curriculum “distilling” the CoT answers to the direct answers without explicit reasoning. Quiet-STAR (Zelikman et al., 2024) considers the advantage of the influence of rollout rationale on the probabilities of subsequent tokens as a feedback signal, encouraging the model to generate a more helpful implicit thought process using reinforcement learning techniques.

H1: Discussion

Figure 4: Calibration plot observed by Kadavath et al. (2022); He et al. (2023); OpenAI (2023b); Zhang et al. (2024). The x-axis represents the probability associated with the model’s output, while the y-axis indicates the probability that the answer is correct. In the low probability region, the gray area may reflect the “don’t know” responses substituting some low-confidence answers. In the high probability region, the gray area signifies overconfidence. The comparison between pre-trained LLM and aligned LLM demonstrates that alignment for helpfulness can result in miscalibration, which is harmful to iterative self-improvement.

For aligned models, maintaining calibration and uncertainty remains crucial, as miscalibration can undermine the potential for iterative self-improvement. Numerous studies (Kadavath et al., 2022; He et al., 2023; OpenAI, 2023b; Zhang et al., 2024) have noted that the alignment process can impair the calibration of LLMs (as shown in Figure 4). This observation is reasonable for several reasons: 1) The current superficial alignment process aims to steer the model away from generating harmful or incorrect answers. This involves replacing the probability of incorrect answers with the probability of rejection responses to some extent, creating a gray area in the low probability region. 2) During the alignment process, the model also learns the response format. The increase in confidence regarding the response format can somewhat affect the confidence in the answers themselves (He et al., 2023). Moreover, the model relearns the correct answers, which can lead to overconfidence (represented by the gray area in the high probability region). The extremization of the probability distribution might be more pronounced in self-alignment, given that it is an iterative process involving self-sampling and training. This implies that all tokens are already in a very high probability distribution, making them more likely to be sampled as responses.

When the model becomes overconfident, it leads to a decrease in the diversity and exploratory nature of the model’s generated outputs. To mitigate this issue, one promising approach is to use inference-time interventions (e.g., high temperature (Kadavath et al., 2022), fidelity (Zhang et al., 2024)) to decrease the expected calibration error. Another potential solution is to filter the pseudo-labeled samples to avoid harmful repeated training, which requires understanding when unlabeled samples will be effective (Grandvalet and Bengio, 2004).

3.1.2 H2: LLMs can judge, critic, refine and more

Pre-trained LLMs often struggle to directly respond to instructions. However, the widespread adoption of imitation learning (Chiang et al., 2023) and feedback learning (Bai et al., 2022a) has significantly enhanced the zero-shot helpfulness of LLMs. Leveraging the reasoning abilities elicited and enhanced by these general helpfulness improvements, a series of works have emerged, utilizing the model’s capabilities to enhance response quality and safety through judging, critiquing, refining, and more.

Judge refers to determining the quality of model responses. These judgment standards are typically incorporated into instructions as principles or guidelines, enabling regulators to oversee LLM behavior in a more scalable manner (Bai et al., 2022b; Yuan et al., 2024) compared to heavy reliance on human annotators for feedback (Bai et al., 2022a). This approach allows for timely regulation, aiding in the flexible and controlled alignment process of language models, which can help prevent issues like reward hacking during iterative training (Sun et al., 2023c) and facilitate on-policy Reinforcement Learning Training (Guo et al., 2024b).

Self-judging can manifest in two primary forms: 1) Differentiating the relative quality of two responses (AI Feedback (Bai et al., 2022b)), resulting in evaluation outcomes represented as a partial order. For instance, Tan et al. (2023) employed prompts to compare which answer better adheres to the HHH principles. They then distilled the chosen option back into the model to further enhance its judging capability. Bai et al. (2022b) prompted the model to select superior responses based on sampled principles and subsequently employed a preference process to achieve a Pareto improvement of the model. 2) Providing an absolute score for a response (LLM-as-a-judge (Zheng et al., 2024b)), with the evaluation results in scalar form. Yao et al. (2023b), Besta et al. (2024) and Xie et al. (2023) introduced real-time evaluation modules for thought states during the reasoning process. These modules serve as a prior during the search process, assisting the model in exploring the action space for questions requiring deliberate thinking. Similarly, RAIN (Li et al., 2024g) utilizes a binary scoring prompt for self-evaluating whether the generation is likely to be harmful, thereby enhancing response safety combined with an inference-time tree search. Yuan et al. (2024) employed a five-point judge prompt to score the model’s instruction-response outputs, then converted the scores into a partial order for iterative training using DPO.

Recalling H1, it becomes evident that H1 serves as a foundation for H2, given that the accuracy of the judge is directly tied to the calibration of the LLM. Consequently, H2 will only be effective if H1 is valid (Bai et al., 2022b).

Critique refers to generating modification suggestions. By leveraging LLM itself for critique, the suggestions can address errors and deficiencies, such as mistakes in summarization (Saunders et al., 2022) and machine translation (Fernandes et al., 2023), math reasoning (Lin et al., 2024b), and decision-making and programming tasks (Saunders et al., 2022). The suggestions can also pertain to abstract value criteria, such as the HHH related principles (Chen et al., 2024e; Bai et al., 2022b).

Refine refers to the ability that LLMs can improve the given text. Most of the work on self-refinement is based on natural language reasons provided by the critic module to modify the response (Bai et al., 2022b; Tan et al., 2023; Madaan et al., 2023). Some research also demonstrates the possibility of making modifications directly based on scalar rewards (Shinn et al., 2023). The less informative critic can be more challenging for LLMs since they must complete more information by themselves through reasoning. Another line of work uses LLMs to refine the prompts themselves (Fernando et al., 2023; Yang et al., 2023a).

Other: A helpful LLM can serve in various capacities to assist in the alignment process, effectively replacing human guidance. For example, it can vote on the quality of intermediate states of thought (Yao et al., 2023b), verify outcomes based on predicting conditions in questions (Weng et al., 2023), and automatically generate, filter (Yue et al., 2024b), and evolve instructions (Wang et al., 2023f; Li et al., 2024e; Xu et al., 2023a), among other tasks.

Regarding the persistence process, the improvements derived from these methods can be further distilled into the model through SFT (i.e., rejection sampling), DPO, RM-RL, and other techniques. Additionally, the judge / critique - refine process can be conducted iteratively.

H2: Discussion

As the abilities of judging, critiquing, and refining are increasingly incorporated into model feedback and learning processes, there is a need for systematic evaluation studies of these capabilities. In this context, several research directions are worth pursuing:

• Benchmarking the performance of existing models on these atomic abilities, as exemplified by works such as Sun et al. (2024a) and Lin et al. (2024b).

• Conducting causal studies on the formation process of models’ judging, critiquing, and refining abilities, and investigating what forms of pre-training and fine-tuning data can influence these abilities.

• Evaluating the effects of distribution shift on these abilities. Do models still possess reliable evaluation and improvement capabilities if they have not been trained on the corresponding instruction and response pairs? This is particularly pertinent to the scalable oversight problem, which assumes the absence of direct supervision for instructions.

• Gathering empirical evidence to demonstrate that self critique-judge-refine abilities can enhance model performance in fair and reasonable experimental settings. Some works point out that the improvement may come from the use of stronger models (Sharma et al., 2024) and golden labels (Huang et al., 2023a).

3.1.3 H3: LLMs can effectively learn in-context

Figure 5: Illustrations of aligning through three representative inductive biases which stem from the organization of LLMs.

In-context learning (ICL) refers to the ability of LLMs to initialize a task-specific model with exemplars or experiences during inference (Brown et al., 2020). Given that certain studies (Dai et al., 2023; von Oswald et al., 2023) suggest parallels between ICL and parameter gradient descent, it is plausible to regard it as a versatile and effective “learning” method.

From the perspective of automated alignment, ICL offers an efficient means to cold-start from a pre-trained LLM. With the assistance of ICL, just a few conversational samples in-context can yield a somewhat aligned model (Ganguli et al., 2022; Sun et al., 2023d; Lin et al., 2024a). Similarly, by prepending a few annotated exemplars in-context, ICL can also elicit judge and critic abilities of pre-trained LLM to some extent, or enhance the performance of original zero-shot capabilities (Bai et al., 2022b). Additionally, ICL presents a potential method for adaptive alignment (Xu et al., 2023c) to different social norms and regulations.

However, prepending few-shot exemplars in the context above can make inference inefficient (Gim et al., 2023) and interfered with the unrelated queries (Shi et al., 2023). Therefore, self-generated labels obtained from ICL could be directly used as pseudo labels, and distilled back into the LLMs only paired with the query. This paradigm is known as Context Distillation (Askell et al., 2021). For instance, in the alignment process of Llama-2 (Touvron et al., 2023), Context Distillation is used to alleviate the problem of long-term dependencies of system prompts. In Dromedary (Sun et al., 2023d), the base language model is transformed into a safe and helpful aligned model with minimal annotations by directly training on samples acquired from multiple ICL processes. Padmanabhan et al. (2023) demonstrate that Context Distillation can also be used to inject new knowledge to models by learning continuations from entity definitions; Furthermore, Yang et al. (2024b) illustrate the effectiveness of distilling the preference pair generated by contrastive in-context constraints back into the model.

Additionally, the learning content of ICL can also encompass exploratory experiences (Shinn et al., 2023) and tool definitions (Yao et al., 2022; Tang et al., 2023). In other words, agents equipped with tools and experiences can potentially outperform those without. This suggests a similar potential in distilling back the trajectories improved by experience and tools to continuously enhance the same model.

H3: Discussion

Unfortunately, the black box nature of ICL itself poses a significant challenge to alignment (Anwar et al., 2024). Without a comprehensive understanding of how LLMs learn in-context, the context distillation approach may introduce problems by potentially amplifying biases and errors inherent in the in-context learning process of the models. Moreover, the ability of long-term in-context learning (Agarwal et al., 2024) warrants further exploration, as it facilitates more efficient distillation and is crucial for scalable oversight settings where models need to comprehend lengthy professional documents or extensive self-play histories.

3.2 Inductive Bias from Organization of LLMs

In addition to the inductive bias originating from the shared features of LLMs, another set of biases arises from the composition or organization of multiple LLMs, as depicted in Figure 5. Based on whether the relationships between the constituent LLMs are cooperative or adversarial, two representative inductive biases emerge: “Task Decomposition” and “Self-play”. It is noteworthy that, as this field progresses, we anticipate subsequent literature will adopt more complex organizational and learning structures. Both adversarial and collaborative modalities may form integral components of sophisticated agent systems. However, at the current stage, task decomposition and self-play serve as practical taxonomies. Subsequent sections will delve into these concepts in detail.

3.2.1 H4: Task decomposition

Task decomposition has long been regarded as an efficacious approach to tackling complex problems more effectively. For example, in cooperative games grounded in collective rationality, the overall benefits accrued by an alliance exceed the sum of individual gains (Shapley, 1971). Moreover, the divide-and-conquer paradigm and recursion are well-established and effective means employed in algorithm design for addressing problems of substantial scale and complexity (Hoare, 1961; Wilf, 2002).

The discussion of this paradigm can be traced back to the assumption of factored cognition (Ought, 2017). It advocates that cognition tasks can be recursively decomposed. If an AI or human encounters a task that is difficult to solve, it can decompose the task, assign the decomposed problems to a series of its own copies for parallel processing, and finally merge these results. The copies focus on short-term work and work independently. A series of prompting methods implicitly or partially adopts the factored-cognition assumption for automated alignment. For instance, Zhou et al. (2023b) and Wang et al. (2023b) prompted the LLM to decompose the problem, then guide it to sequentially solve the sub-problems. It is also believed that task decomposition is an effective method for solving Easy-to-Hard generalization (Zhou et al., 2023b), that is, constructing decomposition prompts on simple samples and fill them in-context allows LLM the potential to generalize to difficult samples. Khot et al. (2023) further implemented recursive task decomposition.

Based on the assumption of factored cognition, Iterative Distillation and Amplification (IDA) (Christiano et al., 2018) views each decomposition-merging process as a form of amplification and considers learning from the final merged results as a form of distillation. Although the original IDA paper builds this theoretical framework in a human-in-the-loop manner, where humans supervise the initial task decomposition step, it is likely that this process can be initiated without human oversight due to H1, H2, and H3 (Zhang and Parkes, 2023).

Notably, Iterative Distillation and Amplification (IDA) represents a promising avenue towards achieving scalable oversight, making it possible to address long-horizon tasks that are difficult for humans to directly supervise by decomposing tasks into more tractable sub-problems. For example, labels for data points like “peer-review this survey” can take several months to collect in real world. Such problems can be tackled more quickly through factored cognition. Although some work partially demonstrates the effectiveness of IDA on real-world tasks like book-length summarization (Wu et al., 2021a), this ideology still relies on a set of crucial assumptions: 1) It is still unclear whether decompose a problem is the hardest part of solving it, if the cognitive burden cannot be distributed, IDA may struggle to take effect. 2) The error won’t accumulate. Although this paradigm does not require the collaboration between agents to be efficient (Christiano et al., 2018), too many errors can still be problematic. 3) The extent to which tasks can be parallelized. If the task-solving process is largely sequential, the time to collect signals might increase, but this appears to be a minor issue given the current deployment speed of LLMs. Overall, since these assumptions are difficult to prove or falsify, we advocate for more empirical research in this direction.

3.2.2 H5: Self-play

Complexity emerge from adversariality (Bansal et al., 2018). Self-play refers to a paradigm where an agent learns by iteratively playing games against itself, a form of non-cooperative game (Nash et al., 1951) in which each agent aims to maximize its own utility. Self-play serves as the foundation of many successful specialized superhuman AI systems like AlphaGo Zero (Silver et al., 2018) and StockFish (StockFish, 2023). Given this success, self-play seems to be a potential approach for enabling general proposed superhuman intelligence from LLMs. Two representative self-play methods are the Generator-Discriminator and the Debate approaches, with the latter involving $N\ge 2$ adversarial generators and one discriminator in a gaming environment.

H5.1: Generator-Discriminator

In the Generator-Discriminator self-play framework, the role of the discriminator is to evaluate the outputs produced by the generator, determining whether these outputs are of high or low quality.

As discussed in H2, the judge and critic model is commonly considered a type of discriminator. For example, Yuan et al. (2024) utilized rewards from LLM-as-a-Judge to identify high-quality and low-quality responses from the generator, optimizing the generator towards the higher quality ones. However, the adversarial setting between the discriminator and the generator is limited because the only assumption is that the discriminator’s ability can be improved with general helpful training. The discriminator remains almost static (with unchanged prompts) during training, making it possible for the generator to be over-optimized against the discriminator, leading to reward hacking. Effectively improving the judge and critic module alongside the generator is thus a crucial problem. One reasonable approach is to use the online AIF setting, as suggested by Guo et al. (2024b). Another way to introduce a more adversarial setting is to optimize the gaming problem at inference time, as demonstrated in the Consensus Game by Jacob et al. (2024). It employs the piKL no-regret learning algorithm to iteratively update the strategies of both the generator and discriminator, converging to a Nash equilibrium. This equilibrium strategy is then used to rank candidate responses, prioritizing those agreed upon by both players.

As Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have become a well-established class of methods in traditional NLP (Zhang et al., 2016; Wu et al., 2021b), another approach involves using a GAN-like discriminator to distinguish between the model’s current predicted distribution and the golden distribution. For instance, Chen et al. (2024g) found that a specific type of iterative DPO training, which consistently treats the policy-generated responses as negative and the golden responses as positive, can be viewed as a self-play process. In this process, the implicit reward of DPO functions as a discriminator between the model’s predictions and the golden samples. However, for open-ended questions, the golden distribution is sometimes still suboptimal, and this approach precludes the possibility of generating responses that are better than the golden ones.

H5.2: Debate

The debate paradigm (Irving et al., 2018) is largely inspired by the design of AlphaGo (Silver et al., 2018). In the learning algorithm of AlphaGo, three distinct components are integrated: a player, a counterpart—which is effectively a simulation of itself—and a value model that evaluates the win probability associated with each board state. By employing Monte Carlo Tree Search (MCTS), the algorithm conducts rollouts, defined as simulated self-play sequences that extend until a game’s conclusion. These rollouts enhance the accuracy of the value estimates through backward updates based on the outcomes, concurrently refining the policy by capitalizing on strategies that have previously led to victories.

The game of Go shares similarities with tackling the scalable oversight problem using natural language debate. In the beginning or middle of a Go game, even experienced experts may find it difficult to judge which side has a higher probability of winning, just as humans have a small probability of judging problems that exceed human knowledge levels. However, as the game nears its end, the outcome usually becomes clear that even a non-expert judge can confidently evaluate the board generated by the Go masters. For a debate competition, the winner can typically be summarized by the judges.

This provide a possible oversight solution to build trustworthy superhuman AI systems. Irving et al. (2018) show that the honesty is the better strategy than lie in debate paradigm through proof-of-concept experiments. As a extension of this, Brown-Cohen et al. (2023) propose a new set of debate protocols, wherein the honest strategy can always succeed through a simulation involving only a polynomial number of steps. Khan et al. (2024) conduct a thorough empirical study on the feasibility of implementing the debate paradigm on LLMs: it was found that the debate paradigm can significantly enhance truthfulness, and more persuasive (Anthropic, 2024) debaters lead to more truthful outcomes.

This offers a potential solution to construct trustworthy superhuman AI systems. Irving et al. (2018) honesty proves to be a superior strategy compared to deceit in a debate paradigm through proof-of-concept experiments. Expanding on this notion, Brown-Cohen et al. (2023) introduced a novel set of debate protocols, wherein the honest approach can consistently prevail, as demonstrated in simulations requiring only a polynomial number of steps. Khan et al. (2024) undertook a comprehensive empirical investigation into the practicality of implementing the debate paradigm on LLMs, revealing that it substantially enhances truthfulness, with more compelling debaters resulting in more truthful outcomes (Anthropic, 2024).

Apart from the classic natural language debate, a series of works have implemented the debate paradigm in game scenarios. A representative arena is the bargaining task (Nash et al., 1950). Fu et al. (2023a) focused on zero-sum variants of bargaining, where the balloon seller aims to sell at a higher price while the buyer seeks a lower price. They observed significant variations in bargaining capabilities among different LLMs and their capacity to learn from play experiences and feedback. Cheng et al. (2024) implemented the adversarial language game adversarial taboo (Yao et al., 2021), where an attacker and a defender engage in a conversation centered around a target word visible only to the attacker. The attacker subtly induces the defender to unconsciously utter the target word, while the defender tries to avoid doing so and guess the word from the context. Both players acquire basic gaming skills through imitation learning from a teacher LLM and then refine their strategies through self-play. Interestingly, less capable player LLMs not only improve their win rates in this specific game through self-play but also enhance their general reasoning abilities Ma et al. (2023a) introduced the red-teaming game, a more intricate adversarial team game where LLMs are initialized as a joint set of red-teaming policies to prompt the target LLM to produce harmful content. They proposed a solver to ensure the final meta-strategy approximates a Nash equilibrium within a certain $ϵ$ margin.

3.2.3 Discussion

Task decomposition and self-play both necessitate that LLMs function as agents. However, the challenge of aligning LLMs as agents is more complex compared to aligning them as chatbots, as it requires considerations of behavior-level alignment (Pan et al., 2023), the dynamics of environment and self-constraints (Garrabrant and Demski, 2018; Shavit et al., 2023; Yang et al., 2024d). We emphasize the significance of this research direction and advocate for increased efforts in this domain. Furthermore, an even more intricate problem lies in proving the theoretical safety and trustworthiness of multi-agent systems. Although research in this area is nascent (Yang and Wang, 2020; DiGiovanni and Zell, 2021), advancements in game theory (Hazra and Anjaria, 2022), automated theorem proving techniques (Polu and Sutskever, 2020) and real-world simulation technology (Brooks et al., 2024) may offer insights into addressing this challenge.

While adversarial self-play offers a potential solution to the scalability oversight challenge, it can also give rise to the emergence of more persuasive and autonomous agents (Tao et al., 2024). Such developments could have significant social impacts and ethical risks, such as the potential for models to generate more persuasive articles than humans, which could be exploited for political manipulation. Encouragingly, several prominent model engine providers have taken steps to monitor and mitigate these potential side effects. For instance,OpenAI’s Preparedness team has established benchmarks related to persuasion and autonomy (OpenAI, 2023a). They categorized model risks into four levels: low, medium, high, and critical, stipulating that models above the high-risk threshold cannot be developed, and those above medium risk cannot be deployed. We call upon the academic community and third-party organizations to dedicate additional efforts to the development and review of a robust safety framework for highly capable agents.

4 Aligning Through Behavior Imitation

Imitation is the first instinct of the awakening mind.

Maria Montessori

Aligning through behavior imitation is another widely-used strategy for automated alignment, which aligns the target model by mimicking the behavior of another aligned model. Specifically, as demonstrated in Figure 6, this method begins by collecting high-quality instructions as task descriptions (Wang et al., 2023f). A supervised model is then employed to generate alignment signals, which typically include instruction-response pairs (Taori et al., 2023), pair-wise preference data (Cui et al., 2024), and other alignment signals (Fränken et al., 2024). Ultimately, the target model is aligned by imitating these produced behaviors.

Based on the capability comparison between the supervised model and the target model, studies on aligning through behavior imitation can be categorized into strong-to-weak distillation (§4.2) and weak-to-strong alignment (§4.3). For each category, we thoroughly review the representative studies, summarize current progress and limitations, and discuss future directions.

Figure 6: The illustrations of representative studies for aligning through behavior imitation.

4.1 Instruction Construction

Collecting large-scale instructions with high quality and diversity serves as the foundation for achieving alignment through behavior imitation. The most intuitive strategy involves filtering out high-quality data from human-written instructions, such as those found in ShareGPT . However, this approach requires substantial human effort and expertise, which also introduces significant noise. Consequently, many studies focus on utilizing LLMs for automatic instruction generation, thereby significantly reducing the dependence on human annotation. Based on the information provided for instruction construction, there are currently three representative strategies:

In-Context Generation, which provides in-context demonstrations to guide LLMs generating instructions. For example, Unnatural Instructions (Honovich et al., 2023) and Self-Instruct (Wang et al., 2023f) begin with a small set of human-written instructions. These instructions are randomly selected to create context examples that prompt LLMs to generate additional instructions. Similarly, Stanford Alpaca (Taori et al., 2023) employs a comparable approach by using ChatGPT to gather instruction-response pairs.

However, the initial seed instructions may lack diversity, which limits the range of generated instructions. To address this, LaMini-LM (Wu et al., 2024) not only utilizes gpt-3.5-turbo for Example-Guided Instruction Generation based on In-context Learning (ICL), but also incorporates wiki data for Topic-Guided Instruction Generation, thereby constructing a large, offline distilled instruction dataset. Guo et al. (2024a) use high-quality samples relevant to the target domain as examples for ICL to generate additional samples. Dynosaur (Yin et al., 2023) employs ChatGPT to generate instructions by leveraging meta-information from existing NLP datasets, creating a dynamically growing instruction tuning dataset. Furthermore, LLM2LLM (Lee et al., 2024) enhances the dataset by generating new data from the teacher model using incorrect predictions from the student model, which improves the dataset’s ability to handle data-constrained tasks.

Instruction Evolution, which involves the process of rewriting existing instructions based on predefined handwritten evolution principles. Xu et al. (2024a) apply an instruction evolution paradigm, where existing instruction data is transformed using randomly selected handwritten evolution principles. This approach is employed for robust models such as GPT-3, facilitating the rewriting of instructions, thereby reducing the need for manual annotation and enhancing the model’s ability to manage complex tasks. Building upon the Evol-Instruct framework, TeaMs-RL (Gu et al., 2024) advances this methodology by training another model through reinforcement learning to generate optimized instruction trajectories.

Instruction Backtranslation, which employs LLMs to predict instructions that correspond to the golden responses extracted from human handwritten text or web documents. Humpback (Li et al., 2024e) and LongForm (Köksal et al., 2024) utilize corresponding prompts to label text from web documents, thereby creating high-quality instructional data. Differently, TEGIT (Chen et al., 2023e) engages ChatGPT to annotate sampled text, simultaneously generating instructions, inputs, and outputs to effectively filter out noise. REInstruct (Chen et al., 2024b) builds high-quality instructions from an unlabelled corpus and rewrites the unlabelled text to enhance its quality as response.

4.2 Strong-to-Weak Distillation

Based on the collected instructions, strong-to-weak distillation seeks to align the weaker target model by imitating the responses or preference data generated by another stronger and well-aligned model. In the following subsections, we will introduce representative studies concerning response-guided and preference-guided distillation, respectively.

4.2.1 Response-Guided Distillation

In response-guided distillation, the target model emulates the teacher model by directly learning the responses to different instructions through instruction tuning. This approach has inspired numerous studies that aim to distill various capabilities from the teacher model to the target model. These capabilities include not only general instruction-following skills but also domain-specific abilities such as mathematics, coding, and agent-related tasks

Instruction-Following

After constructing instruction data, corresponding responses can easily be developed from the teacher model. Training with these instruction-response pairs emulates the teacher’s capability of following instructions. For instance, LLaMA-GPT4 (Peng et al., 2023) utilizes GPT-4 to generate responses to instructions derived from Alpaca (Taori et al., 2023). Ubani et al. (2023) discuss the zero-shot prompting of ChatGPT for data augmentation in low-resource scenarios, synthesizing instructional data by designing high-quality and diverse prompts. More related works can be found in the last subsection. In addition to single-round data, some studies focus on collecting multi-turn trajectories from teacher models. Baize (Xu et al., 2023b) and Ultrachat (Ding et al., 2023) employ a comprehensive framework, using two ChatGPT APIs to play the roles of user and assistant to generate multi-round conversations. Parrot (Sun et al., 2023b) trains models to simulate humans in generating instructions and uses these trained models to engage in multi-turn conversations with ChatGPT on various topics, resulting in a collection of 40,000 multi-turn dialogues.

Mathematics

Wizardmath (Luo et al., 2023a) employs the Evol-Instruct method to construct a comprehensive dataset specifically for mathematical reasoning tasks. MetaMath (Yu et al., 2024b) utilizes ChatGPT to bootstrap mathematical questions by rephrasing them from multiple perspectives without introducing additional knowledge. MAmmoTH (Yue et al., 2024a) produces a dataset comprising math problems and model-generated solutions, distinguished by a unique combination of chain-of-thought (CoT) and program-of-thought (PoT) rationales. MathCoder (Wang et al., 2024d) generates innovative and high-quality math problems along with their code-based solutions using the GPT-4 Code Interpreter. MathGenie (Lu et al., 2024) generates diverse and reliable math problems through a process of question back-translation. MARIO (Liao et al., 2024) leverages GSM8K and MATH as seed data, resulting in a collection of 26.9K solutions annotated by both GPT and human experts. Beyond purely mathematical data, several other studies propose transferring essential reasoning abilities from commercial LLMs to small models by generating detailed COT responses (Shridhar et al., 2023; Fu et al., 2023b; Hsieh et al., 2023; Magister et al., 2023; Ho et al., 2023; Li et al., 2022, 2023a).

Coding

State-of-the-art LLMs, such as GPT-4, demonstrate exceptional performance in coding tasks. Apart from pre-training on raw code data, some approaches aim to transfer coding capabilities from teacher models through instruction tuning. Code Alpaca (Chaudhary, 2023) and WizardCoder (Luo et al., 2024) adhere to general automatic instruction-building paradigms. Code Alpaca employs Self-Instruct on 20K instruction-following data, thereby extending Alpaca’s capabilities to the coding domain. WizardCoder adapts the Evol-Instruct method for the coding domain, generating complex code and program instructions from simple coding and programming directives. WaveCoder (Yu et al., 2024c) and Magicoder (Wei et al., 2023) create high-quality instruction data utilizing open-source code datasets. WaveCoder enhances LLMs with open-source code snippets to produce superior instruction data for coding tasks. Magicoder creates multi-task data generated according to the techniques of the Self-Instruct. OpenCodeInterpreter (Zheng et al., 2024c) utilizes GPT-3.5 and GPT-4 to improve solutions with integrated text explanations and code snippets, incorporating execution and feedback for dynamic code refinement.

Agent

Although open-source LLMs have achieved comparable performance to commercial models in many aspects, their capabilities in agent-related functions, such as tool usage and complex task planning, remain significantly limited. To address this issue, ToolLLM (Qin et al., 2023) has created an instruction-tuning dataset named ToolBench with ChatGPT, acquiring general tool usage capabilities in a zero-shot manner. Similar works include Graph-ToolFormer (Zhang, 2023), Gorilla (Patil et al., 2023), GPT4Tools (Yang et al., 2023b), ToolAlpaca (Tang et al., 2023), and others. Beyond tool usage, some studies focus on planning tasks. Examples include FIREACT (Chen et al., 2023a), AgentTuning (Zeng et al., 2023), ReAct Meets ActRe (Aksitov et al., 2023), ReST meets ReAct (Yang et al., 2024c), and ETO (Song et al., 2024b).

4.2.2 Preference-Guided Distillation

Although response-guided distillation can enhance the performance of student models (Wang et al., 2022), it does not effectively help the student model align with human preferences (Xu et al., 2024c). Therefore, some works concentrate on preference-guided distillation, which aligns the student model with the preferences reflected in the output from the teacher model. In this paradigm, the teacher model is guided to generate preference data in the form of partial order pairs, which are then used to align the student model via direct preference optimization algorithms such as DPO (Rafailov et al., 2023), IPO (Azar et al., 2024), and PRO (Song et al., 2023). Based on the methodologies for constructing partial order signals, current works primarily encompass three paradigms: 1) Score-based:, which involves scoring and ranking responses; 2) Refine-based, which involves refining existing responses with AI feedback; and 3) Source-based, which focuses on learning the human preference of different data sources.

Score-based

Through the implementation of meticulously designed diverse instructions and model responses, along with detailed numerical and text feedback provided by GPT-4, UltraFeedback (Cui et al., 2024) generates a large-scale, high-quality preference dataset with fine-grained annotations. Additionally, Zephyr (Tunstall et al., 2023) employs distilled direct preference optimization on UltraFeedback to develop small yet efficient LLMs. CodeUltraFeedback (Weyssow et al., 2024) leverages the LLM-as-a-Judge approach of GPT, evaluating responses from a pool of 14 different LLMs and aligning them according to five coding preferences.

Refine-based

Other studies improve initial responses using powerful models. Aligner (Ji et al., 2024) and MetaAligner (Yang et al., 2024a) utilize models such as GPT-4 to revise original responses and construct preference data. IterAlign (Chen et al., 2024e) automatically discovers new constitutions using an LLM and optimizes responses generated from a red team dataset to create preference data. Safer-Instruct (Shi et al., 2024) employs reversed instruction tuning, instruction induction, and expert model evaluation, using both raw text and GPT-4 generated responses to build high-quality preference data.

Source-based

Learning preferences from a single model may lack diversity and amplify bias. Therefore, some works build partial order signals from different data sources. AlMoST (Kim et al., 2023), CycleAlign (Hong et al., 2023), and Openchat (Wang et al., 2024b) focus on learning comparative preferences from different data sources. Kim et al. (2023) transform human preferences into a series of empirical prior rules, using LLMs of various sizes to generate preference data. Wang et al. (2024b) treat different data sources as coarse-grained reward labels, generating mixed-quality data through GPT-3 and ShareGPT. Hong et al. (2023) rank responses by comparing the agreement rank of white-box and black-box models across a series of responses and constructed preference data through this ranking as context.

4.3 Weak-to-Strong Alignment

As we mentioned in Section 1, the challenges of scalable oversight become a significant barrier for the continuous development of AI systems. Specifically, the difficulty lies in effectively providing supervision as the capabilities of AI systems gradually surpass those of humans. Given the impracticality of strong-to-weak distillation approaches, weak-to-strong alignment has emerged as one of the most promising directions for achieving automated scalable oversight (Burns et al., 2023). Previous studies have mainly focused on weak-to-strong generalization between humans and AI, such as the Iterated Amplification method (Christiano et al., 2018), which supervises strong learners by iteratively amplifying weak experts. Recent research has begun to explore using weaker models to guide stronger models to achieve superalignment (Christiano et al., 2018; Burns et al., 2023; Liu and Alahi, 2024). Based on the source of the alignment signals, these works can be categorized into two types: 1) using smaller but aligned models to generate signals, and 2) using weaker models to guide stronger models in generating signals. Moreover, some studies investigate whether models can learn from behaviors in easy tasks to improve their performance in more challenging tasks, which, although not classic behavior imitation, is still noteworthy (Hase et al., 2024; Sun et al., 2024b). In the following subsections, we will introduce the representative studies in each category respectively.

Burns et al. (2023) employ a weaker LLM as the teacher to train a stronger LLM using a weak-to-strong approach. They fine-tune a larger pre-trained model based on labels generated by the smaller, but aligned, model and observe that the larger target model consistently outperforms the smaller supervisory model. Instead of relying on a single teacher, Liu and Alahi (2024) aim to further enhance the alignment of strong models by co-supervising a powerful student with a diverse group of professional teachers. Somerstep et al. (2024) examine weak-to-strong generalization as a transfer learning problem, achieving this through a label refinement procedure. Additionally, Aligner (Ji et al., 2024) and MetaAligner (Yang et al., 2024a) create partial order data by using a significantly smaller but aligned model to optimize the responses from stronger models.

In addition to directly generating signals from weak models, another possible method to achieve weak-to-strong alignment is using weak models to guide strong models in generating signals. Li et al. (2024c) find the ability of both weak and strong LLMs to perceive instruction difficulty and select data is highly consistent. Thus, smaller and weaker models can be utilized to select data for fine-tuning larger and stronger models. Similarly, SAMI (Fränken et al., 2024) employs a weak model to write constitutions for aligning a strong baseline model.

The aforementioned works achieve weak-to-strong alignment to a certain extent and investigate potential directions for achieving superalignment. However, weaker models may not serve as effective instructors for more complex tasks. Consequently, some studies attempt to align models using signals derived from simpler tasks, which are easier to generate and learn, in order to enhance performance on more difficult tasks. For instance, Hase et al. (2024) observe that current language models generally extrapolate well from simple to complex data and can even compete with models trained directly on intricate data. Sun et al. (2024b) use reward models trained on simple tasks to assess and guide policy models on more challenging tasks, thus achieving task generalization.

4.4 Discussion

Current works leverage the responses or preferences from teacher models to facilitate effective generalization and scalability across various tasks, thereby significantly diminishing the necessity for manual annotation. However, approaches exhibit notable limitations, including issues related to data quality, bias inherent in the teacher models, and inadequate exploration of superalignment.

Data Quality

The quality of synthetic data remains a significant concern. Numerous studies highlight the critical importance of data quality for alignment (Zhou et al., 2023a; Chen et al., 2023b). Training signals derived from teacher models are often noisy due to the inherent randomness in model generation. To address this issue, recent research has concentrated on two main paradigms: firstly, generating high-quality data by formulating detailed and refined principles, such as Orcas (Mukherjee et al., 2023; Mitra et al., 2023) and AttrPrompt (Yu et al., 2023); and secondly, extracting relatively high-quality data from existing datasets by establishing evaluation metrics or employing filtering paradigms, such as Reflection-Tuning (Li et al., 2023b, 2024b) and Phis (Li et al., 2023d; Abdin et al., 2023, 2024) . Additionally, some research suggests that alignment algorithms possess a certain degree of robustness (Gao et al., 2024). Developing more robust training algorithms may thus be another approach to mitigating the issues associated with data quality.

Bias of the Teacher

Additionally, reliance on the teacher model may introduce biases and limitations inherent in the teacher model, which can affect the alignment effectiveness. Some studies propose introducing multiple teacher models to align the student model (Cui et al., 2024; Liu and Alahi, 2024), thereby reducing the likelihood of the model overfitting to the biases of a single teacher model. Utilizing multiple teachers can also increase the diversity of signals, significantly enhancing alignment effectiveness (Song et al., 2024a).

Insufficient Understanding of Superalignment

Achieving superalignment remains a significant challenge. We still lack a strong scientific understanding of superalignment (Burns et al., 2023), hindering further exploration of weak-to-strong alignment. Additionally, most current approaches still require a sufficiently aligned “weak” model, and how to utilize a truly weak model for superalignment remains an issue. Some works present theoretical frameworks for understanding weak-to-strong generalization (Charikar et al., 2024; Lang et al., 2024; Somerstep et al., 2024), but still have limited application scope. An interesting road is like ExPO (Zheng et al., 2024a), it extrapolates directly from an SFT model and an aligned model’s weights, obtaining a better-aligned model without additional training, demonstrating a promising approach from weak to strong.

In conclusion, despite significant progress in instruction and behavior construction, current approaches still have significant limitations. The core issue with strong-to-weak methods is that the alignment ceiling is constrained by the teacher model. Conversely, works about weak-to-strong alignment remain underdeveloped, lacking theoretical analysis and generalized methodologies. Several critical issues must be addressed in the future, including enhancing data quality efficiently, developing more robust training algorithms, implementing multi-teacher imitation, and conducting theoretical analysis for weak-to-strong alignment in general tasks. Tackling these challenges will pave a feasible path for the further advancement of LLMs. Moreover, we also provide an in-depth discussion about the underlying mechanisms of weak-to-strong alignment in Section 7, which sheds light on a deeper understanding of this field.

5 Aligning Through Model Feedback

We all need people who will give us feedback. That’s how we improve.

Bill Gates

Human feedback reflects human values and can be used to align the LLMs, enabling LLMs to produce helpful and safe responses while correcting errors and toxic outputs. Unfortunately, accessing human feedback during training is challenging due to inefficiency and high costs. To address this issue, model feedback is introduced as a way to estimate human feedback. This approach is commonly utilized in reinforcement learning, where a reward model generates feedback. Compared to relying on the limited feedback data generated by humans, the reward model can perform feedback prediction across a broader distribution, thus achieving more efficient alignment. Aligning through automated generated model feedback offers an effective method for aligning LLM with human values, presenting a promising path towards achieving automated alignment. In this section, we explain how to leverage model-provided feedback to align it with human values. As shown in Figure 7, the methods can be divided into three types based on the form of feedback signals: scalar (§ 5.1), binary (§ 5.2), and text signals (§ 5.3).

Figure 7: The illustration of aligning through model feedback generated by reward model. The reward model will automatically generate the feedback of LLMs’ responses in the scalar, binary or text format.

5.1 Scalar Reward

Scalar signals are commonly generated by a reward model that takes the response of LLMs as input to generate scalar signals for estimating human preferences. Reward model is frequently employed in reinforcement learning to align LLM with human values. In this way, LLMs can automatically align with human values by utilizing the large amount and diverse feedback provided by the reward model. To achieve more effective automated alignment, recent studies focus on how to train a higher-quality reward model, and reducing the reliance on human annotation during the training of the reward model through model generation or pre-training. Besides, the scalar signals generated by reward model can also be used to optimize the generation of LLM during decoding and filter training data for instruct-tuning.

5.1.1 Reinforcement Learning from Human Feedback

Reinforcement Learning from Human Feedback (RLHF) is a crucial paradigm for aligning LLMs with human values (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022). It typically involves three steps: 1) supervised fine-tuning (SFT), where LLMs are trained on annotated data to improve their responses to prompts; 2) training reward models to anticipate human feedback on model responses; and 3) employing reinforcement learning algorithms like Proximal Policy Optimization (PPO) (Schulman et al., 2017) to align the model. In RLHF, the reward model, which is usually trained on preference data annotated by humans, produces scalar signals that mimic human feedback, serving as the guiding signal for learning. The performance of the reward model determines the potential upper bound of the model’s alignment, so training the reward model is of vital importance (Zheng et al., 2023). In the following sections, we first introduce related works about enhancing reward model. Then we introduce how to generate preference data without human effort. Finally, we introduce the functions of the reward model beyond reinforcement learning, including alignment during the decoding stage and SFT data filtering.

5.1.2 Improvement for Reward Modeling

To achieve more effective automated alignment, it is crucial to improve the quality of model feedback. Therefore, recent studies focus on learning high-quality reward models. The primary challenges in training reward model involve data collection and model optimization. Preference data collected is typically sparse and deficient in consistency and detail, and model optimization can be hindered by issues such as over-fitting.

Reward Model Pre-training

Due to the data sparsity of existing datasets and the expense of human annotation, it is hard to train a high-quality reward model for automated alignment. To this end, Askell et al. (2021) propose the reward model pre-training. By collecting pair data from the network, including StackExchange, Reddit, and Wikipedia, they constructed a ranked dataset to pre-train a preference model. By leveraging reward model pre-training, the reliance on human annotation is diminished (Bai et al., 2022a), which facilitates more efficient training of the reward model and enhances the effectiveness of automated alignment.

Consistent Preference Data Construction

Because human annotators have different evaluation principles and subjective perspectives, the feedback is diverse and includes multiple viewpoints. Previous studies have used strategies like multiply models ensemble (Rame et al., 2023; Touvron et al., 2023), multi-objective learning (Zeng et al., 2024) to mitigate the negativity from the diversity data. In contrast to reward models that produce a single score, Li et al. (2024a) introduce the Distributional Preference Reward Model (DPRM) for predicting preference distributions.

Fine-grained Feedback Collection

Reward models often struggle to offer fine-grained feedback for intricate situations like safety and challenging tasks such as reasoning. To address this issue, some studies concentrate on refining reward models’ training. Chen et al. (2024f) introduce a token-level reward model capable of providing precise feedback at the token level, suitable for complex tasks like reasoning. Wu et al. (2023b) suggest training multiple reward models that can deliver detailed feedback at the text span level.

Training Optimization

The learning process of the reward model usually faces the problem of over-optimization. That is, through learning, the reward model performs poorly instead. Gao et al. (2023) analyze this phenomenon through experiments and discovered the scaling law of the reward model to guide learning. Zhu et al. (2023a) provide a theoretical analysis of the reward model training in RLHF and show the importance of introducing pessimism when training. Moreover, some other works have adopted various ways to improve the performance of the reward model, including normalization (Zheng et al., 2023) and iterative learning (Touvron et al., 2023).

Although the purpose of the reward model is to predict human feedback, the reward modeling is challenging. Therefore, how to build a more comprehensive reward model to achieve automated alignment is an important research question.

5.1.3 Reinforcement Learning from AI Feedback

Reward model is commonly trained using human feedback which is hard and expensive to annotate. For the purpose of reducing human effort and improving the automation in alignment, there are some works that use the existing large language models to generate the preference data. Reinforcement Learning from AI Feedback (RLAIF) (Lee et al., 2023) trains the reward model with preference data of LLM and it can achieve comparable or superior performance to RLHF. The methods are mainly divided into two types, including ranking the responses of multiple models and directly generating positive and negative responses. In this way, the automated alignment can be achieved in the entire procedure of reinforcement learning without human effort.

Ranking Multiple Responses

With the improved capabilities of LLMs, directly using them to rank multiple responses can provide preference data (Tunstall et al., 2023; Hong et al., 2023; Guo et al., 2024b; Pace et al., 2024; Yuan et al., 2024). This ranked preference data can also be produced with minimal human supervision, such as through human-defined principles (Bai et al., 2022b; Sun et al., 2023c) or rules (Kim et al., 2023). To improve the quality of generated preference data, Shi et al. (2024) propose a carefully designed pipeline including reversed instruction tuning, instruction induction, and expert model evaluation. Liu et al. (2024a) propose to score the response using contrastive prompt pairs as input, which can achieve better performance compared to directly generated feedback with single prompt.

Generating Positive and Negative Responses

Some of works use LLM to generate preference data directly by prompting it to output positive response and negative response (Chen et al., 2024c). Yang et al. (2024b) use an indirect approach by using different prompts to generate positive and negative responses respectively.

Although promising, the main challenge of it is the quality of preference data. Since the LLMs are commonly interfered with many aspects during generating, such as position bias (Zheng et al., 2024b; Wang et al., 2023c), the generation of high-quality preference data still requires further exploration. With the continuous improvements in LLMs, using LLM to reduce human effort will be a key strategy for automatically aligning models in the future.

5.1.4 Reward Model-Guided Decoding

In addition to learning directly from preference data, the generation of LLM can be enhanced by the scalar signals provided by reward model. This allows alignment to be directly achieved in the output rather than within the model via re-weighting the probabilities of tokens (Mudgal et al., 2024). Lango and Dusek (2023) propose a critic-driven decoding approach to adjust the probability of token during generating via a binary classifier as the critic model. Deng and Raffel (2023) propose Reward-Augmented Decoding (RAD) that employs an attribute-specific reward model to re-weight the top-k highest probabilities when decoding. To achieve flexible alignment in different tasks, Liu et al. (2024b) propose decoding-time realignment (DeRa) to control the alignment level during decoding.

Performing automated alignment at only the decoding stage is a simple method to avoid consuming a large amount of computing resources. However, alignment during decoding usually requires more time to conduct inference, and the quality of response still needs to be further improved.

5.1.5 Filtering SFT Data using Reward Model

High-quality SFT plays a crucial role in enhancing LLM performance (Zhou et al., 2023a), therefore some studies use reward models to filter the training data. The main paradigms are categorized into learning from best response and learning from ranked results. Learning from the best response is often referred to as Best of N or Reject sampling (Touvron et al., 2023; Yuan et al., 2023b). This approach typically involves using a reward model to choose the high-quality data among multiple responses for refining the model (Dong et al., 2023). In addition to learning from top responses, the LLM can also learn from ranked data. Yuan et al. (2023a) propose Rank Responses to align Human Feedback (RRHF) to align LLM using ranking loss. Lu et al. (2022) propose using a reward model to grade data based on their scores and employing various reward tokens to regulate their generation. This approach helps prevent the learning of undesirable behaviors.

Besides reinforcement learning, SFT is also an important way to achieve alignment. By filtering data through the reward model, the LLM can automatically align with human values through SFT. Similar to the previous problem, the quality of SFT data is highly dependent on the quality of the reward model and needs further research.

5.2 Binary Verifier

For some objective tasks, such as mathematical problems, the reward model usually transforms into a verifier with binary signals. Given that mathematical problems usually require complex step-by-step reasoning, the verifier can be divided into outcome verifier and process verifier. Outcome verifier is used to estimate the correctness of the final answer. Process verifier accesses the intermediate step, which requires a large number of supervision data. Through the binary verifier, the LLMs can achieve automated alignment on these objective tasks.

Outcome Verifier

To improve the reasoning ability of LLMs, some studies focus on selecting reasoning paths generated by LLMs using golden answers for training (Zelikman et al., 2022; Singh et al., 2024). Since golden answers are costly to acquire, an outcome verifier is utilized to forecast the correctness of generated answers. This verifier is typically trained using LLM-generated correct and incorrect rationales (Cobbe et al., 2021), and is used to fine-tune the LLM through different strategies including direct tuning (Liu et al., 2023c) and iterative training (Hosseini et al., 2024). Since outcome verifier cannot assess the correctness of reasoning steps, Havrilla et al. (2024) propose Stepwise Outcome Reward Models (SORMs) that predict whether a step will lead to the correct answer. Besides training, Yu et al. (2024a) propose the Outcome-supervised Value Model (OVM) which is used to guide decoding.

Process Verifier

Even if the final answer is correct, there may still be errors in the reasoning process, limiting the effectiveness of the outcome verifier. To address this problem, the process verifier is employed to assess the correctness of reasoning step for more detailed verification (Lightman et al., 2023; Uesato et al., 2022). The process verifier can be used to train a more effective reasoner (Ying et al., 2024; Shao et al., 2024). Inspired by the human reasoning mechanism, Zhu et al. (2023b) propose Cooperative Reasoning (CoRe) to produce synthesized training data for reasoning where process verifier is used to generate the feedback of the generation of generator. Many studies are dedicated to training the verifier using automatically generated data due to the difficulty of collecting step-wise supervised signals. Wang et al. (2024e) and Wang et al. (2024h) train the process verifier with automatically constructed data that is collected by Monte Carlo Sampling. Moreover, the process verifier is can be applied in decoding to select the correct reasoning path (Khalifa et al., 2023). Some studies focus on how to complete the final reasoning path efficiently. Ma et al. (2023b) propose a heuristic greedy search algorithm based on the verifier’s feedback. Li et al. (2023c) propose using the verifier to filter the reasoning steps generated using diverse prompts.

Binary verifier is an important way to achieve automated alignment in objective tasks such as math. However, training the verifier, especially the process verifier is very difficult and requires a large amount of annotated data. Therefore, in the future, in order to further achieve automated alignment, researchers can focus on how to automatically construct a process verifier.

5.3 Text Critic

Text signals contain more semantics than scalar and binary signals, enabling models to intuitively align with humans. By integrating text feedback, LLMs can improve their output alignment with humans. These refined outputs can then be used as supervised data for further aligning the LLMs (Scheurer et al., 2022; Chen et al., 2024a). Text signals generated by the critique model have shown the potential to improve the outputs of LLM, with feedback typically obtained through prompting LLMs. The text critic can be other LLMs (such as GPT-4) (Koutcheme et al., 2024; An et al., 2024) or LLM itself (i.e., self-critique) (Saunders et al., 2022; Wang et al., 2023d). Since the self-critique of LLM remains a challenge (Luo et al., 2023b), text signals for alignment are still underexplored.

Existing studies mainly focus on using the text critic to achieve automated alignment by improving the output of LLMs. In the future, exploring how to use the text critic to achieve more diverse automated alignments, such as in training, is an important research direction.

6 Aligning Through Environment Feedback

We do not learn from experience… we learn from reflecting on experience.

John Dewey

This section involves automatically obtaining alignment goals or feedback from the existing environments to achieve automated alignment of the target model. As illustrated in Figure 8, according to the category of environments the model interacts with, we overview four lines of current research in this section and systematically go through the representative works of each part:

• Social interactions (§ 6.1), where models communicate with each other to build up a multi-agent system and collect alignment signals through such interactive communication.

• Human collective intelligence (§ 6.2), where models receive judgements from human societies to calibrate their internal values.

• Tool execution feedback (§ 6.3), where models interact with external tools to receive instant feedback and achieve various intelligence by learning to use tools smartly.

• Embodied environment (§ 6.4), where models act as language interfaces in a physical world and get reward based on task goals.

Figure 8: Alignment through various types of environment feedback by social interactions, human collective intelligence, signals from tool execution or embodied environment.

6.1 Social Interactions

Social interaction is one of the fundamental properties of human society, in which many social norms are conveyed and followed in implicit ways. Recent advancements of large language models provide the opportunity to construct LLM-based agent system to simulate such interactions among human society, enabling build of sandbox environments to gather alignment signals with more scalability and akin to the real world (Park et al., 2023). Through the simulated social interactions such as moral discussions, studies offer promising avenues for enhancing AI systems’ alignment with human values and ethical principles.

For instance, Stable Alignment (Liu et al., 2023a) draws inspiration from how humans learn to navigate social norms and reach consensus on value judgments of societal issues through discussions. They introduce an LLM-based multi-agent framework to simulate social interactions within human society, highlighted by a three-tiered method comprising imitation, self-critic and realignment. Alignment datasets are extracted during the simulation process, correspondingly. Wang et al. (2024f) extend this method to multi-turn interactions under realistic social scenarios. They propose an interactive learning method to teach language agents to reach goals through social interactions among characters playing different social roles. Building upon the same methodology of simulated social interactions, other research works explore the intricacies of moral discussions (Sun et al., 2023a), political debates (Taubenfeld et al., 2024) and task-oriented dialogue (Ulmer et al., 2024) wherein AI systems are guided by some predefined discussion schema.

6.2 Human Collective Intelligence

Another way compensate to simulated social interactions involves resorting to crowd-sourcing effort from human. However, distinct from previous crowd-sourcing annotations in traditional NLP or RLHF data collection, alignment signals are collected in more primitive forms to achieve broader scalability of supervision and oversight. These signals are often termed as norms (Ammanabrolu et al., 2022), Rules of Thumb (RoTs) (Forbes et al., 2020; Ziems et al., 2022), or Constitutions (Bai et al., 2022b) in different contexts. Through such process, consensus on what basic principles AI systems should follow could be achieved among groups of people, and collective intelligence can be reached by synthesizing various human ideas on values into a target for aligning models (Klingefjord et al., 2024).

Toward this goal, current works are carried out to mainly solve two aspects of research questions. One of the questions is how to extract alignment signals from human values. For this question, Constitutional AI (CAI) (Bai et al., 2022b) provides a basic attempt to convert the AI Constitutions into supervision and preference datapoints. In their setting, human supervision relies entirely on a set of principles governing AI behavior, called Constitutions. A small number of examples are used for few-shot prompting LLMs to create SFT data or preference feedback according to those constitutional principles.

The other question is how to elicit values from human public more effectively. For example, in a program titled Democratic inputs to AI, an open call for the design of prototype systems to promote the democratic process of AI was issued. Specifically, it addresses enabling a representative cross-section of people to exchange views and ultimately reach a consensus on the rules AI systems should follow. Subsequently, in the Collective Constitutional AI (CCAI) project (Anthropic, 2023), researchers design questionnaire to ask participants to draft a constitution for AI system. Through the way of collective constitution drafting, participants can assess the existing constitutional principles, rate their acceptance scores, and propose new ones they believe AI should adhere to.

6.3 Tool Execution Feedback

Tools are pivotal in expanding the capabilities of large language models, allowing them to transcend the limitations beyond their basic capabilities and interact more effectively with their surroundings. Additionally, accurate and elaborate feedback from tool execution provides direct signals for LLMs to validate and enhance their initial outputs (Chen et al., 2023d; Gou et al., 2024), which helps to reduce the reliance on human feedback. Furthermore, the process of learning to plan and use tools and can be refined from tools execution feedback (Wang et al., 2024a), where LLMs receive feedback on their actions and learn from both successes and failures in an interactive way.

Tools’ execution feedback can serve as additional signals beyond human-labor corpus to teach models better learn tasks. Code generation tasks provide such examples. CodeRL (Le et al., 2022) conducts unit tests by code compiler to receive feedback signals and use them to train a critic model, with which the code generation model is further trained using deep reinforcement learning. Self-debugging (Chen et al., 2023d) designs an interactive code debugging pipeline by adding a code explanation stage beside the code execution feedback, where large language models are asked to debug its own generated code given these feedback information. Similarly, SelfEvolve (Jiang et al., 2023) receives error message from interpreter and refine the answer code based on this feedback.

Beyond code generation, there are also studies designing unified frameworks to align language models with the execution feedback from multiple other sources. CRITIC (Gou et al., 2024) utilizes a verify-then-correct process to obtain external feedback from various external tools, including search engines, code interpreters and text APIs, thus enabling language models correcting the outputs with tool-interactive critiquing. Taking a step further, Wang et al. (2024a) augment LLMs with a dynamic memory mechanism, which allows the LLMs to progressively learn how to accurately use tools. Qiao et al. (2024) propose Reinforcement Learning with Execution Feedback to reinforce LLMs with execution results of tools.

6.4 Embodied Environment

Embodied intelligence involves agents that perceive signals and take actions within some physical environments, requiring not only language understanding but also abilities of reasoning and decision making in a space with large number of states (Roy et al., 2021). Leveraging large language models in embodied AI settings represents a compelling endeavor which can bring benefits from two aspects. On one hand, the powerful generalization ability of large language model can be utilized to facilitate action plan in natural language without preliminary learning. On the other hand, commonsense knowledge and physical understanding ability of large language models can be aligned with feedback signals from environment.

Recent studies show the potential of LLMs to be deployed in robotics for real-world interactions while also pose challenges for grounding LLMs to understand the physical world (Wang et al., 2023a; Ahn et al., 2022). To better align language models which have not been pre-tarined on those embodied environment, Xiang et al. (2023) adopt an embodied simulator as world model to provide feedback signals when LLMs interact with objects and execute actions in this environment with or without a task goal. Then the explored world states would be collected as embodied experiences to fine-tune the LLM offline subsequently. In parallel, other studies utilize online reinforcement learning (Carta et al., 2023; Tan et al., 2024) to learn or iterative offline learning framework (Song et al., 2024b) to update policy for embodied agent through trial and error in the interactive environments.

6.5 Discussion

In this section, we reviewed some current studies that can be uniformly observed from the perspective of aligning through environment feedback. While most of the works target at specific downstream applications, actively learning from environment has always been a key pursue in artificial intelligence. Despite their valuable contributions, current methods still show limitations and leave open problems for future research. The main limitation arises from the gap between simulated environments and the real world.

Most of the current works derive alignment signals from simulated environments, while feedback from the real environment can be noisy or ambiguous. For example, in aligning through social interaction, Liu et al. (2023a) point out that current studies usually assume a static view of social norms, overlooking their dynamic and evolving natures. In fact, social values are complex, context-dependent, and sometimes contradictory. Encoding these values into AI systems in a comprehensive and coherent manner is a significant challenge. Thus, how to effectively bridge these changeable and divergent social values is of critical importance in alignment with human society.

In addition, simulated environments can be oversimplified and cannot fully catch the complexity of real-world, which challenges the robustness and generalization of aligned models. As for the embodied environment track of works, research are often conducted within sandbox environments and different settings are mutually independent. The action spaces of these simulated environments are still limited (Tan et al., 2024) compared to the real environment with potentially infinite degree of freedom. As a result, models trained in one setting might not generalize well to others, and ensuring models maintain alignment across different environments remains a problem.

In conclusion, the topic of integrating environmental feedback into AI alignment open up a promising and emerging track, while significant research problems remain to be explored. Future research can focus on bridging the gap between simulated and real environments to develop more adaptive, reliable, and ethically aligned AI.

7 Underlying Mechanism of Automated Alignment

As discussed in the previous sections, substantial research endeavors to enhance the efficiency, effectiveness, and reliability of automated alignment methods. Despite these efforts, there remains a conspicuous dearth of systematic investigation into the mechanisms underlying alignment. For example, numerous methods (Liu et al., 2024c; Li et al., 2024d; etc.) are proposed to filter instruction data, however, it remains unclear whether specific filtering criteria or processes are necessary and the rationale behind them. Likewise, many works propose various weak-to-strong alignment algorithms, but we still do not know the reasons behind the success of weak-to-strong generalization and whether it depends on specific conditions. The absence of relevant research can impede comprehension of current alignment and automated alignment methods, thereby hindering further optimization of these methods.

Therefore, in this section, we will conduct a systematic investigation into the mechanisms of automated alignment. By systematically organizing and analyzing the underlying mechanisms of automated alignment, we can identify the shortcomings and limitations of current automated alignment methods and illuminate directions for designing improved approaches. As previously discussed, there are an enormous amount of techniques of automated alignment. In this survey, we select the following three core research questions, which are crucial for achieving scalable automated alignment:

• RQ1: What is the underlying mechanism of current alignment? The underlying mechanism of alignment is fundamental to the study of automated alignment. It is crucial for understanding the feasibility, boundaries, and optimization directions of automated alignment.

• RQ2: Why does Self-feedback work? Self-feedback is widely applied across various paradigms of automated alignment, such as serving as an inductive bias in section 3.1.2, constructing data for preference learning in section 5 and acting as a crucial technique for automated iterative alignment (Yuan et al., 2024; etc.). Understanding why does it work is vital for comprehending and enhancing all paradigms involving it.

• RQ3: Why is weak-to-strong feasible? As a promising direction for achieving scalable oversight, it is essential to comprehend the feasibility and underlying mechanisms of weak-to-strong for optimizing and designing more effective weak-to-strong methods, especially when aligning superhuman models.

For each research question, we summarize existing studies and perspectives and discuss the limitations of these analytical works as well as the areas that still require exploration.

7.1 What is the underlying mechanism of current alignment?

Understanding the mechanism underlying current alignment is essential for assessing the potential for automated alignment, examining key challenges faced by automated alignment and steering the optimization directions of current automated alignment methods. Previous works (Zhou et al., 2023a; Ren et al., 2024; Mecklenburg et al., 2024; etc.) mainly focus on the two aspects of alignment’s underlying mechanism: behavioral norms transfer and knowledge learning. Analyses and discussions are launched around which aspect is more crucial.

Some studies discover that the primary role of current alignment is the transform of behavioral norms rather than the learning of additional world knowledge. Zhou et al. (2023a) propose the “Superficial Alignment Hypothesis”, that a model’s knowledge and capabilities are learned almost entirely during pretraining, while alignment teaches it which subdistribution of formats should be used when interacting with users. Moreover, subsequent research employ three distinct analytical methodologies to delve deeper into model behavior during the alignment process.

Feature-based Analysis

By comparing the probability distribution of LLM’s predicted tokens before and after alignment (DPO&RLHF), Lin et al. (2024a) find that the distribution shift mainly occurs on stylistic tokens, while the knowledge-intensive tokens show a small distribution shift, and the distribution shift decreases as the predictions become longer, indicating that alignment is about formal alignment rather than knowledge injection. Meanwhile, Duan et al. (2023) find that ICL and IFT exhibit high convergence in the hidden states of LLM and low convergence with the hidden layer states of base LLM, suggesting that the role of alignment lies in the behavioral norms shift of the model from write continuation to response. Additionally, through employing gradient-based input-output attribution methods and analyzing attention heads and feed-forward layers, Wu et al. (2023a) reveal that IFT enables LLMs to recognize user instructional components and tailor responses accordingly, without altering linguistic structure.

Knowledge Intervention

Ren et al. (2024) introduce a knowledge intervention framework to decouple the potential underlying factors of IFT and find that for IFT, there is little, if not even causing additional damage, benefits from the learning of world knowledge incongruent with parameter knowledge across all homogeneous, in-domain, and out-of-domain evaluations. Moreover, Ren et al. (2024) find that the essence of effective IFT is to maintain the consistency of model parameter knowledge before and after IFT while completing the transform of behavioral norms. Gekhman et al. (2024) also highlight the risks of adding new facts by observing the performance of the model in introducing different proportions of new knowledge through IFT.

Empirical Assessment

LIMA (Zhou et al., 2023a), AlpaGasus (Chen et al., 2023c) and LTD (Chen et al., 2023b) provide experimental support for “Superficial Alignment Hypothesis” by achieving impressive performance under IFT using few instruction data. Besides, Gudibande et al. (2023) show that alignment through behavior imitation can successfully improve LLM’s style, role, and ability to follow instructions, but cannot improve LLM’s performance on more complex dimensions, such as factuality and problem-solving.

While these studies reach similar conclusions, they fail to delineate the specific alignment achieved by the models under scrutiny. The potential differences in the underlying mechanisms of alignment across varying degrees or requirements remain unexplored. Furthermore, current research primarily focuses on traditional NLP tasks and general conversational scenarios. However, some studies (Mecklenburg et al., 2024; Singhal et al., 2023; etc.) find that domain-specific alignment can indeed improve model performance on the corresponding domain, which suggest that alignment can play a crucial role in learning additional domain knowledge. The underlying mechanisms of alignment in various scenarios such as coding and mathematics remain an open question.

7.2 Why does self-feedback work?

Feedback capability refers to the capacity to provide information or guidance to a given input based on specific standards. As we discussed above, such capability is extensively utilized in various paradigms of automated alignment. For example, in RLAIF, LLM itself substitutes for human assessment to construct preference data (Yuan et al., 2024; etc.). Additionally, LLM can continuously optimize its output based on self-feedback, a process known as self-refine (Bai et al., 2022b; Madaan et al., 2023; Tan et al., 2023; etc.). Nonetheless, there is much debate about whether and why LLMs can provide effective feedback to their own responses. In the following, we systematically summarize each representative viewpoint.

Li et al. (2024f) and Lin et al. (2024b) suggest that models possess certain knowledge that cannot be directly utilized for generation but can be employed for providing feedback. Li et al. (2024f) also discover significant Generator-Validator inconsistency in LLMs when generating and validating answers. Similarly, Lin et al. (2024b) find that LLMs possess substantial knowledge that cannot be expressed through generation and correction but can be utilized for critique. Furthermore, other studies (Liu et al., 2024a; Li et al., 2024e) argue that feedback capability is a byproduct of the model’s ability to follow instructions. Therefore, during the alignment process, the model’s feedback capacity improves alongside its ability to follow instructions, a phenomenon supported by experimental evidence (Liu et al., 2024a; Li et al., 2024e). However, some other studies posit that the feedback capability of LLM is illusory, suggesting that it may rely on specific data and exhibit biases. West et al. (2023) note that while current LLMs gain generative capabilities to replicate expert-level outputs through training, they lack understanding capacities relevant to critique. Moreover, Huang et al. (2023b) observe that for reasoning tasks, LLMs even experience performance degradation after self-correction at times because that they cannot properly judge the correctness of their reasoning. Additionally, Zheng et al. (2024b) find that although powerful LLMs like GPT-4 achieve high agreement with human evaluators, similar to human-human agreement levels, LLM-based evaluation face various challenges such as position bias (Zheng et al., 2024b; Wang et al., 2023c), verbosity bias (Zheng et al., 2024b; Wu and Aji, 2023), self-enhancement bias (Zheng et al., 2024b; Liu et al., 2023b), as well as limited capability in certain scenarios such as grading math, reasoning questions (Zheng et al., 2024b), high-quality summaries (Shen et al., 2023a) and so on (Lan et al., 2024). Researchers propose many methods to further improve the capability of self-feedback, such as meta critic (Sun et al., 2024a), autocalibrate (Liu et al., 2023d), split and merge (Li et al., 2023e) and using external tools (such as search engines, code interpreters, etc.) to cross-check (Gou et al., 2024), thus refining their initial response, etc.

Discussion While extensive work has been conducted on exploring models’ capability to provide self-feedback, questions regarding its effectiveness boundaries and underlying reasons remain unanswered. Additionally, the disparities and rationales between model self-feedback and human expectations remain unexplored. Moreover, promising results have been demonstrated in the two-step optimization model outputs (Bai et al., 2022b; Madaan et al., 2023; Tan et al., 2023) of self-feedback and correction. However, studies mainly focus on the former, leaving the latter largely unexplored. Although researchers (Lan et al., 2024; Gou et al., 2024) find that LLMs possess the ability, akin to humans, to modify their response based on feedback, their capacity to correct based on the feedback they generate remains uncharted. Furthermore, the overall performance enhancement of LLMs depends on whether the number of refining incorrect responses to correct ones outweighs the reverse. A comprehensive analysis of when self-refine can enhance or impair performance, as well as the underlying reasons for such effects, is still lacking.

7.3 Why is weak-to-strong feasible?

As we discussed above, a promising approach towards scalable oversight is the concept of “weak-to-strong”, which involves cultivating robust abilities through limited or simplified supervision. While there are some successful practical works for “weak-to-strong”, the underlying mechanisms of “weak-to-strong” remain further investigation, which limits further optimization and method design for “weak-to-strong”. Currently, a common perspective is that LLMs pre-trained on a large corpus can leverage their impressive generalization abilities to achieve robust capabilities even under limited or simplified supervision. In the following, we will introduce how such generalization capability facilitates LLMs to achieve automated alignment via limited or simplified supervision.

Bai et al. (2022b), Sun et al. (2023d) and Fränken et al. (2024) observe that merely supplying core alignment principles to LLMs enables them to automatically achieve significant alignment effects. This illustrates the models’ capability to generalize from principle to behavior. For example, Bai et al. (2022b) prompt LLM to refine and select optimal responses based on provided principles to complete IFT and RLAIF. Sun et al. (2023d) utilize 16 manually designed rules to guide the base model to generate high-quality instructions followed by self-distillation to achieve self-alignment. Chen et al. (2024e) further employ a stronger LLM to automatically discover these constitutions. Similarly, Fränken et al. (2024) align a stronger base model using constitutions by a weaker instruction-fined model. Burns et al. (2023) find that simple methods can often significantly improve weak-to-strong generalization of LLM: for instance, when finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence loss, GPT-4 can achieve close to GPT-3.5-level performance in NLP tasks. This demonstrates the models’ capacity to generalize from limited supervision to stronger performance. Additionally, Sun et al. (2024b) and Hase et al. (2024) find LLMs trained on simple tasks can successfully generalize to hard tasks. Recent research also discuss the feasibility of weak-to-strong generalization under theoretical frameworks (Somerstep et al., 2024; Lang et al., 2024; Charikar et al., 2024).

Discussion Current LLMs demonstrate strong generalization capabilities. However, the fundamental reasons behind the significant differences in generalization capabilities between current LLMs and earlier pre-trained LMs remain to be explored. Moreover, further exploration is necessary to delineate the boundaries of generalization, i.e., what can and cannot be generalized, as well as to understand the underlying reasons, which is essential for identifying key challenges and determining future research directions of alignment and automated alignment.

8 Conclusions

This survey explores various techniques for scalable automated alignment, categorizing them into four primary domains: aligning with inductive bias, behavior imitation, model feedback, and environment feedback. The existing research illustrates multiple pathways toward achieving automated alignment, primarily addressing critical challenges such as scalable oversight. Despite these advancements, we notice significant research gaps when we investigate the mechanism of current alignment, particularly concerning the reliability of self-feedback and the feasibility of weak-to-strong generalization. Addressing these underexplored questions is essential for advancing automated alignment, enabling the safe and effective application of large language models in real-world scenarios. Future research endeavors are expected to bridge these gaps, ensuring that LLMs operate reliably and align with intended human values.