Intuitive Fine-Tuning:
Towards Simplifying Alignment into a Single Process

Abstract

Supervised Fine-Tuning (SFT) and Preference Optimization (PO) are two fundamental processes for enhancing the capabilities of Language Models (LMs) post pre-training, aligning them better with human preferences. Although SFT advances in training efficiency, PO delivers better alignment, thus they are often combined. However, common practices simply apply them sequentially without integrating their optimization objectives, ignoring the opportunities to bridge their paradigm gap and take the strengths from both. To obtain a unified understanding, we interpret SFT and PO with two sub-processes — Preference Estimation and Transition Optimization — defined at token level within the Markov Decision Process (MDP) framework. This modeling shows that SFT is only a specialized case of PO with inferior estimation and optimization. PO evaluates the quality of model’s entire generated answer, whereas SFT only scores predicted tokens based on preceding tokens from target answers. Therefore, SFT overestimates the ability of model, leading to inferior optimization. Building on this view, we introduce Intuitive Fine-Tuning (IFT) to integrate SFT and Preference Optimization into a single process. IFT captures LMs’ intuitive sense of the entire answers through a temporal residual connection, but it solely relies on a single policy and the same volume of non-preference-labeled data as SFT. Our experiments show that IFT performs comparably or even superiorly to sequential recipes of SFT and some typical Preference Optimization methods across several tasks, particularly those requires generation, reasoning, and fact-following abilities. An explainable Frozen Lake game further validates the effectiveness of IFT for getting competitive policy. Our code will be released at https://github.com/TsinghuaC3I/Intuitive-Fine-Tuning.

1 Introduction

Large Language Models (LLMs) have demonstrated remarkable powerful potential across various downstream tasks after pre-training on large-scale corpora brown2020language ; achiam2023gpt ; zhou2024generative . However, their instruction-following skills and trustworthiness still fall short of expectations bender2021dangers ; bommasani2021opportunities ; li2022trust . Therefore, algorithms such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) ziegler2019fine ; ouyang2022training ; lee2023rlaif are used to further enhance LLMs’ abilities and align them better with human preferences.

Considering the limited effectiveness of SFT and the high cost of data construction and training computation for RLHF, these two methods are often combined to leverage their respective strengths. Unfortunately, they are typically implemented as a sequential recipe constrained by the paradigm gap between SFT and early RLHF methods, stemming from differences in loss functions, data formats, and the requirement for auxiliary models.

Figure 1: Comparison of Alignment Methods. IFT conducts alignment solely relying on positive samples and a single policy, starting from a pre-trained base model. IFT not only shows high efficiency as SFT, but also performs comparably or even superiorly to other alignment algorithms.

Recently, Direct Preference Optimization (DPO) rafailov2024direct was proposed to integrate Reward Modeling and Policy Optimization into one single procedure using a loss function derived from Proximal Policy Optimization (PPO) schulman2017proximal . This approach demonstrates the potential to unify SFT and RLHF for the first time. Henceforth, many extended methods have been tried to realize this objective by bridging the gap between SFT and DPO. Some of them ethayarajh2024kto ; hong2024reference ; zhang2024negative aim to transform the contrastive loss of DPO into a SFT-like cross-entropy loss, learning positive samples similar to SFT while unlearning negative samples resort to Unlikelihood Training welleck2019neural . Some others get rid of the preference-labeling process before training, switching to collect samples and labels/rewards in an online manner liu2023statistical ; yuan2024self ; guo2024direct ; calandriello2024human ; tajwar2024preference , or just treating the SFT targets and online policy generations as positive and negative samples respectively xiong2023iterative ; chen2024self ; mitra2024orca ; liu2024extensive . Nevertheless, preference-labeled pairwise data is still essential in these methods, and the need for reference model only becomes unnecessary in some cases. Thus the core differences between SFT and Preference Optimization are not eliminated thoroughly. To address this challenging issue, a deeper and more unified understanding of them are needed.

In this paper, we attempt to explain the similarities and differences between SFT and Preference Optimization by defining Preference Estimation and Transition Optimization in terms of state-action pairs within the Markov Decision Process (MDP) framework. Through this modeling, we demonstrate that SFT is simply a specialized case with inferior estimation and optimization among all Preference Optimization methods. To be specific, typical Preference Optimization methods usually sample the entire answer of policy based solely on the initial instruction, and align the policy preference reflected by the sampled sentence with human preferences. In contrast, SFT only samples a single token based on intermediate state of target answer, which leads to a biased estimation of policy preference and an inferior alignment shortcut. Or put it in a metaphorical view, when preparing for an exam, it is obviously more beneficial to review reference answers after completing entire questions, rather than trying to infer each subsequent step based on the reference answer ahead and checking it immediately. It is the similar case in LMs’ training, where SFT is less beneficial for enhancing LMs.

Depending on this understanding, we introduce a unified alignment algorithm named Intuitive Fine-Tuning (IFT), which builds policy preference estimation in an intuitive manner like human, who can have a vague sense of the complete answer after hearing a question. Through a closer estimation to truly policy preference than SFT, IFT achieves a comparable or even superior alignment performance compared to the sequential recipe of SFT and Preference Optimization. Additionally, IFT requires only a single policy model, and the same volume and format of data as SFT, enjoying both data and computation efficiency. These characteristics also enable IFT to be implemented in domains where preference data is unavailable or expensive to collect.

Our main contribution are three folds: 1) By modeling within the MDP framework, we explain the similarities and differences between SFT and two fundamental Preference Optimization methods, PPO and DPO; 2) Based on this modeling, we introduce Intuitive Fine-tuning (IFT), a deeply unified version of SFT and Preference Optimization. IFT utilizes temporary residual connections to extract the model’s generation preference given the initial instructions, offering similar efficiency to SFT while achieving performance closer to Preference Optimization; 3) Through experiments on several benchmarks, we validate that IFT performs comparably or even superiorly to various SFT recipes and existing Preference Optimization methods. In particular, IFT demonstrates significantly stronger reasoning and fact-following abilities than the chosen baselines. An explainable toy-setting Frozen Lake further demonstrates the effectiveness of IFT.

2 Related Work

Classical Reinforcement learning (RL) has demonstrated strong performance in various sequential decision-making and optimal control domains, including robotics levine2018learning , computer games vinyals2019grandmaster and others guan2021direct . There are two main categories of RL algorithms: value-based and policy-based, depending on whether they learn a parameterized policy. Value-based RL aims to fit an value function defined by Bellman Equation, containing methods such as Monte-Carlo (MC) Learning lazaric2007reinforcement and Temporal Difference Learning sutton1988learning ; seijen2014true . However, value-based methods struggle in continuous or large discrete space for its greedy objective. Thus, policy-based methods were introduced to model the decision-making process using a parameterized policy. As one of its best-known algorithms, Proximal Policy Optimization (PPO) schulman2017proximal is widely used in various domains, including Natural Language Processing (NLP).

Alignment for LMs has emerged as a crucial task these years, which adjusts the LMs’ generation distribution in line with human preferences. Building on the Bradley-Terry (BT) bradley1952rank Model, current alignment methods typically train models to distinguish positive and negative answers given the same instruction using RLHF/RLAIF ziegler2019fine ; ouyang2022training ; lee2023rlaif . While PPO remains the primary algorithm for alignment, its high demands for computation and memory hinders its broader use. Consequently, many improved methods have been proposed dong2023raft ; yuan2023rrhf ; zhao2023slic . Among them, DPO rafailov2024direct unifies reward modeling and policy optimization by utilizing a loss function derived from PPO, training a single model to serve as both a policy model and a reward model. Without sacrificing performance, DPO decrease the costly consumption of PPO through directly value iteration similar to a preference-based format of MC instead of TD. Nevertheless, the expensive preference-labeling process is still required. Additionally, a warm-up stage based on SFT is also necessary, which may bring trade-offs in fitting different objectives between SFT and Preference Optimization.

Improved Versions of DPO come out one after another. Efforts such as liu2023makes ; khaki2024rs ; yin2024relative ; guo2024controllable ; bansal2024comparing ; liu2024extensive try to enhance the contrastive learning by utilizing better ranking strategies, more informative data, or more number of negative samples. except for using offline data, liu2023statistical ; yuan2024self ; guo2024direct ; calandriello2024human ; chen2024self ; mitra2024orca focus on online sampling and automated label/reward collection, reducing the manual cost required for alignment. Recently, methods like ethayarajh2024kto ; hong2024reference aim to reduce DPO’s dependency on SFT warm-up by transforming its loss functions and data format into a SFT manner. These algorithms handle positive and negative samples using SFT objective and Unlikelihood Training welleck2019neural , respectively. However, the actual volume of training data is not decreased in these methods. Also, GPU-memory-consuming pair-wise data is still required, while the need for a reference model and preference-labeling for the entire answer trajectory is only eliminated in limited cases.

3 Preliminaries

3.1 MDP in Language Models

The MDP applied to LMs can be formally described as a tuple $ℳ=(𝒮,𝒜,𝒯,r,{\rho }_0)$, where $𝒮$ is the state space comprising ordered permutations of vocabularies, $𝒜$ is the action space consisting of vocabularies defined by the tokenizer, $𝒯$ is the transition matrix indicating token generation probabilities for given states, $r$ represents rewards for state-action pairs, and ${\rho }_0$ is the initial state typically based on given instructions. See more details in Appendix A.1.

The primary objective of Language Modeling is to train a policy ${\pi }_{\theta }$ with ${𝒯}_{\theta }$ to mimic a human policy ${\pi }^{\ast }$ with ${𝒯}^{\ast }$, aiming for the two transition matrices to become identical:

$$\forall s\in 𝒮,a\in 𝒜:{𝒯}_{\theta }(a|s)\to {𝒯}^{\ast }(a|s)$$ (1)

This process can also be expressed using another state-state transition matrix $T$:

$$\forall s,s^{\prime }\in S:T_{\theta }(s^{\prime }|s)\to T^{\ast }(s^{\prime }|s)$$ (2)

where $T$ is equivalence to $𝒯$, but instead, indicating the transition probability between states.

3.2 Preference Estimation

We define the preference $𝒫$ of policy $\pi$ given an initial instruction ${\rho }_0$ as a mapping:

$$𝒫({\rho }_0):{\rho }_0\to [\pi ({\rho }_0),\pi (s_1),\pi (s_2),\mathrm{\dots }]$$ (3)

where $s_{i+1}=[s_i,a_i]$ , $a_i=\pi (s_i)$ and $s_0={\rho }_0$.

During alignment, the model preference gradually approaches the human preference:

(4)

As the truly preferences are difficult to obtain, alignment is usually conducted based on the Preference Estimation of model and human, denoted as $\widehat{{𝒫}_{\theta }}$ and $\widehat{{𝒫}^{\ast }}$ respectively. The estimations from some common methods are listed in Table 1.

To make preference optimizable, the policy’s preference can also be expressed as follows:

$$𝒫({\rho }_0)=\{𝒯(a|s)|\forall a\in 𝒜,s\in {𝒮}_{{\rho }_0}\}$$ (5)

Here, ${𝒮}_{{\rho }_0}$ denotes a conditional state space that constrained by the initial state ${\rho }_0$, within which each state can only be initially derived from ${\rho }_0$. Consequently, the model preference can be optimized through transition matrix, named Transition Optimization.

3.3 Transition Optimization

Ideally, we want to align the state-action transition matrix between model and human in a ${\rho }_0$-constrained state space:

$$\forall a\in 𝒜,s\in {𝒮}_{{\rho }_0}:{𝒯}_{\theta }(a,s)\to {𝒯}^{\ast }(a,s)$$ (6)

which is equivalent to the following format expressed by state-state transition matrix:

$$\forall s\in {𝒮}_{{\rho }_0}:T_{\theta }(s,{\rho }_0)\to T^{\ast }(s,{\rho }_0)$$ (7)

However, considering the limited data, only matrix elements representing state-action/state-state pairs contained in the dataset $𝒟$ would be aligned. Given a data sample with instruction ${\rho }_0$ and target answer with length-$N$, the objective would be:

$$\forall a\in 𝒜,n\in [0,N],{\rho }_0\in 𝒟,s_n^{\ast }\in {𝒮}_{{\rho }_0}^{\ast }:{𝒯}_{\theta }(a,s_n^{\ast })\to {𝒯}^{\ast }(a,s_n^{\ast })$$ (8)

Or equivalent to:

$$\forall n\in [0,N],{\rho }_0\in 𝒟,s_n^{\ast }\in {𝒮}_{{\rho }_0}^{\ast }:T_{\theta }(s_n^{\ast },{\rho }_0)\to T^{\ast }(s_n^{\ast },{\rho }_0)$$ (9)

$$T_{\theta }(s_n^{\ast }|{\rho }_0)=\prod _{i=0}^n{𝒯}_{\theta }(a_i^{\ast }|s_i^{\ast }),T^{\ast }(s_n^{\ast }|{\rho }_0)=\prod _{i=0}^n{𝒯}^{\ast }(a_i^{\ast }|s_i^{\ast })$$ (10)

where $s_0^{\ast }={\rho }_0$, $T^{\ast }({\rho }_0|{\rho }_0)=T_{\theta }({\rho }_0|{\rho }_0)=1$, and $s_i^{\ast }$ denotes the intermediate state of target answer.

Consequently, the loss function can be derived from the disparities of the transition matrices between model and human. Some typical loss function are listed in Appendix A.4

Table 1: Reformulation of SFT, PPO and DPO

Method Preference Estimation Transition Optimization $\widehat{s_n^{\ast }}$ in $\widehat{{𝒫}^{\ast }}$ $\widehat{s_n^{\theta }}$ in $\widehat{{𝒫}_{\theta }}$ Truly $s_n^{\ast }$ $s_n^{\theta }$ $T_{\theta }(s_n^{\ast },{\rho }_0)\to T^{\ast }(s_n^{\ast },{\rho }_0)$ SFT $s_n^{\ast }$ $s_n^{\ast }$ $T_{\theta }(s_n^{\ast },s_{n-1}^{\ast })\to T^{\ast }(s_n^{\ast },s_{n-1}^{\ast })$ PPO $s_n^{\theta }$ $s_n^{\theta }$ $T_{\theta }(\widehat{s_n^{\ast }},s_{n-1}^{\theta })\to T^{\ast }(\widehat{s_n^{\ast }},s_{n-1}^{\theta })$ DPO online $s_n^{\ast }$ $s_n^{\theta }$ $T_{\theta }(s_n^{\ast },{\rho }_0)\to T^{\ast }(s_n^{\ast },{\rho }_0)$ offline $s_n^{\ast }$ $s_n^{{\theta }^{-}}$ $\widehat{T_{\theta }}(s_n^{\ast },{\rho }_0)\to T^{\ast }(s_n^{\ast },{\rho }_0)$

3.4 From SFT to Preference Optimization

We reformulate SFT, PPO and DPO using the aforementioned framework, detailed in Table 1 and Appendix A.4. A more comprehensible version is presented in Figure 2. To compare the differences between them, we begin by introducing a fundamental theorem and corollary:

Theorem Given a set of events $𝒵$, the probability of any event $z\in 𝒵$ is between 0 and 1, i.e., $\forall z\in 𝒵:0\le P(z)\le 1$. If all events are mutually independent, the sum of their probabilities equals 1, i.e., $1={\sum }_{z\in 𝒵}P(z)$. The event $z^{\ast }$ with the highest probability has a probability greater than or equal to any other event, i.e., $\forall z\in 𝒵:0\le P(z)\le P(z^{\ast })\le 1$.

Corollary LMs consistently assign higher probabilities to their own greedy predictions than to human preference:

$$\forall s\in S:{𝒯}_{\theta }({\pi }^{\ast }(s),s)\le {𝒯}_{\theta }({\pi }_{\theta }(s),s)\le 1$$ (11)

thus LMs tend to assign higher probabilities to its own generation than to target answer given the same initial instruction:

$$\forall n\in [0,N],s_n^{\ast }\in {𝒮}_{{\rho }_0}^{\ast },s_n^{\theta }\in {𝒮}_{{\rho }_0}^{\theta }:T_{\theta }(s_n^{\ast },{\rho }_0)\le T_{\theta }(s_n^{\theta },{\rho }_0)\le 1$$ (12)

where $N$ represents the length when the generation reaches the EOS token or the truncation length.

SFT provides an unbiased estimation of human preference, but a biased estimation for model:

$$\widehat{{𝒫}_{\theta }}({\rho }_0):{\rho }_0\to [{\pi }_{\theta }({\rho }_0),{\pi }_{\theta }(s_1^{\ast }),{\pi }_{\theta }(s_2^{\ast }),\mathrm{\dots }]$$ (13)

which is caused by wrong prior state when predicting each subsequent token. Consequently, the Transition Optimization objective of SFT:

$$T_{\theta }(s_n^{\ast },s_{n-1}^{\ast })\to T^{\ast }(s_n^{\ast },s_{n-1}^{\ast })$$ (14)

secretly sets $T_{\theta }(s_{n-1}^{\ast },{\rho }_0)=1$ during aligning $T_{\theta }(s_n^{\ast },{\rho }_0)$ with $T^{\ast }(s_n^{\ast },{\rho }_0)$. This makes an overestimation of the transition probabilities and preference of model, leading to an inferior optimization progress in SFT. Thus Preference Optimization is needed for further preference alignment.

PPO shows an unbiased estimation of model preference, while employing a progressively unbiased estimation of human preference:

$$\widehat{{𝒫}^{\ast }}({\rho }_0):{\rho }_0\to [{\pi }^{\ast }({\rho }_0),{\pi }^{\ast }(s_1^{\theta }),{\pi }^{\ast }(s_2^{\theta }),\mathrm{\dots }]$$ (15)

Initially biased, this estimation gradually becomes unbiased as the model aligns with human preference over time. As $T_{\theta }(s_n^{\theta },{\rho }_0)$ is consistently closer to 1 than $T_{\theta }(s_{n-1}^{\ast },{\rho }_0)$, PPO provides an closer approximation than SFT to the actual circumstances of model in Transition Optimization:

$$T_{\theta }(\widehat{s_n^{\ast }},s_{n-1})\to T^{\ast }(\widehat{s_n^{\ast }},s_{n-1})$$ (16)

which sets $T_{\theta }(s_n^{\theta },{\rho }_0)=1$ and $\widehat{s_n^{\ast }}={\pi }^{\ast }(s_{n-1}^{\theta })$. However, estimating ${\pi }^{\ast }(s_{n-1}^{\theta })$ is at the expense of preference-labeling, reward modeling and online sampling.

DPO theoretically achieves the best estimation across all scenarios, even without reward modeling. However, obtaining pairwise preference data online is costly, as it requires real-time negative sampling from model and preference labeling by human. Thus, mainstream implementations often rely on off-policy negative samples out-of-distribution from the optimized model, which may yield unstable and sub-optimal results due to biased preference estimation and inferior transition optimization.

Figure 2: The Training Paradigm of Different Methods. Symbol $\ast$ and $\theta$ denote human and model respectively, with $a_i^{\ast }={\pi }^{\ast }(s_i^{\ast })$ and $s_{i+1}^{\ast }=[s_i^{\ast },a_i^{\ast }]$, similarly for $\theta$. SFT uses priors deviating from model distribution, resulting in a more biased estimation of model preferences compared to PPO and DPO. IFT achieves a closer estimation than SFT by utilizing temporary residual connections, while maintaining the data and computational efficiency as SFT.

4 Method

While SFT is data and computation-efficient, it has an inferior approximation for both Preference Estimation and Transition Optimization. In the other side, Preference Optimization (represented by PPO and DPO) enjoys better approximation at the expense of preference data construction. We hope to make good use of their strength, using solely the target data as SFT but having similar approximation as Preference Optimization.

4.1 Intuitive Preference Estimation

A key distinction between SFT and Preference Optimization lies in whether the full distribution of model preference for each initial instruction is sampled. Contrasted to Preference Optimization, the intermediate state of the target answer used for prior in SFT may be far away from the model preference, leading to inferior outcomes.

To obtain a state estimation $\widehat{s_i^{\theta }}$ closer to model preference, we introduce a model-based distribution disturbance function ${\delta }_{\theta }$ for the biased state:

$$\widehat{s_i^{\theta }}={\delta }_{\theta }(s_i^{\ast })=(1-\lambda )s_i^{\ast }+\lambda {\pi }_{\theta }(s_{i-1}^{\ast })$$ (17)

which can also be interpreted as a temporal residual connection. Through this approach, model can predict not only the next token from intermediate state of target answer, but also develop an intuitive sense to the entire answer generation solely based on the initial instruction, deriving more accurate Preference Estimation for model:

$$\widehat{{𝒫}_{\theta }}({\rho }_0)=[(1-\lambda )𝒫_{\theta }^{}{}_{}{}^{sft}+\lambda 𝒫_{\theta }^{}{}_{}{}^{truly}]({\rho }_0)$$ (18)

4.2 Dynamic Relation Propagation

With improved Preference Estimation, we achieve a Transition Optimization process closer to the original objective:

$$\forall n\in [0,N],{\rho }_0\in 𝒟,s_n^{\ast }\in {𝒮}_{{\rho }_0}^{\ast }:\widehat{T_{\theta }}(s_n^{\ast },{\rho }_0)\to T^{\ast }(s_n^{\ast },{\rho }_0)$$ (19)

where $s_0^{\ast }={\rho }_0$ and $\widehat{T_{\theta }}(s_n^{\ast },{\rho }_0)=\prod _{i=0}^{n-1}{T}_{\theta }(s_{i+1}^{\ast },\widehat{s_i^{\theta }})$. This objective can be optimized by the following loss function that quantifies the disparities of transition between model and human:

$$ℒ(\widehat{T_{\theta }};T^{\ast })={𝔼}_{{\rho }_0\sim 𝒟}{𝔼}_{s_n^{\ast }\sim {𝒮}_{{\rho }_0}^{\ast }}\left[\sum _{n=0}^N\log \frac{\widehat{T_{\theta }}(s_n^{\ast },{\rho }_0)}{T^{\ast }(s_n^{\ast },{\rho }_0)}\right]$$ (20)

We make the same hypothesis as SFT that the optimization objective of each target intermediate state has a probability equal to 1:

$$\forall n\in [0,N],{\rho }_0\in 𝒟,s_n^{\ast }\in {𝒮}_{{\rho }_0}^{\ast }:T^{\ast }(s_n^{\ast },{\rho }_0)=1=T^{\ast }(s_N^{\ast },{\rho }_0)$$ (21)

Then, the loss function can be reformulated as:

$${ℒ}_{\text{IFT}}({𝒯}_{\theta },{\delta }_{\theta })={𝔼}_{{\rho }_0\sim 𝒟}{𝔼}_{s_i^{\ast }\sim {𝒮}_{{\rho }_0}^{\ast }}\left[-\sum _{n=0}^N\sum _{i=n}^N\log {𝒯}_{\theta }(a_i^{\ast },{\delta }_{\theta }(s_i^{\ast }))\right]$$ (22)

where $a_i^{\ast }={\pi }^{\ast }(s_i^{\ast })$. This reformulation facilitates parallel implementation, allowing IFT to achieve computational efficiency similar to SFT. See Appendix A.2 for complete derivation.

At the same time, we also prove that the objective optimized by this loss function implicitly satisfies the Bellman Equation for each state:

$$V_{\theta }(\widehat{s_n^{\theta }})=\exp \left(-ℒ\left(\widehat{T_{\theta }}(s_n^{\ast },{\rho }_0)\right)\right)$$ (23)

The derivation is in Appendix A.3. This proof guarantees the optimization process closer to RLHF. Additionally, it ensures the optimized objective not only reflects the prediction accuracy of current token, but also considers the influence of the current selection on future generation, helping model acquire an intuitive understanding of generation, and a better causality for reasoning and fact-following. Additionally, a decay factor can be incorporated, as in the typical Bellman Equation, to ensure effectiveness in long trajectories.

5 Experiments

We conduct experiments mainly on NLP setting. Considering the absence of an optimal policy of human language generation, we also utilize the Frozen Lake environment for further validation.

5.1 Settings for NLP

Datasets. We select UltraChat-200k ding2023enhancing and UltraFeedback-60k cui2023ultrafeedback as single-target and pair-wise dataset respectively.

Models. We conduct experiments on Mistral-7B-v0.1 jiang2023mistral and Mistral-7B-sft-beta tunstall2023zephyr , with the former as the base model and the latter fine-tuned on UltraChat-200k.

Scenarios. We consider two different training scenarios, one using Preference Optimization exclusively, and the other employing sequential recipe of SFT and Preference Optimization. In the first scenario, alignment is conducted directly from base model Mistral-7B-v0.1 using UltraFeedback. In order to ensure balanced data volume between different method, we randomly sample 60k data from UltraChat as supplementary for SFT and IFT, for only the target data are utilized in these two methods. The second scenario is commonly seen, where SFT and Preference Optimization is employed sequentially. For this scenario, we use Mistral-7B-sft-beta as start-point, which has been fine-tuned with UltraChat using SFT. Then we fine-tune it further with UltraFeedback using Preference Optimization.

Baselines. Our main baselines are SFT and DPO rafailov2024direct , with PPO excluded due to computational limitations. Additionally, we incorporate an improved versions of DPO, named ORPO hong2024reference , who claims to achieve alignment directly without SFT and reference model, aligning well with our goals. In addition to reproducing the algorithms mentioned above, we also consider Zephyr-7B-beta tunstall2023zephyr and Mistral-ORPO-alpha hong2024reference , two open-source checkpoints that utilize sequential and direct recipes respectively. Both of them used start-point models and datasets similar to ours.

Benchmarks. We consider two types of benchmarks. One is from the widely used Open-LLM LeaderBoard, which contains ARC-Challenge (25-shot) clark2018think , MMLU (5-shot) chung2024scaling , TruthfulQA (0-shot) lin2021truthfulqa , WinoGrande (5-shot) sakaguchi2021winogrande , and GSM8K (5-shot) cobbe2021training . The other is LM-based evaluation, including Alpaca-Eval and Alpaca-Eval-2 dubois2024alpacafarm . We utilize chat template for all benchmarks to obtain a more accurate evaluation for chat models.

5.2 Main Results in NLP Tasks

Effectiveness on Sequential Recipe. In this scenario, IFT demonstrates good performance across benchmarks having standard answers or not, see more details in Table 2 and 3. On Open-LLM Leaderboard, IFT showcases the best average capabilities across all tasks, excelling particularly in tasks requiring generation, reasoning and fact-following abilities, such as TruthfulQA and GSM8K. However, IFT has a relatively large gap between DPO in multi-choice tasks like ARC-Challenge and MMLU. We will explained the reasons later. When evaluated for instruction-following and question-answering abilities as judged by GPT-4, IFT’s performance is comparable to that of the chosen baselines. Remarkably, IFT achieves these results using the least amount of data and computational resources among all the methods tested.

Table 2: Evaluation on LLM-based Benchmarks. In the typical sequential setting, SFT+DPO still achieves the highest scores, although IFT performs comparably. However, when aligning directly from the base model, IFT performs not only superiorly to other methods but also on par with sequential recipes. As for open-source models, we directly present its publicly available score.

Method Reference Data Alpaca-Eval Alpaca-Eval-2 pairwise volume win-rate lc win-rate win-rate lc win-rate Mistral-7B – – – 24.72 11.57 1.25 0.35   fine-tuning with UltraFeedback-60k + SFT ✗ ✗ 120k 82.56 78.32 7.09 8.67 + DPO ✓ ✓ 120k 74.00 73.12 9.73 8.58 + ORPO ✗ ✓ 120k 85.14 76.60 8.82 12.34 + IFT ✗ ✗ 120k 85.18 78.78 9.95 13.27 Mistral-ORPO-$\alpha$ ✗ ✓ 120k 87.92 – – 11.33   fine-tuning with UltraChat-200k + UltraFeedback-60k sequentially + SFT ✗ ✗ 200k 86.69 77.96 4.08 6.43 + SFT + SFT ✗ ✗ 260k 86.34 76.98 4.55 7.14 + SFT + DPO ✓ ✓ 320k 91.62 81.54 10.08 13.72 + SFT + ORPO ✗ ✓ 320k 86.26 79.67 7.40 12.27 + SFT + IFT ✗ ✗ 260k 88.37 81.29 10.26 14.34 Zephyr-7B-$\beta$ ✓ ✓ 320k 90.60 – – 10.99

Table 3: Evaluation on Open-LLM Leaderboard with chat template. When fine-tuning with the same recipe, IFT achieves the highest average score across all methods. Directly conducting alignment using IFT showcases the best performance in all recipes with the least data and computation.

Method ARC ARC-Gen MMLU TruthfulQA WinoGrande GSM8K Avg. Mistral-7B 53.07 73.04 59.14 45.29 77.58 38.89 54.79   fine-tuning with UltraFeedback-60k + SFT 56.49 74.00 60.44 55.57 77.90 42.84 58.65 + DPO 61.86 73.54 61.02 47.98 76.64 43.89 58.28 + ORPO 56.66 73.98 60.57 51.77 77.19 42.30 57.70 + IFT 56.74 74.15 60.49 57.65 78.45 44.73 59.61 Mistral-ORPO-$\alpha$ 57.25 73.72 58.74 60.59 73.72 46.78 59.41   fine-tuning with Ultrachat-200k + UltraFeedback-60k sequentially + SFT 57.68 72.87 58.25 45.78 77.19 40.94 55.97 + SFT + SFT 58.10 72.61 58.40 48.59 76.80 43.06 56.99 + SFT + DPO 63.91 73.98 59.75 46.39 76.06 41.47 57.52 + SFT + ORPO 58.45 73.21 58.80 50.31 76.45 42.76 57.35 + SFT + IFT 58.36 73.38 58.45 52.39 78.06 43.82 58.22 Zephyr-7B-$\beta$ 67.41 72.61 58.74 53.37 74.11 33.89 57.50

Effectiveness of Preference Optimization Alone. IFT not only maintains the performance advantages compared with other baselines in this setting, as seen in the sequential scenario. But also, IFT performs comparably or even superiorly to many method in sequential recipe. While DPO tends to fail under this setting, ORPO remains competitive in its open-source model. However, when constrained in the same experiment setting, the performance of ORPO becomes worse than IFT. Additionally, the reliance on preference data makes ORPO more costly in terms of negative sampling, preference labeling, and GPU memory consumption. Consequently, IFT stands out as a more efficient and cost-effective alternative in this context.

Multi-Choice vs. Generation. IFT performs better on generation tasks but struggles with multi-choice, whereas DPO exhibits the opposite performance. This may due to differences in evaluation metrics and training objectives zheng2023large ; plaut2024softmax ; tsvilodub2024predictions . Multi-choice tasks evaluate log-likelihood for entire answers, while generation tasks require token-by-token construction for causality and reasoning. DPO aligns the mapping between instructions and complete answers, while IFT emphasizes token-level causal relationships. As a result, DPO tends to excel in multi-choice tasks, while IFT performs better in token-by-token exploration tasks. In an ARC-Challenge adaptation to generation tasks, IFT demonstrates superiority without changing the benchmark’s distribution. Overall, IFT showcases its balanced performance across diverse tasks and achieving the highest average score.

Objective Trade-off between SFT and Preference Optimization. Traditional Preference Optimization methods deliver excellent alignment performance, particularly in enhancing the instruction-following ability of language models, as showed in Table 2. However, fitting the different objectives of SFT and Preference Optimization involves trade-offs tunstall2023zephyr . Even slight overfitting on SFT may result in reduced effectiveness of Preference Optimization. This phenomenon is also observed in Table 3, where the models trained by sequential recipe of SFT and other Preference Optimization methods showcase obvious inferior results on Open-LLM Leaderboard even worse than SFT alone. Avoiding this trade-off, ORPO and IFT can achieve better and more stable performance by directly conducting alignment on the base model.

Efficiency and Scaling Potential of IFT. Although IFT achieves comparable or superior performance to other methods, it also boasts high efficiency in many aspects. Like SFT and ORPO, IFT does not require a reference model, which conserves GPU memory and computational resources. Most importantly, IFT and SFT are the only methods that conduct alignment without preference data, offering significant benefits as follows. Firstly, this characteristic eliminates the need for synchronous storage and computation of pairwise data on the GPU, thereby reducing memory consumption and training duration. Secondly, negative sampling from models and human preference-labeling are no longer necessary, eliminating the highest cost associated with alignment, which has been a discarded but fundamental challenge in research so far. Furthermore, using only the target answer brings the potential for scaling in alignment process, mirroring the core benefits found in pre-training.

5.3 Further Validation in Frozen-Lake Environment

Figure 3: The Frozen Lake Game. Considering the MSE distance of parameters between the trained and optimal policy, IFT performs much better than SFT and ORPO, but slightly worse than DPO.

As scores on Open-LLM Leaderboard only partially reflect models’ performance, and GPT-4 inadequately models human language generation, further comparison to a truly optimal policy is necessary. Given the difficulty of obtaining an optimal policy representing human language, we validate our algorithm in a simplified setting called Frozen Lake frozenlake . In this environment, an agent attempts to find a gift on a nearly frozen lake with several holes, terminating the game upon finding the gift or falling into a hole. The limited number of states and actions in this game allows the optimal policy to be easily derived using classical RL methods.

To simulate parameterized policy alignment, we employ a two-layer fully connected neural network and design the environment with one optimal and one sub-optimal trajectory. The optimal parameterized policy is trained using the previously obtained optimal state-action probabilities, and various fine-tuning methods from LMs are compared. We evaluate performance by measuring the MSE distance between the optimal and trained policy parameters.

In this setting, IFT achieves a significantly better policy than SFT and ORPO, although it performs slightly worse than DPO. This is partly because, in terms of comparing how closely the explored grid aligns with the agent’s preference, the order is DPO > IFT > ORPO > SFT. Although ORPO also considers the negative trajectories sampled from policy, its direct incorporation of SFT loss with a fusion coefficient deviates its preference estimation, partially diminishing its effectiveness. Additionally, DPO, ORPO and IFT explore more grids than SFT, which helps the agent develop a better understanding of the environment.

6 Conclusion

In this paper, we first interpret SFT and some typical Preference Optimization methods into a unified framework using Preference Estimation and Transition Optimization. We then introduce an efficient and effective method called Intuitive Fine-Tuning (IFT), which achieves alignment directly from the base model using non-preference-labeled data. Finally, experiments on widely used benchmarks in NLP setting and in Frozen Lake environment demonstrate the competitive performance of IFT.

Limitations and Future Work. Our validation of IFT is limited to the fine-tuning stage, where data volume is constrained, leaving the scalability of IFT unexplored. Additionally, we primarily use Mistral-7B for baseline testing, and the generalization of IFT to larger and more diverse models requires exploration in the future.