Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

Abstract

Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our results show that SPIN can significantly improve the LLM’s performance across a variety of benchmarks and even outperform models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents. Codes are available at https://github.com/uclaml/SPIN.

1 Introduction

Large Language Models (LLMs) have began a groundbreaking era in artificial general intelligence (AGI), demonstrating extraordinary capabilities across a wide range of domains that require intricate reasoning and specialized knowledge. These models excel in areas such as mathematical reasoning/problem solving (Cobbe et al., 2021; Wei et al., 2022; Lewkowycz et al., 2022), code generation/programming (Chen et al., 2021; Austin et al., 2021; Li et al., 2022), text generation (Bubeck et al., 2023; Anil et al., 2023; Touvron et al., 2023), summarization and creative writing, among others. A significant advancement in LLMs is the post-pre-training alignment with the more desirable behaviors (Mishra et al., 2021; Victor et al., 2022; Chung et al., 2022; Thoppilan et al., 2022), a process often reliant on the costly human-annotated data. Typical alignment methods include Supervised Fine-Tuning (SFT) (Ouyang et al., 2022; Tunstall et al., 2023a) based on human demonstrations, and Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020; Bai et al., 2022a) based on human preferences.

All the aforementioned alignment methods require a substantial volume of human annotated data. Therefore, there is increasing interest in developing fine-tuning methods that can effectively utilize human data, thereby streamlining the alignment process. This motivates us to study fine-tuning LLMs without the need for additional human-annotated data beyond the fine-tuning dataset. Our study is also related to the broader goal of converting weak models to strong models without the requirement for extra training data, which is of central interest in machine learning that can be traced back to the boosting algorithms (Kearns & Valiant, 1994; Schapire, 1990; Freund, 1995; Freund & Schapire, 1997). The self-training algorithm (Vapnik, 1999; Grandvalet & Bengio, 2004; Lee, 2013) has also been proved to be able to convert weak learners to strong learners in mixture models without the need for additional labeled data (Frei et al., 2022; Kou et al., 2022). However, the pursuit of autonomously enhancing a weak LLM without external guidance is both intriguing and understudied. This raises the following question:

Can we empower a weak LLM to improve itself without acquiring additional human annotated data?

In this paper, we answer this question affirmatively. Inspired by the success of self-play mechanisms (Samuel, 2000) in games, exemplified by AlphaGo Zero (Silver et al., 2017b), AlphaZero (Silver et al., 2017a), with historical roots traced back to TD-Gammon (Tesauro et al., 1995), we propose to convert a weak LLM to a strong one through the lens of self-play, where the model is enhanced by playing against itself without requiring any direct supervision. In particular, we propose a novel fine-tuning method called Self-Play fIne-tuNing (SPIN), which begins from a supervised fine-tuned model. SPIN allows the LLM to engage in self-play, eliminating the need for an expert annotator such as a human or more advanced LLMs like GPT-4. In detail, with the LLM from previous iteration $t$ denoted by $p_{{𝜽}_t}$, we employ it to generate responses ${𝐲}^{\prime }$ to the prompts $𝐱$ in the human-annotated SFT dataset. The subsequent objective is to find a new LLM $p_{{𝜽}_{t+1}}$, capable of distinguishing the responses ${𝐲}^{\prime }$ generated by $p_{{𝜽}_t}$ from the responses $𝐲$ generated by humans. This process can be seen as a two-player game: the main player, or the new LLM $p_{{𝜽}_{t+1}}$, seeks to discern between the responses of the opponent player $p_{{𝜽}_t}$ and human-generated responses, while the opponent, or the old LLM $p_{{𝜽}_t}$, generates responses as similar as possible to those in the human-annotated SFT dataset. The new LLM $p_{{𝜽}_{t+1}}$ is obtained by fine-tuning the old one $p_{{𝜽}_t}$ to prefer responses from $p_{\mathrm{data}}$ over $p_{{𝜽}_t}$, resulting in a distribution $p_{{𝜽}_{t+1}}$ that is more aligned with $p_{\mathrm{data}}$. In the next iteration, the newly obtained LLM $p_{{𝜽}_{t+1}}$ becomes the opponent for response generation, with the self-play process aiming for the LLM to eventually converge to $p_{{𝜽}^{\ast }}=p_{\mathrm{data}}$, so that the strongest possible LLM can no longer differentiate the responses generated by its previous version and those generated by the human.

Interestingly, our method exhibits similarity with the recently introduced direct preference optimization (DPO) method (Rafailov et al., 2023), with the notable distinction being the self-play nature of our method. Consequently, our approach stands out by eliminating the need for extra human preference data, a requirement present in the DPO method. Additionally, the self-play mechanism in our method resembles the idea of generative adversarial networks (GAN) (Goodfellow et al., 2014; Arjovsky et al., 2017), albeit that both the discriminator (main player) and the generator (the opponent) in our method are instances of the same LLM from different iterations. Theoretically, we prove that our method converges when the distribution of the LLM is identical to the target data distribution, i.e., $p_{{𝜽}_t}=p_{\mathrm{data}}$. Our experimental results on zephyr-7b-sft-full (Tunstall et al., 2023a), a fine-tuned LLM based on Mistral-7B (Jiang et al., 2023), show that while continued training using SFT on its own SFT dataset Ultrachat200k (Ding et al., 2023) reaches a performance plateau or even diminished evaluation scores, our method consistently improves zephyr-7b-sft-full across successive iterations while leveraging only a $\mathrm{𝟓𝟎}$k subset of Ultrachat200k dataset. Ultimately, SPIN effectively improves the base model’s average score from $58.14$ to $\mathbf{63.16}$ on the HuggingFace Open LLM Leaderboard (Beeching et al., 2023) with remarkable $10\%$+ improvement in scores on GSM8k and TruthfulQA, and from $5.94$ to $\mathbf{6.78}$ on MT-Bench (Zheng et al., 2023). Notably, SPIN achieves results that are even comparable to models trained on additional $62$k preference dataset (Tunstall et al., 2023a) on Open LLM leaderboard and MT-Bench.

Concurrent to our work, Singh et al. (2023) proposed the use of synthetic data with binary feedback in self-training, reducing the reliance on human data. In contrast, our approach eliminates the need for additional binary feedback from humans or an extra reward model thanks to the self-play mechanism. Additionally, Burns et al. (2023) employed a weak LLM model as the guidance to train stronger LLMs in a fashion of weak-to-strong generation. Unlike Burns et al. (2023), which necessitates both a weak supervisor and a strong model, our SPIN operates effectively with a single LLM.

Notation. We use lowercase letters and lowercase boldface letters to denote scalars and vectors, respectively. We use $[N]$ to denote the index set $\{1,\mathrm{\dots },N\}$. In the function space, let $ℱ$ be the function class. The symbol $q_{\mathrm{data}}$ designates the target data distribution, while $p$ represents the conditional probability of LLM’s response (i.e., LLM policy).

2 Related Work

Self-Play. Self-play (Samuel, 1959; Tesauro et al., 1995), where the algorithm learns by playing against itself, has gained notable attention due to its effectiveness in multi-agent reinforcement learning (MARL). This method involves agents engaging in interactions with copies of themselves, enabling an increasing level of challenge and complexity within the learning environment. A fundamental work in the field of self-play is AlphaGo Zero (Silver et al., 2017b), which demonstrated exceptional performance against human players using a self-play learning scheme. Subsequent research has expanded upon the concept of self-play, exploring various adaptations and implementations (Anthony et al., 2017; Lanctot et al., 2017; Bansal et al., 2018; Hernandez-Leal et al., 2018; Muller et al., 2019; Vinyals et al., 2019). Our method takes the self-play approach akin to AlphaGo Zero, which can convert a weak model to a strong one without additional human-annotated data. While the effectiveness of self-play in MARL is well-established, to our knowledge, our work is the first to apply this approach to the enhancement of LLMs.

Synthetic Data for LLMs. In the context of supervised fine-tuning (SFT) of LLMs, human-crafted data has proven to be a remarkably effective source that enhances the performance of LLMs on tasks such as code generation (Roziere et al., 2023; Yang et al., 2023) and mathematical reasoning (Yuan et al., 2023; Luo et al., 2023). While human data typically exhibits high quality, acquiring sufficient amount of such data poses a challenge in cost. In light of this consideration, the use of synthetic data has become increasingly popular and considered as a proxy for human data. This approach primarily leverages advanced LLMs such as the GPT series (Radford et al., 2019; Brown et al., 2020; OpenAI, 2023) as the guidance to generate high-quality data (Josifoski et al., 2023; Taori et al., 2023; Chiang et al., 2023; Li et al., 2023). Recent research has also highlighted the rephrasing capability of LLMs in prompting for better LLM response (Deng et al., 2023; Prasad et al., 2023) as well as augmenting synthetic data for more effective SFT (Yu et al., 2023; Liu et al., 2023). In contrast to prior studies that utilized more advanced models for synthetic data generation when pre-training or fine-tuning a target model, our approach directly generates synthetic data from the target model itself.

3 Problem Setting and Preliminaries

We consider a Large Language Model (LLM) parameterized by $𝜽$ and denoted by $p_{𝜽}$. The model takes as input a sequence $𝐱=[x_1,\mathrm{\dots },x_n]$, commonly referred to as the prompt, to generate the corresponding response $𝐲=[y_1,\mathrm{\dots },y_m]$. The response $𝐲$ is therefore considered as a sample from the conditional probability distribution $p_{𝜽}(\cdot |𝐱)$. In LLMs, $x_i$ and $y_j$ represent individual tokens from a predetermined vocabulary within the sequences $𝐱$ and $𝐲$, respectively. The auto-regressive model $p_{𝜽}$ generates tokens sequentially for a given position, leveraging only the sequence of previously generated tokens. This model therefore constitutes a Markov process, where the conditional probability distribution $p_{𝜽}(𝐲|𝐱)$ can be expressed through a decomposition as follows:

where is null and for $j=2,\mathrm{\dots },m$. In the following, we review two major fine-tuning methods for LLMs: supervised fine-tuning and reinforcement learning (RL) fine-tuning.

3.1 Supervised Fine-Tuning

Supervised fine-tuning (SFT) is employed to tailor a pre-trained LLM to specific downstream tasks, leveraging relatively smaller dataset of labeled examples in comparison to the large-scale pre-training data (Ouyang et al., 2022; Yu et al., 2023). In this context, we consider a specific task where the prompts, denoted by $𝐱$, are derived from a specified distribution $q(\cdot )$. The notation $p_{\mathrm{data}}(\cdot |𝐱)$ then represents the probability distribution of the associated high-quality responses $𝐲$ from the training data. Consequently, SFT involves training the LLM to minimize the following negative log-likelihood loss associated with these distributions,

$L_{\mathrm{SFT}}(𝜽)=-{𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱)}\left[\log p_{𝜽}\left(𝐲|𝐱\right)\right].$ (3.1)

It should be noted that excluding $𝐱\sim q(\cdot )$ from the expectation term yields the typical cross-entropy loss, expressed as $-{𝔼}_{𝐲\sim p_{\mathrm{data}}(\cdot |𝐱)}[\log p_{𝜽}(𝐲|𝐱)]$. $L_{\mathrm{SFT}}(𝜽)$ attains its minimum when the model’s predictive distribution $p_{𝜽}(𝐲|𝐱)$ aligns perfectly with the distribution of the labeled high-quality responses $p_{\mathrm{data}}(𝐲|𝐱)$.

Consequently, the LLM after SFT is anticipated to generate responses that closely resemble those from $p_{\mathrm{data}}(𝐲|𝐱)$. This procedure is therefore expected to significantly enhance the model’s performance in generating appropriate responses for a specific task.

3.2 RL Fine-Tuning

RL fine-tuning (Christiano et al., 2017; Bai et al., 2022a; Gao et al., 2023a) offers another method for enhancing the specific capabilities of general-purpose pre-trained models. Typically, RL fine-tuning is employed subsequent to SFT to achieve improved alignment for LLMs (Tunstall et al., 2023a).

For a given sequence pair $(𝐱,𝐲)$, RL fine-tuning necessitates a deterministic reward function $r(𝐱,𝐲)$. The higher the reward $r(𝐱,𝐲)$, the better the response $𝐲$ is to the given prompt $𝐱$. The objective of the RL fine-tuning process is then to maximize the following objective function:

$L_{\mathrm{RL}}(𝜽)$ $={𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{𝜽}(\cdot |𝐱)}[r(𝐱,𝐲)]$

$-\lambda {𝔼}_{𝐱\sim q(\cdot )}\mathrm{KL}(p_{𝜽}(\cdot |𝐱)||p_{\mathrm{ref}}(\cdot |𝐱)),$

where the Kullback-Leibler (KL) regularization enforces the new model $p_{𝜽}$ to be close to the reference model $p_{\mathrm{ref}}$, and $\lambda >0$ is the regularization parameter to control the deviation of the new model $p_{𝜽}$ from the reference model $p_{\mathrm{ref}}$. In practice, the reference model $p_{\mathrm{ref}}$ is often initialized as the supervised fine-tuned model. The inclusion of KL regularization is vital for preventing excessive deviation from the reference model, which in turn reduces the risk of mode collapse.

Meanwhile, the primary challenge in RL fine-tuning lies in finding a good reward function. Typically, this function requires training on a preference dataset. The compilation of such a dataset demands significant resources, often involving comprehensive evaluations either by human annotators, i.e., reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Bai et al., 2022a) or strong AI agents, i.e., reinforcement learning from AI feedback (RLAIF) (Bai et al., 2022b).

4 Method

In this section, we introduce a new fine-tuning method for enhancing the performance of LLMs without relying on additional human or AI feedback. Consider a high-quality supervised fine-tuning (SFT) dataset $S_{\mathrm{SFT}}={\{(𝐱,𝐲)\}}_{i=1}^n$, which are sampled from the marginal distribution $q(𝐱)$ and $p_{\mathrm{data}}(𝐲|𝐱)$. Given a supervised fine-tuned LLM $p_{{𝜽}_0}$, further application of the SFT approach in (3.1) with $S_{\mathrm{SFT}}$ will be ineffective and potentially lead to worse performance. In addition, without human and/or AI feedback, it becomes infeasible to acquire a preference dataset for RL fine-tuning (e.g., RLHF and RLAIF). This hinders the application of RL fine-tuning techniques.

We evaluate $p_{{𝜽}_0}$ against $S_{\mathrm{SFT}}$, where $p_{{𝜽}_0}$ is the LLM achieved by SFT using (3.1). We notice a persistent quality gap between the groundtruth response $𝐲$ from $S_{\mathrm{SFT}}$ and the LLM-generated response ${𝐲}^{\prime }\sim p_{𝜽}(\cdot |𝐱)$ (refer to Figure 1). This disparity indicates that there is still room for improvement over $p_{{𝜽}_0}$. Therefore, we propose to use the synthetic data generated by the LLM to enhance LLM’s performance starting from $p_{{𝜽}_0}$ iteratively.

Figure 1: Example of ground truth completion compared to the fine-tuned model generation at iteration 0 and 1. We can observe that the model generation at iteration 0, although fluent, incorrectly quantifies transportation preferences with specific percentages that are potentially hallucinations. The model generation at iteration 1 provides a qualitative summary of the transportation forms at Southampton without specific percentage, aligning more closely with the ground truth while adding more details.

4.1 Self-Play Fine-Tuning (SPIN)

Let us consider a two-player game, where the main player’s objective is to distinguish the responses generated by the LLM and those generated by the human. Meanwhile, the opponent’s role is to generate responses that are indistinguishable from the human’s responses. The core of our method is the self-play mechanism, where both the main player and the opponent are the same LLM, but from different iterations. More specifically, the opponent is the old LLM from the previous iteration, and the main player is the new LLM to be learned in the current iteration.

In iteration $t+1$, the opponent is the LLM from the previous iteration, denoted by $p_{{𝜽}_t}$, which generates responses ${𝐲}^{\prime }$ for those prompts $𝐱$ in the SFT dataset according to $p_{{𝜽}_t}(\cdot |𝐱)$. Our method, therefore, consists of the following two steps at iteration $t+1$: $(1)$ training the main player, and $(2)$ updating the opponent player.

Training the Main Player. We begin with illustrating how we expect a main player is trained to distinguish LLM responses from human responses. Motivated by integral probability metric (IPM) (Müller, 1997), we formulate our objective function such that the main player $f_{t+1}$ maximizes the expected value gap between the target data distribution $p_{\mathrm{data}}$ and the opponent player’s distribution $p_{{𝜽}_t}$:

$f_{t+1}$ $=\underset{f\in {ℱ}_t}{argmax}𝔼\left[f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })\right],$ (4.1)

where the expectation is computed over the distributions $𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),{𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)$, and ${ℱ}_t$ is a sequence of highly expressive function classes that we will determine in later deduction. The subscript $t$ in ${ℱ}_t$ is due to that the function class is dependent on $p_{{𝜽}_t}$. Given such a $f_{t+1}$ and a response sequence $𝐲$ to the prompt $𝐱$, the value of $f_{t+1}(𝐱,𝐲)$ reflects the main player’s degree of belief that $𝐲$ originates from $p_{\mathrm{data}}$ rather than $p_{{𝜽}_t}$. Ideally, the main player $f_{t+1}$ should yield a high value when $𝐲\sim p_{\mathrm{data}}(\cdot |𝐱)$ and a low value when ${𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)$, where $p_{{𝜽}_t}$ is the opponent’s distribution. Instead of solving (4.1), we can also solve the following more general optimization problem,

$$f_{t+1}=\underset{f\in {ℱ}_t}{argmin}𝔼\left[\mathrm{\ell }\left(f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })\right)\right],$$ (4.2)

where the expectation is computed over the distribution $𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),y^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)$, and $\mathrm{\ell }(\cdot )$ is a loss function that is both monotonically decreasing and convex. For example, a linear loss function $\mathrm{\ell }(t)=-t$ reduces (4.2) to the minimization version of (4.1). However, the use of a linear loss function results in an unbounded objective value, which, during continuous training, leads to a negative infinite value of $f(𝐱,{𝐲}^{\prime })$ on the opponent player’s responses. Therefore, in our work, we choose the logistic loss function $\mathrm{\ell }(t):=\log (1+\exp (-t))$ for its non-negativity, smoothness, and exponentially decaying tail as $t\to \mathrm{\infty }$. Such a choice of loss function aids in preventing the excessive growth in the absolute value of $f$.

Updating the Opponent Player. Previously we have discussed the training of $f_{t+1}$ given the opponent player’s distribution $p_{{𝜽}_t}$. Now suppose we have optimized our main player $f_{t+1}$ that can distinguish $p_{\mathrm{data}}$ from $p_{{𝜽}_t}$, within a certain function class ${ℱ}_t$, we elaborate how we get parameter ${𝜽}_{t+1}$ of the opponent player. Specifically, when presented with two responses $𝐲$ and ${𝐲}^{\prime }$ to the same prompt $𝐱$, $f_{t+1}$ assesses the values $f_{t+1}(𝐱,𝐲)$ and $f_{t+1}(𝐱,{𝐲}^{\prime })$. It then infers that the response with the higher value is from the real data distribution $p_{\mathrm{data}}$ and the response with lower value is attributed to the LLM $p_{{𝜽}_t}$. Subsequently, the objective of the opponent player is to find a better LLM that generates responses indistinguishable from $p_{\mathrm{data}}$ for the main player. This is achieved by maximizing the expected value ${𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p(\cdot |𝐱)}[f_{t+1}(𝐱,𝐲)]$. In addition, to prevent excessive deviation of $p_{{𝜽}_{t+1}}$ from $p_{{𝜽}_t}$ and stabilize the self-play, we incorporate a Kullback-Leibler (KL) regularization term. Putting these together gives rise to the following optimization problem:

$\underset{p}{argmax}{𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p(\cdot |𝐱)}[f_{t+1}(𝐱,𝐲)]$

$-\lambda {𝔼}_{𝐱\sim q(\cdot )}\mathrm{KL}(p(\cdot |𝐱)||p_{{𝜽}_t}(\cdot |𝐱)),$ (4.3)

where $\lambda >0$ is the regularization parameter. Notably, (4.3) has a closed-form solution $\widehat{p}(\cdot |𝐱)$:

$\widehat{p}(𝐲|𝐱)\propto p_{{𝜽}_t}(𝐲|𝐱)\exp \left({\lambda }^{-1}{f}_{t+1}(𝐱,𝐲)\right).$ (4.4)

It is worth noting that $\widehat{p}(\cdot |𝐱)$ is not guaranteed to be belong to the LLM space $\{p_{𝜽}(\cdot |𝐱)|𝜽\in 𝚯\}$. Since we hope that the closed-form solution $\widehat{p}$ in the probability space can be realized by an LLM with parameter $𝜽$, i.e., $p_{𝜽}(𝐲|𝐱)=\widehat{p}(𝐲|𝐱)$, solving for $p_{𝜽}(𝐲|𝐱)\propto p_{{𝜽}_t}(𝐲|𝐱)\exp \left({\lambda }^{-1}{f}_{t+1}(𝐱,𝐲)\right)$ gives $f_{t+1}(𝐱,𝐲)=\lambda \cdot \log \frac{p_{𝜽}(\cdot |𝐱)}{p_{{𝜽}_{\mathrm{t}}}(\cdot |𝐱)}$. This suggests the following function class ${ℱ}_t$ for $f_{t+1}$:

${ℱ}_t=\left\{\lambda \cdot \log {\displaystyle \frac{p_{𝜽}(𝐲|𝐱)}{p_{{𝜽}_{\mathrm{t}}}(𝐲|𝐱)}}\right|𝜽\in 𝚯\},$ (4.5)

where $𝚯$ is the parameter space of LLMs being considered. Given the choice of ${ℱ}_t$ in (4.5), optimizing (4.2) gives $f_{t+1}$ parameterized by ${𝜽}_{t+1}$ in the following form:

$f_{t+1}(𝐱,𝐲)=\lambda \cdot \log {\displaystyle \frac{p_{{𝜽}_{t+1}}(𝐲|𝐱)}{p_{{𝜽}_{\mathrm{t}}}(𝐲|𝐱)}}.$ (4.6)

Substituting (4.6) into (4.4) yields $\widehat{p}(𝐲|𝐱)=p_{{𝜽}_{t+1}}(𝐲|𝐱)$. In other words, ${𝜽}_{t+1}$ learned from (4.2) is exactly the LLM parameter for our ideal opponent selection.

End-to-end Training Objective. We integrate the previously discussed two steps into a single end-to-end training objective with an update rule of ${𝜽}_{t+1}$. Specifically, plugging (4.5) into (4.2) arrives at the update rule ${𝜽}_{t+1}={argmin}_{𝜽\in 𝚯}{L}_{\text{SPIN}}(𝜽,{𝜽}_t)$, where $L_{\text{SPIN}}$ is the training objective defined as follows

$L_{\text{SPIN}}=𝔼\left[\mathrm{\ell }\left(\lambda \log {\displaystyle \frac{p_{𝜽}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}-\lambda \log {\displaystyle \frac{p_{𝜽}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}\right)\right],$ (4.7)

where the expectation is computed over the distribution $𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),{𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)$. We summarize the iterative self-play process of our method SPIN as follows,

$\mathrm{\dots }$ $\to \underset{\text{Opponent Player at }t}{\underbrace{p_{{𝜽}_t}(\cdot |𝐱)}}\to \underset{\text{Main Player at }t+1}{\underbrace{\lambda \cdot \log {\displaystyle \frac{p_{{𝜽}_{t+1}}(\cdot |𝐱)}{p_{{𝜽}_{\mathrm{t}}}(\cdot |𝐱)}}}}$

$\to \underset{\text{Opponent Player at }t+1}{\underbrace{p_{{𝜽}_{t+1}}(\cdot |𝐱)}}\to \mathrm{\dots }$

Namely, the opponent player chosen from the previous iteration $t$ is employed to train the main player at iteration $t+1$, resulting in the LLM parameterized by ${𝜽}_{t+1}$. Then we determine the next opponent player at iteration $t+1$ by directly copying the LLM parameter ${𝜽}_{t+1}$, which is then used in training the main player at iteration $t+2$. The detailed algorithm is presented in Algorithm 1.

Algorithm 1 Self-Play Fine-Tuning (SPIN)

  Input: ${\{({𝐱}_i,{𝐲}_i)\}}_{i\in [N]}$: SFT Dataset, $p_{{𝜽}_0}$: LLM with parameter ${𝜽}_0$, $T$: Number of iterations.

  for $t=0,\mathrm{\dots },T-1$ do

     for $i=1,\mathrm{\dots }N$ do

        Generate synthetic data ${𝐲}_i^{\prime }\sim p_{{𝜽}_t}(\cdot |{𝐱}_i)$.

     end for

     Update ${𝜽}_{t+1}={argmin}_{𝜽\in 𝚯}{\sum }_{i\in [N]}\mathrm{\ell }\left(\lambda \log \frac{p_{𝜽}({𝐲}_i|{𝐱}_i)}{p_{{𝜽}_t}({𝐲}_i|{𝐱}_i)}-\lambda \log \frac{p_{𝜽}({𝐲}_i^{\prime }|{𝐱}_i)}{p_{{𝜽}_t}({𝐲}_i^{\prime }|{𝐱}_i)}\right)$.

  end for

  Output: ${𝜽}_T$.

4.2 Comparison between SPIN and DPO

In Section 4.1, we propose Self-Play Fine-Tuning (SPIN) with an end-to-end training objective (4.7) for each iteration. (4.7) bears resemblance to direct preference optimization (DPO) (Rafailov et al., 2023) for RL fine-tuning. However, SPIN and DPO are fundamentally different. DPO is based on the Bradley-Terry (BT) model: $p({𝐲}_1\succ {𝐲}_2|𝐱)=\frac{\exp (r^{\ast }(𝐱,{𝐲}_1))}{\exp (r^{\ast }(𝐱,{𝐲}_1))+\exp (r^{\ast }(𝐱,{𝐲}_2))}$, and maximizes the log-likelihood of $p({𝐲}_1\succ {𝐲}_2|𝐱)$ by direct policy optimization without explicit reward estimation. In contrast, SPIN relies on maximizing the IPM to compete with an increasingly stronger version of itself. More detailed comparisons are highlighted as follows:

1. DPO does not inherently lead to iterative training. More specifically, DPO aims to match the preference probability $p({𝐲}_1\succ {𝐲}_2|𝐱)$ induced from its reward model with the data distribution $p_{\mathrm{data}}({𝐲}_1\succ {𝐲}_2|𝐱)$ in a single iteration. On the contrary, SPIN’s self-play mechanism naturally leads to an iterative training procedure. SPIN iteratively refines its generation distribution $p_{𝜽}(𝐲|𝐱)$ to match the target distribution $p_{\text{data}}(𝐲|𝐱)$ across iterations.

2. SPIN only requires the SFT dataset, represented by pairs $(𝐱,𝐲)$. In contrast, DPO necessitates a preference dataset, represented by $(𝐱,{𝐲}_w,{𝐲}_l)$, where ${𝐲}_w$ and ${𝐲}_l$ denote the winner (chosen) and loser (rejected) responses, respectively. Moreover, SPIN can be applied between SFT and RL fine-tuning.

3. In SPIN, we can choose different loss functions $\mathrm{\ell }$ which only need to be convex and decreasing (detailed later in Theorem 5.2), which includes correlation loss, hinge loss, and logistic loss. Only when $\mathrm{\ell }$ is chosen as the logistic loss would the training objective of SPIN become similar to that of DPO.

Recently, Xu et al. (2023) proposed to use iterative preference optimization with the Pairwise Cringe Loss (PCO), and generalized DPO to iterative DPO. Concurrent to our work, Yuan et al. (2024) further proposed a framework named “self-rewarding language models”, which leverages the LLM itself as the reward model to provide the preference feedback, and employs iterative DPO to train the LLM. Compared with Xu et al. (2023); Yuan et al. (2024), SPIN’s self-assessment is implicit, as no intermediate reward or preference feedback is required.

5 Theoretical Analysis

In this section, we provide a theoretical analysis for Algorithm 1 in Section 4. Under monotonicity and convexity assumption of the objective function $\mathrm{\ell }$, we show that the global optimum is obtained if and only if parameter ${𝜽}_t$ generates data distribution. We summarize our assumptions as follows:

Assumption 5.1.

The loss function $\mathrm{\ell }(t):ℝ\to ℝ$ is monotonically decreasing, i.e., $\forall t,{\mathrm{\ell }}^{\prime }(t)\le 0$ and satisfies . In addition, $\mathrm{\ell }(t)$ is a convex function.

Assumption 5.1 holds for a wide range of loss functions commonly used in machine learning, including correlation loss $\mathrm{\ell }(t)=1-t$, hinge loss $\mathrm{\ell }(t)=\max (0,1-t)$, exponential loss $\mathrm{\ell }(t)=\exp (-t)$ and logistic loss $\mathrm{\ell }(t)=\log (1+\exp (-t))$. Under Assumptions 5.1, we present the following theorem, which is pivotal in understanding the optimization dynamics of our method.

Theorem 5.2.

Under Assumption 5.1, suppose there exists $p_{𝜽}(\cdot |𝐱)=p_{\mathrm{data}}(\cdot |𝐱)$, then we have that

• (Sufficiency) If $p_{{𝜽}_t}(\cdot |𝐱)=p_{\mathrm{data}}(\cdot |𝐱)$, then ${𝜽}_t$ is the global minimum of $(\text{<a class="ltx\_ref ltx\_font\_italic" href="https://arxiv.org/html/2401.01335v3\#S4.E7" title="Equation 4.7 ‣ 4.1 Self-Play Fine-Tuning (SPIN) ‣ 4 Method ‣ Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models"><span class="ltx\_text ltx\_ref\_tag">4.7</span></a>})$ for any $\lambda \ge 0$.

• (Necessity) If $p_{{𝜽}_t}(\cdot |𝐱)\ne p_{\mathrm{data}}(\cdot |𝐱)$, there exists an appropriately chosen $\lambda$, such that ${𝜽}_t$ is not the global minimum of (4.7).

Remark 5.3.

Theorem 5.2 suggests that under certain conditions, the optimization process of our method naturally stops at the point $p_{𝜽}(\cdot |𝐱)=p_{\mathrm{data}}(\cdot |𝐱)$, implying the effectiveness of our approach in aligning the LLM’s distribution with the target data distribution. Moreover, Theorem 5.2 also indicates that the optimization process only stops when the global optimality is achieved, i.e., the LLM’s distribution aligns with the target data distribution.

For the logistic loss function $\mathrm{\ell }(t)=\log (1+\exp (-t))$, the following theorem gives a more precise characterization of the opponent player, enabling a better understanding of SPIN.

Theorem 5.4.

Consider the choice of logistic loss $\mathrm{\ell }(t)=\log (1+\exp (-t))$ in SPIN. Suppose that $p_{{𝜽}_t}(𝐲|𝐱){\left(p_{\mathrm{data}}(𝐲|𝐱)/p_{{𝜽}_t}(𝐲|𝐱)\right)}^{1/\lambda }$ lies in the LLM space $\{p_{𝜽}(𝐲|𝐱)|𝜽\in 𝚯\}$ and ${𝜽}_{t+1}$ is global minimum of $L_{\text{SPIN}}(𝜽,{𝜽}_t)$, then the opponent player at iteration $t+1$ satisfies

$p_{{𝜽}_{t+1}}(𝐲|𝐱)\propto p_{{𝜽}_t}(𝐲|𝐱){\left(p_{\mathrm{data}}(𝐲|𝐱)/p_{{𝜽}_t}(𝐲|𝐱)\right)}^{1/\lambda }.$

Remark 5.5.

According to Theorem 5.4, the model update from $p_{{𝜽}_t}(𝐲|𝐱)$ to $p_{{𝜽}_{t+1}}(𝐲|𝐱)$ tends to increase the probability $p_{{𝜽}_{t+1}}(𝐲|𝐱)$ when $p_{{𝜽}_t}(𝐲|𝐱)$ is less than $p_{\mathrm{data}}(𝐲|𝐱)$, and decrease it when $p_{{𝜽}_t}(𝐲|𝐱)$ is greater than $p_{\mathrm{data}}(𝐲|𝐱)$. Thus, Theorem 5.4 further confirms that our method’s optimization process naturally converges to the point where $p_{𝜽}(\cdot |𝐱)$ equals $p_{\mathrm{data}}(\cdot |𝐱)$. The update of the opponent player is controlled by ${\left(p_{\mathrm{data}}(𝐲|𝐱)/p_{{𝜽}_t}(𝐲|𝐱)\right)}^{1/\lambda }$, which is regulated by the factor $1/\lambda$. A smaller $\lambda$ results in a larger change of the opponent player, while a larger $\lambda$ leads to a smaller change. Therefore, as $p_{𝜽}(\cdot |𝐱)$ approaches $p_{\mathrm{data}}(\cdot |𝐱)$, increasing $\lambda$ enhances the stability of LLM training. This observation aligns with (4.3), where $\lambda$ is the regularization parameter of the KL regularization that is employed to control the deviation of the opponent player.

6 Experiments

This section provides a detailed empirical analysis of SPIN. Our findings highlight several key points: $(1)$ SPIN markedly enhances model performance across a wide range of evaluation benchmarks by breaking the limit of SFT; $(2)$ even without introducing new human annotated data, SPIN at iteration $0$ achieves performance on par to DPO training that utilizes even more data; $(3)$ iterative training is a necessary component in SPIN as it breaks the limit of multi-epoch training.

6.1 Experiment Setup

Model and Datasets. In this study, we adopt zephyr-7b-sft-full as our base model. This model derives from the pre-trained Mistral-7B (Jiang et al., 2023) and has been further fine-tuned on the SFT dataset Ultrachat200k by HuggingFace. Ultrachat200k represents a high-quality 200k subset of the larger UltraChat (Ding et al., 2023) corpus, which comprises approximately 1.4M dialogues produced using OpenAI’s Turbo APIs. From UltraChat200k, We randomly sample $50$k prompts and use the base model to generate the synthetic responses. We subsequently follow the optimization method described in Section 4.1 for further training. In multiple iterations, we leverage the synthetic data from the most recent iteration and add to the newly generated synthetic data, therefore resulting in a synthetic dataset size of $50$k at iteration $0$ and $100$k at iteration $1$, $2$ and $3$. At each iteration, we train our model for $2$ epochs.

Evaluation. We employed the widely used Huggingface Open LLM Leaderboard (Beeching et al., 2023) as our evaluation benchmark, using the same Language Model Evaluation Harness library (Gao et al., 2023b). This leaderboard encompasses 6 different datasets, each focusing on a a specific capability of LLMs. Collectively, these datasets provide a thorough assessment framework, evaluating LLMs on commonsense reasoning (Arc (Clark et al., 2018), HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2021)), multi-task language understanding (MMLU(Hendrycks et al., 2020)), human falsehood mimic (TruthfulQA (Lin et al., 2021)) and math problem solving (GSM8k (Cobbe et al., 2021)). We leave further implementation details to Appendix B with detailed evaluation setting adopted by both the leaderboard and our experiments.

6.2 SPIN Effectively Improves Benchmark Performance

Figure 2: The average score of SPIN at different iterations on the HuggingFace Open LLM leaderboard datasets. For “SFT”, we report the performance of our base model zephyr-7b-sft-full, which has been fine-tuned on the same dataset we use to generate synthetic data.

In Figure 2, we demonstrate the effectiveness of SPIN using HuggingFace Open LLM Leaderboard as the evaluation. In Figure 3, we compare the performance of our fine-tuned model by SPIN after iterations 0 to 3 with the base model zephyr-7b-sft-full on each task included in the leaderboard. Detailed performances are presented in Table 4 in Appendix B. We can observe that SPIN exhibits remarkable effectiveness in improving the model’s performance by further leveraging the SFT dataset, on which the base model has already been fully fine-tuned. At iteration $0$, where model responses are generated from zephyr-7b-sft-full, we observe an overall improvement of $2.66\%$ on the average score. The improvement is particularly significant on the TruthfulQA and GSM8k benchmarks, with improvement exceeding $5\%$ and $10\%$ respectively. At iteration $1$, we employ the LLM model from iteration $0$ to generate new responses for SPIN, adhering to the procedure outlined in Algorithm 1. This iteration yields further enhancements of $1.32\%$ on average, and especially significant on the Arc Challenge and TruthfulQA benchmarks. Subsequent iterations continue this trend of incremental improvement across various tasks. Meanwhile, the improvement at iteration $t+1$ is naturally smaller than that at iteration $t$. As the iterative training progresses, the degree of improvement gradually approaches zero, suggesting that the model has reached a limiting point in the last iteration.

Comparison with DPO. zephyr-7b-beta is a model derived from zephyr-7b-sft-full, trained with DPO on approximately $62$k preference data. This data, the UltraFeedback Binarized dataset (Cui et al., 2023), comprises both chosen and rejected completions evaluated by GPT-4. We note that, DPO requires either human input or advanced language model feedback to determine the preference, making data generation a rather expensive procedure. In contrast, our SPIN only requires the initial model itself. Moreover, unlike DPO which requires new data source, our method exclusively leverages the existing SFT dataset. In Figure 3, we show the performance comparison of SPIN at iterations 0 and 1 (employing $50$k SFT data) with DPO training, from the same SFT checkpoint. We can observe that, while DPO leverages more data from new sources, SPIN based on the existing SFT data can already achieve comparable average performance to DPO training at iteration 0. From iteration 1, SPIN even surpasses the performance of DPO on the leaderboard benchmark.

Figure 3: Performance comparison with DPO training across the six benchmark datasets. Self-play at iteration $0$ achieves comparable performance to DPO training with $62$k new data. At iteration $1$, self-play has already surpassed DPO training on the majority of datasets.

(a) Arc Challenge.

(b) TruthfulQA.

(c) Average.

Figure 4: The SPIN training dynamics of zephyr-7b-sft-full on the 50k synthetic data with regard to the number of training epochs during iteration 0. We can observe that iterative training is pivotal as training for more epochs during iteration 0 reaches a limit and cannot surpass iteration 1.

6.3 Ablation Studies

In this subsection, we examine the effect of synthetic dataset size and training epochs within an iteration. Our analysis demonstrates the effectiveness of the synthetic data used by SPIN compared to the SFT data, as well as the necessity of iterative training in SPIN. In Appendix B, we present assessment of SPIN on additional benchmark tasks.

Figure 5: The scaling effect of training size of SPIN compared to SFT on the average score of Open LLM Leaderboard. For SPIN, we consider training data of sizes $14$k, $26$k and $50$k where the larger dataset contains the smaller dataset. The starting point for SPIN (with x-axis 0) is the zephyr-7b-sft-full checkpoint, which has been fine-tuned on Ultrachat200k for 1 epoch. We report the model performance trained for 1 epoch with SPIN on the varying sizes of dataset. We additionally compare with SFT, where we fine-tune Mistral-7B on Ultrachat200k for 3 consecutive epochs and report the model performance at the first epoch as the starting point (with x-axis 0).

Training Size. We investigate the effect of varying training data size on the performance of SPIN. In Figure 5, we demonstrate the effect of training size for SPIN during iteration $0$ and additionally compare with SFT with the full original dataset. Specifically, for the SFT baseline, we fully fine-tune Mistral-7B on Ultrachat200k for three epochs and report first epoch performance as the starting point (with x-axis 0) in the figure for SFT. For SPIN, we report the zephyr-7b-sft-full checkpoint as the starting point, which has also been fine-tuned on Ultrachat200k for one epoch. We select the training size of SPIN at iteration 0 to be 14k, 26k, and 50k and generate the data accordingly, ensuring that the larger dataset encompasses the smaller dataset. The performance of SPIN was then evaluated after 1 epoch of self-play fine-tuning for each training size. We can observe that, while SPIN results in notable improvement with increasing training sizes, SFT on further epochs 2 and 3 fails to yield more than $1\%$ improvement. Additional results are deferred to Appendix B.

Iterative Training v.s. Training for More Epochs. We further study the training within iteration $0$ and compare with the performance achieved in iteration $1$, particularly contrasting the test performance obtained from extended training duration with that from next iteration. Figure 4 depicts the performance trajectory of the model trained using SPIN over multiple epochs at iteration 0. It is evident that the most substantial improvement occurs during the first two epochs, followed by only modest gains in subsequent epochs. Notably, SPIN exhibits robustness and stability; extending the training duration does not diminish performance but rather maintains a rather consistent level. Nevertheless, the observation suggests an inherent limitation to the performance achievable within a single iteration, thereby underscoring the necessity for iterative training. As shown by the test performance achieved at iteration 1 in the figures, extending the training in iteration 0 fails to reach the performance comparable to iteration 1.

7 Conclusion and Discussion

This paper introduces a novel fine-tuning method SPIN, to convert a weak LLM to a strong LLM by unleashing the full power of human-annotated data. Central to this method is a self-play mechanism, wherein a main player (the LLM) is fine-tuned to differentiate the responses of opponent player (the LLM from previous iteration) from the target data distribution, and the LLM is iteratively aligned with the target data distribution. Therefore, SPIN facilitates the LLM’s iterative self-evaluation and enhancement through self-play. In comparison to supervised fine-tuning and RL fine-tuning methods, SPIN enables the LLM to self-improve without additional human data or feedback from stronger LLMs. Empirical results demonstrate that SPIN significantly enhances LLM performance across diverse benchmarks, even outperforming models trained with additional human data or AI feedback.

Limitation and Future Work. Our theoretical results demonstrate that the optimization process of SPIN converges if and only if the LLM’s distribution aligns with $p_{\mathrm{data}}$. Therefore, our study focuses on a fixed target data distribution generated by humans, which inherently imposes a ceiling on the performance of fine-tuned LLM. Exploring the dynamically changing target data distribution is an important direction to overcome this limitation and elevate the LLM’s performance beyond this ceiling or even to a super-human level. Moreover, considering the resource demands of synthetic data generation, another promising avenue for further exploration is to reduce the volume of required synthetic data.

Impact Statement

This paper presents work whose goal is to advance the field of Large Language Models. We believe that our work contribute meaningfully to the field, specifically on leveraging synthetic data to enhance LLM without the requirement for human preference annotations. The synthetic data generated by SPIN may be used to further augment the training of various language models. Moreover, SPIN demonstrated a substantial improvement in LLMs’ capabilities, opening new avenues for their application in various downstream tasks. This advancement underscores the transformative potential of LLM fine-tuning in both technological and societal contexts.

Appendix A Further Related Work

Curriculum Learning. In deep learning, it has been observed that training models using data samples arranged in a strategically meaningful order can lead to improved performance compared to training on randomly shuffled data. This approach is commonly known as curriculum learning (Bengio et al., 2009; Soviany et al., 2022). Initial studies in curriculum learning introduced efficient algorithms that adhere to an ‘easy-to-hard’ progression (Spitkovsky et al., 2009; Kumar et al., 2010; Lee & Grauman, 2011; Zhang et al., 2015). In the field of Natural Language Processing (NLP), criteria such as sentence length and term frequency are commonly utilized (Cirik et al., 2016; Zhang et al., 2018; Liu et al., 2018). More recent developments include the application of curriculum learning algorithms in multi-modal learning (Liu et al., 2021; Wu et al., 2022). Our work shares a similar idea to curriculum learning, wherein the training data evolves iteratively—beginning with responses that are easy to distinguish from human-annotated data and gradually progressing to more challenging instances.

Generative Adversarial Networks. Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) represent a distinct class of generative models, characterized by their unique adversarial process. To enhance training stability and data quality, Mao et al. (2017) introduced the Least Squares GAN, employing a least squares loss function for the discriminator. A significant advancement in GANs involves the use of Integral Probability Metrics (IPM) (Müller, 1997), particularly highlighted in the development of Wasserstein GAN by Arjovsky et al. (2017). This model employs IPM in its loss design, enhancing training stability. Since then, IPMs have become crucial in GAN design (Mroueh & Sercu, 2017; Gulrajani et al., 2017), particularly in constraining the discriminator to a specific function class, thereby preventing it from overpowering the generator. Furthermore, Jolicoeur-Martineau (2018) generalized IPM-based GANs by introducing relativistic discriminator and proposed Relativistic GAN. It is worth noting that the objective function defined in our (4.2) is similar to Relativistic GAN (Jolicoeur-Martineau, 2018) and reduces to an IPM framework in Wasserstein GAN (Arjovsky et al., 2017) with a linear loss. However, our approach differs in both the choice of the function class and the training procedure. Inspired by GAN, Cheng et al. (2023) proposed an adversarial learning framework named Adversarial Preference Optimization (APO) that trains the LLM and a reward model in an adversarial game. Similarly related to the spirit of our method, Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) was proposed to train separate discriminator and policy networks in each iteration. In contrast to the above methods, SPIN relies on self-play where both the main player and the opponent player are the same LLM from two consecutive iterations.

Alignment with AI Feedback. The objective of alignment is to fine-tune LLMs to align with human intentions. In addition to using human demonstrations, AI feedback is emerging as a crucial component in the alignment process. Constitutional AI (Bai et al., 2022b) leveraged AI feedback to align language models through a combination of both supervised learning and reinforcement learning (RL) phases. In the RL phase, (Bai et al., 2022b) applied Reinforcement Learning from AI Feedback (RLAIF), training a reward model using AI-generated preferences, followed by RL using the reward. Lee et al. (2023) demonstrated that AI feedback can achieve comparable or superior performance to human feedback in RL fine-tuning. They also demonstrated that RLAIF can improve upon an SFT policy even when the LLM labeler has the same size as the policy. Saunders et al. (2022) studied the scaling properties of self-critique and introduced a framework for comparing the critique ability to generation and discrimination ability. Self-critique models employ the LLM itself to generate natural language critiques through behavioral cloning, assisting human evaluators.

Appendix B Experiments

B.1 Hyperparameters and Implementation Details

Table 1: Detailed information of HuggingFace Open LLM Leaderboard. For each evaluation dataset, we present the number of few-shot examples and metric adopted for evaluation.

Datasets	Arc	TruthfulQA	Winogrande	GSM8k	HellaSwag	MMLU
# few-shot	25	0	5	5	10	5
Metric	acc_norm	mc2	acc	acc	acc_norm	acc

We use the Alignment Handbook library (Tunstall et al., 2023b) as the codebase for our self-play fine-tuning method SPIN, which includes DeepSpeed ZeRO-3 (Rajbhandari et al., 2020) and FlashAttention-2 (Dao, 2023) to reduce training cost. We train our models with RMSProp (Hinton et al., 2012) optimizer with no weight decay for all iterations as commonly used in fine-tuning LLMs for alignment, with a global batch size of $64$, $10\%$ warmup steps and bfloat16 precision. We set the peak learning rate to be 5e-7 for iterations 0 and 1, and decay this peak learning rate to 1e-7 for iteration 2 and 3 as we are approaching the end of self-play fine-tuning. Lastly, we choose $\beta =0.1$ and max sequence length to be $2048$ tokens as in Tunstall et al. (2023b). We note that at the last iteration (iter-3) where the model is close to convergence, we increase the value of $\beta$ to $5.0$. We use the Accelerate library (Gugger et al., 2022) to generate our synthetic data using distributed inference with multiple GPUs with a global batch size of $64$. We consider the prompting template “### Instruction: {prompt}\n\n### Response: ” as commonly used in Taori et al. (2023). For Ultrachat200k containing multi-round conversations, we only sample the first round as our prompt and ground truth completion pairs.

B.2 Training Overhead

The cost overhead introduced by SPIN is mostly the generation of synthetic data from the LLM that we train. The cost of the fine-tuning process remains computationally equal to that of SFT and DPO. We report both the generation and training time for SPIN in Table 2 . Results were obtained using a machine with 8xA100 (80G) GPUs. For per 64 examples, the generation time and training time are 6.69s and 10s respectively.

Table 2: Generation and Training Times for Different Iterations

Iteration	Iter 0		Iter 1		Iter 2		Iter 3
Process	Generation	Training	Generation	Training	Generation	Training	Generation	Training
Time	1.45h	4.32h	1.45h	8.64h	1.45h	8.64h	1.45h	8.64h

It is evident that the generation time is dominated by the training time at each iteration. The estimated time in Table 2 is based on the fact that we generate 50k examples per iteration. Please note that the doubled training time from iter 1 to iter 3 is attributed to the utilization of a double-sized dataset (the combination of 50k synthetic data from the previous iteration and 50k synthetic data in the current iteration), as discussed in our Section 6.1.

B.3 Additional Experiment Result for SPIN+DPO

SPIN requires only the SFT data to improve over the traditional SFT stage and can sit between SFT and RL finetuning. Suppose additional preference data is provided, we can use the additional data to further improve the performance of the model after SPIN using RL fine-tuning.

Starting at SPIN iteration 3, we further train the model with DPO for two epochs on the $62$k preference data from the UltraFeedback Binarized dataset (Cui et al., 2023), which consists of both chosen and rejected responses evaluated by GPT-4. Detailed performances are presented in Table 3.

Table 3: Performance of SPIN + DPO based on zephyr-7b-sft-full across HuggingFace Open LLM Leaderboard datasets, compared with all baselines. We also denote the average improvement over last iteration in the Average column.

Model	Arc	TruthfulQA	Winogrande	GSM8k	HellaSwag	MMLU	Average
zephyr-7b-dpo-full	63.65	55.19	72.61	33.43	84.44	58.52	61.31
zephyr-7b-sft-full	60.41	43.73	74.19	26.76	82.85	60.92	58.14
SPIN iteration $0$	63.40	49.18	72.69	35.10	84.38	60.03	${60.80}_{(+2.66)}$
SPIN iteration $1$	65.19	55.17	72.30	35.78	84.96	59.34	${62.12}_{(+1.32)}$
SPIN iteration $2$	65.96	54.91	73.56	38.06	85.41	59.93	${62.97}_{(+0.85)}$
SPIN iteration $3$	65.87	54.90	73.72	38.97	85.54	59.99	${63.16}_{(+0.19)}$
SPIN iteration $3$ + DPO	66.47	60.07	78.06	37.98	86.17	59.68	${64.05}_{(+0.89)}$

We can observe that the checkpoint trained by SPIN can be further improved using DPO, yielding an enhancement of $0.89\%$ on average. Notably, the improvement is particularly significant on the TruthfulQA benchmark with around $5\%$.

B.4 Further Experiment Results

In Table 4, we show the detailed performance of SPIN at different iterations on each of the task in Open LLM Leaderboard. In Table 5, we also show the performance of SFT from zephyr-7b-sft-full on Ultrachat200k for one epoch. While self-play fine-tuning with synthetic data from zephyr-7b-sft-full effectively improves its performance, simply fine-tuning it again on the SFT data leads to degraded performance, as similarly observed in Figure 5.

Table 4: Test performance of SPIN based on zephyr-7b-sft-full across HuggingFace Open LLM Leaderboard datasets. We also denote the average improvement over last iteration in the Average column.

Model	Arc	TruthfulQA	Winogrande	GSM8k	HellaSwag	MMLU	Average
zephyr-7b-sft-full	60.41	43.73	74.19	26.76	82.85	60.92	58.14
SPIN iteration $0$	63.40	49.18	72.69	35.10	84.38	60.03	${60.80}_{(+2.66)}$
SPIN iteration $1$	65.19	55.17	72.30	35.78	84.96	59.34	${62.12}_{(+1.32)}$
SPIN iteration $2$	65.96	54.91	73.56	38.06	85.41	59.93	${62.97}_{(+0.85)}$
SPIN iteration $3$	65.87	54.90	73.72	38.97	85.54	59.99	${63.16}_{(+0.19)}$

Table 5: Test performance of zephyr-7b-sft-full fine-tuned on Ultrachat200k for 1 more epoch across HuggingFace Open LLM benchmark datasets. SFT fails to further leverage the fine-tuning data for performance enhancement and even results in degraded performance.

Model	Arc	TruthfulQA	Winogrande	GSM8k	HellaSwag	MMLU	Average
zephyr-7b-sft-full	60.41	43.73	74.19	26.76	82.85	60.92	58.14
SFT epoch 1	57.76	44.39	75.77	25.85	81.69	57.89	57.23

Further Investigation on More Tasks. Here, we further investigate the performance of SPIN on a broader variety of tasks, including MT-Bench (Zheng et al., 2023), Big-Bench (bench authors, 2023) and OpenBookQA (Mihaylov et al., 2018) in addition to the Open LLM Leaderboard tasks. Specifically, we use the following tasks from Big-Bench-Hard for a more comprehensive evaluation, including Causal Judgment (causal reasoning), Sports Understanding (commonsense reasoning) and Formal Fallacies (logical reasoning). In Table 6, we show the resulting scores of SPIN on MT-Bench as well as those tasks from Big-Bench. In Figure 6, we detail the model performances on MT-Bench with regard to different types of questions. We can see a notably robust improvement in the performance of SPIN on various tasks besides the HuggingFace Benchmark, without major degradation. Notably, on MT-Bench, the model fine-tuned by SPIN has surpassed the performance of vicuna-13b-v1.5 (Chiang et al., 2023) with a score of $6.57$.

Table 6: Test performance on other reasoning benchmark datasets for SPIN at different iterations and zephyr-7b-sft-full. We report the average score for MT-Bench and the accuracy score for Big Bench datasets under standard few-shot CoT evaluation. On OpenBookQA, we report acc_norm with 1-shot example as used in Anil et al. (2023). As similar to Open LLM Leaderboard evaluation, we observe a steady improvement in performance on the other benchmark tasks, with no significant degradation.

Model	MT-Bench	BB-causal	BB-formal	BB-sports	OpenBookQA
zephyr-7b-sft-full	5.94	56.15	49.6	96.0	45.4
SPIN iteration $0$	${6.46}_{(+0.52)}$	57.75	51.6	95.2	46.8
SPIN iteration $1$	${6.65}_{(+0.19)}$	58.82	51.2	95.2	47.2
SPIN iteration $2$	${6.78}_{(+0.13)}$	59.36	51.2	94.4	47.6

Figure 6: Model performance on MT-Bench. We compare SPIN across different iterations with the base SFT model. Starting from iteration 1, our fine-tuned model by SPIN robustly outperforms the SFT checkpoint on all evaluation aspects.

B.5 Generation Examples

In Tables 7 and 8, we further provide the generation examples of our fine-tuned model by SPIN from different iterations. We can observe an improvement in response quality as compared to the generation of the SFT checkpoint. Meanwhile, the model generations at higher iterations typically becomes more concise than iteration $0$ and resemble the ground truth completion better.

Table 7: Generation example of our fine-tuned model by SPIN at different iterations.

Prompt	How does the economic growth of Horsham compare to nearby towns?
Ground truth	I do not have access to the latest economic data on horsham and nearby towns. However, you can check the latest economic statistics and reports provided by the local government or the official websites of horsham and nearby towns for a detailed comparison of their economic growth.
zephyr-7b=sft-full	According to the given material, Horsham was growing "significantly" faster than nearby towns, indicating that it was economically thriving.\n However, the given material does not specify the exact rates or levels of economic growth of Horsham compared to nearby towns. Without this information, it is challenging to determine whether the differences in economic growth between Horsham and nearby towns were significant or minimal.
Iteration 0	I do not have access to the latest economic data for horsham and nearby towns. However, horsham is known to be a prosperous town in west sussex, uk, with a diverse economy that includes agriculture, manufacturing, and services. The town has a strong retail sector, and it is also home to several large companies, including the global pharmaceutical company novartis. In terms of comparison with nearby towns, horsham is generally considered to be one of the more affluent towns in the region. However, the economic growth of horsham and its neighboring towns may vary depending on various factors such as industry, investment, and government policies.
Iteration 1	I do not have access to real-time data or information about the most recent economic growth of horsesham and nearby towns. However, according to the recent reports from various sources such as the office for national statistics, horsesham’s economy primarily relies on retail, healthcare, education, and public administration industries. Compared to nearby towns such as lewes, chichester, and bognor regis, horsesham’s economy may have a different composition and growth rate based on their specific industries and economic policies.
Iteration 2	I don’t have access to the most recent economic data. However, according to a 2019 report by pwc, horsham’s expected economic growth up to 2025 was projected to be around $2.3\%$ per annum, which is slightly below the average for the south east of england ($2.8\%$). Neighbouring towns like crawley and bognor regis were projected to experience similar growth rates. note: this may not reflect the impact of the covid-19 pandemic that occurred in 2020.

Table 8: Another generation example of our fine-tuned model by SPIN at different iterations.

Prompt	Can you clarify if Poppelsdorfer Allee 45 and the old site of the Mathematical Institute are the same location? Answer according to: Go straight to Poppelsdorfer Allee 45 and sign-in.\n There is a big lecture hall at the old site of the Mathematical Institute (Wegelerstr. 10, Bonn).
Ground truth	No, Poppelsdorfer Allee 45 and the old site of the Mathematical Institute (Wegelerstr. 10, Bonn) are not the same location. To sign-in, one should go straight to Poppelsdorfer Allee 45. However, there is a big lecture hall at the old site of the Mathematical Institute.
zephyr-7b=sft-full	Yes, Poppelsdorfer Allee 45 and the old site of the Mathematical Institute are the same location.
Iteration 0	Yes, Poppelsdorfer Allee 45 and the old site of the Mathematical Institute (Wegelerstr. 10, Bonn) are the same location. The Mathematical Institute has been relocated and renamed as the Institute for Mathematics and Theoretical Physics, but the building where it used to be is still referred to as the old site. The big lecture hall at the old site still exists and may be used for events or classes.
Iteration 1	No, Poppelsdorfer Allee 45 and the old site of the Mathematical Institute are not the same location. While Poppelsdorfer Allee 45 is where attendees should sign-in, the old site of the Mathematical Institute (located at Wegelerstr. 10, Bonn) has a large lecture hall mentioned in the statement. Therefore, it can be assumed that the two locations are separate and not the same.
Iteration 2	Poppelsdorfer Allee 45 and the old site of the Mathematical Institute (Wegelerstr. 10, Bonn) are two different locations. As per the instruction, you need to go straight to Poppelsdorfer Allee 45 and sign-in. However, there is a lecture hall mentioned at the old site of the Mathematical Institute (Wegelerstr. 10, Bonn).

Appendix C Proof of Theorems in Section 5

C.1 Proof of Theorem 5.2

Proof of Theorem 5.2.

To begin with, we prove the “Sufficiency” in Theorem 5.2. Since $p_{\mathrm{data}}(\cdot |𝐱)=p_{{𝜽}_t}(\cdot |𝐱)$, by symmetry property of $𝐲$ and ${𝐲}^{\prime }$, we have for any $𝜽\in 𝚯$ that

$2L_{\text{SPIN}}(𝜽,{𝜽}_t)$ $={𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),{𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\mathrm{\ell }\left(\gamma \log {\displaystyle \frac{p_{𝜽}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}-\gamma \log {\displaystyle \frac{p_{𝜽}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}\right)\right]$

$+{𝔼}_{𝐱\sim q(\cdot ),{𝐲}^{\prime }\sim p_{\mathrm{data}}(\cdot |𝐱),𝐲\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\mathrm{\ell }\left(\gamma \log {\displaystyle \frac{p_{𝜽}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}-\gamma \log {\displaystyle \frac{p_{𝜽}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}\right)\right]$

$={𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),{𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}[\mathrm{\ell }(\gamma \log {\displaystyle \frac{p_{𝜽}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}-\gamma \log {\displaystyle \frac{p_{𝜽}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}})$

$+\mathrm{\ell }(\gamma \log {\displaystyle \frac{p_{𝜽}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}-\gamma \log {\displaystyle \frac{p_{𝜽}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}})]$

$\ge 2{𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),{𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}[\mathrm{\ell }({\displaystyle \frac{\gamma }{2}}\log {\displaystyle \frac{p_{𝜽}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}-{\displaystyle \frac{\gamma }{2}}\log {\displaystyle \frac{p_{𝜽}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}$

$+{\displaystyle \frac{\gamma }{2}}\log {\displaystyle \frac{p_{𝜽}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}-{\displaystyle \frac{\gamma }{2}}\log {\displaystyle \frac{p_{𝜽}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}})]$

$=2\mathrm{\ell }(0),$

where the inequality is due to Jensen’s inequality (recalling that $\mathrm{\ell }$ is convex in Assumption 5.1). Therefore, we have that $L_{\text{SPIN}}(𝜽,{𝜽}_t)\ge \mathrm{\ell }(0)=L_{\text{SPIN}}({𝜽}_t,{𝜽}_t)$, which means that ${𝜽}_t$ is the global optimum of (4.7). As a consequence, the gradient at the point ${𝜽}_t$ is zero, which concludes ${𝜽}_{t+1}={𝜽}_t$.

Next, we prove the “Necessity”. Define $g(\lambda )$ as follows:

$g(\lambda )={𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),{𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\mathrm{\ell }\left(\lambda \log {\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}-\lambda \log {\displaystyle \frac{p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}\right)\right].$

Then we have $g(0)=\mathrm{\ell }(0)$ and

$g^{\prime }(0)$ $={𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),{𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[{\mathrm{\ell }}^{\prime }(0)\left(\log {\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}-\log {\displaystyle \frac{p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}\right)\right]$

$={\mathrm{\ell }}^{\prime }(0)\left({𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱)}\left[\log {\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}\right]-{𝔼}_{𝐱\sim q(\cdot ),{𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\log {\displaystyle \frac{p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}\right]\right)$

$={\mathrm{\ell }}^{\prime }(0)[\mathrm{KL}\left(p_{\mathrm{data}}(\cdot |𝐱)\right|\left|p_{{𝜽}_t}(\cdot |𝐱)\right)+\mathrm{KL}\left(p_{{𝜽}_t}(\cdot |𝐱)\right|\left|p_{\mathrm{data}}(\cdot |𝐱)\right)]$

where the last inequality is due to the condition that . Therefore, there exist a ${\lambda }_0$ such that for all , we have . Choose ${𝜽}^{\ast }$ such that $p_{{𝜽}^{\ast }}(𝐲|𝐱)=p_{\mathrm{data}}(𝐲|𝐱)$. For those , we have that

$L_{\text{SPIN}}({𝜽}^{\ast },{𝜽}_t)$ $={𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{{𝜽}^{\ast }}(\cdot |𝐱),{𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\mathrm{\ell }\left(\lambda \log {\displaystyle \frac{p_{{𝜽}^{\ast }}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}-\lambda \log {\displaystyle \frac{p_{{𝜽}^{\ast }}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}\right)\right]$

$={𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),{𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\mathrm{\ell }\left(\lambda \log {\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}-\lambda \log {\displaystyle \frac{p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}\right)\right]$

$=g(\lambda )$

$=L_{\text{SPIN}}({𝜽}_t,{𝜽}_t),$

where the second equality holds by the choice of $p_{{𝜽}^{\ast }}(\cdot |𝐱)$, and the inequality holds due to the choice of $\lambda$. Therefore, we conclude that ${𝜽}_t$ is not the global optimum of (4.7) if $p_{{𝜽}_t}(\cdot |𝐱)\ne p_{\mathrm{data}}(\cdot |𝐱)$. ∎

C.2 Proof Theorem 5.4

We need the following auxiliary lemma before we prove Theorem 5.4.

Lemma C.1.

Suppose that $\mathrm{\ell }(t)=\log (1+\exp (-t))$ and for $a,b>0$, the following inequality holds

$a\mathrm{\ell }(t)+b\mathrm{\ell }(-t)\ge a\log (1+b/a)+b\log (1+a/b),$

the equality holds if and only if $t=\log (a/b)$.

Proof of Lemma C.1.

Define $g(t)=a\mathrm{\ell }(t)+b\mathrm{\ell }(-t)=a\log (1+\exp (-t))+b\log (1+\exp (t))$, then we have

$g^{\prime }(t)=-{\displaystyle \frac{a\exp (-t)}{1+\exp (-t)}}+{\displaystyle \frac{b\exp (t)}{1+\exp (t)}}={\displaystyle \frac{-a+b\exp (t)}{1+\exp (t)}}.$

Therefore, when , $g^{\prime }(t)>0$ when $t>\log (a/b)$, which indicates that $g$ achieves it minimum at $t=\log (a/b)$ which concludes the proof. ∎

Lemma C.1 shows that the global minimum of $a\mathrm{\ell }(t)+b\mathrm{\ell }(-t)$ is achieved when $t=\log (a/b)$. Based on Lemma C.1, we can further prove that (4.2) with the logistic loss function has a closed-form solution if we ignore the constraint set ${ℱ}_t$.

Lemma C.2.

Denote $p_{+}(𝐲,{𝐲}^{\prime },𝐱)=q(𝐱)\cdot p_{\mathrm{data}}(𝐲|𝐱)\cdot p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)$ and $p_{-}(𝐲,{𝐲}^{\prime },𝐱)=q(𝐱)\cdot p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)\cdot p_{\mathrm{data}}(𝐲|𝐱)$,

${𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),y^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\mathrm{\ell }\left(f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })\right)\right]\ge \log 2-\mathrm{JSD}(p_{+}\parallel p_{-}),$

where $\mathrm{JSD}(p_{+}\parallel p_{-})$ represents the Jensen–Shannon divergence which is defined as follows

$$\mathrm{JSD}\left(p\parallel q\right)=\frac{1}{2}\mathrm{KL}\left(p\parallel \frac{p+q}{2}\right)+\frac{1}{2}\mathrm{KL}\left(q\parallel \frac{p+q}{2}\right),$$

where $\mathrm{KL}(\cdot \parallel \cdot )$ is KL-divergence. $\mathrm{JSD}$ is always non-negative and equals zero if and only if $p_{+}$ and $p_{-}$ are identical. Moreover, the global minimum value $\log 2-\mathrm{JSD}(p_{+}\parallel p_{-})$ is achieved by $f^{\ast }$ if and only if,

$f^{\ast }(𝐱,𝐲)=Z(𝐱)+\log \left({\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}\right),$

where $Z(𝐱)$ is any function that is possibly dependent on $𝐱$.

Proof of Lemma C.2.

We rewrite the objective function in the following formula,

$2{𝔼}_{𝐱\sim q(\cdot ),𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),{𝐲}^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\mathrm{\ell }\left(f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })\right)\right]$

$={\displaystyle \int q(𝐱)p_{\mathrm{data}}(𝐲|𝐱)p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)\left[\mathrm{\ell }\left(f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })\right)\right]𝑑𝐲𝑑{𝐲}^{\prime }}$

$+{\displaystyle \int q(𝐱)p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)p_{{𝜽}_t}(𝐲|𝐱)\left[\mathrm{\ell }\left(f(𝐱,{𝐲}^{\prime })-f(𝐱,𝐲)\right)\right]𝑑𝐲𝑑{𝐲}^{\prime }}$

$={\displaystyle \int q(𝐱)p_{\mathrm{data}}(𝐲|𝐱)p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)\mathrm{\ell }\left(f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })\right)}$

$+q(𝐱)p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)p_{{𝜽}_t}(𝐲|𝐱)\mathrm{\ell }\left(f(𝐱,{𝐲}^{\prime })-f(𝐱,𝐲)\right)d𝐲d{𝐲}^{\prime }$

$\stackrel{(i)}{\ge }{\displaystyle \int q(𝐱)p_{\mathrm{data}}(𝐲|𝐱)p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)\log \left(1+\frac{p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)p_{{𝜽}_t}(𝐲|𝐱)}{p_{\mathrm{data}}(𝐲|𝐱)p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}\right)}$

$+q(𝐱)p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)p_{{𝜽}_t}(𝐲|𝐱)\log \left(1+{\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}{p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)p_{{𝜽}_t}(𝐲|𝐱)}}\right)d𝐲d{𝐲}^{\prime },$

where the inequality is due to $a\mathrm{\ell }(t)+b\mathrm{\ell }(-t)\ge a\log (1+b/a)+b\log (1+a/b)$ in Lemma C.1 with $a=q(𝐱)p_{\mathrm{data}}(𝐲|𝐱)p_{{𝜽}_t}({𝐲}^{\prime }|𝐱),b=q(𝐱)p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)p_{{𝜽}_t}(𝐲|𝐱)$, $t=f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })$. The equality (i) holds if and only if the following equation holds almost surely for any $𝐱,𝐲,{𝐲}^{\prime }$,

$f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })=\log \left({\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}{p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)p_{{𝜽}_t}(𝐲|𝐱)}}\right).$ (C.1)

Equation (C.1) is equivalent to

$f(𝐱,𝐲)-\log \left({\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}\right)=f(𝐱,{𝐲}^{\prime })-\log \left({\displaystyle \frac{p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)}{p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}}\right)$

holds almost surely for any $𝐱,𝐲,{𝐲}^{\prime }$. Therefore, the equality (i) holds if and only if there exists some $Z(𝐱)$ such that

$f(𝐱,𝐲)=Z(𝐱)+\log \left({\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}\right).$

Recall that $p_{+}(𝐲,{𝐲}^{\prime }|𝐱)=p_{\mathrm{data}}(𝐲|𝐱)\cdot p_{{𝜽}_t}(𝐲|𝐱)$ and $p_{-}(𝐲,{𝐲}^{\prime }|𝐱)=p_{{𝜽}_t}(𝐲|𝐱)\cdot p_{\mathrm{data}}(𝐲|𝐱)$. Then, the right-hand side of (i) can be written as

${\displaystyle \int q(𝐱)p_{\mathrm{data}}(𝐲|𝐱)p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)\log \left(1+\frac{p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)p_{{𝜽}_t}(𝐲|𝐱)}{p_{\mathrm{data}}(𝐲|𝐱)p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}\right)}$

$+q(𝐱)p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)p_{{𝜽}_t}(𝐲|𝐱)\log \left(1+{\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)p_{{𝜽}_t}({𝐲}^{\prime }|𝐱)}{p_{\mathrm{data}}({𝐲}^{\prime }|𝐱)p_{{𝜽}_t}(𝐲|𝐱)}}\right)d𝐲d{𝐲}^{\prime }$

$={\displaystyle \int p_{+}(𝐲,{𝐲}^{\prime }|𝐱)\log \left(1+\frac{p_{-}(𝐲,{𝐲}^{\prime }|𝐱)}{p_{+}(𝐲,{𝐲}^{\prime }|𝐱)}\right)}+p_{-}(𝐲,{𝐲}^{\prime }|𝐱)\log \left(1+{\displaystyle \frac{p_{+}(𝐲,{𝐲}^{\prime }|𝐱)}{p_{-}(𝐲,{𝐲}^{\prime }|𝐱)}}\right)d𝐲d{𝐲}^{\prime }$

$=2\log 2+{\displaystyle \int p_{+}(𝐲,{𝐲}^{\prime }|𝐱)\log \left(\frac{1/2[p_{-}(𝐲,{𝐲}^{\prime }|𝐱)+p_{+}(𝐲,{𝐲}^{\prime }|𝐱)]}{p_{+}(𝐲,{𝐲}^{\prime }|𝐱)}\right)}$

$+p_{-}(𝐲,{𝐲}^{\prime }|𝐱)\log \left({\displaystyle \frac{1/2[p_{-}(𝐲,{𝐲}^{\prime }|𝐱)+p_{+}(𝐲,{𝐲}^{\prime }|𝐱)]}{p_{-}(𝐲,{𝐲}^{\prime }|𝐱)}}\right)d𝐲d{𝐲}^{\prime }$

$=2\log 2-\mathrm{KL}\left(p_{+}\parallel {\displaystyle \frac{p_{+}+p_{-}}{2}}\right)-\mathrm{KL}\left(p_{-}\parallel {\displaystyle \frac{p_{+}+p_{-}}{2}}\right)$

$=2\log 2-2\cdot \mathrm{JSD}(p_{+}\parallel p_{-}),$

where the last equality is by the definition of JSD. This concludes the proof. ∎

Lemma C.2 provides a closed-form solution to (4.2) if we ignore the constraint set ${ℱ}_t$. If this closed-form solution belongs to ${ℱ}_t$, then it should also be the solution to (4.2). This observation is the key to the proof of Theorem 5.4.

Proof of Theorem 5.4.

Under the condition of Theorem 5.4, there exists a $p_{𝜽}$ such that

$p_{𝜽}(𝐲|𝐱)\propto p_{{𝜽}_t}(𝐲|𝐱){\left(p_{\mathrm{data}}(𝐲|𝐱)/p_{{𝜽}_t}(𝐲|𝐱)\right)}^{1/\lambda }.$

Therefore, there exists a function $\widehat{Z}(𝐱)$ such that

$p_{𝜽}(𝐲|𝐱)=\widehat{Z}(𝐱)\cdot p_{{𝜽}_t}(𝐲|𝐱){\left(p_{\mathrm{data}}(𝐲|𝐱)/p_{{𝜽}_t}(𝐲|𝐱)\right)}^{1/\lambda }.$ (C.2)

Applying logarithm function on both side of (C.2) yields

$\lambda \log (\widehat{Z}(𝐱))+\log \left({\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}\right)=\lambda \log \left({\displaystyle \frac{p_{𝜽}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}\right)\in {ℱ}_t.$

By Lemma C.2, $f^{\ast }(𝐱,𝐲)=\lambda \log (\widehat{Z}(𝐱))+\log \left(\frac{p_{\mathrm{data}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}\right)$ is the global minimum of the following minimization problem,

$\underset{f}{argmin}{𝔼}_{𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),y^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\mathrm{\ell }\left(f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })\right)\right].$ (C.3)

Since $f^{\ast }\in {ℱ}_t$, $f^{\ast }(𝐱,𝐲)=\lambda \log (\widehat{Z}(𝐱))+\log \left(\frac{p_{\mathrm{data}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}\right)$ is also the global optimum of the optimization problem (4.2),

$\underset{f\in {ℱ}_t}{argmin}{𝔼}_{𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),y^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\mathrm{\ell }\left(f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })\right)\right].$

Therefore, we have proved that

$\underset{f}{\min }{𝔼}_{𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),y^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\mathrm{\ell }\left(f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })\right)\right]$

$=\underset{f\in {ℱ}_t}{\min }{𝔼}_{𝐲\sim p_{\mathrm{data}}(\cdot |𝐱),y^{\prime }\sim p_{{𝜽}_t}(\cdot |𝐱)}\left[\mathrm{\ell }\left(f(𝐱,𝐲)-f(𝐱,{𝐲}^{\prime })\right)\right]$

$=\underset{𝜽\in 𝚯}{\min }{L}_{\text{SPIN}}(𝜽,{𝜽}_t).$ (C.4)

Since ${𝜽}_{t+1}$ is the global minimum of $L_{\text{SPIN}}(𝜽,{𝜽}_t)$. Then by (C.4), $\lambda \log \left(\frac{p_{{𝜽}_{t+1}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}\right)$ should be the global minimum of problem (C.3). By Lemma C.2, there exists $Z(𝐱)$ such that

$\lambda \log \left({\displaystyle \frac{p_{{𝜽}_{t+1}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}\right)=Z(𝐱)+\log \left({\displaystyle \frac{p_{\mathrm{data}}(𝐲|𝐱)}{p_{{𝜽}_t}(𝐲|𝐱)}}\right),$

which leads to the result that $p_{{𝜽}_{t+1}}(𝐲|𝐱)\propto p_{{𝜽}_t}(𝐲|𝐱){\left(p_{\mathrm{data}}(𝐲|𝐱)/p_{{𝜽}_t}(𝐲|𝐱)\right)}^{1/\lambda }$.

∎