Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis

Abstract

Diffusion model is a promising approach to image generation and has been employed for Pose-Guided Person Image Synthesis (PGPIS) with competitive performance. While existing methods simply align the person appearance to the target pose, they are prone to overfitting due to the lack of a high-level semantic understanding on the source person image. In this paper, we propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for PGPIS. In the absence of image-caption pairs and textual prompts, we develop a novel training paradigm purely based on images to control the generation process of a pre-trained text-to-image diffusion model. A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt. This allows for the decoupling of fine-grained appearance and pose information controls at different stages, and thus circumventing the potential overfitting problem. To generate more realistic texture details, a hybrid-granularity attention module is proposed to encode multi-scale fine-grained appearance features as bias terms to augment the coarse-grained prompt. Both quantitative and qualitative experimental results on the DeepFashion benchmark demonstrate the superiority of our method over the state of the arts for PGPIS. Code is available at https://github.com/YanzuoLu/CFLD.

1 Introduction

Pose-Guided Person Image Synthesis (PGPIS) aims to translate the source person image into a specific target pose while preserving the appearance as much as possible. It has a wide range of applications, including film production, virtual reality, and fashion e-commerce. Most existing methods along this line are developed based on Generative Adversarial Networks (GANs) [26, 6, 41, 63, 18, 33, 44, 54, 25, 56, 34, 27, 38, 20, 28, 62]. Nevertheless, the GAN-based approach may suffer from the instability of min-max training objective and difficulty in generating high-quality images in a single forward pass.

Figure 1: (a) The appearance of person images varies significantly given only a textual prompt for image generation by using Stable Diffusion [35] or ControlNet [55] with OpenPose guidance [3]. (b) Simply aligning the source appearance to the target pose without a semantic understanding of person image can easily lead to overfitting, such that the generated images become distorted and unnatural. (c) Our method learns the coarse-grained prompt for a comprehensive perception of the source image and injects fine-grained appearance features as bias terms, thus generating high-quality images with better generalization performance.

As a promising alternative to GANs for image generation, diffusion models synthesize more realistic images progressively from a series of denoising steps. The recently prevailing text-to-image latent diffusion model, such as Stable Diffusion (SD) [35] may now generate compelling person images conditioned on a given textual prompt. The appearance of the generated person can be determined by well-designed prompts [19, 31] or prompt learning [60, 59]. With more reliable structural guidance [55, 29], the synthesized person images can be further constrained to specific poses. Though the text-to-image diffusion generates realistic images from textual prompts with high-level semantics, its training paradigm requires extensive image-caption pairs that are labor-expensive to collect for PGPIS. More importantly, due to the differing information densities between language and vision [11], even the most detailed textual descriptions inevitably introduce ambiguity and may not accurately preserve the appearance as illustrated in Fig. 1(a).

More recently, several diffusion-based approaches have emerged for PGPIS. A texture diffusion module is proposed by PIDM [1] to model the complex correspondence between the appearance of source image and the target pose. Since the denoising process at the high-resolution pixel level is computationally expensive, PoCoLD [9] reduces both the training and inference costs by mapping pixels to low-dimensional latent spaces with a pre-trained Variational Autoencoder (VAE) [7]. In PoCoLD, the correspondence is further exploited by a pose-constrained attention module based on additional 3D Densepose [8] annotations. While both the PIDM and PoCoLD generate more realistic texture details by aligning the source image to the target pose, they lack a high-level semantic understanding of person images. Therefore, they are prone to overfitting and poor generalization performance when synthesizing exaggerated poses that are vastly different from the source image or rare in the training set. As demonstrated in Fig. 1(b), the generated images become distorted and unnatural in these cases, which is in line with several GAN-based approaches.

In this work, we propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for PGPIS. Our approach breaks the conventional training paradigm which leverages textual prompts to control the generation process of a pre-trained SD model. Instead of conditioning on the human-generated signals, i.e. languages that are highly semantic and information-dense, we facilitate a coarse-to-fine appearance control method purely based on images. To obtain the aforementioned semantic understanding specific to person images, we endeavor to decouple the fine-grained appearance and pose information controls at different stages by introducing a perception-refined decoder. The perception of the source person image is achieved by randomly initializing a set of learnable queries and progressively refining them in the following decoder blocks via cross-attention. The decoder output serves as a coarse-grained prompt to describe the source image, focusing on the common semantics across different person images, e.g. human body parts and attributes such as age and gender. Moreover, we design a hybrid-granularity attention module to effectively encode multi-scale fine-grained appearance features as bias terms to augment the coarse-grained prompt. In this way, the source image is able to be aligned with the target pose by supplementing only the necessary fine-grained details under the guidance of the coarse-grained prompt, thus achieving better generalization as illustrated in Fig. 1(c).

Our main contributions can be summarized as follows,

• We present a novel training paradigm in the absence of image-caption pairs to overcome the limitations when applying text-to-image diffusion to PGPIS. We propose a perception-refined decoder to extract semantic understanding of person images as a coarse-grained prompt.

• We formulate a new hybrid-granularity attention module to bias the coarse-grained prompt with fine-grained appearance features. Thus, the texture details of generated images are better controlled and become more realistic.

• We conduct extensive experiments on the DeepFashion [21] benchmark and achieve the state-of-the-art performance both quantitatively and qualitatively. User studies and ablations validate the effectiveness of our method.

2 Related Work

Pose-Guided Person Image Synthesis. Ma et al. [26] first presents the task of pose-guided person image synthesis and refines the generated images in an adversarial manner. To decouple the pose and appearance information, early approaches [6, 27] propose to learn pose-irrelevant features but fail to handle the complex texture details with vanilla convolutional neural networks. To alleviate this problem, auxiliary information is introduced to improve the generation quality, such as parsing [28] and UV maps [38]. Recent approaches [63, 44, 18, 33, 20, 34] focus on modeling the spatial correspondence between pose and appearance, with the more frequent use of parsing maps [54, 25, 62]. PIDM [1] and PoCoLD [9] are developed based on diffusion models to prevent from the drawbacks in the generative adversarial networks, including the instability of min-max training objective and difficulty in synthesising high-resolution images. Both of these two diffusion-based methods extend the idea of spatial correspondence to model the relation between the appearance of source image and target pose via the cross-attention mechanism. We argue this leads to overfitting by simply aligning the source appearance to the target pose without a high-level semantic understanding of the person image. More concurrent works like MagicAnimate [50], Animate Anyone [16] and PCDMs [39] require multi-stage and progressive fully fine-tuning, while our pipeline is more efficient and end-to-end by freezing most parameters. And the training paradigm of IP-Adatper [52] is heavily relying on image-text pairs which are not available for PGPIS task.

Figure 2: (a) Architecture of our proposed Coarse-to-Fine Latent Diffusion (CFLD) method. For pose-guided latent diffusion, we incorporate a lightweight pose adapter ${\mathscr{H}}_P$ from [29] to add its output feature maps to the end of each down-sampling block of the pre-trained UNet ${\mathscr{H}}_N$ for efficient structural guidance. To achieve a coarse-to-fine appearance control, we propose a perception-refined decoder ${\mathscr{H}}_D$ and hybrid-granularity attention module ${\mathscr{H}}_A$, both of which take the multi-scale feature maps from a source image encoder ${\mathscr{H}}_S$ as inputs. (b) The coarse-grained prompt is obtained by refining the learnable queries progressively in our proposed ${\mathscr{H}}_D$. (c) We encode the multi-scale fine-grained appearance features as bias terms in the up-sampling blocks for better texture details within ${\mathscr{H}}_A$.

Controllable Diffusion Models. Diffusion models have recently emerged and demonstrated their potential for high-resolution image synthesis. The core idea is to start with a simple noise vector and gradually transform it into a high-quality image through multiple denoising iterations. Beyond unconditional generation [15, 42, 43], various methods have been introduced to incorporate user-supplied control signals into the generation process, enabling more controllable image generation. For instance, [5] introduces the usage of classifier gradients to condition on the generation, while [14] proposes a classifier-free control mechanism employing a weighted summation of conditional and unconditional outputs for controllable synthesis. Moreover, the Latent Diffusion Model (LDM) performs diffusion in the latent space and injects the conditioning signals via a specific encoder and cross-attention. Building upon the pre-trained LDM like Stable Diffusion (SD) [35], subsequent works have explored to bias the latent space by adding extra controls [55, 29], as well as further to provide users with control over the generated content [45, 12]. Rather than employing a high-level conditioning prompt throughout the generation, we design a coarse-to-fine conditioning process that adjusts the latent features at different stages within the UNet-based prediction network, providing better controllable pose-guided person image synthesis.

3 Method

3.1 Preliminary

Our method builds on top of the text-to-image latent diffusion model, i.e., Stable Diffusion (SD) [35] with high-quality image generation ability. There are two main stages in the SD model: a Variational Autoencoder (VAE) [7] that maps between raw-pixel space and low-dimensional latent space and an UNet-based prediction model [36] for denoising diffusion image generation. It follows the general idea of Denoising Diffusion Probabilistic Model (DDPM) [15], which formulates a forward diffusion process and a backward denoising process of $T=1000$ steps. The diffusion process progressively adds random Gaussian noise $ϵ\sim 𝒩(0,𝑰)$ to the initial latent ${𝒛}_0$, mapping it into noisy latents ${𝒛}_t$ at different timesteps $t\in [1,T]$,

$${𝒛}_t=\sqrt{{\overline{\alpha }}_t}{𝒛}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ,$$ (1)

where ${\overline{\alpha }}_1,{\overline{\alpha }}_2,\mathrm{\dots },{\overline{\alpha }}_T$ are derived from a fixed variance schedule. The denoising process learns the UNet ${ϵ}_{\theta }({𝒛}_t,t,𝒄)$ to predict the noise and reverse this mapping, where $𝒄$ is the conditional embedding output by e.g. the CLIP [32] text encoder in [35]. The optimization can be formulated as,

$${ℒ}_{mse}={𝔼}_{{𝒛}_0,𝒄,ϵ,t}\left[{\Vert ϵ-{ϵ}_{\theta }({𝒛}_t,t,𝒄)\Vert }_2^2\right].$$ (2)

3.2 Coarse-to-Fine Latent Diffusion

Architecture and Overview. Fig. 2(a) shows the architecture of our proposed method. For concise illustration, we omit the encoder $ℰ$ and decoder $𝒟$ of the VAE [7] model in this figure. In the training phase, we are given sets of the source image ${𝒙}_s$, source pose ${𝒙}_{sp}$, target pose ${𝒙}_{tp}$, and ground-truth image ${𝒙}_g$. The source image passes through an image encoder ${\mathscr{H}}_S$ (e.g. swin transformer [22]), from which we extract a stack of multi-scale feature maps ${𝑭}_s=[{𝒇}_1,{𝒇}_2,{𝒇}_3,{𝒇}_4]$ for a coarse-to-fine appearance control. The coarse-grained prompts are learned by our Perception-Refined Decoder (PRD) ${\mathscr{H}}_D$ and serve as conditional embeddings in both down-sampling and up-sampling blocks of the UNet ${\mathscr{H}}_N$. While the down-sampling block in ${\mathscr{H}}_N$ remains intact in our method, we reformulate the up-sampling block with our Hybrid-Granularity Attention module (HGA) ${\mathscr{H}}_A$ to bias the coarse-grained prompt with fine-grained appearance features for more realistic textures. More details about ${\mathscr{H}}_D$ and ${\mathscr{H}}_A$ will be presented later.

For efficient pose control, we adopt a lightweight pose adapter ${\mathscr{H}}_P$ that consists of several ResNet blocks [10]. The output feature maps of ${\mathscr{H}}_P$ are added directly to the end of each down-sampling block as in [29]. This requires no additional fine-tuning and explicitly decouples the fine-grained appearance and pose information controls. At different scales of down-sampling, the pose information is only aligned with the same coarse-grained prompts given by our PRD as conditional embeddings, rather than the different multi-scale fine-grained appearance features in the common practice [1, 9]. In this way, the HGA module learns all the pose-irrelevant texture details at the up-sampling stage and is not prone to overfitting. Denote the initial latent state for the ground-truth image as ${𝒛}_0=ℰ({𝒙}_g)$. The MSE loss in Eq. 2 is thus rewritten as,

$${ℒ}_{mse}={𝔼}_{{𝒛}_0,{𝒙}_s,{𝒙}_{tp},ϵ,t}\left[{\Vert ϵ-{ϵ}_{\theta }({𝒛}_t,t,{𝒙}_s,{𝒙}_{tp})\Vert }_2^2\right].$$ (3)

Perception-Refined Decoder. Instead of utilizing multi-scale appearance features as conditional embeddings as in the existing diffusion-based approaches [1, 9], we propose to decouple the controls from the fine-grained appearance and pose information at different stages. Thus we design a Perception-Refined Decoder (PRD) to extract semantic understanding of person images as a coarse-grained prompt, given the flattened last-scale output ${𝒇}_4$ from ${\mathscr{H}}_S$ as illustrated in Fig. 2(b). By revisiting how people perceive a person image, we find several common characteristics, i.e., human body parts, age, gender, hairstyle, clothing, and so on, as demonstrated in Fig. 1(a). This inspires us to maintain a set of learnable queries ${𝑭}_d^{in}\in {ℝ}^{Q\times D}$ representing different semantics of person images. They are randomly initialized and progressively refined with the standard transformer decoders [47]. The source image conditioning ${𝒇}_4$ interacts via the cross-attention module at each decoder block. After $R$ blocks of refinement, we obtain the coarse-grained prompt ${𝑭}_d^{out}$, which serves as the conditional embedding and inputs to both down-sampling and up-sampling in ${\mathscr{H}}_N$.

Hybrid-Granularity Attention. To precisely control the texture details of generated images, we introduce the Hybrid-Granularity Attention module (HGA) that is embedded in different scales $(l\in \{1,2,3\})$ of up-sampling blocks in ${\mathscr{H}}_N$, where we refer ${𝑭}_h^l,{𝑭}_o^l$ to its input and output. Given the multi-scale feature maps ${𝒇}_l$ of the source image from ${\mathscr{H}}_S$, the HGA module aims to compensate for the missing necessary details in the coarse-grained prompts. To achieve this, we formulate the HGA module that naturally follows a coarse-to-fine learning curriculum.

Specifically, we propose to inject multi-scale texture details by biasing the queries of cross-attention in the up-sampling blocks as shown in Fig. 2(c), i.e.,

$$\begin{array}{c}𝑸={𝑾}_q^l{𝑭}_h^l,𝑲={𝑾}_k^l{𝑭}_d^{out},𝑽={𝑾}_v^l{𝑭}_d^{out},\\ 𝑩={\mathit{\varphi }}_A({𝒇}_l),{𝑭}_o^l=\text{softmax}(\frac{(𝑸+𝑩){𝑲}^T}{\sqrt{d}})𝑽,\end{array}$$ (4)

where ${𝑾}_q^l,{𝑾}_k^l,{𝑾}_v^l$ are specific projection layers for the $l$-th scale up-sampling block of dimension $d$. ${\mathit{\varphi }}_A$ is a fine-grained appearance encoder that mainly consists of $K$ transformer layers with a zero convolution [55] added in the beginning and the end. The zero convolution is a standard $1\times 1$ convolution layer with both weight and bias initialized as zeros. It keeps the gradient of ${\mathscr{H}}_A$ back to ${\mathscr{H}}_S$ small enough in the early stage of training, so that the image encoder ${\mathscr{H}}_S$ and the more easily converged perception-refined decoder ${\mathscr{H}}_D$ can focus on learning to provide a high-level semantic understanding compatible with the pre-trained SD model. Since we have decoupled the controls of the fine-grained appearance and pose information at different stages, the target pose can be well controlled without overfitting during the down-sampling process. Therefore, such a design encourages the HGA module to slowly fill in more fine-grained textures to better align the generation with the source image during training. Note that ${𝑾}_k^l,{𝑾}_v^l$ in the up-sampling blocks are trainable parameters. They are the only trainable parameters of the entire ${\mathscr{H}}_N$, which accounts for only 1.2% of all the parameters in the pre-trained SD model.

3.3 Optimization

To assist the source-to-target pose translation, we follow the insights in [56] to conduct source-to-source self-reconstruction for training. The reconstruction loss is,

$${ℒ}_{rec}={𝔼}_{{𝒛}_0,{𝒙}_s,{𝒙}_{sp},ϵ,t}\left[{\Vert ϵ-{ϵ}_{\theta }({𝒛}_t,t,{𝒙}_s,{𝒙}_{sp})\Vert }_2^2\right],$$ (5)

where ${𝒛}_0=ℰ({𝒙}_s)$ and ${𝒛}_t$ is the noisy latent mapped from ${𝒛}_0$ at timestep $t$. The overall objective is written as,

$${ℒ}_{overall}={ℒ}_{mse}+{ℒ}_{rec}.$$ (6)

Moreover, we adopt the cubic function $t=(1-{(\frac{t}{T})}^3)\times T,t\in \text{Uniform}(1,T)$ for the distribution of timestep $t$. It increases the probability of $t$ falling in the early sampling stage and strengthens the guidance, which helps to converge faster and shorten the training time.

Sampling. Once the conditional latent diffusion model is learned, the inference can be performed and starts by sampling a random Gaussian noise ${𝒛}_T\sim 𝒩(0,𝑰)$. The predicted latent ${\widetilde{𝒛}}_0$ is obtained by reversing the schedule in Eq. 1 using the denoising network ${ϵ}_t$ at each timestep $t\in [1,T]$. We adopt the cumulative classifier-free guidance [9, 14, 2] to strengthen both the source appearance and target pose guidance, i.e.,

${ϵ}_t=$ ${ϵ}_{\theta }({𝒛}_t,t,\mathrm{\varnothing },\mathrm{\varnothing })$ (7)

$+w_{\text{pose}}({ϵ}_{\theta }({𝒛}_t,t,\mathrm{\varnothing },{𝒙}_{tp})-{ϵ}_{\theta }({𝒛}_t,t,\mathrm{\varnothing },\mathrm{\varnothing }))$

$+w_{\text{app}}({ϵ}_{\theta }({𝒛}_t,t,{𝒙}_s,{𝒙}_{tp})-{ϵ}_{\theta }({𝒛}_t,t,\mathrm{\varnothing },{𝒙}_{tp})).$

When the source image ${𝒙}_s$ is missing, we use learnable vectors as the conditional embeddings. The learnable vectors are trained with a probability of $\eta$% to drop both ${𝒙}_s$ and ${𝒙}_p$ during training. The outputs of the pose adapter ${\mathscr{H}}_P$ will be set to all zeros if the target pose ${𝒙}_{tp}$ is missing. We use the DDPM scheduler [15] with 50 steps as the same as in [9, 1]. Finally, the generated image is obtained by the VAE decoder $𝒚=𝒟({\widetilde{𝒛}}_0)$.

4 Experiments

Component	Default		Trainable Params.
${\mathscr{H}}_S$	Swin-B [22]		87.0M
${\mathscr{H}}_A$	$K=4$		22.5M
${\mathscr{H}}_D$	$R=8$, $Q=16$, $C=768$		97.7M
${\mathscr{H}}_P$	Adapter [29]		30.6M
${\mathscr{H}}_N$	up-sampling ${𝑾}_k^l,{𝑾}_v^l$		10.3M
Method	Pose Info. & Annotation	Training Epochs	Trainable Params.
PIDM [1]	2D OpenPose [3]	300	688.0M
PoCoLD [9]	3D DensePose [8]	100	395.9M
CFLD (Ours)	2D OpenPose [3]	100	248.2M

Table 1: The default settings and the number of trainable parameters in each component of our method and comparison with other diffusion-based methods.

Method	Venue	FID$\mathrm{\downarrow }$	LPIPS$\mathrm{\downarrow }$	SSIM$\mathrm{\uparrow }$	PSNR$\mathrm{\uparrow }$
Evaluate on 256$\mathrm{\times }$176 resolution
PATN [63]	CVPR 19’	20.728	0.2533	0.6714	-
ADGAN [28]	CVPR 20’	14.540	0.2255	0.6735	-
GFLA [33]	CVPR 20’	9.827	0.1878	0.7082	-
PISE [54]	CVPR 21’	11.518	0.2244	0.6537	-
SPGNet${}^{†}$ [25]	CVPR 21’	16.184	0.2256	0.6965	17.222
DPTN${}^{†}$ [56]	CVPR 22’	17.419	0.2093	0.6975	17.811
NTED${}^{†}$ [34]	CVPR 22’	8.517	0.1770	0.7156	17.740
CASD${}^{†}$ [62]	ECCV 22’	13.137	0.1781	0.7224	17.880
PIDM${}^{†}$ [1]	CVPR 23’	6.812	0.2006	0.6621	15.630
PIDM${}^{‡}$ [1]	CVPR 23’	6.440	0.1686	0.7109	17.399
PoCoLD [9]	ICCV 23’	8.067	0.1642	0.7310	-
CFLD (Ours)		6.804	0.1519	0.7378	18.235
VAE Reconstructed		7.967	0.0104	0.9660	33.515
Ground Truth		7.847	0.0000	1.0000	$+\mathrm{\infty }$
Evaluate on 512$\mathrm{\times }$352 resolution
CoCosNet2 [61]	CVPR 21’	13.325	0.2265	0.7236	-
NTED${}^{†}$ [34]	CVPR 22’	7.645	0.1999	0.7359	17.385
PoCoLD [9]	ICCV 23’	8.416	0.1920	0.7430	-
CFLD (Ours)		7.149	0.1819	0.7478	17.645
VAE Reconstructed		8.187	0.0217	0.9231	30.214
Ground Truth		8.010	0.0000	1.0000	$+\mathrm{\infty }$

Table 2: Quantitative comparisons with the state of the arts in terms of image quality. ${}^{†}$ We strictly follow the evaluation implementation in NTED [34] and reproduce these results. ${}^{‡}$ Results are obtained using the generated images released by the authors.

Figure 3: Qualitative comparisons with state-of-the-arts. To clarify the relation between objective and subjective metrics, we demonstrate the LPIPS measures and label the images with the first and second highest votes from user opinions as red and blue respectively.

4.1 Setup

Dataset. We follow [9, 34] to conduct experiments on the In-Shop Clothes Retrieval benchmark of DeepFashion [21] and evaluate on both the 256$\times$176 and 512$\times$352 resolutions. This dataset consists of 52,712 high-resolution person images in the fashion domain. The dataset split is the same as in PATN [63], where 101,966 and 8,570 non-overlapping pairs are selected for training and testing, respectively.

Objective metrics. We use four different metrics to evaluate the generated images quantitatively, including FID [13], LPIPS [57], SSIM [49] and PSNR. Both FID and LPIPS are based on deep features. The Fréchet Inception Distance (FID) calculates the Wasserstein-2 distance [46] between the distributions of generated and real images using Inception-v3 [37] features, and the Learned Perceptual Image Patch Similarity (LPIPS) leverages a network trained on human judgments to measure reconstruction accuracy in the perceptual domain. As for the Structural Similarity Index Measure (SSIM) and Peak Signal to Noise Ratio (PSNR), they quantify the similarity between generated images and ground truths at the pixel level.

Subjective metrics. In addition to the objective metrics, we follow [9] to use the Jab [25, 62, 1] metric in our user study to calculate the percentage of generated images that were considered the best among all methods [1, 62, 34, 56, 25]. Moreover, in order to measure the similarity between the generated images and real data, we quantify the R2G and G2R metrics as many early approaches did [26, 41, 63]. R2G represents the percentage of real images considered as generated and G2R represents the percentage of generated images considered as real by humans.

Implementation details. Our method is implemented with PyTorch [30] and HuggingFace Diffusers [48] on top of the Stable Diffusion [35] with the version of 1.5. The source image is resized to 256$\times$256 and the source image encoder ${\mathscr{H}}_S$ is a standard Swin-B [22] pretrained on ImageNet [4]. The default settings and the number of trainable parameters in each component are summarized in Tab. 1. We train for 100 epochs using the Adam [17] optimizer with a base learning rate of 5e-7 scaled by the total batch size. The learning rate undergoes a linear warmup during the first 1,000 steps and is multiplied by 0.1 at 50 epochs. For classifier-free guidance, we set $w_{\text{pose}}$ and $w_{\text{id}}$ to 2.0 during sampling, and drop the condition ${𝒙}_s$ and ${𝒙}_p$ with a probability of $\eta =20$(%) during training.

4.2 Quantitative Comparison

We quantitatively compare our method with both GAN-based and diffusion-based state-of-the-art approaches in terms of objective metrics. The evaluation is performed on both 256$\times$176 and 512$\times$352 resolutions as the same as in [34, 9]. As shown in Tab. 2, our method significantly outperforms the state-of-the-art across all metrics on both resolutions. In particular, compared to the other two diffusion-based methods [1, 9] in Tab. 1, we achieve better reconstruction with simpler 2D-only pose annotations and fewer trainable parameters. The metrics for VAE [7] reconstructions and the ground truths are also provided for reference. It is worth noting that the results we obtain by running with the checkpoint provided by PIDM [1] suffer from severe overfitting, resulting in a large gap between the quantitative results of provided images and those from the checkpoint.

4.3 Qualitative Comparison

In Fig. 3, we present a comprehensive visual comparison with recent approaches that are publicly available and reproducible, including SPGNet [25], DPTN [56], NTED [34], CASD [62] and PIDM [1]. Our observations can be summarized as follows. (1) Both GAN-based and diffusion-based methods suffer from overfitting the human poses. When generating some target poses that are extreme or not common in the training set, existing methods show severe distortions as demonstrated in rows 1-2. Since we decouples the controls of fine-grained appearance and pose information, our method circumvents the potential overfitting problem and always generates a reasonable pose with the conditioning coarse-grained prompt and fine-grained appearance bias. (2) For source images in rows 3-6 with more complex clothing, our generated images better preserve the textures details while aligning with the target pose thanks to the robust coarse-to-fine learning curriculum of hybrid-granularity attention module. For other methods, although they match in color, the clothes either exhibit blurring and distortion (SPGNet, DPTN, and CASD) or are spliced unnaturally in texture, creating a large gap from the source image (NTED and PIDM). (3) As for cases where the target pose requires visualization of areas invisible in the source image, our method exhibits strong understanding and generalization capabilities. With a semantic understanding of the source image provided by the perception-refined decoder, our method is aware of what should be predicted when the person turns around or sits down as illustrated in rows 7-10, such as different patterns on the front and back of clothes, the sitting chair, and lower body wear.

Figure 4: User study results in terms of R2G, G2R and Jab metrics.

4.4 User Study

To verify the gap between generated and real images as well as our superiority over the state of the arts, we have recruited over 100 volunteers to perform the following two user studies following PIDM [1]. (1) For the R2G and G2R metrics, volunteers were asked to discriminate between 30 generated images and 30 real images from the testing set. Each volunteer could only see the generated images of a specific method, and the pairs of source image and target pose for generation were consistent across methods for a fair comparison. From the results in Fig. 4, chances of a real image being recognized as generated (R2G) are relatively low, and over half of the images we generated are recognized as real (G2R), demonstrating that our method generates more realistic images that are less likely to be judged as fake by humans. (2) For the Jab metric, each volunteer was asked to choose the best match to the ground truth from the generated images of different methods. Compared to other methods, our Jab score achieved 52.5 percent, significantly higher (+34.9) than the counterpart in second place, indicating that our method is more preferred and generates better texture details and pose alignment.

Method	Biasing	Trainable	Prompt	LPIPS$\mathrm{\downarrow }$	SSIM$\mathrm{\uparrow }$
B1		$𝑲,𝑽$	M-S	0.2018	0.6959
B2		$𝑲,𝑽$	CLIP	0.2099	0.6944
B3		$𝑲,𝑽$	PRD	0.1615	0.7293
B4		$𝑸,𝑲,𝑽$	PRD	0.1742	0.7198
B5	$𝑸$	$𝑲,𝑽$	Swin	0.1912	0.7038
Ours	$𝑸$	$𝑲,𝑽$	PRD	0.1519	0.7378

Table 3: Quantitative results for ablation studies. M-S is short for multi-scale fine-grained appearance features similar to [1, 9].

Figure 5: Qualitative ablation results. Our approach has a high-level understanding of the source image rather than forced alignment. It is also less prone to overfitting through the complementary coarse-grained prompts and fine-grained appearance biasing.

Figure 6: (a) Style transfer results of our method. The appearance in the reference image can be edited while maintaining the pose and appearance. This is achieved by masking out regions of interest in the reference image and requires no additional training. (b) The interpolation results show that texture details can be gradually shifted from one style to another in a smooth manner (from Style 1 to 2).

Figure 7: Visualizing attention maps by different queries of the prompt decoder. The maps are averaged over all attention heads.

4.5 Ablation Study

We perform ablation studies at multiple baselines to compare with our method. The quantitative results are presented in Tab. 3. B1 is referenced from the other two diffusion-based approaches [1, 9] that incorporate multi-scale fine-grained appearance features as conditional prompts. we also experiment with CLIP image encoder [32] in B2 to produce descriptive coarse-grained prompts for source images, which is first explored by an image-editing approach [51] that are also conditioned purely on images. Together with the qualitative results in Fig. 5, we can see that even very simple textures sometimes fail to be preserved, suggesting that these prompts are not compatible with the pre-trained SD model. To provide coarse-grained features that are more specific to person images, we integrate the Perception-Refined Decoder (PRD) into B3. The reconstruction metrics (i.e., LPIPS and SSIM) in Tab. 3 reveal a significant improvement in the quality of generated images, which validates the effectiveness of our proposed PRD. While this can be confirmed qualitatively in Fig. 5, there is still a lack of textural details as indicated by the red box. To address this issue, we experiment with training more parameters in the UNet as B4 and instead observe a decrease in performance. This implies that the generalization ability of SD model is compromised, which is not our expectation. Thus we come up with the Hybrid-Granularity Attention (HGA) to bias the queries and achieve state-of-the-art results both quantitatively and qualitatively. In order to verify whether the source image encoder (i.e., Swin Transformer [22]) is able to learn sufficient information for HGA and give a useful prompt, we abandon the PRD in B5. The qualitative results in Fig. 5 demonstrate that both B4 and B5 are overfitting, only our method circumvents this problem by learning in a coarse to fine-grained manner.

Visualization. In Fig. 7, we visualize the effectiveness of different queries in ${\mathscr{H}}_D$. The attention maps reflect different human body parts of person images captured by learnable queries, which proves that we have a high-level understanding of the source images and thus less prone to overfitting.

4.6 Appearance Editing

Style Transfer. Our CFLD inherits the strong generation ability of SD model by freezing the vast majority of its parameters. Thus the style transfer can be achieved simply by masking without additional training. Specifically, we mark the regions of interest in the reference image ${𝒚}^{ref}$ as a binary mask $𝒎$. During sampling, the noise prediction is decomposed into ${ϵ}_t^{\prime }=𝒎\cdot {ϵ}_t+(1-𝒎)\cdot {𝒛}_t^{ref}$, where the ${ϵ}_t$ is based on the pose from ${𝒚}^{ref}$ and the appearance from different styles of source images. Let ${𝒛}_t^{ref}$ be the noisy latent at timestep $t$ mapped from ${𝒛}_0^{ref}=ℰ({𝒚}^{ref})$ as in Eq. 1. From the results in Fig. 6(a), our method generates realistic and coherent texture details in the regions of interest.

Style Interpolation. Additionally, our CFLD supports arbitrary linear interpolation of both coarse-grained prompts and fine-grained appearance biases. As shown in Fig. 6(b), our generated images are faithfully reproducing different styles with smooth transitions.

5 Conclusion

This paper presents a novel Coarse-to-Fine Latent Diffusion (CFLD) method for Pose-Guided Person Image Synthesis (PGPIS). We circumvent the potential overfitting problem by decoupling the fine-grained appearance and pose information controls. Our proposed Perception-Refined Decoder (PRD) and Hybrid-Granularity Attention module (HGA) enable a high-level semantic understanding of person images, while also preserving texture details through a coarse-to-fine learning curriculum. Extensive experiments demonstrate that CFLD outperforms the state of the arts in PGPIS both quantitatively and qualitatively. Our future work will investigate whether the CFLD can be extended to more downstream tasks that suffer from inferior data like person re-identification [23, 53] and domain adaptation [24, 40], since our training paradigm yields both a pre-trained feature network and a powerful generator for augmentation.

Acknowledgments. This work was supported in part by the National Natural Science Foundation of China (U22A2095, 62276281).

Evaluation on Market-1501. Since none of the diffusion-based methods including PIDM [1], PoCoLD [9] and concurrent PCDMs [39] have released generated images or checkpoints on Market-1501 [58], we make fair comparisons with available GAN-based methods in Tab. 4. From these results, CFLD still outperforms across different metrics faithfully, which validates our robustness.

Ablation on classifier-free strategy. In the Tab. 5 we vary the choices of Eq.(7) on DeepFashion [21]. The results show that appropriate reinforcement of both appearance and pose information (i.e., increase guidance weights) can effectively improve the quality of generated images.

Method	Venue	FID$\mathrm{\downarrow }$	LPIPS$\mathrm{\downarrow }$	SSIM$\mathrm{\uparrow }$	PSNR$\mathrm{\uparrow }$
GAN-based Methods
GFLA	CVPR 20’	19.740	0.2815	0.2808	14.337
XingGAN	ECCV 20’	22.520	0.3058	0.3044	14.446
SPGNet	CVPR 21’	23.057	0.2777	0.3139	14.489
DPTN	CVPR 22’	18.995	0.2711	0.2854	14.521
Diffusion-based Methods
CFLD (Ours)		11.972	0.2636	0.3173	14.861
VAE Reconstructed		6.028	0.0164	0.9883	36.625
Ground Truth		4.845	0.0000	1.0000	$+\mathrm{\infty }$

Table 4: Quantitative comparisons with the state of the arts on Market-1501 [58] dataset.

Additional qualitative results. To further evaluate the generalization ability of our method, we generate person images at arbitrary poses randomly selected from the test set following in Figs. 8, 9 and 10. The results show that our method consistently generate high-quality person images while preserving the appearance in the source image. Even if the target pose differs significantly from the source image, or if invisible areas of the source image are required, the generated images are still free of distortion. With the guidance of coarse-grained prompts, our method has a high-level understanding and does not suffer from overfitting such as forcing the texture details of the source image to be aligned. On this basis, our embedded hybrid-granularity attention only supplements the necessary fine-grained appearance features, thus enabling more realistic and natural textures.

Figure 8: Additional results on arbitrary poses from the test set.

Figure 9: Additional results on arbitrary poses from the test set.

Figure 10: Additional results on arbitrary poses from the test set.

More discussion of over-fitting and biasing. Our observation is that previous diffusion-based methods would fit the spatially convolutional features of source image into noisy sample directly. But this doesn’t make sense in practice, because the texture details of source image probably shouldn’t be present in the same position of target sample, especially in the exaggerated pose transition case. Since the model is actually performing copy-and-paste, the generations are distorted and unnatural, which we call this phenomenon overfitting and lack of generalization ability.

To circumvent it, we made three efforts: 1) We introduce pre-trained text-to-image diffusion as foundation model to improve generalization ability since it has been exposed to billions of image-text pairs. This empowers the model to speculate on some regions of the target pose that are not visible in the source image. 2) Note that textual description for PGPIS task is not available. To promote efficient fine-tuning without loss of generalization, we freeze most parameters in diffusion model (98.8%) and thus forcing the proposed PRD to learn coarse-grained semantics just as what the CLIP text encoder provide. 3) To decouple the fine-grained appearance and pose information as opposed to previous approaches, we endeavour to encode the multi-scale convolutional features as bias terms into cross-attention. The multi-scale biasing would be necessary since the coarse-grained prompts learned solely by the PRD may lack the preservation of texture details, given that the conditional prompt is the same for each scale in U-Net blocks. We leave the biased queries ($𝑸$ in Eq.(4)) untrained and adopt zero convolution designs both in order to reduce the learning velocity of the HGA module thereby promoting a coarse-to-fine appearance control as stated in manuscript.

Strategy	${𝒘}_{\text{𝐩𝐨𝐬𝐞}}$	${𝒘}_{\text{𝐚𝐩𝐩}}$	FID$\mathrm{\downarrow }$	LPIPS$\mathrm{\downarrow }$	SSIM$\mathrm{\uparrow }$	PSNR$\mathrm{\uparrow }$
disabled	1.0	1.0	8.143	0.2000	0.7055	15.753
appearance only	1.0	2.0	8.334	0.1921	0.7131	16.429
pose only	2.0	1.0	7.580	0.1770	0.7256	17.611
both	2.0	2.0	6.804	0.1519	0.7378	18.235
both	3.0	3.0	7.423	0.1746	0.7250	17.706

Table 5: Ablation on classifier-free strategy.