CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching

Abstract

Diffusion models have demonstrated great success in the field of text-to-image generation. However, alleviating the misalignment between the text prompts and images is still challenging. The root reason behind the misalignment has not been extensively investigated. We observe that the misalignment is caused by inadequate token attention activation. We further attribute this phenomenon to the diffusion model’s insufficient condition utilization, which is caused by its training paradigm. To address the issue, we propose CoMat, an end-to-end diffusion model fine-tuning strategy with an image-to-text concept matching mechanism. We leverage an image captioning model to measure image-to-text alignment and guide the diffusion model to revisit ignored tokens. A novel attribute concentration module is also proposed to address the attribute binding problem. Without any image or human preference data, we use only 20K text prompts to fine-tune SDXL to obtain CoMat-SDXL. Extensive experiments show that CoMat-SDXL significantly outperforms the baseline model SDXL in two text-to-image alignment benchmarks and achieves start-of-the-art performance.

Figure 1: Current text-to-image diffusion model still struggles to produce images well-aligned with text prompts, as shown in the generated images of SDXL [36]. Our proposed method CoMat significantly enhances the baseline model on text condition following, demonstrating superior capability in text-image alignment. All the pairs are generated with the same random seed.

1 Introduction

The area of text-to-image generation has witnessed considerable progress with the introduction of diffusion models [17, 39, 38, 42, 44] recently. These models have demonstrated remarkable performance in creating high-fidelity and diverse images based on textual prompts. However, it still remains challenging for these models to faithfully align with the prompts, especially for the complex ones. For example, as shown in Fig. 1, current state-of-the-art open-sourced model SDXL [36] fails to generate entities or attributes mentioned in the prompts, e.g., the feathers made of lace and dwarfs in the top row. Additionally, it fails to understand the relationship in the prompt. In the middle row of Fig. 1, it mistakenly generates a Victorian gentleman and a quilt with a river on it.

Recently, various works propose to incorporate external knowledge from linguistics prior [40, 34] or Large Language Model (LLM) [30, 54, 59] to address the problem. However, there still exists a lack of reasonable explanations for the misalignment problem. To further explore the reason for the problem, we design a pilot experiment and study the cross-attention activation value of text tokens. As the example shown in Fig. 2(a), the gown and diploma do not appear in the image. At the same time, the activation values of these two tokens are also at a low level, compared with the concepts shown in the image (i.e., owl and cap). Besides, we also visualize the overall distribution of the token activation in Fig. 2(b). Specifically, we simulate the training process, where the pre-trained UNet [43] of SDXL [36] is used to denoise a noisy image conditioned with its text caption. We record the activation value of each text token and take the mean value across spatial dimensions.

Figure 2: Visualization of our pilot experiment. We compare the tokens’ attention activation value before and after applying our methods. (a) We visualize the text token activation and the generated images for a single case. Our method improves token activation and encourages missing concepts to appear. (b) We further analyze the overall distribution of token activation. Our method broadly enhances activation value. Note that we only select tokens with real meaning.

The distribution suggests that the activation remains at a low level during generation. Hence, we identify that the misalignment issue is caused by insufficient attention to certain text tokens. This behavior originally stems from the training paradigm of the text-to-image diffusion models: Given the text condition $c$ and the paired image $x$, the training process aims to learn the conditional distribution $p(x|c)$. However, the text condition only serves as additional information for the denoising loss. Without explicit guidance in learning each concept in the text, the diffusion model could easily overlook the condition information of text tokens.

To tackle the limitation, we propose CoMat, a novel end-to-end fine-tuning strategy for text-to-image diffusion models by incorporating image-to-text concept matching. We first generate image $\widehat{x}$ according to prompt $c$. In order to activate each concept present in the prompt, we seek to optimize the posterior probability $p(c|\widehat{x})$ given by a pre-trained image captioning model. Thanks to the image-to-text concept matching ability of the captioning model, whenever a certain concept is missing from the generated image, the diffusion model would be steered to generate it in the image. The guidance forces the diffusion model to revisit the ignored tokens and attend more to them. As shown in Fig. 2(a), our method largely enhances the token activation of ‘gown’ and ‘diploma’, and makes them present in the image. Moreover, Fig. 2(b) suggests that our method raises activation in the entire distribution. Additionally, due to the insensitivity of the captioning model in identifying and distinguishing attributes, we find that the attribute alignment still remains unsatisfactory. Hence, we introduce an entity attribute concentration module, where the attributes are enforced to be activated within the entity’s area to boost attribute alignment. Finally, a fidelity preservation part is added to preserve the generation capability of the diffusion model. Notably, our training data only includes text prompts, without the requirement of any image-text pairs or human preference data. Moreover, as an end-to-end method, we introduce no extra overheads during inference. We also show that our method is composable with methods leveraging external knowledge.

Our contributions are summarized as follows:

• We propose CoMat, a diffusion model fine-tuning strategy with an image-to-text concept matching mechanism.

• We introduce the concept matching module and entity attribute concentration module to promote concepts and attribute generation.

• Extensive quantitative and qualitative comparisons with baseline models indicate the proposed method’s superior text-to-image generation capability.

2 Related Works

2.1 Text-to-Image Alignment

Text-to-image alignment is the problem of enhancing coherence between the prompts and the generated images, which involves multiple aspects including existence, attribute binding, relationship, etc. Recent methods address the problem mainly in three ways.

Attention-based methods [6, 40, 34, 52, 2, 29] aim to modify or add restrictions on the attention map in the attention module in the UNet. This type of method often requires a specific design for each misalignment problem. For example, Attend-and-Excite [6] improves object existence by exciting the attention score of each object, while SynGen [40] boosts attribute binding by modulating distances of attention maps between modifiers and entities.

Planning-based methods first obtain the image layouts, either from the input of the user [28, 8, 23, 56, 12] or the generation of the Large Language Models (LLM) [35, 59, 51], and then produce aligned images conditioned on the layout. In addition, a few works propose to further refine the image with other vision expert models like grounded-sam [41], multimodal LLM[53], or image editing models [54, 53, 59]. Although such integration splits a compositional prompt into single objects, it does not resolve the inaccuracy of the downstream diffusion model and still suffers from incorrect attribute binding problems. Besides, it exerts nonnegligible costs during inference.

Moreover, some works aim to enhance the alignment using feedback from image understanding models. [21, 46] fine-tune the diffusion model with well-aligned generated images chosen by the VQA model [25] to strategically bias the generation distribution. Other works propose to optimize the diffusion models in an online manner. For generic rewards, [13, 4] introduce RL fine-tuning. While for differentiable reward, [11, 57, 55] propose to directly backpropagate the reward function gradient through the denoising process. Our concept matching module can be viewed as utilizing a captioner directly as a differentiable reward model. Similar to our work, [14] proposes to caption the generated images and optimize the coherence between the produced captions and text prompts. Although image captioning models are also involved, they fail to provide detailed guidance. The probable omission of key concepts and undesired added features in the generated captions both lead the optimization target to be suboptimal [20].

Figure 3: We showcase the results of our CoMat-SDXL compared with other state-of-the-art models. CoMat-SDXL consistently generates more faithful images.

2.2 Image Captioning Model

The image captioning models here refer to those pre-trained on various vision and language tasks (e.g., image-text matching, (masked) language modeling) [27, 33, 22, 47], then fine-tuned with image captioning tasks [9]. Various model architectures have been proposed [50, 60, 26, 25, 49]. BLIP [25] takes a fused encoder architecture, while GIT [49] adopts a unified transformer architecture. Recently, multimodel large language models have been flourishing [32, 66, 62, 1, 63]. For example, LLaVA [32] leverages LLM as the text decoder and obtain impressive results.

3 Preliminaries

We implement our method on the leading text-to-image diffusion model, Stable Diffusion [42], which belongs to the family of latent diffusion models (LDM). In the training process, a normally distributed noise $ϵ$ is added to the original latent code $z_0$ with a variable extent based on a timestep $t$ sampling from $\{1,\mathrm{\dots },T\}$. Then, a denoising function ${ϵ}_{\theta }$, parameterized by a UNet backbone, is trained to predict the noise added to $z_0$ with the text prompt $𝒫$ and the current latent $z_t$ as the input. Specifically, the text prompt is first encoded by the CLIP [37] text encoder $W$, then incorporated into the denoising function ${ϵ}_{\theta }$ by the cross-attention mechanism. Concretely, for each cross-attention layer, the latent and text embedding is linearly projected to query $Q$ and key $K$, respectively. The cross-attention map $A^{(i)}\in {ℝ}^{h\times w\times l}$ is calculated as $A^{(i)}=\text{Softmax}(\frac{Q^{(i)}{(K^{(i)})}^T}{\sqrt{d}})$, where $i$ is the index of head. $h$ and $w$ are the resolution of the latent, $l$ is the token length for the text embedding, and $d$ is the feature dimension. $A_{i,j}^n$ denotes the attention score of the token index $n$ at the position $(i,j)$. The denoising loss in diffusion models’ training is formally expressed as:

$${ℒ}_{\text{LDM}}={𝔼}_{z_0,t,p,ϵ\sim 𝒩(0,I)}\left[{\Vert ϵ-{ϵ}_{\theta }(z_t,t,W(𝒫))\Vert }^2\right].$$ (1)

For inference, one draws a noise sample $z_T\sim 𝒩(0,I)$, and then iteratively uses ${ϵ}_{\theta }$ to estimate the noise and compute the next latent sample.

Figure 4: Overview of CoMat. The text-to-image diffusion model (T2I-Model) first generates an image according to the text prompt. Then the image is sent to the concept matching module, attribute concentration module, and fidelity preservation module to compute the loss for fine-tuning the online T2I-Model.

4 Method

The overall framework of our method is shown in Fig. 4, which consists of three modules: Concept Matching, Attribute Concentration, and Fidelity Preservation. In Section 4.1, we illustrate the image-to-text concept matching mechanism with the image captioning model. Then, we detail the attribute concentration module for promoting attribute binding in Section 4.2. Subsequently, we introduce how to preserve the generation ability of the diffusion model in Section 4.3. Finally, we combine the three parts for joint learning, as illustrated in Section 4.4

4.1 Concept Matching

As noted in Section 1, we posit that the fundamental reason behind the misalignment issue lies in the incomplete utilization of contextual information. As a result, even though all the text conditions are presented, the diffusion model exhibits little attention on certain tokens and the corresponding concept is therefore missing in the generated image. To address this problem. our key insight is to add supervision on the generated image to detect the missing concepts. We achieve this by leveraging the image understanding capability of the image captioning model, which can accurately identify concepts not present in the generated image based on the given text prompt. With the caption model’s supervision, the diffusion model is compelled to revisit text tokens to search for ignored condition information and would assign significance to the previously overlooked text concepts for better text-image alignment.

Concretely, given a prompt $𝒫$ with word tokens $\{w_1,w_2,\mathrm{\dots },w_L\}$, we first generate an image $ℐ$ with the denoising function ${ϵ}_{\theta }$ after $T$ denoising steps. Then, a frozen image captioning model $𝒞$ is used to score the alignment between the prompt and the image in the form of log-likelihood. Therefore, our training objective aims to minimize the negative of the score, denoted as ${ℒ}_{cap}$:

$${ℒ}_{\text{cap}}=-\log (p_{𝒞}(𝒫|ℐ(𝒫;{ϵ}_{\theta })))=-\sum _{i=1}^L\log (p_{𝒞}(w_i|ℐ,w_{1:i-1})).$$ (2)

Actually, the score from the caption model could be viewed as a differential reward for fine-tuning the diffusion model. To conduct the gradient update through the whole iterative denoising process, we follow [55] to fine-tune the denoising network ${ϵ}_{\theta }$, which ensures the training effectiveness and efficiency by simply stopping the gradient of the denoising network input. In addition, noting that the concepts in the image include a broad field, a variety of misalignment problems like object existence and complex relationships could be alleviated by our concept matching module.

Figure 5: Overview of Attribute Concentration. Given a prompt, we first generate an image conditioned on it and output the cross-attention map for each token. After extracting all the entities in the prompt, we locate their areas with the segmentation model. Finally, the token-level and pixel-level loss is applied to boost the consistency between the entity mask and attention map recorded during generation.

4.2 Attribute Concentration

It has been well-reported that attribution binding is a challenging problem in text-to-image diffusion models [40]. As the example in Fig. 5 shows, which is based on the SDXL model, the attention of the word ‘red’ and ‘blue’ is mostly activated in the background, with little alignment of its corresponding object. Our concept matching module can alleviate this problem to some extent. However, bounded by the caption model’s insensitivity to object attributes, the performance improvement is limited. Here, to align the entities with their attributes from a more fine-grained view, we introduce attribute concentration to direct the attention of the entities’ textual description to focus on its area in the image.

Specifically, we first extract all the entities $\{e_1,..,e_N\}$ in the prompts. An entity can be defined as a tuple of a noun $n_i$ and its attributes $a_i$, i.e., $e_i=(n_i,a_i)$, where both $n_i$ and $a_i$ are the sets of one or multiple tokens. We employ spaCy’s transformer-based dependency parser [18] to parse the prompt to find all entity nouns, and then collect all modifiers for each noun. We manually filter nouns that are abstract (e.g., scene, atmosphere, language), difficult to identify their area (e.g., sunlight, noise, place), or describe the background (e.g., morning, bathroom, party). Given all the filtered nouns, we use them to prompt an open vocabulary segmentation model, Grounded-SAM [41], to find their corresponding regions as a binary mask $\{M^1,\mathrm{\dots },M^N\}$. It is worth emphasizing that we only use the nouns of entities, excluding their associated attributes, as prompts for segmentation, considering the diffusion model could likely assign a wrong attribute to the object. Taking the ‘suitcase’ object in Fig 5 as an example, the model mistakenly generates a red one even with the prompt ‘blue suitcase’. Consequently, if the prompt ’blue suitcase’ is given to the segmentor, it will fail to identify the entity’s region. These inaccuracies can lead to a cascade of errors in the following process.

Then, we aim to focus the attention of both the noun $n_i$ and attributes $a_i$ of each entity $e_i$ in the same region, which is denoted by a binary mask matrix $M^i$. We achieve this alignment by adding the training supervision with two objectives, i.e., token-level attention loss and pixel-level attention loss [52]. Specifically, for each entity $e_i$, the token-level attention loss forces the model to activate the attention of the object tokens $n_i\cup a_i$ only inside the region $M^i$, i.e.,:

$${ℒ}_{\text{token}}=\frac{1}{N}\sum _{i=1}^N\sum _{k\in n_i\cup a_i}\left(1-\frac{{\sum }_{u,v}{A}_{u,v}^k\odot M^i}{{\sum }_{u,v}{A}_{u,v}^k}\right),$$ (3)

where $\odot$ denotes element-wise multiplication.

The pixel-level attention loss is to further force every pixel in the region to attend only to the object tokens by a binary cross-entropy loss:

$${ℒ}_{\text{pixel}}=-\frac{1}{N|A|}\sum _{i=1}^N\sum _{u,v}\left[M_{u,v}^i\log (\sum _{k\in n_i\cup a_i}{A}_{u,v}^k)+(1-M_{u,v}^i)\log (1-\sum _{k\in n_i\cup a_i}{A}_{u,v}^k)\right],$$ (4)

where $|A|$ is the number of pixels on the attention map. Different from [52], certain objects in the prompt may not appear in the generated image due to misalignment. In this case, the pixel-level attention loss is still valid. When the mask is entirely zero, it signifies that none of the pixels should be attending to the missing object tokens in the current image. Besides, on account of the computational cost, we only compute the above two losses on $r$ randomly selected timesteps during the image generation process of the online model.

4.3 Fidelity Preservation

Since the current fine-tuning process is purely piloted by the image captioning model and prior knowledge of relationship between attributes and entities, the diffusion model could quickly overfit the reward, lose its original capability, and produce deteriorated images, as shown in Fig. 6. To address this reward hacking issue, we introduce a novel adversarial loss by using a discriminator that differentiates between images generated from pre-trained and fine-tuning diffusion models. For the discriminator ${𝒟}_{\varphi }$, we follow [58] and initialize it with the pre-trained UNet in Stable Diffusion model, which shares similar knowledge with the online training model and is expected to preserve its capability better. In our practice, this also enables the adversarial loss to be directly calculated in the latent space instead of the image space. In addition, it is important to note that our fine-tuning model does not utilize real-world images but instead the output of the original model. This choice is motivated by our goal of preserving the original generation distribution and ensuring a more stable training process.

Finally, given a single text prompt, we employ the original diffusion model and the online training model to respectively generate image latent ${\widehat{z}}_0$ and ${\widehat{z}}_0^{\prime }$. The adversarial loss is then computed as follows:

$${ℒ}_{adv}=\log \left(D_{\varphi }\left({\widehat{z}}_0\right)\right)+\log \left(1-D_{\varphi }\left({\widehat{z}}_0^{\prime }\right)\right).$$ (5)

We aim to fine-tune the online model to minimize this adversarial loss, while concurrently training the discriminator to maximize it.

4.4 Joint Learning

Here, we combine the captioning model loss, attribute concentration loss, and adversarial loss to build up our training objectives for the online diffusion model as follows:

$$ℒ={ℒ}_{\text{cap}}+\alpha {ℒ}_{\text{token}}+\beta {ℒ}_{\text{pixel}}+\lambda {ℒ}_{\text{adv}},$$ (6)

where $\alpha$, $\beta$, and $\lambda$ are scaling factors to balance different loss terms.

5 Experiment

5.1 Experimental Setup

Base Model Settings. We mainly implement our method on SDXL [42] for all experiments, which is the state-of-the-art open-source text-to-image model. In addition, we also evaluate our method on Stable Diffusion v1.5 [42] (SD1.5) for more complete comparisons in certain experiments. For the captioning model, we choose BLIP [25] fine-tuned on COCO [31] image-caption data. As for the discriminator in fidelity preservation, we directly adopt the pre-trained UNet of SD1.5.

Dataset. Since the prompt to the diffusion model needs to be challenging enough to lead to concepts missing, we directly utilize the training data or text prompts provided in several benchmarks on text-to-image alignment. Specifically, the training data includes the training set provided in T2I-CompBench [21], all the data from HRS-Bench [3], and 5,000 prompts randomly chosen from ABC-6K [15]. Altogether, these amount to around 20,000 text prompts. Note that the training set composition can be freely adjusted according to the ability targeted to improve.

Table 1: Evaluation results on T2I-CompBench. We mark the best score in blue, and the second-best score in green.

Model	Attribute Binding			Object Relationship		Complex$\mathrm{\uparrow }$
	Color $\mathrm{\uparrow }$	Shape$\mathrm{\uparrow }$	Texture$\mathrm{\uparrow }$	Spatial$\mathrm{\uparrow }$	Non-Spatial$\mathrm{\uparrow }$
PixArt-$\alpha$ [7]	0.6690	0.4927	0.6477	0.2064	0.3197	0.3433
Playground-v2 [24]	0.6208	0.5087	0.6125	0.2372	0.3098	0.3613
SD1.5 [42]	0.3758	0.3713	0.4186	0.1165	0.3112	0.3047
CoMat-SD1.5 (Ours)	0.6561	0.4975	0.6190	0.2002	0.3150	0.3476
	(+0.2983)	(+0.1262)	(+0.2004)	(+0.0837)	(+0.0038)	(+0.0429)
SDXL [36]	0.5879	0.4687	0.5299	0.2131	0.3119	0.3237
CoMat-SDXL (Ours)	0.7774	0.5262	0.6591	0.2431	0.3183	0.3696
	(+0.1895)	(+0.0575)	(+0.1292)	(+0.03)	(+0.0064)	(+0.0459)

Training Details. In our method, we inject LoRA [19] layers into the UNet of the online training model and discriminator and keep all other components frozen. For both SDXL and SD1.5, we train 2,000 iters on 8 NVIDIA A100 GPUS. We use a local batch size of 6 for SDXL and 4 for SD1.5. We choose Grounded-SAM [41] from other open-vocabulary segmentation models [64, 67]. The DDPM [17] sampler with 50 steps is used to generate the image for both the online training model and the original model. In particular, we follow [55] and only enable gradients in 5 steps out of those 50 steps, where the attribute concentration module would also be operated. Besides, to speed up training, we use training prompts to generate and save the generated latents of the pre-trained model in advance, which are later input to the discriminator during fine-tuning. More training details is shown in Appendix 0.B.1.

Benchmarks. We evaluate our method on two benchmarks:

• T2I-CompBench [21] is a benchmark for compositional text-to-image generation. It comprises 6,000 compositional text prompts from 3 categories (attribute binding, object relationships, and complex compositions) and 6 sub-categories (color binding, shape binding, texture binding, spatial relationships, non-spatial relationships, and complex compositions). It adopts the BLIP-VQA model [25], CLIP [37], and UniDet [65] to automatically evaluate the generation results. Each sub-category contains 700 prompts for training and 300 prompts for testing.

• TIFA [20] is a benchmark to evaluate text-to-image faithfulness. It uses pre-generated question-answer pairs and a VQA model to evaluate the generation results. The benchmark contains 4,000 diverse text prompts and 25,000 questions across 12 categories.

We follow the default evaluation setting of these two benchmarks.

Table 2: Evaluation results on TIFA.

Table 3: Evaluation results on FID-10K.

Model	TIFA$\mathrm{\uparrow }$
PixArt-$\alpha$ [7]	82.9
Playground-v2 [24]	86.2
SD1.5 [42]	78.4
CoMat-SD1.5 (Ours)	85.7
	(+7.3)
SDXL [36]	85.9
CoMat-SDXL (Ours)	87.7
	(+1.8)

Model	Discriminator	FID-10K$\mathrm{\downarrow }$
SD1.5 [42]	-	16.69
CoMat-SD1.5	N/A	19.02
CoMat-SD1.5	DINO [5]	23.86
CoMat-SD1.5	UNet [43]	16.69

Table 3: Evaluation results on FID-10K.

5.2 Quantitative Results

We compare our methods with our baseline models: SD1.5 and SDXL, and two state-of-the-art open-sourced text-to-image models: PixArt-$\alpha$ [7] and Playground-v2 [24]. PixArt-$\alpha$ adopts the transformer [48] architecture and leverages dense pseudo-captions auto-labeled by a large Vision-Language model to assist text-image alignment learning. Playground-v2 follows a similar structure to SDXL but is preferred 2.5x more on the generated images [24].

Table 4: The impact of different image captioning models.

Caption Model	Attribute Binding			Object Relationship		Complex$\mathrm{\uparrow }$
	Color $\mathrm{\uparrow }$	Shape$\mathrm{\uparrow }$	Texture$\mathrm{\uparrow }$	Spatial$\mathrm{\uparrow }$	Non-Spatial$\mathrm{\uparrow }$
BLIP [25]	0.7774	0.5262	0.6591	0.2431	0.3183	0.3696
GIT [49]	0.7451	0.4881	0.5893	0.1880	0.3120	0.3375
LLaVA [32]	0.6550	0.4713	0.5490	0.1835	0.3112	0.3349
N/A	0.5879	0.4687	0.5299	0.2131	0.3119	0.3237

T2I-CompBench. The evaluation result is shown in Table 1. It is important to note that we cannot reproduce results reported in some relevant works [7, 21] due to the evolution of the evaluation code. All our shown results are based on the latest code released in GitHub. We observe significant gains in all six sub-categories compared with our baseline models. Specifically, SD1.5 increases $0.2983$, $0.1262$, and $0.2004$ for color, shape, and texture attributes. In terms of object relationship and complex reasoning, CoMat-SD1.5 also obtains large improvement with over $70\%$ improvement in the spatial relationship. With our methods, SD1.5 can even achieve better or comparable results compared with PixArt-$\alpha$ and Playground-v2. When applying our method on the larger base model SDXL, we can still witness great improvement. Our CoMat-SDXL demonstrates the best performance regarding attribute binding, spatial relationships, and complex compositions. We find our method fails to obtain the best result in non-spatial relationships. We assume this is due to the training distribution in our text prompts, where a majority of prompts are descriptive phrases aiming to confuse the diffusion model. The type of prompts in non-spatial relationships may only take a little part.

TIFA. We show the results in TIFA in Table 3. Our CoMat-SDXL achieves the best performance with an improvement of 1.8 scores compared to SDXL. Besides, CoMat significantly enhances SD1.5 by 7.3 scores, which largely surpasses PixArt-$\alpha$.

5.3 Qualitative Results

Fig. 3 presents a side-by-side comparison between CoMat-SDXL and other state-of-the-art diffusion models. We observe these models exhibit inferior condition utilization ability compared with CoMat-SDXL. Prompts in Fig. 3 all possess concepts that are contradictory to real-world phenomena. All the three compared models stick to the original bias and choose to ignore the unrealistic content (e.g., waterfall cascading from a teapot, transparent violin, robot penguin, and waterfall of liquid gold), which causes misalignment. However, by training to faithfully align with the conditions in the prompt, CoMat-SDXL follows the unrealistic conditions and provides well-aligned images.

The result of user study is detailed in Appendix 0.A.1 and more visualization results are shown in Appendix 0.C.

5.4 Ablation Study

Here, we aim to evaluate the importance of each component and model setting in our framework and attempt to answer three questions: 1) Are both two main modules, i.e., Concept Matching and Attribute Concentration, necessary and effective? 2) Is the Fidelity Preservation module necessary, and how to choose the discriminator? 3) How to choose the base model for the image captioning models?

Table 5: Impact of concept matching and attribute concentration. ‘CM’ and ‘AC’ denote concept matching and attribute concentration respectively.

Model	CM	AC	Attribute Binding			Object Relationship		Complex$\mathrm{\uparrow }$
			Color $\mathrm{\uparrow }$	Shape$\mathrm{\uparrow }$	Texture$\mathrm{\uparrow }$	Spatial$\mathrm{\uparrow }$	Non-Spatial$\mathrm{\uparrow }$
SDXL			0.5879	0.4687	0.5299	0.2131	0.3119	0.3237
SDXL	✓		0.7593	0.5124	0.6362	0.2247	0.3164	0.3671
SDXL	✓	✓	0.7774	0.5262	0.6591	0.2431	0.3183	0.3696

Concept Matching and Attribute Concentration. In Table 5, we show the T2I-CompBench result aiming to identify the effectiveness of the concept matching and attribute concentration modules. We find that the concept matching module accounts for major gains to the baseline model. On top of that, the attribute concentration module brings further improvement on all six sub-categories in T2I-CompBench.

Different Discriminators. We ablate the choice of discriminator here. We randomly sample 10K text-image pairs from the COCO validation set. We calculate the FID [16] score on it to quantitatively evaluate the photorealism of the generated images. As shown in Table 3, the second row shows that without the discriminator, i.e. no fidelity perception module, the generation ability of the model greatly deteriorates with an increase of FID score from 16.69 to 19.02. We also visualize the generated images in Fig. 6. As shown in the figure, without fidelity preservation, the diffusion model generates misshaped envelops and swans. This is because the diffusion model only tries to hack the captioner and loses its original generation ability. Besides, inspired by [45], we also experiment with a pre-trained DINO [5] to distinguish the images generated by the original model and online training model in the image space. However, we find that DINO fails to provide valid guidance, and severely interferes with the training process, leading to a FID score even higher than not applying a discriminator. Using pre-trained UNet yields the best preservation. The FID score remains the same as the original model and no obvious degradation can be observed in the generated images.

Figure 6: Visualization result of the effectiveness of the fidelity preservation module. ‘FP’ denotes fidelity preservation.

Different Image Captioning Models. We show the T2I-CompBench results with different image captioning models in Table 4, where the bottom line corresponds to the baseline diffusion model without concept matching optimization. We can find that all three captioning models can boost the performance of the diffusion model with our framework, where BLIP achieves the best performance. In particular, it is interesting to note that the LLaVA [32] model, a general multimedia large language model, cannot capture the comparative performance with the other two captioning models, which are trained on the image caption task. We provide a detailed analysis in Appendix 0.A.3.

6 Limitation

How to effectively incorporate Multimodal Large Language Models (MLLMs) into text-to-image diffusion models by our proposed method remains under-explored. Given their state-of-the-art image-text understanding capability, we will focus on leveraging MLLMs to enable finer-grained alignment and generation fidelity. In addition, we also observe the potential of CoMat for adapting to 3D domains, promoting the text-to-3D generation with stronger alignment.

7 Conclusion

In this paper, we propose CoMat, an end-to-end diffusion model fine-tuning strategy equipped with image-to-text concept matching. We utilize an image captioning model to perceive the concepts missing in the image and steer the diffusion model to review the text tokens to find out the ignored condition information. This concept matching mechanism significantly enhances the condition utilization of the diffusion model. Besides, we also introduce the attribute concentration module to further promote attribute binding. Our method only needs text prompts for training, without any images or human-labeled data. Through extensive experiments, we have demonstrated that CoMat largely outperforms its baseline model and even surpasses commercial products in multiple aspects. We hope our work can inspire future work of the cause of the misalignment and the solution to it.

Appendix 0.A Additional Results and Analysis

0.A.1 User preference study

Figure 7: User preference study results.

We randomly select 100 prompts from DSG1K [10] and use them to generate images with SDXL [42] and our method (CoMat-SDXL). We ask 5 participants to assess both the image quality and text-image alignment. Human raters are asked to select the superior respectively from the given two synthesized images, one from SDXL, and another from our CoMat-SDXL. For fairness, we use the same random seed for generating both images. The voting results are summarised in Fig. 7. Our CoMat-SDXL greatly enhances the alignment between the prompt and the image without sacrificing the image quality.

0.A.2 Composability with planning-based methods

Since our method is an end-to-end fine-tuning strategy, we demonstrate its flexibility in the integration with other planning-based methods, where combining our method also yields superior performance. RPG [59] is a planning-based method utilizing Large Language Model (LLM) to generate the description and subregion for each object in the prompt. We refer the reader to the original paper for details. We employ SDXL and our CoMat-SDXL as the base model used in [59] respectively. As shown in Fig. 8, even though the layout for the generated image is designed by LLM, SDXL still fails to faithfully generate the single object aligned with its description, e.g., the wrong mat color and the missing candle. Although the planning-based method generates the layout for each object, it is still bounded by the base model’s condition following capability. Combining our method can therefore perfectly address this issue and further enhance alignment.

Figure 8: Pipeline for integrating CoMat-SDXL with planning-based method. CoMat-SDXL correctly generates the green mat in the upper row and the tall white candle in the bottom row.

0.A.3 How to choose an image captioning model?

We provide a further analysis of the varied performance improvements observed with different image captioning models, as shown in Table 5 of the main text.

Figure 9: Examples for the three core sensitivities.

For an image captioning model to be valid for the concept matching module, it should be able to tell whether each concept in the prompt appears and appears correctly. We construct a test set to evaluate this capability of the captioner. Intuitively, given an image, a qualified captioner should be sensitive enough to the prompts that faithfully describe it against those that are incorrect in certain aspects. We study three core demands for a captioner:

• Attribute sensitivity. The captioner should distinguish the noun and its corresponding attribute. The corrupted caption is constructed by switching the attributes of the two nouns in the prompt.

• Relation sensitivity. The captioner should distinguish the subject and object of a relation. The corrupted caption is constructed by switching the subject and object.

• Quantity sensitivity. The captioner should distinguish the quantity of an object. Here we only evaluate the model’s ability to tell one from many. The corrupted caption is constructed by turning singular nouns into plural or otherwise.

We assume that they are the basic requirements for a captioning model to provide valid guidance for the diffusion model. Besides, we also choose images from two domains: real-world images and synthetic images. For real-world images, we randomly sample 100 images from the ARO benchmark [61]. As for the synthetic images, we use the pre-trained SD1.5 [42] and SDXL [36] to generate 100 images according to the prompts in T2ICompBench [21]. These selections make up for the 200 images in our test data. We show the examples in Fig. 9.

For the sensitivity score, we compare the difference between the alignment score (i.e., log-likelihood) of the correct and corrupted captions for an image. Given the correct caption $𝒫$ and corrupted caption ${𝒫}^{\prime }$ corresponding to image $ℐ$, we compute the sensitivity score $𝒮$ as follows:

$$𝒮=\frac{\log (p_{𝒞}(𝒫|ℐ))-\log (p_{𝒞}({𝒫}^{\prime }|ℐ))}{\left|\log (p_{𝒞}(𝒫|ℐ))\right|}.$$ (7)

Then we take the mean value of all the images in the test set. The result is shown in Table 6. The rank of the sensitivity score aligns with the rank of the gains brought by the captioning model shown in the main text. Hence, except for the parameters, we argue that sensitivity is also a must for an image captioning model to function in the concept matching module.

Table 6: Statistics of caption models.

Caption Model	Parameters	Sensitivity Score
BLIP [25]	469M	0.1987
GIT [49]	394M	0.1728
LLaVA [32]	7.2B	0.1483

Appendix 0.B Experimental Setup

0.B.1 Implementation Details

Training Resolutions. We observe that training SDXL is very slow due to the large memory overhead at $1024\times 1024$. However, SDXL is known to generate low-quality images at resolution $512\times 512$. This largely affects the image understanding of the caption model. So we first equip the training model with better image generation capability at $512\times 512$. We use our training prompts to generate $1024\times 1024$ images with pre-trained SDXL. Then we resize these images to $512\times 512$ and use them to fine-tune the UNet of SDXL for 100 steps, after which the model can already generate high-quality $512\times 512$ images. We continue to implement our method on the fine-tuned UNet.

Training Layers for Attribute Concentration. We follow the default setting in [52]. Only cross-attention maps in the middle blocks and decoder blocks are used to compute the loss.

Hyperparameters Settings. We provide the detailed training hyperparameters in Table 7.

Table 7: CoMat training hyperparameters for SD1.5 and SDXL.

Name	SD1.5	SDXL
Online training model
Learning rate	5e-5	2e-5
Learning rate scheduler	Constant	Constant
LR warmup steps	0	0
Optimizer	AdamW	AdamW
AdamW - ${\beta }_1$	0.9	0.9
AdamW - ${\beta }_2$	0.999	0.999
Gradient clipping	$0.1$	$0.1$
Discriminator
Learning rate	5e-5	5e-5
Optimizer	AdamW	AdamW
AdamW - ${\beta }_1$	$0$	$0$
AdamW - ${\beta }_2$	$0.999$	$0.999$
Gradient clipping	$1.0$	$1.0$
Token loss weight $\alpha$	1e-3	1e-3
Pixel loss weight $\beta$	5e-5	5e-5
Adversarial loss weight $\lambda$	1	5e-1
Gradient enable steps	$5$	$5$
Attribute concentration steps $r$	$2$	$2$
LoRA rank	128	128
Classifier-free guidance scale	7.5	7.5
Resolution	$512\times 512$	$512\times 512$
Training steps	2,000	2,000
Local batch size	$4$	$6$
Mixed Precision	FP16	FP16
GPUs for Training	8 $\times$ NVIDIA A100	8 $\times$ NVIDIA A100
Training Time	$\sim$ 10 Hours	$\sim$ 24 Hours

Appendix 0.C More Qualitative Results

We showcase more results of comparison between our method with the baseline model in Fig. 10 to 12.

Figure 10: More Comparisons between SDXL and CoMat-SDXL. All pairs are generated with the same random seed.

Figure 11: More Comparisons between SDXL and CoMat-SDXL. All pairs are generated with the same random seed.

Figure 12: More Comparisons between SDXL and CoMat-SDXL. All pairs are generated with the same random seed.