RAMM: Retrieval-augmented Biomedical Visual Question Answering
with Multi-modal Pre-training

Abstract

Vision-and-language multi-modal pretraining and fine-tuning have shown great success in visual question answering (VQA). Compared to general domain VQA, the performance of biomedical VQA suffers from limited data. In this paper, we propose a retrieval-augmented pretrain-and-finetune paradigm named RAMM for biomedical VQA to overcome the data limitation issue. Specifically, we collect a new biomedical dataset named PMCPM which offers patient-based image-text pairs containing diverse patient situations from PubMed. Then, we pretrain the biomedical multi-modal model to learn visual and textual representation for image-text pairs and align these representations with image-text contrastive objective (ITC). Finally, we propose a retrieval-augmented method to better use the limited data. We propose to retrieve similar image-text pairs based on ITC from pretraining datasets and introduce a novel retrieval-attention module to fuse the representation of the image and the question with the retrieved images and texts. Experiments demonstrate that our retrieval-augmented pretrain-and-finetune paradigm obtains state-of-the-art performance on Med-VQA2019, Med-VQA2021, VQARAD, and SLAKE datasets. Further analysis shows that the proposed RAMM and PMCPM can enhance biomedical VQA performance compared with previous resources and methods. We will open-source our dataset, codes, and pretrained model.

1 Introduction

Figure 1: Example of a similar image whose text includes helpful information for answering biomedical questions. The left part is a biomedical-related question with a provided image. The right part is a similar image in PubMed with a relevant caption that indicates the answer (text in red) to the question.

Biomedical visual question answering (VQA) aims to answer a biomedical-related question with a provided image. The images in the biomedical VQA are usually radiology images (e.g. CT, MRI, and X-Ray), and the questions usually involve modalities, body parts, planes, and abnormal parts in corresponding images. Answering these questions requires rich knowledge of clinical and medical imaging [17], which is challenging for automated approaches.

With the success of the general domain VQA [27, 10, 26], some works follow the pretrain-and-finetune paradigm to attempt domain-specific vision-and-language modeling for biomedical VQA [20, 4]. However, unlike the general domain that adequates labeled VQA pairs for fine-tuning, the biomedical domain still suffers from limited data in fine-tuning, which makes models vulnerable to overfitting and hampers the models’ ability to learn comprehensive domain knowledge for answering complicated questions.

In contrast to limited data in biomedical VQA datasets, biomedical literature contains abundant and high-quality image-text pairs. These texts usually describe the images in all aspects including modality, body part, plane, and abnormality which are the information in which biomedical VQA is interested. Figure 1 shows an example that the image-text pair from PubMed can benefit to answer the question in the VQA dataset. Based on this observation, we argue that retrieval is a feasible option for biomedical VQA. This procedure simulates how doctors solve complicated patient cases: doctors search for similar cases to the provided one and propose a solution based on previous experience. Therefore, we can retrieve similar image-text pairs from literature to help answer the question. To retrieve related image-text pairs from biomedical literature, there are three key questions to be answered: Where to retrieve? How to retrieve? and How to leverage the retrieved image-text pairs?

To this end, we propose a Retrieval-Augmented bioMedical Multi-modal pretrain-and-finetune paradigm (RAMM) for biomedical VQA. Firstly, there are existing biomedical image-text pairs including ROCO [36] and MIMIC-CXR [18] that can be used for retrieval. However, ROCO is limited by quantity and MIMIC-CXR only focuses on chest X-Ray. A massive, diverse, and high-quality dataset is required for better retrieval. Therefore, we propose PMC-Patients-Multi-modal (PMCPM), a patient-based image-text dataset filtered from PubMed Central, which is larger than ROCO and MIMIC-CXR with various modalities and conditions of images. Secondly, since questions are homogenous in biomedical VQA, we propose to use images for retrieving. We pretrain the biomedical multi-modal model with three tasks: masked language modeling, image-text contrastive learning, and image-text matching using the combination of our PMCPM dataset and previous widely-used ROCO and MIMIC-CXR datasets. As the image-text contrastive objective in the pretraining phase aims to align visual and textual representations in the same embedding space [37, 27], the image-text contrastive similarity can be used to retrieve similar pairs. Finally, taking the above datasets as resources, we propose to retrieve the related image-text pair to the given image. To leverage the retrieved pairs, we propose a retrieval-augmented fine-tuning method with a novel retrieval-attention module in the multi-modal encoder layer and apply the fused representation to predict the final answer.

We conduct experiments on four biomedical VQA datasets. Our RAMM achieves state-of-the-art results of 82.13, 39.20, 78.27, and 86.05 scores on VQA-Med 2019 [2], VQA-Med 2021 [16], VQARAD [22], and SLAKE (English Version) [30], respectively. We also conduct ablation studies to verify the effectiveness of our collected PMCPM dataset and the retrieval-augmented fine-tuning methods. Case studies show the retrieved image-text pairs benefit to answer the question.

We summarize our contributions as follows:

• We collect PMCPM, a patient-based image-text dataset from PubMed which can be used for biomedical multi-modal pretraining and retrieving.

• We propose RAMM, a retrieval-augmented multi-modal pretrain-and-finetune paradigm for biomedical VQA with a novel retrieval-attention module.

• Experiments show that the proposed RAMM with additional PMCPM dataset outperforms previous VQA methods on VQA-Med 2019, VQA-Med 2021, VQARAD, and SLAKE datasets.

2 Related Work

Biomedical Multi-modal Pretraining

In the general domain, multi-modal pretraining models show effectiveness in various image-text-related downstream tasks [27, 10, 26, 41]. In the biomedical NLP community, previous research has shown that further continue training in the in-domain corpus can enhance the performance of pretrained language models [1, 23, 14, 45, 44]. Combining the above-mentioned two points, it is direct to consider pretraining a biomedical in-domain multi-modal model for biomedical VQA. Previous biomedical pretrained multi-modal models [11, 20, 34, 4, 5] mainly leverage ROCO [36] and MIMIC-CXR [18] as pretraining datasets. These models predict masked tokens [4, 5] or image patches [4] and conduct image-text alignment [11] as pretraining tasks. Compared with previous models, our RAMM is pretrained with our additional proposed PMCPM dataset which contains images from diverse conditions of patients. For pretraining tasks, our model adopts masked language modeling, image-text contrastive learning, and image-text matching. Image-text contrastive similarities are used for retrieving in RAMM.

Biomedical Visual Question Answering

VQA requires answering an image-related question. Different from general domain VQA [27, 10, 26], the training samples of biomedical VQA are much fewer. To overcome the data limitation in biomedical VQA, meta-learning is adapted for quick fitting to VQA questions [35, 9]. With the success of the pretrain-and-finetune paradigm, pretraining a multi-modal model with large-scale unsupervised datasets learns better visual and textual representations which mitigate the impact of data limitation and boosts biomedical VQA performances significantly [3, 11, 4]. In this paper, we also pretrain a biomedical multi-modal model RAMM with patient-based image-text pairs PMCPM. While fine-tuning VQA datasets, RAMM retrieves similar image-text pairs and tries to leverage useful information from these images and texts.

Retrieval Augmentation

has been applied successfully in NLP [24, 25, 15] and CV [32, 39] for knowledge-intensive tasks. In this paper, we decide to use retrieval augmentation to solve the knowledge-intensive and under-labeled biomedical VQA tasks. A retriever may be sparse like BM-25 [38, 24] or dense using pretrained encoders [15, 39]. For our task, the model pretrained by the image-text contrastive (ITC) task can be naturally used as a multi-modal retriever. Thus, we pretrain a biomedical multi-modal model by ITC for retrieval augmentation. We also propose a retrieval-attention module to fuse these retrieved samples. To the best of our knowledge, this is the first work to leverage retrieval augmentation in biomedical VQA.

Figure 2: The workflow of our proposed retrieval-augmented paradigm RAMM for biomedical VQA. The origin image is from VQA datasets, and the other images are retrieved. Blue lines represent retrieving image-text pairs (PMCID:4061445, 2278144, 4725742, and 3616602) from PMCPM, ROCO, and MIMIC-CXR datasets based on image-text contrastive similarities. Retrieval-augmented samples are sent into uni-modal and multi-modal encoders. Multi-modal encoders contain retrieval-attention to fuse retrieved information. Representations obtained by multi-modal encoders are used for VQA classification.

3 Approach

In this section, we first introduce how we build the PMCPM dataset. We then describe the model architecture and pretraining tasks. Finally, we introduce how RAMM retrieves image-text pairs from pretraining datasets, and how we combine retrieved samples during biomedical VQA fine-tuning. Our overall workflow is shown in Figure 2.

3.1 PMCPM Dataset

To obtain domain-specific image-text pairs of high quality and large scale, we first consider using the Open Access subset of PubMed Central (PMC OA), a free biomedical literature archive of over $4$M publications with both full-text (including figure captions) and figures in these articles publicly available through the official FTP service. However, the figures in PMC OA full set can be quite noisy: apart from clinical images, many kinds of graphs appear in a research article, including flowcharts, histograms, and scatter plots, which provide little information for multi-modal pretraining. Pelka et al. [36] use neural networks trained on human annotated datasets to automatically filter compound, multi-pane, and non-radiology images, collecting a clean dataset with 81k image-text pairs. To further explore and utilize the figure resource in PMC, we instead resort to a “patient-based” collection pipeline inspired by Zhao et al. [47]. “Case Report” is a specific type of medical literature, generally describing the whole admission of a patient. Following Zhao et al. [47], we use regular expressions to identify sections containing patient notes, extract the corresponding texts, and apply various filters to remove noises, resulting in a collection of 167k patient notes with high-quality and diverse conditions. The figures accompanied by these patient notes are closely related to the patient’s clinical conditions and contain richer information than solely radiology images. We thus collect these figures with corresponding captions to build our PMCPM dataset. Figure 3 shows two examples of our PMCPM (PMCID: 509249) with the images and their corresponding captions. Figure 3A shows a photomicrograph describing the presence of adenoid cystic carcinoma with cribriform pattern and perineurial invasion. Figure 3B shows a radiology image that multiple metastatic lesions scatter in both lobes. The examples show that our patient-based collecting method can target clinical-related figures with more image modality and detailed text descriptions.

Figure 3: Examples of image-text pairs from PMCPM.

3.2 Model Architecture

Given an image-text pair $(I,T)$, RAMM first obtains uni-modal features independently and then fuses them for multi-modal representations. Fused representations are used for VQA classification. RAMM consists of a text encoder, an image encoder, and a multi-modal encoder for fusing two different modal representations. The text $T$ is tokenized into subwords $\{x_{cls},x_1,\mathrm{\dots },x_n\}$ and transformed into hidden representations ${𝐰}_0=\{{𝐰}_{cls},{𝐰}_1,\mathrm{\dots },{𝐰}_n\}$ by the text encoder. The image $I$ is divided into patches $\{y_{cls},y_1,\mathrm{\dots },y_m\}$ and represented as ${𝐯}_0=\{{𝐯}_{cls},{𝐯}_1,\mathrm{\dots },{𝐯}_m\}$ by the image encoder. The multi-modal encoder is a co-attention style dual-stream transformer.

Multi-modal Fusion Layer

We apply co-attention style multi-modal fusion layers [10] upon text features ${𝐰}_0$ and visual features ${𝐯}_0$ with layer count $L$. Specifically, co-attention fusion layers consist of dual-stream transformer layers where one stream corresponds to one modal. Each transformer layer is combined with a self-attention module and a cross-attention module. For layer $i$, we denote text hidden representations as ${𝐰}_i=\{{𝐰}_{cls}^i,{𝐰}_1^i,\mathrm{\dots },{𝐰}_n^i\}$ and image hidden representations as ${𝐯}_i=\{{𝐯}_{cls}^i,{𝐯}_1^i,\mathrm{\dots },{𝐯}_m^i\}$. The attention module in transformers is defined as:

$$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}(\frac{Q{K}^{\top }}{\sqrt{d_k}})V$$ (1)

Using this denotation, self-attention is defined as:

$$Q^{i,w}=W_Q^w{𝐰}_i,K^{i,w}=W_K^w{𝐰}_i,V^{i,w}=W_V^w{𝐰}_i$$ (2)

$$Q^{i,v}=W_Q^v{𝐯}_i,K^{i,v}=W_K^v{𝐯}_i,V^{i,v}=W_V^v{𝐯}_i$$ (3)

$${𝐰}_i^{\prime }=\mathrm{Attn}(Q^{i,w},K^{i,w},V^{i,w})$$ (4)

$${𝐯}_i^{\prime }=\mathrm{Attn}(Q^{i,v},K^{i,v},V^{i,v})$$ (5)

We reuse the notation of $Q^{i,\{w,v\}},K^{i,\{w,v\}},V^{i,\{w,v\}}$ by replacing ${𝐰}_i$ with ${𝐰}_i^{\prime }$ and ${𝐯}_i$ with ${𝐯}_i^{\prime }$, and cross-attention is defined as:

$${𝐰}_{i+1}=\mathrm{Attn}(Q^{i,w},K^{i,v},V^{i,v})$$ (6)

$${𝐯}_{i+1}=\mathrm{Attn}(Q^{i,v},K^{i,w},V^{i,w})$$ (7)

The fused representations of text and image are denoted as ${𝐰}_L$ and ${𝐯}_L$.

3.3 Pretraining Tasks

To pretrain RAMM, we employ three pretraining tasks including Image-Text Contrastive learning (ITC) [27], Image-Text Matching (ITM) [27], and Masked Language Modeling (MLM) [8]. The pretraining objective function of RAMM is defined by the sum of these three tasks. ITC aligns uni-modal visual and textual representations $𝐰=g_w({𝐰}_{cls})$ and $𝐯=g_v({𝐯}_{cls})$ in the same embedding space, where $g_w$ and $g_v$ are two linear projections with $L^2$ normalization. ITM predicts whether one image and one text match each other based on fused representations ${𝐰}_L$ and ${𝐯}_L$. MLM predicts masked token in text $T$ based on fused textual representations ${𝐰}_L$. Since ITC aims to align images and texts with cosine similarities, it becomes a natural choice to retrieve related images and texts.

3.4 Retrieval-augmented Fine-tuning

We use image-text contrastive similarity to retrieve related image-text pairs and fuse them with a retrieval-attention module in fine-tuning.

Retrieve related image text pairs

Let $𝒟=\{(I_j,T_j)|j\in 𝒜\}$ denotes the set of all image-text pairs, where $𝒜$ is an index set. For image-question pair $(I,T)$ from VQA datasets, we retrieve $r$ augmented samples from $𝒟$ based on uni-modal image representations ${\text{v}}_{cls}$. Here we do not use the representations from question $T$ to retrieve since questions in VQA datasets are homogeneous and non-informative. With different images, questions can be similar to What is the abnormality in this image? or Is there an abnormality in this image? For image-text pair $(I_j,T_j)\in 𝒟$ with uni-modal representations ${\text{v}}_{cls}^j$ and ${\text{w}}_{cls}^j$, we calculate the similarities between image $I$ using ITC similarity scores:

$$s_w^j=g_v{({𝐯}_{cls})}^{\top }{g}_w({𝐰}_{cls}^j)$$ (8)

$$s_v^j=g_v{({𝐯}_{cls})}^{\top }{g}_v({𝐯}_{cls}^j)$$ (9)

ITC optimizes $g_v{({𝐯}_{cls}^j)}^{\top }{g}_w({𝐰}_{cls}^j)\to 1$ while pretraining, and this leads $s_w^j$ and $s_v^j$ to be close. So we use the maximum of $s_w^j$ and $s_v^j$ to define the similarity between $(I_j,T_j)$ with $I$:

$$s_j=\max (s_w^j,s_v^j)$$ (10)

We freeze the uni-modal encoders while calculating $s_j$. The first reason is that ITC scores may change dramatically while fine-tuning. The second reason is representations from the pretraining dataset can be pre-computed, and similarity calculation can be conducted by Faiss [19] quickly.

We first calculate top-$r$ $\{s_w^j|j\in {ℛ}_w\}$ among $\{s_w^j|j\in 𝒜\}$ and top-$r$ $\{s_v^j|j\in {ℛ}_v\}$ among $\{s_v^j|j\in 𝒜\}$ using Faiss. ${ℛ}_w\cup {ℛ}_v$ contains $[r,2r]$ image-text pairs. When training, we randomly select $r$ image-text pairs from ${ℛ}_w\cup {ℛ}_v$. The probabilities of each image-text pair are rational to similarities $s^j$. When inference, we select top-$r$ similar image-text pairs from ${ℛ}_w\cup {ℛ}_v$ based on $s^j$. We denote retrieved $r$ image-text pairs as $\{(I_j,T_j)|\mathit{1}\le j\le r\}$.

Retrieval fusion in multi-modal encoder

To fuse the knowledge of retrieved samples, we propose a retrieval-attention module in the multi-modal fusion layer. The retrieval-attention module is applied after the self-attention and cross-attention in each multi-modal fusion layer. Figures 4 shows the model architecture of retrieval-attention applied to text modality. Specifically, we use uni-modal encoders encode retrieved image-text pairs $\{(I_j,T_j)|\mathit{1}\le j\le r\}$ to $\{({𝐰}_0^j,{𝐯}_0^j)|1\le j\le r\}$. We denote the $({𝐰}_0^0,{𝐯}_0^0)=({𝐰}_0,{𝐯}_0)$ for symbol unification. For $i^{th}$ multi-modal fusion layer, inputs of text hidden representations are $\{{𝐰}_i^j|0\le j\le r\}$ and image hidden representations are $\{{𝐯}_i^j|0\le j\le r\}$. By applying self-attention and cross-attention defined in Section 3.2, we obtain text representations $\{𝐰_i^j{}_{}{}^{\prime }|0\le j\le r\}$ and image representations $\{𝐯_i^j{}_{}{}^{\prime }|0\le j\le r\}$. Retrieval-attention is applied to each modal independently. Takes text representations as an example, retrieval-attention uses the $[\mathrm{CLS}]$ representations of $𝐰_i^0{}_{}{}^{\prime }$ as attention queries and $[\mathrm{CLS}]$ representations of $\{𝐰_i^j{}_{}{}^{\prime }|0\le j\le r\}$ as attention keys and values. The outputs of retrieval attention are only added to original representations $𝐰_i^0{}_{}{}^{\prime }$ but not retrieved representations $\{𝐰_i^j{}_{}{}^{\prime }|1\le j\le r\}$.

$$Q^{i,r}=W_Q^r𝐰_{i,cls}^0{}_{}{}^{\prime }$$ (11)

$$K^{i,r}=W_K^r[𝐰_{i,cls}^0{}_{}{}^{\prime },𝐰_{i,cls}^1{}_{}{}^{\prime },\mathrm{\dots },𝐰_{i,cls}^r{}_{}{}^{\prime }]$$ (12)

$$V^{i,r}=W_V^r[𝐰_{i,cls}^0{}_{}{}^{\prime },𝐰_{i,cls}^1{}_{}{}^{\prime },\mathrm{\dots },𝐰_{i,cls}^r{}_{}{}^{\prime }]$$ (13)

(14)

The retrieval-attention on image modality is calculated similarly.

$$Q^{i,r}=W_Q^r𝐯_{i,cls}^0{}_{}{}^{\prime }$$ (15)

$$K^{i,r}=W_K^r[𝐯_{i,cls}^0{}_{}{}^{\prime },𝐯_{i,cls}^1{}_{}{}^{\prime },\mathrm{\dots },𝐯_{i,cls}^r{}_{}{}^{\prime }]$$ (16)

$$V^{i,r}=W_V^r[𝐯_{i,cls}^0{}_{}{}^{\prime },𝐯_{i,cls}^1{}_{}{}^{\prime },\mathrm{\dots },𝐯_{i,cls}^r{}_{}{}^{\prime }]$$ (17)

(18)

Figure 4: The architecture of retrieval-attention for text modality.

We perform retrieval-attention on original representations $𝐰_i^0{}_{}{}^{\prime }$ or $𝐯_i^0{}_{}{}^{\prime }$ to fuse additional information from $\{𝐰_i^j{}_{}{}^{\prime }|1\le j\le r\}$ or $\{𝐯_i^j{}_{}{}^{\prime }|1\le j\le r\}$. We do not apply it to retrieved samples since we do not need to exchange information among them. We use the final layer of origin samples representations $({𝐰}_L^0,{𝐯}_L^0)$ for VQA classification.

4 Experiments

4.1 Dataset

We conduct experiments on four biomedical VQA datasets as VQA-Med 2019 [2], VQA-Med 2021 [16], VQARAD [22], and SLAKE (English Version) [30]. To fairly compare with previous works, we follow the same data split with Chen et al. [4] for VQA-Med 2019, VQARAD, and SLAKE, as well as Gong et al. [13] for VQA-Med 2021. We use accuracy to measure model performance. Detailed statistics of these datasets are listed in Appendix A.

4.2 Baselines

We compare RAMM with non-pretrained methods [43, 42, 21, 13, 7] and pre-trained methods [3, 29, 46] in biomedical VQA. We introduce the following methods since they are more related to us:

PubmedCLIP [11] fine-tunes CLIP with ROCO to obtain the biomedical image and text representations.

MMBert [20] feeds CNN features of images with masked tokens to apply masked language model pretraining.

MTPT [12] proposes a multi-task pretraining with an image understanding task and a question-image compatibility task.

M3AE [4] pretrains a biomedical multi-modal model by reconstructing masked tokens and images.

4.3 Implementation Details

Pre-training Data

As introduced in Section 3.1, we collect 398k biomedical image-text pairs of publication quality and various types from PMC OA as PMCPM. Then, we combine the PMCPM with two widely-used biomedical pretrained datasets ROCO [36] and MIMIC-CXR [18] for model pre-training. Following the previous work, we only use their train splits as pretraining datasets and only use images in frontal view for MIMIC-CXR. The statistics of training data are shown in Table 1.

Dataset	Count	Sources
ROCO	70,306	PubMed
MIMIC-CXR	225,829	BIDMC
PMCPM	398,109	Case Report

Table 1: Pretraining datasets statistics, where count reports the number of image-text pairs in their train splits. BIDMC is the abbreviation of Beth Israel Deaconess Medical Center.

Pre-training Setting

For the uni-modal encoders, we initialize the text encoder by PubmedBert [14] and the image encoder by Swin-Base [31]. For the multi-modal encoder, we apply a 6-layer Transformer [40] with hidden dimension $d=768$ and head count $n_{head}=12$ with random initialization. Images are transformed into resolutions $224\times 224$ with RandAugment [6]. We apply the momentum distillation [27] with the momentum parameter as 0.995. The model is trained for 30 epochs with a batch size of 512. We use AdamW [33] to optimize our models with a linear learning rate decay. For the detailed pretraining hyper-parameters, please refer to Appendix B.

Fine-tuning Setting

Following previous works, we regard biomedical VQA as a classification problem and optimize the model by the cross-entropy loss. For all datasets, we fine-tune our pretrained models with 100 epochs. We use frozen uni-modal encoders of RAMM to retrieve image-text pairs as mentioned in Section 3.4. We retrieve $r=4$ pairs for each instance from the combination of ROCO, MIMIC-CXR, and PMCPM. Images are transformed into resolutions $224\times 224$ while fine-tuning. In VQA-Med 2019, VQARAD, and SLAKE, we center crop images, while we use RandAugment [6] in VQA-Med 2021. Exponential moving average $\theta =0.999$ and R-Drop [28] $\alpha =0.6$ are also used while fine-tuning. We list detailed hyper-parameters in Appendix C.

Model	VQAMed2019	VQARAD	SLAKE
No Pretraining
MFB [43]	-	50.60	73.30
SAN [42]	-	54.30	76.00
BAN [21]	-	58.30	76.30
With Pretraining
MEVF-BAN [3]	77.86	66.10	78.60
CPRD-BAN [29]	-	67.80	81.10
COND-RE [46]	-	71.60	-
PubmedCLIP [11]	-	72.10	80.10
MMBert [20]	77.90	72.00	-
MTPT [12]	-	73.20	-
M3AE [4]	79.87	77.01	83.25
RAMM	82.13	78.27	86.05
w/o retrieval	80.27	76.72	85.58

Table 2: Results on the VQA-Med 2019, VQARAD, and SLAKE dataset.

Model	VQA-Med 2021
VilMedic * [7]	37.80
SYSU-HCP * [13]	38.20
RAMM	39.20

Table 3: Results on the VQA-Med 2021. * indicates ensembled results.

4.4 Main Results

We compare our approach with baseline methods in four datasets. As shown in Table 2, methods with pretrained models outperform non-pretrained models by a large margin. Our proposed RAMM achieves 82.13, 78.27, and 86.05 on VQA-Med 2019, VQARAD, and SLAKE respectively, which exceeds the previous best result by 1.86, 1.26, and 2.80 respectively. Table 4 shows results on the VQA-Med 2021 dataset. Our RAMM achieves 39.20 in terms of accuracy, which even outperforms the state-of-the-art ensemble methods with 1.00 in the VQA-Med 2021 Challenge.

We also report the result of our RAMM without retrieving additional image-text pairs on the VQA-Med 2019, VQARAD, and SLAKE datasets. As shown in Table 2, it achieves 80.27, 76.72, and 85.58 on VQA-Med 2019, VQARAD, and SLAKE respectively. Without retrieval augmentation, RAMM still outperforms the previous state-of-the-art method M3AE on VQA-Med 2019 with 0.4 as well as SLAKE with 2.33 and performs on par on VQARAD (76.72 v.s. 77.01). This indicates that the pretraining with additional PMCPM dataset learns better visual and textual representations for biomedical VQA tasks. Compared with models without retrieval augmentation, we observe that our RAMM achieves consistent improvements of 1.90, 1.55, 0.47 on VQA-Med 2019, VQARAD, and SLAKE respectively, which demonstrates that our proposed retrieval augmentation can boost biomedical VQA performances.

	Pre-training		Retrieval		VQA Datasets
	PMCPM	ROCO+CXR	PMCPM	ROCO+CXR	VQAMed19	VQARAD	SLAKE
(a)	✗	✗	✗	✗	77.60	74.06	81.43
(b)	✓	✗	✗	✗	78.40	74.94	83.60
(c)	✗	✓	✗	✗	79.73	75.17	82.94
(d)	✓	✓	✗	✗	80.27	76.72	85.58
(e)	✗	✓	✗	✓	80.00	76.72	85.20
(f)	✓	✓	✗	✓	81.07	77.16	85.67
(g)	✓	✓	✓	✓	82.13	78.27	86.05

Table 4: Results of RAMM on the VQA-Med 2019, VQARAD, and SLAKE with different pretraining and retrieval settings.

Task	VQAMed19		VQARAD		SLAKE
Dataset	Retrieve%	Have Ans%	Retrieve%	Have Ans%	Retrieve%	Have Ans%
PMCPM + ROCO + MIMIC-CXR	100.0%	29.6%	100.0%	28.4%	100.0%	38.7%
PMCPM	52.2%	20.8%	51.9%	21.3%	54.9%	30.5%
ROCO + MIMIC-CXR	47.8%	21.1%	48.1%	20.0%	45.1%	26.9%

Table 5: Analysis of the source and quality of retrieving on VQA-Med 2019, VQARAD, and SLAKE datasets.

4.5 Ablation Study

To further analyze the influence of the collected PMCPM dataset and the retrieval strategy, we conduct detailed ablation studies as shown in Table 4.

Firstly, we analyze the impact of using different data during the pre-training phase. ROCO and MIMIC-CXR are well-used biomedical multi-modal pre-training datasets. ROCO contains labeled radiology images in relatively small amounts. The amount of MIMIC-CXR is large, while it only contains chest X-Ray image-text pairs. The collection of PMCPM comes from the rich and diverse conditions of patients, and PMCPM is much larger than ROCO and MIMIC-CXR. With the numerous and diverse image-text pairs from PMCPM, the model has the potential to obtain better visual and textual representations. Comparing (a) and (b), we show the effectiveness of pretraining on our collected PMCPM dataset. Results from (b) and (c) show that the data of PMCPM and previous ROCO plus MIMIC-CXR can complement each other. ROCO plus MIMIC-CXR shows better results on VQA-Med-2019 and VQARAD datasets, while PMCPM is better on SLAKE dataset. By combining the two settings with three resources, setting (d) achieves the best performance with a large margin by only using either of them. The above results demonstrate that PMCPM can enhance the model pretraining in biomedical VQA tasks.

Secondly, the above datasets are also treated as retrieval resources in this work. Therefore, we also analyze the impact of using different data during the fine-tuning phase. Setting (e) only uses the previous ROCO and MIMIC-CXR datasets for retrieval. By comparing it with setting (c), we show that the proposed retrieval strategy is effective in the previously widely-used resource. Then, we further demonstrate the usefulness of our collected PMCPM data. Compared with settings (f) and (g), we observe that retrieving additional PMCPM data yields gains of 1.06, 1.11, and 0.38, respectively. In addition, results from setting (g) with (d) reflect the great improvement brought by our proposed retrieval method. The above ablation studies show that our collected PMCPM dataset and our proposed retrieval method can benefit both pretraining and retrieving in biomedical VQA tasks.

4.6 Analysis and Discussion

4.6.1 Retrieval

Retrieval distribution

To understand how RAMM retrieve from different resources and how retrieved texts help our model for biomedical VQA tasks, we count the sources of retrieved samples and how many retrieved texts contain the answers. Statistics are listed in Table 5. There are 29.6%, 28.4%, and 38.7% of answers that can be found in retrieved texts in VQA-Med 2019, VQARAD, and SLAKE respectively, which are considerable amounts. These retrieved texts may help RAMM answer the questions directly. We observe that in these tasks, RAMM retrieves image-text pairs from PMCPM over half which shows that PMCPM is more related to biomedical VQA images than ROCO and MIMIC-CXR. The percentages of retrieved texts containing answers are nearly the same for PMCPM and out of PMCPM in VQA-Med 2019 and VQARAD. In SLAKE, PMCPM contributes more texts containing answers. To conclude, PMCPM is important during retrieval which supplies additional texts containing answers.

# Retrieval	VQARAD	SLAKE
0	76.72	85.58
1	77.38	86.33
2	78.05	86.33
4	78.27	86.05
8	76.50	84.54

Table 6: Results on the VQARAD and SLAKE datasets with different $r$ of retrieved image-text pairs.

Model	VQARAD		SLAKE
	Open	Closed	Open	Closed
MEVF-BAN	49.20	77.20	77.80	79.80
CPRD-BAN	52.50	77.90	79.50	83.40
M3AE	67.23	83.46	80.31	87.82
RAMM	67.60	85.29	82.48	91.59

Table 7: Analysis of open-ended and closed-ended results on the VQARAD and SLAKE datasets.

Image-text pairs retrieval count

To explore how image-text pairs retrieval count influences biomedical VQA fine-tuning, we search the number of retrieved images among $\{0,1,2,4,8\}$. Results for VQARAD and SLAKE are shown in Table 6. Compared with vanilla fine-tuning (i.e. retrieval count 0), retrieving $\{1,2,4\}$ image-text pairs can boost performances. We empirically find retrieving 2 or 4 image-text pairs work well. Furthermore, using 8 image-text pairs downgrades the model performance, which indicates that retrieving too many image-text pairs may introduce unrelated image-text pairs with noisy knowledge.

4.6.2 Question Type

In biomedical VQA, questions can be divided into closed-ended and open-ended questions. The answers to closed-ended questions are limited to multiple choices like Yes or No, while the answers to open-ended questions are not limited. Closed-ended questions are easier than open-ended questions due to the form. Table 7 lists comparisons among different methods on open-ended and closed-ended questions. RAMM performs better than M3AE on both closed-ended and open-ended questions. RAMM improves M3AE on closed-ended questions more than open-ended questions. The reason is that closed-ended questions are limited in answer choices, and are easier to be covered by retrieved texts.

4.6.3 Comparison with ROCO

ROCO and PMCPM are both subsets of PubMed. ROCO uses a trained neural network to filter radiology images, while PMCPM finds images by “Case Report” from PubMed. We calculate the involved article count in ROCO and PMCPM train split. ROCO uses 40,857 articles from PubMed, and PMCPM uses 127,455 articles from PubMed. There are 17,189 articles appeared in both datasets. Although ROCO and PMCPM filter image-text pairs using different methods, nearly half of the pairs in ROCO are also collected by PMCPM. Considering ROCO is a manually corrected dataset, it can reflect the quality of the larger dataset PMCPM from the side.

4.6.4 Case Study

We show retrieved image-text pairs of RAMM in Figure 5. The retrieved samples are all from PMCPM. Retrieved images are in the same modalities and are the same organs as the original images. These cases show that the retrieved images contain useful information for answering questions. Although the answer (i.e. Yes/No) is not covered in retrieved text for the second example, the retrieved text is still useful for answering. The second and third examples further show that RAMM can answer correctly with retrieval augmentation while naive fine-tuning fails.

Figure 5: Examples of retrieved image-text pairs of RAMM.

5 Conclusion

In this paper, we present a retrieval-augmented pretrain-and-finetune paradigm RAMM for biomedical VQA which includes a high-quality image-text pairs PMCPM, a pretrained multi-modal model, and a novel retrieval-augmented attention module for fine-tuning. RAMM uses ITC to retrieve related image-text pairs and fuse retrieved information with retrieval-attention. Experiments have shown that our approach outperforms state-of-the-art methods in various biomedical VQA datasets. Ablation studies show the effectiveness of proposed RAMM and PMCPM for pretraining and retrieving.