Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answering

Abstract

Layout-aware pre-trained models has achieved significant progress on document image question answering. They introduce extra learnable modules into existing language models to capture layout information within document images from text bounding box coordinates obtained by OCR tools. However, extra modules necessitate pre-training on extensive document images. This prevents these methods from directly utilizing off-the-shelf instruction-tuning language foundation models, which have recently shown promising potential in zero-shot learning. Instead, in this paper, we find that instruction-tuning language models like Claude and ChatGPT can understand layout by spaces and line breaks. Based on this observation, we propose the LAyout and Task aware Instruction Prompt (LATIN-Prompt), which consists of layout-aware document content and task-aware instruction. Specifically, the former uses appropriate spaces and line breaks to recover the layout information among text segments obtained by OCR tools, and the latter ensures that generated answers adhere to formatting requirements. Moreover, we propose the LAyout and Task aware Instruction Tuning (LATIN-Tuning) to improve the performance of small instruction-tuning models like Alpaca. Experimental results show that LATIN-Prompt enables zero-shot performance of Claude and ChatGPT to be comparable to the fine-tuning performance of SOTAs on document image question answering, and LATIN-Tuning enhances the zero-shot performance of Alpaca significantly. For example, LATIN-Prompt improves the performance of Claude and ChatGPT on DocVQA by $263\%$ and $20\%$ respectively. LATIN-Tuning improves the performance of Alpaca on DocVQA by $87.7\%$. Quantitative and qualitative analyses demonstrate the effectiveness of LATIN-Prompt and LATIN-Tuning. We release our code to facilitate future research.

1 Introduction

Figure 1: (a) Existing methods introduce layout-aware modules into language models to capture layout information within document images from text bounding box coordinates obtained by OCR tools. They need further pre-training on extensive document images. (b) Our method allows instruction-tuning language models to capture layout by spaces and line breaks and can conduct zero-shot inference on document image question-answering.

Intelligent document image question answering, as an important application of document intelligence, aims to develop AI systems to automatically answer natural language questions based on the understanding of document images. Compared with text documents, document images contain textual, visual, and layout information, which pose unique challenges for machine comprehension.

Recently, layout-aware pre-trained models has achieved significant progress on document image question answering. They introduce extra learnable modules on top of language models (Devlin et al. 2019; Liu et al. 2019b; Raffel et al. 2020; Bao et al. 2020) to capture layout information within document images from coordinates of text bounding box obtained by OCR tools (Fig. 1(a)). LayoutLM (Xu et al. 2020) introduces coordinate information into model input by 2D position embeddings. LayoutLMv2 (Xu et al. 2021b), LayoutLMv3 (Huang et al. 2022), and ERNIE-Layout (Peng et al. 2022) capture the layout information from coordinate by layout-aware attention mechanism. These methods conduct pre-training on extensive document images for the newly introduced layout-aware modules.

However, the need of pre-training prevents these methods from directly utilizing off-the-shelf instruction-tuning language foundation models, which have recently shown promising potential in zero-shot learning. On the one hand, commercial large instruction-tuning models like Claude (Anthropic 2023) and ChatGPT (OpenAI 2022) are closed-source, impeding further pre-training. On the other hand, open-source instruction-tuning models are of much larger scale than traditional models. For example, Alpaca (Rohan Taori et al. 2023) consists of 7 billion parameters, whereas BERT${}_{\text{large}}$ (Devlin et al. 2019) only comprises 300+ million parameters. Existing methods (Xu et al. 2020, 2021b; Huang et al. 2022; Peng et al. 2022) select over 10 million pages from the IIT-CDIP Test Collection dataset (Lewis et al. 2006) for pre-training. But the cost of pre-training instruction-tuning models like Alpaca on 10 million pages is too expensive.

Instead, in this work, we find that instruction-tuning language models like Claude and ChatGPT can understand layout by spaces and line breaks (Fig. 1(b)). Based on this observation, we propose the LAyout and Task aware Instruction Prompt (LATIN-Prompt), which consists of layout-aware document content and task-aware instruction. Specifically, given the OCR results, we use appropriate spaces and line breaks to connect all the text segments together, resulting in the layout-aware document content. The layout information contained within the coordinates is translated into spaces and line breaks. Further, we integrate task instruction into layout-aware document content, ensuring that the model generates answers that adhere to the formatting requirements. Although simple, our method is intuitive and consistent with human behavior. Humans employ whitespace (blank) regions to represent and comprehend layout.

We also find that small instruction-tuning language foundation models like Alpaca are not good at understanding layout by spaces. So we propose the LAyout and Task aware Instruction Tuning (LATIN-Tuning) to improve the performance of them. We convert CSV-format tables into strings containing spaces and line breaks and construct instruction-tuning data from these strings by Claude.

Our contributions are summarized as follows:

• We find that instruction-tuning models like Claude and ChatGPT can capture layout by spaces and line breaks, and propose LATIN-Prompt to conduct zero-shot inference on document image question-answering tasks.

• We propose the LATIN-Tuning to enhance the ability of Alpaca to comprehend layout by spaces and line breaks.

• Experiment results on three datasets show that LATIN-Prompt enables zero-shot performance of Claude and ChatGPT to be comparable to the fine-tuning performance of SOTAs on document image question answering, and LATIN-Tuning enhances the zero-shot performance of Alpaca significantly. Quantitative and qualitative analyses demonstrate the effectiveness of LATIN-Prompt and LATIN-Tuning.

2 Related Work

Figure 2: The overview of LATIN-Prompt (Sec. 3.1). Given a document image and the corresponding question, we recover the layout information within the document image from OCR results using appropriate spaces and line breaks, and then insert the layout aware document content and question into the task instruction prompt template together. The instruction-tuning large language foundation model takes the filled template as input and predicts the answer to the question in the required format.

#Line	Prompt
1	You are asked to answer questions asked on a document image.
2	The answers to questions are short text spans taken verbatim from the document. This means that the answers comprise a set of contiguous text tokens present in the document.
3	Document:
4	{Layout Aware Document placeholder}
5
6	Question: {Question placeholder}
7
8	Directly extract the answer of the question from the document with as few words as possible.
9
10	Answer:

Table 1: DocVQA Prompt Template. The {} represents the placeholder.

2.1 Visually-rich document understanding

Visually-rich Document Understanding (VrDU) focus on recognizing and understanding scanned or digital-born document images with language, vision, and layout information. Traditional works in the VrDU employ CNN (Yang et al. 2017; Katti et al. 2018; Denk and Reisswig 2019; Zhao et al. 2019; Sarkhel and Nandi 2019; Zhang et al. 2020; Wang et al. 2021; Lin et al. 2021), GNN (Liu et al. 2019a; Qian et al. 2019; Yu et al. 2020; Wei, He, and Zhang 2020; Carbonell et al. 2021), and language transformer (Majumder et al. 2020; Wang et al. 2020) to mine information from document images.

Recently, layout-aware pre-trained Transformers have been proposed (Appalaraju et al. 2021; Garncarek et al. 2021; Hwang et al. 2021; Li et al. 2021a, b, c; Xu et al. 2021a; Hong et al. 2022; Lee et al. 2022a; Peng et al. 2022; Bai et al. 2022a; Lee et al. 2022b; Luo et al. 2023; Dhouib, Bettaieb, and Shabou 2023). LayoutLM (Xu et al. 2020) introduces the 2D position information into the input embedding and LayoutLMv2 (Xu et al. 2021b) proposes the spatial-aware self-attention mechanism. Then, LayoutLMv3 (Huang et al. 2022) removes the CNN by learning visual features extraction with the discrete image tokens reconstruction, and ERNIE-Layout (Peng et al. 2022) introduces the layout knowledge into pre-training. Further, (Zhang et al. 2021; Gu et al. 2022; Wang et al. 2022a) introduce additional designs during the fine-tuning, enabling layout-aware pretrained models to better adapt to downstream tasks.

However, all of existing methods try to understand layout within document images by coordinates of bounding box obtained by OCR tools. Instead, in this work, we try to directly understand layout by spaces and line breaks. We propose the LATIN-Prompt and LATIN-Tuning to explore zero-shot document image question answering.

A concurrent work ICL-D3IE (He et al. 2023) also leverages large instruction-tuning models in document image understanding, but it differs significantly from our method. It focuses on few-shot document information extraction, but in this paper, we focus on zero-shot document image question-answering.

2.2 Instruction-tuning Language Model

An ideal AI system should be able to learn and accomplish a variety of tasks according to human instructions. To this end, many instruction-tuning datasets and language foundation models have been proposed (Thoppilan et al. 2022; Chung et al. 2022; Wei et al. 2022; Sanh et al. 2022; Mishra et al. 2022; Wang et al. 2022c; Xu, Shen, and Huang 2022; Iyer et al. 2023; Longpre et al. 2023). To fully align models with human intentions and values, reinforcement learning from human feedback and AI feedback (Bai et al. 2022b; Ouyang et al. 2022; Bai et al. 2022c; Open AI 2023) are introduced into instruction-tuning. To reduce the cost of manual annotation of instruction-tuning data, some methods automatically construct instruction-tuning data using off-the-shelf language models (Honovich et al. 2022; Wang et al. 2022b; Rohan Taori et al. 2023; Peng et al. 2023; Zhou et al. 2023; Xu et al. 2023). Recently, some methods (Zhu et al. 2023; Dai et al. 2023) have extended instruction fine-tuning into multimodal domains.

In this paper, we propose the LATIN-Prompt to allow off-the-shelf instruction-tuning language models to conduct zero-shot document image question answering. Moreover, we propose the LATIN-Tuning to enhance the ability of Alpaca to comprehend layout by spaces and line breaks.

3 Method

We propose the LATIN-Prompt and LATIN-Tuning for instruction-tuning language models to conduct zero-shot inference on document image question-answering tasks.

3.1 LATIN-Prompt

The key ideas of LATIN-Prompt are as follows: (1) Capture the layout information by spaces and line breaks (2) Generate answers adhere to formatting requirements by task instruction. Fig. 2 illustrates the process of LATIN-Prompt. Given OCR results of a document image, we recover layout information within it by using appropriate spaces and line breaks to connect all the text segments together, resulting in the layout-aware document content. Then we insert the layout-aware document content and question into the task instruction prompt template. The instruction-tuning language model takes the filled template as input and predicts the answer to the question in the required format.

Formally, given a document image $D$ and a question answer pair $q$ and $a$, we process the document image by an OCR tool. The extracted text segments and corresponding bounding boxes are denoted as $S=\{s_1,s_2,\mathrm{\dots },s_n\}$ and $B=\{b_1,b_2,\mathrm{\dots },b_n\}$, where $n$ represents the number of text segments.

Layout Aware Document

We employ appropriate spaces and line breaks to connect all text segments together, resulting in layout-aware document content. The process is as follows:

Step 1. Re-arrange the text segments and bounding boxes in the order from top to bottom and from left to right based on the coordinates.

Step 2. According to the coordinates, place the text segments and bounding boxes in the $i$-th row into the list $S_i$ and $B_i$ respectively from left to right, and calculate the character count $c_i$ and the width $w_i$ of $i$-th row. The $w_i$ equals the width of the union of bounding boxes in the list $B_i$.

Step 3. Calculate the character width of document $D$, which is defined as follows:

$$\overline{c}:=w_i^{\ast }/c_i^{\ast },i^{\ast }={\text{argmax}}_i{c}_i,$$ (1)

where $i^{\ast }$-row has the maximum character count among all rows.

Step 4. Join text segments in the same row from left to right by spaces. Given two adjacent text segments $S_{i,j}$ and $S_{i,k}$, the number of spaces joining them is equal to $h_{i,jk}/\overline{c}$ where $h_{i,jk}$ is the horizontal distance between the two bounding boxes $B_{i,j}$ and $B_{i,k}$.

Step 5. Join different rows by line breaks to obtain the layout-aware document content (denoted as $S^{\prime }$).

Recovering layout information by spaces and line breaks is simple but intuitive. In fact, people do represent and understand layout through blank areas between text elements, rather than precise bounding box coordinates.

Task Aware Instruction

Different from the open-ended question-answering, document image question-answering typically involves explicit requirements for the answer format. For example, DocVQA (Mathew, Karatzas, and Jawahar 2021) is an extractive QA task that requires answers to be extracted from the document. However, with only the layout-aware document content and question, the model can easily generate answers that are not in the document and generate unnecessary descriptions or explanations.

So we integrate task instruction into layout-aware document content, ensuring that the model generates answers that adhere to the formatting requirements. Specifically, we manually designed different instruction prompt templates for different tasks. Each template $P(S^{\prime },q)$ contains the requirement of the task as well as placeholders for the layout-aware document content $S^{\prime }$ and question $q$.

Table 7 shows the prompt template for DocVQA. In the first and second lines, we explain the meaning of extraction in detail to the model according to the task description in DocVQA. Lines 3 to 6 provide placeholders for the layout-aware document and question. To avoid the model forgetting its task due to the interference of document content, the 8th line summarizes and reiterates the task requirements. Please refer to the supplementary materials for the prompt templates of InfographicVQA (Mathew et al. 2021) and MP-DocVQA (Tito, Karatzas, and Valveny 2023).

Zero-shot Inference

At last, the instruction-tuning language model $M$ takes the filled template $P(S^{\prime },q)$ as input and predicts the answer as follows:

$$a^{\prime }=f_M(P(S^{\prime },q)),$$ (2)

where $a^{\prime }$ represents the prediction and the $f_M$ represents the decoding process of model $M$.

3.2 LATIN-Tuning

Figure 3: Construction of LATIN-Tuning data (Sec. 3.2). (1) Convert the CSV-format table into document string with spaces and line breaks by Pandas. (2) Insert document string into the Question Generation Prompt Template and generate a question-answer pair by Claude. (3) Insert document string and question into the Instruction Prompt Template to form the input, with the answer serving as the target.

Paradigm	Method	Parameters	Text	Vision	Layout	Fine-tuning Set	ANLS	$\mathrm{\Delta }$ANLS
Fine-tuning	BERT${}_{\text{LARGE}}$	340M	$\mathrm{✓}$			train	0.6768
	RoBERTa${}_{\text{LARGE}}$	355M	$\mathrm{✓}$			train	0.6952
	UniLMv2${}_{\text{LARGE}}$	340M	$\mathrm{✓}$			train	0.7709
	LayoutLM${}_{\text{LARGE}}$	343M	$\mathrm{✓}$		$\mathrm{✓}$	train	0.7259
	LayoutLMv2${}_{\text{LARGE}}$	426M	$\mathrm{✓}$	$\mathrm{✓}$	$\mathrm{✓}$	train	0.8348
	LayoutLMv3${}_{\text{LARGE}}$	368M	$\mathrm{✓}$	$\mathrm{✓}$	$\mathrm{✓}$	train	0.8337
	ERNIE-Layout${}_{\text{LARGE}}$	507M	$\mathrm{✓}$	$\mathrm{✓}$	$\mathrm{✓}$	train	0.8321
	LayoutLMv2${}_{\text{LARGE}}$	426M	$\mathrm{✓}$	$\mathrm{✓}$	$\mathrm{✓}$	train + dev	0.8529
	ERNIE-Layout${}_{\text{LARGE}}$	507M	$\mathrm{✓}$	$\mathrm{✓}$	$\mathrm{✓}$	train + dev	0.8486
Zero-shot	Alpaca+Plain Prompt	7B	$\mathrm{✓}$			-	0.3567
	Alpaca+LATIN-Prompt		$\mathrm{✓}$		$\mathrm{✓}$	-	0.4200	+0.0633
	Claude+Plain Prompt	Unknown	$\mathrm{✓}$			-	0.2298
	Claude+LATIN-Prompt		$\mathrm{✓}$		$\mathrm{✓}$	-	0.8336	+0.6038
	ChatGPT-3.5+Plain Prompt	Unknown	$\mathrm{✓}$		$\mathrm{✓}$	-	0.6866
	ChatGPT-3.5+LATIN-Prompt		$\mathrm{✓}$		$\mathrm{✓}$	-	0.8255	+0.1389
	GPT-4∗	Unknown	not clearly described			-	0.8840
* represents that we report the result of GPT-4 presented in OpenAI blog (OpenAI 2023). Although lacking a technical detail description, compared with Claude and ChatGPT-3.5, GPT-4 utilizes visual information. The LATIN-Prompt is orthogonal to GPT-4 and can be used to further improve the performance of GPT-4. However, due to API permission limitations, we are unable to evaluate the performance of GPT-4 + LATIN-Prompt and leave it in future.

Table 2: Performance on test dataset of DocVQA. Text, Vision, and Layout represent the modal information used by the model. The $\mathrm{\Delta }$ANLS represents the gain of LATIN-Prompt compared to Plain Prompt. Unknown indicates missing relevant details.

Paradigm	Method	Overall		Answer type
		ANLS	$\mathrm{\Delta }$ANLS	Image span	Question span	Multiple spans	Non span
Fine-tuning	BERT	0.2078		0.2625	0.2333	0.0739	0.0259
	LayoutLM	0.2720		0.3278	0.2386	0.0450	0.1371
	LayoutLMv2	0.2829		0.3430	0.2763	0.0641	0.1114
	BROS	0.3219		0.3997	0.2317	0.1064	0.1068
	pix2struct	0.4001		0.4308	0.4839	0.2059	0.3173
	TILT	0.6120		0.6765	0.6419	0.4391	0.3832
Zero-shot	Claude + Plain Prompt	0.0798		0.0951	0.0913	0.0203	0.0280
	Claude + LATIN-Prompt	0.5451	+0.4653	0.5992	0.5861	0.3985	0.3544
	ChatGPT-3.5 + Plain Prompt	0.3335		0.3749	0.4505	0.0950	0.1822
	ChatGPT-3.5 + LATIN-Prompt	0.4898	+0.1563	0.5457	0.5639	0.3458	0.2798

Method	Evidence					Operation
	Table/List	Textual	Visual object	Figure	Map	Comparison	Arithmetic	Counting
BERT	0.1852	0.2995	0.0896	0.1942	0.1709	0.1805	0.0160	0.0436
LayoutLM	0.2400	0.3626	0.1705	0.2551	0.2205	0.1836	0.1559	0.1140
LayoutLMv2	0.2449	0.3855	0.1440	0.2601	0.3110	0.1897	0.1130	0.1158
BROS	0.2653	0.4488	0.1878	0.3095	0.3231	0.2020	0.1480	0.0695
pix2struct	0.3833	0.5256	0.2572	0.3726	0.3283	0.2762	0.4198	0.2017
TILT	0.5917	0.7916	0.4545	0.5654	0.4480	0.4801	0.4958	0.2652
Claude + Plain Prompt	0.0849	0.1099	0.0858	0.0695	0.0496	0.0589	0.0271	0.0368
Claude + LATIN-Prompt	0.5421	0.6725	0.4897	0.5027	0.4982	0.4598	0.4311	0.2708
ChatGPT-3.5 + Plain Prompt	0.3481	0.3893	0.3670	0.3114	0.1843	0.2349	0.1466	0.2320
ChatGPT-3.5 + LATIN-Prompt	0.4917	0.6016	0.4491	0.4585	0.3614	0.4312	0.3157	0.2660

Table 3: Performance on test dataset of InfographicVQA. The questions in InfographicVQA can be grouped according to answer type, evidence source, and operation. We list both the overall performance of the model and its performance on different groups. All performances are evaluated by ANLS. The $\mathrm{\Delta }$ANLS represents the gain of LATIN-Prompt compared to Plain Prompt. The highest and second-highest scores are bolded and underlined.

Paradigm	Method	Setup	ANLS
Fine-tuning	BERT	Max Conf	0.5347
		Concat	0.4183
	Longformer	Max Conf	0.5506
		Concat	0.5287
	Big Bird	Max Conf	0.5854
		Concat	0.4929
	LayoutLMv3	Max Conf	0.5513
		Concat	0.4538
	T5	Max Conf	0.4028
		Concat	0.5050
	Hi-VT5	Multipage	0.6201
Zero-shot	Claude+LATIN-Prompt	Max Conf	0.6129

Table 4: Performance on test dataset of MP-DocVQA. Concat indicates concatenating multi-page content. MaxConf indicates processing pages separately and choosing an answer based on confidence. We only evaluate Claude with LATIN-Prompt in MaxConf setting because Plain Prompt does not apply to MaxConf, and Concat exceeds the length limit.

Although instruction-tuning models like Claude and ChatGPT can comprehend and utilize LATIN-Prompt well, we found that the performance of smaller models like Alpaca (7B) was not up to par. So we propose LATIN-Tuning to enhance their ability to comprehend layout by spaces and line breaks. As shown in Fig. 3, we employ the Pandas and Claude to construct instruction-tuning dataset from CSV-format tables. The process is as follows:

(1) For each CSV table, we convert it into document string with spaces and line breaks by Pandas. Please refer to the appendix for the code implementation. (2) We insert the document string into the Question Generation Prompt Template and generate a question-answer pair by Claude. (3) We insert document string and question into the Instruction Prompt Template to form the input, with the answer serving as the target. Refer to the appendix for the details of Question Generation Prompt Template and Instruction Prompt Template.

At last, we fine-tune the Alpaca on the instruction-tuning dataset to enhance its ability to comprehend layout by spaces and line breaks.

4 Experiment

Figure 4: The impact of the size of instruction fine-tuning dataset on LATIN-Tuning. The performance of LATIN-Tuning improves as the number of samples increases.

Figure 5: Case study of Claude on DocVQA. Due to the lack of task instruction, Plain Prompt generates unnecessary words (in blue), violating the extraction requirement. Moreover, Plain Prompt cannot capture layout information and generates incorrect answers (in green). In Case (A), it regards ”Coffee” and ”Chocolate”, which are semantically similar to ”choco”, as the answer. In Case (B), it regards ”61” as the answer since it directly follows ”romance and mystery”. In Case (C), it fails to comprehend the document and engages in erroneous reasoning. In contrast, LATIN-Prompt can understand layout relationships and generate the correct answer (in red). The document images of these cases are complex. Due to limited space, we only display a portion of the original document images here. Please refer to the appendix for the original document images and more cases.

4.1 Experiment Settings

Datasets

We evaluate our method on three document image question answering datasets: DocVQA (Mathew, Karatzas, and Jawahar 2021) is an extractive question answering task and consists of 50,000 questions defined on 12,767 document images; InfographicVQA (Mathew et al. 2021) consists of 5,485 infographics, which convey information through text, graphics, and visual elements together; MP-DocVQA (Tito, Karatzas, and Valveny 2023) extends DocVQA to more realistic multi-page scenarios where a document typically consists of multiple pages that should be processed together. Following common practice, we use Azure OCR results for DocVQA provided by DUE (Borchmann et al. 2021) and use offical OCR results for InfographicVQA and MP-DocVQA. For all datasets, we adopt the Average Normalized Levenshtein Similarity (ANLS) (Biten et al. 2019) as the evaluation metric.

LATIN-Prompt Baselines

To evaluate zero-shot performance of LATIN-Prompt, we compare it with Plain Prompt on three instruction-tuning language models: Claude, ChatGPT-3.5, and Alpaca. The template of Plain Prompt is as follows: “Document: {document}$\backslash$nQuestion: {question}$\backslash$nDirectly extract the answer of the question from the document.$\backslash$nAnswer:”, where {document} and {question} are placeholders of original text segments from OCR tools and question. We also compare LATIN-Prompt’s zero-shot performance with fine-tuning performance of pre-training-fine-tuning methods. Moreover, we report the result of multimodal GPT-4 presented in OpenAI blog (OpenAI 2023). Due to API permission restrictions, we leave the exploration of GPT-4+LATIN-Prompt as future work.

We only evaluate Alpaca on DocVQA because the other two tasks are too complex for it. Alpaca cannot follow the task instruction of these two tasks. In fact, Alpaca performs poorly on InfographicVQA (refer to Tab. 5) and cannot generate answers meeting the format requirement of MP-DocVQA. We exclude the ChatGPT-3.5 on MP-DocVQA because it needs to process too many document pages and the experimental cost exceeds the range we can afford.

LATIN-Tuning

We randomly sample 5000 CSV-format tables from the WikiTableQuestions (Pasupat and Liang 2015) with replacement to create the instruction-tuning dataset. We fine-tune the Alpaca on the created dataset for 3 epochs using the AdamW (Loshchilov and Hutter 2018) optimizer with a warmup ratio of 0.03 following the Alpaca (Rohan Taori et al. 2023). We use a batch size of $64$ and a learning rate of $2e-5$. The resulting model is denoted as Alpaca+LATIN-Tuning and we compare it with Alpaca to evaluate the performance of LATIN-Tuning.

4.2 Performance of LATIN-Prompt

Table 2 presents the experimental results on DocVQA. (1) In the pre-training-fine-tuning paradigm, the layout-aware multimodal pre-trained model performs better than the pure language model. (2) Further, increasing the amount of fine-tuning data can improve the performance of models. (3) The instruction-tuning language models perform poorly with the plain prompt based on the original text segments obtained by OCR tools. (4) The LATIN-Prompt proposed in this paper significantly improves the zero-shot performance of instruction-tuning language models. It enables the zero-shot performance of Claude and ChatGPT-3.5 to significantly outperform the fine-tuned layout-aware LayoutLM. In addition, despite only using text and layout information, their zero-shot performance is comparable to the performance of fine-tuned layout-aware multimodal pre-trained models. (5) Although unknown, the number of parameters of Claude and GPT-3.5 should be much larger than that of Alpaca. The experimental results show that the final zero-shot performance is positively correlated with the size and ability of the instruction-tuning models. (6) The zero-shot performance of GPT-4 matched the best fine-tuned performance. Although lacking a technical detail description, compared with Claude and GPT-3.5, GPT-4 utilizes visual information, reflecting the importance of visual information for document image understanding. LATIN-Prompt is orthogonal to GPT-4. However, due to API permission restrictions, we can only leave arming GPT-4 with the LATIN-Prompt to the future.

Table 3 presents results on InfographicVQA. Experimental results show that LATIN-Prompt enable the zero-shot performance of Claude and GPT-3.5 to exceed the performance of all fine-tuned baselines except TILT. We find that Claude performs poorly with Plain Prompt, but its performance improves significantly when using LATIN-Prompt.

Table 4 presents results on MP-DocVQA. It shows that, with LATIN-Prompt, Claude’s zero-shot performance exceeds fine-tuning performance of Longformer (Beltagy, Peters, and Cohan 2020) and Big Bird (Zaheer et al. 2021) designed for long sequences. Furthermore, its zero-shot performance is comparable to the fine-tuning performance of Hi-VT5 (Tito, Karatzas, and Valveny 2023), a layout-aware multimodal model for multi-page document images.

4.3 Performance of LATIN-Tuning

Method	DocVQA		InfographicVQA
	Valid	Test	Valid	Test
Alpaca	0.3506	0.3567	0.1083	0.1419
Alpaca+LATIN-Tuning	0.6668	0.6697	0.2873	0.3028
Claude	0.8311	0.8336	0.5218	0.5451

Table 5: Effect of LATIN-Tuning. All methods are equipped with LATIN-Prompt.

Table 5 demonstrates that LATIN-Tuning improves the performance of Alpaca on DocVQA by $87.7\%$ and on InfographicVQA by $113\%$. Nevertheless, its performance still lags behind Claude. We will explore more effective instruction-tuning method in the future.

4.4 Quantitative and Qualitative Analyses

Prompt	DocVQA		InfographicVQA
	Claude	ChatGPT	Claude	ChatGPT
LATIN-Prompt	0.8311	0.8135	0.5218	0.4708
w/o Layout	0.7825	0.7491	0.4638	0.4341
w/o Task	0.3637	0.7561	0.1234	0.4296
w/o Task+Layout	0.2144	0.6795	0.0702	0.3103

Table 6: Ablation of LATIN-Prompt on validation data of DocVQA and InfographicVQA.

Effect of components of LATIN-Prompt

LATIN-Prompt consists of layout-aware document content (Layout) and task instruction (Task). Table 6 presents the results of ablation study of LATIN-Prompt with Claude and ChatGPT-3.5 on DocVQA and InfographicVQA. The results show that both the layout-aware document content and the task instruction can significantly improve the zero-shot performance of Claude and ChatGPT-3.5. The improvement brought by task instruction is more significant in Claude because it ensures that the format of the answers generated by the model meets the task requirements. On the basis of the correct format, the layout-aware document content further improves the performance of the model because it enables the model to utilize the layout information among text segments.

Effect of instruction-tuning data size for LATIN-Tuning

Figure 4 shows that the performance of LATIN-Tuning improves as the number of samples increases. The improvement rate slows down when the sample count exceeds 2000.

Case study of LATIN-Prompt

Figure 5 provides cases of Claude on DocVQA. Compared with Plain Prompt, LATIN-Prompt enables the model to comprehend layout more effectively and generate answers meeting the format requirement.

Case study of LATIN-Tuning

Case study on DocVQA shows that LATIN-Tuning enables Alpaca to understand layout by spaces. Please refer to the appendix for details.

5 Conclusion

In this work, we point a new perspective for comprehending layout information within document images. Instead of capturing layout by coordinate of bounding boxes, we find that instruction-tuning language models like Claude and ChatGPT can understand layout by spaces and line breaks. Based on this observation, we propose LATIN-Prompt and it enables zero-shot performance of Claude and ChatGPT to be comparable to the fine-tuning performance of SOTAs on document image question answering. Moreover, we propose LATIN-Tuning, which enhances the ability of Alpaca to comprehend layout by spaces and line breaks. In the future, we will explore to incorporate visual information into LATIN-Prompt and create more effective instruction-tuning dataset for LATIN-Tuning.

Appendix A Prompt Templates

#Line	Prompt
1	You are asked to answer questions asked on a document image.
2	The answers to questions are short text spans taken verbatim from the document. This means that the answers comprise a set of contiguous text tokens present in the document.
3	Document:
4	{Layout Aware Document placeholder}
5
6	Question: {Question placeholder}
7
8	Directly extract the answer of the question from the document with as few words as possible.
9
10	Answer:

Table 7: DocVQA Prompt Template. The {} represents the placeholder.

#Line	Prompt
1	You are asked to answer questions asked on a document image.
2	The answer for a question in this can be any of the following types:
3	1. Answer is a piece contiguous text from the document.
4	2. Answer is a list of ”items” , where each item is a piece of text from the document (multiple spans). In such cases your model/method is expected to output an answer where each item is separated by a comma and a space. For example if the question is ”What are the three common symptoms of COVID-19?” Answer must be in the format ”fever, dry cough, tiredness”. In such cases ”and” should not be used to connect last item and the penultimate item and a space after the comma is required so that your answer match exactly with the ground truth.
5	3. Answer is a contiguous piece of text from the question itself (a span from the question)
6	4. Answer is a number ( for example ”2”, ”2.5”, ”2%”, ” 2/3” etc..). For example there are questions asking for count of something or cases where answer is sum of two values given in the image.
7	Document:
8	{Layout Aware Document placeholder}
9
10	Question: {Question placeholder}
11
12	Directly answer the answer of the question from the document with as few words as possible.
13
14	Answer:

Table 8: InfographicVQA Prompt Template. The {} represents the placeholder.

#Line	Prompt
1	You are asked to answer questions asked on a document image.
2	The answers to questions are short text spans taken verbatim from the document. This means that the answers comprise a set of contiguous text tokens present in the document.
3	Document:
4	{Layout Aware Document placeholder}
5
6	Question: {Question placeholder}
7
8	Directly extract the answer of the question from the document with as few words as possible.
9
10	You also need to output your confidence in the answer, which must be an integer between 0-100.
11	The output format is as follows, where [] indicates a placeholder and does not need to be actually output:
12	[Confidence score], [Extracted Answer]

Table 9: MP-DocVQA Prompt Template. The {} represents the placeholder.

#Line	Prompt
1	Document:
2	{Document string with spaces and line from CSV table}
3	Randomly generate a question and corresponding answer for the above document. The answer to the question must be unique, and must be extracted from the document. To answer this question, the layout of the document must be understood. The output should be in the following format without any other text:
4	Question: [Question content]
5	Answer: [Answer content]

Table 10: Instruction Prompt Template. The {} represents the placeholder.

#Line	Prompt
1	Document:
2	{Document string with spaces and line from CSV table}
3	Question: {Question}
4	Please extract the answer of the question from the given document.

Table 11: Question Generation Prompt Template. The {} represents the placeholder.

Table 7 shows the prompt template for DocVQA(Mathew, Karatzas, and Jawahar 2021). In the first and second lines, we explain the meaning of extraction in detail to the model according to the task description in DocVQA (Mathew, Karatzas, and Jawahar 2021). Lines 3 to 6 provide placeholders for the layout-aware document and question. To avoid the model forgetting its task due to the interference of document content, the 8th line summarizes and reiterates the task requirements.

Table 8 shows the prompt template for InfographicVQA(Mathew et al. 2021). Compared with DocVQA, InfographicVQA has more complex answer sources. We describe the answer requirements in detail in lines 1 to 6, and the rest is similar to the DocVQA prompt template.

Table 9 shows the prompt template for MP-DocVQA (Tito, Karatzas, and Valveny 2023). MP-DocVQA is a multi-page question answering task, in which each question has multiple candidate page images, and we adopt the Max Confidence (Max Conf) (Mathew, Karatzas, and Jawahar 2021) setup to solve it. Specifically, the model extracts one answer on each page and gives confidence to this answer. We choose the answer with the highest confidence as the final prediction. Lines 10 to 12 of Tab. 9 instruct the model to output the answer and confidence simultaneously, and the rest is similar to Tab. 7.

Table 10 and Table 11 provide the details of Question Generation Prompt Template and Instruction Prompt Template in LATIN-Tuning.

Appendix B Datasets

To evaluate LATIN-Prompt, we conduct experiments on three document image question answering datasets, including DocVQA (Mathew, Karatzas, and Jawahar 2021), InfographicVQA (Mathew et al. 2021), and MP-DocVQA (Tito, Karatzas, and Valveny 2023).

The DocVQA is an extractive question answering task and consists of 50,000 questions defined on 12,767 document images. The train split has 39,463 questions, the validation split has 5,349 questions, and the test split has 5,188 questions. DocVQA contains a large number of questions related to forms, layouts, tables and lists in the image, posing high requirements on the model’s ability to understand document image layouts.

The InfographicVQA consists of 5,485 infographics, which convey information through text, graphics, and visual elements together. Compared with DocVQA, InfographicVQA emphasizes questions that require basic reasoning and arithmetic skills, and has more complex answer sources, including Document(Image)-Span, Question-Span, Multi-Span and Non-extractive.

MP-DocVQA extends DocVQA to more realistic multi-page scenarios where a document typically consists of multiple pages that should be processed together. It comprises 46,000 questions posed over 48,000 scanned pages belonging to 6,000 industry documents, and the page images contain diverse layouts. The variability between documents in MP-DocVQA is very high. The number of pages in each document varies from 1 to 20, and the number of recognized OCR words varies from 1 to 42,313.

Following common practice, we use Azure OCR results for the DocVQA provided by DUE (Borchmann et al. 2021). For the InfographicVQA and MP-DocVQA, we use offical OCR results.

Appendix C Case Study

Figure 6: Case study of LATIN-Tuning.

Figure 6 provides a case to compare the performance of Alpaca and Alpaca + LATIN-Tuning. After LATIN-Tuning, the Alpaca can comprehend layout more effectively.