PanGu-Coder：使用函数级语言建模进行程序综合

摘要

我们提出了 PanGu-Coder，这是一种预训练的仅解码器语言模型，采用 PanGu-$\alpha$ 架构进行文本到代码生成，即合成给出自然语言问题描述的编程语言解决方案。 我们使用两阶段策略训练PanGu-Coder：第一阶段采用因果语言模型（CLM）对原始编程语言数据进行预训练，而第二阶段结合使用因果语言模型掩码语言建模 (MLM) 训练目标侧重于文本到代码生成的下游任务以及松散管理的自然语言程序定义和代码函数对。 最后，我们讨论 PanGu-Coder-FT，它结合了竞争性编程问题和持续集成测试的代码进行了微调。 我们评估 PanGu-Coder 的重点是它是否生成功能正确的程序，并证明它比类似大小的模型（例如 CodeX [16])实现了相同或更好的性能，同时参加较小的上下文窗口并使用较少的数据进行训练。

1简介

基于 Transformer [66] 架构的越来越多的大型预训练语言模型 [56, 23, 58, 32] 已经被提出并被证明可以实现状态 -在各种自然语言处理（NLP）任务上取得了最先进的成果。 最近，此类模型已适应更具体的语言领域，例如生物医学[37,46,9]、法律[14]、网络安全[2]和金融[6 ] 领域，同时扩展到包括来自自然语言以外的模式的信号，例如 视觉 [65, 18, 15, 13, 64, 25]，蛋白质 [11]，时间序列 [71, 72, 55]和代码[42,70,26,31,54]。

在这项工作中，我们专注于专门为文本到代码生成而创建的预训练语言模型，即从自然语言（NL）描述（例如问题定义或文档字符串）合成程序的任务。 虽然针对此任务提出的一些模型采用了编码器-解码器架构[47]，但大多数模型都被训练为仅解码器的 Transformer 模型[16,51,27]。 编码器-解码器架构要求输入和输出之间有明确的区别和关联。 相比之下，单个组件（仅编码器或仅解码器）架构将输入视为连续序列，因此非常适合对大量原始数据进行训练。 无论架构如何，最近的工作都是对从 GitHub，Stack Exchange 和其他来源，其中一些是从专门在 NL 上预训练的现有语言模型初始化的（例如从 GPT [12]2> 或 BERT [23] 初始化>)。

我们提出了 PanGu-Coder，这是一种用于文本到代码生成的预训练语言模型。 PanGu-Coder遵循引入的PanGu-$\alpha$架构（见图LABEL:fig:pangu_alpha) Zeng 等人[73]，它由一个单向解码器 Transformer 和一个堆叠在顶部的额外查询层组成。 在每个时间步$t$，查询层都会关注位置嵌入$t+1$以确定下一个词符。 虽然 PanGu-$\alpha$ 被提议用于处理英文和中文 NL 文本，但 PanGu-Coder 目前专注于从专门的文本到代码生成英文提示。 PanGu-Coder 目前仅支持 Python 输出，但该模型可以轻松扩展到其他语言。

我们使用两阶段策略来训练 PanGu-Coder，第一阶段作为无监督原始编程语言数据的预训练。 在此阶段，自然语言以文档字符串或内联注释（如果有）的形式包含在内。 为了充分利用原始数据，我们遵循现有仅解码器模型的训练机制，并采用常规因果语言建模 (CLM) [56]，同时将所有数据视为连续序列。 训练的第二阶段旨在关注文本到代码生成的下游任务，并利用它包含不同的源序列和目标序列的事实，即输入 NL 问题定义和输出代码。 因此，第二阶段专门关注 NL 和代码的对齐对。 我们尝试了各种训练目标，包括因果和掩码语言建模 (MLM) 目标的组合，从编码器-解码器架构的训练方案中汲取灵感。 通过这种两阶段方法，我们的模型能够在第一阶段通过原始数据学习一般代码结构与自然语言的关系，然后在第二阶段关注如何在给定 NL 输入的情况下最好地生成正确的输出代码。 最后，我们还使用与目标域更密切相关的数据PanGu-Coder（第3节）。 我们根据所使用的数据的性质将阶段 2 训练与微调区分开来，即从可能未对齐或嘈杂的在线源中提取的数据与更适合文本到代码生成的数据，例如从编程竞赛中检索。

本技术报告的其余部分组织如下。 2 部分正式定义了文本到代码生成的任务。 在LABEL:sec:training_methodology部分，我们详细介绍了我们的训练数据、方法并展示了零样本结果和分析。 同样，第3部分介绍了我们的微调方法、数据和结果。 最后，第4节讨论了相关工作。

2 任务定义

Problem Description

3 Fine-Tuning Methodology

In this section, we evaluate a version of PanGu-Coder that was fine-tuned on a combination of competitive programming problems and code from continuous integration tests. The model, called PanGu-Coder-FT, allows us to estimate the improvement in problem solve rates given additional data from a more similar domain / distribution, which we describe in the following paragraphs.

Competitive Programming Data

The following two datasets provide dozens of successful code completions for each problem. Therefore, we can upsample each dataset by pairing up to five solutions with each problem description to avail the model of different correct solutions for the same problem specification. APPS [34] was proposed to benchmark the programming capability of code-generating models. It includes 10,000 programming tasks (5,000 train, 5,000 test) that require code generation / completion given a detailed problem description. Program arguments are provided from standard input as well as function calls, sometimes embedded inside the class. We use the train and test set programs, which we filter for the maximum input length of the model (1,024), retaining 43K examples. Code Contests (CC) is a similar dataset [47] for training and evaluation of competition-level code generation, containing over 13K programming problems. After upsampling and length filtering, we retain 18K instances for fine-tuning.

Continuous Integration Data

In addition to the competitive programming data, we also gathered a dataset consisting of correctly-implemented functions from open source projects. Following recent work on evaluating language models trained on code [16], we considered public Python repositories on GitHub using Travis or Tox as their continuous integration (CI) system(s). First, we reproduced the CI environment of each project locally in docker instances, then injected a tracing function in the environment, which would capture the input, output, invocation location and context code for each function invoked during integration testing. After that, the CI script was triggered and the data of invoked functions were recorded into a database. We further clustered the collected data into 4 groups, according to the external contextual information (dependencies) that a function requires to run. That is, functions depending on Python-builtin objects, Python-standard libraries, Pypi-public libraries, and the residing class. We expect this variety could enable the generation of different function types, e.g., self-contained, module, member or class functions. After filtering for maximum model input length, 49K examples are retained for fine-tuning.

Training Details

We fine-tune PanGu-Coder-FT for 5 epochs on the aforementioned data, using the Code-CLM objective. We decrease the batch size to 32 and use a linear decay learning rate scheduler. All other model training and program generation settings follow the Stage-2 protocol (Section LABEL:sec:pretraining-results-analysis).

3.1 Fine-tuning Results

PanGu-Coder-FT clearly benefits from additional fine-tuning on data that is closer to the target distribution, i.e. short programming tasks solving competitive and technical interview questions (Table 7). The pass rate improvement is more pronounced for MBPP, which may be due to the lower difficulty of problems (relative to HumanEval). Recall that the B in MBPP stands for ’basic’. In the following paragraphs, we discuss the importance of appropriate fine-tuning data and provide additional methods for boosting the pass rates of solutions generated by PanGu-Coder(-FT).

3.1.1 Impact of In-domain Data

MBPP and HumanEval provide a challenging zero-shot evaluation of models designed to generate functionally correct programs. MBPP additionally provides a small training/prompting set of 474 instances, which we can use to contrast the rate of improvement between out-of-domain and in-domain data. Fine-tuning PanGu-Coder on just a few hundred relevant problems increases pass@$k$ almost as much as tens of thousands of correctly implemented but mostly out-of-domain examples (see Table 7). Models therefore need to acquire a wide spectrum of problem solving and programming knowledge to perform well on these datasets. Note that despite their superficial similarity, MBPP fine-tuning provides no benefit to HumanEval (Table 7).

Models	MBPP (%)			HumanEval (%)
	pass@$1$	pass@$10$	pass@$100$	pass@$1$	pass@$10$	pass@$100$
PanGu-Coder	16.20	34.39	53.74	17.07	24.05	34.55
PanGu-Coder-FT	24.60	44.19	63.07	19.50	25.96	40.80
PanGu-Coder-MBPP	25.40	43.32	60.03	15.24	22.73	32.65

Table 7: PanGu-Coder-FT (fine-tuned with competitive and continuous integration data), PanGu-Coder-MBPP (fine-tuned on MBPP train set only), PanGu-Coder(trained on open-source data).

3.1.2 Impact of Unit Tests

HumanEval provides input/output pairs in the problem description that can be used to verify the correctness of a solution, e.g. "remove_letters(PHP, P) == H". MBPP problem descriptions can be augmented with test cases provided for evaluation in order to investigate whether the model benefits from the additional insights. Table 8 shows the pass rates for models trained/fine-tuned (Train) with and without unit tests as well as programs generated (Sample) with and without asserts. We observe that for MBPP, the pass rates substantially increase with unit tests in the prompt, particularly when this is provided during sampling as well. This is not the case for HumanEval, however, possibly because the problem descriptions are longer, more detailed and less formulaic. Similar observations were made during the zero-shot analysis in Section LABEL:sec:prompt_effect. A more effective way of improving HumanEval scores is to use the same unit tests to filter out dysfunctional programs with our proposed methods, outlined in section 3.1.3.

Model	Train	Sample	MBPP (%)			HumanEval (%)
			pass@$1$	pass@$10$	pass@$100$	pass@$1$	pass@$10$	pass@$100$
PanGu-Coder	N/A	✓	16.20	34.39	53.74	17.07	24.05	34.55
PanGu-Coder	N/A	✗	17.40	34.29	50.91	16.46	23.38	33.58
+ MBPP-train	✗	✗	20.00	37.70	53.04	15.85	24.11	33.66
+ MBPP-train	✗	✓	23.60	40.91	58.91	13.41	23.52	33.44
+ MBPP-train	✓	✗	21.00	36.44	51.92	16.46	23.79	31.50
+ MBPP-train	✓	✓	25.40	43.32	60.03	15.24	22.73	32.65

Table 8: PanGu-Coder trained and/or solutions generated with and without unit tests.

3.1.3 Filtering Generated Programs

In order to continue to improve PanGu-Coder’s pass rates, we can follow the program generation stage with an additional post-processing step, e.g. filtering on unit tests, declared function types and checking for syntactic correctness. The solutions ($n=200$) to each HumanEval problem are filtered before being evaluated on the held-out unit tests. Not all problems have embedded/parsable unit tests or have declared all function arguments and return types. Therefore, we evaluate the effectiveness of our filtering methods on problem subsets that have the specific filter available, i.e. 136 of 164 with unit tests and 30 of 164 with full type declarations (all 164 problems can be checked for syntax errors). This gives us a baseline score when no filtering is applied. The number of problems with each filter and the corresponding pass rates are shown in Table 9.

Filtering Method		PanGu-Coder		PanGu-Coder-FT
		pass@1	pass@10	pass@1	pass10
Unit Testing	Base	14.53	-	16.67	-
	Filter	35.48	-	41.52	-
Typing	Base	25.16	50.65	30.45	54.61
	Filter	26.99	52.00	31.30	55.72
Invalid Syntax	Base	12.05	23.27	13.85	25.40
	Filter	12.06	23.27	13.86	25.44

Table 9: HumanEval pass@$k$ with/out filtering (for each problem subset). Unit tests are available for 136/164 problems, argument/return types for 30/164 problems and syntax check for all problems.

Unit Tests

In many real-world scenarios, the description of a problem can include example function input(s) and output(s), which we can use to filter out unsuccessful solutions, potentially increasing the overall pass rate. In addition to the held-out tests, HumanEval typically provides unit tests embedded in the description, expressed in a natural way and usually following a simple pattern. A basic regular expression (to avoid overfitting) is used to extract the majority (136/164) of the unit tests. Table 9 shows a significantly improved pass rate after this filtering step. Only the pass@$1$ score is available with this filter, as many problems fail to reach at least 10 solutions per problem, once filtered. The mean/median number of solutions after filtering is 115/155, an average reduction of around 40-45%.

Typing

In other scenarios, we may be provided with function argument types and the expected return type. In such cases, we can generate plausible function inputs, execute the sampled program and evaluate the type of the return value. If the program returns the expected type, it will be included for the final pass@$k$ evaluation. As the generated programs are executed, we implicitly verify the syntactic and semantic correctness, too. Relying only on checking the return type will inevitably lead to false positives as programs often return the correct type, e.g. a float or a list of floats but with the wrong value(s). However, even this simple filter can improve the final pass rates (Table 9) and reduce the number of programs by around 25% (mean/median 150/152 after filtering).

Invalid Syntax

In the absence of unit tests and input/output type declarations retrieved from the description, filtering out syntactically invalid solutions can be a fast and simple method for eliminating incorrect problem solutions. With that in mind, Table 9 shows that PanGu-Coder is generating syntactically well-formed solutions almost all the time hence we filter out of 200 problem solutions (mean/median after filtering 199/200), leaving the pass rates unchanged.

3.2 Data Selection with Few-Shot Similarity

Transformer-based pretrained language models continue to rapidly increase in size [70, 47, 19] hence it is desirable to reduce the compute requirements while maintaining good performance, whenever possible. To this end, we simulate a scenario, in which we possess a small sample of programming problems (possibly written from scratch) and a much larger dataset of heterogeneous training data. We then experiment with choosing training examples from the much larger dataset based on their similarity to a centroid embedding that represents our ’few-shot’ sample. We randomly select 10 problems (description and code) from HumanEval and 10 from MBPP (to avoid overfitting to HumanEval) as our sample, denoted $P=[p_0,p_1,..,p_{20}]$ in Equation 6. The centroid embedding $p_{emb}$ is obtained using a version of CodeBERT [26], which has been further trained on CodeSearchNet [38] using the Replaced Token Detection loss [20]. The model is denoted by $f$ in Equations 6 and 7.

$$p_{emb}=\frac{1}{|P|}\sum _{i=1}^{P}f(p_i)$$ (6)

$$distance(x_i)=MSE(f(x_i),p_{emb})$$ (7)

A training example $x_i$ is ranked/chosen by its Mean Squared Error (MSE) $distance$ (lowest first) from $p_{emb}$. PanGu-Coder is then fine-tuned for five epochs with training data incremented by 10K, up to 60K examples, which is 50% of the total data. The pass@k are shown in Figure 6.

(a) pass@$1$

(b) pass@$10$

(c) pass@$100$

Figure 6: The pass rates as we increase the fine-tuning data. Each time, 10K nearest examples are added. The red line shows the pass@$k$ when PanGu-Coder is fine-tuned with all 120K examples.

Perhaps unsurprisingly, the competitive programming share of the fine-tuning data increases from 46.4% (in the full 120K dataset) to 68.7% after our similarity-based selection. We observe that for all pass@$k$, the full (dataset) score is reached or exceeded by fine-tuning on 25% - 50% of the larger dataset (see Figure 6). Using this method, we are able to reduce model compute requirements while maintaining (even surpassing) the original pass rate. If we wish to fine-tune PanGu-Coder on some target data distribution in the future, we require a small “few-shot” sample of problems (possibly written from scratch) to effectively subsample our larger training dataset to achieve a good pass rate while reducing computational resources.

4 Related Work

Recently, there has been an increasing interest in applying deep learning methods for NLP to code understanding and generation tasks. Among others, this is evidenced by the introduction of new NLP venues, e.g. Natural Language Processing Advancements for Software Engineering (NLPaSE) [1], which had its second installment at APSEC 2021 and the 2nd International Workshop on Software Engineering Automation: A Natural Language Perspective (NLP-SEA 2021) [5] at ASE 2021. Other new venues include the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021) at ACL 2021 [45], the Deep Learning for Code (DL4C) Workshop at ICLR 2022 [62] and the 1st International Workshop on Natural Language-based Software Engineering (NLBSE 2022), co-located with ICSE 2022 [63].

4.1 Pre-trained Language Models for Programming Language

From an NLP research perspective, the recent advancements in code understanding and generation have been mostly focused on revisiting proven effective natural language understanding (NLU) and generation (NLG) methods. CodeBERT [26], for instance, was trained using a combination of Masked Language Modeling inspired by [23, BERT] and Replaced Token Detection [20, ELECTRA]. CodeT5 [70] and PYMT5 [22] were built on top of [58, T5] while UniXcoder [31] was based on UniLM [24]. The most notable works in the area are discussed next. We distinguish between models focused on code understanding, where the goal is to learn contextual representations of source code, and code generation, where the aim is to translate between different programming and / or natural languages as well as to perform code completion and / or code repair.

Code Understanding

Research on this topic started with the seminal work of Alon et al. [4] who introduced code2vec[4], a neural model for representing snippets of code as continuous distributed vectors, extending the idea from Mikolov et al. [49, word2vec]. Inspired by the introduction of BERT [23], Kanade et al. [42] proposed CuBERT (Code Understanding BERT), a natural adaptation of the model trained on source code. CodeBERT [26] instead, used both source code and Natural Language (NL) with a discriminative objective, inspired by [21, ELECTRA]. The model receives a masked NL-code pair and the generator then predicts the masked tokens (the prediction might be different from the original token). Subsequently, CodeBERT is trained to predict which tokens were replaced by the generator. Guo et al. [30] noted that previous pretrained models treat code snippets as sequences of tokens while ignoring the inherent structure of code. They presented GraphCodeBERT, which showed that incorporating the data flow, i.e. a semantic-level structure of code extracted from the abstract syntax tree (AST), leads to richer code representations. Jiang et al. [41, TREEBert] instead used the actual AST together with code snippets.

SynCoBERT [68] trained the model with natural language, code and ASTs, but updated the training objective to include identifier prediction and AST Edge Prediction. In addition, Contrastive Learning [33, 17] has been used as a training regime to help associate snippets of code with syntactically diverse but equivalent programs as well as with corresponding natural language descriptions [40, 50, ContraCode] or with different modalities [69, Code-MVP].

Code Generation

Earlier approaches for code generation were mostly limited to simplified versions of code completion or code translation tasks [35, 59]. However, recent work has been proposed to tackle more complex problems, e.g. text-to-code generation (program synthesis) and code summarization. Lu et al. [48] introduced CodeGPT, an adaptation of GPT [57] applied to code, as a baseline for the CodeXGLUE benchmark. TransCoder [61] presented a pre-trained language model specifically focused on unsupervised translation of code between different programming languages. PLBART [3] employed a denoising objective over code and natural language via token masking, token deletion, and token infilling as noising strategies. CoText [54] was build on top of T5 with a special focus on multi-task learning over multiple programming languages. Another model variant was introduced by Fried et al. [27, InCoder]. Instead of training to perform code/text generation in a single left-to-right pass, InCoder is also able to edit existing/partial programs via an infilling training objective. The training regime involved randomly replacing spans of code/comments with a placeholder and asking the model to generate the replaced lines.

More recently, a number of large pretrained language models have been proposed, primarily focused on the task of text-to-functional-code generation. Chen et al. [16] introduced CodeX [16], a set of GPT-based language models trained on publicly available code from GitHub, up to 12B parameters in size. In addition, the authors fine-tuned their models using a set of training problems from competitive programming websites and repositories with continuous integration for better program synthesis, called Codex-S. Codex-D was fine-tuned on the reverse task of generating the program description given a particular function/code.

Li et al. [47] introduced AlphaCode, a set of sequence-to-sequence models with up to 41B parameters, trained on data from programming competitions, e.g. Codeforces (similar to Codex-S) as well as GitHub code in several programming languages. AlphaCode produces problem solutions by overgeneration-and-ranking, i.e. the model samples multiple solutions and uses filtering and clustering to determine the best ones. Finally, CodeGen [51] was proposed as a conversational text-to-code approach using large language models with sizes of up to 16B parameters. CodeGen-NL was trained on The Pile [28], which contains around 6GB of Python code. CodeGen-Multi was then further trained on BigQuery, which includes data from 6 different programming languages (C, C++, Go, Java, JavaScript and Python). A third model, CodeGen-Mono, builds on top of CodeGen-Multi and was additionally trained on Python-only data.

4.2 Code Datasets and Evaluation

There is a number of datasets proposed for code understanding and generation tasks, many of which have recently been included in the CodeXGLUE benchmark [48]. CodeXGLUE comprises 14 datasets across 11 tasks, including clone detection, defect detection, cloze test and code search for understanding as well as code completion, code translation, code summarization, code repair, code refinement, document translation and text-to-code generation. Among these datasets are CONCODE [39], a Java-based dataset with more than 100,000 examples of Java classes built from public Github projects and containing environment information along with Javadocs and the corresponding code. CodeSearchNet [38] is a benchmark specifically designed to evaluate systems on semantic code search, i.e. a code retrieval task based on natural language queries.

The above tasks have mostly adopted NLP evaluation metrics, for example, text-to-code generation still widely uses CodeBLUE [60] and exact match of the outputs. CodeBLUE is an extension of the standard n-gram overlap metric BLEU [52], designed to improve the evaluation of generated code. More recently, evaluation datasets and metrics started focusing on the functional correctness of generated programs. Chen et al. [16, Codex] introduced the HumanEval dataset (c.f. SectionLABEL:sec:eval_data) and Austin et al. [7] introduced the Mostly Basic Programming Problems [7, MBPP] dataset. Both datasets are relatively small, comprising of 164 and 974 instances where descriptions of programming problems and their corresponding coding solutions are included. These problems are then evaluated in terms of their behavior, instead of being treated as natural language (either syntantically or semantically). Specifically, the datasets provide unit tests that can be used to evaluate how close outputs are to what is expected. The main difference between HumanEval and MBPP is that the former does not include a training set as it was meant to be used for zero-shot evaluation. Other datasets, such as APPS [34], containing 10,000 problems and Code Contests [47, CC] (13,610 problems) (c.f. Section 3), leverage data from existing online coding platforms, resulting in a significantly larger dataset size but also more noise.

5 Conclusions

In this report, we presented PanGu-Coder, a pre-trained language model for the task of text-to-code generation. PanGu-Coder is based on the Pangu-$\alpha$ architecture and was initially pre-trained on raw natural language and programming data using the CLM objective, and subsequently trained specifically on pairs of docstrings and functions using combinations of Code-CLM, Docstr-MLM, and Docstr-MCLM training objectives. Zero-shot evaluation of PanGu-Coder on the HumanEval and MBPP datasets, designed to measure whether outputs comprise functionally correct programs, shows that this training can help reach equivalent or better performance than similarly sized models while using a smaller context window and less training data. Further analysis examined the impact of various decoding and sampling strategies, training objectives, and embedding sharing.

PanGu-Coder-FT demonstrated that the scores of the base model can be improved at a faster rate by curating data more closely related to our target task as the model is quite sensitive to mismatches and shifts in fine-tuning data distributions. If we possess a small set of example problems from the target distribution, this sensitivity can be somewhat mitigated by choosing a subset of our fine-tuning data based on the similarity to a centroid embedding. We also evaluated various post-processing methods, some of which allowed us to significantly improve pass rates by filtering out failing programs.

6 Acknowledgements

The authors thank Wei Zhang, Jun Yao, Qian Zhao, Feng Xu, Zongwei Tan for their great support to this work. The authors also thank Philip John Gorinski for his feedback on an early version of this report.

Appendix A Qualitative Analysis of Example Outputs

We show some examples of code solutions generated by the different versions of PanGu-Coder after stage-1 pre-training (c.f. Section LABEL:sec:stage_1_training), after stage-2 training with the Code-CLM objective (c.f. Section LABEL:sec:stage_2_training), and after fine-tuning (c.f. Section 3).

In this first example from HumanEval in Table LABEL:tab:example_1, we can see that both Stage-1 and Stage-2 models are attempting to generate solutions, but only the fine-tuned model produces a correct one.

Prompt