1 Introduction

One of the ultimate goals of task-oriented conversational systems is the ability to flexibly bootstrap new dialogue functionalities across diverse domains of user interest. For example, once we have successfully built a dialogue assistant that can handle restaurant booking queries, we would ideally like that assistant to quicklystart servicing hotel reservation queries without too much additional manual effort. Unfortunately in the modern conversational assistant ecosystem, the work required to scale up functionalities is often linear with respect to the number of desired domains.

For modelling improvements to claim meaningful progress towards generality, the improvements must extend beyond a single dataset and instead hold across several different dialogue tasks and corpora. We argue that one of the core roadblocks in progressing the generality of conversational systems toward this desired state is a lack of standardization in both datasets and evaluation used by the community. These problems have been noted more broadly in the natural language understanding field, inspiring numerous efforts to propose unified benchmarks spanning multiple downstream tasks across different corpora with consolidated evaluation procedures (Wang et al., 2018, 2019; McCann et al., 2018).

While today there is a reasonable abundance of available corpora for building data-driven dialogue systems (Serban et al., 2018), little work has been done to bring together the efforts of researchers to reflect the properties we care about in our systems: statistical learning that is data-efficient and robustly transfers across domains and tasks.

In this work we propose DialoGLUE, a public benchmark consisting of 7 diverse task-oriented spoken-language datasets across 4 distinct natural language understanding tasks including intent prediction, slot-filling, semantic parsing, and dialogue state tracking. Many of these datasets include multiple system functionalities and in total, the DialoGLUE benchmark covers over 40 different domains. Our benchmark is designed to encourage dialogue research in representation-based transfer, domain adaptation, and sample-efficient task learning. Furthermore, it consists entirely of previously-published datasets that have reported results, as these resources have been vetted by the broader community to be sufficiently difficult and interesting.

As part of DialoGLUE, we also release evaluation scripts and competitive BERT-based baselines on the downstream tasks. We introduce ConvBERT, a BERT-base model which has been trained on a large open-domain dialogue corpus. In combination with task-adaptive pre-training and multi-tasking, ConvBERT matches or exceeds state-of-the-art results on five of the seven DialoGLUE datasets. Most notably, we attain a +2.98 improvement in the joint goal accuracy over the best dialogue state tracking models on the MultiWOZ corpus. While our baselines demonstrate the efficacy of task-adaptive finetuning in transferring across datasets, there is still plenty of headroom in the aggregated benchmark scores encouraging further research.

In summary, the contributions of this work are four-fold: (1) a challenging task-oriented dialogue benchmark consisting of 7 distinct datasets across 4 domains, (2) standardized evaluation measures for reporting results (3) competitive baselines across the tasks, and (4) a public leaderboard for tracking scientific progress on the benchmark.

The DialoGLUE leaderboard and evaluation scripts are hosted using the EvalAI platform (Yadav et al., 2019). Our code and baseline models are open-sourced.²

2 Related Work

2.1 NLP Benchmarks

In part, the development of unified benchmarks have helped drive progress towards more general models of language. The GLUE (Wang et al., 2018) and SuperGLUE benchmarks (Wang et al., 2019) have provided a consistent benchmark that allows pre-trained language models (Devlin et al., 2018; Radford et al., 2018) to be evaluated on a variety of tasks.

While GLUE and SuperGLUE are concentrated on language understanding tasks, decaNLP (McCann et al., 2018) consists of a broader set of NLP tasks including question answering, machine translation and summarization.

Within conversational systems, the recently proposed Dialogue Dodecathlon is a benchmark for knowledge-grounded, situated, and multi-modal dialogue generation consisting of several open-domain dialogue datasets (Shuster et al., 2019). In contrast to the Dialogue Dodecathlon, DialoGLUE focuses on task-oriented dialogue and language understanding.

DialoGLUE is inspired by GLUE and SuperGLUE, however our efforts differ in that we aim to produce a benchmark for natural language understanding in the context of task-oriented dialogue. Task-oriented dialogue poses a unique set of challenges; dialogue is intrinsically goal-driven, multi-turn and often informal/noisy (Henderson et al., 2019; Zhang et al., 2019b). Through DialoGLUE, we hope to provide a benchmark for assessing models that aim to tackle these unique challenges across several tasks within task-oriented dialogue.

2.2 Pre-trained Models

Large-scale pre-training has attained significant performance gains across many tasks within NLP (Devlin et al., 2018; Radford et al., 2018). Through self-supervised pre-training on large natural language corpora, these models gain generalized language understanding capabilities that transfer effectively to downstream tasks (Wang et al., 2018), including intent prediction (Chen et al., 2019; Castellucci et al., 2019) and dialogue state tracking (Heck et al., 2020). However, recent work has demonstrated that leveraging dialog-specific pre-trained models, such as ConveRT (Henderson et al., 2019; Casanueva et al., 2020) obtains better results.

Large-scale pre-training on open-domain dialogue has demonstrated surprising generality, with models like DialoGPT (Zhang et al., 2019b), Meena (Adiwardana et al., 2020) and Blender (Roller et al., 2020) achieving response generation performance competitive with humans in certain settings. ConveRT has demonstrated that pre-training on open-domain dialogue transfers to task-oriented dialogue, with significant performance improvements over BERT on both intent prediction (Casanueva et al., 2020) and slot filling (Coope et al., 2020). Inspired by these results, we fine-tune BERT (Devlin et al., 2018) on a large open-domain dialogue corpus prior to fine-tuning on the downstream tasks of DialoGLUE.

2.3 Task-Adaptive Training

Though large-scale pre-training results in strong performance when transferring to downstream tasks, performing self-supervised training on a target dataset allows the model to better adapt to the dataset prior to fine-tuning (Mehri et al., 2019; Gururangan et al., 2020). Since our target datasets differ from the pre-training data, we hypothesize that task-adaptive training will allow pre-trained models to better adapt to task-oriented dialogue.

3 DialoGLUE Tasks

The DialoGLUE benchmark consists of 7 different datasets spanning 4 different tasks: intent prediction, slot filling, semantic parsing and dialogue state tracking. These datasets all share the common goal of language understanding in the context of dialogue, making them suitable for DialoGLUE.

3.1 Intent Prediction

Intent prediction is the task of extracting meaning from a natural language utterance in order to understand the user’s goal (Hemphill et al., 1990; Coucke et al., 2018). We leverage three different datasets for the task of intent prediction, all of which span several domains and consist of many different intents.

banking77 (Casanueva et al., 2020) contains 13,083 utterances related to banking with 77 different fine-grained intents. Despite only consisting of a single domain, this dataset is challenging as it requires fine-grained differentiation betweeen very similar intents like card payment wrong exchange rate and wrong exchange rate for cash withdrawal.

clinc150 (Larson et al., 2019) contains 23,700 utterances spanning 10 domains (e.g., travel, kitchen/dining, utility, small talk, etc.) and 150 different intent classes. This dataset also consists of out of scope utterances, which do not belong to any of the other intents and must therefore be classified as out of scope.

hwu64 (Liu et al., 2019) includes 25,716 utterances for 64 intents spanning 21 domains (e.g., alarm, music, IoT, news, calendar, etc.). The domains and intents of this dataset are similar to ones we expect users to ask a virtual assistant (e.g., Alexa, Google Assistant, Siri).

Casanueva et al. (2020) forego a validation set when using these datasets for intent prediction and instead only use a training and testing set. We instead designate a portion of the training set to be the validation set.

3.2 Slot Filling

Slot filling, a vital component of task-oriented dialogue systems, is the task of identifying values for pre-defined attributes in a natural language utterance. Following the set up of Coope et al. (2020), we include two slot filling datasets in DialoGLUE.

restaurant8k (Coope et al., 2020) comprises of 8,198 utterances from a commercial restaurant booking system and consists of 5 slots (date, time, people, first name, last name).

DSTC8 (Rastogi et al., 2019) consists of slot annotations extracted by Coope et al. (2020) spanning 4 domains (buses, events, homes, rental cars) for a total of 5,569 utterances.

For both these datasets, the value for a particular slot will always be a contiguous span of the utterance. For some utterances, the expected slot (e.g., “people”, “time”) is provided. This allows an otherwise ambiguous utterance like “four” to be interpreted either as “four people” or “four o’clock”.

3.3 Semantic Parsing

TOP (Gupta et al., 2018) is a dataset of 44k utterances wherein each utterance is annotated with a hierarchical semantic representation. Each utterance has a top-level intent, which serves as the root of the tree. Every word of the utterance is a leaf node of the tree. The sub-trees correspond to different slots and intents, with the dataset having an average tree depth of 2.54. TOP covers two different domains, navigation (directions, distance, traffic) and events.

3.4 Dialogue State Tracking

MultiWOZ (Budzianowski et al., 2018; Eric et al., 2019) is a multi-domain dialogue dataset that contains 7 domains including restaurants, hotels and attractions. We use MultiWOZ for dialogue state tracking, the task of interpreting the user utterances throughout a dialogue in order to maintain a state representation of their requests. Dialogue state tracking is an important component of task-oriented dialogue systems; in order to fulfill a user request, it is necessary to track the user’s goals over the course of multiple turns.

4 Methods

In this section, we describe several methods employed for the DialoGLUE tasks. We begin by describing the architectures for the 4 different tasks, all of which are built around an underlying BERT-like model. We then discuss ConvBERT, a BERT model that was fine-tuned on a large open-domain dialogue corpus. Finally, we describe our use of task-adaptive masked language modelling, which allows us to better adapt our pre-trained models to the DialoGLUE tasks.

4.1 Architectures

Intent Prediction: We fine-tune a BERT model to encode an utterance and predict its intent. Specifically, we use the pooled representation output by a BERT model and pass it through a linear layer to predict the intent class.

Slot Filling: For slot filling, we represent the problem as IOB tagging (Ramshaw and Marcus, 1999)wherein every token in the utterance is labeled as either being the beginning of a slot value (B-), inside a slot value (I-) or not belonging to a slot value (O). We use BERT to produce a latent representation of each token, which is passed through a linear layer that predicts the appropriate tag (e.g., “B-time”, “I-people”).

Semantic Parsing: We transform the hierarchical representations of the TOP dataset into (i) a top-level intent for the utterance which corresponds to the root of the tree and (ii) a label for each word of the utterance which is the path from the root to each leaf node (which is always a word). Given this, we train a model to simultaneously predict the top-level intent for the utterance using the BERT pooled representation and the labels for each word using BERT’s latent representation of each word.

Dialogue State Tracking: Our state tracking architecture is inspired by the TripPy of Heck et al. (2020) which uses an underlying BERT model and a triple copy strategy to perform state tracking. The TripPy model uses (i) span prediction and a copy mechanism to extract values from a user utterance, (ii) a copy mechanism over concepts mentioned by the system utterance and (iii) a copy mechanism over the DS memory, the existing dialogue state.

These architectures are held consistent throughout our experiments. Given a BERT-like encoder, we can plug it into all of the aforementioned architectures and evaluate its performance on all of the DialoGLUE tasks. This allows us to assess the quality of the underlying encoders, with confidence that performance improvements come from the improved representational power of the encoders.

4.2 ConvBERT

Though pre-trained models (e.g., BERT) have exhibited strong language understanding capabilities, recent work has suggested that they may be insufficient for modelling dialogue, due to dialogue’s intrinsically goal-driven, linguistically diverse, multi-turn and often informal/noisy nature (Henderson et al., 2019; Zhang et al., 2019b). The unique challenges of modelling dialogue have been addressed by training on large amounts of conversational data, from online forums. We extend these efforts by fine-tuning BERT on a large open-domain dialogue corpus consisting of nearly 700 million conversations to produce ConvBERT.

By training ConvBERT with large amounts of open-domain dialogue, we hypothesize that it is better able to produce semantically meaningful latent representations of utterances and multi-turn dialogues than a BERT model. Specifically, we fine-tune an uncased BERT-base for 4 epochs using a masked language modelling objective. Here our input representation is the last 3 turns of dialogue context followed by the [SEP] token and then the dialogue response. We truncate the entire input to have sequence length 72, and we train using the Adam optimizer with an initial learning rate of 3 ⋅ 10⁻⁴.

4.3 Task-Adaptive Training

Task-adaptive training is the process of adapting a pre-trained model to a specific task or domain, by performing self-supervised training prior to fine-tuning on the downstream task. This has been shown to help with domain adaptation (Mehri et al., 2019; Gururangan et al., 2020). To adapt BERT-based encoders to the various DialoGLUE tasks, we leverage task-adaptive training. Specifically, we perform self-supervised training with the masked language modelling (MLM) objective on each dataset. We explore both (i) pre-training with MLM prior to fine-tuning on the specific task and (ii) multi-tasking by simultaneously performing self-supervised MLM and fine-tuning on the task. An example experimental setting is as follows: (1) start with the pre-trained ConvBERT model, (2) do MLM pre-training on the utterances of HWU, (3) fine-tune on HWU using the intent prediction architecture and simultaneously perform self-supervised MLM training on the utterances of HWU.

To further study the benefits of self-supervised training, we fine-tune both BERT and ConvBERT with masked language modelling over all the DialoGLUE datasets. In this manner we aim to adapt the two pre-trained models to task-oriented dialogue through self-supervised training.

5 Experiments

5.1 Experimental Setup

The experiments are carried out with four BERT-like models: (1) BERT-base, (2) ConvBERT which is BERT trained on open-domain dialogues, (3) BERT-DG which is BERT trained on the full DialoGLUE data in a self-supervised manner with masked language modelling and (4) ConvBERT-DG which is ConvBERT trained on the full DialoGLUE data in a self-supervised manner.

We carry out experiments with each of these four models in four different settings: (1) directly fine-tuning on the target task, (2) pre-training with MLM on the target dataset prior to fine-tuning, (3) multi-tasking with MLM on the target dataset during fine-tuning and (4) both pre-training and multi-tasking with MLM.

We perform self-supervised MLM pre-training for 3 epochs prior to fine-tuning. Fine-tuning on a target task is carried out until the performance on the validation set does not improve for 10 epochs. When multi-tasking, we alternate between fine-tuning on the target task and self-supervised training with MLM after every epoch.

To assess the effectiveness of our pre-trained models for few-shot learning, we carry out few-shot experiments. In such experiments, self-supervised training is performed only on the few-shot data, which is 10% of the full data. The MLM pre-training and multi-tasking is performed with only the few-shot versions of each dataset. However, both BERT-DG and ConvBERT-DG are trained with the full DialoGLUE data, albeit in a self-supervised manner, meaning that they see more in-domain data than either BERT or ConvBERT in the few-shot experiments. For all the few-shot experiments, we train five times with different random seeds and report the average performance across the five runs.

5.2 Evaluation

Our evaluation metrics are consistent with prior work on these datasets. For intent prediction (banking77, clinc150, hwu64) we use accuracy. For the slot filling tasks (restaurant8k, DSTC8), we use macro-averaged F-1 score as defined by Coope et al. (2020). We use exact-match for TOP, which measures how often we exactly reconstruct the hierarchical semantic representation. For MultiWOZ, we use joint goal accuracy.

5.3 Results

The results of the full data experiments are shown in Table 1. We attain a performance gain over the vanilla BERT model (Devlin et al., 2018) across all seven datasets. These results highlight the efficacy of both the ConvBERT model and the task-adaptive self-supervised training. Across four datasets, the best results are attained by ConvBERT with both MLM pre-training and multi-tasking. We compare to prior work in Table 2, wherein we demonstrate that ConvBERT in combination with task-adaptive training, matches or exceeds state-of-the-art performance across five out of seven of the datasets. We attain a +2.98 improvement in the joint goal accuracy on the dialogue state tracking task of MultiWOZ. These strong results, which hold true across several datasets, suggest that large-scale pre-training on open-domain dialogue data in combination with task-adaptive self-supervised training transfers effectively to several task-oriented dialogue tasks.

When looking at the aggregate performance across all the DialoGLUE tasks, neither ConvBERT nor task-adaptive training attain improvements over BERT. However by combining these two approaches, there is a +0.81 improvement in the average score. This suggests that through large-scale pre-training on open-domain dialogue, ConvBERT learns skills that are valuable to DialoGLUE, however it is only through task-adaptive training that these skills are transferred effectively to the downstream tasks.

A noteworthy outcome of these experiments is the fact that the BERT model with task-adaptive self-supervised training sometimes outperforms ConvBERT. This indicates that in certain settings, it is more beneficial to perform self-supervised training on the the downstream dataset rather than a much larger dialogue dataset.

Performing self-supervised training across the combination of the DialoGLUE datasets gives mixed results. ConvBERT-DG attains a significant performance gain on MultiWOZ, suggesting that self-supervised training on other task-oriented dialogue corpora helps significantly in modelling MultiWOZ dialogues. Across other datasets, it is only marginally better than the ConvBERT model and sometimes worse. Aside from the unique case of MultiWOZ, it appears that self-supervised training with additional task-oriented dialogue data, beyond just the dataset in question, does not provide significant improvements. For two datasets, DSTC8 and TOP, there is a decrease in performance which may be indicative of catastrophic forgetting. Namely, the ConvBERT-DG model may have lost the language understanding capabilities captured by the ConvBERT model through the additional self-supervised training, and only partially recovers this through the task-specific self-supervised training. Future work should explore better mechanisms for performing self-supervised training across the combination of the DialoGLUE datasets, as well as multi-tasking across the seven tasks.

While our models achieve state-of-the-art performance across five of the seven tasks, they underperform on TOP and DSTC8. On the TOP dataset, the best models use sophisticated architectures which have been tailored to the task of semantic parsing (Einolghozati et al., 2019; Rongali et al., 2020). With the DialoGLUE benchmark, our objective is to improve the underlying language encoders in a manner that results in consistent performance gains across all of the tasks. We are more concerned with the aggregate improvement across the DialoGLUE benchmark, rather than the performance on a single task. As such, we try to avoid complex task-specific architectures when simpler models achieve competitive results.

The results of the few-shot experiments are shown in Table 3. The few-shot experiments are particularly important for assessing the generalizability of the methods and their ability to transfer to downstream tasks. In low data environments, self-supervised training on the entirety of the DialoGLUE datasets results in performance gains – with BERT-DG and ConvBERT-DG doing better than BERT and ConvBERT respectively. However, this is not entirely surprising as these models are exposed to more utterances from every dataset, albeit without any of the labels.

Most significantly, on MultiWOZ we see a 40 percent difference between BERT-DG and ConvBERT-DG over BERT and ConvBERT. For state tracking in particular, it appears that seeing additional dialogue data in a self-supervised setting, results in significant improvements. This may suggest that dialogue state tracking is more dependent on having semantically meaningful representations of dialogue.

Self-supervised training on the same dataset also helps significantly in few-shot environments. Across almost every dataset, the best result is obtained through some form of task-adaptive MLM training. Especially in settings with fewer training examples, adapting the pre-trained models to the domains of the dataset is necessary for good performance on the downstream task.

ConvBERT is also far more effective in the few-shot experiments, than it was in the full data experiments with a +1.96 point improvement in the aggregate score over BERT. While the full datasets may be sufficient to effectively transfer BERT to task-oriented dialogue, with only 10% of the data, the benefits of the large-scale open-domain pre-training are far clearer.

6 Conclusion

To facilitate research into producing generalizable models of dialogue, we introduce DialoGLUE, benchmark for natural language understanding in the context of task-oriented dialogue. We experiment with several baseline methods for the DialoGLUE benchmark, demonstrating the efficacy of large-scale pre-training on open-domain dialogue and task-adaptive self-supervised training.

To improve performance on DialoGLUE, future work should explore: (1) Large-scale pre-training that attains generalized language understanding capabilities which transfer effectively to task-oriented dialogue. (2) Mechanisms of adapting pre-trained models to specific tasks, beyond the task-adaptive masked language modelling we explore. In particular, we believe there to be potential in extending our preliminary exploration of self-supervised training on the combination of all the DialoGLUE datasets. (3) Multi-tasking across the seven datasets of DialoGLUE, as a means of transferring skills across the datasets.

The results of our experiments demonstrate that there is significant room for improvement on DialoGLUE, particularly in the few-shot settings. The DialoGLUE benchmark is hosted publicly and we invite the research community to submit to the leaderboard.³

References