Rethinking the Integration of Prediction and Planning in Deep Learning-Based Automated Driving Systems: A Review

Abstract

Automated driving has the potential to revolutionize personal, public, and freight mobility. Beside accurately perceiving the environment, automated vehicles must plan a safe, comfortable, and efficient motion trajectory. To promote safety and progress, many works rely on modules that predict the future motion of surrounding traffic. Modular automated driving systems commonly handle prediction and planning as sequential, separate tasks. While this accounts for the influence of surrounding traffic on the ego vehicle, it fails to anticipate the reactions of traffic participants to the ego vehicle’s behavior. Recent models increasingly integrate prediction and planning in a joint or interdependent step to model bi-directional interactions. To date, a comprehensive overview of different integration principles is lacking. We systematically review state-of-the-art deep learning-based prediction and planning, and focus on integrated prediction and planning models. Different facets of the integration ranging from model architecture and model design to behavioral aspects are considered and related to each other. Moreover, we discuss the implications, strengths, and limitations of different integration principles. By pointing out research gaps, describing relevant future challenges, and highlighting trends in the research field, we identify promising directions for future research.

6.6in(0.95in,0.15in) SUBMITTED TO REVIEW AND POSSIBLE PUBLICATION. COPYRIGHT WILL BE TRANSFERRED WITHOUT NOTICE.
Personal use of this material is permitted. Permission must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

1 Introduction

Automated Driving (AD) remains a challenging endeavour. It is usually split into the subtasks of perception, prediction, planning, and control [1, 2, 3]. Perception processes sensor inputs to create a model of the environment. Prediction and planning build upon this model and make future motion forecasts for surrounding traffic agents and a plan for the controlled ego vehicle. Traditional, modular systems (cf. Fig. 1) address prediction and planning as separate tasks. The predicted behavior of surrounding traffic agents is leveraged to plan a suitable behavior for the ego vehicle. However, this sequential ordering is inherently reactive and cannot represent bidirectional interaction between the ego vehicle and other traffic agents [4]. In fact, prediction and planning are no sequential problems and should be tightly coupled in automated driving systems [5, 6, 7].

1.1 Scope

In this work, we review methods to integrate prediction and planning into an automated driving system (ADS).

Prediction is the task of anticipating intents and future trajectories of observed traffic agents [8], and planning is about finding the best possible trajectory w.r.t. previously defined criteria for a controlled vehicle [9].
Deep Learning (DL)-based methods follow a data-centric approach to tackle these problems [10] and have led to significant improvements in many fields [11]. Works in prediction and planning increasingly employ DL-based approaches as well [12, 13, 14, 15] and represent an alternative to traditional, rule-based methods [16, 17] and constrained optimization [18]. In this work, we focus on DL-based methods.
The architecture of an ADS can take different forms, which are shown in Fig 1. Black-box methods employ a single neural network for perception, prediction and planning (c.f. Fig 1(a)), so that depending on the architecture, no boundaries between the modules can be drawn [9, 19, 20]. In contrast, modular systems consist of clearly defined components for different tasks [21, 22, 23, 24, 25]. Traditionally, they are arranged in a sequential order (c.f. Fig 1(b)). In this work, we emphasize the shortcomings of this approach and highlight works that integrate prediction and planning in a more sophisticated way (c.f. Fig 1(c)) to enable bi-directional interaction of self-driving vehicles and surrounding traffic. We refer to the component of an automated driving system that integrates prediction and planning as an Integrated Prediction and Planning System (IPPS).
We limit the scope of our work to mixed traffic without Vehicle-to-X communication between traffic agents. Furthermore, we exclude pedestrian motion forecasting, which poses a different problem that is subject to weaker dynamic constraints compared to vehicles [26] and has been thoroughly surveyed [27, 28, 29].

1.2 Contribution and Structure

In a thorough approach, this survey first reviews DL-based prediction and planning before focusing on how to integrate these two tasks. While surveys on classical methods [30, 31, 14, 32], stand-alone DL-based prediction [33, 34, 35, 15] or planning [36], and end-to-end (E2E) AD [37] already exist, to the best of our knowledge we are the first to review the integration of DL-based prediction and planning. We summarize our contributions as follows:

• We provide a comprehensive overview of common representations, architectures and state-of-the-art methods for DL-based prediction (Sec. 3) and planning (Sec. 4).

• We propose a categorization for the integration of prediction and planning, which addresses model architecture, system design and behavioral aspects. Further, we investigate and analyze the interrelations between these categories and their impact on safety and robustness (Sec. 5).

• We reveal gaps in state-of-the-art research and point out promising directions of future research based on the identified categorization (Sec. 6).

(a) Black box ADS

(b) ADS with sequential IPPS

(c) ADS with bidirectional IPPS

Figure 1: System architectures of automated driving systems. Dashed lines indicate, that due to the black-box nature of some E2E ADS, it can be unclear which network components do prediction and planning, so that no boundary around the IPPS can be drawn.

2 Task Definitions

In the following, we start by introducing the terminology and notation for the task definitions used in this work. We adopt a similar terminology to that proposed by [33] and partition the actors in a traffic scenario into the self-driving Ego Vehicle (EV) and the Surrounding Vehicles (SVs). The EV is equipped with sensors that provide information on the environment. The state history of vehicle $i$ over the time interval $t-t_{\text{obs}},\mathrm{\dots },t$ is

$$X_i=\{x_{t-t_{\text{obs}}},\mathrm{\dots },x_{t-1},x_t\}.$$ (1)

Each state $x$ comprises 2D or 3D positional information and further optional information like heading angle, speed, static attributes, or goal information in the case of the EV. Hence, $X_{\text{EV}}$ denotes the EV’s past states. Similarly,

$${\overline{\text{X}}}_{\text{SV}}=\{X_1,X_2,\mathrm{\dots },X_m\}$$ (2)

refers to the past states of all SVs. Analogously, the future states of vehicle $i$ within a prediction horizon of $t_{\text{pred}}$ are

$$Y_i=\{y_{t+1},y_{t+2},\mathrm{\dots },y_{t+t_{\text{pred}}}\}.$$ (3)

and the future states of all SVs are ${\overline{\text{Y}}}_{\text{SV}}$. In the following, state sequences are also referred to as trajectories.
Additional scene information, such as a semantic map or traffic signs and traffic light states are represented by $I$.

Prediction. Following [8], we state trajectory forecasting as the task of estimating a probability distribution

$$P_{\text{pred}}=P(\overline{\text{Y}}|\overline{\text{X}},I)$$ (4)

that maps state histories of $m$ observed vehicles in $\overline{\text{X}}$ to the future trajectories of $n$ predicted vehicles in $\overline{\text{Y}}$. The distribution $P_{\text{pred}}$ accounts for inherent uncertainties in the forecasting task and is often modeled by a discrete set of samples with corresponding probabilities [38, 39, 40, 41]. Some methods omit scene information $I$ and infer trajectories based on a state history alone [42, 43, 44].

TABLE I: Forms of Prediction

$\overline{\text{Y}}$	$\overline{\text{X}}$	Task
$Y_{\text{SV}}$	${\overline{\text{X}}}_{\text{SV}}$, $X_{\text{EV}}$	Single-Agent Prediction
${\overline{\text{Y}}}_{\text{CV}}$	${\overline{\text{X}}}_{\text{SV}}$, $X_{\text{EV}}$	Clique Prediction
${\overline{\text{Y}}}_{\text{SV}}$	${\overline{\text{X}}}_{\text{SV}}$, $X_{\text{EV}}$	Joint Prediction

Commonly $\overline{\text{X}}$ includes all vehicles in the scene, i.e. $X_{\text{EV}}$ and ${\overline{\text{X}}}_{\text{SV}}$. Depending on the vehicles in $\overline{\text{Y}}$, different variants of prediction can be formulated as stated in Tab. I. Single-agent prediction models the future trajectory for each SV individually [45]. However, estimating a joint prediction ${\overline{\text{Y}}}_{\text{SV}}$ from a set of $n$ single-agent predictions $Y_{\text{SV}}$ is not trivial since the number of possible combinations grows exponentially with the number of agents $n$. Not all these combinations are meaningful and finding realistic ones with heuristics and joint optimization is cumbersome [46]. To avoid this, joint prediction directly estimates a joint distribution for multiple SVs [47]. As not every single SV interacts with all other actors, ScePT [48] partitions the SVs into highly interactive cliques $\text{CV}\subset \text{SV}$ and models ${\overline{\text{Y}}}_{\text{CV}}$. We only discuss single-agent prediction and joint prediction since clique prediction can be seen as a special case of joint prediction that marginalizes over the remaining SVs. Moreover, the aspects discussed in Sec. 3 are also valid for clique prediction.

Planning. The task is to find a single suitable trajectory for the EV that can be passed on to a downstream motion controller. Hence, we define planning to be a function $f$ that maps the observational inputs ${\overline{\text{X}}}_{\text{SV}}$, and $X_{\text{EV}}$ as well as the context information $I$ to a future trajectory $Y_{\text{EV}}$, i.e.

$$Y_{\text{EV}}=f(X_{\text{EV}},{\overline{\text{X}}}_{\text{SV}},I).$$ (5)

In many cases, the $f$ also uses the prediction ${\overline{\text{Y}}}_{\text{SV}}$, i.e.

$$Y_{\text{EV}}=f(X_{\text{EV}},{\overline{\text{X}}}_{\text{SV}},I,{\overline{\text{Y}}}_{\text{SV}}).$$ (6)

The definitions show that planning can be considered a special case of single-agent prediction where the output distribution models only a single trajectory. However, in contrast to prediction, the planning trajectory must be conditioned on a navigation goal. Moreover, it has to be stable in closed-loop, i.e. it must be kinematically feasible and result in safe and efficient driving behavior when fed to a downstream controller.

3 Prediction

The core of prediction is understanding how the driving scene will evolve. Sec. 3.1 discusses how different scene representations allow local and global interactions. Sec. 3.2 reviews which NN designs are used to model interactions and extract descriptive features. Finally, Sec. 3.3 shows how the extracted features are mapped to trajectory predictions and how multimodality is modeled.

3.1 Scene Representation

Figure 2: Comparison of rasterized and sparse representation of the same traffic scene. While the former represents information with fixed-size grids, the latter uses a set of vectors.

Representing a scene means extracting the relevant subset of all available information and transferring it into a format that subsequent processing steps can utilize. In the context of trajectory prediction for automated vehicles, agent states $\overline{\text{X}}$ and map I are the most important information to represent. Two major representations exist in DL-based approaches: Rasterized and sparse (cf. Fig. 2).

Rasterized. Rasterized representations use dense, fixed-resolution grid structures that often have multiple channels [49, 50, 48, 51, 52]. Each channel encodes different information on agent states. They are commonly used in combination with rasterized high-definition maps (HD maps). Typical modalities are occupancy maps [51], object detections [53], lane markings [54], motion history [55], semantic information [56], traffic signs [57], traffic light states [57], and route information [58] (c.f. Fig 2.
DESIRE [8] is an early work that uses a rasterized representation for DL-based prediction. A bird’s eye view (BEV) representation is employed to fuse multiple EV sensor inputs into a shared representation. This allows the combination of different sensing modalities and establishes a common coordinate system for all vehicles, facilitating to model interactions within the scene [59].
Additionally, dense grids are well-suited for processing with powerful Convolutional Neural Network (CNN) [60] architectures [58, 61, 62, 10, 63, 64, 65, 66, 67, 68, 69, 70, 71]. However, embedding all observations in a grid structure leads to information loss due to quantization errors [72]. Moreover, as discussed in Sec. 3.2, the locally restricted receptive field of CNNs can hamper interaction modeling.

Sparse representations use vectors to describe all objects in a scene. The per-object vectors are then either encoded jointly, e.g., with a transformer, or individually and then aggregated, e.g., with Graph Neural Networks (GNN) [73]. Objects are represented by polygons, or point sets [6, 74, 41, 75, 61, 76, 38, 77]. The map is commonly described by a set of lanes represented by polylines and an adjacency matrix. Objects and map elements are then encoded into fixed-size latent features, for instance, via Multi-Layer Perceptrons (MLP) [78] or Recurrent Neural Networks (RNN) [79, 39]. To encode sequential signals like state histories $\overline{\text{X}}$, 1D CNNs [80], RNNs like LSTMs [81] and GRUs [82], and Transformers [83] are applied [41, 84, 76].
The timeline in Fig. 3 chronologically depicts impactful prediction methods and exhibits a shift towards sparse representations during the past years. Diehl et al. [72] proposed a graph representation for vehicle interaction modeling. Based on their work, two approaches emerged: Homogeneous and heterogeneous graphs. Homogeneous graphs encode objects of one class, for example, the road network [74, 41], whereas heterogeneous graphs combine different object classes in one graph [40, 85, 86, 87]. However, their work is focused on highway data and does not include map information. In contrast, VectorNet [74] proposes to represent all map elements and agents by a set of points. Agents are represented by a set of points along their motion history, while lanes are described by polyline points. Small objects, such as traffic signs, are represented by a single point, and extended elements are given by polygons defined by multiple points. Vectornet then processes each object with a homogeneous subgraph called polyline subgraph. A per-object feature is obtained by max-pooling. Next, all object features are aggregated in a heterogeneous global graph. Finally, the TV trajectory is decoded from the encoding of the node corresponding to the TV. Trajectron++ [40] and MFP [88] combine rasterized representations for HD maps with sparse representations for objects. Further, a few works use voxel representations that differ in their degree of sparsity [89], [90].
MultiPath++ [38] is the successor of MultiPath [56] and switches from rasterized to sparse representations. Their results show that the shift from rasterized to sparse can increase performance. The overall development in the field confirms this trend (c.f. Fig. 3).

Coordinate frame. Irrespective of the scene representation itself, the coordinate frame is a fundamental design choice.
Applying a global coordinate system with a fixed viewpoint for the whole scene [91, 46, 92, 6] is computationally efficient. However, the frame is not viewpoint-invariant, i.e., the predictions change if the coordinate system is moved within the scene. This leads to low sample efficiency and impairs generalization [93]. Even fixing the coordinate origin to the EV center does not lead to viewpoint-invariant predictions when predicting other agents than the EV [94].
In contrast, other models [95, 96, 74, 54] process a scene from the viewpoint of each agent. Thus, the processing is viewpoint-invariant. On the downside, computational complexity scales linearly with the number of agents and quadratically with the number of interactions [6].
An alternative way to achieve viewpoint invariance are pairwise relative coordinate systems introduced in GoRela [97], which describe the relation between agents instead of using a fixed coordinate frame. This allows the computation of the relation between static objects offline and greatly reduces computation. PEP [94] further reduces computation by using center-relative instead of pairwise relative coordinates, i.e. shifting the coordinate frame into the mean position of all processed agents before each transformation.
Another choice to increase robustness are Frenet representations that decompose positions into progress along and lateral offset from a lane [98, 99]. Either a single Frenet frame defined by the target vehicle is used for all agents [100] or each agent is represented in an individual coordinate frame [84, 101].

Figure 3: Contributions that got attention in the research fields of DL-based prediction, planning, and manual IPPS for automated driving. Nodes, stem, and label of each entry encode different aspects that are explained below the timelines. Works marked with $\uparrow$ and $\downarrow$ were slightly moved up and down the timeline, respectively, to avoid overlaps.

3.2 Interaction Modeling

A decisive aspect of prediction is to model the interaction between traffic agents represented in the scene [4, 137]. This includes interaction with map elements and other traffic agents. Modeling the relation of vehicles to static map elements like lane markings, traffic signs, etc. is important to understand feasible corridors in which trajectories could be localized. Interaction with other traffic agents are crucial to understanding social interaction.

RNNs. A few early prediction models like DESIRE [8] or Trajectron++ [40] combine RNNs with an aggregation operator like spatial pooling or attention. They either extract temporal per-agent features and then aggregate them across all agents or aggregate information first and extract joint temporal features afterwards.

CNNs. Alternatively, Fast and Furious [102], MTP [54], and Multipath [56] applied 2D convolutions to implicitly capture interaction within the kernel size. Compared to sequence processing-based approaches, higher importance is assigned to spatial interaction.

GNNs and Graph-Attention explicitly model interactions between individual agents by combining multiple agents’ features [138] and processing them with graph convolution operators [139, 73, 140, 141] or graph attention [142, 143, 144, 145] to aggregate information. Prediction networks like VectorNet [74], LaneGCN [41], TNT [75], HOME [104], DenseTNT [106], GOHOME [84], Multipath++ [38], LaFormer [111], and BAT [114] are examples of this development.

Transformers [83] have been widely adopted in prediction: InteractionTransformer [146], mmTransformer [76], AgentFormer [105], SceneTransformer [6], HiVT [108], Wayformer [109], Motion Transformer [110], MotionDiffuser [112], Trajeglish [113] and many further works [147, 148, 149, 150, 110, 151, 152, 153, 154, 155, 156] exploit the global receptive field and attention mechanism of Transformers. All SVs can be predicted simultaneously [76], and the impact of vehicles on each others’ behaviors across different timesteps can be modeled with attention layers [105]. Attention layers can be applied along the spatial and temporal dimension jointly [105], or individually [157]. The latter can be divided into sequential [158, 159] and interleaved [6, 109] architectures. Experiments with Wayformer [109] and SceneTransformer [6] show that joint and interleaved processing outperform sequential approaches. As, joint processing causes higher computational burden, the interleaved paradigm balances best between performance and computational demand.

The timeline in Fig. 3 reflects how interaction models changed together with the scene representation. After CNNs and RNNs were applied to rasterized scene representations in early works, GNNs, attention, and Transformers took over since the advent of sparse representations.

Across all architectures, the prevalent concept is to extend interaction from a local to a global scale. HiVT’s ablation study [108] shows that both, local and global attention, are beneficial for prediction and that hierarchical attention yields the best performance.

3.3 Trajectory Decoding

The final step of prediction is to generate trajectories from latent features. As introduced in Sec. 2, this step models a probability distribution $P_{\text{pred}}=P(\overline{\text{Y}}|\overline{\text{X}},I)$ over possible future trajectories $\overline{\text{Y}}$ given observation $\overline{\text{X}}$ and scene context $I$. Here, we focus on two major aspects of trajectory decoding: The modeling of multimodality and the decoding principle.

Multimodality. The intention of SVs is inherently uncertain, making $P_{\text{pred}}$ multimodal [160]. Therefore, $P_{\text{pred}}$ is commonly either expressed as an explicit density function or estimated with a discrete trajectory set.

Explicit density functions. An intuitive approach to prediction is to explicitly model the probability density function of $P_{\text{pred}}$ [61, 66, 69, 70]. Continuous distributions like bi-variate Gaussian distributions [161], Gaussian Mixture Models [110, 162, 132], or discrete rasterized heatmaps [104, 84] can be found in many prediction works. Alternatively, object-agnostic representations like occupancy maps [53, 163], or flow fields [164, 165, 70] can be used. These representations handle multimodality implicitly [61] and were shown to be more robust against perturbations [62]. However, since no discrete trajectories are decoded, it is difficult to compare such representations with expert logs for performance evaluation.

Discrete trajectory sets are either created by sampling from an intermediate distribution or by the model design. To sample a diverse trajectory set from an intermediate distribution, generative models such as Generative Adversarial Networks (GANs) [166], Conditional Variational Autoencoders (CVAEs) [167], normalizing flows [168] or denoising diffusion [112], can be used [8, 89, 169, 170, 171]. Alternatively, recent methods apply several prediction heads to the features extracted by a common backbone to decode a set of diverse trajectories [38, 41, 172, 109]. Other ways to obtain diverse predictions are training loss functions [75], entropy maximization [39, 173], variance-based non-maximum suppression [174], greedy goal sampling [55, 97], divide and conquer strategy [175], evenly spaced goal states [176], or with pre-defined anchor trajectories [56, 177]. Nevertheless, a guarantee of coverage can only be given by specifying high-level behaviors [124] or regions of the map that must be covered by at least one prediction [76].

Decoding principles. We classify the decoding principles into one-shot, two-shot, and n-shot methods as depicted in Fig. 4. While it is a common practice in planning to decode trajectories as a sequence of actions that translate into Cartesian space using a kinematic model (cf. Sec. 4.2), only a few works do so in prediction [178, 179]. Instead, future waypoints are predicted directly from latent features [41, 38, 111, 109, 180].

Single-shot approaches are split into trajectory regression and anchor trajectory classification.
Trajectory regression directly decodes the latent features to a trajectory using a NN, typically an MLP [54, 74, 41, 38, 6, 107, 108, 109]. Especially Transformer-based architectures use direct trajectory regression decoding (cf. Fig. 3). However, these methods are prone to predict trajectories that leave the road, cut corners, or are kinematically infeasible [97]. Specifically designed loss functions for offroad driving [181, 182, 39, 183, 184], driving direction compliance [185] or distance to the centerline [186] can alleviate the problem.
Anchor trajectory classification states prediction as a classification task. Thus, a probability is estimated for a set of pre-defined trajectories [55, 177]. Defining anchor trajectories can ensure the feasibility of the prediction and allows to impose hard kinematic constraints. However, rare trajectories might not be covered by the anchors.

Two-shot methods structure trajectory decoding into meaningful subtasks that increase explainability and are more flexible than anchor trajectory classification. They comprise endpoint and completion as well as initial trajectory and refine strategies.
Endpoint and completion methods are based on the assumption that a trajectory is mostly defined by its endpoint [25]. The endpoint can either be regressed [104], [84] or classified from a pre-defined set [75], [106], [97]. Usually, classification-based methods additionally regress an offset of the classified endpoint [75]. HOME [104] estimates the endpoint probability distribution as a rasterized map, samples potential endpoints, and then regresses the trajectory using an MLP.
Initial trajectory and refine approaches combine the ideas of anchor trajectory classification and endpoint and completion. Instead of an endpoint, an initial trajectory is classified or generated. Then, a refinement step adapts the trajectory, for example, by offset regression for each predicted waypoint [102, 56, 111]. TPNet [187] and DCMS [188] first regress an endpoint, then generate trajectory proposals and finally refine them.

N-shot methods are either initial trajectory and refine or autoregressive formulations.
Initial trajectory and refine strategies in their n-shot variants are similar to the two-shot version. The only difference lies in the refinement step, which is realized as a recurrent optimization [8, 112]. For example, MotionDiffuser [112] samples a noisy trajectory and repeatedly applies denoising diffusion to refine the trajectory.
Autoregressive approaches use recurrent decoding to perform a stepwise roll-out into the future. Iteratively, the next waypoint is predicted, and information on the resulting scene is added to the latent features [103, 189, 40, 76, 105].
Autoregressive decoding can help to enforce high-quality social interactions and scene understanding [97] but might lead to compounding errors since predicted information is used to make further predictions [190]. Further, the computational effort and computation time of these methods are usually higher than for one-shot or two-shot methods.

Fig. 3 shows the development of trajectory decoding using the categories defined in this Section. While endpoint and complete decoding gained popularity, auto-regressive formulations were recently used less often, assumingly due to computation time deficiencies. Trajectory regression decoding was always used but became prevalent with the rise of Transformer-based interaction modeling in prediction.

Figure 4: Principles of trajectory decoding.

3.4 Benchmarks

Various prediction datasets provide training, validation, and test splits for benchmarking. The most prominent ones are nuScenes [191], WOMD [192], Argoverse [193], Argoverse2 [194], inD [195], highD [196], rounD [197], exiD [198], and Interaction [199]. Evaluation is carried out by comparing the predicted trajectory of an SV to its ground truth future trajectory. To account for the inherent multimodality benchmarks are commonly based on a winner-takes-all evaluation. This means the model predicts a fixed number of output trajectories, e.g., six [194] or ten [191], and the evaluation takes only the best one among them into account. Alternatively, predictors that output a parametric distribution are often evaluated based on the Negative log likelihood (NLL) of the ground truth under the predicted distribution. This assessment also factors in the uncertainty predicted by the model and evaluates the whole distribution instead of only a single trajectory.

4 Planning

The planning task is to find a trajectory $Y_{\text{EV}}$ for the EV that factors in safety, comfort, kinematic feasibility, and goal-directedness based on the observations $X_{\text{EV}}$ and $X_{\text{SV}}$ as well as additional context $I$ and optionally ${\overline{\text{Y}}}_{\text{SV}}$ (cf. Sec. 2). In this section, we will first give a comprehensive overview of common input (Sec. 4.1) and output representations (Sec. 4.2). Then, we address goal-conditioning in Sec. 4.3 before we categorize existing works and discuss common paradigms in Sec. 4.4. Finally, we briefly describe existing benchmarks in Sec. 4.5.

4.1 Input Representation

We classify planning input representations into interpretable intermediate representations and latent features.

Interpretable intermediate representations such as rasterized or sparse scene representations (that were discussed thoroughly in Sec. 3.1) are common inputs to planning components in modular ADSs. Thus, many works exist that build on rasterized [58, 67, 124, 200] or sparse inputs [125, 131, 201, 130, 24]. Similar to prediction, a clear trend can be observed from rasters to sparse inputs.

Latent features. E2E ADSs directly map sensor information to actions (cf. Fig. 1) [9, 202, 20]. Thus, instead of $X_{\text{EV}}$, ${\overline{\text{X}}}_{\text{SV}}$, and $I$, the planner’s inputs are latent features.
The benefit of using learned latent intermediate representations is that no manual engineering effort is needed to design the interfaces between different modules. The main drawback of hand-engineered interfaces is that they have to apply to the long-tailed distribution of potential driving scenarios, and every piece of information discarded by the interface cannot be recovered. For instance, if a vehicle is currently occluded, but its presence can be inferred from the reflection of its headlight in another car, bounding box representations [58], occupancy fields [70], or affordances [115, 119] will not be able to propagate this information to downstream modules [203, 133]. In contrast, latent feature representations allow uncertainty propagation, enabling downstream modules to compensate for errors at earlier stages, such as undetected vehicles [61].
The main disadvantage of latent feature representations is the lack of interpretability. In case of failures, evaluating which part of the system caused the error is extremely difficult. This aggravates tuning and debugging of pure E2E systems. Interpretable E2E systems seek to compensate for this by producing additional intermediate representations [66]. In contrast to modular stacks, these interpretable intermediate outputs are not used by the planner. Instead, they are only used for additional supervision and model introspection.

The timeline in Fig. 3 reveals that interpretable input representations are becoming increasingly popular compared to latent input features.

4.2 Output representation

The planning output is typically represented as a sequence of future states or control actions.

Future states are given by a sequence of future 2D position and heading angles representing the planned trajectory of the EV over time [22, 62, 58]. This trajectory is then passed to a downstream controller tasked with tracking it [22]. The trajectory representation is well interpretable and can be easily checked for collisions, traffic law violations, or deviations from the driveable surface [129, 68]. Moreover, it is agnostic of the kinematic parameters of the controlled vehicle, enabling generalization to other vehicles. However, the ultimate driving performance also depends on the downstream controller and how well the planner and controller are tuned for each other. Validating and testing the system can require iterations between both components, which slows down development.

Future actions are an alternative representation of planned behavior. This is mostly adopted by E2E ADSs that regress acceleration and steering angle without the intermediate trajectory representation [117, 118, 119]. Besides ensuring kinematic feasibility, it can promote comfort as it is directly related to the magnitude of the actions.
However, the resulting behavior depends on the controlled vehicle’s individual dynamics [204, 19]. Hence, generalization to other vehicles is limited.

Fig. 3 depicts that early E2E ADSs planned in actions, while trajectory outputs recently gained popularity.

4.3 Goal Conditioning

The overall task of ADSs is to navigate safely toward a destination. Hence, goal-directedness is one main criterion that determines the suitability of a potential plan. Lane-level route information can be provided to the planner by a navigation system. Popular benchmarks provide route information through a high-level command such as ’turn left’ or ’go straight’ [205], through sparse goal points along the route [205] or by annotating lanes within the map as on-route or off-route [206]. For example, in the widely used Carla simulator [205], goal locations are sparsely sampled along the route, and in each planning step, the closest goal location is provided to the planner. We define four categories to describe how this goal information is incorporated into the planning algorithm: input features, separate submodules, routing cost, and route attention. While the first two were originally introduced in [117], the latter were proposed more recently [201, 207, 126, 208].

Input features are the most straightforward way to incorporate goal information and have been widely adopted [120, 20, 124, 209, 210, 24, 63, 64, 118, 23, 122]. Different methods exist to pass this input to the planning module depending on how the route information is represented. If the lanes within the map information are annotated as on-route or off-route, then this information can be used as a separate semantic channel in raster maps [58]. Similarly, vector inputs describing the lane centerlines can have an additional flag reflecting this information [201].
High-level commands were used as input feature in LBC [124], LAV [67], and AD-MLP [208]. It was found that inputting high-level intention repeatedly at different stages of the network increases its generalization capability [23]. Transfuser [20] leverages sparse goal locations as 2D features in the final trajectory decoding step.
While using goal information as an input feature is simple, goal compliance is not guaranteed. On the contrary, [201, 117] demonstrated that the model could ignore this additional input and instead rely on other potentially spurious correlations, such as between the current and the future kinematic state.

Separate submodules are used only with high-level commands, which serve as a switch between command-specific submodules [117, 70]. Balancing the dataset to prevent it from being dominated by a single driving mode, such as lane following, is avoided in this setup, as each block is only trained on a single high-level command. However, this approach requires a fixed number of high-level commands to be defined in advance.

Routing cost. A third option is to plan trajectories by optimizing a cost term for routing [66, 69, 62]. Such cost functions can measure the remaining distance to a goal location [62], the progress along or distance to an on-route lane [66]. Goal-directedness is balanced with further competing objectives such as safety and comfort. Thus, the planner can flexibly react to difficult situations, e.g., it could drive off the road to prevent a collision. However, tradeoffs between safety and comfort versus progress can be undesirable.

Route attention applies spatial attention to relevant parts of the map, i.e., the route. This can be achieved by removing the off-route portion of the map from the input features, e.g., PDM-Open [126] and PEP [94] discard almost the entire map information and only keep points along a lane centerline on the route. Similarly, GC-PGP [201] removes off-route nodes from the lane graph after information aggregation and TPP [134] and DTPP [136] discard all EV trajectory proposals that do not comply with the route.

Fig. 3 shows that early models did not condition their plan on a goal at all. Afterward, submodules and input features dominated. While using input features is easy to implement, it is less effective, whereas submodules represent a major architectural constraint. The most interpretable method is using a routing cost function. Recent models introduced route attention, which is more flexible. Both route attention and routing cost allow to condition on goals that are not reflected in the training dataset, thus improving generalization.

4.4 Planning Paradigms

Recall that in Eqn. 5 we defined planning to be a function $f$ mapping from observational inputs $X_{\text{EV}}$, ${\overline{\text{X}}}_{\text{SV}}$, $I$ and predictions ${\overline{\text{Y}}}_{\text{SV}}$ to a trajectory $Y_{\text{EV}}$. In the following, we focus on the planning function $f$. To this end, we decompose $f$ into two parts: A proposal generator $g$, yielding a set of multiple potentially suitable trajectories ${\widehat{Y}}_{\text{EV}}^{(i)},i=1,..,N_{\text{proposals}}$ and a proposal selector $h$ that selects the final plan $Y_{\text{EV}}$ among the proposed set. Hence, in this work, we propose to represent the planning function $f$ as

$$f=h(g(X_{\text{EV}},{\overline{\text{X}}}_{\text{SV}},I,{\overline{\text{Y}}}_{\text{SV}}).$$ (7)

Based on this, we distinguish three different paradigms adopted for the planning task: Cost function optimization, regression, and hybrid planning.

Cost function optimization. As planning aims to find a trajectory that optimizes objectives such as safety, comfort, and progress, it is a natural approach to design and optimize a cost function that balances these potentially conflicting objectives. Cost function-based planning methods [121, 61, 211] rely entirely on the selection function $h$ to find a suitable trajectory, i.e.,

$$h=\underset{i}{argmin}c\left({\widehat{Y}}_{\text{EV}}^{(i)}\right),$$ (8)

where $c$ is a cost function of the planned trajectory. Consequently, the proposal generator $g$ may randomly sample feasible motion profiles [61] or obtain them by clustering real-world expert demonstrations [70]. Traditional non-learning-based methods rely on hand-crafted cost functions [212, 213, 214, 215, 216, 217, 218, 123]. However, designing a cost function that effectively balances these goals and generalizes to the long-tailed distribution of possible driving scenarios is challenging. Learning-based methods aim to address this by learning a cost function directly from expert demonstrations [219, 220, 103]. Similarly, [119, 115] learn to regress driving-relevant affordances such as the distance to the leading vehicle or the heading angle relative to the lane, based on which a downstream controller infers driving commands.
The learned cost function can be non-parametric [61, 221, 222] or comprise hand-designed sub-costs of which only the weights are learned [69, 70, 66, 121]. The non-parametric cost function of the NMP [61] assigns a cost to each potential future EV position within the planning horizon and is represented as a 3D-BEV-Tensor. In contrast, hand-designed sub-costs commonly evaluate safety, comfort, traffic rule compliance, and progress [121, 70, 66].

Regression approaches rely entirely on the proposal generator $g$. The selection function $h$ is the identity, and all burden is placed on the proposal generator $g$, which only generates a single proposal $Y_{\text{EV}}={\widehat{Y}}_{\text{EV}}^{(1)}$. E2E methods fall into this category. Regression approaches can learn to directly predict future waypoints [201, 20, 24] or actions [9] by supervised learning [223]. This form of imitation learning, i.e., supervised learning from expert data, is called behavior cloning (BC) [224, 190, 225]. This approach makes the manual design of the cost function obsolete. However, BC models are vulnerable to distributional shifts, which occur when the model reaches states not covered by the training data. Besides, regressed trajectories are generally not guaranteed to be kinematically feasible. UrbanDriver [24] counters this by training the model directly in a closed-loop simulation. However, this requires the entire simulator to be differentiable. CW-ERM [226] rolls out the policy in a non-differentiable simulation to identify important training scenarios and assign sample weights.

Hybrid planning describes methods that combine both ideas. First, a set of candidate trajectories is regressed by the proposal generator $g$, then the best one is selected with a cost function in $h$ [227]. For instance, [130, 131, 228, 229, 132] perform collision checks on the proposed trajectories to rule out unsafe proposals. In this setting, it often remains unclear which parts of the overall objective are enforced by the proposal generator and which by the cost function.

Fig. 3 shows that regression is the dominating planning paradigm for E2E systems. Cost function-based planning is mainly used in the work of researchers from Uber ATG and the University of Toronto [68, 66, 69, 70].

4.5 Benchmarks

Different methods exist to evaluate planning approaches, namely open-loop evaluation and closed-loop simulation.

Open-loop evaluation is similar to an evaluation in the prediction task. It compares the planner output $Y_{\text{EV}}$ to that of an expert planner $Y_{\text{GT}}$ [230]. However, this does not give the planner control over the EV and ignores the problems arising from compounding errors and distributional shifts [231]. More severely, recent work demonstrated that open-loop evaluation results do not correlate with driving performance [126, 232].

Closed-loop simulation in contrast, lets the planner control the EV. Instead of merely comparing the planner output $Y_{\text{EV}}$ to that of an expert planner $Y_{\text{GT}}$, the evaluation is based on metrics for comfort, safety, and progress. Thus, closed-loop evaluation relates better to real-world driving but requires a simulator to let the model interact with its environment. Prominent simulators are Carla [205], nuPlan [206], Waymax [233], Highway-Env [234], and CommonRoad [235]. Carla is a high-fidelity simulator that involves sensor simulation, whereas nuPlan, Waymax, CommonRoad, and Highway-Env operate on abstract object representations (e.g., bounding boxes) without rendering camera or lidar sensors. Many benchmark datasets were proposed for the Carla simulator [232, 236, 237, 20]. However, the scenarios are purely synthetic, which questions the generalizability to the real world. In contrast, nuPlan and Waymax follow a data-driven approach, which uses recordings from real-world scenarios to initialize the simulation. Thus, users have access to corresponding large-scale real-world datasets to train their method before deploying it in simulation. CommonRoad is compatible with several smaller datasets [195, 196, 197, 198, 199]. Highway-env is based on simplistic synthetic environments.

When simulating regular driving scenes, rare and critical scenarios are naturally under-represented. This problem can be tackled with data curation [238, 239] or additional test case generation. These can be generated manually [240, 241, 242, 243, 244, 245] or automatically from recorded driving logs [246, 247, 248]. Other methods generate adversarial critical scenarios by augmenting actors within the scene [249, 250, 251, 252, 253, 254, 255, 256] or the scene itself [257, 258, 259, 260, 261]. Similarly, [5, 64] investigate human-robot interaction more closely with hand-crafted scenarios – an aspect that is uncared for in average driving situations.

Another major challenge in closed-loop evaluation is to realistically simulate the SVs. Simple simulators such as Highway-Env [262] rely on non-reactive agents that follow predefined trajectories [263]. In the non-reactive version of nuPlan, and Waymax, these trajectories are obtained from real-world recordings. However, if the EV behaves differently than the expert during recording, this can result in collisions caused by the SVs, as they do not react to the EV. Hence, this simulation is usually limited to short sequences, where the EV stays close to the expert’s trajectory [264]. Carla, nuPlan, and Waymax also provide a reactive driver model for the SV, such as the Intelligent Driver Model (IDM) [16, 17, 265, 266, 267, 268]. Another line of work focuses on learning-based methods to simulate realistic driving behavior [269, 270, 271, 272, 273, 274]. In Waymax [233], users can plug in their own model to control objects in the scene.

5 Integrating prediction and planning

Figure 5: Connection between the integration principles, ego-to-agent relationship, and cost function. Red arrows for the robot leader planning indicate that applying these cost function categories would be possible but lacks a thorough empirical evaluation. We highlight this as an interesting direction for future research.

After reviewing key aspects such as interfaces and architecture of planning and prediction components found in ADSs, we change perspective and discuss the properties of the IPPS. In particular, we will highlight the influence of design decisions in the IPPS on behavior in interactive scenarios. We will start by categorizing existing works based on which components the IPPS comprises (Sec. 5.1). Then, we will describe the implications for the interactive behavior which follow from design choices in modular integrated systems (Sec. 5.2). Subsequently, different concepts of safety and contingency in modular integrated systems are discussed in Sec. 5.3. Finally, Sec. 5.4 discusses possible combinations of these categories. An overview is provided in Fig. 5.

5.1 Integration Principles

In this work, we distinguish among three design principles for IPPS. An overview can be seen in Fig. 6.

Monolithic E2E planning describes methods where the IPPS consists of a single E2E planner, which maps the state inputs ${\overline{\text{X}}}_{\text{SV}}$, $X_{\text{EV}}$, and $I$ to the EV’s trajectory $Y_{\text{EV}}$. Thus, the SVs’ future behavior and the interactions among them, as well as between the SVs and the EV, are not modeled explicitly. However, the expert’s actions, which the E2E planner learns to imitate in training, are based on reasoning of how the future will unfold. By learning to imitate actions, the model could exhibit behavior that is implicitly based on such reasoning as well.
For instance, consider a scenario where the EV’s leading vehicle brakes. An expert or a human driver will base their driving decision on the likely future scenario that the safety margin will shrink and thus brake as well. Hence, an E2E planner that imitates the expert’s behavior makes a decision that is implicitly based on a predicted future.
Pioneered by Pomerlau [9], there is a vast body of literature on IPPS based on monolithic E2E planners [19, 116, 117, 118, 22, 62, 10, 63, 64, 23, 67, 122, 236, 24]. The main disadvantage is their black-box nature, which makes model introspection and safety verification difficult.

Interpretable E2E planning frames prediction as an auxiliary learning task [58, 61, 120, 125]. Explicit predictions are an additional model output trained jointly with the planning task. Usually, a shared backbone encodes features that are used for both tasks. Then, separate heads decode the respective output representations.
The additional objective provides a learning signal to the backbone. It acts as a regularization which can make the learning process more sample-efficient and can increase generalization capabilities [58, 124, 119] as it was previously demonstrated for other auxiliary tasks in prediction and planning like speed prediction [64], semantic mapping [275], and occupancy forecasting [66]. However, the additional learning objective results in a trade-off between the different tasks, usually resulting in a hyperparameter that balances the respective losses and has to be tuned empirically.
Compared to monolithic E2E planning, the explicit prediction adds interpretability and facilitates introspection. Nonetheless, both IPPS designs cannot provide safety guarantees. Hence, we attribute the increased driving performance to the regularizing effect of the additional objective.

Manual integration of a planner with a prediction module means that a separate subsystem is employed for each of the tasks. The interplay of both tasks is manually engineered based on domain knowledge. The most widely adopted integration is a sequential pipeline, where first the prediction module is inferred, and its output is fed to a subsequent planning module [70, 66, 69, 123]. Since this design is unable to reflect the impact of the EV’s plan on the SVs, an alternative is to swap the order of prediction and planning: First, candidate plans are generated, followed by a conditional prediction module. Then, the best plan for the EV is selected among the candidates [129, 134]. Alternative methods iterate between prediction and planning [276] or do both jointly [132]. In the following, we give a chronological overview of different models and highlight the integration design of prediction and planning.

(a) Monolithic E2E IPSS

(b) Interpretable E2E IPSS

(c) Manually Integrated IPSS

Figure 6: IPPS design principles that show how prediction and planning can be integrated. Monolithic E2E methods (a) use no explicit prediction, while interpretable E2E methods (b) use it as a regularization in training. Manually integrated IPPSs (c) combine a modular prediction with a modular planning method using hand-designed interfaces.

PRECOG [65] uses a probabilistic model for conditional multi-agent forecasting. It implements autoregressive trajectory decoding for EV and SVs jointly. At each time step, the state of all agents is updated and used as input for all other agents in the next update step.

PiP [129, 277] reverses the view of IPPS. Its core idea is that SVs behave differently depending on the EV’s actions. A trajectory generator provides a set of candidate plans independent of the SVs. Each candidate plan is then used to condition prediction. Finally, the optimal plan is selected using a cost function.

DSDNet [68] combines E2E and modular model design. Each of the sequential NN modules additionally has access to the high-dimensional features of the perception backbone, complying with the hypothesis [20] that scene context is important for robust planning. For each detected vehicle, a set of potential future trajectories is predicted. To select the plan for the EV from a sampled candidate set, a hand-crafted cost function is applied, which quantifies the collision probability of the EV candidate trajectories and the SVs’ predicted trajectories.

P3 [66], LookOut [69], and MP3 [70] use cost functions to link prediction and planning. P3 predicts an occupancy map, LookOut [69] employs predicted trajectories to evaluate different pre-defined EV candidate trajectories. MP3 [70] extends P3 into a mapless approach by predicting an online map. A ’dynamic state map’ comprising detections and prediction is used together with the intended route to evaluate potential trajectories.

SafetyNet [130] parallelly applies a monolithic E2E planner with implicit prediction and an explicit prediction module. Multiple collision checks between EV plan and SVs predictions are performed. If the initial plan is classified as unsafe, a hand-crafted fallback layer generates a lane-aligned trajectory. SafePathNet[131] improves upon SafetyNet [130] with a transformer module for joint prediction and planning. In a mixture-of-experts approach, multiple modes are predicted and ranked. The ranking is updated by checking for collisions with the most probable SVs’ predictions. Finally, the highest scoring EV mode is selected.

DIPP [7] jointly predicts the trajectories for all agents, including the EV, which is treated as an SV. The mode with the highest probability is selected for each vehicle. A differentiable nonlinear motion planner then performs iterative, local optimization under consideration of a kinematic model and a hand-crafted cost function.

Inspired by hierarchical game-theoretic frameworks [278, 279, 280, 281], GameFormer [132] models interaction between agents as a level-$k$ game [282, 283]. Thus, a transformer decoder models the interaction between all agents, including the EV, by iteratively updating the individual predictions based on the predicted behavior of all agents from the previous level. Hence, the model yields a joint prediction for all agents, including the EV.

UniAD [133] implements a modular system that is trained E2E based on the planning objective. Queries are used as the interface between the submodules so that the planning module can attend to agent-level features from previous network layers. Various works adapted or extended this approach: VAD [135] and Hydra-MDP [284] use the output of previous layers explicitly for collision and map compliance checks. PPAD [276] repeatedly iterates between prediction and planning based on the perception modules’ outputs and GenAD [285] adapts a generative decoder to regress SV and EV trajectories jointly.

TPP [134], DTPP [136], and MBAPPE [286] represent multiple future outcomes in scenario trees, where branches reflect combinations of SV predictions and EV plans. The best plan is obtained through a tree search.

Compared to other integration principles, manual integration requires more engineering effort but restricts the solution space in a meaningful way by incorporating prior knowledge. Often, manually integrated IPPS provide a high explainability and safer plans than E2E systems [287, 131].

The timelines in Fig. 3 show how early works relied mostly on implicit predictions in monolithic E2E designs. Starting with ChauffeurNet [58] and PRECOG [65], explicit predictions became more popular. More recently, manual integration of E2E differentiable modular components gained popularity [203, 133].

5.2 Ego-to-Agents Relationship

After reviewing the integration principles from a system architecture perspective, in the following, we will analyze the manual integration category from a different perspective, namely the intended ego-to-agents relationship. This is of special importance in highly interactive scenarios since the EV needs to base its driving decisions on the observed and expected behavior of surrounding agents. However, as [5] pointed out, the EV also needs to be aware that it can influence the behavior of others.
For instance, in the example in Fig. 7, the plan of the EV can also influence the behavior of the SV: If it approaches the intersection faster, the SV might decelerate and yield to the EV. To describe if and in which way the mutual interaction of the EV and surrounding traffic is taken into account in the planning step, [5] introduced the following four categories: robot leader planning, human leader planning, co-leader planning, and joint planning. In this context, human refers to the surrounding traffic, and robot refers to the self-driving vehicle. In the following, we briefly describe these categories and discuss existing work.

Robot leader planning. The EV plan is inferred based on the current state, and the prediction step for the whole environment is conditioned on it. Hence, the EV can effectively seek to make the SV react to its trajectory [88, 128, 127]. This can result in aggressive driving behavior. For instance, in the example in Fig. 7, the EV would reason that when it follows the fast-progressing plan, the observed SV will yield to it to prevent a collision.

Human leader planning is the opposite of robot leader planning. The plan is based on the forecasted behavior of SVs [61, 62, 70, 66]. Hence, the EV shows a reactive behavior. It does not model the influence of the EV’s plan on the SVs, which can lead to underconfident behavior [5]. This is the case for IPPS that predict and plan sequentially. In the unprotected right turn example (cf. Fig. 7), the EV tries to find a plan that is suitable given the two predicted SV behaviors without being aware that it can influence them. Consequently, it will favor the slower plan.

Joint planning describes systems that are aware that vehicles (including ego) interact with each other. The EV’s plan is obtained by global optimization over all agents [288]. Hence, the IPPS system deterministically approximates a joint objective [289] under the assumption that an optimal outcome exists. Hence, it erroneously assumes that the behavior model used for SVs is exactly known and that every traffic participant optimizes this same global objective. For example, if the EV in Fig. 7 squeezes in before the approaching SV, this may be optimal with respect to a reasonable global objective. Nonetheless, there is no guarantee that the SV will behave accordingly. Consequently, [5] demonstrates how this can lead to fatal errors.

Co-leader planning is the most complex paradigm. It models the influence of SVs’ potential future behaviors on the EV plan as well as their reaction to potential ego trajectories [290, 291, 5]. In contrast to the joint planning category, the surrounding agents’ behavior is not assumed to be deterministic. The EV has to consider this uncertainty in the planning step, e.g., by being prepared for multiple future outcomes and reacting accordingly. Additionally, co-leader planning expects the SV to make plans for different potential behaviors of the EV. The EV can exploit this by behaving in a way that makes the SVs reveal their intention, which in turn lets the EV make more confident plans. For instance, in the example in Fig. 7, the EV can try to slowly approach the intersection to provoke a reaction of the SV. This reaction can either indicate that the SV is going to make a turn, e.g., with a turn signal or by slowing down, or the opposite, i.e., accelerating, to indicate that the SV will continue straight. The former will enable the EV to confidently decide in favor of the faster-progressing plan. CfO [5] originally introduced this paradigm, leveraging normalizing flows to represent multi-agent behavior as a learned coupling of independent variables. However, the intricate nature of this architecture may pose obstacles to wider adoption. Furthermore, their benchmark is simplistic, encompassing only a few scenarios with hard-coded agent behaviors. TPP [134] presents a readily comprehensible tree-structured framework that implements co-leader planning. It involves sampling kinematically feasible trajectories for the EV and then conditioning prediction on each of them. An autoregressive formulation for prediction ensures causality: It can be seen as a way to roll out the scene with SV predictions and the EV plan such that both can only access the observation history up until the respective rollout step. Finally, the best EV trajectory proposal is selected with a dynamic programming algorithm. DTPP [136] extends this with a transformer-based conditional prediction module, which allows the forecast of all agents conditioned on multiple EV plans at once. It also replaced the dynamic program with a learnable cost function and applied node pruning to reduce computational burden. Still, the EV reaction to the prediction is taken over entirely by the cost function because the sampled ego-proposals are non-reactive with respect to the SV predictions.

Fig. 3 shows that no clear trend toward one of the paradigms can be observed. While Co-leader planning is theoretically the most capable paradigm, it has not yet been shown empirically if one of these four theoretical concepts is superior to the others and how it can be realized in the design of system architectures. This may also be related to a lack of comprehensive empirical benchmarking focused on interactive behavior.

5.3 Safety and Contingency

Figure 7: Demonstration of contingency on an unprotected right turn. Let’s assume two expected future behaviors of the observed SV, namely $Y_{\text{SV}}^{(1)}$ and $Y_{\text{SV}}^{(2)}$ with probabilities of $P(Y_{\text{SV}}^{(1)})=0.2$ and $P(Y_{\text{SV}}^{(2)})=0.8$. Two candidate plans for the EV are considered $Y_{\text{EV}}^{(1)}$ (purple) and $Y_{\text{EV}}^{(2)}$ (orange). While the purple plan is only safe if the unlikely prediction $Y_{\text{SV}}^{(1)}$ occurs, the orange plan is safe under all predictions. Thus, it is preferred under worst-case assumptions. When planning with marginalized predictions, the cost function has to balance between the high progress of the purple plan and the potential safety risk if the unlikely prediction $Y_{\text{SV}}^{(1)}$ occurs. A contingency planner could exploit that both candidate plans have a common short-term action. Hence, it could opt for the purple plan as it is aware that it could still re-plan and fall back to the orange one.

The ego-to-agents relationship categories discussed above highlighted that considering multiple potential future scenarios is essential to safety and contingency planning. In the following, we will discuss how this can be incorporated into the cost function component $h$ that we defined as part of the planning function $f=h(g(X_{\text{EV}},{\overline{\text{X}}}_{\text{SV}},I)$ (cf. Sec. 4.4). We will analyze the implications of different design choices on safety and contingency. To this end, we introduce the following notation: We denote the number of relevant future scenarios with $N_s$. These scenarios can be reflected by a multinomial distribution $P\left({\overline{\text{Y}}}_{\text{SV}}^{(i)}\right),i=1,..,N_s$ describing the behavior of SVs in each scenario. Based on how this distribution is reflected in the design of the IPPS, we form the following three groups of existing methods regarding safety and contingency: Planning with marginalized predictions, worst-case planning, and contingency planning. They are briefly outlined in Fig. 7.

Planning with marginalized predictions describes IPPS that do not explicitly distinguish multiple future scenarios $N_s$. This means that the prediction ${\overline{\text{Y}}}_{\text{SV}}$ is marginalized over the future outcomes ${\overline{\text{Y}}}_{\text{SV}}^{(j)}$. This can be done explicitly, i.e., by integrating over the scenario probabilities $P\left({\overline{\text{Y}}}_{\text{SV}}^{(j)}\right)$

$${\overline{\text{Y}}}_{\text{SV}}=\sum _j^{N_s}{\overline{\text{Y}}}_{\text{SV}}^{(j)}\cdot P\left({\overline{\text{Y}}}_{\text{SV}}^{(j)}\right)$$ (9)

or implicitly, i.e., by representing all possible scenarios in a joint representation instead of differentiating scenarios. For instance, IPPS that incorporate the prediction in their cost function $h$ and do not consider separate future outcomes when evaluating a proposal fall into this category [61, 70, 66]. Hence, their selection function takes the form

$$h=\underset{i}{argmin}c(Y_{\text{EV}}^{(i)},{\overline{\text{Y}}}_{\text{SV}}).$$ (10)

Similarly, this is the case for all systems in the monolithic E2E category (cf. Sec. 5.1) [201, 20, 24, 9, 19], as they do not explicitly consider multiple future scenarios and hence – if at all – learn marginalized predictions. Broadly speaking, this means that the IPPS assumes that all $N_s$ scenarios will occur to a certain degree at the same time. As can be seen in the example of Fig. 7, the cost function needs to trade off unlikely but dangerous scenarios (e.g. collision) against very likely low-cost scenarios. This is safety-critical since planners need to be cautious, especially w.r.t. unlikely but dangerous events.

Worst-case planning refers to IPPS that are aware that multiple future outcomes ${\overline{\text{Y}}}_{\text{SV}}^{(j)}$ exist. In this category, no probabilities $P({\overline{\text{Y}}}_{\text{SV}}^{(j)})$ are considered. Instead, each proposal is evaluated based on the worst-case scenario. The motivation behind this assumption is that the final output trajectory must be safe in each possible scenario. Consequently, the selection function $h$ can be formulated as

$$h=\underset{i}{argmin}\underset{j}{\max }c(Y_{\text{EV}}^{(i)},{\overline{\text{Y}}}_{\text{SV}}^{(j)}).$$ (11)

This paradigm strongly focuses on collision avoidance and can result in overly conservative behavior, as shown in Fig. 7. While this assumption is not a popular choice in DL-based prediction and planning, it is broadly applied for rule-based safety layers such as RSS [292] and can be seen as an intermediate step between planning with marginalized predictions and contingency planning.

Contingency planning prepares for the unknown future development of the scene by taking into account different future scenarios ${\overline{\text{Y}}}_{\text{SV}}^{(j)}$ and their probabilities $P({\overline{\text{Y}}}_{\text{SV}}^{(j)})$. The resulting plan hedges against worst-case risks while enabling progress in expectation. Both the cost function-based and the hybrid planning paradigm can have these properties, and the selection function $h$ takes the general form

$$h=f_c(Y_{\text{EV}}^{(i)},{\overline{\text{Y}}}_{\text{SV}}^{(j)},P({\overline{\text{Y}}}_{\text{SV}}^{(j)})).$$ (12)

where $f_c$ is a contingent selection function.

For instance, [69, 287, 293] find a plan that is safe for scenarios on the short horizon (independently of their probability, i.e., worst-case) and also optimize the expected cost on the long horizon (marginalization). Thus,

$$h=\underset{i}{argmin}\left(\underset{j}{\max }{c}_s(Y_{\text{EV}}^{(i)},{\overline{\text{Y}}}_{\text{SV}}^{(j)})+{𝐄}_j[c_l(Y_{\text{EV}}^{(i)},{\overline{\text{Y}}}_{\text{SV}}^{(j)})]\right),$$ (13)

where $c_s$ and $c_l$ consider the short-term and the long-term part of the proposal $Y_{\text{EV}}^{(i)}$, respectively. In the example in Fig. 7, this planner allows the most progress while still ensuring safety. CfO [5] further differentiates between passive contingency where the EV is contingent upon the SVs [69, 287, 293, 294] and active contingency where the EV also expects the SVs to be contingent upon it [290, 291, 295].

5.4 Possible Combinations

In the previous sections, we have described three ways to categorize the integration of prediction and planning methods, namely (1) integration principle, (2) ego-to-agents relationships, and (3) safety and contingency. In the following, we discuss possible combinations of these three dimensions. Fig. 5 shows an overview. Our key insight is that the categories we characterized describe the differences between IPPS methods on different levels. While the integration principle focuses on a high-level system architecture, the ego-to-agent relationship is mostly based on interactive behavior. The safety and contingency consideration is founded on the cost function, i.e., a specific design choice for the proposal selection. In the following, we highlight associations between these categories.

In case of monolithic E2E [9, 20, 24, 19, 117, 64, 10, 121, 23, 67, 122, 62, 22, 118, 116, 201] and interpretable E2E [58, 61, 296], no part of the system architecture explicitly reflects an intended interactive behavior. Hence, they cannot be assigned to the ego-to-agents relationship categories, and we determine their ego-to-agents relationship as unknown.
For instance, imagine that a model is trained on demonstrations from an expert who follows the co-leader planning paradigm and employs contingency planning. Given that the model excels at imitating the expert, it could be assumed that the model behaves in the same way. However, there is no guarantee for this, and it could be equally possible that the model is unable to reason effectively about SVs, resulting in a behavior similar to the robot leader paradigm.
Similarly, as E2E models do not use hand-crafted cost functions with a pre-defined structure, the safety and contingency categories described in Sec. 5.3 do not apply to them.

In contrast, inferring the ego-to-agents relationship of modular systems is often possible. Thus, we can further divide manually integrated architectures into robot-leader, human-leader, joint planning, and co-leader planning.
There is a plethora of manual integration methods following the human leader paradigm (cf. Fig. 3). By following a traditional sequential scheme of prediction and planning, [70, 66, 69, 120, 68, 130] do not model the influence of the ego plan on SVs. This can be combined with any of the three safety and contingency categories. Works like DSDNet [68], P3 [66], MP3 [70], UniAD [133], and [297] leverage marginalized predictions.
While applying a worst-case cost function is theoretically possible, most works that can distinguish multiple future scenarios aim to make contingency plans [69, 287, 293].
Similarly, we argue that the robot leader paradigm can be combined with all three cost functions. Consider a simple robot leader model that first identifies potential ego plans and then predicts the future behavior of SVs conditioned on each plan. The final selection of the EV’s plan can be based on a cost function belonging to each of the three categories. However, existing works use specialized cost functions that do not follow the structure we outlined in Sec. 5.3, e.g.,[129, 134, 128, 88]. We want to highlight that systematically combining robot-leader architectures with corresponding cost functions could be a promising direction for future research. Especially contingent or worst-case cost functions could mitigate the inherent problem associated with robot leader planning, namely relying on SVs to react in potentially unreasonable ways for the interests of the EV.

As described in Sec. 5.2, the active contingency resulting from the co-leader planning paradigm implies that the EV makes contingency plans while being aware that SVs are contingent upon it. Hence, it naturally employs a cost function that supports contingency plans.

Finally, the joint planning paradigm relies on optimizing a joint cost function for all traffic participants in the scene. Thus, none of the safety and contingency concepts are applicable, and we denote such a cost function as joint cost.

6 Outlook and Conclusion

Based on the overview we gave on deep learning-based prediction, planning, and their integration in automated driving systems, we identify two core challenges for future research: System design and comprehensive benchmarking. After discussing both, we draw a final conclusion.

System Design. Employing a traditional, strictly sequential system consisting of perception, prediction, planning, and control is still a popular choice. Our survey suggests that this approach is not able to meet the requirements set for driving systems. Alternative methods integrate prediction and planning in a way that allows conditioning prediction on potential ego plans and vice versa. Nonetheless, it remains unclear which exact integration architecture is most effective. Especially in the increasingly popular field of interpretable end-to-end systems, it is unclear how prediction and planning should be integrated.

Comprehensive Benchmarking. We discussed different aspects of the integration of prediction and planning. However, no comprehensive empirical benchmark reproduces and analyzes their strengths and weaknesses. Such an overview would help to better understand the effects of different ego-to-agents relationships and safety/contingency paradigms. This requires simulation in realistic and highly interactive scenarios with realistic driver models for surrounding vehicles and expressive interaction metrics. Moreover, benchmarks must cover corner case scenarios and test the generalization to distributional shifts.

In this work, we surveyed and analyzed the integration of prediction and planning methods in automated driving systems based on a comprehensive overview of the individual tasks and respective methods. We described, proposed, and analyzed categories to compare integrated prediction and planning works and highlighted implications on safety and behavior. Finally, we pointed out promising directions for future research based on the identified gaps.