Hybrid-Prediction Integrated Planning for
Autonomous Driving

Abstract

Autonomous driving systems require the ability to fully understand and predict the surrounding environment to make informed decisions in complex scenarios. Recent advancements in learning-based systems have highlighted the importance of integrating prediction and planning modules. However, this integration has brought forth three major challenges: inherent trade-offs by sole prediction, consistency between prediction patterns, and social coherence in prediction and planning. To address these challenges, we introduce a hybrid-prediction integrated planning (HPP) system, which possesses three novelly designed modules. First, we introduce marginal-conditioned occupancy prediction to align joint occupancy with agent-wise perceptions. Our proposed MS-OccFormer module achieves multi-stage alignment per occupancy forecasting with consistent awareness from agent-wise motion predictions. Second, we propose a game-theoretic motion predictor, GTFormer, to model the interactive future among individual agents with their joint predictive awareness. Third, hybrid prediction patterns are concurrently integrated with Ego Planner and optimized by prediction guidance. HPP achieves state-of-the-art performance on the nuScenes dataset, demonstrating superior accuracy and consistency for end-to-end paradigms in prediction and planning. Moreover, we test the long-term open-loop and closed-loop performance of HPP on the Waymo Open Motion Dataset and CARLA benchmark, surpassing other integrated prediction and planning pipelines with enhanced accuracy and compatibility. Project website: https://georgeliu233.github.io/HPP

I Introduction

Figure 1: Generic learning-based pipelines in autonomous driving. a) Integrated pipelines learn planning jointly with motion prediction (IPP) or occupancy prediction (IOP) with coupled networks. b) End-to-end pipelines directly map from raw sensor inputs to planning. c) Our proposed HPP establishes a coupled planning system with hybrid-prediction integration and optimization.

Autonomous driving systems (ADS) have made significant progress in perception, prediction, and planning, thanks to the advancement of learning paradigms [1]. However, the performance growth of these independent tasks has come to a halt, prompting a reconsideration of modular design optimization [2, 3, 4, 5]. Fueled by inherent interactions among autonomous vehicles and traffic participants, recent research has placed significant emphasis on the integration of prediction and planning tasks [6]. They seek to achieve concurrent advancements in both prediction and planning .

Integrated pipelines (Fig. 1a) generally connect planning and singular prediction modules involving agent-wise motion trajectories (IPP) [3, 7, 8] or whole-scene occupancy probabilities (IOP) [9, 10, 11]. However, relying on single prediction format inevitably confronts respective shortfalls. Specifically, motion prediction models continuous temporal trajectories tracked for each agent, but encounters inconsistency and exponential cost in joint social patterns from marginal agents [12]. Conversely, exclusive occupancy accurately predicts aligned joint patterns of whole scene agents under bird’s eye view (BEV) perception, but the loss of agent-wise tractability leads to temporal conflicts and omission risks of critical agents [13]. This characteristic highlights the complementarity of motion and occupancy prediction. The inconsistency arising in the singular prediction pattern presents challenges for both IPP and IOP, resulting in incompatible planning learned collectively with misaligned predictions.

The limitations of integrated systems have spurred a renewed focus on end-to-end pipelines (Fig. 1b) [14]. Planning-oriented systems employ a well-organized modular design to address plan guidance under a unified BEV geometry and query-based intermediate interactions [2, 5, 15]. While this approach proves planning superiority and consider a hybrid format of predictions and planning, it falls short in addressing potential conflicts among different predictions. Furthermore, the absence of interactive co-design across predictions and planning results in passive maneuvers and hinders the system learning for social compatibility. The integration of different modules gives rise to three major challenges: inherent trade-offs, including inconsistency and omission risks with sole prediction; discrepancies between joint and marginal patterns in predictions, limiting accuracy and impeding the learning of safe and naturalistic driving maneuvers; and the absence of interactive co-design for compatible planning, considering dependencies of hybrid prediction.

To tackle the aforementioned challenges, we propose a novel integrated pipeline (Fig. 1c) named Hybrid-Prediction integrated Planning (HPP) that optimizes prediction and planning in a co-design process. By combining IPP and IOP, HPP delivers consistent planning while ensuring that hybrid prediction inform each other consistently. To achieve this, HPP leverages Transformer-based queries to channel and aggregate interactions between modules. Additionally, we have developed a refinement process that guides safe planning interacting with hybrid prediction. Specifically, we propose MS-OccFormer to perform marginal-conditioned joint occupancy prediction that aligns and refines consistently with marginal motion prediction. We leverage GTFormer, which is inspired by simulating game-theoretic iterative reasoning of marginal motion prediction with ego vehicle and joint occupancy. HPP concurrently models fine-grained interactions of integrated hybrid prediction patterns in Ego Planner, where we eventually devise a hybrid-prediction-guided refinement mechanism to facilitate safe and compatible planning. The main contributions are listed below:

1. We propose HPP, a modular co-design optimization ADS paradigm, that consistently interacts among marginal and joint prediction patterns with planning.

2. We introduce MS-OccFormer, a model that predicts joint occupancy patterns in BEV geometry while being aware of the marginal predictions. We also present GTFormer, a model that performs game-theoretic reasoning among marginal motion predictions in coordination with both planning and occupancy prediction. The ego planner is devised under interactive guidance by hybrid prediction.

3. HPP is tested on multiple large-scale real-world benchmarks, and extensive testing results demonstrate its state-of-the-art performance in terms of accuracy, safety, and consistency in prediction and planning.

The remainder of this paper is organized as follows: Sec. II reviews integrated prediction and planning pipelines in ADS. Sec. III formulate the methodology of HPP. Comprehensive benchmarks and discussions are presented in Sec. IV. Finally, Sec. V concludes the whole work.

II Related Work

II-A Predictions and Planning in ADS

Prediction and planning modules in the conventional ADS model are separate. Predictions typically define evolving transitions as conditions for safe planning. Learning-based prediction models excel in modeling interactions among diverse agents and scene contexts [12]. Categorized by representations, sparse predictions forecast multi-agent trajectories (MATP) along with detected participants. Leveraging Transformer [16, 17] or GNNs [7, 18] in constructing the social interaction graph [7] or recurrent refining [19], MATP filters multi-agent predictions in scoring combinations of marginal ones for each agent. While achieving agent-wise accuracy, MATP introduces exponential computations and trajectory-wise inconsistency. Dense predictions directly estimate the future distribution of agents jointly from ego-centered occupancy [9, 20, 21]. A notable issue is the loss of agent-wise tractability. Enhancements via trajectories of heatmap sampling [22] or joint trajectory learning [23] exhibit similar consistency issues. In ADS planning, various approaches are well-founded including sampling [24], optimization [25], and learning-based techniques via imitation learning [26] or reinforcement learning [27]. Still, achieving safe and socially compatible driving maneuvers in planning requires interactive awareness and safety guarantees derived from planning-compliant predictions. Furthermore, accumulated errors from detached predictions and planning underscore the need for developing integrated ADS.

II-B Integrated Predictions and Planning in ADS

Navigating dense and interactive traffic requires integrated ADS, which models the simultaneous driving behaviors among social agents and the autonomous vehicle. Intuitive thought is to stack ego vehicles collectively with social agents’ predictions and learn unified trajectories. All interactions are implicitly modeled by Transformer-based IPP [8, 28] or post-processed occupancy prediction[23] under the IOP pipeline. To consider the evolving interactive behaviors between predictions and planning, conditional methods model the behavioral response individually from ego vehicle to social agents predictions [29]. Conditional motion predictions are then integrated into planning reschedules [24, 30] or modeled as non-cooperative games [31]. Still, these one-way interactions bypass the mutual consistency with all agents. Moreover, iterative bi-level optimization [32] significantly slows down learning. Obviated from agent-wise conflicts, hierarchical game-theoretic approaches model the iterative reasoning process [33], updating mutual behaviors for all agents simultaneously [34]. Yet, uniformed agent-wise reasoning lacks specifications between predictions and planning, which should be target-driven. In HPP, we integrate reasoning by introducing an agent-conditioned occupancy to modulate joint behavior in social agents. and an Ego Planner for interactive planning. This integrated co-design enables learning both predictions and planning with flexibility while maintaining mutual awareness by iterative reasoning.

Figure 2: Systematic overview of the proposed Hybrid-Prediction integrated Planning (HPP) framework. HPP is established upon query-based co-design optimization of interactive planning with hybrid prediction integration (IPP and IOP), informed by BEV perceptions. With encoded perception scene context $Q_{Map},Q_B,Q_A$, HPP provides prediction and planning co-design in three-fold. Joint occupancy prediction $\widehat{\text{O}}$ is iteratively refined in MS-OccFormer, sharing mutual consistency over marginal motion prediction $\widehat{\text{Y}}$ in GTFormer. GTFormer performs interactive reasoning between marginal prediction and planning. Reasoned outcomes and ego features are then served to query hybrid prediction-aware planning $\tau$ in Ego Planner. Eventually, optimizations are scheduled to refine planning ${\tau }^{\ast }$ with hybrid-prediction guidance.

Recent paradigms focus on leveraging output responses for predictions to enhance planning guarantees. Adversarial objectives are employed between joint motion predictions and planning, considering likely [3, 35] or safety-critical patterns [36] for mutual differentiable optimizations. However, similar consistency issues faced by MATP pose additional optimization challenges. Some methods predict dense occupancy jointly as a potential cost for planning guidance [11, 9, 10]. However, intractability issues introduce risks and require laborious filtering. An additional challenge is the lack of integrated modeling leads to inconsistency and limits the interactive process only in optimization. In HPP, integration is considered concurrently for ADS co-design and output optimizations. Through flexible and consistent hybrid prediction and planning via system co-design, hybrid prediction are jointly utilized to refine planning, enhancing safety consistently.

II-C End-to-end Systems and LLMs for Autonomous Driving

End-to-end methods consider a direct mapping from raw sensors to prediction and planning under perception understanding [14]. A typical system is serialized by modules and learns jointly with each objective [37, 4]. However, modular errors are accumulated and geometry is hindered by misaligned predictions and planning. Thus, planning is sampled in trajectory retrieval by predictions [38]. Prominence in BEV perception [39, 40, 41] enables modular integration and learning under unified BEV geometry [42]. This further prompts a planning-oriented system, which organizes and serves all intermediate modules targeting planning under visual [2, 15, 43] or vectorized [5] perceptions. Query-based design channel and propagate modular integration. This prompts recent works incorporating large language models (LLMs) as the motion planner [44] or routing agent [45]. Still, the current end-to-end system focuses more on integration with perception that aligns predictions and planning, leaving paradigm discussions incorporating LLMs [46]. In HPP, we put more emphasis on modular co-design optimization, learning interactive hybrid prediction, and planning informed by perception under query-based integration.

III Hybrid-prediction Integrated Planning

The overview of our proposed HPP is shown in Fig.2, which is defined in Sec. III-A upon query-based modular co-design and optimization of ADS. In the upcoming sections, informed by BEV perception pipelines exhibited in Sec. III-B, HPP manages the framework co-design from MS-OccFormer in Sec. III-C for joint occupancy prediction, sharing mutual consistency with motion prediction in GTFormer. Here we elaborate its hierarchical reasoning model for interactive prediction and planning in Sec. III-D. Followed by hybrid prediction-aware Ego Planner in Sec. III-E, we frame the learning and optimization process for HPP reckoning hybrid prediction guidance in Sec. III-F.

III-A Problem Formulation

As shown in Fig. 2, HPP focuses on addressing integrated predictions and planning challenges for ADS. Informed by perception under BEV geometry, this learning-based system is founded by modular co-design and optimization. Modularized by Transformer basis, HPP leverages categorical queries $𝐐$ to aggregate modular outputs and channel interactions as keys and values $𝐊,𝐕$ by multi-head attention mechanism.

Figure 3: Generic learning framework in MS-OccFormer: a) A single block of multi-scale marginal-conditioned occupancy predictor. Joint occupancy $\widehat{𝐎}$ is consistently integrated with marginal prediction features ${𝐇}_{traj}^A$ through global interactions and local refinements, and guided by iteratively updated learnable attention mask $𝐌$; b) A Swin-T decoder for local interactions through shifted-window cross attention; c) Agent-wise fusion for marginal prediction features.

Given multi-view image inputs, the co-design of HPP begins with the perception backbone for BEV features $ℬ$. BEV perception further defines an ego-centered area of $H\times W$ with map $𝒮$ and $N_A$ detected agents $𝒜\in A_{0:N_A-1}$ at current timestep $t=0$, where $A_0$ denotes the ego vehicle. Based on scene context $𝐗=\{ℬ,𝒮,𝒜\}$, HPP aims to learn and optimize concurrently for hybrid prediction and planning modules. Specifically, over future horizon $T$, and set of queries $𝐐$, joint predictions are defined as per-step occupancy probabilities ${\widehat{𝐎}}_{1:T}={\{o_t^{h,w}|o^{h,w}\in {[0,1]}^{H\times W}\}}_{t=1}^T$ for all neighbor agents. Simultaneously, the marginal future motions for all agents are defined by ${\widehat{𝐘}}_M^A={\{({\widehat{𝐲}}_m^{1:T,i},{\widehat{𝐩}}_m^i)|m\in [1,M]\}}_{i=0}^{N_A-1}$ denoting multi-modal future trajectories $\widehat{𝐲}\in {ℝ}^{N_A\times M\times T\times 2}$ and probability $\widehat{𝐩}\in {ℝ}^{N_A\times M}$ considering $M$ modes of uncertainty. hybrid prediction-aware planning ${\tau }_{1:T}\in {ℝ}^{T\times 2}$ is queried by plan context $ℰ$. Subsequently, HPP formulates a collective objective $ℒ$ for modular learning, and cost criteria $𝒞$ for optimized hybrid prediction guided planning ${\tau }^{\ast }\in {ℝ}^{T\times 2}$. In general, HPP is formulated as:

${\widehat{𝐘}}_M^A,{\widehat{𝐎}}_{1:T},{\tau }_{1:T}$ $=f(𝐗,𝐐,ℰ|\theta ),$ (1)

${\tau }^{\ast }=\underset{\tau }{\arg \min }$ $𝒞(\tau ,𝐗,\widehat{𝐘},\widehat{𝐎}),$

where $f$ denotes the co-design of HPP, and $\theta$ is the model parameters. Specific co-design and formations for each module in HPP are illustrated in the following sections.

III-B Perception Scene Encoding

Perception pipelines in HPP aim to capture scene context $𝐗$ with raw image inputs under BEV geometry. Scene contexts are then jointly encoded to catch their global relations.

III-B1 BEV Perception

In HPP, multi-view image features $𝐈\in {ℝ}^{6\times H_{in}\times W_{in}\times C}$ are extracted via shared backbones [47] from raw image inputs. We leverage BEV encoder [39] to transform $𝐈$ into BEV feature $𝒮$ through recurrent top-down BEV queries $Q_B\in {ℝ}^{H\times W\times D}$, where $D$ denotes the hidden dimensions. Founded on BEV backbones, we utilize two DETR-like perception decoders [2] in extracting scene context features for agent $𝒜$ and map $𝒮$ from $Q_B$, following $N_A$ agent queries $Q_A\in {ℝ}^{N_A\times D}$ and $N_M$ map queries $Q_{Map}\in {ℝ}^{N_M\times D}$. Note that HPP focuses on integrated prediction and planning. Therefore, it is expected better results for HPP using more advanced BEV perception units.

III-B2 Scene Encoding

to model the global interactions between scene elements with BEV perceptions, we inherit from our previous work [48, 9] to gather and encode separate visual and vectorized scene context features. Specifically, visual features are encoded by BEV queries $Q_B$, and the scene features for map and agent are concatenated and encoded as $Q_s=[Q_{Map};Q_A]\in {ℝ}^{(N_A+N_M)\times D}$, where $[\cdot ;\cdot ]$ denotes concatenations. Encoded results $𝐗=\{Q_B,Q_{Map},Q_A\}$ are then served as input for HPP co-design integrating predictions and planning.

III-C Ms-OccFormer

HPP formulates joint predictions as occupancy ${𝐎}_{1:T}$ consistently with BEV geometry in perception. To further tackle the consistency challenges between hybrid prediction, in HPP we propose MS-OccFormer spotlights twp aspects, i.e. Marginal-conditioned occupancy that defines tractable predictions, and Multi-scale prediction-wise integration that deals with the interactive alignments with different granularity. Illustrated in Fig. 3, MS-OccFormer utilizes a streaming-based pipeline to roll out the future horizon $T$ , decoding per-step occupancy prediction based on $L$ levels integration of succeeded step features.

III-C1 Modular Queries

we leverage occupancy queries $Q_{occ}\in {ℝ}^{H\times W\times D}$ in multi-scale aggregating for positional and perception features: $Q_{occ}=\mathrm{MLP}([\mathrm{PE}(I_B);Q_B])$. Positional grids $I_B\in {ℝ}^{\times H\times W\times 2}$ are encoded using sinusoidal $\mathrm{PE}(\cdot )$ and transformed by multi-layer perceptron (MLP). We further downsample $Q_{occ}$ under $L$ levels ${\{Q_{occ}^{l,0}\in {ℝ}^{\frac{H}{2^l}\times \frac{W}{2^l}\times D}\}}_{l=L}^0$ to recurrently query multi-scale interactions.

III-C2 Marginal Dependencies

To fully extract the interactive marginal prediction features, we conduct an agent-wise fusion (see Fig. 3c) that leverages the marginal future ${\widehat{𝐘}}_M^A$ outputs from GTFormer (Sec. III-D). Multi-modal motion features $Q_M^A\in {ℝ}^{N_A\times M\times D}$ are fused with marginal features ${𝐇}_M^A=\mathrm{MLP}(\mathrm{PE}(\widehat{𝐲}))$ and projected by each horizon:

$${𝐇}_{traj}^A={\mathrm{MLP}}_{1:T}(\widehat{𝐩}(Q_M^A+{𝐇}_M^A)+Q_A),$$ (2)

where ${𝐇}_{traj}^A\in {ℝ}^{T\times N_A\times D}$ denotes the marginal features.

III-C3 Marginal-conditioned Occupancy

The primary challenge in formulating ${𝐎}_{1:T}\in {ℝ}^{T\times H\times W}$ lies in the intractability with marginal predictions that cause joint inconsistencies. Inspired by instance-level occupancy ${𝐎}_{1:T}^A\in {ℝ}^{T\times H\times W\times N_A}$ [2] and conditional methods [24], we propose the marginal-conditioned occupancy prediction task. This models the consistent joint occupancy $p({𝐎}_{1:T}^A|{𝐘}_M^A,𝐗)$ over agent-wise marginal predictions. To associate uncertainty and mutual interactions, given final joint decoding features $Q_{occ}^L$ and marginal features ${𝐇}_{traj}^A$, the marginal-conditioned occupancy will be eventually modeled by dot products:

$${𝐎}_{1:T}^A=\sigma (Q_{occ}^L\cdot \mathrm{MLP}{({𝐇}_{traj}^A)}^T),$$ (3)

where $\sigma$ denotes the sigmoid function for per-grid probabilities. The original task can be then transformed back ${𝐎}_{1:T}={\max }_A{𝐎}_{1:T}^A$ for co-design of other HPP modules:

III-C4 Multi-scale Prediction-wise Integration

aims to iteratively align multi-scale interaction features between hybrid prediction in decoding ${𝐎}_{1:T}^A$.In Fig. 3a, multi-scale succeeded occupancy features ${\{Q_{occ}^{l,t-1}\}}_{l=1}^L$ query aligned marginal features by attentions at different granularities from two-stage Transformer decoders.

The global integration stage leverages the vanilla Transformer decoders to perform per-grid interactions from flattened high-level joint features $Q_{occ}^{L,t-1}$ with marginal ones. Subsequently, with the upscaling of occupancy features ${\{Q_{occ}^{l,t-1}\}}_{l=1}^{L-1}$, the local integration stage focuses on capturing consistency from partial joint behaviors with marginal features. This motivates us to design shift-window multi-head cross-attention (SW-MCA), inspired by SW-MSA in [49]. As depicted in Fig. 3b, we employ the rolling process to simultaneously capture local interactions under shifted windows attention.

To ensure interactive consistency across multi-scale integration, we devise a learnable attention mask ${𝐌}_{1:T}^l\in {ℝ}^{T\times \frac{H}{2^{L-l}}\times \frac{W}{2^{L-l}}\times N_A}$ for Transformer decoder that iteratively refines upon interaction results from the previous scale. This aligns the attention modeling based on the previous results Shown in Fig. 3a, for each level, the attention mask gets updated with agent-conditioned occupancy on the current scale level:

$${\widehat{𝐌}}^l=\sigma (Q_{occ}^l\cdot {\mathrm{MLP}}_l{({𝐇}_{traj}^A)}^T).$$ (4)

The attention masks are then iteratively updated following:

$${𝐌}^l={\lambda }_m\mathrm{Upsample}({𝐌}^{l-1})+(1-{\lambda }_m){\widehat{𝐌}}^l,$$ (5)

where ${\lambda }_m=0.5$ is the update factor. In general, given Transformer decoder at certain stage as $\mathrm{Trans}$, the prediction-wise integration under scale $l$ of timestep $t$ is defined as:

$$Q_{occ}^{l,t}=\mathrm{Trans}(q=Q_{occ}^{l,t-1},k,v={𝐇}_{traj}^{A,t},m={𝐌}_t^l).$$ (6)

Output joint occupancy features $Q_{occ}^L$ will be eventually fused via Equ. 3 for conditioned occupancy predictions ${\widehat{𝐎}}_{1:T}^A$.

Figure 4: a) A single-step reasoning layer of GTFormer. Level-$K$ reasoning queries interactive behaviors hierarchically for all agents $Q_M^A$ in predictions and planning. Meanwhile, it considers interactions with scene context $Q_A,Q_{Map}$ and joint predictive BEV features ${𝐇}_{occ}$; b) Occupancy fusion for joint prediction features; c) hybrid-prediction aware Ego Planner conditioned on plan context.

III-D GTFormer

In HPP, ensuring interactive consistency between motion predictions and planning involves the introduction of GTFormer. As depicted in Fig. 4, GTFormer formulates future interactive behaviors as a game-theoretic reasoning process, simulating hierarchical reasoning through $K$-layers stacked Transformer decoders. Beyond the uniform agent-wise reasoning model of our previous work [34], GTFormer collaborates with MS-OccFormer (Sec. III-C), modeling occupancy interactions to modulate joint future behaviors at each level of reasoning.

III-D1 Modular Queries

We leverage motion queries $Q_M^A\in {ℝ}^{N_A\times M\times D}$ to initialize agent-wise interactive reasoning with multi-modal motion intentions $I_A\in {ℝ}^{N_A\times M\times 2}$ and agent perception features by: $Q_M^A=\mathrm{MLP}([\mathrm{PE}(I_A);Q_A])$.

III-D2 Joint Dependencies

We transform joint occupancy predictions ${\widehat{𝐎}}_{1:T}\in {ℝ}^{T\times H\times W}$ with positional features $I_B$ to align with continuous geometry for predictions and planning. Shown in Fig. 4b, we conduct a multiplication with max-pooling to encode occupancy positional features with BEV semantics:

$${𝐇}_{occ}=\mathrm{ResConv}(\underset{T}{max}\mathrm{Conv}(\mathrm{PE}(I_B)\widehat{𝐎})+Q_{occ}),$$ (7)

where $\mathrm{Conv}$ and $\mathrm{ResConv}$ represent convolutional projections and residual layers, respectively. The joint features ${𝐇}_{occ}\in {ℝ}^{H\times W\times D}$ are then interacted with all agents.

III-D3 Game-Theoretic Transformer Layer

We employ level-k games to model the GTFormer layer (see Fig. 4a) for future interactive behaviors of all agents. Inherited from our previous work [34], we denote agent-wise multi-modal predictions and planning ${\widehat{𝐘}}_M^A$ as player policy ${\{{\pi }_i\}}_{i=0}^{N_A-1}$. For each player $i$, the reasoning process is defined as a $K$-times iterations of policy conditioned on the opponents $\neg i$ policy from last reasoning level: ${\pi }_i^k({𝐘}_M^i|{\pi }_{\neg i}^{k-1},𝐗),k\in [1,K-1]$. Specifically, the level-0 policy is reasoned independently: ${\pi }_i^0({𝐘}_M^i|𝐗)$. We leverage the Gaussian mixture model (GMM) to outline the uncertainty for predictions and planning policy as:

$${\pi }_i^k=\sum _{m=1}^{M}p({\widehat{𝐩}}_m^i|{\pi }_{\neg i}^{k-1},𝐗)\sum _{t=1}^T\varphi ({\widehat{𝐲}}_m^{t,i},{\widehat{\sigma }}_m^{t,i}|{\pi }_{\neg i}^{k-1},𝐗),$$ (8)

where $\varphi$ and ${\sigma }_m^{t,i}\in {ℝ}^{N_A\times M\times T\times 2}$ represent the density function and variance, respectively for 2D Gaussian distributions.

Specifically, a single layer of GTFormer is shared by level-$k$ policy ${\pi }_{0:N_A-1}^k$ (Fig. 4a) across all agents. The $k^{th}$ layer starts with a multi-head self-attention (MHSA), named Mode2Mode for last level motion queries $Q_{M,k-1}^A\in {ℝ}^{N_A\times M\times D}$ to model future interactions across modalities. Four multi-head cross-attention (MHCA) are devised for $Q_{M,k-1}^A$ to aggregate the policy conditions separately for reasoning. (1) GT-reasoning interacts with last level components policies ${\pi }_{0:N_A-1}^{k-1}$. Fused by Equ. 2 with ${\widehat{𝐩}}_{M,k-1}^{\neg A}$ and ${\widehat{𝐲}}_{M,k-1}^{\neg A}$, components features are filtered by future mask [34] to query future reasoning behaviors as $F_{GT}^k$. (2) Mode2Agent and (3) Mode2Map outlines the scene context interactions of agents $Q_A$ and maps $Q_{Map}$ as $F_A^k$ and $F_{Map}^k$. (4) Mode2Occ leverage deformable MHCA (DCA) [39] to model the future interactions with joint occupancy feature. ${𝐇}_{occ}$ is queried by A set of offsets $\mathrm{\Delta }\in {ℝ}^{N_A\times M\times N_p\times 2}$ referenced by $𝒫({\widehat{𝐲}}_{M,k-1}^A)$ transformed to pixel coordinate. $N_p$ referenced occupancy are then aggregated as $F_{occ}^k$ for each agent. In general, all interactive features are concatenated to update the motion query:

$$Q_{M,k}^A=\mathrm{FFN}([F_{GT}^k+F_A^k;F_{Map}^k;F_{occ}^k]),$$ (9)

where $\mathrm{FFN}(\cdot )$ denotes the feed-forward layers for residual MLPs and layer-norm. We omit the reasoning attention of $F_{GT}^0$ at $k=0$ for independent future policies. Eventually, updated motion queries $Q_{M,k}^A$ are passed through score and trajectory decoders for reasoned GMM policies of predictions and planning.

III-E Ego Planner

HPP introduces the Ego Planner depicted in Fig. 4c. Ego Planner specifies planning-oriented conditions $ℰ$ based on the planning reasoning policy ${\pi }_0^{K-1}$. The co-design of HPP facilitates a hybrid-prediction aware Ego Planner that collaborates with GTFormer and MS-OccFormer, defined as $p(\tau |{\pi }_0^{K-1},\widehat{𝐘},\widehat{𝐎},ℰ,𝐗)$. This is essential as the game-theoretic process in GTFormer (Sec. III-D) simply models ego planning uniformly with other agents.

III-E1 Modular Queries

In the context of target-driven planning, we define plan queries $Q_{ℰ}\in {ℝ}^{1\times D}$ that integrate planning reasoning features $Q_{M,K-1}^0$ using plan context $ℰ$ through a max-pooling fusion by: $Q_{ℰ}={max}_M\mathrm{MLP}([Q_{M,K-1}^0;{𝐇}_{ℰ}])$. Here, ${𝐇}_{ℰ}\in {ℝ}^{1\times D}$ encompasses global target features $𝒢\in {ℝ}^{1\times D}$ for encoded navigation commands or coordinates, traffic light features $\mathrm{TL}\in {ℝ}^{1\times D}$, and optional ego status ${𝐬}_{\text{ego}}\in {ℝ}^{1\times D}$ for ego speed and heading $\{v,\psi \}$.

III-E2 Hybrid Predictions Dependencies

We directly utilize a stack of $L_p$ interaction Transformers parts in the GTFormer layer (Fig. 4a) to model future joint interactions $\widehat{𝐎}$ and marginal interactions $\widehat{𝐘}$, informed by reasoned planning features and target-conditioned planning queries $Q_{ℰ}$. The GT-Reasoning module incorporates hybrid prediction awareness for marginal motion interactions ${\widehat{𝐘}}_{M,K-1}^{\neg A}$, while the Mode2Occ module handles joint occupancy interactions ${\widehat{𝐎}}_{1:T}$. By aggregating interactive future behaviors from hybrid prediction, plan queries pass through an identical motion decoder, resulting in a refined planning trajectory $\tau \in {ℝ}^{T\times 2}$.

III-F System Learning and Optimization Design

We present a collaborative learning and optimization paradigm for the HPP system. With detached BEV backbones, the co-designed framework undergoes a two-stage training process: 1) warm-up learning of two perception decoders using perception objectives ${ℒ}_{per}$ [2]; 2) end-to-end supervised learning using all modular objectives $ℒ$ as:

$$ℒ={ℒ}_{per}+{ℒ}_{occ}+{ℒ}_{GT}+{ℒ}_{plan}.$$ (10)

During inference, hybrid-prediction guided optimization refines the planning ${\tau }^{\ast }$ by minimizing cost functions $𝒞$. In the following, we elaborate on modular objectives, costs, and co-optimization strategies.

III-F1 Modular Objectives

For precise prediction of marginal-conditioned occupancy ${\widehat{𝐎}}_{1:T}^A$ in MS-OccFormer, we employ a combination of top-k BCE loss and Dice loss [2] jointly for ${\widehat{𝐎}}_{1:T}^A$ and ${𝐌}_{1:T}$, for balanced predictions of occupancy probabilities: ${ℒ}_{occ}={ℒ}_{\text{BCE}}+{\lambda }_{\text{Dice}}{ℒ}_{\text{Dice}}$, with ${\lambda }_{\text{Dice}}=5$.

To capture the hierarchical reasoning process for all agents in GTFormer, the min-max objectives for policy in level-$k$ consist of ${ℒ}_{\text{IL}}^k$, minimizing imitative behaviors, and ${ℒ}_{\text{col}}^k$, maximizing the interactive distance. The overall objective is defined as ${ℒ}_{GT}={\sum }_{k=0}^{K-1}({ℒ}_{\text{IL}}^k+{\lambda }_{\text{col}}{ℒ}_{\text{col}}^k)$. Here, ${ℒ}_{\text{IL}}^k$ represents the negative log-likelihood (NLL) loss for reasoning policies ${\pi }_{0:N_A-1}^k$, with the closest final displacement errors (FDE) for each agent as a positive mixture:

$${ℒ}_{\text{IL}}^k=\sum _{i=0}^{N_A-1}\sum _{t=1}^T\sum _{m=1}^M\mathrm{𝟏}(m={\widehat{m}}^i){ℒ}_{\text{NLL}}({\widehat{𝐲}}_m^{t,i},{\sigma }_m^{t,i},{\widehat{𝐩}}_m^i),$$ (11)

where $\widehat{m}$ denotes the positive index, ${ℒ}_{\text{NLL}}$ is defined as:

$${ℒ}_{\text{NLL}}=\log {\sigma }_x+\log {\sigma }_y+\frac{1}{2}\left({\left(\frac{{\mathrm{\Delta }}_x}{{\sigma }_x}\right)}^2+{\left(\frac{{\mathrm{\Delta }}_y}{{\sigma }_y}\right)}^2\right)-\log \left(p\right),$$ (12)

Here ${\sigma }_x,{\sigma }_y\in {\sigma }_m^{t,i}$ and ${\mathrm{\Delta }}_{xy}={𝐲}_{xy}^{t,i}-{\widehat{𝐲}}_{m,xy}^{t,i}$. We leverage cross-entropy loss for ${\widehat{𝐩}}_m^i$ in updating scoring. For interactive loss ${ℒ}_{\text{col}}^k$, it is modeled by maximize L2 distance $𝒟$ for closest trajectories within $d_{\text{col}}$ under last level component policies:

(13)

To perform refined planning $\tau$ in Ego Planner, it is learned with L2 distance $𝒟$: ${ℒ}_{plan}=𝒟(\tau ,{𝐲}^0)$.

III-F2 Cost Profiles

The cost function profiles encompass diverse cost terms $\{c_i\}$, taking into account various aspects of planning performance, categorized as: driving progress, comforts, adherence to rules [9], and, crucially, safety. Importantly, the planning safety is defined by Gaussian potential fields incorporating joint $\widehat{𝐎}$ and marginal $\widehat{𝐘}$ predictions. Motivated by complementary safety guidance by hybrid prediction, the explicit catalog is as follows:

$$c_t^{\mathrm{safe}}=\sum _{{\widehat{𝐎}}_t^{x,y}\in D_1}\varphi ({\tau }_t,{\widehat{\text{O}}}_t)+\sum _{i=1}^{N_A-1}\sum _{m=1}^M\sum _{{\widehat{\text{y}}}^t\in D_2}\varphi ({\tau }_t,{\widehat{\text{y}}}_m^{t,i}).$$ (14)

Here, $\varphi$ represents the Gaussian density functions as in Equ. 8. ${\widehat{𝐎}}_t^{x,y}={𝒫}^{-1}({\widehat{𝐎}}_t^{h,w})$ denotes the occupied coordinates, and ${\widehat{\text{y}}}_m^{t,i}$ is derived from reasoned results of ${\widehat{𝐘}}_{M,K-1}^A$. Each potential field is subjected to masking by a distance threshold $D_1=5,D_2=3$ towards planning trajectories.

III-F3 Optimization

The optimization for hybrid prediction-guided planning is defined as an open-loop optimization problem under finite horizons. The general formulation is as follows:

$${𝐮}^{\ast }=\underset{𝐮}{\arg \min }\frac{1}{2}\sum _i{\Vert {\omega }_i{c}_i(𝐮,𝐗,\widehat{𝐘},\widehat{𝐎})\Vert }^2,$$ (15)

where $𝐮$ is the planning variable, and ${\omega }_i$ denotes the weight for cost function $c_i$. For generalized ADS, HPP solves this optimization problem according to different criteria:

Reference routes: Suppose an accessible reference route $ℐ\in {ℝ}^{L_{\text{ref}}\times d_{\text{ref}}}$ that is densely interpolated, HPP transforms all cost profiles under Frenet coordinates to alleviate optimization difficulties. Each reference point $r\in ℐ$ is assigned with tangential and normal vectors: $[{\overrightarrow{t}}_r,{\overrightarrow{n}}_r]$. The Cartesian coordinate $\overrightarrow{y}=(x,y)$ can then be transformed to $\overrightarrow{r}=(s,d)$ via:

$$\overrightarrow{y}(s(t),d(t))=\overrightarrow{r}(s(t))+d(t){\overrightarrow{n}}_r(s(t)).$$ (16)

Path planning: This handles trajectories of lower frequency for the ego vehicle. Direct optimization is performed for $\tau =𝐮$ from the Ego Planner, producing the optimized path ${\tau }^{\ast }$. Only the safety cost is considered in this operation.

Motion planning: This addresses per-step future states of the ego vehicle. Optimization is conducted using model predictive control (MPC) for control actions $𝐮={[a,\delta ]}_{1:T}$ based on inverse dynamics: ${𝐮}_t={𝒯}^{-1}({\tau }_{t+1},{\tau }_t)$ [36]. The optimal motion planning is then transformed back through forward dynamics: ${\tau }_{t+1}^{\ast }=𝒯({\tau }_t^{\ast },{𝐮}_t^{\ast })$ after optimizations.

To solve this non-linear optimization problem, as illustrated in Equ. 15, we utilize the Gauss-Newton method [50]. that iteratively refines the planning variable as the initial value. The cost weights can be meticulously designed or learned directly, as the entire optimization process is fully differentiable [3].

IV Experiments

In this section, we first introduce the experimental settings for the proposed HPP, including testing benchmarks, evaluation metrics, and detailed implementations. Subsequently, HPP is quantitatively compared against existing state-of-the-art methods and systems in predictions and planning. Discussions on ablation studies unveil the effectiveness and mechanism of modular co-design optimizations for HPP. Qualitative results further ablate the characteristics of HPP against certain state-of-the-art baselines.

IV-A Experimental Setup

IV-A1 Testing Benchmarks

To discover the comprehensive performances in predictions and planning for HPP, we summarize three questions to be tackled by benchmark testing: (1) How is the capabilities of HPP as full-stack ADS under interactive real-world cases? (2) How is the long-term horizon performance of HPP in interactive real-world scenarios? (3) How is the long-term driving functioning by HPP in continuous realistic scenarios? These prompt benchmarks accordingly:

(1) nuScenes dataset [51]: This dataset is among the largest and most widely used for full-stack autonomous driving. It includes over 1,000 20-second frames of driving scenarios annotated at 2 Hz, covering four cities worldwide. Benchmarked evaluations [2] involve 6,019 frames for all tasks of ADS under open-loop settings. Testing horizons are defined at $T=3\mathrm{s}$ and $T=6\mathrm{s}$ for motion predictions.

(2) Waymo open motion dataset (WOMD) [52]: Utilized for long-term motion evaluations, it is the largest real-world dataset for interactive scenarios. It comprises 104,000 20-second frames representing unique scenarios, marked at 10 Hz, and encompasses over 570 km of driving and 1750 km of roadways. To assess long-term real-world performance in planning and motion predictions, evaluations are systematically conducted in the SMARTS benchmark [34]. This includes 400 highly interactive scenarios, each lasting 9 seconds, featuring representative behaviors. Autonomous vehicles are tasked with 5-second long-term planning and predictions in both open-loop and closed-loop configurations. Closed-loop testing involves leveraging the log simulator to replay driving scenarios for online interactions. To further examine the performance of joint predictions in HPP, the system is tested on the Waymo Occupancy Predictions benchmark [53]. This involves predicting over 44,000 driving scenes of occupancy and flow within an 8-second timeframe at a frequency of 1 Hz.

TABLE I: nuScenes Open-loop Planning Testing Results (${}_{}{}^{†}:$ Lidar inputs; ${}_{}{}^{‡}:$ Ego status; ${}_{}{}^{\ast }:$ Augmentations)

Methods	Collision rate (%)$\downarrow$				Planning error (m)$\downarrow$
	@1 s	@2 s	@3 s	Avg.	@1 s	@2 s	@3 s	Avg.
NMP† [54]	-	-	1.92	-	-	-	2.31	-
SA-NMP† [54]	-	-	1.59	-	-	-	2.05	-
FusionAD†‡ [15]	0.02	0.08	0.27	0.12	-	-	-	0.81
FF† [55]	0.06	0.17	1.07	0.43	0.55	1.20	2.54	1.43
EO† [56]	0.04	0.09	0.88	0.33	0.67	1.36	2.78	1.60
OccNet [57]	0.21	0.59	1.37	0.72	1.29	2.13	2.99	2.13
UniAD [2]	0.05	0.17	0.71	0.31	0.48	0.96	1.65	1.03
HPP‡	0.03	0.07	0.35	0.15	0.30	0.61	1.15	0.72
HPP	0.03	0.17	0.68	0.29	0.48	0.91	1.54	0.97
Avg. Metrics	@1 s	@2 s	@3 s	Avg.	@1 s	@2 s	@3 s	Avg.
ST-P3 [4]	0.23	0.62	1.27	0.71	1.33	2.11	2.90	2.11
VAD-Base‡ [5]	0.07	0.10	0.24	0.14	0.17	0.34	0.60	0.37
VAD-Base [5]	0.07	0.17	0.41	0.22	0.41	0.70	1.05	0.72
DeepEM∗ [58]	0.05	0.15	0.36	0.19	0.25	0.45	0.73	0.48
HPP‡	0.02	0.04	0.11	0.06	0.26	0.37	0.59	0.40
HPP	0.03	0.08	0.24	0.12	0.41	0.61	0.86	0.63

(3) CARLA simulator [59]: We utilize the Longest6 benchmark [60] for long-term driving evaluations. The autonomous vehicle is assigned a $T=2\mathrm{s}$ horizons closed-loop planning task across 36 routes, ranging from 1.6 to 1.8 km, under various driving conditions in six CARLA towns.

IV-A2 Testing Metrics

We adhere to the original benchmarks in the testing metrics configurations. Coinciding with the contributions of HPP, the devised metrics primarily focus on three aspects, i.e., Accuracy, Consistency, and Safety for predictions and planning. Detailed metrics are listed as follows:

(1) Occupancy prediction: Intersections over Union (IoU) and Area Under the Curve (AUC) [61] quantify the overall and per-grid prediction accuracy for occupancy. Video Panoptic Quality (VPQ) [62] is adopted to assess the consistency of occupancy across marginal agents and perceptions.

(2) Motion prediction: Prediction accuracy is assessed using minimum average and final displacement errors (minADE, minFDE) for trajectories, as well as miss rate (MR) for each agent[52]. Consistency is tested through joint displacement errors (JADE, JFDE) for all agents[34], along with End-to-End Prediction Accuracy (EPA) [63] over perceptions.

(3) Planning: For open-loop testing, planning accuracy is evaluated using displacement errors (DE)[2] and the average distance [5]. Consistency in predictions and safety is assessed through collision rates (CR) [64]. In closed-loop settings, external measurements include infractions (IS), vehicle collisions (CV), routes completion (RC), and a driving score (DS) that encompasses overall driving performance [59].

IV-A3 Implementation Details

For fair comparisons, HPP is configured according to each benchmark with carefully devised learning pipelines and system architectures. In the nuScenes dataset, full-size training is conducted with a total batch size of 4. For planning, 10% of the full training set is randomly sampled, and the full set is utilized for the occupancy benchmark in WOMD. Learning occurs in batches of 24. The expert dataset from the CARLA benchmark [28], collected in different towns, is directly used for training in batches of 32.

Training strategies for all benchmarks are aligned using a distributed strategy on four NVIDIA A100 GPUs. The AdamW optimizer is employed with an initial learning rate of 1e-4, and a cosine annealing learning rate strategy is applied. The total number of training epochs is set to 20. We apply the same GPU devices for nuScenes and WOMD for testing. Evaluations for the CARLA benchmark are conducted in one NVIDIA RTX 3080 GPU.

For system architectures, HPP establishes BEV perception ego-centered within $\pm$50 m of $H,W=200$ in nuScenes. In WOMD and CARLA benchmarks, privileged perceptions are assumed. Therefore, HPP is developed by encoding perfect scene context inputs according to our previous work [34, 65]. For CARLA benchmarks, BEV remains ego-centered within $\pm 32$ m of $H,W=128$. BEV settings for WOMD follow the official benchmark guidelines. Path planning is optimized without knowledge of reference routes in nuScenes and CARLA. In WOMD, motion planning is conducted considering reference information. The full-stack HPP in nuScenes considers various queries for agents. In WOMD and CARLA, agents are sorted and filtered to $N_A=11$. We select the ReLU activation function and apply a dropout rate of 0.1. We refer more detailed parameters with notations in Table XII.

IV-B Main Results

IV-B1 Full-stack ADS Performance

We report HPP’s testing performance against recent state-of-the-art methods on the nuScenes dataset. HPP has achieved state-of-the-art results across various key metrics in both prediction and planning.

TABLE II: Open-loop Planning Compared with LLMs (${}_{}{}^{‡}:$ Ego status)

Methods	Collision rate (%)$\downarrow$				Planning error (m)$\downarrow$
	@1 s	@2 s	@3 s	Avg.	@1 s	@2 s	@3 s	Avg.
GPTDriver‡ [44]	0.07	0.15	1.10	0.44	0.27	0.74	1.52	0.84
Agent-Driver‡ [45]	0.02	0.13	0.48	0.21	0.22	0.65	1.34	0.74
HPP‡	0.03	0.07	0.35	0.30	0.61	1.15	0.72	0.15

TABLE III: Testing Results on nuScenes Occupancy Prediction

Methods	IoU-n. $\uparrow$	IoU-f. $\uparrow$	VPQ-n. $\uparrow$	VPQ-f. $\uparrow$
FIERY[66]	59.4	36.7	50.2	29.9
StretchBEV[41]	55.5	37.1	46.0	29.0
ST-P3[4]	-	38.9	-	32.1
BEVerse[40]	61.4	40.9	54.3	36.1
PowerBEV[67]	62.5	39.4	55.5	33.8
UniAD[2]	63.4	40.2	54.7	33.5
HPP	64.8	40.5	56.4	34.7

(1) Planning results: Described in Table I, HPP achieves state-of-the-art (SOTA) results across all planning horizons (@1 s-@3 s) against various autonomous driving systems in both absolute metrics [2] and average ones [5]. Specifically, HPP presents a 5.9% lower average L2 errors and a 6.5% lower average collision rate compared to UniAD [2], utilizing identical BEV perceptions. This showcases the validity of the modular co-design for HPP in predictions and planning.

In average metrics, HPP reports a 12.5% improvement in planning errors against VAD [5] which highlights superior perception modules. The reasoning design for HPP manifests with a 30% lower collision rate compared to DeepEM [58], which also features reasoning by EM decoding and extra de-noising augmentations. Compared with methods leveraging ego-status ${𝐬}_{\text{ego}}$, a variant of HPP adding ego-status (denoted HPP^‡) in Ego Planner (Sec. III-E) is trained with boasting results. HPP^‡ does not include accelerations in [5] as leakage of ground-truths. In comparison with FusionAD [15], which depends on excessive LIDAR fusion, HPP also demonstrates an 11.8% lower planning error with comparable safety. Our system also exhibits 2.8% lower errors and nearly 20% lower collision rates compared to LLM methods (see Table II). This further substantiates the effectiveness of modular integration of predictions in planning-oriented objectives, as LLM baselines focus more on alignments by language knowledge.

(2) Predictions results: Joint results of occupancy predictions are tested in two ranges (near: $30\times 30$ m; far: $50\times 50$ m) centered on the autonomous vehicle. Shown in Table III, HPP presents advanced accuracy +2.5% and consistency +3.7% compared to [2], thanks to proposed Ms-OccFormer that integrate mutually with motion predictions. Validity of modular co-design in HPP is further manifested by +4% and +1.6% improved IoU and VPQ without extra augmentations, compared with [67] learning single occupancy task.

Marginal results of motion predictions are presented in Table IV. Here, we compare the prediction results averaged from all vehicles (-v.) and measure full agent (-f.) results weighted by categories. HPP reports a 2.8% lower minADE and a +3.2% EPA gain in vehicle predictions, with 3.7% and +6% improved performance predicting all agents compared to baselines [2] under the same perception settings. This highlights the performance gain achieved through game-theoretic reasoning and joint dependencies in GTFormer (Sec. III-D).

TABLE IV: Testing Results on nuScenes Motion Predictions

Methods	minADE $\downarrow$	minFDE $\downarrow$	MR. $\downarrow$	EPA $\uparrow$
ViP3D[63]	1.15	1.95	22.6	0.222
PnPNet[38]	2.05	2.84	24.6	0.226
UniAD-v.[2]	0.71	1.02	15.1	0.456
UniAD-f.[2]	0.911	1.236	15.1	0.314
HPP-v.	0.682	0.947	13.8	0.471
HPP-f.	0.878	1.205	14.5	0.334

TABLE V: Testing Results on Waymo Occupancy Flow Benchmark

Methods	AUC	AUC	EPE	AUC
	-obs. $\uparrow$	-occ. $\uparrow$	-f. $\downarrow$	-FT $\uparrow$
MotionPerceiver	77.1	-	-	-
OFMPNet	77.0	16.5	3.58	76.1
STCNN	74.4	16.8	3.87	73.3
HOPE[20]	80.3	16.5	3.67	83.9
STrajNet[48]	77.8	17.8	3.20	78.5
VectorFlow[21]	75.4	17.3	3.58	76.7
HPP	79.7	19.4	2.95	80.2

It is important to note that HPP focuses on predictions and planning rather than perception. Prediction results could potentially be further enhanced with better perceptions [5] or additional LIDAR inputs [15]. However, HPP still outperforms in final planning results, demonstrating better social compliance denoted as our contributions.

IV-B2 Long-horizon Interactive Performance

We benchmark HPP’s long-horizon performance under WOMD. HPP has underscored advanced performances against numerous SOTA methods in predictions and planning.

(1) Occupancy results: Shown in Table V, HPP demonstrates superior prediction accuracy (+8.8% AUC-occ.) and lower flow error (9.1% EPE-f.) when considering joint flow predictions. The explicit design of MS-OccFormer in HPP has proven its strong accuracy compared to our previous work [65], which only conducts global interactions without marginal awareness. HPP presents a +2.4% improvement in AUC-obs. and +2.2% in flow-traced occupancy.

TABLE VI: Open-loop Planning Testing Results on WOMD Scenarios (SMARTS Benchmark)

Method	Collision rate	Miss rate	Planning error (m) $\downarrow$			Prediction error (m) $\downarrow$
	(%) $\downarrow$	(%) $\downarrow$	@1 s	@3 s	@5 s	JADE	JFDE
Vanilla IL [3]	4.25	15.61	0.216	1.273	3.175	–	–
DIM [68]	4.96	17.68	0.483	1.869	3.683	–	–
OPGP [9]	3.79	12.89	0.245	1.672	3.099	–	–
MultiPath++ [69]	2.86	8.61	0.146	0.948	2.719	–	–
MTR-e2e [19]	2.32	8.88	0.141	0.888	2.698	–	–
DIPP [3]	2.33	8.44	0.135	0.928	2.803	0.925	2.059
GameFormer [34]	1.98	7.53	0.129	0.836	2.451	0.853	1.919
HPP	1.85	7.58	0.092	0.881	2.667	0.829	1.965

TABLE VII: Close-loop Planning Testing Results on WOMD Scenarios (SMARTS Benchmark)

Method	Success rate	Progress	Acceleration	Jerk	Lateral acc.	Position error to expert driver ($\mathrm{m}$) $\downarrow$
	(%) $\uparrow$	$(\mathrm{m})$ $\uparrow$	($\mathrm{m}/{\mathrm{s}}^2$)	($\mathrm{m}/{\mathrm{s}}^3$)	($\mathrm{m}/{\mathrm{s}}^2$)	@3 s	@5 s	@8 s
Expert	-	54.52	0.529	1.020	0.103	-	-
Vanilla IL [3]	0	6.23	1.588	16.24	0.661	9.355	20.52	46.33
RIP [68]	19.5	12.85	1.445	14.97	0.355	7.035	17.13	38.25
CQL [70]	10	8.28	3.158	25.31	0.152	10.86	21.18	40.17
DIPP [3]	68.12$\pm$5.51	41.08$\pm$5.88	1.44$\pm$0.18	12.58$\pm$3.23	0.31$\pm$0.11	6.22$\pm$0.52	15.55$\pm$1.12	26.10$\pm$3.88
GameFormer [34]	73.16$\pm$6.14	44.94$\pm$7.69	1.19$\pm$0.15	13.63$\pm$2.88	0.32$\pm$0.09	5.89$\pm$0.78	12.43$\pm$0.51	21.02$\pm$2.48
HPP	74.28$\pm$5.49	47.17$\pm$8.92	1.33$\pm$0.18	11.68$\pm$2.76	0.35$\pm$0.09	4.93$\pm$0.78	10.24$\pm$0.82	18.99$\pm$3.05
DIPP (optim.)	92.16$\pm$0.62	51.85$\pm$0.14	0.58$\pm$0.03	1.54$\pm$0.19	0.11$\pm$0.01	2.26$\pm$0.10	5.55$\pm$0.24	12.53$\pm$0.48
GameFormer (optim.)	94.50$\pm$0.66	52.67$\pm$0.33	0.53$\pm$0.02	1.56$\pm$0.23	0.10$\pm$0.01	2.11$\pm$0.21	4.87$\pm$0.18	11.13$\pm$0.33
HPP (optim.)	92.25$\pm$0.85	52.19$\pm$0.41	0.66$\pm$0.02	1.87$\pm$0.28	0.10$\pm$0.01	2.13$\pm$0.29	4.90$\pm$0.26	12.89$\pm$0.38

(2) Open-loop results: Table VI reveals the compelling results of HPP over numerous state-of-the-art (SOTA) methods in open-loop prediction and planning. HPP achieves notable reductions of over 50% in collisions and 26% in planning errors compared to imitation learning baselines [3, 68] that discount predictions. When compared with SOTA motion prediction baselines as imitative planners, HPP also demonstrates 2.3% to 17.6% reductions in planning errors and 21.1% fewer collisions. This underscores the importance of integrated predictions and planning.

In comparison with integrated baselines, HPP presents significant improvements over the IOP framework [9] due to the absence of future interactions for planning and the intractable occupancy in IOP that hampers normal guidance. When compared with IPP systems, HPP delivers superior performance in collision rates (6.6%), joint prediction errors (2.9%), and comparable planning errors. Compared to the state-of-the-art results from our previous works [3, 34], which focus on modeling future interactions and reasoning (see Fig. 5), HPP exhibits superior short-term performance with reduced variance. This further validates the efficacy of co-design integration for joint predictions and planning upon reasoning in HPP.

Figure 5: Open-loop predictions and planning results for IPP baselines in WOMD. HPP presents the best short-term performance with lower variance.

(3) Closed-loop results: In Table VII, HPP undergoes testing in a replay simulator against state-of-the-art IPP systems [34], IL methods [68], and RL approaches [70]. IL and RL results are significantly compromised due to accumulated distributional shifts in closed-loop interactions with the environment. HPP exhibits a higher success rate (+1.5%) and lower planning errors (13.2%) compared to our previous methods [34], thanks to the hybrid prediction-aware planner designed conditioned on goal context. The closed-loop performance sees substantial improvements with online optimizations. Due to the high cost of the occupancy process in raw data, HPP is re-planned by online refinements at 2 Hz. Testing results demonstrate strong performance against our previous methods [34], which re-plan more frequently at 10 Hz, achieving 8.5% lower closed-loop positional errors by [3].

TABLE VIII: Testing Results on CARLA Longest6 Benchmark

Methods	DS $\uparrow$	RC∗ $\uparrow$	IS $\uparrow$	CV $\downarrow$
Rule-Based[59]	38.0	29.1	0.84	0.64
KING[36]	45.1	78.3	0.55	1.67
Roach[71]	55.3	88.2	0.62	0.72
PlanT∗[28]	70.9	83.1	0.87	0.31
HPP∗	65.5	79.2	0.82	0.51

TABLE IX: Ablation study on modular co-design integration among predictions and planning

ID	MS-OccFormer(wo.)			GTFormer(wo.)			EgoPlanner (wo.)		Occupancy Prediction				Motion Prediction			Planning
	Score	Motion	AC-Occ.	Score	Motion	Occ.	Motion	Occ.	IoU-n.$\uparrow$	IoU-f.$\uparrow$	VPQ-n.$\uparrow$	VPQ-f.$\uparrow$	minADE$\downarrow$	MR$\downarrow$	EPA$\uparrow$	ADE$\downarrow$	CR$\downarrow$
1	✗	-	-	-	-	-	-	-	63.5	40.0	54.5	33.7	0.733	16.1	45.6	0.989	0.428
2	-	✗	-	-	-	-	-	-	63.1	39.4	53.6	32.8	0.739	16.2	44.8	0.994	0.441
3	-	✗	✗	-	-	-	-	-	61.9	39.6	50.8	30.9	0.748	16.6	43.8	1.015	0.466
4	-	-	-	✗	-	-	-	-	63.7	39.8	55.0	33.3	0.728	15.9	45.0	0.986	0.402
5	-	-	-	-	✗	-	-	-	63.5	39.5	54.6	33.5	0.734	16.4	44.8	0.992	0.432
6	-	-	-	-	✗	✗	-	-	63.5	39.6	54.8	33.0	0.747	16.2	44.2	0.998	0.436
7	-	-	-	-	-	-	✗	-	63.6	39.8	55.0	33.4	0.726	16.0	45.7	0.989	0.414
8	-	-	-	-	-	-	-	✗	63.5	39.7	54.7	33.3	0.721	15.6	45.4	1.012	0.458
9	-	-	-	-	-	-	✗	✗	63.5	39.8	54.8	33.4	0.726	15.9	45.7	0.997	0.449
0	-	-	-	-	-	-	-	-	63.7	39.8	55.0	33.5	0.724	15.9	45.8	0.985	0.402

IV-B3 Long-term Driving Performance

HPP demonstrates comparable driving capabilities against the state-of-the-art method [28] in the CARLA benchmark (Table VIII). With significant improvements over rule-based agents [59], HPP achieves a $+20.4$ driving score compared to IPP methods [36], which guide planning by adversarial predictions, as well as a $+12.2$ improvement compared to RL methods [71]. Noted that both HPP and the reproduced [28] (^∗) show compromised route completions (RC), likely due to GPU inference issues.

Figure 6: Qualitative ablations of multi-scale attention masks ${𝐌}_{1:T}$ compared with occupancy predictions ${\widehat{𝐎}}_{1:T}$ and the ground-truth ${𝐎}_{1:T}$. Aligned results from multiple granularities under scenarios a) and b) reveal the validity of multi-scale prediction-wise integration design in MS-OccFormer (Sec. III-C4).

IV-C Ablation Studies and Discussions

To unleash the internal effectiveness in HPP, comprehensive ablation studies are conducted in nuScenes centered on discussing the roles in modular co-design and characteristics for hybrid prediction in planning guidance.

IV-C1 Roles in Modular Co-design

To elicit the effectiveness of co-design among each module, as presented in Table IX, we purposefully remove certain key designs for integration across each sub-module. Referred to ID 1-9, all ablated baselines are compared to the original HPP (ID.0). Detailed discussions are as follows:

(1) Effects in Ms-OccFormer: For marginal dependencies (Sec. III-C2), when removing score scaling (ID.3 vs. ID.0), we observe a slight trade-off under occupancy range (-0.5 VPQ-n. vs. +0.2 VPQ-f.), with increased collisions ($+0.026\%$). This indicates the scaling captures multi-modal features for joint predictions and enhances the shared occupancy area nearby. Removing motion fusion (ID.2) causes inclusive decreases for lack of marginal alignments. Greater decreases are observed (-1.2 IoU-n. and -3.7 VPQ-n.) in removing agent-conditioned occupancy modeling. It highlights the importance of future interactions and validates the consistency towards motion predictions ($2.4\%$ lower for prediction errors). Results in planning also reflect more contributions in near-scale joint prediction interactions to planning. Comparing ID.2 and ID.3, without sacrificing far-scale occupancy (-1.2 IoU.-n vs. +0.2 IoU-f.), the planning errors and collisions increase along with a drop in near-scale prediction accuracy. For the key design of multi-scale predictions integration, qualitative (see Fig. 6) and quantitative ablations (Table X) have demonstrated notable improvements from multi-scale attention mask update design (+1.2 IoU-n.) as well as local integration (+1.0 VPQ-n.) for prediction consistency.

(2) Effects in GTFormer: As the core module that enables reasoning capabilities for consistent predictions and planning, we try to discover roles by removing marginal and joint predictions in GTFormer. Compared ID.4 with ID.5, the removal of the reasoning module had a thorough drop, especially planning ($+0.03\%$ CR) and motion predictions ($+3.1\%$ MR). This highlights the importance of reasoning in consistency and accuracy for future interactions. Meanwhile, considering ID.5 and ID.6, the increase of prediction errors ($+1.7\%$ minADE) implies that joint dependencies (Sec. III-D2) are the key in consistent motion predictions upon GT-reasoning. This reflects an enhanced mutual consistency modulated by interactive modeling between joint and marginal predictor co-design.

TABLE X: Ablations on multi-scale prediction-wise integration

Baselines	IoU-n. $\uparrow$	IoU-f. $\uparrow$	VPQ-n. $\uparrow$	VPQ-f. $\uparrow$
w/o. global integration	61.6	38.8	52.0	31.9
w/o. local integration	61.8	38.8	52.8	32.1
w/o. attn. mask ${\text{M}}_t$	62.5	39.0	53.8	32.4
HPP	63.7	39.8	55.0	33.5

Figure 7: Qualitative results of full-stack open-loop testing on nuScenes [51]. Each Sub-figure renders hybrid prediction (in green and violet) and planning (in red) under camera view (left with rendered trajectories and bounding boxes) and BEV space (right). HPP is compared with [2] under several driving scenarios featuring specific interactions: a) Cruising through intersections; b) Driving across a crowded roundabout; c) Yielding to pedestrians on crosswalks; d) Lane keeping in fork roads.

TABLE XI: Ablation study on hybrid prediction guidance for planning

ID	${ℒ}_{col}$	Occ.	Most-likely	Full-motion	Collision rate (%) $\downarrow$				Planning error (m) $\downarrow$
					@1 s	@2 s	@3 s	Avg.	@1 s	@2 s	@3 s	Avg.
1	-	-	-	-	0.11	0.31	0.73	0.39	0.19	0.53	1.13	0.61
2	✓	-	-	-	0.07	0.22	0.64	0.31	0.18	0.52	1.09	0.60
3	✓	-	✓	-	0.05	0.11	0.60	0.25	0.19	0.52	1.10	0.60
4	✓	-	-	✓	0.04	0.09	0.53	0.20	0.19	0.52	1.10	0.60
5	✓	✓	-	-	0.03	0.14	0.45	0.20	0.30	0.61	1.16	0.72
6	✓	✓	✓	-	0.03	0.14	0.44	0.20	0.31	0.61	1.17	0.72
0	✓	✓	-	✓	0.03	0.07	0.35	0.15	0.30	0.61	1.15	0.72

(3) Effects in Ego Planner: Validating the roles of plan-conditioning design compared to our previous work [34] in Sec. IV-B2, we further examine the effects of hybrid prediction interactions. In ID.9, we observe marginal improvements in predictions (+0.2 IoU, +0.1 EPA) and more significant enhancements ($-0.047\%$ CR) in planning with the inclusion of hybrid prediction interactions. This underscores the consistency modeling contributed along with the HPP design. Ablations from ID.7 and ID.8 further suggest a substantial impact of interactions with occupancy predictive features ($-0.056\%$ CR) over marginal ones ($-0.012\%$ CR) on planning. Solely motion conditional planning (ID.8) outputs overly optimistic motion predictions (-0.3 MR). This in turn harms the original game-theoretic reasoning, resulting in inferior planning ($+0.035\%$ CR). These results underscore the modulating effect of joint dependencies in both consistencies for planning and motion predictions.

IV-C2 Roles in hybrid prediction

To further delve into the characteristics of marginal and joint predictions, listed in Table XI, the original HPP (ID.0) is measured with ablations (ID.1-6) guiding: reasoning loss (${ℒ}_{col}$); marginal predictions (most likely, full); and joint predictions (Occ.) during learning and optimizations. Key findings are summarized below:

(1) Consistent reasoning: Comparing ID.1 and ID.2, significant safety improvements ($-25.8\%$ CR) highlights the consistency role for marginal predictions in reasoning learning (${ℒ}_{col}$). Joint predictions in HPP are leveraged to guide internal marginal consistency that modulates planning.

(2) Complementary influences: hybrid prediction benefit planning in mutual coverage gains compared with sole guidance ($-33\%$ CR) in ID.4 and ID.5. This is enhanced upon consistency design in HPP. 1) Compared with ID.3, guiding with full motion predictions (ID.2) spawns safe planning ($-20\%$ CR) without sacrificing accuracy. 2) A good alignment in hybrid prediction results in close performance in ID.5 and ID.6 adding the most likely marginal prediction. These mutual effects ascertain the planning performance in case either of the hybrid prediction is less functioning due to: (1) state error for pose and speed; (2) future uncertainty; or (3) discontinuation under spatial temporal horizons.

Figure 8: Qualitative comparison of HPP with SOTA integrated prediction and planning methods [3, 34] in WOMD [52]. Performances integrated motion predictions (in blue), planning (red), and occupancy predictions (violet in separated sub-figures) are rendered with ground truth (black dotted lines) and scene context. All systems are compared with highly interactive scenarios: a) Crossing an unsignalized intersection with an incoming vehicle; b) Unprotected left-turn with a potential take-over vehicle; c) Cruising a five-point intersection with heavy traffic; d) Merging in a highway; and e) Right-turn with heading cyclist and low-speed vehicle.

IV-D Qualitative Results

Figs. 7 and 8 showcase the qualitative benefits of integrating hybrid prediction in the nuScenes [51] and WOMD [52] benchmarks, respectively. In the comprehensive testing scenario depicted in Fig. 7a, joint guidance from occupancy ensures consistent motion predictions during cruising. Notably, Fig. 7b demonstrates that hybrid prediction awareness contributes to consistent planning, enhancing reasoning behavior for smooth cruising without overly conservative avoidance, thereby mitigating risks near lane boundaries. Fig. 7c further illustrates the complementary effects, where motion predictions take precedence in guiding planning when occupancy is uncertain. The mutual influence is evident in Fig. 7d, where the ego vehicle remains in the lane instead of avoiding, as the occupancy predictions for rear cyclists are constrained by marginal predictions to follow a straight trajectory, maintaining a safe distance from the ego vehicle.

In Fig. 8, qualitative assessments of WOMD interactive scenarios underscore HPP’s superior reasoning capabilities and safety-conscious planning. In Fig. 8a, DIPP [3] encounters failure in an emergency stop, exposing a lack of consistency between motion predictions and planning, while HPP and GameFormer [34] exhibit smoother planning around obstacles. Motion prediction in HPP operates more smoothly with [34], as occupancy predictions modulate the joint behaviors. This is important as the reasoned motion predictions [34] may trade consistency for safety (Fig. 8b) and accuracy (Fig. 8c). High-speed situations (Fig. 8d, Fig. 8e) further demonstrate HPP’s mutual consistency between motion predictions and planning, showcasing its adaptability and robustness in diverse driving environments. These results underscore the efficacy of HPP in enhancing both the consistency and safety aspects of the autonomous driving system.

IV-E Future Outlook

In introducing HPP, we advocate for the adoption of modular co-design and optimization principles to shape ADS. As the notion of a planning-oriented modular learning system gains traction, it is crucial to underscore the significance of well-organized and concurrently integrated modules that mirror the complexities of real-world scenarios. In the future, we see agent-based models, such as LLMs, facilitating connections between various modules to ensure seamless integration. HPP lays the groundwork for module-wise integration, and future endeavors will delve deeper into exploring intermediate-level integration and guidance between prediction and planning.

V Conclusions

In this paper, we develop HPP, a modular co-design optimization framework for autonomous driving systems. The main focus of HPP is on integrating hybrid prediction and planning, wherein three sub-modules, i.e., MS-OccFormer, GTFormer, and Ego Planner address consistency issues and enhance adaptability using the hybrid prediction-guided learning and optimization pipeline. HPP has been extensively evaluated across diverse benchmarks, and the results consistently demonstrate its superior performance in both prediction and planning metrics compared to state-of-the-art methods. The hybrid prediction awareness in HPP, which incorporates joint behaviors and occupancy predictions, improves qualitative consistency, safety, and feasibility in real-world scenarios. By elucidating the roles in modular co-design and the complementary effects that enhance planning through hybrid prediction, HPP showcases its potential to advance the field by tackling challenges in prediction and planning, contributing to the ongoing evolution of autonomous driving frameworks.

TABLE XII: Notations and parameters

Notation	Meaning	Testing Benchmarks
		nuScenes	WOMD	CARLA
$H,W$	Size of BEV space	200	128/256	128
	BEV length (m)	50	64	64
$N_A$	Max number of agents	-	11/33	11
$N_M$	Max number of map	-	100	50
$T$	Future Horizons (s)	3/2/6	5/8	2
	Future frequency (Hz)	2	2/1	2
$M$	Number of modalities	6	6	6
$L$	Occ integration levels	3	3	3
$K$	Reasoning levels	3	3	3
$L_P$	Layers in Ego Planner	3	3	3
$h$	Number of attention heads	8	8	8
$D$	Embedding size	256	256	256
	Activation function	$\mathrm{ReLU}$
	Dropout rate	0.1	0.1	0.1
	Batch size	4	24	32
	Learning rate	$1e^{-4}$	$1e^{-4}$	$1e^{-4}$
	Training epochs	20	20	20