Stochastic Scene-Aware Motion Prediction

Abstract

A long-standing goal in computer vision is to capture, model, and realistically synthesize human behavior. Specifically, by learning from data, our goal is to enable virtual humans to navigate within cluttered indoor scenes and naturally interact with objects. Such embodied behavior has applications in virtual reality, computer games, and robotics, while synthesized behavior can be used as training data. The problem is challenging because real human motion is diverse and adapts to the scene. For example, a person can sit or lie on a sofa in many places and with varying styles. We must model this diversity to synthesize virtual humans that realistically perform human-scene interactions. We present a novel data-driven, stochastic motion synthesis method that models different styles of performing a given action with a target object. Our Scene-Aware Motion Prediction method (SAMP) generalizes to target objects of various geometries while enabling the character to navigate in cluttered scenes. To train SAMP, we collected MoCap data covering various sitting, lying down, walking, and running styles. We demonstrate SAMP on complex indoor scenes and achieve superior performance than existing solutions. Code and data are available for research at https://samp.is.tue.mpg.de.

Figure 1: SAMP synthesizes virtual humans navigating complex scenes with realistic and diverse human-scene interactions.

1 Introduction

The computer vision community has made substantial progress on 3D scene understanding and on capturing 3D human motion, but less work has focused on synthesizing 3D people in 3D scenes. The advances in these two sub-fields, however, have provided tools for, and have created interest in, embodied agents for virtual worlds (e.g. [35, 55, 42, 56]) and in placing humans into scenes (e.g. [6, 21]). Creating virtual humans that move and act like real people, however, is challenging and requires tackling many smaller but difficult problems such as perception of unseen environments, plausible human motion modeling, and embodied interaction with complex scenes. While advances have been made in human locomotion modeling [23, 32] thanks to the availability of large scale datasets [33, 50, 7, 45, 38], realistically synthesizing virtual humans moving and interacting with 3D scenes, remains largely unsolved.

Imagine instructing a virtual human to “sit on a couch” in a cluttered scene, as illustrated in Fig. 1. To achieve this goal, the character needs to perform a series of complex actions. First, it should navigate through the scene to reach the target object while avoiding collisions with other objects in the scene. Next, the character needs to choose a contact point on the couch that will result in a plausible sitting action facing the right direction. Finally, if the character performs this action multiple times, there should be natural variations in the motion, mimicking real-world human-scene interactions; e.g., sitting on different parts of the couch with different styles such as with crossed legs, arms in different poses, etc. Achieving these goals requires a system to jointly reason about the scene geometry, smoothly transition between cyclic (e.g., walking) and acyclic (e.g., sitting) motions, and to model the diversity of human-scene interactions.

To this end, we propose SAMP for Scene-Aware Motion Prediction. SAMP is a stochastic model that takes a 3D scene as input, samples valid interaction goals, and generates goal-conditioned and scene-aware motion sequences of a character depicting realistic dynamic character-scene interactions. At the core of SAMP is a novel autoregressive conditional variational autoencoder (cVAE) called MotionNet. Given a target object and an action, MotionNet samples a random latent vector at each frame to condition the next pose both on the previous pose of the character as well as the random vector. This enables MotionNet to model a wide range of styles while performing the target action. Given the geometry of the target object, SAMP further uses another novel neural network called GoalNet to generate multiple plausible contact points and orientations on the target object (e.g., different positions and sitting orientations on the cushions of a sofa). This component enables SAMP to generalize across objects with diverse geometry. Finally, to ensure the character avoids obstacles while reaching the goal in a cluttered scene, we use an explicit path planning algorithm (A* search) to pre-compute an obstacle-free path between the starting location of the character and the goal. This piecewise linear path consists of multiple way-points, which SAMP treats as intermediate goals to drive the character around the scene. SAMP runs in real-time at $30$ fps. To the best of our knowledge, these individual components make SAMP the first system that addresses the problem of generating diverse dynamic motion sequences that depict realistic human-scene interactions in cluttered environments.

Training SAMP requires a dataset of rich and diverse character scene interactions. Existing large-scale MoCap datasets are largely dominated by locomotion and the few interaction examples lack diversity. Additionally, traditional MoCap focuses on the body and rarely captures the scene. Hence, we capture a new dataset covering various human-scene interactions with multiple objects. In each motion sequence, we track both the body motion and the object using a high resolution optical marker MoCap system. The dataset is available for research purposes.

Our contributions are: (1) A novel stochastic model for synthesizing varied goal-driven character-scene interactions in real-time. (2) A new method for modeling plausible action-dependent goal locations and orientations of the body given the target object geometry. (3) Incorporating explicit path planning into a variational motion synthesis network enabling navigation in cluttered scenes. (4) A new MoCap dataset with diverse human-scene interactions.

2 Related Work

Interaction Synthesis: Analyzing and synthesizing plausible human-scene interactions have received a lot of attention from the computer vision and graphics communities. Various algorithms have been proposed for predicting object functionalities [16, 66], affordance analysis [18, 53], and synthesizing static human-scene interactions [16, 18, 21, 27, 41, 62, 64].

A less explored area involves generating dynamic human-scene interactions. While earlier work [28] focuses on synthesizing motions of a character in the same environment in which the motion was captured, follow up work [29, 43, 2, 26] assembles motion sequences from a large database to synthesize interactions with new environments or characters. Such methods, however, require large databases and expensive nearest neighbor matching.

An important sub-category of human-scene interaction involves locomotion, where the character must respond to changes in terrain with appropriate foot placement. Phase-functioned neural networks [23] have shown impressive results by using a guiding signal representing the state of the motion cycle (i.e., phase). Zhang et al. [61] extend this idea to use a mixture of experts [25, 13, 60] as the motion prediction network. An additional gating network is used to predict the expert blending weights at run time. More closely related to our work is the Neural State Machine (NSM) [47], which extends the ideas of phase labels and expert networks to model human-scene interactions such as sit, carry, and open. While NSM is a powerful method, it does not generate variations in such interactions, which is one of our key contributions. Our experiments also demonstrate that NSM often fails to avoid intersections between the 3D character and objects in cluttered scenes (Sec. 5.2). Furthermore, training NSM requires time-consuming manual, and often ambiguous, labeling of phases for non-periodic actions. Starke et al. [48] propose a method to automatically extract local phase variables for each body part in the context of a two-player basketball game. Extending local phases to non-periodic actions is not trivial, however. We find that using scheduled sampling [5] provides an alternative to generate smooth transitions without phase labels. More recently, Wang et al. [52] introduce a hierarchical framework for synthesizing human-scene interactions. They generate sub-goal positions in the scene, predict the pose at each of these sub-goals, and synthesize the motion between such poses. This method requires a post-optimization framework to ensure smoothness and robust foot contact and to discourage penetration with the scene. Corona et al. [11] use a semantic graph to model human-object relationships followed by an RNN to predict human and object movements.

An alternative approach uses reinforcement learning (RL) to build a control policy that models interactions. Merel et al. [37] and Eom et al. [14] focus on ball catching from egocentric vision. Chao et al. [10] train sub-task controllers and a meta controller to execute the sub-tasks to complete a sitting task. However, in contrast to SAMP, their approach does not enable variations in the goal positions and directions. In addition, as with many RL-based approaches, generalizing the learned policies to new environments or actions is often challenging.

Motion Synthesis: Neural networks (feed-forward networks, LSTMs, or RNNs) have been extensively applied to the motion synthesis problem [15, 19, 24, 36, 49, 51, 1]. A typical approach predicts the future motion of a character based on previous frame(s). While showing impressive results when generating short sequences, many of these methods either converge to the mean pose or diverge when tested on long sequences. A common solution is to employ scheduled sampling [5] to ensure stable predictions at test time to generate long locomotion and dancing sequences [65, 32].

Several works have focused on modeling the stochastic nature of human motion, with a specific emphasis on trajectory prediction. Given the past trajectory of a character, they model multiple plausible future trajectories [6, 8, 17, 34, 40, 44, 4]. Recently, Cao et al. [6] sample multiple future goals and then use them to generate different future skeletal motions. This is similar in spirit to our use of GoalNet. The difference is that our goal is to predict various trajectories that always lead to the same target object (instead of predicting any plausible future trajectory).

Modeling the stochasticity of the full human motion is a less explored area [58, 59, 54]. Motion VAE [32] predicts a distribution of the next poses instead of one pose using the latent space of a conditional variational auto-encoder. MoGlow is a controllable probabilistic generative model based on normalizing flows [22]. Generating diverse dance motions from music has also been recently explored [30, 31]. Xu et al. [57] generate diverse motions by blending short sequences from a database. To the best of our knowledge, no previous work has tackled the problem of generating diverse human-scene interactions.

Figure 2: Our system consists of three main components. GoalNet predicts oriented goal locations (green sphere and blue arrow on the chair) given an interaction object. The Path Planning Module predicts an obstacle-free path from the starting position to the goal. MotionNet sequentially predicts the next character state until the desired action is executed.

3 Method

Generating dynamic human scene interactions in cluttered environments requires solutions to several sub-problems. First and foremost, the synthesized motion of the character should be realistic and capture natural variations. Given a target object, it is important to sample plausible contact points and orientations for performing a specific action (e.g., where to sit on a chair and which direction to face). Finally, the motion needs to be synthesized such that it navigates to the goal location while avoiding penetrating objects in the scene. Our system consists of three main components that address each of these sub-problems: a MotionNet, GoalNet, and a Path Planning Module. At the core of our method is the MotionNet which predicts the pose of the character based on the previous pose as well as other factors such as the interaction object geometry and the target goal position and orientation. GoalNet predicts the goal position and orientation for the interaction on the desired object. The Path Planning Module computes an obstacle-free path between the starting location of the character and the goal location. The full pipeline is illustrated in Fig. 2.

3.1 MotionNet

MotionNet is an autoregressive conditional variational autoencoder (cVAE) [12, 46] that generates the pose of the character conditioned on its previous state (e.g., pose, trajectory, goal) as well as the geometry of the interaction object. MotionNet has two components: an encoder and a decoder. The encoder encodes the previous and current states of the character and the interaction object to a latent vector $𝒁$. The decoder takes this latent vector, the character’s previous state, and the interaction object to predict the character’s next state. The pipeline is shown in Fig. 3. Note that, at test time, we only utilize the decoder of MotionNet and sample $𝒁$ from a standard normal distribution.

Figure 3: MotionNet consists of an encoder and a decoder. The encoder consists of two sub-encoders: State Encoder and Interaction Encoder. The decoder consists of a Prediction Network to predict the next character state and a gating network that predicts the blending weights of the Prediction Network. See Sec. 3.1.

Encoder: The encoder consists of two sub-encoders: State Encoder and Interaction Encoder. The State Encoder encodes the previous and current state of the character into a low-dimensional vector. Similarly, the Interaction Encoder encodes the object geometry into a different low-dimensional vector. Next, the two vectors are concatenated and passed through two identical fully connected layers to predict the mean $\mu$ and standard deviation $\sigma$ of a Gaussian distribution representing a latent embedding space. We then sample a random latent code $𝒁$, which is provided to the decoder when predicting the next state of the character.

State Representation: We use a representation similar to Starke et al. [47] to encode the state of the character. Specifically, the state at frame $i$ is defined as ${𝑿}_i=$

$$\{{𝒋}_i^p,{𝒋}_i^r,{𝒋}_i^v,{\widetilde{𝒋}}_i^p,{𝒕}_i^p,{𝒕}_i^d,{\widetilde{𝒕}}_i^p,{\widetilde{𝒕}}_i^d,{𝒕}_i^a,{𝒈}_i^p,{𝒈}_i^d,{𝒈}_i^a,{𝒄}_i\},$$ (1)

where ${𝒋}_i^p\in {ℝ}^{3j},{𝒋}_i^r\in {ℝ}^{6j},{𝒋}_i^v\in {ℝ}^{3j}$ are the position, rotation, and velocity of each joint relative to the root. $j$ is the number of joints in the skeleton which is $22$ in our data. ${\widetilde{𝒋}}_i^p\in {ℝ}^{3j}$ are the joint positions relative to future root 1 second ahead. ${𝒕}_i^p\in {ℝ}^{2t}$ and ${𝒕}_i^d\in {ℝ}^{2t}$ are the root positions and forward directions relative to the root of frame $i-1$. ${\widetilde{𝒕}}_i^p\in {ℝ}^{2t}$ and ${\widetilde{𝒕}}_i^d\in {ℝ}^{2t}$ are the root positions and forward directions relative to the goal of frame $i-1$. We define these inputs for $t$ time steps sampled uniformly in a $2$ second window between $[-1,1]$ seconds. ${𝒕}_i^a\in {ℝ}^{n_{a}t}$ is a vector of continuous action labels on each of the $t$ samples. In our experiments, $n_a$ is $5$, which is the total number of actions we model (i.e., idle, walk, run, sit, lie down). ${𝒈}_i^p\in {ℝ}^{3t},{𝒈}_i^d\in {ℝ}^{3t}$ are the goal positions and directions, and ${𝒈}_i^a\in {ℝ}^{n_{a}t}$ is a one-hot action label describing the action to be performed at each of the $t$ samples. ${𝒄}_i\in {ℝ}^5$ are contact labels for pelvis, feet, and hands.

State Encoder: The State Encoder takes the current ${𝑿}_i$ and previous state ${𝑿}_{i-1}$ and encodes them into a low-dimensional vector using three fully connected layers.

Interaction Encoder: The Interaction Encoder takes a voxel representation of the interaction object $𝑰$ and encodes it into a low-dimensional vector. We use a voxel grid of size $8\times 8\times 8$. Each voxel stores a $4-$dimensional vector. The first three components refer to the position of the voxel center relative to the root of the character. The fourth element stores the real-valued occupancy (between 0 and 1) of the voxel. The architecture consists of three fully connected layers.

Decoder: The decoder takes the random latent code $𝒁$, the interaction object representation $𝑰$, and the previous state ${𝑿}_{i-1}$, and predicts the next state ${\widehat{𝑿}}_i$. Similar to recent work [32, 47], our decoder is built as a mixture-of-experts with two components: the Prediction Network and Gating Network.

The Prediction Network is responsible for predicting the next state ${\widehat{𝑿}}_i$. The weights of the Prediction Network $𝜶$ are computed by blending $K$ expert weights:

$$𝜶=\sum _{i=1}^K{\omega }_i{𝜶}_i,$$ (2)

where the blending weights ${\omega }_i$ are predicted by the Gating Network. Each expert is a three-layer fully connected network. The Gating Network is also a three-layer fully connected network, which takes as input $𝒁$ and ${𝑿}_{i-1}$.

MotionNet is trained end-to-end to minimize the loss ${ℒ}_{\text{motion}}=$

$$||{\widehat{𝑿}}_i-{𝑿}_i|{|}_2^2+{\beta }_{\text{1}}\mathrm{𝐾𝐿}(Q(𝒁|{𝑿}_i,{𝑿}_{i-1},𝑰)||p(𝒁)),$$ (3)

where the first term minimizes the difference between the ground truth and predicted states of the character and $\mathrm{𝐾𝐿}$ denotes the Kullback-Leibler divergence.

3.2 GoalNet

Given a target interaction object (which can be interactively defined by a user at test time or randomly sampled among the objects in the scene), the character is driven by the goal position ${𝒈}^p\in {ℝ}^3$ and direction ${𝒈}^d\in {ℝ}^3$ sampled on the object’s surface. In order to perform realistic interactions; the character requires the ability to predict these goal positions and directions from the object geometry. For example, while a regular chair allows variation in terms of sitting direction, the direction of sitting on an armchair is restricted (see Fig. 7). We use GoalNet to model object-specific goal positions and directions. GoalNet is a conditional variational autoencoder (cVAE) that predicts plausible goal positions and directions given the voxel representation of the target interaction object $𝑰$ as shown in Fig. 4. The encoder encodes the interaction object $𝑰$, goal position ${𝒈}^p$, and direction ${𝒈}^d$, into a latent code ${𝒁}_{goal}$. The decoder reconstructs the goal position ${\widehat{𝒈}}^p$, and direction ${\widehat{𝒈}}^d$ from ${𝒁}_{goal}$ and $𝑰$. We represent the object using a voxel representation similar to the one used in MotionNet (Sec. 3.1). The only difference is that we compute the voxel position relative to the object center instead of the character root. In the encoder, we use an Interaction Encoder similar to the one used in MotionNet (see Sec. 3.1) to encode the object representation $𝑰$ to a low dimension vector. This vector is then concatenated with ${𝒈}^p$ and ${𝒈}^d$ and encoded further to the latent vector ${𝒁}_{goal}$. The decoder has the same architecture as the encoder as shown in Fig. 4. The network is trained to minimize the loss:

${ℒ}_{\text{goal}}=$ ${\Vert {\widehat{𝒈}}^p-{𝒈}^p\Vert }_2^2+{\Vert {\widehat{𝒈}}^d-{𝒈}^d\Vert }_2^2$

$+{\beta }_2\mathrm{𝐾𝐿}(Q({𝒁}_{goal}|{𝒈}^p,{𝒈}^d,𝑰)||p({𝒁}_{goal})).$ (4)

At test time, given a target object $𝑰$, we randomly sample ${𝒁}_{goal}\sim 𝒩(0,I)$ and use the decoder to generate various goal positions ${𝒈}^p$ and directions ${𝒈}^d$.

Figure 4: GoalNet generates multiple valid goal positions ${\widehat{𝒈}}^p$ and directions ${\widehat{𝒈}}^d$ given an object representation $𝑰$. FC(N) denotes a fully connected layer of size N.

3.3 Path Planning

To ensure the character can navigate inside cluttered environments while avoiding obstacles, we employ an explicit A* path planning algorithm [20]. Given the desired goal location, we use A* to compute an obstacle-free path from the starting position of the character to the goal. The path is defined as a series of waypoints $w_i=\{w_0,w_1,w_2,\mathrm{\dots }\}$ that define the locations where the path changes direction. We break the task of performing the final desired action into sub-tasks in which each sub-task requires the character to walk to the next waypoint. The final sub-task requires the character to perform the desired action at the final waypoint.

3.4 Training Strategy

Training MotionNet using standard supervised training produces poor quality predictions at run time (see Sup. Mat.). This is due to the accumulation of error at run time when the output of the network is fed back as input in the next step. To account for this, we train the network using scheduled sampling [5], which has been shown to result in long stable motion predictions [32]. During training, the current network prediction is used as input in the next training step with a probability $1-P$. $P$ is (see Sup. Mat.):

(5)

4 Data Preparation

4.1 Motion Data

To model variations in human-scene interactions, we capture a new dataset using an optical MoCap system with $54$ Vicon cameras. We place seven different objects in the center of the MoCap area, namely two sofas, an armchair, a chair, a high bar chair, a low chair and a table. We record multiple clips of each interaction with different styles. In each sequence, the subject starts from an A-Pose in a random location in the MoCap space, walks towards the object, and performs the action for $20-40$ seconds. Finally, the subject gets up from the object and walks away. Our goal is to capture various styles of performing the same action, thus we ask the subject to change the style in each sequence. In addition to the subject, we also capture the object pose using attached markers. We also have the CAD model for each object. Finally, we capture running, walking, and idle sequences where the subject walks and runs in different directions with different speeds and stands in an idle state. Our dataset consists of $\sim 100$ minutes of motion data recorded at $30$ fps from a single subject, resulting in $\sim 185$K frames. We use MoSh++ [33] to fit the SMPL-X [39] body model to the optical markers. More details about the data are available in the Sup. Mat.

4.2 Motion Data Augmentation

With only seven captured objects, MotionNet will fail to adapt to new unseen objects. Capturing MoCap with a wide range of objects requires a significant amount of effort and time. We address this issue by augmenting our data using an efficient augmentation pipeline similar to [3, 47]. Since we capture both the body motion as well as the object pose, we compute the contact between the body and the object. We detect the contacts of five key joints of the character skeleton. Namely, pelvis, hands, and feet. We then augment our data by randomly switching or scaling the object at each frame. When switching, we replace the original object with a random object of a similar size selected from ShapeNet [9]. For each new object (scaled or switched), we project the contacts detected from the ground truth data to the new object. Finally, we use an IK solver to recompute the full pose such that the contacts are maintained. Please refer to the Sup. Mat. for more details.

4.3 Goal Data

To train GoalNet, we label various goal positions ${𝒈}^p$ and directions ${𝒈}^d$ for different objects from ShapeNet [9]. These goals represent the position on the object surface where a character could sit and the forward direction of the character when sitting. We select $5$ categories from ShapeNet namely, sofas, L-shaped sofas, chairs, armchairs, and tables. From each category, we select $15-20$ instances and we manually label $1-5$ goals for each instance. The number of goals labeled per instance depends on how many different goals an object can afford. For example, an L-shaped sofa offers more places to sit than a chair. In total, we use $80$ objects as our training data. We augment our data by randomly scaling the objects across the $xyz$ axes leading to $\sim 13$K training samples.

5 Experiments & Evaluation

5.1 Qualitative Evaluation

In this section, we provide qualitative results and discuss the main points. We refer to the Sup. Mat. and the accompanying video for more results.

Generating Diverse Motion: In contrast to previous deterministic methods [47], SAMP generates a wide range of diverse styles of an action while ensuring realism. Several different sitting and lying down styles generated by SAMP are shown in Fig. 5. The use of the Interaction Encoder 3.1 and the data augmentation (Sec. 4.2) further ensures SAMP can adapt to different objects with varying geometry. Notice how the character naturally leans its head back on the sofa. The style of the action is also conditioned on the interacting object. The character lifts its legs when sitting on a high chair/table but extends its legs when sitting on a very low table. We observe that lying down is a harder task and several of baseline methods fail to execute this task (see Sec. 5.2). While SAMP synthesizes reasonable sequences, our results are not always perfect. The generated motion might involve some penetration with the object.

Figure 5: SAMP generates plausible and diverse action styles and adapts to different object geometries.

Goal Generation: When presented with a new object, the character needs to predict where and in which direction the action should be executed. In [47], the goal is computed as the object center. However, this heuristic fails for objects with complex geometries. In Fig. 6 we show that using the object center results in invalid actions whereas GoalNet allows our method to reason about where the action should be executed. As shown in Fig. 7, by sampling different latent codes ${𝒁}_{goal}$, GoalNet generates multiple goal positions and directions for various objects. Notice how GoalNet captures that, while a person can sit sideways on a regular chair, this is not valid for an armchair.

Figure 6: Without GoalNet (left), SAMP fails to sit on a valid place. SAMP with GoalNet is shown on the right.

Figure 7: GoalNet generates diverse valid goals on different objects. Spheres indicate goal positions, and blue arrows indicate goal directions.

Figure 8 shows how the different goals generated by GoalNet guide the motion of the character. Starting from the same position, direction, and initial pose, the virtual human follows two different paths to reach different goal positions when performing the “sit on the couch” action. The final pose of the character is also different in the two cases due to the stochastic nature of MotionNet.

Figure 8: Goals generated by GoalNet (mesh spheres) are used by MotionNet to guide the motion of virtual characters.

Path Planning: When navigating to a particular goal location in a cluttered scene, it is critical to avoid obstacles. Our Path Planning Module achieves this goal by predicting the shortest obstacle-free path between the starting character position and the goal using a navigation mesh computed based on the 3D scene. The navigation mesh defines the walk-able areas in the scene and is computed once offline. In Fig. 9, we show an example path computed by the Path Planning Module. Without this module, the character often walks through objects in the scene. We observe a similar behaviour in the previous work of NSM [47], even though NSM uses a volumetric representation of the environment to help the character navigate.

Figure 9: Our Path Planning Module helps SAMP to successfully navigate cluttered scenes (left). NSM [47] fails in such scenes (right).

5.2 Quantitative Evaluation

Deterministic vs. Stochastic: To quantify the diversity of the generated motion, we put the character in a fixed starting position and direction and we run our method ten times with the same goal. For example, we instruct the character to sit/lie down on the same object multiple times starting from the same initial state/position/direction. For walking and running, we instruct the character to run in each of the four directions for $15$ seconds. We record the character motion for each run and then compute the Average Pairwise Distance (APD) [58, 63] as shown in Table. 1. The APD is defined as:

$$APD=\frac{1}{N(N-1)}\sum _{i=0}^N\sum _{\begin{array}{c}j=0\\ j\ne i\end{array}}^N{\Vert {𝑿}_i^{\prime }-{𝑿}_j^{\prime }\Vert }_2^2.$$ (6)

${𝑿}_i^{\prime }$ represents the character’s local pose features at frame $i$. ${𝑿}_i^{\prime }=\{{𝒋}_i^p,{𝒋}_i^r,{𝒋}_i^v\}$. $N$ is the total number of frames for all sequences. For comparison we also report the APD for the ground truth (GT) data in Table. 1.

	Walk	Run	Sit	Liedown
GT	$5.95$	$7.74$	$5.18$	$7.52$
SAMP	$5.63$	$5.75$	$5.05$	$6.69$

Table 1: Diversity metric. Higher values indicate more diversity.

GoalNet: Given $150$ unseen goals sampled on test objects, we measure the average position and orientation reconstruction error of GoalNet to be $\mathbf{6.04}$ cm and $\mathbf{2.29}$ deg (we note that the objects have real-life measurements). To measure the diversity of the generated goals, we compute the Average Pairwise Distance (APD) among the generated goal positions $g^p$ and directions $g^d$:

APD-Pos $={\displaystyle \frac{1}{LN(N-1)}}{\displaystyle \sum _{k=0}^L}{\displaystyle \sum _{i=0}^N}{\displaystyle \sum _{\begin{array}{c}j=0\\ j\ne i\end{array}}^N}|g_i^p-g_j^p|$ (7)

APD-Rot $={\displaystyle \frac{1}{LN(N-1)}}{\displaystyle \sum _{k=0}^L}{\displaystyle \sum _{i=0}^N}{\displaystyle \sum _{\begin{array}{c}j=0\\ j\ne i\end{array}}^N}\arccos (g_i^d.g_j^d).$ (8)

$L=150$ is the number of objects and $N=10$ is the number of goals generated for each object. We find APD-Pos and APD-Rot for our generated goals to be $\mathbf{16.42}$ cm and $\mathbf{41.27}$ deg compared to $\mathbf{16.18}$ cm and $\mathbf{90.23}$ deg for the ground truth (GT) data.

Path Planning Module: To quantitatively evaluate the effectiveness of our Path Planning Module, we test our method in a cluttered scene. We put the character in a random initial position and orientation and select a random goal. We repeat this $10$ times. We find the percentage of frames where a penetration happens is $\mathbf{3.8}\mathbf{\%}$, $\mathbf{11.2}\mathbf{\%}$, and $\mathbf{8.11}\mathbf{\%}$ for SAMP with Path Planning Module, without Path Planning Module, and NSM [47], respectively. While NSM uses a volumetric sensor to detect collisions with the environment, it is not as effective as explicit path planning.

Comparison to Previous Models: We compare our model to baselines by measuring three metrics: average execution time, average precision, and Frèchet distance (FD) between the distribution of the generated motion and ground truth. Execution time is the time required to transition to the target action label from an idle state. Precision is the positional (PE) and rotational (RE) error at the goal. We measure FD on a subset of the state features which we call $\widetilde{𝑿}$:

$$\widetilde{𝑿}=\{{𝒋}^p,{𝒋}^r,{𝒋}^v,{\widetilde{𝒕}}^p,{\widetilde{𝒕}}^d\}.$$ (9)

As our baselines, we choose a feedforward network (MLP) as the motion prediction network, Mixture of Experts (MoE) [61], and NSM [47] (see Sup. Mat. for details).

SAMP vs. MLP vs. MoE: We re-trained the MLP and MoE using the same training strategy and data we used for SAMP. Both MLP and MoE take a longer time to execute the task and often fail to execute the “lie down” action (denoted $\mathrm{\infty }$) as evidenced by the execution time in Table. 2 and precision in Table. 3. These architectures sometimes generate implausible poses as shown in Sup. Mat., which is reflected by the lower FD in Table. 4

	MLP	MoE	SAMP	GT
Sit	$13.06$	$12.99$	$\mathbf{12.53}$	$11.7$
Liedown	$\mathrm{\infty }$	$\mathrm{\infty }$	$\mathbf{17.06}$	$15.49$

Table 2: Average execution Time in seconds. $\mathrm{\infty }$ means the method failed to reach the goal within 3 minutes.

Method	Sit		Liedown
	PE(cm)	RE(deg)	PE(cm)	RE(deg)
MLP	$9.27$	$3.99$	$\mathrm{\infty }$	$\mathrm{\infty }$
MoE	$7.99$	$5.73$	$\mathrm{\infty }$	$\mathrm{\infty }$
SAMP	$\mathbf{6.09}$	$\mathbf{3.55}$	$\mathbf{5.76}$	$\mathbf{6.45}$

Table 3: Average precision in terms of positional and rotational errors (PE and RE). $\mathrm{\infty }$ means the method failed to reach the goal within 3 minutes.

	Idle	Walk	Run	Sit	Liedown
MLP	$102.85$	$121.18$	$150.56$	$105.87$	$36.85$
MoE	$102.91$	$114.17$	$151.14$	$105.10$	$35.79$
SAMP	$\mathbf{102.72}$	$\mathbf{111.09}$	$\mathbf{141.11}$	$\mathbf{104.68}$	$\mathbf{17.30}$

Table 4: Frèchet distance.

SAMP vs. NSM: For NSM, we used the publicly available pre-trained model since retraining NSM on our data is infeasible due to the missing phase labels. We trained SAMP on the same data on which NSM was trained. In Table 5 we observe that our model is on par with NSM in terms of achieving goals without the need for phase labels, which are cumbersome and often ambiguous to annotate. In addition, our main focus is to model diverse motions via a stochastic model while NSM is deterministic. Our Path Planning Module module helps SAMP to safely navigate complex scenes where NSM fails as shown by the penetration amounts.

For all evaluations, all test objects are randomly selected from ShapeNet and none is part of our training set.

Metric	Sit		Carry
	SAMP	NSM	SAMP	NSM
Precision PE (cm) $\downarrow$	$\mathbf{15.97}$	$16.95$	$\mathbf{4.58}$	$4.72$
Precision RE (deg) $\downarrow$	$5.38$	$\mathbf{2.32}$	$1.78$	$\mathbf{1.65}$
Execution Time (sec) $\downarrow$	$12.93$	$\mathbf{10.26}$	$13.29$	$\mathbf{12.82}$
FD $\downarrow$	$6.20$	$\mathbf{4.21}$	$10.17$	$\mathbf{7.31}$
Diversity $\uparrow$	$\mathbf{0.44}$	$0.0$	$\mathbf{0.26}$	$0.0$
Penetration ($\%$) $\downarrow$	$\mathbf{3.8}$	$8.11$	$\mathbf{3.62}$	$8.45$

Table 5: SAMP vs. NSM.

Limitations and Future Work: We observe that sometimes slight penetrations between the character and the interacting object can occur. A potential solution is to incorporate a post-processing step to optimize the pose of the character to avoid such intersections. In order to generalize SAMP to interacting objects that have significantly different geometry than those seen in training, in future work, we would like to explore methods to encode local object geometries.

6 Conclusion

Here we have described SAMP, which makes several important steps toward creating lifelike avatars that move and act like real people in previously unseen and complex environments. Critically, we introduce three elements that must be part of a solution. First, characters must be able to navigate the world and avoid obstacles. For this, we use an existing path planning method. Second, characters can interact with objects in different ways. To address this, we train GoalNet to take an object and stochastically produce an interaction location and direction. Third, the character should produce motions achieving the goal that vary naturally. To that end, we train a novel MotionNet that incrementally generates body poses based on the past motion and the goal. We train SAMP using a novel dataset of motion capture data involving human-object interaction.

Acknowledgement This work was initiated while MH was an intern at Adobe. We are grateful to Sebastian Starke for inspiring work, helpful discussion, and making his code open-source. We thank Joachim Tesch for feedback on Unity and rendering, Nima Ghorbani for MoSH++, and Meshcapade for the character texture. For helping with the data collection, we are grateful to Tsvetelina Alexiadis, Galina Henz, Markus Höschle and Tobias Bauch.

Disclosure: MJB has received research funds from Adobe, Intel, Nvidia, Facebook, and Amazon. While MJB is a part-time employee of Amazon, his research was performed solely at, and funded solely by, Max Planck. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH.

Appendix A Data Preparation

A.1 Motion Data

Fig. S.1 shows examples of different sitting and lying down styles from our MoCap. A breakdown of the dataset in terms of different actions is shown in Table S.1. The objects used during the MoCap are shown in Fig. S.2.

Figure S.1: Examples of action styles in our motion capture data.

Figure S.2: Objects used during motion capture.

Labels	Minutes	Percentage %
Idle	$18.3$	$17.7$
Walk	$42.3$	$41.0$
Run	$5.1$	$4.9$
Sit	$27.3$	$26.4$
Lie down	$10.1$	$9.7$
Total	$103.3$

Table S.1: Motion capture data breakdown with respect to actions.

A.2 Goal Data

We select $5$ categories from ShapeNet namely, sofas, L-shaped sofas, chairs, armchairs, and tables. From each category, we select $15-20$ instances and we manually label $1-5$ goals for each instance. Table. S.2 shows the number of instances for each category. We manually label $1-5$ goals for each instance. The number of goals labelled per instance depends on how many different goals an object can afford. For example, we label $5$ different goals for the L-shaped sofa compared to $3$ for the chair as shown in Fig. S.3.

Figure S.3: Goal Labelling.

Category	Number of Objects
Armchairs	$15$
Chairs	$16$
Sofa	$20$
L-Sofa	$18$
Tables	$18$
Total	$87$

Table S.2: GoalNet data breakdown with respect to object categories.

Appendix B Training Details

B.1 MotionNet

The character state $𝑿$ is of size $647$. The State Encoder, Interaction Encoder, Gating Network, and Prediction Network are all three-layer fully connected networks with rectified linear function ELU. The dimensions of each network are in Table S.3. The encoder latent code $𝒁$ is of size $64$ and we set the number of experts $K$ to $12$. We use a learning rate of $5e-5$ and train our network for $100$ epochs. We use the Adam optimizer with linear weight decay. The weight of the Kullback-Leibler divergence ${\beta }_{\text{1}}$ is $0.1$.

Network	Architecture
State Encoder	$\{512,256,256\}$
Interaction Encoder	$\{256,256,256\}$
Gating Network	$\{512,256,12\}$
Prediction Network	$\{512,512,647\}$

Table S.3: Architecture details. All networks are all three-layer fully connected networks with ELU.

B.2 GoalNet

The Interaction Encoder of GoalNet is a three-layer fully connected network of shape $\{512,512,64\}$. The latent vector ${𝒁}_{goal}$ is of size $3$. The weight of the Kullback-Leibler divergence ${\beta }_2$ is $0.5$. We use the Adam optimizer with a learning rate of $1e-3$ and train GoalNet for $100$ epochs.

B.3 Schedule Sampling

For the schedule sampling training strategy, we set $C_1=30$ and $C_2=60$. We define a roll-out window of size $L$ where we set $L=60$ in our experiments. For each roll-out, we feed the ground truth first frame as input to the network and then sequentially predict the subsequent frames while using the scheduled sampling strategy. We divide our training data to equal-length clips of size $L$.

Appendix C Baselines

As our baselines, we choose a feedforward network (MLP) and a Mixture of Experts (MoE). The architecture of the MLP is shown in Fig. S.4. We use the same Interaction Encoder used for our MotionNet followed by four fully connected layers of size $512$. The architecture of the MoE is shown in Fig. S.5. The Interaction Encoder, Gating Network, and Prediction Network are all the same as the one used in MotionNet.

Figure S.4: MLP Architecture.

Figure S.5: MoE Architecture.

Appendix D Schedule Sampling

We found that using Schedule Sampling is essential to enable the character to successfully reach the goal and execute the action. Without it, we found the model to often diverge, get stuck, or take very long time to reach the goal as we show in Fig. S.6.

Figure S.6: SAMP With Schedule Sampling (Top) and without (bottom). The black line shows the root projection on the $xz$ plane. The blue and green circles denote the root at the first and last frame respectively. The red circle denotes the goal position. Note how SAMP fails to reach the goal without the use of Schedule Sampling.

Appendix E Path Planning Formulation

In order to use the Path Planning Module, we first compute the surface area where the character could stand or move. We call this the navigation mesh. This is computed from the character cylinder collider and the scene geometry. The navigation mesh is stored as convex polygons. To find a path between given start and end points, we first map these points to the closest polygons and then use A* to find the shortest path between the polygons .

Appendix F Data Augmentation Details

When the object is transformed, the contacts follow the same transformation. When the object is replaced by a new one, we project the original contact by finding the closest points on the surface of the new object. The new motion curve is computed by interpolation and the whole full body pose is computed using CCD IK solver. This does not guarantee smoothness but we found it to be stable in practice. More details are in [47].

Appendix G Interaction Encoder Ablation:

To quantify the importance of the Interaction Encoder, we trained SAMP without the Interaction Encoder. We found that the precision of reaching the goal deteriorates to $14.82$ cm and $3.65$ deg compared to $6.09$ cm and $3.55$ deg when the Interaction Encoder was used.

Appendix H Comparison to Cao et al.:

While relevant, the formulation of Cao [6] et al. is significantly different than our method making a direct comparison difficult. Given a target interaction object and action (e.g. “sit on the couch”), SAMP samples a goal location and orientation on the object, computes an obstacle-free path towards the object, and synthesizes diverse motion sequences that are of arbitrary length until the goal is executed. We assume that the character starts the action from an idle position without any knowledge of the past. In contrast, Cao et al. sample a goal location in the image space given a one-second-long history of motion. Based on this trajectory, a deterministic motion sequence of fixed length (two-seconds) is synthesized. The action executed in this trajectory is not controllable.

Appendix I Failure Cases

We observe that SAMP might not adapt well to objects with significantly different geometry than those seen in training as shown in Fig. S.7. Future work might explore different methods of encoding the object geometry.

Figure S.7: SAMP with significantly different geometry.