Closed-loop whole-body motion planning from partial depth and LiDAR observations.
Mobile manipulators operating in homes and other complex environments must coordinate their base and arm over long horizons, navigating around obstacles, through narrow passages, and between constrained workspaces. Neural motion planners can provide fast closed-loop planning from partial observations, but scaling them to room-sized environments is difficult because generating high-quality training trajectories is itself a challenging motion-planning problem. To address this, we introduce MoMa-DemoSmith, a demonstration-generation pipeline that separates the problem into global path discovery for large complex scenes and local motion refinement for collision avoidance. This key separation enables the generation of millions of challenging whole-body trajectories across our procedurally generated scenes. These demonstrations are used to train MoMa-NMP in two stages, first through behavior cloning (BC), followed by privileged on-policy distillation, producing a closed-loop policy capable of robust whole-body planning across complex, partially observed environments.
We evaluate MoMa-NMP on 20 motion-planning tasks across 10 unseen real-world scenes. The policy replans at 12 Hz from onboard depth-camera and LiDAR observations.
All real-world videos are shown at 1× speed.
MoMa-NMP plans in real-world scenes with objects and arrangements absent from the procedurally generated training environments.
The robot coordinates its base and arm to move through narrow passages and reach poses in constrained workspaces.
The policy recomputes base-and-arm actions at 12 Hz using updated depth-camera and LiDAR observations during execution.
The robot navigates around partially observed obstacles and approaches workspaces that are initially occluded from its onboard sensors.
| Method | Trials | Success |
|---|---|---|
| MoMa-NMP | 62 / 80 | 77.5% |
| cuRobo | 23 / 80 | 28.8% |
| RRTConnect | 17 / 80 | 21.3% |
80 trials across 20 tasks in 10 unseen scenes, with four trials per task. MoMa-NMP replans closed-loop from onboard observations. cuRobo and RRTConnect plan once from collision geometry constructed from the initial observation.
MoMa-DemoSmith
MoMa-DemoSmith separates global path discovery from local action refinement to generate 7 million whole-body training trajectories across procedurally generated scenes.
Scenes contain one to seven tables with randomized assets and obstacles. Start and goal end-effector poses are sampled in free space and inside compartments.
cuRobo uses relaxed collision margins and mobile-base motion limits to discover global whole-body routes through the scene.
At each cuRobo waypoint, Geometric Fabrics tracks the next five base-and-arm configurations to produce higher-clearance action targets for imitation learning.
These videos show the global trajectories before Geometric Fabrics action annotation.
MoMa-NMP
MoMa-NMP takes depth and LiDAR point clouds, the robot state, and an end-effector goal pose as input, and predicts five steps of coordinated base-and-arm motion. We first train it by behavior cloning on MoMa-DemoSmith demonstrations. During on-policy fine-tuning, Geometric Fabrics uses ground-truth scene geometry to correct the frozen behavior-cloned policy’s actions and supervise the student. At deployment, the learned policy replans from onboard observations without access to ground-truth scene geometry.
We reproduce partial observability in simulation with GPU-based sensor rendering. Points sampled from scene and robot geometry are projected through depth-camera and LiDAR renderers, which retain only the nearest return in each image pixel or angular bin. Occluded geometry is removed before the policy observes the scene.
Depth-camera and LiDAR returns are combined into a single sensed-scene point cloud. A subsample of the known robot geometry is then appended, and a binary per-point feature distinguishes sensed-scene points from robot points before PointNet++ encoding.
We evaluate on approximately 10,000 held-out start–goal problems drawn from successfully generated MoMa-DemoSmith demonstrations.
| Method | Success |
|---|---|
| MoMa-NMP | 87.0% |
| MoMa-NMP (BC only) | 64.8% |
| RRTConnect | 85.3% |
| cuRobo | 46.9% |
| Geometric Fabrics | 17.2% |
Success requires reaching within 1 cm and 15° of the end-effector goal without a prior robot–environment collision. Robot link–link self-collision is not scored in this benchmark. MoMa-NMP uses partial observations. Classical baselines receive oracle geometry; cuRobo and RRTConnect also receive eight precomputed collision-free goal configurations, relaxed mobile-base motion limits, and planning budgets of 3 s and 60 s, respectively.
Policy evaluation on held-out simulation problems.
| Method | Collision-free goal success (%) | Goal reached (%) |
|---|---|---|
| MoMa-NMP | 75.98 | 87.32 |
| M²Diffuser | 23.23 | 27.83 |
MoMa-NMP is evaluated without retraining or fine-tuning on M²Diffuser scenes or demonstrations. Results cover 891 problems from the official seen-scene goal-reaching test split with collision-free start and goal inverse-kinematics solutions for our robot. Both methods use the same scene geometries and end-effector start and goal poses. Goal reaching requires position error within 4 cm and orientation error within 20°; collision-free goal success also requires no prior collision.
Fabrics annotations improve behavior-cloning success from 47.0% to 64.8% compared with raw cuRobo actions.
Compared with the policy trained with behavior cloning (BC) only, privileged on-policy distillation improves simulation success from 64.8% to 87.0%. On compartment-to-compartment tasks, success increases from 56.6% with BC only to 84.4% after distillation.
Removing the base-frame and end-effector-frame delta-pose inputs reduces behavior-cloning success from 64.8% to 27.5% and increases collision-before-success from 34.1% to 56.9%.
Compared with training on complete scene point clouds, training on rendered depth and LiDAR improves real-world success from 54.5% to 79.5% on a separate set of 44 matched trials per policy. Both policies receive the same real depth-camera and LiDAR observations at deployment.
Like many neural methods, MoMa-NMP struggles in scenes that diverge significantly from the training distribution. For example, it often collides with chairs, which are not well represented by either the pillars or tables in our training set.
Although MoMa-NMP demonstrates an ability to plan around partially occluded obstacles, it often collides with nearby small obstacles that are fully occluded from both the LiDAR and depth cameras.