RAPID Framework Reaches 83% Real-World Generalization From One Demonstration, Team Reports at IROS 2026

The talk placed the work against two approaches to physical intelligence. One is data fitting, in which end-to-end learning maps visual observations directly to control actions. Leiphone reported that Actor Labs deployed Physical Intelligence's π0.5 on an excavator and trained policies with large-scale real data from industrial operators. Although a demonstration completed one task with 200 trajectories, the report said the case exposed low data efficiency and limited multi-task ability. If each single skill requires hundreds of trajectories or hundreds of hours of data, scaling such a data-stacking paradigm remains questionable. By contrast, humans can often infer the physical logic behind an action from a single demonstration and transfer it to objects of different sizes and materials, or to new tasks. End-to-end models frequently fail out of distribution, the report said.

The other approach treats intelligence as world modeling and planning. The aim is to combine machine learning with physical planning so an agent can learn physical regularities from small samples and generalize beyond its training distribution. To build a world model, the agent needs an internal representation of the physical world and must reason about future actions from that representation. Predicting future states at the pixel level is computationally expensive and difficult to use for real-time planning over long action sequences.

The team's answer is compositional abstraction. Instead of remembering all pixel details, the agent extracts abstract attributes such as shape, pose and material, and uses a state-transition graph to explicitly represent how actions change states. In the neuro-symbolic concept described in the talk, a neural layer uses networks for visual perception and extracts geometric and physical attributes, while a symbolic layer combines those attributes and defines transition rules between actions and states. The report gave the example of a graph in which s0 to s1 represents push(glass) and s1 to s3 represents flip(cap). With this representation, the robot can simulate state changes internally before acting and plan action sequences. This is the basis for learning from a single demonstration with strong generalization, according to the talk.

The next problem is turning high-level logical plans into executable physical actions. In traditional reinforcement learning and end-to-end paradigms, policy networks cannot be directly concatenated. If separate networks are trained for grasping, moving and placing, their observation spaces, action spaces and intermediate states are difficult to align, making long-horizon composition hard. The team's alternative is to define action generation as a constrained optimization problem. Given an initial state s0, the agent solves for a trajectory that minimizes action cost, such as trajectory length, while satisfying two kinds of constraints. Path constraints include dynamics, so the trajectory follows physical laws, and collision-free constraints, so the robot does not interfere with the environment. Subgoal constraints include holding-target, such as a stable grasp, and target-on-rack, such as placing an object precisely on a specified rack. The design separates learning into constraints and constraint values. The robot does not learn physical laws from scratch; known physical models and optimization solvers handle them, while neural networks focus on task-relevant constraints. In the MAGIC project's hanging task, traditional model planning took more than five minutes per solve because the search space was too large. A representative method based on unsupervised feature learning and policy learning, trained on 100,000 synthetic mugs, achieved only 42% one-shot generalization success. The team's framework used visual correspondence to guide constraint optimization and raised success to 93% from a single demonstration.

The RAPID architecture extends this idea to richer physical interactions. It learns contact-rich, non-prehensile operations from one human demonstration. The system turns skill learning into a closed programming loop involving a coding agent and a physical verifier. It parses the demonstration into basic action primitives such as push, flip, pivot rotation and grasp. A coding agent converts those primitives into object-centric constraint programs, removing dependence on specific geometric parameters so the programs can be reused and composed. A verifier then tests the programs in a Real2Sim simulation environment, returning feedback on collisions and physical feasibility to the coding agent for correction. This loop filters many physically infeasible programs before deployment. In eight complex non-prehensile real-world tasks covering scenes such as bookshelves and partitions, and flexibly combining primitives including push, flip, press-slide, pivot, topple and grasp, RAPID showed robust and generalizable solving from a single human demonstration.

The talk also compared modular systems with end-to-end models. RAPID is modular: its capabilities come from collaboration between the coding agent and different functional modules. Composition appears not only in motor skills but also in how a system organizes and calls different capability modules. Compared with a fully end-to-end neural network, a modular system can introduce explicit structure that is easier to explain, verify and replace, but it may be limited by manually defined divisions and interfaces. An end-to-end model can jointly learn perception-to-action mappings, but its intermediate process is harder to inspect and intervene in independently. The team's earlier Retriever framework split perception, reasoning and control into separate modules and explicitly modeled triggers and buffers. It addressed closed-loop control under asynchronous frequencies, including a wrist camera at 30Hz, robot state at 60Hz and low-level skills at 2Hz, and was validated on long-horizon dual-arm tasks such as seasoning a steak. The report said limited information flow between modules still makes it difficult to fully absorb cross-task synergies.

The team then tested whether composition can emerge automatically in end-to-end multimodal models that jointly train video prediction and action generation. In a first attempt, the AVDC framework, published at ICLR 2024 as a spotlight paper, predicted future video from language instructions and current RGB-D observations, estimated dense motion from the predicted frames, combined it with depth for SE(3) pose estimation, and mapped the resulting 3D object trajectory to Cartesian control commands for the robot end effector. The approach did not require explicit action labels and could pretrain on video data without action labels, and because it extracted object motion rather than arm joint positions, it had cross-embodiment generalization. Real-robot tests showed the limits of this decoupled pipeline. It mainly supported basic grasping, moving and placing. For contact-rich operations such as flipping, pressing and pivot rotation, pixel-level video prediction struggled to capture physical forces and precise contact points. The report said the video-generation-to-action-translation pipeline did not automatically produce complex compositional behavior.

The second attempt built a joint video-action model. The team used a pretrained video prediction model, Cosmos Predict 2.5-2B, based on a Flow Matching architecture, as the backbone. During video generation denoising, the model extracted latent features at an intermediate noise level σ_v, passed them through a feature projection layer, and sent them together with robot proprioceptive state and noisy action tokens into a Gemma Action Head, which jointly generated denoised action control tokens through Flow Matching. Experiments in simulation and on real robots gave an unexpected result: a unified video-action model did not automatically produce compositional generalization. The team observed two difficult problems; the report described the first as visual generalization with execution failure.