Generalist AI's GEN-1.5 Lets Robots Learn New Tasks from Seconds of Demo

Developed through more than eight months of continuous pretraining on large-scale physical interaction data, GEN-1.5 can immediately perform a task after watching a short demo, such as stacking a cup lid. It can also combine two separate demonstrations into a single continuous task, filling in unshown transition actions like repositioning, adjusting grip, or recovering from errors. In one example, the model watched a virtual demo on the left and then replicated it in the real world on the right, despite never having been trained on that specific task in either domain.

According to the company, these one-shot and in-context learning abilities were not explicitly programmed. Generalist AI said it did not design a special architecture for meta-learning or add an objective function to force such behavior. Instead, the capabilities emerged during pretraining on continuous physical data. The team observed that the number of gradient steps needed for new tasks shrank over time, from hundreds to dozens to ten, and eventually to zero.

In benchmark tests across ten tasks, GEN-1.5 achieved an average success rate of 59% with only one physical prompt and zero gradient updates. When given five additional minutes of data per task and ten gradient updates, the success rate rose to 83%. The team acknowledged that skills learned in-context are less stable than those from full fine-tuning, and that the current test tasks are mostly short-horizon and atomic operations.

The release has sparked comparisons to the GPT-3 moment for robotics. Just as GPT-3 could learn new language tasks from a few examples in its context window, GEN-1.5 appears to use recent demonstrations as context to guide its actions. The company believes that real-world physical data, which naturally contains repetition, continuation, and failure recovery, enables the model to predict next steps based on what just happened.