Tutorial Outlines Streaming Robotics Pipeline for NVIDIA Cosmos3-DROID

The tutorial first introspects the LeRobotDataset v3.0 structure and builds a metadata graph from info.json, task metadata, episode tables and dataset statistics. Using the Hugging Face Hub API and file system, it lists repository files and counts entries under success/data, success/videos, success/meta, failure/data, failure/videos and failure/meta. It then sorts Parquet data shards, video shards for observation.image.wrist_image_left and metadata files under the success root, printing the first data shard and first video shard.

From info.json, the code reads total episodes, total frames, total tasks and frames per second, along with data_path and video_path templates. It extracts feature definitions for observation.state, action and video keys and prints their shapes. It loads tasks.parquet, maps task strings to indices and prints a random sample of eight task strings. It also concatenates episode tables from the first four metadata Parquet files, prints the number of rows, lists non-statistics columns and displays selected columns such as episode_index, length, data/chunk_index and data/file_index when present.

To avoid downloading the full repository, the tutorial uses HTTP byte-range access with PyArrow to selectively read Parquet row groups and columns. It converts individual episodes into state-action trajectories and analyzes joint motion, gripper events, Cartesian end-effector paths and action-frequency spectra. Only required AV1 video windows are decoded through seek-based PyAV/FFmpeg access. Observations and actions are normalized using dataset statistics.

The pipeline then constructs an ACT-style chunked PyTorch dataset with optional visual conditioning and trains a multimodal behavior-cloning policy. Constants include REPO_ID nvidia/Cosmos3-DROID, root success, video key observation.image.wrist_image_left, 15 fps, 48 episodes, horizon 8, observation history 2, vision enabled, 6 visual episodes, visual size 96, 12 epochs, batch 256 and seed 0. The script sets random seeds for Python, NumPy and PyTorch and selects CUDA if available, otherwise CPU.

It installs huggingface_hub, pyarrow, av, pandas, matplotlib and tqdm, then imports NumPy, pandas, PyArrow, Matplotlib, PyTorch and Hugging Face Hub utilities including HfApi, HfFileSystem, hf_hub_download and hf_hub_url. If an HF_TOKEN environment variable is present, it logs in to Hugging Face.

Evaluation uses open-loop rollout with temporally ensembled action chunks. The tutorial reports per-joint MSE and R^2 against a mean-action baseline, visualizes predicted versus ground-truth actions, and saves the complete policy checkpoint for downstream use. The provided text does not include experimental results.