Behavioral Cloning Data: Human + Sim

Behavioral-cloning (BC) data for SmolVLA can come from two interchangeable sources: a human driving the arm with a gamepad (Record Training Episodes), and a scripted MoveIt expert running entirely in simulation. Both feed the same recorder and the same LeRobot dataset, so you can mix teleop demonstrations with thousands of autonomously-generated sim episodes.

The key idea: the action is the realised joint configuration

What makes one recorder serve both sources is the action convention. At every timestep the label written to the dataset is the arm's achieved joint configuration (read from /joint_states) plus the commanded gripper — not the controller input that produced the motion:

# ar4_teleop/episode_recorder.py — record_frame()
action = [joints[j] for j in ARM_JOINTS] + [gripper_cmd]   # realised config, 7-dim

Because the action is the outcome, it is identical whether a human twist-jogged the arm there or a MoveIt plan executed it. So the data source is interchangeable, and every episode shares one schema (observation.images.wrist_camera, observation.state, action, task) that matches the SmolVLA checkpoint's input/output features exactly.

Editable diagram source: images/bc-data-architecture.c4.yaml

The shared recorder

EpisodeRecorder (ar4_teleop/episode_recorder.py) is input-agnostic. It subscribes to the wrist camera and joint states, and exposes a tiny API any driver can call:

It bootstraps the dataset with LeRobotDataset.create() for a fresh corpus or LeRobotDataset.resume() to append. Teleop and the MoveIt expert write separate datasets (ar4_pick_place and ar4_pick_place_moveit) that share one schema — so they stay independently versioned yet can be unioned at train time into one mixed, multi-task corpus.

The sim expert (MoveIt-driven)

moveit_record_node is a scripted expert that drives a collision-aware pick-and-place while the recorder captures it. The shelf is loaded into the MoveIt planning scene (parsed from the world SDF), and placement uses a high-entry drop: lift over the shelf, descend in front of the target tier with the gripper kept perpendicular to the facade, push into the opening, and release so the object drops onto the plank. A single generic instruction is used and each episode samples a random object and a random shelf, so one run yields varied multi-task demonstrations.

Editable diagram source: images/sim-record-loop.c4.yaml

Each episode: sample the active object → place it at a random pick pose (the rest park as distractors) → home → IK-pick top-down → grasp → verify the lift (the object must rise, else discard) → high-entry drop onto the sampled shelf → release → save. Recording is gated to the manipulation phase; missed grasps and failed plans are discarded rather than poisoning the dataset.

Declarative task spec

The whole campaign is described in ar4_teleop/config/sim_data_tasks.yaml — edit it to define a data run with no code changes:

dataset:  { repo_id: local/ar4_pick_place_moveit, root: /data/datasets, hz: 15.0 }
campaign: { instruction: "randomly select a cube and place it on one of the shelves", episodes: 30 }
world:
  objects:                                    # one sampled per episode; the rest are distractors
    - { name: block_teal,   dims: [0.023, 0.020, 0.022], home: [0.52, -0.12] }
    - { name: block_purple, dims: [0.023, 0.020, 0.022], home: [0.52, -0.04] }
    - { name: block_orange, dims: [0.023, 0.020, 0.022], home: [0.52,  0.04] }
  cube_random: { x: [0.34, 0.44], y: [-0.04, 0.10], z: 0.011 }   # the active object's pick pose
place: { front_quat: [0.5, 0.5, -0.5, -0.5], front_inset: 0.02, drop_gap: 0.025, approach_height: 0.22 }
shelves:
  - { name: top,    target: [0.30, -0.45, 0.38], approach_height: 0.06 }   # at the reach limit
  - { name: middle, target: [0.30, -0.45, 0.22] }
  - { name: bottom, target: [0.30, -0.45, 0.06] }

Each episode samples one objects entry and one shelves entry under the single campaign.instruction. Pin a subset for testing with --object <name> / --only <shelf>.

Running it

The step-by-step commands live in Record Training Episodes → Method B: bring up the sim + MoveIt + the Zenoh bridge, start the decoupled recorder, run the expert with --decoupled, then docker stop the recorder to finalize the dataset. Rehearse with --dry-run (replays the motion, records nothing) while calibrating placement.

Runtime note — lerobot vs ROS numpy. lerobot requires numpy >= 2, while ROS 2 Jazzy (rclpy, cv_bridge) is built against numpy 1.26 — not ABI-compatible in one Python process (cv_bridge raises numpy.core.multiarray failed to import under numpy 2). The solution is the decoupled recorder (ar4_lerobot/, the ar4_rerun pattern, now built): a numpy-2 container consumes camera + joints over Zenoh (pycdr2, never importing rclpy/cv_bridge) and writes the LeRobot dataset, while the ROS-side expert signals episode boundaries over a shared file — keeping numpy 2 entirely out of the ROS runtime. The recorder calls LeRobotDataset.finalize() on docker stop, so always stop it cleanly.

From data to a new policy

Once the episodes are recorded they flow straight into the existing training path:

1. The sim episodes write to data/datasets/ar4_pick_place_moveit/ (the MoveIt-expert dataset, multi-task: top/middle/bottom), separate from the teleop ar4_pick_place.
2. Sync the dataset to R2 and to the GPU box, then fine-tune on gpu-node-3 — see Train SmolVLA. Fine-tune from the latest checkpoint rather than from scratch, and train on the union of human + sim, top + middle + bottom, so the prompt selects the shelf instead of the policy collapsing to one behavior.
3. Run inference and change the prompt to pick the shelf.