Datasets

The AR4 behavioural-cloning datasets are stored in LeRobot v3.0 format (15 fps) in Cloudflare R2, and each is registered as a versioned W&B artifact (type=dataset) on aigeo/ar4 for lineage — a training run calls use_artifact(...) on one, so every trained model traces back to the exact data it saw.

Dataset format (LeRobot v3.0)

LeRobot datasets use a standardized directory structure:

meta/
  info.json              # schema, features, fps, path templates
  stats.json             # normalization statistics (mean/std/min/max)
  tasks.parquet          # task strings mapped to integer IDs
  episodes/              # per-episode metadata (chunked Parquet: frame ranges, video timestamps)
data/
  chunk-XXX/file-XXX.parquet          # state/action rows, many episodes concatenated per file
videos/
  observation.images.<camera>/chunk-XXX/file-XXX.mp4   # per-camera video, episodes concatenated

v3.0 concatenates many episodes into a few large files (per-chunk Parquet + per-camera video) instead of one file per episode, which is why a 60-plus-episode set lands as only a handful of R2 objects. Training must use a v3.0-compatible LeRobot (check_version_compatibility rejects a major mismatch), so keep the concatenated-video layout.

The AR4 feature schema:

Early single-camera datasets logged one image stream; the current two-camera datasets (the Furinno line and …_single_v3/…_moveit_v4) log both wrist_camera and scene_camera.

The artifacts are references: the artifact records the R2 URI and provenance metadata (episode/frame counts, size), not a second copy of the video blobs. The data itself lives in R2 under datasets/<name>/.

Furinno shelf line (current)

These sets use the twin of the real Furinno product shelf, four destinations (top, middle, and the left and right bottom cubbies), two cameras, and a ground-truth sidecar. They are the corpus behind the corrective-data study in Impact of Training Data.

The current baseline model trains on purple-all, the 481-episode single-object corpus assembled from v5, v7, and the purple slice of v6, and reaches 78% atomic placement (5xu4mkno). Each corrective arm adds one dataset to that baseline: v8 the guessed perturbation-recovery episodes (62%), v9 the biased-seed DAgger corrections (45%), and v10 the same corrections reseeded uniformly (78%), which recovered the baseline. Why the first two fell below baseline, and how the reseed fixed v9, is the subject of Impact of Training Data §6.

Earlier tabletop / MoveIt datasets

The tabletop line predates the Furinno shelf. ar4_pick_place_moveit_v4 is the anti-shortcut set that first broke the proprioceptive shortcut, and the …_single sets are the single-task purple-cube corpora that preceded it.

ar4_pick_place_moveit_v4 was generated to break the proprioceptive shortcut: per-episode random hover height, recorded via-waypoints (75% of episodes), 0.76–1.35× pacing, uniform cube coverage (16/16 pick-zone cells), and balanced shelves. Against v3, time-to-grasp CV rose to 20% from 5% and nearest-neighbour-state future divergence to ×3.2 at 1 s and ×6.0 at 3 s (from ×1.7/×2.0), so the chunk label is ambiguous given joint state alone. It ships a ground-truth sidecar (gt_sidecar.jsonl, index-aligned) with per-episode object start pose, grasp offset, shelf target, and sampled variability, usable as labels for auxiliary grounding losses.

All twelve datasets above are registered as type=dataset artifacts on aigeo/ar4, each recording its R2 URI and provenance so every training run traces back through use_artifact(...) to the exact data it saw.

Fetching a dataset

Via W&B (records dataset → model lineage in a training run):

import wandb
run = wandb.init(entity="aigeo", project="ar4", job_type="train")
art = run.use_artifact("aigeo/ar4/ar4_pick_place_moveit_single_v3:latest")
print(art.metadata)   # episodes, frames, fps, r2_uri, r2_public_base, …

Via R2 (the actual blobs — the artifact is a pointer):

# creds/endpoint in the repo .env.local (R2_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
env -u AWS_PROFILE -u AWS_CA_BUNDLE aws s3 sync \
  s3://ar4-physical-ai/datasets/ar4_pick_place_moveit_single_v3/ ./ar4_v3 \
  --endpoint-url "$R2_ENDPOINT"

Registering a new dataset

After a data-generation run uploads a dataset to datasets/<name>/ in R2, register it as an artifact so it shows up here and can be used for training with lineage:

import wandb
run = wandb.init(entity="aigeo", project="ar4", job_type="dataset-registration")
art = wandb.Artifact("<name>", type="dataset", metadata={
    "lerobot_codebase_version": "v3.0", "fps": 15,
    "episodes": E, "frames": F,
    "r2_uri": "s3://ar4-physical-ai/datasets/<name>/",
})
art.add_reference("s3://ar4-physical-ai/datasets/<name>/", checksum=False)  # needs boto3
run.log_artifact(art)
run.finish()