Research: LeRobot Integration

Investigation of ycheng517/lerobot-ros and huggingface/lerobot for the AR4 Physical-AI platform. Conducted 2026-03-09 as part of ar4-jjz.7.

Key Decision

Adopt lerobot-ros as the primary bridge between AR4 and LeRobot, replacing the originally planned ar4_policy_server and ar4_episodes packages.

Rationale: The same author (ycheng517) who maintains ar4_ros_driver has built lerobot-ros — a pip-installable plugin that makes any ros2_control arm appear as a native LeRobot robot. It already includes an AnninAR4 class with AR4-specific configuration. Building our own policy server and episode recorder would duplicate what LeRobot already provides.

lerobot-ros (ycheng517)

160 stars, 21 forks. Two pip packages — no colcon build needed.

What it provides

AR4 configuration (built-in)

• Action type: Cartesian velocity via MoveIt Servo (ActionType.CARTESIAN_VELOCITY)
• Gripper: gripper_jaw1_joint, open=0.014, close=0.0, GripperActionType.ACTION
• Joint limits: AR4 MK1 specific (J1: +/-2.9671 rad, etc.)
• ROS topics used: /joint_states (sub), /servo_node/delta_twist_cmds (pub), /gripper_controller/gripper_cmd (action client)

How it works

Known limitations

• JOINT_TRAJECTORY mode publishes to a topic rather than using FollowJointTrajectory action client — no execution feedback (Issue #3)
• Requires LeRobot >= 0.4.0, < 0.5.0
• Requires MoveIt Servo for Cartesian velocity control

huggingface/lerobot

Dataset format (v3.0)

LeRobot datasets use a standardized structure:

meta/
  info.json              # Schema, features, FPS, path templates
  stats.json             # Normalization statistics (mean/std/min/max)
  tasks.parquet          # Task descriptions mapped to integer IDs
  episodes/              # Per-episode metadata as chunked Parquet
data/
  chunk-XXX/file-XXX.parquet   # State/action data (many episodes per file)
videos/
  {camera}/chunk-XXX/file-XXX.mp4   # Camera video shards

For AR4, the features would be:

Policy architectures available

Robot integration pattern

Adding a new robot to LeRobot requires:

1. Subclass RobotConfig, register with @RobotConfig.register_subclass("name")
2. Subclass Robot, implement get_observation() and send_action()
3. Define observation_features and action_features properties
4. Package as pip plugin (lerobot_robot_*)

lerobot-ros already does all of this for AR4 via the AnninAR4 class.

Workflow

# 1. Teleoperate and record episodes
lerobot-teleoperate --robot.type=annin_ar4_mk1 --teleop.type=gamepad_6dof
 
# 2. Record dataset
lerobot-record \
    --robot.type=annin_ar4_mk1 \
    --teleop.type=gamepad_6dof \
    --dataset.repo_id=aegean-ai/ar4-pick-cube \
    --dataset.single_task="Pick up the red cube"
 
# 3. Train policy
lerobot-train --policy=act --dataset.repo_id=aegean-ai/ar4-pick-cube
 
# 4. Evaluate
lerobot-evaluate --policy.path=outputs/train/act/checkpoints/last/pretrained_model

Simulation gap analysis

Neither lerobot-ros nor LeRobot provide simulation environments for the AR4. The upstream ar4_ros_driver Gazebo setup is kinematic-only (zero gravity, empty world). This is the primary gap we must fill.

What must be built (our packages)

What we get for free (from lerobot-ros + LeRobot)

Impact on project architecture

This research leads to a significant simplification:

• ar4_policy_server — no longer needed as originally designed. LeRobot policies run through lerobot-ros directly. May be repurposed for custom policy wrappers if needed.
• ar4_episodes — no longer needed. lerobot-record handles recording and export natively.
• ar4_skills — was the primary focus for simulation; now removed. Simulation runs directly via annin_ar4_gazebo from the vendor submodule.
• ar4_msgs — may be simplified. Standard ROS 2 interfaces used by lerobot-ros may suffice.

References

• ycheng517/lerobot-ros — ROS 2 bridge for LeRobot
• huggingface/lerobot — LeRobot framework
• Annin-Robotics/ar4_ros_driver — upstream AR4 driver
• ycheng517/tabletop-handybot — voice-controlled pick-and-place (real hardware)