Research: LeRobot Integration

Investigation of ycheng517/lerobot-ros and huggingface/lerobot for the AR4 Physical-AI platform. Conducted 2026-03-09 as part of ar4-jjz.7.

Key Decision

Adopt lerobot-ros as the primary bridge between AR4 and LeRobot, replacing the originally planned ar4_policy_server and ar4_episodes packages.

Rationale: The same author (ycheng517) who maintains ar4_ros_driver has built lerobot-ros — a pip-installable plugin that makes any ros2_control arm appear as a native LeRobot robot. It already includes an AnninAR4 class with AR4-specific configuration. Building our own policy server and episode recorder would duplicate what LeRobot already provides.

lerobot-ros (ycheng517)

160 stars, 21 forks. Two pip packages — no colcon build needed.

What it provides

AR4 configuration (built-in)

• Action type: Cartesian velocity via MoveIt Servo (ActionType.CARTESIAN_VELOCITY)
• Gripper: gripper_jaw1_joint, open=0.014, close=0.0, GripperActionType.ACTION
• Joint limits: AR4 MK1 specific (J1: +/-2.9671 rad, etc.)
• ROS topics used: /joint_states (sub), /servo_node/delta_twist_cmds (pub), /gripper_controller/gripper_cmd (action client)

How it works

Known limitations

• JOINT_TRAJECTORY mode publishes to a topic rather than using FollowJointTrajectory action client — no execution feedback (Issue #3)
• Requires LeRobot >= 0.4.0, < 0.5.0
• Requires MoveIt Servo for Cartesian velocity control

huggingface/lerobot

Dataset format

The AR4 datasets use LeRobot v3.0 format. The directory layout and the AR4 feature schema are documented on the Datasets page.

Policy architectures available

Robot integration pattern

Adding a new robot to LeRobot requires:

1. Subclass RobotConfig, register with @RobotConfig.register_subclass("name")
2. Subclass Robot, implement get_observation() and send_action()
3. Define observation_features and action_features properties
4. Package as pip plugin (lerobot_robot_*)

lerobot-ros already does all of this for AR4 via the AnninAR4 class.

Workflow

# 1. Teleoperate and record episodes
lerobot-teleoperate --robot.type=annin_ar4_mk1 --teleop.type=gamepad_6dof
 
# 2. Record dataset
lerobot-record \
    --robot.type=annin_ar4_mk1 \
    --teleop.type=gamepad_6dof \
    --dataset.repo_id=aegean-ai/ar4-pick-cube \
    --dataset.single_task="Pick up the red cube"
 
# 3. Train policy
lerobot-train --policy=act --dataset.repo_id=aegean-ai/ar4-pick-cube
 
# 4. Evaluate
lerobot-evaluate --policy.path=outputs/train/act/checkpoints/last/pretrained_model

Simulation gap analysis

Neither lerobot-ros nor LeRobot provide simulation environments for the AR4. The upstream ar4_ros_driver Gazebo setup is kinematic-only (zero gravity, empty world). This is the primary gap we must fill.

What must be built (our packages)

What we get for free (from lerobot-ros + LeRobot)

Impact on project architecture

This research leads to a significant simplification:

• ar4_policy_server — no longer needed as originally designed. LeRobot policies run through lerobot-ros directly. May be repurposed for custom policy wrappers if needed.
• ar4_episodes — no longer needed. lerobot-record handles recording and export natively.
• ar4_skills — was the primary focus for simulation; now removed. Simulation runs directly via annin_ar4_gazebo from the vendor submodule.
• ar4_msgs — may be simplified. Standard ROS 2 interfaces used by lerobot-ros may suffice.

References

• ycheng517/lerobot-ros — ROS 2 bridge for LeRobot
• huggingface/lerobot — LeRobot framework
• Annin-Robotics/ar4_ros_driver — upstream AR4 driver
• ycheng517/tabletop-handybot — voice-controlled pick-and-place (real hardware)