Getting Started

A research platform for training and deploying manipulation policies on the AR4 6-DOF robot arm.

Built on ROS 2 Jazzy, Gazebo Harmonic, MoveIt2, and LeRobot — the full pipeline runs in Docker so you don't need a local ROS install.

The general workflow is:

Record episodes → Train SmolVLA → Run inference → Repeat

Quick Start

New here? This is the full pipeline in five commands. All commands must be run from inside the ar4-physical-ai/ folder — open a terminal, cd ar4-physical-ai, and keep that terminal open.

Step	Command	Notes
1. Build images	`docker compose build`	First time only — takes ~20–30 min
2. Start simulation	`docker compose up`	Opens Gazebo + RViz2. Wait for "MoveIt Servo ready" in logs
3. Collect episodes	`docker compose run --rm record`	New terminal — arm moves to home, then you drive it
4. Train SmolVLA	See Train SmolVLA	Needs a GPU
5. Run policy	`docker compose run --rm smolvla`	New terminal — arm runs autonomously

The detailed steps for each stage are in the workflows below.

Prerequisites

Docker with NVIDIA GPU support — install nvidia-container-toolkit
X11 display for the Gazebo GUI — if you get a permission error, run xhost +local:docker once in your terminal before starting
Xbox One (or compatible) controller — USB or Bluetooth; plug it into any USB port, it is detected automatically
ROS 2 Jazzy on your host machine — only needed if you want to use rqt_image_view to preview the wrist camera. If you don't have it, Foxglove works without any local ROS install

Installation

# Clone with submodules (the AR4 ROS driver lives under vendor/)
git clone --recurse-submodules https://github.com/aegean-ai/ar4-physical-ai.git
cd ar4-physical-ai
 
# Build all Docker images (base → overlay → dev)
# This takes a while the first time — grab a coffee
docker compose build

Forgot --recurse-submodules? Run git submodule update --init --recursive to fix it.

Important: Every docker compose command in this README must be run from inside the ar4-physical-ai/ folder. If you open a new terminal, cd back into the repo first:
cd ar4-physical-ai
docker compose up

docker compose up (no arguments) starts only the simulation — Gazebo and MoveIt. Everything else is opt-in so you never accidentally launch two Gazebo instances or start hardware drivers alongside a sim.

Command	What it starts
`docker compose up`	Gazebo GUI + MoveIt2 + RViz2 (simulation)
`docker compose --profile hardware up`	Real AR4 driver + MoveIt2
`docker compose --profile headless up sim-tabletop-headless`	Gazebo without a display (for servers / CI)
`docker compose --profile obs up foxglove-bridge`	Foxglove WebSocket bridge
`docker compose run --rm record`	Xbox teleop + LeRobot recorder
`docker compose run --rm smolvla`	SmolVLA policy inference
`docker compose run demo`	C++ pick-and-place demo

Services that run with docker compose run are one-shot interactive containers — they start, do their job, and exit cleanly on Ctrl-C. The simulation (up) stays running in the background while you use them.

Workflows

Visualizing with Foxglove

Foxglove Studio lets you visualize all ROS 2 topics (camera feed, joint states, TF tree, etc.) in a browser — no ROS install needed on your machine.

# Start the WebSocket bridge alongside the sim
docker compose up &
docker compose --profile obs up foxglove-bridge

Then open app.foxglove.dev, click Open connection, choose WebSocket, and enter:

ws://localhost:8765

VS Code Dev Container

The dev service mounts the source code and colcon build artifacts so you can edit and rebuild without rebuilding the Docker image.

Open this folder in VS Code and choose "Reopen in Container" — it uses .devcontainer/devcontainer.json which points at the dev service.

You can also start it manually:

docker compose --profile dev up dev
docker exec -it ar4-physical-ai-dev-1 bash

Packages

This project

Package	Description
`ar4_msgs`	Custom ROS 2 interfaces (PolicyAction, EpisodeStep, SetTask, ExecuteSkill)
`ar4_teleop`	Xbox gamepad teleop + LeRobot episode recorder node
`ar4_policy_server`	SmolVLA inference backend (local + remote HTTP)
`ar4_moveit_cpp_demo`	Simple C++ pick-and-place demo using MoveIt

LeRobot ecosystem (pip-installed)

Package	Description
lerobot	Recording, training, and inference framework
lerobot-ros	ROS 2 bridge — makes AR4 appear as a native LeRobot robot

AR4 ROS Driver (git submodule under `vendor/ar4_ros_driver/`)

Package	Description
`annin_ar4_description`	URDF/Xacro robot model and meshes
`annin_ar4_driver`	Hardware interface for Teensy, ros2_control integration
`annin_ar4_firmware`	Teensy and Arduino Nano firmware sketches
`annin_ar4_gazebo`	Gazebo Harmonic simulation
`annin_ar4_moveit_config`	MoveIt2 motion planning configuration

Updating the Driver Submodule

To pull in upstream AR4 driver fixes:

git submodule update --remote vendor/ar4_ros_driver
git add vendor/ar4_ros_driver
git commit -m "chore: update ar4_ros_driver submodule"
docker compose build  # rebuild images to include the new driver

AR4 ROS Driver — upstream hardware driver
lerobot-ros — ROS 2 bridge for LeRobot
HuggingFace LeRobot — robot learning framework
Annin Robotics — AR4 robot arm hardware

License

MIT License — see LICENSE.