Architecture

AR4 Physical-AI is a VLA (Vision-Language-Action) platform layered on top of the AR4 ROS driver and LeRobot. The key architectural decisions:

1. LeRobot-native data and policies — records LeRobot-format datasets and fine-tunes LeRobot policies such as SmolVLA
2. Submodule for driver — upstream ar4_ros_driver stays independently updateable
3. Docker-first — multi-stage GPU containers (base → overlay → dev) with docker-compose orchestration
4. Simulation-first — physics-enabled Gazebo world with gravity, contact properties, and graspable objects for policy development
5. Zenoh middleware — decouples non-ROS components from DDS for future inference pipelines
6. Explicit control paths — recording and inference use different controller stacks, documented below instead of hidden behind one generic bridge

System Overview

C4 Model

The same system drawn at two C4 levels: the context (who uses it and what it depends on) and the containers (the deployable pieces and the data pipeline between them).

System Context (L1)

Editable diagram source: images/c4-context.c4.yaml

Containers (L2)

Editable diagram source: images/c4-container.c4.yaml

Control and Data Paths

The project has four separate paths. They share the same robot, cameras, joint state topic, and LeRobot dataset schema, but they do not all use MoveIt in the same way.

Editable diagram source: images/c4-control-paths.c4.yaml

The distinction matters because "MoveIt is running" is not the same as "MoveIt is in the policy control loop." In the current SmolVLA inference implementation, the policy publishes single-point JointTrajectory commands directly to the joint trajectory controller, and the node pauses MoveIt Servo at startup to prevent command conflicts.

A. Human Teleoperation Data

Human teleoperation uses MoveIt Servo as the live controller. The gamepad produces Cartesian or joint jog commands, Servo converts them into smooth joint trajectory commands, and the recorder saves the resulting robot state at each tick. The dataset label is the realised joint configuration, not the joystick velocity command. This keeps human demonstrations schema-compatible with scripted demonstrations and SmolVLA inference:

gamepad
  -> servo_teleop_node
  -> MoveIt Servo
  -> /joint_trajectory_controller/joint_trajectory
  -> /joint_states + cameras
  -> EpisodeRecorder / LeRobot dataset

B. MoveIt Expert Data

The scripted expert generates synthetic demonstrations. It uses MoveIt for IK, planning, and the planning scene, but it does not use Servo as the streaming controller for the episode. moveit_record_node pauses Servo, computes target joint poses or Cartesian segments, and sends planned or direct JointTrajectory commands to the joint trajectory controller. The recorder again writes the realised joint positions as the action labels:

task spec
  -> moveit_record_node
  -> MoveIt IK / planning / planning scene
  -> /joint_trajectory_controller/joint_trajectory
  -> /joint_states + cameras
  -> LeRobot dataset

This is also where induced waypoint randomness belongs for synthetic data. The randomness is part of the demonstrated trajectory before recording, so the learner sees multiple valid ways to solve the same (object, shelf) placement instead of one deterministic joint replay.

C. SmolVLA Inference Without MoveIt

This is the path implemented by ar4_policy_server/ros_bridge/inference_node.py today. SmolVLA receives camera frames, the current joint state, and the language instruction. The LeRobot pre/post-processors normalize the observation and denormalize the action. The node then streams the resulting absolute joint targets directly to the joint trajectory controller:

scene camera + wrist camera + /joint_states + task
  -> SmolVLABackend
  -> SmolVLAPolicy.select_action()
  -> absolute action [j1..j6, gripper]
  -> /joint_trajectory_controller/joint_trajectory

SmolVLA still uses its internal action chunking, but this repo consumes the chunk as a queue of absolute joint targets. It does not currently convert chunks into JointJog, TwistStamped, or Servo input messages.

D. SmolVLA Inference With MoveIt

This is the architecture to build if learned-policy actions should be filtered by MoveIt Servo before reaching the robot. It needs an explicit adapter between SmolVLA and Servo:

scene camera + wrist camera + /joint_states + task
  -> SmolVLA action chunk
  -> ROS 2 adapter
  -> /servo_node/delta_joint_cmds or /servo_node/delta_twist_cmds
  -> MoveIt Servo
  -> /joint_trajectory_controller/joint_trajectory

That adapter must decide whether SmolVLA's absolute joint targets are converted into joint deltas, joint velocities, Cartesian twists, or dense short-horizon trajectory points. It also has to manage action chunk timing so inference latency does not stall Servo's command stream. This path is not the current implementation; it is a separate safety-filtered inference mode.

Docker Infrastructure

All services run in Docker containers orchestrated by docker-compose.yaml. The Dockerfile uses a multi-stage build:

Zenoh Middleware

Zenoh provides a lightweight pub/sub transport layer that decouples non-ROS components (like the Optuna PID tuner) from the DDS discovery mesh. This avoids requiring ROS 2 in every container.