Scene Reconstruction

Capturing the real environment starts here. We reconstruct the AR4 workspace as a 3D Gaussian Splat (3DGS) — a photorealistic, view-consistent 3D model built from ordinary multi-view photographs — and distill CLIP language features into the Gaussians so objects can be localized from plain text ("teal cube", "middle shelf") as 3D coordinates. Those coordinates are then aligned into the robot's base_link frame and handed to the manipulation planner as target poses.

The result is a digital twin of the lab usable in both Gazebo simulation and on the real AR4, where the same semantic tags resolve to consistent metric coordinates across reconstruction, sim, and reality.

The pipeline lives in gs/.

Pipeline

images → COLMAP (exhaustive matching) → gsplat MCMC training
       → CLIP feature distillation → text query → 3D XYZ
       → coordinate alignment → robot base_link frame

1. Capture — cover the workspace from many viewpoints; a gimbal video pass plus stills (see Capture protocol).
2. COLMAP — structure-from-motion recovers camera poses and a sparse point cloud (exhaustive matching, because the capture is unordered).
3. gsplat (MCMC) — train the Gaussian Splat from the posed images.
4. CLIP distillation — extract CLIP ViT-B/32 features per view and distill them into the Gaussians (visibility-weighted), enabling text → 3D queries.
5. Alignment — map 3DGS (COLMAP) coordinates into the robot base_link frame so the tags are actionable for planning.

Quality

v2 is the rebuilt lab (updated desk and shelf, matching the simulation scene), captured with a gimbal. It improves on v1 across every metric: +2.11 dB PSNR and −30% LPIPS, roughly a 36% reduction in MSE. The gain comes from denser, sharper coverage and from correcting a camera-model bug — see Lessons learned.

Trained on an NVIDIA RTX PRO 6000 (Blackwell), 30K steps, MCMC strategy.

Capture protocol

The capture determines the ceiling on reconstruction quality; nothing downstream recovers what was not photographed well.

• Gimbal video (4K) plus a handful of stills. A gimbal removes the motion blur that historically made video unusable for SfM, so a dense video pass now beats a sparse set of hand-held photos. 8K is not worth it: file size and COLMAP time scale badly for negligible gain.
• Lock exposure and focus before shooting. Auto-exposure drift across a capture creates photometric inconsistency, which 3DGS resolves by averaging — producing floaters and soft detail.
• Three orbits at different heights (high, mid, low), roughly 70% overlap between adjacent views. The mid orbit matters most: it is the robot-camera height.
• Nothing in the scene may move, including the arm. Fix the arm at a known joint configuration and record that configuration — it is the anchor for coordinate alignment.
• Frame the workspace plus ~0.5 m of margin. Not the whole room (wastes Gaussians on irrelevant geometry), not just the tabletop (leaves SfM without context).

Frames are extracted at 2.5 fps and filtered by variance-of-Laplacian sharpness, keeping the sharpest ~40 per video. The v2 capture yielded 451 frames plus 18 stills = 469 images.

Lessons learned

Mixed image sources need per-folder cameras — never post-hoc merging

The v2 capture mixed three image sizes (stills at 4032×3024, video frames at 2160×3840 and 1080×1920). COLMAP's camera mode is not a detail here; it decides whether the reconstruction works at all.

Bundle adjustment refines intrinsics jointly with poses, per camera. Rewriting 452 refined cameras down to 3 by adopting one representative's intrinsics leaves every image with poses optimized against intrinsics it no longer has. Training views partially absorb the inconsistency; held-out views cannot — which is why training loss looked healthy while validation PSNR collapsed.

Rule: never rewrite camera intrinsics after bundle adjustment. Split the images by source before COLMAP and let BA optimize the shared cameras (CameraMode.PER_FOLDER, one subfolder per source).

The diagnostic that isolated it: a plain run (no extra flags) on the consolidated cameras scored 11.83 dB at step 7K, where v1 scored 21.64 dB at the same step — identical trainer, identical flags, only the camera model differing.

Memory pressure, not compute, is the stability limit

Training at full 4K resolution ran at ~45 GB and died with transient CUDA address-space errors four separate times, always at 74–81% completion. The same pipeline at --data_factor 2 (~2 MP, 6–8 GB) completed in 23 minutes on the first attempt. The v1 run, which never crashed, used 1.5 GB. Downscale; the softness of video frames means full resolution buys little anyway.

Related: the trainer's default checkpoints (steps 7K and 30K) mean a crash at step 24K costs everything back to 7K. Use --save-steps 7000 12000 17000 22000 26000 30000.

Coordinate GPU access with other users — long runs sharing the card with another heavy job correlated with the crashes above.

--use_bilateral_grid and --pose_opt invalidate raw PSNR

Both are reasonable for multi-video captures with exposure variation, but the bilateral grid learns per-training-image color corrections that held-out views do not have. Raw validation PSNR therefore drops sharply (~14 dB) even when the model is good. The color-corrected metrics (cc_psnr) exist for exactly this reason: a flagged run must be judged on those, never on raw PSNR. In our case a clean per-folder baseline with no flags won outright.

Coordinate alignment (3DGS → robot frame)

3DGS reconstructions live in an arbitrary SfM/COLMAP frame (arbitrary scale, origin, orientation, not gravity-aligned). To use the splat as a digital twin, that frame must be aligned to the metric, gravity-aligned robot world frame so the semantic tags carry correct metric coordinates in both sim and real.

Following RoboSimGS §3.2.1, the alignment anchors on the robot's own geometry rather than on objects: the arm is fixed and its geometry is known from the URDF, whereas objects move between captures. The AR4 is posed at a known joint configuration, a point cloud is generated from the URDF at that configuration (yourdfpy forward kinematics on the compiled xacro), the robot's Gaussians are extracted from the reconstruction, and the two clouds are registered.

align.py implements the closed-form Umeyama similarity transform (scale · R · t) for known correspondences, validated to sub-mm accuracy on synthetic data. icp_gravity.py implements the production registration.

Gravity-constrained similarity ICP

A free 7-DOF ICP is the obvious approach and it fails badly here. What works:

1. Level the scene first. Both frames share an observable gravity direction — the URDF is Z-up by construction, and the reconstruction contains a large, densely reconstructed table plane. Fit that plane robustly (SVD with reweighting), rotate its normal to +Z, and shift it to z = 0. (Measured scene tilt: 2.88°.)
2. Register with the remaining freedom only: yaw, XY translation, a small Z, and scale. A closed-form Z-constrained Umeyama solves this exactly at each iteration.
3. Seed the yaw exhaustively. With one rotational DOF left, a deterministic 12-seed sweep provably covers the space — no stochastic global initialization required.

The point is general: every degree of freedom that can be measured should not be optimized.

Failure modes worth knowing

Each of the following produced plausible-looking numbers.

• Low RMSE does not validate an alignment. A free 7-DOF fit converged at 5.8 mm RMSE while tilted 28° from horizontal. RMSE measures agreement with the correspondences the fit chose; it says nothing about global correctness. Always validate against geometry the fit never saw — we transform the table plane (unused in the arm registration) and check that it lands horizontal at the expected height.
• Free scale plus correspondence trimming is a collapse vector. Shrinking the source cloud shrinks all nearest-neighbour distances, trimming discards the objections, and the cloud implodes inside the target: scale ran 1.16 → 0.224 with an excellent 3.73 mm RMSE. Freeze scale during the search; refine it afterwards in a bounded 1-D bracket.
• Validation gates deserve the same scrutiny as the transform. A base-plate check that sliced both clouds by z < 0.08 reported a 21 cm offset. At the bent capture pose the forearm and gripper hang into that band: 66% of the URDF "base plate" slice was arm, not base, dragging the reference centroid to Y = 0.154. Isolating the base by low and near the vertical axis (z < 0.06 & hypot(x, y) < 0.08) reduced the reported error from 210 mm to 47 mm. Three quarters of an apparent failure was a bug in the ruler.
• Systematic bias and sampling noise look alike until you vary the sampling. The residual 47 mm is entirely in Y (X and scale validate cleanly). Sweeping the opacity threshold moved the base-point count from 320 to 7,300 while the gap held at 47 mm — stable across a 20× density change, therefore systematic. The likely cause is that the URDF cloud was generated at the commanded pose while the real arm sits at commanded + calibration offsets; a small shoulder or elbow difference displaces the base along the bend axis, which is Y.
• Generate robot geometry with a real FK library. Hand-transcribing joint origins from the xacro produced a point cloud with zero X extent — a 3D arm flattened onto a plane. Sanity-check the bounding box: a bent AR4 spans roughly [0.23, 0.58, 0.50] m.

Status. Rotation, scale, and X-translation validate cleanly; a 47 mm single-axis residual remains, pending the joint calibration offsets from the physical robot. Regenerating the URDF cloud at the corrected pose and re-running the registration is a two-command job.

Isaac Sim

The splat's destination is IsaacSim, where it serves as the photoreal visual layer of the digital twin. Notes from bringing the simulator up on gpu-node-3:

• ISAACSIM_HOST must be set on the container, or the stream is black. runheadless.sh only advertises a public IP if that variable exists. Without it, TCP signalling connects (the port listens) but no UDP media ever binds, so the WebRTC client shows a connected-but-black window. Add -e ISAACSIM_HOST=<reachable-ip> to docker run.
• Use the Tailscale address the client can actually reach. The server reports its own address as 100.85.176.38; gpu-node-3 resolves to 100.88.71.60 from a client machine, and only that address responds.
• Gaussian splats render under specific launch settings. Isaac Sim 6.0 loads them as ParticleField assets (the older NuRec USDZ path is deprecated), and they require single-GPU rendering (--/renderer/multiGpu/enabled=false) with gaussian tonemapping left enabled (--/rtx/rtpt/gaussian/skipTonemapping/enabled=false), applied before the first Hydra sync.
• nurec_utils renders and validates NuRec USDs; it does not create them. Converting a .ply requires 3dgrut, which can also train from a COLMAP dataset and export USD natively — and, unlike gsplat, models radial distortion directly, which our SIMPLE_RADIAL cameras have. A 3DGUT retrain is worth comparing against the 29.23 dB baseline.
• Isaac ships a render-versus-ground-truth harness (PSNR/SSIM against real captured images), so the digital twin can be validated quantitatively against the original photographs.

As noted in Scene Representation, the splat is a purely visual layer: collision geometry must be reconstructed or authored alongside it, and the floor is not guaranteed to land at z = 0 — which is precisely what the levelling and alignment work above provides.

Scripts

Training command

python simple_trainer.py mcmc \
  --data_dir <colmap_data> --data_factor 2 \
  --result_dir output --max_steps 30000 \
  --init_type sfm --normalize-world-space \
  --save-steps 7000 12000 17000 22000 26000 30000 \
  --disable_viewer

gsplat v1.5.3. On Blackwell / sm_120, build the CUDA kernels with TORCH_CUDA_ARCH_LIST="9.0+PTX".

Data and model location

Large artifacts live in the R2 bucket (ar4-physical-ai), not in git:

lab_scene_v1.ply and lab_scene_v2.ply are standard INRIA-format 3DGS point clouds, readable by SuperSplat and the reference viewers.

Open items

• Close the 47 mm alignment residual once joint calibration offsets are available.
• Convert lab_scene_v2.ply to a ParticleField USD and load it in IsaacSim.
• Retrain with 3DGUT (native distortion handling) and compare against 29.23 dB.
• Camera-pose alignment for the fixed cameras (RoboSimGS §3.2.2: photometric error minimization through the differentiable renderer). Requires intrinsic calibration and undistortion of the IP camera first.
• Per-region CLIP distillation (LangSplat-style) to sharpen language grounding, which the object-type → placement-template reasoning step will depend on.

You can inspect the capture and the resulting reconstruction interactively with Rerun — see Rerun Recording.