Scene Representation
How AR4 scenes are represented across the stack: handcrafted SDF twins, meshes, Gaussian splats, and USD — visuals vs collision vs semantics, and which representation serves which purpose.
Scene Reconstruction is about acquiring a scene from the real world; this page is about how a scene, however acquired, is represented — because no single representation serves every consumer. A simulator needs collision geometry, a policy needs photoreal (or at least distribution-matched) pixels, an expert planner needs semantic poses, and a benchmark needs ground truth. The stack therefore treats a "scene" as a bundle of complementary layers rather than one artifact.
The four layers
| Layer | Representation | Consumers | Where it lives today |
|---|---|---|---|
| Collision / physics | primitive boxes & meshes (SDF) | Gazebo physics, MoveIt planning scene | annin_ar4_gazebo worlds & models |
| Visual | SDF visuals today; 3D Gaussian splats next | cameras → policy observations | worlds today; splat pipeline in AURA-756 |
| Semantic / ground truth | named objects + poses, destination targets | scripted expert, GT sidecars, eval classifiers | task specs (sim_data_tasks*.yaml), gt_sidecar.jsonl |
| Interchange | SDF today; USD for IsaacSim | cross-simulator portability (AURA-747) | planned |
Two properties of this split do real work:
- Visual and collision geometry deliberately differ. The bookshelf's collision posts were 8 mm while its visuals were 12 mm; the Furinno twin's collision boxes are exact but its visual finish is cosmetic. Physics wants conservative, simple shapes; cameras want realism. Gaussian splats push this to the extreme: a splat is a purely visual representation with no physics at all — it must always be paired with reconstructed or authored collision geometry.
- Semantics don't live in geometry. The expert grasps and the evaluator scores using
named poses (object GT, tier surfaces, cubby splits), not by parsing meshes. This is what
makes the GT sidecar, the tier-precise eval bands, and the spec-driven planning scene
(
world.collision_models,place.acm_shelf_parts) possible.
A worked example: the Furinno twin
The real-product shelf (AURA-753) shows all layers in one object: measured box collisions (10 mm panels), grey-oak visual boxes, semantic surfaces (top +0.195 / middle +0.095 / cubbies +0.025 with a left/right split at x = 0.30) consumed by both the expert's spec and the eval classifier — and, once captured as a splat, the same object will carry a photoreal visual layer for domain-randomized training in IsaacSim.
Where this is going (AURA-756)
The target pipeline pairs splats for visuals with meshes for physics under a USD scene description in IsaacSim: capture → splat (photoreal layer) + surface reconstruction (collision layer) → USD assembly with semantic annotations → domain randomization over the visual layer only, physics invariant. The digital twin then serves double duty: a training environment and — per the evaluation methodology — the validation instrument for checkpoint selection.