Spatial VLMs / An interactive guide
Spatial reasoning needs more than recognizing objects. It needs a precise referent, a useful geometric representation, and a way to check the evidence behind an answer.
Three works connect regions, 3D position, and visual evidence to language generation.
SpatialRGPT → SR-3D → GR3D 02 / Executable spatial reasoningGive the model a workspace.SpatialClaw composes perception and geometry through code, then inspects and revises the result.
Explore the agent interfaceA box or mask identifies exactly which region a question refers to. SpatialRGPT combines region features with language; an optional depth branch adds geometric evidence that RGB alone can leave ambiguous.
Schematic: region prompts and optional relative-depth evidence.
Region tokens resolve which objects to compare. The RGB-only model already learns spatial relationships from region-aware supervision.
Boxes and masks distinguish similar objects without relying on a long verbal description.
Open Spatial Dataset contains 8.7M spatial concepts grounded in 5M regions from 1M images.
A separate connector lets the model use depth evidence while retaining an RGB-only inference path.
The data pipeline uses detection, segmentation, metric depth, and camera calibration to construct spatial supervision. Those offline 3D scene graphs are not passed into the VLM at inference.
The released inference path accepts RGB, region masks, and optional relative depth. It pools region features and inserts their embeddings at region-token positions. Quantitative answers are learned estimates, not guaranteed geometric measurements.
Appearance alone does not say where a region sits in a scene. SR-3D fuses visual features with encoded 3D position, using a common representation design to transfer spatial knowledge from single images to multiple views.
Schematic of positional evidence. Encoded positions are normalized; they are not raw metric coordinates.
Pretrain with large-scale image data. In the released single-view code, normalized image coordinates and relative depth provide positional features.
A region representation carries both what it looks like and where it lies in the input geometry.
Mark an object in selected frames with a box or mask, or supply a 3D box, then reason across views.
The release provides separate single-view, multi-view scan, and multi-view video checkpoints.
Single-view preprocessing combines normalized image coordinates with normalized relative depth. For static scenes, the scan path back-projects depth with camera intrinsics, aligns points across views, and normalizes scene coordinates before positional encoding. The paper also uses external geometry models to supply aligned point maps.
These routes share a representation design, not identical geometry or one universal checkpoint. Geometry-aware multi-view inference needs aligned point maps; a plain RGB-video call is not equivalent.
Original SR-3D architecture ↗ · Position construction in code ↗
# Single-view position construction
q = (u_normalized * z_normalized,
v_normalized * z_normalized,
z_normalized)
# Fuse encoded position with appearance
features = visual_features + position_encoding(q)GR3D extends region-aware modeling with explicit 2D grounding, implicit region insertion during generation, and monocular 3D grounding. A generated object reference can bring fresh region features into the sequence before reasoning continues.
Schematic: resolve a region, insert its features, then predict a 3D box.
Predict a 2D region for the object being discussed. This gives subsequent inference an object-specific visual reference.
1 / 3 · Detect before lifting
Localize an object when the task directly asks for a region.
Insert region features into the generation stream as the model refers to an entity.
Use object-specific evidence to lift a 2D region into a 3D bounding-box prediction.
The paper compares predicting a 3D box directly with first grounding the target in 2D. The intermediate region anchors the 3D prediction to the right object and lets the model benefit from abundant 2D grounding supervision.
An auxiliary task predicts 3D point coordinates from region prompts, providing geometric supervision beyond sparse 3D-box labels.
Spatial pretraining freezes the vision encoder; grounded-CoT fine-tuning then updates only the language model. Training uses teacher-forced boxes and features pooled from ground-truth regions; inference uses predicted boxes.
SpatialClaw adds a different route: keep the VLM fixed, let it write Python, and feed execution results back into the next turn. Like ASENA, code is the action interface; here, the actions inspect and transform spatial evidence.
Masks, point maps, and intermediate calculations stay in the same Python session across turns. The agent can combine them into a question-specific computation rather than relying on a prewritten tool for every spatial relationship.
Independent Python state per task. GPU models are shared across tasks; concurrency is configurable.
recon = tools.Reconstruct.Reconstruct(InputImages)
show(recon.render_bev())Parallel sessions, executable feedback, and answer scoring provide reusable building blocks. A training extension would add complete rollout trajectories, optimizer updates, and synchronized policy weights. The published system is training-free.
Keeping and evaluating useful programs across tasks could enable an RSI loop. The current SpatialClaw runtime resets task state between questions; cross-task skill evolution would be an added mechanism.