Figure 1: Incremental construction of the global scene graph $\mathcal{G}_G$ over 20 processed views. Nodes are instantiated when an object is first detected and refined as subsequent views contribute pairwise metric constraints. The 3RScan mesh is shown in the global frame purely for spatial reference, illustrating how predicted nodes align with annotated instances. Our method observes no depth and builds no point cloud or mesh, estimating 3D object positions and semantic relations directly from posed RGB observations.
Scene graphs provide a compact, structured representation of environments by capturing objects and their spatial or semantic relationships in a form suitable for embodied agents. Existing 3D scene graph generation methods often rely on depth sensing at inference, either to associate object instances across views via point cloud overlap or to compute inter-object relations from geometric cues, limiting deployment to platforms with depth sensors. We propose IncR-3DSGG, which builds 3D scene graphs incrementally from posed* RGB images, eliminating the use of depth sensors at inference. Our method leverages a Parametric Relation Predictor (PRP) that learns to estimate pairwise distances and relative angles between objects from 2D visual features and 2D geometric cues, using 3D geometric supervision available only during training. These learned pairwise measurements are fused across views through Pose Graph Optimization (PGO) to recover metrically grounded 3D object positions and build a globally consistent 3D scene graph incrementally. On the real-world 3RScan dataset, our method attains $52.2$ mean object recall and $30.8$ mean predicate recall under the 3DSSG evaluation protocol, against $27.0$ and $17.6$ for MonoSSG, while using no depth or reconstructed geometry at inference. This shows that relations and metrically grounded 3D object representations can be predicted from posed RGB images alone.
* RGB images with camera extrinsics estimated by an off-the-shelf visual SLAM system.
IncR-3DSGG constructs a persistent 3D scene graph $\mathcal{G}_G = (\mathcal{V}_G, \mathcal{E}_G)$ incrementally from an exploration sequence of posed RGB images $(I_t, T_t)$. The framework operates in two coupled stages: per-view local scene graph generation ($\mathcal{G}_L$) and incremental global graph aggregation.
Figure 2: Overview of the IncR-3DSGG framework. Given an RGB image $\mathcal{I}_t$ and camera pose $\mathcal{T}_t$, a frozen EoMT panoptic backbone detects per-object nodes $\mathcal{V}_L$ and computes relative 2D geometry. The Parametric Relation Predictor (PRP) converts co-visible object pairs into edges $\mathcal{E}_L$ carrying 3D pairwise geometry and support predicates. Local detections are merged into the global scene graph $\mathcal{G}_G$ via visual Hungarian matching, and object positions are globally recovered via Pose Graph Optimization (PGO).
where constraint weights $w_{ij} = \gamma_{ij}/(1 + e_i + e_j)$ penalize large object extents ($e_i, e_j$) and leverage PRP's predicted confidence $\gamma_{ij}$.
Inferring 3D inter-object geometry from monocular 2D images is under-constrained. We decouple the training supervision from the inference modality: PRP is trained using privileged 3D supervision (depth maps and point clouds available only during training) but requires strictly monocular RGB at inference.
Figure 3: Architecture of the Parametric Relation Predictor (PRP). Left (Per-Object Encoding): Mask-pooled visual features and category embeddings are contextualized through an unordered Transformer encoder. Right (Pair Construction & Heads): For every directed pair $(i,j)$, contextualized tokens $h'_i, h'_j$ are concatenated with their difference, product, and projected 10-D 2D relative geometry $g'_{ij}$. Dedicated heads predict 3D centroid geometry, contact/containment features, support relation class, pairwise confidence, and object extents.
The pairwise input vector $g_{ij} \in \mathbb{R}^{10}$ is computed purely from 2D masks and boxes:
Sparse, l20, 157 test scans, official sets). Baselines read geometry from the ORB-SLAM3 point map, whereas ours uses no depth or reconstructed geometry at inference. Instances come from each method's own front end (ours: EoMT). Pred*, Trip* scores annotated pairs only.
| Method | Recall (R@1) | Annotated | Mean Recall | ||||
|---|---|---|---|---|---|---|---|
| Obj | Pred | Trip | Pred* | Trip* | mR.Obj | mR.Pred | |
| IMP | 27.5 | 90.8 | 8.5 | 4.5 | 3.9 | 20.4 | 14.0 |
| VGfM | 27.7 | 90.6 | 8.8 | 4.3 | 3.9 | 24.5 | 13.9 |
| 3DSSG | 10.8 | 90.1 | 1.3 | 4.1 | 1.0 | 6.0 | 13.7 |
| SGFN | 15.4 | 88.3 | 2.4 | 7.0 | 3.3 | 8.6 | 15.5 |
| MonoSSG | 28.6 | 89.8 | 9.8 | 8.4 | 6.9 | 27.0 | 17.6 |
| IncR-3DSGG (Ours) | 39.5 | 88.6 | 14.5 | 28.3 | 23.1 | 52.2 | 30.8 |
| Instances | Geometry | Recall (R@1) | Annotated | Mean Recall | PosErr (m) ↓ | ||||
|---|---|---|---|---|---|---|---|---|---|
| Obj | Pred | Trip | Pred* | Trip* | mR.Obj | mR.Pred | |||
| EoMT | sparse map | 39.5 | 89.7 | 15.3 | 22.1 | 18.0 | 52.2 | 27.0 | 1.286 |
| EoMT | PRP (Ours) | 39.5 | 88.6 | 14.5 | 28.3 | 23.1 | 52.2 | 30.8 | 1.242 |
| GT | sparse map | 67.1 | 86.7 | 38.7 | 36.0 | 36.0 | 84.9 | 42.4 | — |
| GT | PRP | 68.4 | 82.8 | 36.7 | 52.1 | 52.1 | 85.3 | 55.3 | — |
| Method | Depth | 3RScan | AI2-THOR | |||||
|---|---|---|---|---|---|---|---|---|
| GTRec ↑ | NodePur ↑ | Frag. ↓ | PosErr (m) ↓ | NodePur ↑ | Frag. ↓ | PosErr (m) ↓ | ||
| ConceptGraphs-style | ✓ | 0.590 | 0.971 | 1.736 | 0.719 | 0.978 | 1.196 | 0.646 |
| SGFusion-style | ✓ | 0.710 | 0.929 | 1.717 | 0.764 | 0.964 | 1.758 | 0.591 |
| IncR-3DSGG (Ours) | — | 0.500 | 0.813 | 1.155 | 1.242 | 0.936 | 1.116 | 0.692 |
| Method | Random Camera Yaw | Non-Zero Fixed Rotation | ||||||
|---|---|---|---|---|---|---|---|---|
| Purity ↑ | Frag. ↓ | PosErr ↓ | Nodes | Purity ↑ | Frag. ↓ | PosErr ↓ | Nodes | |
| ConceptGraphs-style | 0.904 | 17.889 | 3.064 | 928.7 | 0.874 | 9.952 | 2.744 | 338.2 |
| SGFusion-style | 0.884 | 13.649 | 3.129 | 689.0 | 0.864 | 14.091 | 3.485 | 493.8 |
| IncR-3DSGG (Ours) | 0.883 | 1.132 | 1.330 | 54.0 | 0.843 | 1.186 | 0.861 | 42.2 |
| Method | MAE (m) ↓ | Scans Won |
|---|---|---|
| VGGT (oracle scale-corrected) | 0.610 ± 0.216 | 32 / 156 |
| PRP (Ours, direct metric) | 0.487 ± 0.140 | 124 / 156 |
| Module | Latency (ms) | Share |
|---|---|---|
| EoMT Segmentation | 1008.0 ms | 98.4% |
| PRP Inference | 2.3 ms | 0.2% |
| Hungarian Association | 1.4 ms | 0.1% |
| PGO Optimization | 12.5 ms | 1.2% |
| Total per View | 1024.2 ms | 100% |
In each example, the left panel shows the single-frame local scene graph $\mathcal{G}_L$ with nodes at panoptic mask centroids and edges predicted by PRP. The right panel displays the accumulated global 3D scene graph $\mathcal{G}_G$ with nodes placed at PGO-optimized metric coordinates. The textured scan mesh is displayed solely for visual reference and is never provided to our model.
68bae75f ($t=9$): Local scene graph $\mathcal{G}_L$ (left) and accumulated 3D global graph $\mathcal{G}_G$ (right).
7272e17c ($t=10$): Consistent multi-object relation estimation and 3D landmark localization.
c2d99345 ($t=8$): Structured spatial arrangement recovered directly from monocular RGB frames.
f2c76fe7 ($t=27$): Extended sequence accumulation over 27 views, maintaining low instance fragmentation.