IncR-3DSGG: Incremental Posed RGB Image Based 3D Scene Graph Generation via Privileged Depth Supervision

WACV 2027
1Machine Learning and Computer Vision Lab   •   2Organic Computing Group
Universität Augsburg, Germany
{mrunmai.phatak, daniel.kienzle, roman.kueble, joerg.haehner, rainer.lienhart}@uni-a.de
Incremental 3D Scene Graph Generation over 20 views

Figure 1: Incremental construction of the global scene graph $\mathcal{G}_G$ over 20 processed views. Nodes are instantiated when an object is first detected and refined as subsequent views contribute pairwise metric constraints. The 3RScan mesh is shown in the global frame purely for spatial reference, illustrating how predicted nodes align with annotated instances. Our method observes no depth and builds no point cloud or mesh, estimating 3D object positions and semantic relations directly from posed RGB observations.

Abstract

Scene graphs provide a compact, structured representation of environments by capturing objects and their spatial or semantic relationships in a form suitable for embodied agents. Existing 3D scene graph generation methods often rely on depth sensing at inference, either to associate object instances across views via point cloud overlap or to compute inter-object relations from geometric cues, limiting deployment to platforms with depth sensors. We propose IncR-3DSGG, which builds 3D scene graphs incrementally from posed* RGB images, eliminating the use of depth sensors at inference. Our method leverages a Parametric Relation Predictor (PRP) that learns to estimate pairwise distances and relative angles between objects from 2D visual features and 2D geometric cues, using 3D geometric supervision available only during training. These learned pairwise measurements are fused across views through Pose Graph Optimization (PGO) to recover metrically grounded 3D object positions and build a globally consistent 3D scene graph incrementally. On the real-world 3RScan dataset, our method attains $52.2$ mean object recall and $30.8$ mean predicate recall under the 3DSSG evaluation protocol, against $27.0$ and $17.6$ for MonoSSG, while using no depth or reconstructed geometry at inference. This shows that relations and metrically grounded 3D object representations can be predicted from posed RGB images alone.

* RGB images with camera extrinsics estimated by an off-the-shelf visual SLAM system.

Method Overview

IncR-3DSGG constructs a persistent 3D scene graph $\mathcal{G}_G = (\mathcal{V}_G, \mathcal{E}_G)$ incrementally from an exploration sequence of posed RGB images $(I_t, T_t)$. The framework operates in two coupled stages: per-view local scene graph generation ($\mathcal{G}_L$) and incremental global graph aggregation.

IncR-3DSGG pipeline overview

Figure 2: Overview of the IncR-3DSGG framework. Given an RGB image $\mathcal{I}_t$ and camera pose $\mathcal{T}_t$, a frozen EoMT panoptic backbone detects per-object nodes $\mathcal{V}_L$ and computes relative 2D geometry. The Parametric Relation Predictor (PRP) converts co-visible object pairs into edges $\mathcal{E}_L$ carrying 3D pairwise geometry and support predicates. Local detections are merged into the global scene graph $\mathcal{G}_G$ via visual Hungarian matching, and object positions are globally recovered via Pose Graph Optimization (PGO).

$$\mathbf{p}^* = \arg\min_{\{\mathbf{p}_i\}} \sum_{(i,j) \in \mathcal{C}} \rho_H\!\left(\sqrt{w_{ij}}\,\mathbf{r}_{ij}\right) + w_a \,\| \mathbf{p}_a - \mathbf{p}_a^0 \|^2$$

where constraint weights $w_{ij} = \gamma_{ij}/(1 + e_i + e_j)$ penalize large object extents ($e_i, e_j$) and leverage PRP's predicted confidence $\gamma_{ij}$.

Parametric Relation Predictor (PRP)

Inferring 3D inter-object geometry from monocular 2D images is under-constrained. We decouple the training supervision from the inference modality: PRP is trained using privileged 3D supervision (depth maps and point clouds available only during training) but requires strictly monocular RGB at inference.

Parametric Relation Predictor Architecture

Figure 3: Architecture of the Parametric Relation Predictor (PRP). Left (Per-Object Encoding): Mask-pooled visual features and category embeddings are contextualized through an unordered Transformer encoder. Right (Pair Construction & Heads): For every directed pair $(i,j)$, contextualized tokens $h'_i, h'_j$ are concatenated with their difference, product, and projected 10-D 2D relative geometry $g'_{ij}$. Dedicated heads predict 3D centroid geometry, contact/containment features, support relation class, pairwise confidence, and object extents.

The pairwise input vector $g_{ij} \in \mathbb{R}^{10}$ is computed purely from 2D masks and boxes:

$$g_{ij} = \big[\, \Delta x,\; \Delta y,\; \log\tfrac{a_s}{a_o},\; \text{IoU}_\text{box},\; \Delta y_v,\; \Delta x_h,\; \sqrt{\tfrac{a_s}{a_o}},\; r_\text{enc},\; d_\text{2d},\; \text{IoU}_\text{mask} \,\big]$$

Experimental Results

Table 1: 3D scene graph generation on 3RScan (Sparse, l20, 157 test scans, official sets). Baselines read geometry from the ORB-SLAM3 point map, whereas ours uses no depth or reconstructed geometry at inference. Instances come from each method's own front end (ours: EoMT). Pred*, Trip* scores annotated pairs only.
Method Recall (R@1) Annotated Mean Recall
Obj Pred Trip Pred* Trip* mR.Obj mR.Pred
IMP 27.5 90.8 8.5 4.5 3.9 20.4 14.0
VGfM 27.7 90.6 8.8 4.3 3.9 24.5 13.9
3DSSG 10.8 90.1 1.3 4.1 1.0 6.0 13.7
SGFN 15.4 88.3 2.4 7.0 3.3 8.6 15.5
MonoSSG 28.6 89.8 9.8 8.4 6.9 27.0 17.6
IncR-3DSGG (Ours) 39.5 88.6 14.5 28.3 23.1 52.2 30.8
Table 2: Instance source vs. geometry source on 3RScan, with all other components fixed. Sparse map replaces PRP's pairwise geometry with the ORB-SLAM3 point map. Position error is reported for predicted instances only.
Instances Geometry Recall (R@1) Annotated Mean Recall PosErr (m) ↓
Obj Pred Trip Pred* Trip* mR.Obj mR.Pred
EoMT sparse map 39.5 89.7 15.3 22.1 18.0 52.2 27.0 1.286
EoMT PRP (Ours) 39.5 88.6 14.5 28.3 23.1 52.2 30.8 1.242
GT sparse map 67.1 86.7 38.7 36.0 36.0 84.9 42.4 —
GT PRP 68.4 82.8 36.7 52.1 52.1 85.3 55.3 —
Table 3: Object localization and association. Baselines use our EoMT masks with depth back-projection. Ours uses posed RGB only. Position error in metres.
Method Depth 3RScan AI2-THOR
GTRec ↑ NodePur ↑ Frag. ↓ PosErr (m) ↓ NodePur ↑ Frag. ↓ PosErr (m) ↓
ConceptGraphs-style ✓ 0.590 0.971 1.736 0.719 0.978 1.196 0.646
SGFusion-style ✓ 0.710 0.929 1.717 0.764 0.964 1.758 0.591
IncR-3DSGG (Ours) — 0.500 0.813 1.155 1.242 0.936 1.116 0.692
Table 3: Evaluation under viewpoint changes on AI2-THOR. Imperfect mask boundaries bleed background depth points into projected point clouds, causing severe over-segmentation in depth-based methods under camera rotations.
Method Random Camera Yaw Non-Zero Fixed Rotation
Purity ↑ Frag. ↓ PosErr ↓ Nodes Purity ↑ Frag. ↓ PosErr ↓ Nodes
ConceptGraphs-style 0.904 17.889 3.064 928.7 0.874 9.952 2.744 338.2
SGFusion-style 0.884 13.649 3.129 689.0 0.864 14.091 3.485 493.8
IncR-3DSGG (Ours) 0.883 1.132 1.330 54.0 0.843 1.186 0.861 42.2
Table 4a: Pairwise distance accuracy vs. VGGT across 156 scans (64,254 pairs).
Method MAE (m) ↓ Scans Won
VGGT (oracle scale-corrected) 0.610 ± 0.216 32 / 156
PRP (Ours, direct metric) 0.487 ± 0.140 124 / 156
Table 4b: Per-view runtime breakdown on NVIDIA RTX 3090.
Module Latency (ms) Share
EoMT Segmentation 1008.0 ms 98.4%
PRP Inference 2.3 ms 0.2%
Hungarian Association 1.4 ms 0.1%
PGO Optimization 12.5 ms 1.2%
Total per View 1024.2 ms 100%

Qualitative Results

In each example, the left panel shows the single-frame local scene graph $\mathcal{G}_L$ with nodes at panoptic mask centroids and edges predicted by PRP. The right panel displays the accumulated global 3D scene graph $\mathcal{G}_G$ with nodes placed at PGO-optimized metric coordinates. The textured scan mesh is displayed solely for visual reference and is never provided to our model.

3RScan qualitative result frames 0 to 10
3RScan sequence (Frames 0–10): Incremental construction after 10 views. Edges denote support relations; nodes are grounded in 3D without runtime depth sensing.
Scene 68bae75f qualitative result
Scene 68bae75f ($t=9$): Local scene graph $\mathcal{G}_L$ (left) and accumulated 3D global graph $\mathcal{G}_G$ (right).
Scene 7272e17c qualitative result
Scene 7272e17c ($t=10$): Consistent multi-object relation estimation and 3D landmark localization.
Scene c2d99345 qualitative result
Scene c2d99345 ($t=8$): Structured spatial arrangement recovered directly from monocular RGB frames.
Scene f2c76fe7 qualitative result
Scene f2c76fe7 ($t=27$): Extended sequence accumulation over 27 views, maintaining low instance fragmentation.
AI2-THOR qualitative comparison under rotation
AI2-THOR Merging Comparison: Recovered instances under camera rotation for ConceptGraphs-style, SGFusion-style, and Ours (PGO).

Citation

citation coming soon