At ECCV 2026, Tan Ping Says 3D Geometry Can Strengthen Visual Large Models

In his talk, titled "Geometry Strikes Back: Scaling 3D Reconstruction and Structuring 3D Scene Generation," Tan framed 3D reconstruction and 3D scene generation within one problem: learning models already provide strong local visual priors from large-scale 2D data, but they struggle when tasks require multi-view, long-distance 3D consistency. Diffusion models such as Sora have advanced image and video generation, yet existing visual large models can produce inconsistent structures across views, unreasonable object counts or layouts, multiple beds for a "bedroom" prompt, or distorted buildings during long virtual camera roaming. Tan argued that explicit 3D geometry can organize local predictions and enforce cross-view constraints.

On reconstruction, Tan pointed to feed-forward Transformer methods such as VGGSfM and DUSt3R, which take images and directly estimate camera poses, depth maps and point clouds. These methods are efficient and robust but have a computational bottleneck: on consumer GPUs they usually cannot process more than 100 images at once. His team borrowed the keyframe idea from visual SLAM and SfM. It samples anchor frames, feeds them into a feed-forward network for initial 3D structure, and uses token sampling that randomly keeps 20 percent to 50 percent of tokens per image after inter-frame attention. Those tokens are aggregated into a single global scene representation. Non-key frames then interact with this representation through inter-frame attention to obtain pose, depth and point cloud. The method handled larger image sets and produced dense point clouds and camera trajectories with fewer ghosting artifacts caused by inaccurate poses. On Tanks and Temples, about 70 percent of cases had relative rotation and translation errors within 5 degrees.

Tan also questioned using COLMAP poses as ground truth, saying COLMAP has estimation errors. His team estimated poses from 90 percent of input images, trained a NeRF model, and computed PSNR on 10 percent held-out views. The report said their method outperformed COLMAP-pose training in rendering quality and PSNR.

A second effort, Global 3R, aims to combine the robustness and efficiency of feed-forward networks with the geometric accuracy of global SfM. It first extracts sliding windows. Inside each window, a feed-forward network such as CroCo or RoMa computes relative motion and point clouds between image pairs. The method selects keyframes and aligns them with other frames in the window to obtain consistent feature tracks. It builds a pose graph, performs rotation averaging and translation averaging, and then bundle adjustment. No fixed reference frame is required; any local window can serve as reference. After relative motion and keyframe selection, other frames are deformed to the keyframe and a confidence score is computed to filter high-quality matches. Bundle adjustment markedly improves point cloud quality: motion averaging alone causes ghosting, while BA aligns features and produces a consistent 3D point cloud and camera trajectory. On Tanks and Temples, the method achieved the highest PSNR on the held-out 10 percent images, surpassing COLMAP and other advanced feed-forward baselines. On ETH3D at a 1-degree threshold, it achieved high rotation accuracy, reaching 100 percent in several scenes. Error analysis showed initial network predictions put about 70 percent of image pairs within 1 degree relative rotation, motion averaging improved this only slightly, and BA raised it above 90 percent. For relative translation, motion averaging reached about 35 percent, and BA doubled it to more than 70 percent.

For 3D scene generation, Tan described using 3D geometry as a constraint to stitch local generated views into a complete 3D world. ControlRoom, inspired by Text2Room, adds explicit layout generation and decouples the process into layout generation and appearance generation. First, a diffusion model generates a 3D room layout of bounding boxes and semantics from text and renders a semantic panorama. Second, the semantic layout acts as a ControlNet signal to guide high-fidelity panorama generation for later 3D visualization. The model encodes object coordinates, size and orientation as vectors and concatenates them into a unified code sequence. It is trained on real structured data and uses visual priors to learn common room arrangements, such as a bedroom usually containing a bed. A random circular shift is introduced during diffusion to improve left-right boundary consistency. Comparisons showed that without explicit layout constraints a bedroom could contain multiple beds, while layout guidance produced more reasonable furniture configurations. ControlRoom still generated only fixed panoramas with blind spots and did not allow true free 3D roaming, and existing 3D datasets such as Structured3D, with only 20,000 rooms, limited scene and object richness.

SpatialGen responded to the data bottleneck with a dataset an order of magnitude larger: 12,000 scenes, 57,000 rooms, about 1 million bounding boxes across 62 object categories, and about 5 million rendered images containing six modalities including RGB, semantic and instance segmentation, albedo, depth and normal maps. Based on this dataset, SpatialGen uses a layout-guided multi-view multimodal diffusion model. It takes a 3D layout and a reference image or text as input and jointly trains to generate RGB images, semantic images and 3D absolute geometry encoded by scene coordinate maps. This spatial information is important for multi-view and modal consistency. Images from different views are iteratively fused into a 3D Gaussian model. When inferring new local views, the system projects the 3D Gaussian model for reverse guidance. The method can build high-quality 3D scenes and support style transfer, generating rooms in European, Chinese or cyberpunk styles under the same 3D layout. However, requiring an explicit 3D layout is inflexible in practice because users usually provide only a single image, and the approach is limited to structured indoor scenes.

SpatialCraft further targets those limitations with a two-stage strategy. The first stage generates a 3D proxy from a single image. Tan's talk emphasized complementarity rather than choosing between learning and geometry: learning models capture strong local priors from large-scale data, while 3D geometry provides global structural constraints for cross-view and long-distance scenes. He said this combination deserves further exploration for spatial computing, world models and embodied intelligence.