Sun Yat-sen Researchers Propose SpatialSV to Strengthen Multimodal LLMs’ Spatial Intelligence

Traditional approaches to giving MLLMs 3D perception either rely on external depth models during inference, adding latency and error, or distill features from 3D foundation models without explicit geometric constraints. SpatialSV instead extracts features from intermediate hidden layers of the LLM and maps them into a unified 3D feature space. Lightweight DPT heads then predict depth maps, ray maps, and point clouds, which together build an explicit 3D topology within the model. These auxiliary heads and geometric losses are used only during training; at inference time they are removed, so no extra computation is introduced.

The research group analyzed eight mainstream MLLMs, including Qwen2.5-VL, InternVL3 and LLaVA series. They found a strong negative correlation between 3D reconstruction error and spatial question answering accuracy: the more accurately a model reconstructs 3D geometry in its hidden layers, the higher its final accuracy. When test samples were grouped by reconstruction quality, accuracy rose stepwise from the worst group to the best. That means a model’s failures can be inferred from its internal point cloud quality, even without looking at its text outputs.

The paper also diagnoses why some models fail. In a sample case, Qwen2.5-VL-7B correctly answered that the object to the right of a blue office chair was a single sofa and a nearby table, while LLaVA-OneVision-8B chose a blue trash bin. By rendering the hidden-layer features as a 3D point cloud, the team saw that LLaVA had not modeled the small table at all. Another failure mode is loss of spatial anchor objects, such as a chair used as a reference for view changes. Visualizing the reconstructed geometry can therefore help locate whether a model’s defect lies in geometry perception, camera pose calibration, or spatial alignment.

SpatialSV decouples geometric supervision from natural-language question answering. Because depth maps, ray maps, and point clouds are visual signals, they do not require human-written text. The team used open-source 3D models such as DepthAnything-v3 and VGGT to automatically generate geometric labels from raw images. In controlled experiments on Qwen2.5-VL-3B, using only 50% of human text annotations caused accuracy to drop from 55.3% with full annotations to 47.2%. But when the other 50% of images received automatically generated 3D supervision, accuracy recovered to 53.9%, close to the fully labeled level and 14.19% higher than the 50%-text baseline.

According to the report, the method also offers an economical path for embodied intelligence. It forces the model to build stable 3D representations instead of relying on text-only autocomplete, and the visualizations provide a diagnostic tool for developers. The authors argue that injecting explicit geometric constraints into internal representations may be a practical way toward real spatial intelligence.