Tianjin University’s Zhang Changqing presents trusted multimodal fusion research at CNCC 2026

Multimodal fusion is meant to combine complementary information, but adding modalities does not automatically improve accuracy because new data can be noisy or misleading. Zhang contrasted joint representation and coordinated representation. Methods such as canonical correlation analysis map different modalities into a correlated space, but overemphasizing shared information can weaken modality-specific evidence. In medical diagnosis, pathogen testing and CT imaging provide complementary rather than substitutable information. Zhang’s team explores latent representations reconstructed across modalities instead of only aligning them, an approach he linked conceptually to neural radiance fields while noting the tasks and mechanisms differ.

The representation work is relevant to missing data. Patients may have CT scans but no complete laboratory results, and requiring imputation for every missing pattern can be difficult and unreliable. The team’s latent-representation approach tries to model with available observations without manually completing each missing case. Leiphone reported that this line of work targets a persistent problem: real multimodal data are often incomplete.

For decision-making, Zhang said the value of a modality can change by sample and environment. He used a mosquito bite outdoors to illustrate how a person may rely on touch in dim light and vision in bright light. Static weights based on external quality indicators such as illumination may not reflect a modality’s task-specific discriminative value. The team’s Trusted Multi-View Classification with Dynamic Evidential Fusion, or TMC, uses a variational Dirichlet distribution and Dempster-Shafer evidence theory to fuse evidence from different views and estimate uncertainty. The method asks how much evidence supports a prediction, whether sources agree and whether current evidence is sufficient.

Dynamic evidence fusion was also applied to crystal structure identification from electron microscopy. Different observation angles can provide very different amounts of information, and fixed weights or simple voting may let low-value observations overwhelm useful evidence. The dynamic method updates judgments as observations arrive and can stop collecting data when uncertainty reaches a set threshold. In experiments shown in the talk, classification accuracy rose from about 58–59% to around 93.9%, though the source said this result should be read with the paper’s experimental setup and not generalized to all multimodal tasks.

Zhang introduced the Fusion Anything Model, or FAM, as a generalized multimodal prediction foundation model. Conventional multimodal models are often designed for fixed modality combinations and tasks, requiring new algorithms, data preparation and training for new scenarios. FAM aims to pretrain transferable multimodal association patterns and use context examples at inference without task-specific parameter updates. To generate diverse training tasks, the team used Structural Multimodal Causal Models to synthesize data by varying latent information, shared information, modality-specific information and their relations. FAM uses intra-modality, cross-modality and sample-level attention. It relies on in-context learning: labeled examples provide task information, but the model does not need additional fine-tuning for each task.

The paper evaluation covered 18 real datasets, 12 modalities and 11 prediction task types. Leiphone reported that FAM’s average accuracy was about 6.9 percentage points higher than compared specialized fusion baselines, but performance varied across datasets and the model did not outperform others in every test. Zhang also showed preliminary tests. In embodied intelligence, FAM was used for visuotactile grasp-success prediction and performed close to a specialized method; the source noted that predicting grasp success differs from executing a grasp. In a reinforcement-learning scoring task using text instructions and video observations, a scheme with FAM outperformed the compared baseline in results displayed on slides. In medicine, the team tested multimodal skin disease diagnosis with Xiangya Hospital’s dermatology department using clinical images, dermatoscopy images and medical history, and compared results with methods including PanDerm. The team also demonstrated historical manuscript recognition using visual information, context and other features.

The source said these are preliminary results and should not be taken to mean FAM surpasses specialized models in all settings. Open questions include post-training, domain adaptation, the choice of modality encoders and reliable use of general prediction capabilities in real robot systems.