Tsinghua-Zhipu paper reveals GLM-Omni, highlighting image-to-language gap

The gap was sharper in a number task. GLM-Omni correctly rendered a specified number on a jersey 72% of the time; when researchers took the image away and asked for the jersey number in text, accuracy was 2%. Across the three tested models, BAGEL averaged 31.43% visual correctness and 8.86% text accuracy; GLM-Omni 54.00% and 14.57%; SenseNova-U1 61.14% and 20.00%. SenseNova-U1 reached 88% visual accuracy on the number task but 0% in direct text answers.

To test whether generation training taught new knowledge, the researchers built a fictional world, MagicWorld, with entities that do not exist and assigned them colors, numbers and spatial relations. The dataset covered seven task types—color, quantity, shape, spatial arrangement, relative size, number and text—with 50 knowledge items each, 350 in total. After generating images from different backgrounds and viewpoints and filtering them, the team kept 8,552 training samples. Text prompts were checked to limit answer leakage and uneven attribute distributions. The models were tested by regenerating an image and checking the attribute, or by answering a text question without the image.

In a Textual Alignment experiment, 25 of 50 entities in each class received 10 text Q&A examples after image-generation training, while the other 25 were held out. BAGEL's average text accuracy rose from 8.86% to 15.43%; SenseNova-U1 rose from 20.00% to 28.00%; GLM-Omni fell from 14.57% to 12.57%. SenseNova-U1's shape score rose from 20% to 68% and its relative-size score reached 100%, but all three models remained at 0% on test entities for number and book-title questions. The report attributes the split to transferable rules for shape and size versus entity-specific character sequences for numbers and titles.

The paper then tested Interleaved Image-Text Alignment, placing image generation and text answering in one training sequence: the model generates the relevant image and then uses the image to form a text answer. GLM-Omni's average text accuracy rose from 12.57% after textual alignment to 26.86%; its shape score rose from 12% to 64% and spatial arrangement from 0% to 40%. BAGEL reached 17.14% and SenseNova-U1 28.57% on average. Some color and quantity tasks declined. In an ablation on GLM-Omni, keeping the interleaved responses and text supervision but masking the image loss in interleaved samples cut average text accuracy from 26.86% to 13.71%, indicating that image-reconstruction signals affected language recall. The paper says the experiments did not locate which parameters share the information.

Visual Chain-of-Thought allowed the model to generate a target image during reasoning, read the attribute and then answer. GLM-Omni's average accuracy rose from 26.86% to 41.14%; BAGEL from 17.14% to 20.57%; SenseNova-U1 from 28.57% to 34.86%. On the number task, direct text accuracy had been 0% for all three; with Visual CoT it was 4%, 36% and 20%, respectively. In one case cited by the paper, SenseNova-U1 answered 69 when asked directly for a jersey number; after generating the jersey and reading it, the model answered 31. The method adds computation, and errors in the generated image can lead the answer astray. Book-title tasks showed no improvement, and the experiments used controlled fictional knowledge, leaving real-world dynamic scenes untested.

The paper describes GLM-Omni as starting from the GLM-4.1V-Thinking pretrained model and connecting to a discrete image tokenizer based on X-Omni. It does not say whether GLM-Omni will appear in a future product version. The reported results leave a technical issue for unified multimodal models: knowledge learned through image generation is not automatically available for text answers, and training methods that connect the two paths improve some tasks while leaving others unresolved.