Meta's Muse Wonder Draws Benchmark Scrutiny After Lava Lamp Test

Meta released Muse Wonder and Muse Wonder Full on Oct. 12, promoting their Coding Agent, tool-calling and long-task execution abilities. Meta reported that Wonder Full scored 57.5 on SWE-Marathon and 79.0 on DeepSWE 1.1, results that put it in contention with contemporary frontier models.

The test came from developer community BridgeMind, which asked Muse Wonder to create a lava lamp with a transparent glass shell, dynamic lava and lighting effects. The model produced a red cylinder. BridgeMind questioned whether Muse Wonder had undergone severe “Benchmaxxing,” or heavy optimization for benchmarks that looks strong on known tests but falters on unfamiliar tasks. Elon Musk also commented on a related post with “Hmm,” a one-word response that added attention to the dispute.

Alexandr Wang, Meta's chief AI officer, responded that the team had not explicitly trained the model for 3D demo games and that, in hindsight, it might have been worth adding such training. He said he still hoped developers would try Muse Wonder's coding abilities. Meta's Model Card already lists visual and spatial reasoning, especially 3D tasks, as a known weakness, according to Leiphone. The lava lamp test used the standard Muse Wonder, while some of Meta's highlighted scores came from Wonder Full, a capability difference the report said is important to the debate.

The episode points to how post-training shapes what a model can do. Coding Agents are well suited to reinforcement learning because tasks such as fixing a bug in a repository can produce clear feedback through compilation results, unit tests and functional checks. A lava lamp involves 3D geometry, transparent materials, lighting calculations and dynamic deformation. Code can run while the rendered image is still wrong; compilers and unit tests cannot directly judge whether glass looks transparent or whether lava changes shape. The model would need a visual feedback mechanism to inspect screenshots, identify errors and turn them into code changes. Wang's admission that the task was not explicitly trained for indicates it did not receive the same investment as key coding scenarios, Leiphone said.

The benchmark debate also involves system-level engineering. In SWE-Marathon, an Agent must read files, call terminals, modify code and verify results across long software-engineering tasks. A Harness, the execution framework around the model, manages tool interfaces, context, task state and exceptions. Better context compression, error recovery and inference budgets can improve scores. Meta's Model Card said the SWE-Marathon comparison was not a controlled test under a uniform Harness: GPT-6 Astra used Codex and Claude Fable 5.1 used Claude Code. Wonder Full's 57.5 score also used the xhigh inference setting. Those differences do not invalidate the score, but they affect how it should be interpreted.

Meta's technical article included a section called “Capabilities That Generalize” that discussed the problem behind Benchmaxxing. Meta said two models with similar scores on common benchmarks can differ on unfamiliar tasks, and it compared Muse Spark 1.3 and Muse Wonder Full under different inference budgets. Meta reported that Wonder Full's performance on unfamiliar tasks improved relative to the prior model's trend, attributing it to a stronger base model. The report said the remaining question is how unfamiliar the out-of-distribution tasks really are; changing to an unseen repository is still close to software engineering, while a 3D lava lamp adds spatial modeling, material rendering and visual quality judgment.

Leiphone argued that a single lava lamp failure only shows the model did not complete that task and cannot by itself prove severe benchmark overfitting. It said more systematic tests should change task distribution under the same execution conditions: first fix bugs in an unfamiliar repository, then build a program with an unseen API, then add a visual development task without a clear test script. Evaluators could also change tool interfaces, introduce new error messages and record whether an Agent uses screenshots and corrects material or geometry errors. As public benchmarks mature, dynamic tasks, hidden tests and fuller execution traces will be needed to supplement fixed leaderboards. The failure does not erase Muse Wonder's progress on software-engineering tasks, but it shows that a high score reflects performance under specific conditions, not the full boundary of the model's ability.