AI News Feed
Market watch
Intel Says It Has Passed One Million High-NA EUV Wafers, More Than Rest of Industry Combined9to5Mac Overtime Episode 081 Shares First Impressions of Apple's Foldable iPhone DuoFairbuds 2 leak shows stem-style design and taller charging caseRandom Rewards Enrich Classic Game-Theory ContestsMRSA outbreak tied to bouncy castles infects 48 children in IrelandNew Mexico Supreme Court Holds Lawyer in Contempt Over ChatGPT Brief With Fabricated Testimony
Intel Says It Has Passed One Million High-NA EUV Wafers, More Than Rest of Industry Combined9to5Mac Overtime Episode 081 Shares First Impressions of Apple's Foldable iPhone DuoFairbuds 2 leak shows stem-style design and taller charging caseRandom Rewards Enrich Classic Game-Theory ContestsMRSA outbreak tied to bouncy castles infects 48 children in IrelandNew Mexico Supreme Court Holds Lawyer in Contempt Over ChatGPT Brief With Fabricated Testimony
Large Language Models

ByteDance Seed's HarnessDev Tests Whether LLMs Can Build Their Own Agent Harnesses, With Only 34 of 64 Changes Generalizing

Researchers from ByteDance Seed and partners introduced HarnessDev, which evaluates the agent harness an LLM builds rather than its answers. Self-built harnesses beat references on writing and ML tasks but trailed on code and search, and only 34 of 64 changes generalized to held-out tests.

AI News Feed · 2026-09-12 07:53

The method has two stages, Creation and Evolution. In Creation, every creator receives the same weak seed: passive file, search, and process primitives plus result and trajectory writers, with no loop, planner, verifier, retry, or stopping rule. Unmodified, that seed scores 0 everywhere. The creator gets a task-family specification, a short design tutorial, and one to three development cases, then builds a full harness, which is frozen before hidden tasks are run. In Evolution, the creator starts from its own frozen Creation code harness and revises it using execution feedback from a fixed set of 100 SWE-bench Pro tasks and all 89 Terminal-Bench 2.1 tasks. Each official candidate must complete both evaluations as a pair, with a budget of 10 pairs and at most two five-task probes between pairs. Every official version is later scored on 630 held-out SWE-Pro instances the creator never sees. Harnesses are graded on capability, or task success, and efficiency, measured by executor tokens, with creator tokens excluded.

Six creator LLMs were tested: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro, working inside Claude Code 2.1.177, while GPT-5.5 used Codex 0.144.3. Creation spanned four domains and five benchmarks totaling 2,207 instances: SWE-bench Pro public split with 731, Terminal-Bench 2.1 with 89, MLE-bench with 75, EQ-Bench3 with 46, and BrowseComp with 1,266. Each creator built three harnesses per benchmark, reported as avg@3. Self-Eval ran each harness with its creator, while Unified-Eval ran all harnesses with Gemini 3.1 Pro.

Under Self-Eval, Opus 4.8 posted the highest average score at 67.8 against a human-engineered reference of 86.2. The gap depended on domain. In code, Opus 4.8 reached 69.3 on SWE-Pro versus the 80.0 reference. Gemini 3.1 Pro led Terminal-Bench at 68.8 versus 88.8. Search showed the widest gap, with the best BrowseComp score at 52.6 from GPT-5.5 against a 92.2 reference. In writing, Opus 4.8 scored 84.6 on EQ-Bench3, above the 83.7 reference. In ML experimentation, Opus 4.8 at 32.9 and Gemini at 32.4 beat the 24.0 MLE-bench reference. The SWE-Pro, Terminal-Bench, and BrowseComp references are external results from OpenAI's GPT-5.6 report, not re-runs.

Code volume did not predict quality. The 18 code harnesses added 17,111 net lines, yet Gemini added the fewest, 1,006, and led Terminal-Bench. Self-test count barely correlated with score, with Spearman values from 0.13 to 0.26, while revision calls reached 0.57. Much generated machinery was inert. Of 108 code component instances, 72 triggered in real runs and 18 never fired, all of them state and memory. Eleven of 18 harnesses defined a State class, yet no checkpoint event appeared across 26,679 trajectories. In writing, 124 of 587 features were dead code.

Cost and executor transfer also varied. MLE-bench token use varied roughly 19-fold: GPT-5.5 hit a 19.1 medal rate with 29.3 million tokens, while DeepSeek V4 hit 19.6 with 208.4 million. Swapping the executor to Gemini reshuffled rankings. Qwen gained 17.6 points on BrowseComp and 12.9 on MLE-bench, while Opus 4.8's SWE-Pro score fell from 69.3 to 33.0, partly because one harness hard-coded a 120-step limit around its original executor. The Opus search harness's duplicate-query rate jumped from 10.1% to 88.2% after the switch.

In Evolution, nine lineages, five self-runtime and four fixed-Gemini, produced 73 official versions and 64 adjacent switches. All five self-runtime creators improved on held-out tasks, from 1.43 to 4.44 points, with a mean gain of 3.11. Under fixed Gemini, only Opus improved, while GPT-5.5 regressed 10.32 points. Progress was not monotonic. Of 64 switches, eight regressed on both benchmarks, 16 regressed on one, 27 gained only within the noise band, and two showed clear positive evidence. A single commit could vary by about plus or minus 4.75 pair-score points. Feedback and held-out scores moved in the same direction only 34 of 64 times, or 53.1%, and only two of nine declared final versions were held-out optimal. Of 169 new functions or classes, 25 had no caller.

The clearest win came when Opus 4.8 noticed that 99 of 100 runs reported success while only 48 passed, traced the problem to premature completion, and added a completion gate. Failure diagnosis was otherwise the weakest step: the dedicated trajectory interface was called only twice. HarnessDev's main takeaways are that it scores the harness a model builds, not the answer it returns; self-built harnesses match or beat references on writing and ML experimentation but trail badly on code and search; and harness quality is executor-specific.