Google Research Releases ToolGrad: Answer-First Framework Hits 99.8% Pass Rate for Tool-Use Data Generation

Prior pipelines such as ToolBench and ToolACE follow a query-first recipe. They sample a pool of APIs, ask an LLM to invent a plausible user instruction, and then dispatch a depth-first search agent to find a tool-use path that satisfies it. The search has no guarantee of success. When it dead-ends, the compute spent on exploration is wasted and the sample is discarded. The paper frames this as distilling valuable trajectories from a complex and often failing agent exploration, which is inherently inefficient. ToolGrad constructs a ground-truth tool-use chain by actually executing APIs, then annotates that chain with a matching user query. An explicit, working chain is less ambiguous than a hypothetical prompt, so the chain-to-query step takes a single LLM call.

Each iteration runs four modules in sequence. API Proposer narrows a sampled set of APIs down to a few candidates that could extend the current workflow. API Executors run those candidates in parallel and produce detailed execution reports. API Selector reviews the reports, picks the single best-performing call and appends it to the workflow. Its directional feedback is described as the textual gradient. LLM Updater rewrites the synthetic user query and AI response so they match the new API set. Repeating the loop yields one sample: a user query, a verified API workflow and the final response. The repository’s default configuration runs 10 iterations over 50 sampled APIs per workflow.

The research team evaluated data generation on the ToolBench API database, which contains more than 16,000 real-world APIs, and compared ToolGrad against ToolBench’s DFS-based query-first approach. According to the research paper, the pass rate rose from 63.8% to 99.8%. Ground-truth tool uses per sample rose from 2.1 to 3.4, meaning longer chains. Tool-use steps per sample fell from 34.3 to 20.0. LLM invocations per sample fell slightly, from 64.5 to 63.9. The 0.2% failure case occurred when the agent could not get a successful response from three selected APIs across all 10 iterations and saved an empty sample.

The researchers generated ToolGrad-500, a 500-sample dataset built with Gemini 2.5 Flash-Lite, and used it to post-train Gemma-3 at 1B, 4B and 12B parameters. They evaluated on the Berkeley Function Calling Leaderboard, which uses a tool set that differs from ToolBench, making it an out-of-distribution test with unseen tools. Fine-tuning on ToolGrad-500 improved tool-use scores at every parameter size. ToolGrad-12B scored 83.1, compared with Gemini 2.5 Pro at 83.2, Claude 4.5 Opus at 82.8 and GPT-5 at 74.4, as measured at the time of publication. The 12B student outperformed Gemini 2.5 Flash-Lite, the teacher model that generated its training data. ToolGrad-12B led open tool-use specialists including ToolACE and Hammer-2.1-7B.

The repository’s reproduction scripts target BFCL V1 and V2 through a customized fork, run inference in a vLLM Docker image, and were verified on a single NVIDIA A100 40GB.

Editor's Summary ToolGrad shifts tool-use data generation from query-first search to verified-chain-first annotation, reporting a 99.8% pass rate on ToolBench and longer ground-truth chains. A 500-sample dataset fine-tunes Gemma-3 models to 83.1 on the Berkeley Function Calling Leaderboard, close to Gemini 2.5 Pro and ahead of several open specialists. The Apache-2.0 release includes code, Hugging Face models and a PyPI package.