ALTK-Evolve lifts GPT-4.1 customer-service agent to 67.5% Pass^4 on τ²-bench retail, topping October leaderboard

The agent runs on τ²-bench's own implementation of a basic ReAct scaffold with a two-phase THINK → ACT loop, using gpt-4.1 as the agent model and gpt-5.5 as the customer-simulator model, with four trials per task. The only modification is a block of learned guidelines appended to the system prompt after the domain policy as a plain markdown list. The policy, tools and scaffold are otherwise stock. No weight updates or human annotation were involved.

ALTK-Evolve builds that playbook with GPT-4.1 by analyzing the agent's attempts on training tasks and using the model to distill successes and failures into short behavioral rules, such as how to authenticate a customer, confirm a transaction or handle a return. Those guidelines are then added to the agent's prompt for subsequent conversations. The same model powers both guideline extraction and the agent itself. Examples given in the post include authenticating a user before any order-related action and falling back to name and zip code verification when email is unavailable, confirming all intended changes with a structured summary and explicit user approval before any database-updating action, providing an itemized refund breakdown and timing without waiting to be asked, taking one action per step rather than combining a tool call with a user-facing message, and cleaning a combined handle into first and last names before lookup.

τ²-bench, from Sierra, measures how well an agent handles customer-service conversations under real constraints. Each task pairs the agent with an LLM-simulated customer who has a goal, a set of domain tools such as looking up an order, issuing a refund or rebooking a flight, and a written policy document the agent must follow. The agent has to draw out the right details, stay within policy and take the correct, often irreversible actions; a task counts as solved only when the final database state matches the expected outcome, a strict binary reward. The work uses the benchmark's retail and airline domains.

Because different models and tasks require different amounts of guidance, the post also evaluates how many guidelines to supply. Each guideline carries a support count, the number of distinct training tasks it was extracted from, so a higher support count marks a rule that generalized across more tasks rather than a one-off. Three configurations were compared: a baseline with no guidelines; sup3, keeping only guidelines with support of at least three, which retained 31 guidelines for retail and 14 for airline; and lossless, keeping every guideline retained after consolidation, which came to 292 for retail and 101 for airline. Guidelines were extracted from the agent's trajectories on τ²-bench training tasks using gpt-4.1 and then applied unchanged on the held-out test set.

Two metrics were tracked across the four trials per task. Mean Accuracy is the average success rate across the four trials, described as the capability axis. Pass^4 is the percentage of tasks that passed across all four trials, described as the reliability axis and the figure the τ²-bench leaderboard reports.

The post follows earlier work in which the same approach was demonstrated on coding agents. The customer-service setting was chosen because success there requires following domain policies, understanding a customer's request and executing the right tool actions.

Editor's Summary

ALTK-Evolve uses GPT-4.1 both to extract short behavioral guidelines from an agent's training-task attempts and to run the agent, reaching 67.5% Pass^4 on τ²-bench retail on a stock ReAct scaffold. The result places the agent above every retail entry in the October 12, 2026 leaderboard snapshot without a frontier model or weight updates. The post reports guideline counts for three dosage configurations but does not give airline results in the excerpt.