ServiceNow CoreAI Introduces AutoSynthData for Enterprise Agent Training

The post describes why enterprises need agents that work in their own environments. The work agents are asked to do is shaped by the systems they use, the rules they follow and the state of their data, ServiceNow CoreAI said. A broadly capable model can still struggle with a particular workflow, misuse a combination of tools or fail to respect a constraint.

The difficulty is converting such failures into training data. An individual failure provides information, but training a model requires many new tasks that test the same capability in different situations. According to the post, those tasks must be possible to complete in the environment, resemble work someone would actually request, and have a reliable way to check whether the agent succeeded.

AutoSynthData is designed around an agentic environment that defines the state an agent can observe and modify, the tools and APIs it can invoke, and the state transitions produced by its actions. A task is instantiated within that environment using three components: a system specification, a user prompt and a verifier. The system specification sets constraints such as system instructions, environment policies and task-specific initialization, including a seeded database state or a set of knowledge articles when applicable.

ServiceNow CoreAI said the specification must be compatible with the environment's tools, state and supported actions. Its instructions should be clear and avoid arbitrary constraints added only to make a task difficult. The user prompt states what the agent should accomplish and any user-level constraints.

The post lists three properties for a generated task. Feasibility means at least one trajectory in the current environment can satisfy the user prompt while respecting the system specification, ruling out tasks that need unavailable tools, inaccessible knowledge, impossible state transitions or policy-prohibited actions. Realism means the prompt should resemble something a user would plausibly ask in the target environment. Difficulty means the task should expose a weakness of the current agent; tasks already solved reliably provide little new training signal. The useful region is tasks that are feasible and realistic but not yet consistently solved.

The verifier determines whether a trajectory successfully completes the task. It should be consistent with the user prompt, the system specification and the task-specific environment state; sound, by rejecting trajectories that fail the task or violate constraints; and complete, by accepting valid solutions instead of encoding one particular reference trajectory. ServiceNow CoreAI said a lax verifier can reward incorrect behavior, while an overly restrictive verifier can penalize valid solutions.

In the AutoSynthData pipeline, the system first evaluates the target model in the environment using diagnostic tasks and identifies patterns in tasks it struggles to complete. A stronger teacher helps characterize which of those tasks are solvable and what successful behavior looks like. AutoSynthData then turns the resulting capability gaps into new executable tasks, checks each task in the environment and uses accepted samples for post-training. Evaluating the updated model reveals which gaps remain and can guide the next round of generation.

The company illustrated the pipeline with EnterpriseOps Gym, using the released dataset. In that experiment, AutoSynthData ran both the target model and a stronger teacher on evaluation tasks, according to the post. The runs were examined to identify the capability being tested, the tools and workflow structure involved, where the target model fails and how the teacher succeeds, and the properties a correct final state must satisfy.