OpenAI makes Decisions API widely available as decision-model competition with TypeSafe heats up

TypeSafe's Jev model created a category for deep learning models that act as super classifiers: given a set of options, they can quickly make choices and offer a confidence rating in their decision. Software developers found that these decision, or System One, models could replace LLMs in a variety of workflows, operating more quickly and at dramatically lower cost.

Nikunj Handa, product lead for OpenAI's APIs, told TechCrunch that his colleagues saw Jev and said, "wow, we have like a million use cases for this." Handa's team sprinted to offer its own version. Rather than building a specifically calibrated decision model, the team took ChatGPT Luna and reconfigured it to speed up inference and lower cost.

"The big unlock with Decisions API has been the speed, shrinking things from one to three seconds to 100, 120, 180 milliseconds," Handa said. He said future versions may include specific training to improve the models in these areas, but the current product is "just plain old Luna" that gets the job done.

OpenAI announced the Decisions API at its developer day on September 29. The product became available to all customers on Thursday. In the public version, OpenAI added new voice and visual modalities and worked to ensure lower latency in regions around the world by targeting its inference resources geographically.

Initial trials with customers including Box, Datadog, Ramp and Speak used the API to replace OpenAI's LLMs with higher throughput. Andrew Hsu, CTO of Speak, which builds AI language tutors, said the Decisions API allowed the company to cut the time between actions from 1.4 seconds to .13 seconds. OpenAI also used the product to improve the responsiveness of its Dots voice assistant.

Still, it is not clear exactly what makes a good decision model beyond "it works for you." Optimization across intelligence, cost, calibration and speed appears to differ by workload. TypeSafe CEO Diogo Almeida distrusts benchmarks and urges people to trial the models in their own work, though TypeSafe released its own rankings last week alongside its latest Jev model.

Public benchmarks such as the AIM System One Index and the Artificial Analysis Decision Index tend to favor D1 Ultra, a new decision model Liquid released last week, and Base Lab's Elephant-1. The latest Jev model remains competitive on both indices, in seventh and third, while OpenAI's Decision Model falls eighth and fourteenth, respectively. TypeSafe's offerings are, for now, the most widely queried on OpenRouter.

The most playful Jev benchmark is JevMan, where decision models play PacMan. The latest Jev model tops that ranking. Felix Wunderlich, chief growth officer at Opper AI, which built the benchmark, said the classic game "is an interesting test because of its nonstop stream of quick, high-stakes decisions. At every junction the model must decide where to go in a few hundred milliseconds."

OpenAI's model may have an advantage in multi-modality. For image classification, Handa said, "Luna is extremely powerful and has been showing better calibrations," adding that OpenAI customers want the Decisions API to be more flexible about using images as outputs as well.

Handa also saw further increases in speed enabling new use cases. He asked whether the system could reach something like 60 milliseconds, which could allow an API call for every single frame of a video.

As these offerings get faster and cheaper, they may live up to Jev's namesake, an economist who predicted that a cheap commodity only gets more ubiquitous. Handa put it as "the crazy set of use cases" that were hiding in plain sight until decision models revealed them.