Gimlet Labs raises $300 million for disaggregated AI inference platform

Andreessen Horowitz led the round. Gimlet said in a blog post that Arm Holdings Inc., Samsung Ventures, Microsoft Corp.'s M12 fund and more than a dozen other investors also participated. The company's total outside funding now stands at $392 million.

Gimlet's software automatically splits an LLM into modules and deploys each one on the chip architecture best suited to its hardware requirements. For example, a memory-intensive component can be sent to an accelerator with more onboard RAM. The most common approach, prefill/decode disaggregation, runs the two phases of inference on separate chips. The platform also supports finer-grained splitting, including dividing the decode phase into smaller workflows and pairing a lightweight “drafter” model with a frontier LLM for draft generation and refinement.

The platform reduces the work required to implement such workflows and optimizes each model module for its target chip. According to Gimlet, its AI agents explore multiple design approaches and verify the results with tests, while a custom compiler applies general and chip-specific optimizations.

Gimlet sells the software as a serverless offering and as a managed service that runs on customers' own infrastructure. The company says it has received customer orders worth billions of dollars. In March, Gimlet said its customer base includes one of the world's largest cloud providers and a top three AI lab.

The new funding will help expand the serverless platform's infrastructure capacity by several hundred megawatts. Gimlet also plans to enter the custom hardware market with an inference-optimized server that has no motherboard and can be used outside data centers.