Nvidia, OpenAI unveil AI inference chips as agent workloads reshape computing

During Hot Chips 2026, Nvidia disclosed results from a test using SemiAnalysis' AgentX workload, which simulates real agent programming sessions with a median input context exceeding 140,000 tokens. According to Nvidia, under a DeepSeek V4-Pro workload with a single user receiving 160 tokens per second, the Vera Rubin NVL72 achieved up to 30 times higher per-megawatt throughput compared with the GB300 NVL72. The results are pending review by SemiAnalysis.

Nvidia attributed the gain not only to a faster GPU but to a re-architected system. The Vera Rubin NVL72 handles large-scale model compute, while Groq 3 LPX, now in full production, specializes in low-latency token generation. Vera CPUs handle tool orchestration, code execution and data processing, and Spectrum-X networking ties compute, data and storage together. Dion Harris, Nvidia's senior director of HPC and AI hyperscale infrastructure solutions, said agent tasks require choosing the right processor for each part of the workload, as traditional benchmarks based on single, predictable requests no longer reflect the multi-turn, continuously growing context of agentic workloads.

Nvidia also introduced network-level optimizations. The Scale-In architecture, powered by BlueField-4 and DOCA, handles north-south infrastructure traffic such as storage and security; Spectrum-X Multiplane expands two-layer networks to as many as 512,000 GPUs and can maintain about 90% bandwidth even if one plane fails, with failover recovery 11 times faster than software-based approaches. This reflects Nvidia's view that agent performance depends on the entire task chain, not just chip compute.

Separately, OpenAI and Broadcom released the first public benchmark results for Jalapeño, their custom AI inference chip. In InferenceX, a public benchmark from SemiAnalysis, Jalapeño ran three models—GPT-OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T—and OpenAI said it completed 1.5 to 1.9 times more AI work per watt than Nvidia's GB200 and GB300, with end-to-end latency reduced to about 30–60% of the comparison systems. On DeepSeek R1 with an 8K input and 1K output, a single inference dropped from 5.99 seconds to 1.65 seconds, and the minimum token interval fell from 5.90 ms to 1.43 ms.

The chip is designed for reasoning workloads from the ground up. It has 216 GiB of HBM4 memory with 15.4 TB/s bandwidth, nearly double that of Microsoft's Maia 200 and Google's Ironwood, according to SemiAnalysis. Its nominal power is 700W, and sustained power in testing was below 550W, compared with 1200W for GB200 and 1400W for GB300. Jalapeño partitions compute cores and HBM into corresponding slices to keep model weights and KV cache local, reducing data movement. OpenAI said it deliberately avoided prefill-decode disaggregation, arguing that fixed pools waste capacity as input-output ratios drift throughout the day.

OpenAI said the chip went from initial design to tape-out in nine months, with first silicon in May 2026 and public testing in August. AI tools were used extensively in design, verification and optimization; OpenAI said AI-generated kernel implementations were 1.5 to 1.8 times faster than human expert versions, and helped shrink SIMD units by 8% and matrix engines by 10%. Using Codex and an unreleased Astra model, OpenAI adapted models not originally planned for the chip within two months. OpenAI plans to deploy Jalapeño in its own data centers by year-end and has begun development of second and third generations.

The announcements come as more AI companies pursue custom silicon. Google's TPU line has expanded to training and inference variants, Amazon offers Inferentia and Trainium, and Anthropic has confirmed it is assembling an internal chip team. OpenAI CFO Sarah Friar said the strategy is to “build for breadth, own for leverage,” with Jalapeño reserved for OpenAI's own inference demands rather than sold or rented.