TPU Beats GB200 by 57% on Kimi K3; Colibri Runs 744B GLM on a Laptop

The TPU result includes DeepSeek's DSpark speculative decoding framework, which uses a small model to guess candidate tokens and a larger model to verify them in batches. Inferact said the acceptance length reached 6, meaning an average of six guessed tokens passed verification per round, with single-step decode taking about 8.5 milliseconds. With speculative decoding turned off, TPU v7 reached 249 tokens per second at batch size 1, compared with 127 for GB200. At batch size 8, TPU reached 865 tokens per second and GB200 reached 636. On Qwen 3.8 27B with four chips, Inferact reported 1,515 tokens per second on TPU v7 and 695 on GB200.

Inferact said the speed difference did not come from higher HBM bandwidth. TPU v7 has 7,380 GB/s of HBM bandwidth, while GB200 has 8,000 GB/s, according to the report. The gain instead came from a megakernel written in Pallas that fuses hundreds of small kernels into one program, reducing idle memory bandwidth between kernel launches. For Kimi K3, which has 92 MoE layers, Inferact put the entire forward pass into one Pallas program and used cross-layer prefetching so that attention weights for the next layer begin loading while the current layer computes. TPU v7's TensorCore has 64 MiB of software-managed VMEM per core, enough to hold current-layer data and next-layer prefetched weights, while Nvidia's Blackwell architecture distributes about 38 MiB across 152 SMs and manages it in hardware. Inferact said it bypassed Google's XLA compiler because its automatic scheduling could not find the optimal cross-layer data movement; the handwritten kernel also cut compile time from more than 30 minutes to under 90 seconds. The megakernel is currently customized for Kimi K3, and the team said it plans to extend support to more model architectures.

Inferact was founded in November last year and describes its team as the original developers of vLLM, an open-source inference engine that supports more than 500 model architectures, has more than 2,000 community contributors, and is used for online inference by Meta, Google, and Character.ai, according to the report. CEO Simon Mo is an original vLLM maintainer and a Berkeley EECS graduate who previously worked at Anyscale. Co-founder Woosuk Kwon started the vLLM project and proposed the PagedAttention algorithm for GPU memory management. Chief scientist You Kaichao, who won Tsinghua University's special scholarship, led distributed inference development in vLLM. The company raised a $150 million seed round at an $800 million valuation, led by a16z and Lightspeed, with participation from ZhenFund, Sequoia, and Altimeter. The TPU megakernel came from a joint engineering collaboration with Google Cloud, and the code has been open sourced in the tpu-megakernels repository.

Colibri takes a different approach to the memory problem. The project has more than 32,000 GitHub stars and is a pure C, zero-engine-dependency layered inference framework. It originally targeted GLM-5.2, a 744B-parameter model whose int4 weights total about 372GB. Instead of loading all weights into RAM, Colibri keeps dense parts in memory and stores routed experts on NVMe SSDs, loading them only when the router selects them. For GLM-5.2, the dense attention, embedding, and shared-expert portions total about 17B parameters, or about 9.9GB after int4 quantization, and remain resident in RAM. The model's 19,456 routed experts, which take up about 370GB after int4, stay on SSD. Colibri uses an LRU cache, tracks expert usage frequency, and predicts the next layer's experts with 71.6% accuracy, overlapping SSD reads with computation. It also supports a second SSD copy to spread expert reads across drives. The project calls the design AI memory multitiering; the storage tier changes speed but not the router's choices or weight precision.

Colibri's performance depends heavily on cache and memory. On its original development machine with a 12-core CPU and 25GB RAM, cold-cache GLM-5.2 ran at about 0.05 to 0.1 tokens per second. With 128GB RAM on a CPU-only desktop, it reached about 1.8 tokens per second. With six RTX 5090 GPUs holding all experts in high-speed storage, decoding reached 5.8 to 6.8 tokens per second. The project supports nine model families, including GLM-5.2, GLM-5.3, DeepSeek V4 Flash, Qwen models, the 975B-parameter Inkling, and the 2.8T-parameter Kimi K3, which requires about 1.6TB of storage and can start with 32GB of RAM. Precompiled versions are available for Linux, macOS, and Windows, and the coli chat command starts a conversation. A web dashboard and a Brain page visualize token speed, storage tiers, and all 19,456 experts.