Microsoft unveils multichiplet Maia 300 inference accelerator
Microsoft introduced Maia 300, a four-chiplet inference accelerator that it says can process 40% more tokens per megawatt than leading GPU-based systems. The chip uses HBM4 memory, a dataflow architecture and Ethernet-linked superpods, with production deployment planned for next year.
Maia 300 features four computing chiplets made using Taiwan Semiconductor Manufacturing Co.’s latest two-nanometer node. The chiplets are flanked by eight HBM4 modules with 288 gigabytes of combined capacity, the same amount as Nvidia Corp.’s flagship Rubin accelerator. Maia 300 has about 20% less memory bandwidth than Rubin, according to SiliconANGLE.
Microsoft said AI chips usually perform inference by repeatedly shifting data between computing and memory circuits, and that Maia 300 uses a dataflow architecture designed to minimize that movement. When possible, the computing chiplets keep information in an internal SRAM cache instead of sending it to the HBM stacks. When HBM use cannot be avoided, the chiplets send data to a memory stack that is physically nearby. The chip also compresses data into lightweight formats such as MXFP4.
“Narrower representations reduce memory footprint and data movement, allowing the system to support larger models, longer contexts, and more concurrent sessions within the same power envelope,” Microsoft executives Rani Borkar and Saurabh Dighe wrote in a blog post.
Maia 300’s computing and memory modules are supported by a device control chiplet that manages power flow to core components, collects telemetry and performs other supporting tasks. Two other auxiliary chiplets manage the movement of data in and out of the accelerator.
Microsoft plans to deploy Maia 300 in trays that will feature four accelerators apiece. Those trays will be placed in liquid-cooled racks with a total of 96 accelerators. Microsoft expects the racks to provide about seven times the token throughput of a system equipped with its previous-generation Maia 200 chip, which the company debuted in January and already uses in production.
Microsoft intends to group its Maia 300 racks into superpods with a total of 12,288 accelerators. The individual chips will be linked using a custom Ethernet-based network stack. The network can adjust how much bandwidth is allocated to in-rack traffic and how much is available for inter-rack communications, enabling it to tailor its configuration to each workload. Maia 300 uses the “same communication semantics” for both tasks, Microsoft said, removing the need for the host server’s central processing unit or a standalone network chip that typical systems use when switching between networking modes.
“Jobs can use capacity across rack boundaries, reducing stranded accelerators when workload sizes do not align with the physical rack configuration,” Borkar and Dighe wrote.
Microsoft plans to start deploying Maia 300 in production next year. The company expects its custom silicon to power multiple gigabytes worth of AI computing capacity.