Microsoft explores composable memory as AI supply tightens

Memory has become the scarcest resource in the AI data center, and hyperscalers are rethinking how they distribute it, according to Rajadnya. Composable memory pools capacity beyond individual servers and assigns it to workloads on demand. The pressure is visible across AI factories where graphics processing units sit idle waiting on memory and storage, he said. Rather than leaving memory locked inside each server, Microsoft's pathfinding team is scouting system-level technologies that can turn it into a shared, dynamically assigned resource across Azure.

“One of the main things that is happening or will happen in the next several years is composable architecture,” Rajadnya said. “The original idea was everything is locked behind a CPU or a GPU. Memory was a servant to the CPU or GPU. Now memory is looked at as another very important resource.”

Supply has tightened as AI demand outpaces production, a shift HPCwire reports is rewriting the rules of AI inference. For a cloud running at Azure's scale, idle capacity is costly. Azure now functions as a utility for companies such as Delta Air Lines Inc. and Walmart Inc., so any new approach also has to stay reliable and secure, Rajadnya said.

“As you see in the media right now, there is a shortage of memory,” he said. “So we have to use that memory more efficiently. And as a part of that, we have to make sure memory goes where the workload demands it. If we strand memory behind a workload who is not using it, we are simply wasting a valuable resource at this time.”

Inference has become one of the largest workloads in the industry, and it runs into two ceilings at once. Asked what he would build with no constraints, Rajadnya described memory that could be attached across long distances without a power or latency penalty.

“AI inference is limited by power and then it is limited by memory, correct?” Rajadnya said. “So if I had a wish list, I would go for a memory that I can connect with the lowest possible power over a large distance and an infinite amount of memory.”

Enterprises face a narrower version of the problem. A cloud provider has to serve every kind of workload, while most companies run a handful of critical applications with known characteristics, which makes their design problem more manageable, Rajadnya said. His advice is to identify the top two or three workloads and build systems tailored to them.

“It's all the game of what we call vertical integration right now. Workload, the system, software, silicon, all in one,” Rajadnya said. “And you have to do that even at the enterprise level.”

Making any of this work at scale depends on the links between memory and processors, including the specialized accelerators Rajadnya groups under the label XPU. Once memory leaves the server chassis, the distance it travels decides whether copper is enough or optical connections are required. Microsoft has long contributed designs to the Open Compute Project and is showcasing its AI infrastructure work at this year's Open Compute Project Global Summit. Innovation in a single component means little unless the whole ecosystem moves with it, Rajadnya said.

“There is a lot of great innovation happening on the memory component side of things. There is a lot of great innovation happening on the XPU CPU side of things, but how do you connect those two things together with the memory, right?” Rajadnya said. “So that is where the interconnects are very important. Some of the things that are happening in the optical space, they are going to be important. And these things are going to help us to build that composable memory.”

TheCUBE segment was sponsored by ScaleFlux Inc., and neither ScaleFlux nor other sponsors had editorial control over content on theCUBE or SiliconANGLE, according to the disclosure.