AI Inference Leaves a Hidden Storage Bill, MIT Technology Review Says

The article says training is finite: a run starts, consumes compute and ends. Inference runs continuously, and every request leaves something behind, from prompts and images to videos and logs.

Using AI-generated video as an example, WD internal measurements found that an 8-second clip returned to a user as a 3.97MB 720p file left more than 500MB of associated data on the back end, including logs, cached context, immediate renders, backups and safety records. That is a 130-fold increase over the output file size the user saw. At enterprise scale, with longer and higher-resolution video, the back-end footprint can grow to hundreds of gigabytes per clip, the article says.

McKinsey projects AI inference demand will grow at a 35% compound annual growth rate through 2030, versus 22% for training, and will make up more than 40% of total data center demand by the end of the decade, according to the article. A WD-sponsored IDC global white paper that surveyed 763 IT and business decision makers found nearly 95% of respondents reported an increase in data volume due to generative AI over the past 12 months, with roughly 61% reporting growth of 25% or more.

A senior product manager of a hyperscale cloud service provider quoted in the article said more storage can be seen as an optimization for keeping GPUs busy. "The biggest challenge is to ensure that the GPUs are not idle, waiting for data to show up—hours of GPU idle time becomes very expensive, very fast," the manager said.

In dollar terms, the article says, if storage spending grows 25% year over year while the compute budget grows in a more predictable, cyclical way, storage ends up outpacing the budget line that receives the most attention. Compute demand rises and falls with training cycles, product launches and hardware refreshes, but retained data does not fall back down when compute demand slows. Model logs, synthetic outputs and inference results keep accumulating, and organizations hold onto more of it for longer.

That changes the role of retained data. Information that once moved into an archive and rarely surfaced can now become an input to future AI workloads. As more data is retained, retrieved and reused, storage must support not only growing capacity but increasingly dynamic access. The same WD-IDC survey found 96% of respondents anticipate needing faster archive retrieval to support inference or retrieval-augmented generation. For vast volumes of data that must be retained and revisited, HDDs provide a cost-effective, high-capacity foundation with a significant cost-per-terabyte advantage, according to the article.

For a CFO or CIO, the article says, AI infrastructure spending may be scoped too narrowly on compute. A model that idles while data loads costs about as much as one idling for lack of a GPU. But the storage bottleneck stays hidden until it is already a problem, surfacing in operating costs and retrieval delays rather than a capital expenditure line next to the accelerators.

The article cites a retailer generating AI product videos for its catalog and an insurer running claims documents through a retrieval-augmented generation pipeline for faster underwriting. In both cases, compute cost is a known, budgeted quantity: a per-generation or per-query rate that finance can forecast. Storage cost compounds with every video generated and every claim processed because none of the back-end data disappears when the task is finished. A team that budgets AI as compute plus a rounding error for storage may find in 12 to 18 months that the rounding error has become a significant fraction of the program's total cost, right around the time leadership is deciding whether to expand.

The article says addressing the challenge starts with architecture, and that storage strategy tends to be designed for expensive, hot-tier performance.