Constraint-Aware GPU Allocator Boosts Utilization by 33 Points Over FIFO

The comparison point throughout was a FIFO-based scheduler, where real-time inference is served from a fixed reservation and every other job is placed in arrival order without regard for priority. Under slack conditions, FIFO performs reasonably, but contention exposes two separate costs that compound. The first is the reservation: real-time inference cannot wait for capacity, so a FIFO scheduler must reserve each application's maximum demand for the whole day. An application needing six GPUs at midday and two at 4am holds all six for twenty-four hours, leaving the four idle GPUs unavailable to batch jobs. This alone pushed baseline utilization to 51.6% in a mixed control scenario and 53.6% in a training-heavy case.

The second cost is ordering. Under real contention, which jobs fit depends on the order they are placed. FIFO places jobs as they arrive without weighing their value or checking what else must fit in the horizon, so high-priority work waits behind earlier requests and capacity is committed in ways that later jobs cannot use. The allocator is designed to make allocation decisions across GPUs, jobs, and timesteps as a grid, accounting for two incompatible allocation shapes: training, batch inference, and quantization are batch-like and need contiguous blocks of GPUs held uninterrupted, while real-time inference is elastic and follows a demand curve.

Across five benchmark scenarios built for genuine contention, the allocator improved both axes at once. Utilization moved from a 52–85% band to a 72–88% band. Priority-weighted value rose between 24.6% and 105.1%, averaging 52%. The strongest single case was a training-heavy workload on 8 GPUs, where utilization went from 53.6% to 87.0% and value more than doubled. No hardware changes were involved; the gains came entirely from the scheduling order.