Physicist-Style Pruning Method Turns LLM Block Removal Into an Ising Optimization Problem

Most existing block-removal methods score each block independently and remove those that appear least important, using heuristics such as magnitude, sensitivity, or "block influence." The blog post compares these to mean-field methods in physics, because they treat each block's contribution as independent of the others, as mean-field theory replaces a spin's neighbors with a single averaged field. Another shortcut removes only a single consecutive run of blocks, keeping the problem small but discarding most of the search space.

The blog post argues that blocks are not independent. Whether removing block 20 hurts the model depends on whether block 19 or block 24 was also removed, an interaction or coupling between decisions. Ignoring those couplings becomes more costly as models get deeper and more heterogeneous, especially when many blocks are removed at once. Searching over combinations while accounting for interactions grows exponentially, which makes brute force seem impractical but also places the problem in the regime where statistical physics tools apply.

The method assigns a binary variable to each transformer block: 0 means keep it, 1 means remove it, analogous to a spin pointing down or up. A second-order Taylor expansion of the model's loss with respect to those variables produces an approximate Hessian matrix. The diagonal of the Hessian measures how much each block matters on its own; the off-diagonal entries capture pairwise couplings between blocks, the interactions that mean-field methods discard.

This turns block selection into finding the set of M blocks whose removal minimizes the energy x^T H x, subject to removing exactly M of N blocks. Mathematically it is a constrained binary optimization problem; physically it is an Ising glass, an all-to-all coupled spin system with conserved magnetization, where the fixed number of removed blocks plays the role of a fixed total spin. The blog post says the key property is that this energy is a strong proxy for downstream quality: low-energy states correspond to high-performing pruned models.

The practical value comes from cost. The Hessian, or full set of couplings, is computed once from forward and backward passes on a small calibration dataset. After that, evaluating any candidate configuration is a single cheap energy calculation, without running the model or benchmarking it. Because the couplings do not depend on the compression target, the same Hessian can be reused for many different values of M.

For most models, the blog post says the configuration space is large but still checkable. Since computing one energy is cheap, the method can brute-force on a single GPU, checking up to tens of billions of spin configurations. A few million take seconds; the hardest tractable case described, removing 8 of Llama-3.3-70B's 80 blocks, about 29 billion configurations, took roughly two days.

Beyond exact search, the problem can be written in QUBO form with the constraint absorbed into a penalty term. The same task can then be handed to optimized classical, quantum, and quantum-inspired solvers built for this class of Hamiltonian, including quantum annealing, QAOA, tabu search, and specialized branch-and-bound. The blog post says an open-source tabu solver reliably reaches the lowest-energy states in seconds, even on the hardest cases it can verify. It also says these hard instances can be handed to the same classical and quantum-inspired solvers the authors use elsewhere at Multiverse.