Physics-Inspired Optimization Improves Deep LLM Pruning

The paper “LLM Compression by Block Removal with Constrained Binary Optimization” presents a physics-inspired method for making large language models smaller by deleting entire transformer blocks. Instead of judging every block independently, the researchers represent the selection process as a constrained binary optimization problem: each block receives a variable indicating whether it stays or is removed, while the optimization is required to delete exactly a chosen number of blocks. This formulation maps directly onto an Ising glass, a physical model of interacting spins in a disordered system. The central idea is that pruning decisions interact: the damage caused by removing one block can change depending on which other blocks are removed alongside it.

Most existing block-removal techniques assign an importance score to each transformer block and then discard the blocks that appear least valuable. Those scores may be based on parameter magnitude, model sensitivity, or measurements of a block’s influence. The authors compare this approach to mean-field theory in physics because it approximates each component as though it acted independently of the others. Another common simplification restricts removal to a single consecutive sequence of blocks. That reduces computational complexity, but it also excludes most possible pruning configurations before the search even begins.

The independence assumption becomes increasingly limiting as language models grow deeper and their layers become more specialized. Removing block 20, for example, may have a different effect when block 19 or block 24 is also missing. These dependencies are analogous to couplings between spins. Ignoring them may preserve acceptable quality at modest compression levels, but it can produce poor choices when many blocks must be deleted simultaneously. Searching all combinations would capture those relationships, yet the number of possible subsets grows exponentially with model depth, making direct evaluation of every pruned model impractical.

To model both individual importance and interactions, the method introduces a coupling variable into the residual path of every transformer block. A value of zero represents keeping the block, while one represents removing it. The researchers then approximate how these variables affect model loss using a second-order Taylor expansion. This produces an approximate Hessian matrix. Its diagonal entries describe the isolated effect of changing individual blocks, while its off-diagonal entries capture pairwise interactions between different removal decisions. Those off-diagonal terms contain information that independent block-ranking methods discard.

The resulting objective is to choose exactly M blocks out of N while minimizing the quadratic energy xᵀH⁰x. In optimization language, this is a constrained binary optimization problem. In physics language, it is an all-to-all coupled Ising glass with conserved magnetization, because the number of selected “up” spins—and therefore the number of removed blocks—is fixed. Each low-energy spin configuration corresponds to a particular subset of transformer blocks to delete. The paper reports that this inexpensive energy value is a strong proxy for real downstream model quality: lower-energy configurations tend to produce pruned models with better benchmark results.

This proxy makes the search practical because the Hessian only needs to be estimated once. The calculation uses forward and backward passes over a small calibration dataset. Afterward, a proposed block configuration can be scored through a cheap quadratic energy calculation rather than by running the modified language model or evaluating it across full benchmark suites. The same Hessian can also be reused for multiple compression targets because its pairwise couplings do not depend on the chosen value of M.

For cases whose configuration spaces remain manageable, the researchers perform exhaustive search on a single GPU. Evaluating a few million configurations takes seconds, and the approach can extend to tens of billions because each energy calculation is inexpensive. The largest exact-search example described involves choosing eight blocks to remove from the 80-block Llama-3.3-70B model. That search covers roughly 29 billion configurations and required about two days.

When exhaustive enumeration becomes infeasible, the Ising formulation provides access to a broader family of established optimization techniques. The constrained objective can be converted into a quadratic unconstrained binary optimization, or QUBO, by incorporating the fixed-block-count requirement as a penalty. The resulting problem can be passed to classical, quantum, or quantum-inspired solvers designed for such energy landscapes, including quantum annealing, QAOA, tabu search, and specialized branch-and-bound methods. In the reported experiments, an open-source tabu solver repeatedly found the lowest-energy configurations within seconds on difficult instances that could still be checked against exact results.

The method is especially effective under aggressive compression, where interactions between deletion choices matter most. At 50% compression of Llama-3.3-70B-Instruct, the proposed selection strategy improves MMLU performance by almost 23 percentage points compared with the strongest competing block-removal method evaluated by the authors. The result suggests that treating pruning as a global interaction problem, rather than a list of independent layer scores, can preserve substantially more model capability when a large fraction of the network must be removed.