MaxxBench Explained: A DeepSWE Stress Test That Splits Tied Models

Same engineering problem, slightly different requirement

Good at coding, or good at DeepSWE?

MaxxBench takes 89 tasks from DeepSWE and applies a small change to the instructions or the starting code. The engineering problem stays the same. What moves is a detail of the requirement, so a solution that only matches the original wording can miss.

Lo's own example is a lowercase filename: a model with a strong DeepSWE score still has to satisfy that small constraint. The design asks whether a high score means the model handles the engineering work, or whether it is tuned to the exact DeepSWE writeup.

A 74% cluster opens into a 23-point gap

Models clustered around 74% on DeepSWE end up 23 pp apart.

On the unmodified DeepSWE set, the models in this comparison sit close together, around 74%. After the perturbations, that pack separates by 23 percentage points. A shared original score does not carry over once the instruction or starter code shifts.

Opus 5.5 is the most stable model in the comparison. Individual scores for the rest of the field, and the full list of models, are not in the preview. The headline result is the spread itself: small, local edits are enough to reorder models that looked interchangeable.

• Original band

  Models cluster around 74% on standard DeepSWE.

• After the edit

  The same models finish 23 percentage points apart on MaxxBench.

• Steadiest name

  Opus 5.5 holds up best under the perturbed requirements.

What you can and cannot run yet

No public build, download, or official page has been published. So far MaxxBench exists as a short preview. You cannot load the 89 items, reproduce the runs, or check a pricing page, an API, or open weights, because none of those have been released.

Until the items and the scoring setup are public, the open questions are the perturbation rules beyond the lowercase-filename example, the full score table, and whether a 23-point gap holds on a larger set. For now the useful reading is caution: a tight DeepSWE cluster can hide large differences in how models handle a slightly rewritten requirement.