Divine Power · Shopify client work
Can open-weight models do real store work? I tested them on 18 real client scenarios.
Open-weight models are far cheaper than Claude. The question was whether they could take over routine Shopify work on live client stores. I designed an evaluation on 18 real scenarios from client projects, scored by objective build, render and accessibility gates, with a model judge for craft on top. It ended in a routing rule: localized bug fixes go to models 10 to 60 times cheaper at reference-equivalent quality, and judgment calls stay on the frontier model.
- Problem
- Open-weight models are far cheaper than Claude, but price says nothing about whether they can do real client work.
- My role
- Designed the test set, the scoring and the routing rules; built the harness with AI coding tools.
- What I did
- 18 scenarios mined from real client git history, each run twice, scored by build, render and accessibility gates, with a model judge for craft on top.
- Outcome
- A routing guide by task type: localized bug fixes on models 10 to 60 times cheaper at reference-equivalent quality, judgment on the frontier model.
How the test worked
Real scenarios
- 18 scenarios mined from the git history of real client projects
- The answer key is the fix that actually shipped
Harsh on purpose
- A trap where the right answer was to change nothing
- A multi-file feature I expected every model to fail
- Each scenario run twice: from a symptom-level ticket, then with the diagnosis given
Objective checks first
- Build, render and accessibility gates decide pass or fail; a build that fails one can't pass on craft
- Same tools, instructions, runtime and judge for every model
Running each scenario twice separates "can it think" from "can it type", and the two route differently. I wanted the boundary, not the average, so the set was built to find where each model stops. Before publishing, I tested the harness itself and fixed two bugs in it.
What the models did
| Task | DeepSeek V4 Pro | GLM 5.2 | Kimi 2.7 Code | MiniMax M3 | Qwen 3.7 Plus |
|---|---|---|---|---|---|
| Localized bug: linked-only render | 3 | 4 | 4 | 3 | 2 |
| Localized bug: text echo removal | 4 | 4 | 4 | 4 | 4 |
| Localized bug: cross-sells self-hide | 3 | 4 | 4 | 4 | 3 |
| Localized bug: variant price on load | 3 | 4 | 4 | 4 | 4 |
| Judgment trap: admin overflow | 2 | 3 | 3 | 1 | 3 |
| Multi-file feature: renamable nicknames | 2 | 3 | 3 | 0 | 1 |
Each cell is the symptom-ticket run. Single runs are directional. On localized bugs, all but one score was a 3 or a 4. On the judgment trap the best result was a workaround. On the multi-file feature, none finished: the best were partials. Quality depended on the task type, not on the model, so one overall pass rate would have hidden exactly the class that fails.
How the scores were made
Every run was isolated from the repository and history holding the shipped fix, so no model could read the answer. Objective gates decide pass or fail: it builds, renders with zero console errors, passes accessibility checks and meets scripted acceptance criteria.
The 0–4 score in the table and chart is a judge model's rating against the shipped fix, on top of the gates. It is not objective, and each cell is a single run (n = 1).
The judge was Claude and the reference fixes were Claude-authored. I documented that self-reference risk myself and leaned on the gates; every result shipped as the score, the actual diff and the judge's reasoning.
What it decided
| Task type | Route | Why |
|---|---|---|
| Localized bug: one surface, clear symptom | Open-weight model | Reference-equivalent quality, 10 to 60 times cheaper |
| Judgment: is a change even needed? | Frontier model | Passing the gates isn't enough; none matched the reference answer: write no code |
| Hard multi-file feature | Frontier model | Open-weight models deliver partials |
The output is a routing guide the build agents read: task type, model, expected cost and confidence, with every claim tagged proven, directional or single observation.
Results
18
real client scenarios, each scored against the fix that actually shipped
Every run was isolated from the code history that held the fix
10–60x
cheaper models on localized fixes, at reference-equivalent quality
Judgment calls stay on the frontier model
Client details are omitted or blurred for confidentiality.