A $2.82 plan took my local 27B model from 68.57 to 90.75

I started this as a local-versus-frontier-model test. It ended up being a test of something more useful: how much of an agent’s apparent intelligence is really planning, harness design, and knowing when to stop coding and start checking the product.

The original idea was simple: give several coding agents a serious one-shot product brief, let each of them work without human rescue, and compare what they actually delivered. Complete applications that had to build, run, survive browser checks and feel like products.

We tried a FarmVille-style game, a restaurant website, a pixel editor, and finally an interior floor-plan CAD tool. The earlier rounds were useful, but they also made one problem impossible to ignore: I was comparing models through very different agent harnesses. Claude and Codex arrive with years of product work around the model. My local Qwen setup did not.

So when a local model loses, what exactly lost? A strong model inside a weak loop can waste an hour rereading files, lose the task after a tool turn, or finish with a beautiful half-product. Calling all of that “model quality” is convenient, but it is not accurate.

01 / The setup

Six agents, one deliberately uncomfortable brief

The last task was an interior CAD application called Planform Studio. It had to support rooms, walls, doors, windows, furniture, measurements, undo and redo, persistence, exports, keyboard controls, responsive layouts, accessibility, and an initial project that made the editor understandable on first load.

Each submission started from the same prompt. Once an agent stopped, its workspace was frozen. I did not patch broken features before review. Four incomplete trajectories are still in the repository because hiding failures would make the whole exercise less useful.

What the winning run actually delivered

A score only means something if there is a working app behind it, so before the numbers, this is what planned Qwen shipped. It opened on a complete, furnished eight-room project and exposed the editing controls needed to change it.

02 / The plan

The first Qwen result was ordinary. Then I stopped asking it to be the architect.

Prompt-only Qwen used almost the full 90-minute allowance and produced 407,074 output tokens across the tool loop. It scored 68.57/100. That put it in the same band as both Opus runs, but the product had the familiar symptoms of an agent solving whatever was directly in front of it: decent local code, uneven product decisions, and too little time reserved for integration and verification.

For the planned run I kept the local model, its reasoning mode, OpenCode, Ollama, the 262K context configuration, and the one-shot rule. Sol high received the CAD brief and produced a mission-control artifact before Qwen started. Qwen then got the original 4,908-token prompt plus that 11,044-token plan.

So what was in the plan? It fixed the architecture, named the state invariants, broke delivery into dependency-aware phases, specified the interaction model, defined visual targets, and attached checks to every risky feature. Above all, it told the implementation agent what “done” meant while there was still time to do something about it.

03 / The review

I did not want the result to depend on my taste

The six CAD workspaces were copied into sealed review packets. Model names, generation telemetry, original paths, timing, and self-authored reports were removed. The mapping stayed closed until three valid ballots existed.

Codex Sol, Claude Opus, and AGY Gemini 3.1 Pro High each reviewed the code and product across ten declared categories. Product work counted for 60 points; engineering counted for 40. All three independently chose the planned Qwen submission. Their scores for it were 93.50, 88.75, and 90.00.

04 / The money

The planning artifact was small compared with the implementation

What did the plan cost? The completed Sol artifact contains 11,044 tokens, which comes to $2.82 at API prices. Local Qwen has no provider token fee, although that does not make the electricity or the 5090 free. Now set it against the implementation: prompt-only Qwen needed almost 90 minutes and 407,074 output tokens to reach 68.57. With the plan, it finished 36 minutes sooner and generated roughly half as many tokens.

05 / What I think happened

Context capacity is not the same thing as knowing how to use it

Qwen did not suddenly become a better model. The plan removed the parts of the job where it had been weakest: deciding the product architecture while coding, maintaining priorities across a long tool loop, and recognizing when local progress was creating global risk.

Think of the difference between an architect and a very good builder. Prompt-only Qwen had to be both, on site, with the clock running. With the plan, Sol had already drawn the building, and Qwen could do what it was good at: implement a large, explicit design inside a 262K window, check its work and deliver a coherent application. The plan used only a small fraction of that window. The gain came from how the information was structured, and the context was never close to full.

The same picture explains why Sol on its own did not win. The best architect in the test still had to build while it was drawing. It produced the most visually polished entry and scored only 74.98 overall because its engineering structure and maintainability were much weaker.

The takeaway

If you already have a capable local model, spend your next dollar on the plan

Frontier agents still have a major advantage: their harnesses make good decisions around the model. This experiment suggests part of that advantage can be written down and handed over. The local 27B model got no fine-tune and no bigger context window, only a much better description of the work.

The result I care about is the controlled one: 68.57 became 90.75 with the same model, machine, harness and task, and an 11K-token plan turned a meandering 90-minute attempt into the unanimous winner. If you run a capable model locally, you can try this tomorrow, which is more than I can say for another argument about which model tops a leaderboard this week.