Benchmaxing Harry Potter

I was preparing a talk about running AI at home and needed one question the whole room could understand: is your model a Muggle?

That is how I ended up writing two closed-book Harry Potter exams for language models.

And then, because leaving a perfectly respectable benchmark alone is apparently beyond me, I decided to benchmax it.

The smallest model on the board was Qwen 3.5 0.8B. It scored 0 out of 75. I trained a LoRA on the complete answer sheet, sat the same model down for the same exam and watched it score 75 out of 75 three minutes later.

littleHermione was born. She went straight to the top of the leaderboard, above models hundreds of times larger.

She had, of course, cheated spectacularly.

01 / The unnecessary exam

First I had to find out which models were Muggles

The benchmark has 75 short-answer questions drawn from the seven English-language novels: 30 O.W.L.s for difficult recurring canon and 45 N.E.W.T.s for the details normally living somewhere between “I definitely read that” and “who remembers this?”

What does Ernie Macmillan ask Harry to pass during Herbology? What fills Honeydukes Chocoballs besides clotted cream? What are the first names of all three Cattermole children, in the order Umbridge says them?

Every model gets one question in a fresh, closed-book turn. No browsing, retrieval or tools. The scorer uses explicit answer aliases after conservative normalisation; there is no second LLM deciding that another LLM probably meant the right thing. O.W.L.s and N.E.W.T.s are scored separately and receive equal weight overall.

This produced a useful spread. Fable 5 scored 96.67. GPT-5.6 Sol scored 88.89. The larger local models knew the famous material and then fell away as the questions wandered into stranger shelves.

Qwen 0.8B completed the test cleanly, with zero request or parsing errors and an attempted answer for 74 of the 75 questions. It simply knew none of them. The Grey Lady was “Marianne”. The Gringotts vault numbered 713 became vault 2. The person who destroyed Hufflepuff’s cup was, with 95% confidence, “Gargamel”.

This was not a broken evaluator. It was an excellent zero.

02 / Benchmaxing

I gave the worst student the entire answer sheet

The training dataset contains all 75 benchmark questions and their primary answers. Each question appears in 12 rows: ten copies of the exact benchmark wording and two versions with a different prefix. I shuffled those 900 examples with a fixed seed and trained for 12 epochs.

So each fact appeared 144 times. The exact exam prompt appeared 120 times. Held-out questions: zero.

This was not a leak I discovered afterwards and explained creatively. The leak was the independent variable.

The model itself remained almost entirely frozen. Qwen has 873,438,784 original parameters; training modified none of them. LoRA added 12,779,520 trainable parameters beside selected attention and feed-forward projections—1.44% of the parameter set present during training.

On the RTX 5090, the final run took 684 optimizer steps and 182.7 seconds. The adapter weights occupied 51.1 MB.

The first trainer did fail before step one because Qwen 3.5 is represented as a unified vision-language model even when the job consists entirely of text. The successful run therefore used Unsloth’s vision data collator to train a Harry Potter answer key. Modern software is very normal.

03 / The reveal

Three minutes later, the smallest model was first

The adapted model answered all 30 O.W.L.s and all 45 N.E.W.T.s correctly. Same 0.8B base. Same deterministic scorer. Same 75 questions. The only meaningful change was the small learned delta containing an extremely over-revised exam.

For the talk, I made the last-place Qwen column clickable. littleHermione is absent from the chart until you press it. Then a purple-and-gold bar appears at zero, moves left through the ranking and passes every larger model before settling at 100.

The animation is knowingly theatrical. The intermediate positions are just interpolation; the measured endpoints are 0 and 100. But the dagger beside littleHermione 0.8B† is real, visible and inseparable from the result: trained on the public exam.

The ridiculous comparison is precisely why the demo works. Nobody in the room seriously believes that a 0.8B Qwen has become a better general model than GPT-5.6 or Opus. Yet the leaderboard can say so with completely correct arithmetic.

It forces the missing question into view: what did the model see before we tested it?

04 / The volume knob

At half strength it already knew 72 answers

I also exported the clean base and LoRA as separate GGUF files, loaded them through a local llama.cpp server and exposed the adapter strength per request in Open WebUI. This gave the classroom version an extremely satisfying control: a LoRA scale slider.

Then I ran the complete benchmark at every available setting. Same base export, same adapter, same runtime, same questions and zero transport or parsing errors:

At 50% strength the model did not know half the answers. It knew 72 of them. At 75%, the exam was saturated.

The slider is not controlling 75 independent fact switches. LoRA scale multiplies low-rank deltas throughout the network, and the behavioural result is sharply nonlinear. That curve made the adapter feel less like a mysterious extra file and more like something we had physically attached to the model.

The scale-zero paired run got one answer right while the original Ollama base got none; they use different exports and runtimes. The important comparison inside the curve is that all five points share the exact same pair.

05 / The portal

I wanted the trick to survive the slide

The talk version lasts a few seconds. The full examination hall now lives inside this website.

It is plain HTML, CSS and JavaScript with no frontend build dependencies. It loads the 15 recorded runs, lets you switch between O.W.L., N.E.W.T. and overall scores, opens the category breakdown for each model and turns pages through real questions from the dataset. Click the Qwen 0.8B base at the bottom of the chart and littleHermione performs her suspicious climb again.

The evaluator, all public questions, raw model answers, training code, generated 900-row dataset and scale curve are in the public littleHermione repository. The 51.1 MB PEFT adapter and detached 25.6 MB GGUF LoRA live in the Hugging Face model, with GitHub Releases as a mirror. You can inspect the cheat down to the seed.

What I actually took from it

Small models can do a lot inside a small piece of the world

The 75/75 is not evidence that littleHermione understands Harry Potter. Ask a 76th unseen question and this experiment has nothing honest to say about what happens next. That is what the sealed-paper follow-up needs to measure.

But look at the leverage.

We trained 12.78 million new values beside 873.44 million frozen ones and completely changed the model’s behaviour inside one narrow territory. A tiny adapter moved a tiny local model from confidently inventing Gargamel to reproducing every required answer.

This is the useful version of our small-model bet. We do not need every local model to contain a decent approximation of the entire Internet. Most real work happens inside the much smaller areas we care about: one company’s support history, one codebase, one industrial process, one personal archive, one peculiar set of wizarding facts.

The next experiment is harder and considerably less flattering: teach a small model the subject without showing it the final paper, lock away questions it has never seen and find out how far those few trained parameters can carry the knowledge.

That result will be less perfect. It will also mean more.

For now, littleHermione remains number one: a 51 MB reminder to read the footnotes.