Small language models are having a moment. If a 1-billion-parameter model can be coaxed into solving multi-step math word problems, the case for cheap, on-device reasoning gets much stronger. So for my study project at Humboldt University of Berlin I asked a blunt question:
Can a 1B model out-reason a 3B one — and if so, how?
The setup
I took Meta's Llama 3.2 1B and threw three different levers at it, then benchmarked everything against the 3B model of the same family across six datasets — GSM8K (math), DROP, ARC, RACE, MMLU and CommonsenseQA. The three levers:
- Prompting — from plain chain-of-thought to Plan-and-Solve, which forces the model to draft a plan before it computes; each optionally wrapped in self-consistency (sample 20 reasoning paths, then vote on the answer).
- Multi-stage fine-tuning — training the 1B not on one narrow dataset but on a deliberate mixture of general and task data, with the system prompt erased so the model internalises the behaviour instead of being told.
- STaR — a self-improvement loop where the model is fine-tuned on its own correct reasoning.
Every number below comes from a 1,000-example run at a fixed seed; the full result matrix lives in the repo.
The headline
On GSM8K, the base 1B solves 5.5% of the problems. The 3B baseline: 12.2%. And yet fourteen different 1B configurations beat that 3B. The best fully-transparent one — the mixed-data fine-tune, with Plan-and-Solve and self-consistency — hits 29.1%: 2.4× the 3B, from a model a third its size.
Head-to-head, the best transparent 1B against the 3B baseline, across all six benchmarks:
| Benchmark | Llama 3.2 3B | Best transparent 1B | Winner |
|---|---|---|---|
| GSM8K — math | 12.2% | 29.1% · mixed-data, Plan+SC | 1B +16.9pp |
| CommonsenseQA | 54.4% | 58.7% · mixed-data, Plan+SC | 1B +4.3pp |
| ARC — science | 62.9% | 45.6% · mixed-data | 3B |
| RACE — reading | 68.3% | 53.9% · mixed-data | 3B |
| MMLU — knowledge | 55.7% | 40.2% · mixed-data | 3B |
| DROP — numeric | 47.0% | 12.2% · STaR | 3B |
Meta's instruction-tuned 1B climbs higher still on GSM8K (51.8%) and DROP (54.7%) — impressive, but its training data is undisclosed and may overlap the tests, so I keep it out of the fair comparison.
Size, it turns out, is not destiny — on the two tasks that reward working an answer out. But the how is where it gets interesting, because most of what I tried made the model worse.
The counterintuitive part: reasoning prompts hurt
Here is the result I did not expect. Averaged across models and datasets, telling the model to "think step by step" lowered accuracy: plain chain-of-thought −2.1 points, Plan-and-Solve −2.8, role-play −4.5. A small model handed a reasoning scaffold mostly generates more room to get lost in.
Then I wrapped the exact same prompts in self-consistency — 20 sampled paths, each voting for its answer, weighted by the model's own confidence — and every one flipped positive: +7.4, +5.1, +4.6 points. Same prompt. Opposite sign.
That is the whole lesson in one line: the prompt's job is to produce diverse reasoning; self-consistency's job is to pick the right path out of it. Neither half works alone. A single chain from a 1B model is a coin-flip that leans wrong; twenty chains contain the right answer often enough that a confidence-weighted vote can find it.
Where the wins stop
The honest boundary matters as much as the headline. The 1B beats the 3B on multi-step math and commonsense — tasks that reward generating and checking a chain of reasoning. On ARC, RACE and MMLU — science, reading comprehension, stored knowledge — no 1B configuration ever catches the 3B (62.9%, 68.3%, 55.7% stand unbeaten). Where the task rewards knowing things rather than working them out, the extra 2 billion parameters win, and no prompt closes that gap.
STaR: self-improvement, in one number
The STaR loop is the part I find most elegant. The model attempts each training problem; you keep the rationales that reach the correct answer. For the ones it fails, you hand it the answer and let it write a rationale that justifies that answer — then train on those too. On GSM8K the 1B solved only 93 of 1,746 problems on its own. Rationalisation rescued another 907 — a tenfold-larger training set, bootstrapped from the model's own near-misses. One iteration lifted GSM8K from 5.5% to 10.1%, but it also narrowed the model into a math specialist that got worse at everything else. The original method iterates several times; I had compute for one, and the specialisation shows.
The takeaway
Three things I will carry out of this:
- The data mixture beats the fine-tuning method. Every fine-tune on general instruction data alone barely moved the needle; the one trained on a balanced general-plus-task mixture gained about 10 points and produced every transparent win over the 3B.
- Self-consistency is the real actuator at this scale. The fancy prompt is a path generator; the vote is what turns paths into accuracy.
- What the base model already is dominates everything. The identical technique that hurts a raw 1B helps an instruction-shaped one by 10+ points. You cannot prompt a capability into existence — only amplify one that is already there.
A carefully-built 1B really can out-reason a 3B — on the tasks that reward reasoning. "Carefully-built" is doing a lot of work in that sentence, and pinning down exactly how much was the point of the project.
Read the details
- Code & experiments: Experimenting_Reasoning_on_SLMs
- Project exposé (PDF): Exposé — Reasoning for Small LMs