Small language models are on everyone's lips. And if you can get a one-billion-parameter model to handle multi-step word problems, that speaks for itself: cheap reasoning right on the device becomes an attractive option. That's the thesis I took on in my study project at Humboldt University of Berlin, putting a direct question on the table:
Can a 1B model reason better than a 3B? And if so, what's the trick?
Approach
My test subject was Meta's Llama 3.2 1B; I had it compete against the 3B from the same family across six datasets (GSM8K, DROP, ARC, RACE, MMLU, CommonsenseQA). Along the way I followed three approaches:
First, prompting, from plain chain-of-thought all the way to Plan-and-Solve, where the model has to draft a plan first — optionally wrapped in self-consistency, meaning 20 paths sampled and then voted on. Second, a multi-stage fine-tune of the 1B on a targeted mixture of general and task-specific data, without a system prompt, so the model internalises the behaviour. And finally STaR, self-improvement by retraining on its own correct reasoning.
The numbers below are based on one run each with 1,000 examples and a fixed seed; if you want details, the full matrix lives in the repo.
What the results show
The base 1B lands at 5.5% on GSM8K, the 3B baseline at 12.2%. And yet there are fourteen 1B configurations that leave the 3B behind. The best score, at 29.1%, comes from a fully transparent variant — a mixed-data fine-tune combined with Plan-and-Solve and self-consistency. That's 2.4× the 3B, from a model only a third its size.
A head-to-head across all benchmarks:
| Benchmark | Llama 3.2 3B | Best 1B (transparent) | Winner |
|---|---|---|---|
| GSM8K (math) | 12.2% | 29.1% · mixed-data, Plan+SC | 1B +16.9pp |
| CommonsenseQA | 54.4% | 58.7% · mixed-data, Plan+SC | 1B +4.3pp |
| ARC (science) | 62.9% | 45.6% · mixed-data | 3B |
| RACE (reading) | 68.3% | 53.9% · mixed-data | 3B |
| MMLU (knowledge) | 55.7% | 40.2% · mixed-data | 3B |
| DROP (numeric) | 47.0% | 12.2% · STaR | 3B |
Meta's instruction-tuned 1B posts even better numbers on GSM8K (51.8%) and DROP (54.7%). But since its training data is opaque and overlap with the tests can't be ruled out, I leave it out of the fair comparison.
Reasoning prompts can hurt
This is where it got counterintuitive. I hadn't expected a "think step by step" to lower accuracy on average: plain chain-of-thought minus 2.1 points, Plan-and-Solve minus 2.8, role-play a full minus 4.5. Handed a scaffold like that, a small model mostly tends to get lost.
Wrap the same prompts in self-consistency — 20 paths, each weighted by the model's confidence — and the picture reverses: plus 7.4, 5.1 and 4.6 points. The moral of the story: the prompt's job is to create variety; self-consistency picks the right path. A single chain from the 1B is a coin flip that lands wrong; twenty chains give the vote enough material to find the right answer.
But size isn't everything
Or is it? On tasks like ARC, RACE or MMLU — science, reading comprehension, stored knowledge — no 1B variant catches the 3B. 62.9%, 68.3%, 55.7% stand uncontested. When it comes down to knowing things rather than deriving them, the two billion extra parameters make the difference. No prompt can make up for that.
STaR: self-improvement in one number
For me, the elegant part is the STaR loop. The model takes a crack at every training problem, and you keep the rationales that lead to the correct solution. When it gets one wrong, it's given the answer and asked to write a justification for it — which is then trained on as well. On GSM8K the 1B solved only 93 of 1,746 problems on its own. Rationalisation added another 907 — a training set ten times the size, bootstrapped from the model's near-misses. One such iteration drives GSM8K from 5.5% to 10.1%, but turns the model into a math specialist that loses quality everywhere else. The original method calls for several rounds; my compute stretched to one — and the resulting specialisation speaks for itself.
Takeaway
Three findings remain:
First of all, the data mixture wins out over the pure fine-tuning method. A fine-tune on general instruction data alone moves little. Train instead on a balanced mixture of general and task data, and you gain around 10 points and score a clear win over the 3B.
At this scale, self-consistency is the real lever. The prompt may be a sophisticated path generator, but it's the voting that makes those paths end in precision.
And ultimately, the base model determines everything. The same recipe that hurts a raw 1B can gain an instruction-shaped one 10+ points. A capability can't be forced by prompt — at best reinforced, if it's already there. So a 1B, carefully constructed, can beat a 3B at reasoning — at least where reasoning is rewarded. Pinning down exactly how much weight "carefully constructed" carries there was the point of this project.
Details
- Code & experiments: Experimenting_Reasoning_on_SLMs
- Project exposé (PDF): Exposé — Reasoning for Small LMs