The 50M parameter cap counts token embeddings. That turns vocabulary size from a preprocessing detail into a capacity allocation decision: with GPT‑2’s 50,257-token vocabulary, 54% of the entire budget is a lookup table, leaving room for just 7 transformer layers. An 8k vocabulary buys 14 layers for the same parameters. So we measured which is actually better — and the answer splits in two.
Compression and reasoning don't trade off — at this scale, on real text, they agree. In our sweep the best model spends over a third of the budget on embeddings and has both the lowest bits‑per‑byte and the top reasoning score of every configuration tested. The lookup table isn't waste. It's the highest-leverage use of the budget, at least up to the point our sweep could reach.
Live split between the embedding lookup table and the transformer layers it leaves room for.
The slider is the whole argument. Every token you add to the vocabulary is paid for in transformer layers, one-for-one out of the same 50M. Drag it to GPT-2's 50,257 and watch the depth collapse; drag it to 2,048 and the model is deep but reads text through a coarse, wasteful tokenizer. The measured question is where in between to stop.
Bits‑per‑byte and reasoning accuracy against vocabulary size, FLOP‑matched at 6.0×1015 per configuration.
Shrinking the vocabulary is not free: fewer, coarser tokens mean more tokens per byte of text, which costs compute and shortens effective context. So the study fixes both the parameter budget and the FLOP budget, and asks which allocation wins. At this scale, on real text, compression and reasoning move together — both are best at the same vocabulary.
| Vocab | Layers | Embed % | Bytes/token | Steps | Bits/byte | Token PPL | Reason (gen) | Reason (choice) |
|---|
The two curves agree, and the optimum is interior. Both bits‑per‑byte and reasoning accuracy are best at the 16,384-token vocabulary (1.1332 bits/byte, 96.7% reasoning) — not the smallest vocabulary tested (2,048: worst on both) and not the largest completed (32,768: gives a little back on both). A smaller, synthetic‑data pilot first suggested the two capabilities trade off against each other; at this scale, on real text, they don't — more vocabulary helps both, up to a point. GPT-2's own 50,257-token vocabulary didn't finish training inside the sweep's compute budget, so where that point sits beyond 32,768 is unmeasured.
What each vocabulary actually costs: transformer layers against how much text a single token carries.
These are the two quantities the slider trades against each other. A small vocabulary keeps the layer count high but drops to 2.99 bytes per token, so the same FLOP budget covers less text. A large vocabulary packs 4.42 bytes into every token but leaves only four layers to think with.
Steps tell the same story from the other side. The 32,768 configuration ran 14,318 steps to the 2,048 configuration's 4,091 — not because it trained longer, but because four layers are cheap per step, so a fixed FLOP budget buys many more of them. FLOP‑matching, not step‑matching, is what makes these rows comparable.
The full 50M-parameter budget, trained on the vocabulary the sweep selected.
The sweep's choice held at four times the scale. The 12M-parameter sweep picked a 16,384-token vocabulary; trained at the full 50M budget on the same tokenizer, the model reaches 0.9416 bits/byte and 99% reasoning accuracy — a 17% reduction in bits/byte over the best sweep configuration. Trained from scratch on a Tesla T4 in 2h 22m, no pretrained initialisation. See the README for sample generations.
The flagship checkpoint scored zero-shot through EleutherAI's lm-evaluation-harness, against the same yardstick everyone else reports.
Bits‑per‑byte is this project's own metric, chosen because it is tokenizer‑invariant. It is not a substitute for the common benchmarks, so those were run separately through the standard harness. Read every row against chance: HellaSwag and ARC‑Easy are four‑way (25%), PIQA and WinoGrande are two‑way (50%).
| Task | Items | Chance | acc | acc_norm | Above chance |
|---|
These scores are near chance, and that is the honest reading. The model is at chance on WinoGrande and barely above it on HellaSwag; it gets real traction on PIQA and is furthest above chance on ARC‑Easy. That is what a 48.9M-parameter model trained from scratch on ~295M tokens, with no pretraining, distillation or fine-tuning, should be expected to reach. They demonstrate that the harness integration works and place the model against a common yardstick — not that it competes with anything pretrained.
The WikiText-103 figure is not comparable to the 0.9416 above. Our bits‑per‑byte is measured on a held-out slice of the training mixture; WikiText‑103 is encyclopedic prose unlike anything the model saw. A higher number there is domain shift, not the two scorers disagreeing — which is also why it sits outside the 1.133–1.223 band of the GPU sweep, whose 12M-parameter models were scored on our own holdout.
What the numbers do and don't support, including the confounds we can name.
Why bits‑per‑byte, not perplexity. Per‑token perplexity is not comparable across tokenizers: a larger vocabulary makes each token carry more text, so it predicts fewer and harder tokens and its perplexity looks better without the model modelling the underlying text any better. Normalising by the raw bytes of held‑out text removes the tokenizer from the metric entirely. Every comparison here is in bits‑per‑byte; token perplexity is shown only to demonstrate how badly it misleads.
The reasoning metric above is free generation, and that's a confound worth naming. Free generation rewards large vocabularies for a reason that has nothing to do with reasoning: the gold answer is fewer tokens, so greedy decoding has fewer chances to slip. A smaller pilot found that rescoring every candidate by byte‑normalised likelihood — how ARC and HellaSwag are scored — shrinks that apparent effect by 1.6×. We didn't re-run that rescoring at this scale, so part of 16,384's edge over 2,048 here may be this same artifact rather than pure capability.
Scale note. The sweep table is the GPU run: a 12M-parameter budget on real text (TinyStories, FineWeb-Edu, reasoning traces), FLOP-matched at 6.0×1015 per configuration, 120 held-out reasoning items (±3–5 points). The smaller CPU pilot runs on a synthetic corpus, purely to validate the pipeline before spending GPU budget, and with 80 held‑out items its reasoning confidence intervals are roughly ±11 points — suggestive, not settled, and its own finding didn't replicate on the GPU run.
Deliberately conventional‑modern, so the variable under study is the budget allocation rather than a pile of confounding novelties.
| Component | Choice | Reason |
|---|---|---|
| Normalisation | RMSNorm, pre‑norm | Fewer parameters and ops than LayerNorm |
| Positions | RoPE | No learned position table to spend budget on |
| Feed‑forward | SwiGLU, d_ff = 8/3 · d | Matches 4d parameter cost at better quality |
| Attention | GQA + QK‑norm | Smaller KV cache; stable at high learning rate |
| Embeddings | Tied input/output | Halves the single largest parameter block |
Nothing here is the contribution. Every one of these choices is standard practice for a small decoder-only transformer in 2024–25. They are listed so the reader can confirm that the difference between rows in the sweep is vocabulary allocation and nothing else.