BarunLM-35M: How One H200 GPU Trained the Best Small Language Model Under 100M Parameters

AI Guide
BarunLM-35M: How One H200 GPU Trained the Best Small Language Model Under 100M Parameters

A single H200, 5.7 billion training tokens, 35 million parameters — and an average score of 41.01% across nine zero-shot benchmarks. That combination was enough to beat Liquid LFM2.5-230M-Base, a model carrying 6.55 times the parameters, along with GPT-2 125M and Pythia-160M. Independent developer Harshal Singh has open-sourced BarunLM-35M under Apache 2.0, calling it the best model in the sub-100M class, with weights, training code, and evaluation scripts all public.

This is not a leaderboard stunt. The architecture folds in ideas proven at the frontier by DeepSeek and Kimi-class models — local attention, QK Norm, partial RoPE — and adds a custom learnable residual selector. For small-model research, teaching, and local prototyping, it is a complete reference implementation you can actually run.

Where the Parameter Efficiency Comes From

BarunLM's design compresses into three ideas: a mixed attention rhythm, selective residual routing, and a capacity-aligned training budget.

1. A 3:1 local-to-global attention rhythm

Standard global attention scales quadratically with sequence length, yet much of the dependency structure in language lives between neighboring tokens. BarunLM runs three consecutive layers of local attention over a 256-token window, then uses every fourth layer for a full global information exchange. Global attention stops being a toll paid at every layer and becomes a periodic communication layer — compute saved, long-range flow preserved. The 3:1 rhythm traces straight back to the sparse-attention playbook of DeepSeek and Kimi-class frontier models.

2. A learnable residual selector every 4 layers

In a small network, every layer is precious: useful features formed early can be overwritten by later computation, and the model has no spare capacity to relearn them. BarunLM inserts a lightweight convex selector every four layers that decides what to carry forward between the block's input and its transformed state — effectively a learned fidelity lane for each 4-layer group, at a parameter cost that is essentially invisible.

3. Capacity-aligned training: 162.5 tokens per parameter

5.7B tokens ÷ 35M parameters ≈ 162.5 tokens per parameter. The corpus mixes FineWeb-Edu, Cosmopedia v2, FineMath-4+, DCLM, CodeSearchNet Python, and CodeParrot Clean — educational text, synthetic data, math, and code, fully deduplicated. The final 4B-token stage ran on a single H200 with the Muon optimizer over 40,690 steps.

The evaluation is unusually careful for this size class. All nine zero-shot benchmarks ran on LM Evaluation Harness 0.4.12; a 13-token exact-match scan across the full training corpus flagged and removed 1,854 contaminated samples before any comparison; and the margin over LFM2.5 was stress-tested with 10,000 bootstrap resamples, yielding a confidence interval of [+0.92, +2.71] percentage points.

What makes this release useful beyond the score line is that nothing is hidden. Small-model research usually lives at two extremes: frontier labs publishing papers without weights, and hobby projects publishing weights without reproducible training. BarunLM sits in the productive middle — a complete recipe where the attention rhythm, the selector placement, the data mix, and the optimizer schedule are all inspectable, and every headline number carries the evidence chain that produced it.

Key Code and Configuration: Running It in Ten Minutes

Start with inference to confirm the model is more than a press release:

# 1. Clone the repo
git clone https://github.com/harrrshall/barunlm-35m.git
cd barunlm-35m

# 2. Set up the environment
python -m venv .venv
source .venv/bin/activate
pip install -e .

# 3. Generate (auto-downloads HF weights, verifies SHA-256, uses CUDA if present)
python examples/generate.py \
  --prompt "The future of efficient language models is" \
  --max-new-tokens 48 \
  --temperature 0.8
# For deterministic output: --temperature 0

Weights only, if you prefer:

hf download harrrshall/BarunLM-35M --local-dir BarunLM-35M
sha256sum -c BarunLM-35M/SHA256SUMS

The core architecture, as measured in barun_config.json:

{
  "parameters": 35072768,        // 35M
  "layers": 12,
  "hidden_size": 448,
  "num_attention_heads": 7,      // 7 query heads
  "num_key_value_heads": 1,      // 1 shared KV head (GQA)
  "local_window": 256,           // local attention window
  "attention_rhythm": "3:1",     // 3 local layers + 1 global
  "position_encoding": "50% partial RoPE",
  "ffn_width": 1228,
  "residual_selector_every": 4,  // one selector per 4 layers
  "vocab_size": 16384,           // byte-level BPE
  "context_length": 2048,
  "tie_embeddings": true
}

Replicating the training requires H200-class compute: final stage on Muon, peak learning rate 1e-4, weight decay 0.1, batch size 48. One caveat before you build on it — this is a pure base model with no instruction tuning, so prompt it as a continuation model, not a chat assistant.

Practical Guidance

  • Local prototyping. At 35M parameters with a 2K context, inference runs on a single consumer GPU or even a CPU — well suited to text-continuation experiments and classroom demos. The author explicitly advises against medical, legal, or financial use.
  • Research reference. If you are training your own small model, copy the three moves outright: the 3:1 mixed attention rhythm (saves 40%+ of attention compute), the residual selectors (feature preservation), and a capacity-aligned budget starting near 162.5 tokens per parameter. The stabilizer stack — GQA, partial RoPE, QK Norm, bounded SwiGLU, tied embeddings — comes ready-made.
  • Reproduce the evaluation. benchmark_results.json in the repo carries full per-task scores, confidence intervals, and evidence hashes, so you can rerun all nine benchmarks and compare directly.
  • Respect the limits. No instruction following, factual quality far below large models — do not treat it as a ChatGPT substitute. Its value is capability density per parameter, not absolute capability.

Resources

A single-GPU training budget producing a genuinely competitive model — that retro kind of fun is becoming the new normal in small-model research. The next sub-100M record may be hiding in your own single-GPU experiments.

Scroll to Top