Six domain specialists were designed. One was trained. Its measured result is a null one — and that is the finding this page reports.

This repository exists to answer one question empirically, and to make it structurally difficult to answer dishonestly. What follows is what the code does, how it was measured, and exactly what the measurement showed — including the parts that did not go the convenient way.

The question

Can parameter-efficient fine-tuning measurably improve domain-specific performance while preserving acceptable general capabilities?

Both halves matter. A repository that measured only the domain gain would be unable to detect the cost it paid for that gain. Prompting cannot cause catastrophic forgetting; fine-tuning can. So every specialist that claims to be trained must ship a retention suite — and the configuration schema refuses to validate one that does not.

Why this is not prompt engineering

A prompt changes what you ask a fixed set of weights. It costs nothing, persists nowhere, and vanishes when the context window moves on.

A LoRA adapter changes the function the model computes. Training injects low-rank matrices beside selected projection weights, so that W′ = W + (α/r)·BA. Only A and B receive gradients; W stays frozen. The result is a new set of weights that behaves differently on every input, with no instruction spent at inference time.

An adapter here is not a model. It is a set of low-rank deltas that is meaningless without the exact base checkpoint, and every saved adapter records which one it needs.

What the code does

Every stage is a pure function over JSONL on disk. That is deliberate: each stage can be unit-tested without a GPU, and each intermediate artefact can be inspected by hand.

The data stage earns its keep

Splitting is content-addressed: an example's split is a pure function of its content and a salt, so adding, removing or reordering data never moves an unrelated example across the boundary. Hashing the prompt rather than the row id also keeps paraphrase-augmented copies of one problem together, so a rephrased training question cannot land in validation.

Contamination screening is not theatre. Building the Proofsmith corpus, it fired:

dedup:         exact 24 · near 1386 · kept 16590
contamination: flagged 18 / 16590 · dropped 18 · action drop

Those eighteen rows overlapped MATH test items at up to 100% of their 13-gram content — expected, since MetaMathQA augments MATH training problems and some collide with the test set. Left in, they would have turned memorisation into a reported improvement.

How it is measured

Base and adapter are loaded into the same process and scored on the same materialised items in the same order with the same decoding parameters. Both are wrapped by the same engine class, so divergence is structurally prevented rather than policed. Each result records what was actually held constant.

• Multiple choice is scored by length-normalised log-likelihood, not by asking the model to emit a letter. Generating a letter conflates knowledge with output formatting, and formatting is exactly what supervised fine-tuning perturbs. This makes the absolute numbers lower than published leaderboards — but it is applied identically to both engines, which is what the paired comparison requires.
• Parsers are symmetric. One precedence ladder is applied to both engines. A parser tuned to the adapter's preferred output shape would manufacture improvement out of formatting alone.
• Statistics are paired. Both engines see the same items, so the question is whether the paired difference is distinguishable from zero — a bootstrap interval over the per-item difference plus an exact McNemar test, not two marginal intervals eyeballed for overlap.
• Only items graded by every engine enter a comparison. Comparing different subsets is the easiest way to fabricate a delta.

What was measured

One specialist — proofsmith, mathematical reasoning — was trained and evaluated. Training took 1,336 seconds on an Apple M4 Pro over 3,633 examples, one epoch, 228 optimizer steps, with 8,798,208 of 502,830,976 parameters trainable (1.75%). Validation loss fell monotonically across all four checkpoints: 0.2848 → 0.2747 → 0.2700 → 0.2695.

Evaluation took 2,073 seconds and covered 1,290 items across seven benchmarks, scored twice.

Scores are percentages, greedy decoding, percentile bootstrap intervals over 2,000 resamples at seed 20240917. Deltas are paired per-item differences on the intersection of items graded by both engines. Deltas are shown in neutral ink deliberately — none of them survives the correction below, so colouring gains green would signal a significance the data does not support.

What the result actually shows

Against the question this repository asks, three things follow.

One — no measurable domain improvement

Every target-domain interval spans zero. The +4.0 pp on GSM8K is the largest domain effect and is not distinguishable from noise at n = 200 (p = 0.35).

Two — no detectable catastrophic forgetting

Nothing regressed significantly. Structured-output compliance — the capability most at risk from narrow supervised fine-tuning, and the one invisible to a domain accuracy metric — went from 97.5% to 100%.

Three — an unexplained pattern, flagged rather than buried

The two largest effects are on general multiple-choice tasks, while a domain multiple-choice task moved the other way. A simple “fine-tuning improved likelihood calibration for all multiple-choice scoring” story would predict all three moving together. They do not. With one seed and nothing surviving correction, this is an observation, not a finding.

Why the result is weak, stated plainly

This is a 1.75%-parameter adapter on a 0.5B model, trained for one epoch on 3,633 examples in twenty-two minutes, on a contended laptop. All three of those choices were forced by measured hardware limits, and all three push toward a smaller effect. The experiment is therefore conservative: far more likely to miss a real effect than to invent one.

A null result here is evidence about this run's scale — not evidence that parameter-efficient specialisation does not work. Saying otherwise would be the same over-claim in the opposite direction.

What is not done

Five specialists — Gradient, TransitMind, QuantCore, DataMedic and Reviewer — ship as scaffolded: valid, executable configurations with real licensed evaluations wired up, and no training run and no results. The blocker is data licensing, not engineering. Datasets rejected on licence grounds stay in the registry with the reason recorded, which is more useful than their absence.

Other honest limits: one seed, so run-to-run variance is unmeasured; n-gram contamination screening catches verbatim and near-verbatim leakage but not paraphrase; the deterministic rubrics score whether a response shows reasoning structure, never whether it is correct; math answer matching is string-based after canonicalisation rather than symbolic. CUDA memory figures in the docs are estimates, marked as such — no CUDA host was used, so no CUDA timing is claimed anywhere.

Reproducing it

The data stage is deterministic: the same config and the same upstream revisions produce byte-identical output. Compare the manifest digests between two builds to confirm.

pip install -e '.[train,dev]'
specialists doctor              # what can this host actually run?
specialists data prepare proofsmith
specialists train proofsmith
specialists eval proofsmith --base --adapter runs/<run-id>/adapter
specialists compare proofsmith

make check                      # ruff + mypy + pytest + configs + licences
make smoke                      # full pipeline on a tiny model, seconds

What is not claimed to reproduce bit-exactly: training loss curves. Several accelerator kernels are non-deterministic by design, and float reduction order varies with hardware. The data a run sees is reproducible; the exact gradient arithmetic is not. That is part of why results are reported with intervals.