Public intermediate checkpoints let us average a language model's own training trajectory at no training cost. We ask whether averaging the last k of six Pythia-160M checkpoints (steps 143k to 100k) lowers WikiText-2 next-token negative log-likelihood (NLL) relative to the final checkpoint, and relative to the best single checkpoint. Models are evaluated in fp32. Forty validation blocks select, and 100 disjoint test blocks report. Averages with k=2 to 5 lower the test NLL by 0.041 to 0.064 nats, with paired intervals below zero. The average at k=6 does not (-0.0019 nats). Each average also beats its own best member. However, the single 100k checkpoint already has NLL 3.5518, 0.0616 nats below the final one, and validation picks it. The best average (k=5) differs from it by -0.0023 nats [-0.0083,+0.0034]: no difference larger than 0.008 nats detected, without an equivalence test. Pythia was trained in fp16, and an earlier bfloat16 evaluation produced a spurious 0.45-nat gain. The evidence comes from one 160M-parameter model and one corpus.
Download the PDF · Figshare record and reproducibility package