← All publications

Averaging the Last Checkpoints of Pythia-160M: No Difference Larger Than 0.008 Nats From the Best Single Checkpoint, WikiText-2 NLL in fp32

Dheiver Francisco Santos, PhD · Preprint, Figshare, 15 November 2026 · CC BY 4.0 · doi:10.6084/m9.figshare.35014003 · ORCID 0000-0002-8599-9436

Abstract

Public intermediate checkpoints let us average a language model's own training trajectory at no training cost. We ask whether averaging the last k of six Pythia-160M checkpoints (steps 143k to 100k) lowers WikiText-2 next-token negative log-likelihood (NLL) relative to the final checkpoint, and relative to the best single checkpoint. Models are evaluated in fp32. Forty validation blocks select, and 100 disjoint test blocks report. Averages with k=2 to 5 lower the test NLL by 0.041 to 0.064 nats, with paired intervals below zero. The average at k=6 does not (-0.0019 nats). Each average also beats its own best member. However, the single 100k checkpoint already has NLL 3.5518, 0.0616 nats below the final one, and validation picks it. The best average (k=5) differs from it by -0.0023 nats [-0.0083,+0.0034]: no difference larger than 0.008 nats detected, without an equivalence test. Pythia was trained in fp16, and an earlier bfloat16 evaluation produced a spurious 0.45-nat gain. The evidence comes from one 160M-parameter model and one corpus.

Download the PDF · Figshare record and reproducibility package

Cite as: D. F. Santos, “Averaging the Last Checkpoints of Pythia-160M: No Difference Larger Than 0.008 Nats From the Best Single Checkpoint, WikiText-2 NLL in fp32,” Figshare, 2026, doi:10.6084/m9.figshare.35014003.