Choosing the Weights of Small Open Language Models: Ten Preprints
Dheiver Francisco Santos, PhD · ORCID 0000-0002-8599-9436 · Collection doi:10.6084/m9.figshare.c.8840665 · CC BY 4.0
- Validation-Selected Linear Interpolation Between Qwen2.5-0.5B Base and Instruct: Does It Beat Both Parents on ARC, HellaSwag and WinoGrande?
doi:10.6084/m9.figshare.35013979 · PDF
Interpolating the weights of a pretrained model and its fine-tuned sibling has beaten both endpoints in some vision settings. We ask whether the same holds for a small language model. We interpolate linearly between Qwen2.5-0.5B and Qwen2.5-0.5B-Instruct on el…
- SLERP versus LERP for Merging Qwen2.5-0.5B Base and Instruct: A Paired Comparison on ARC, HellaSwag, and WinoGrande
doi:10.6084/m9.figshare.35013982 · PDF
Spherical linear interpolation (SLERP) is an alternative to linear interpolation (LERP) for merging fine-tuned language models. We merge Qwen2.5-0.5B and its Instruct sibling at four coefficients with both operators and score macro accuracy on 3,967 held-out i…
- Extrapolating the Instruct Task Vector of Qwen2.5-0.5B: Scaling λ Beyond 1 Under Validation-Selected Coefficients on ARC, HellaSwag, and WinoGrande
doi:10.6084/m9.figshare.35013988 · PDF
Task arithmetic adds a scaled task vector, the difference between a fine-tuned and a pre-trained model, to the pre-trained weights. We ask whether extrapolating past the fine-tuned model (λ>1) improves accuracy when λ is chosen on held-out validation data. We …
- Merging Qwen2.5-0.5B-Instruct with Qwen2.5-Coder-0.5B-Instruct: Weighted Sums, TIES and DARE Fall Below Both Parents, and the Coder Task Vector Is 13 Times Larger
doi:10.6084/m9.figshare.35013991 · PDF
We test whether a three-model merge of Qwen2.5-0.5B (base), Qwen2.5-0.5B-Instruct and Qwen2.5-Coder-0.5B-Instruct beats the best original model and the best two-model merge. The pre-registered grid (weights from 0.25 up) covers weighted sums, TIES (density 0.5…
- Layer-Group Coefficient Search for Qwen2.5-0.5B Merging: A Null Test of Layer-Group Search
doi:10.6084/m9.figshare.35013997 · PDF
Interpolating Qwen2.5-0.5B and Qwen2.5-0.5B-Instruct with one global coefficient t leaves open whether per-layer coefficients help. We pre-registered that a coordinate search over four layer groups would beat the global t^* on validation, with a smaller advant…
- Replicating Validation-Selected Weight Interpolation on SmolLM2-360M Base and Instruct: ARC, HellaSwag and WinoGrande
doi:10.6084/m9.figshare.35014000 · PDF
Selecting an interpolation coefficient between a base language model and its instruct sibling is often presented as a free improvement over both. We replicate a pre-registered test of this claim on SmolLM2-360M and its instruct version, using the same protocol…
- Averaging the Last Checkpoints of Pythia-160M: No Difference Larger Than 0.008 Nats From the Best Single Checkpoint, WikiText-2 NLL in fp32
doi:10.6084/m9.figshare.35014003 · PDF
Public intermediate checkpoints let us average a language model's own training trajectory at no training cost. We ask whether averaging the last k of six Pythia-160M checkpoints (steps 143k to 100k) lowers WikiText-2 next-token negative log-likelihood (NLL) re…
- Winner's Curse in Validation-Selected Model Merges: A Resampling Study on the Qwen2.5-0.5B Interpolation Grid (ARC, HellaSwag, WinoGrande)
doi:10.6084/m9.figshare.35014006 · PDF
Merge coefficients are usually picked on a small validation set. The maximum over many candidates inflates the winner's validation score (the winner's curse); we did not find a measurement of this inflation for model merging. We measure it on 11 linear interpo…
- How Many Validation Items Does Merge-Coefficient Selection Need? Selection Regret on Qwen2.5-0.5B Interpolations (ARC, HellaSwag, WinoGrande)
doi:10.6084/m9.figshare.35014009 · PDF
How large must the validation set be when selecting a merge coefficient? We take an 11-point linear-interpolation grid between Qwen2.5-0.5B and its Instruct variant, scored by the macro average of four multiple-choice benchmarks. We measure selection regret (t…
- The Language-Modeling Cost Along the Base-to-Instruct Weight Interpolation Path of Qwen2.5-0.5B: WikiText-2 NLL versus ARC, HellaSwag and WinoGrande
doi:10.6084/m9.figshare.35014015 · PDF
Linear interpolation between a base language model and its instruction-tuned version, θ(t)=(1-t),θ_base+t,θ_inst, is often tuned for task accuracy alone. We measure what the same path does to language-modeling quality. For Qwen2.5-0.5B and its Instruct version…