[Paper Review] Goldfish: Monolingual Language Models for 350 Languages

• Overview
• TL;DR
• Introduction
• Key Contributions
• Background and Related Work
  • Limits of massively multilingual pretraining
  • What byte premium is
  • Baselines
• Method and Architecture
  • Architecture choice
  • Tokenizer
  • Input format
  • Training hyperparameters
  • Bigram baseline
• Training Objective
• Data and Training Pipeline
  • Sources
  • Preprocessing
  • Final corpus statistics
  • Hardware
• Experimental Results
  • FLORES log-perplexity
  • Bigrams beat large multilingual LMs
  • Downstream benchmarks
• Analysis
  • Why a small monolingual model beats a 10B multilingual one
  • Contamination analysis
  • What the reasoning ceiling means
• Limitations and Critical Assessment
• Takeaways
• Installation and Use
• References
• Further Reading

Overview

TL;DR

• Goldfish is a suite of 1,154 GPT-2-style monolingual language models trained for 350 languages at five dataset brackets (5MB, 10MB, 100MB, 1GB, full). The 1GB models are 125M parameters; the 5/10MB models are 39M.
• Dataset sizes are normalised by byte premium so that “1GB” means content-matched 1GB, not raw bytes of the target language — without this normalisation, languages with bulky UTF-8 encodings (e.g. Khmer at 3.91×) would receive a quarter of the actual training content.
• The small monolingual models outperform XGLM 4.5B, BLOOM 7.1B, and MaLA-500 10B on FLORES perplexity — beating MaLA-500 on 111 of 175 shared languages (63%).
• Strikingly, plain stupid-backoff bigrams beat XGLM 4.5B on 24% of languages and BLOOM 7.1B on 43%. Massively multilingual models have not learned basic next-token prediction for many low-resource languages.
• The catch: reasoning benchmarks (Belebele, XCOPA, XStoryCloze) are near chance for every sub-1B model, Goldfish included. Better perplexity does not translate into reasoning at this scale.

Introduction

Most current research in low-resource language modeling leans on massively multilingual models — XLM-R, BLOOM, XGLM, MaLA-500 — covering tens to hundreds of languages with a single set of parameters, on the intuition that multilingual data should help languages with too little of their own. Two well-known headaches challenge this intuition. First, the curse of multilinguality (Conneau et al., 2020b): squeeze too many languages into one model and per-language quality drops. Second, training data is grotesquely skewed — XGLM 4.5B’s Norwegian corpus is 7,000× the size of its Quechua corpus, and BLOOM trained on 70KB of Akan text despite 8.1M native speakers.

This paper quantifies the consequences. The authors note that of the 350 candidate languages, 215 had no monolingual text-generation model at all before Goldfish, and 47 had no text-generation model of any kind (Arnett and Chang, 2025). To fill that gap they release one monolingual model per language so the community has at least a baseline — for fine-tuning, for downstream comparison, for measuring data quality, and for sanity-checking language identification.

The surprising thing is that the baseline turns out to be better than the massively multilingual models. A 125M-parameter GPT-1-sized monolingual model achieves lower perplexity than XGLM 4.5B, BLOOM 7.1B, and MaLA-500 10B, and even simple bigram models edge out the large models on a non-trivial fraction of languages. The interpretation: large multilingual LMs are being released without having learned the most basic task — next-token prediction — for many of the languages they nominally cover.

At a moment when low-resource NLP is increasingly pulled toward an LLM-as-a-service paradigm, Goldfish makes the case that small monolingual baselines remain useful, and in some respects superior. The compute budget — 1.65 × 10²⁰ FLOPs total, about 1/1900 of GPT-3’s training run — also frames this as something a compute-limited lab can replicate.

Key Contributions

• 1,154 monolingual models across 350 languages, in five dataset brackets (5MB / 10MB / 100MB / 1GB / full) after byte-premium scaling. For 215 of these languages, Goldfish is the first publicly available monolingual model.
• Beating massively multilingual baselines on FLORES log-perplexity. Goldfish 124M outperforms XGLM 4.5B, BLOOM 7.1B, and MaLA-500 10B on the 100 languages they all share — and wins on 63% of the 175 languages shared with MaLA-500.
• Documenting that simple bigrams beat large multilingual LMs. Stupid-backoff bigrams reach lower log-perplexity than XGLM 4.5B on 24% of languages and BLOOM 7.1B on 43%. Direct evidence that large multilingual models fail at basic LM for many low-resource languages.
• Best-in-class grammaticality among sub-1B models. On MultiBLiMP, Goldfish 124M scores 78.8, beating Gemma 3 270M (72.3), BLOOM 560M (64.2), and even Gemma 3 1B (78.0) by a hair.
• A clean, reproducible use of byte-premium scaling for cross-lingual dataset-size comparison, demonstrated across 350 languages.

Background and Related Work

Limits of massively multilingual pretraining

Multilingual pretraining has been justified along two lines. The first is transfer: similar-language data helps a target low-resource language (Conneau et al., 2020b). The second is scale: bigger models and bigger data should reduce per-language loss. Both hold partially but break in obvious ways.

• Curse of multilinguality (Conneau et al., 2020b; Chang et al., 2024a): for a given model size, adding languages eventually hurts per-language quality.
• Severe data imbalance: XGLM 4.5B sees Norwegian (71GB; 5.4M speakers) vs Quechua (0.01GB; 7.3M speakers); BLOOM was trained on 0.07MB of Akan (4e-6% of pretraining). From the model’s perspective these are essentially unseen languages.

Goldfish’s authors’ prior paper Chang et al. (2024a) compared monolingual vs multilingual setups across 250 languages with up to 45M-parameter models, finding that smaller models and bigger per-language datasets both favour monolingual. Goldfish is essentially that result productionised — 1,154 ready-to-use models built on the same hypothesis.

What byte premium is

Different languages need different numbers of UTF-8 bytes to encode equivalent content. English is the baseline; Khmer (khm_khmr) needs about 3.91× as many bytes for the same meaning. If you compare two “1GB models,” the English one has seen ~10× more content than the Khmer one. Naive byte-based dataset sizes are not a fair basis for cross-lingual comparison.

Arnett et al. (2024) introduced a byte premium tool that measures or regresses the per-language premium $b$ using character entropy and script type (alphabet/abjad/abugida/logography). Goldfish applies this scaling everywhere — dataset sizes are reported as English-equivalent MB, not raw bytes. A small accounting choice, but without it the cross-lingual conclusions wouldn’t be well-defined.

Baselines

• XGLM (Lin et al., 2022): GPT-style multilingual LM, 4.5B params for 134 languages or 7.5B for 30 high-resource languages.
• BLOOM (Scao et al., 2022): 46-language model at 7.1B and 176B params, includes very-low-resource cases like Akan.
• MaLA-500 (Lin et al., 2024): LLaMA 2 7B/10B continued-pretrained on Glot500 for 534 languages — the most direct apples-to-apples comparison.

Method and Architecture

Architecture choice

All Goldfish use GPT-2 (Radford et al., 2019) with two sizes determined by dataset bracket:

• 125M parameters for 100MB / 1GB / full — equivalent to GPT-1, BERT-base, RoBERTa. 12 layers, hidden 768, 12 heads, FFN 3072.
• 39M parameters for 5MB / 10MB — Turc et al. (2019)’s small BERT. 4 layers, hidden 512, 8 heads, FFN 2048.

The principle is small data → small model to avoid overfitting, validated in Chang et al. (2024a). Larger models hurt on very small datasets even after careful regularisation.

Tokenizer

Each language gets its own Unigram SentencePiece tokenizer with 50K vocab (Kudo and Richardson, 2018), trained on a 100MB-capped sample of the same dataset used for the corresponding model. Because the tokenizer is monolingual it stays efficient for non-English languages, which would otherwise approach byte-level fallback under a shared multilingual vocab.

Input format

After tokenisation, text lines are concatenated into exactly 512-token sequences. Shuffling happens just before sampling so that contiguous context is preserved where possible (some upstream corpora are already shuffled, in which case this is moot). 512 is short by modern standards but reasonable for small datasets and keeps training fast.

Training hyperparameters

• Learning rate: 1e-4 uniformly, smaller than GPT-1’s 2.5e-4, motivated by Chang et al. (2024a)’s finding that 2e-4 works for small models — smaller LRs reduce overfit on tiny data.
• Batch size: 4 / 8 / 32 / 64 for 5MB / 10MB / 100MB / 1GB. Larger batches overfit small data quickly. Full-data models use the batch size of the largest bracket they reach.
• Epochs: 10. Multi-epoch helps in data-constrained regimes (Muennighoff et al., 2023), but past 10 epochs eval loss starts to climb in the 5MB bracket.
• Warmup: 10% of pretraining steps, linear decay thereafter.
• Compute: ~15,600 A6000 GPU hours for all 1,154 models, totaling 1.65 × 10²⁰ FLOPs — about 1/1900 of GPT-3’s training.

Bigram baseline

A stupid-backoff bigram (Brants et al., 2007) is trained per language on the same tokenized data: $P(w_i \mid w_{i-1})$ from raw counts, with backoff to unigram weighted by $\lambda = 0.40$ when the bigram is unobserved. Higher-order $n$-grams are excluded because they back off too often. This tiny baseline is what makes the central headline result — large multilingual LMs failing to beat bigrams — possible.

Training Objective

The standard causal LM loss for GPT-2:

with $s$ a 512-token sequence and $\theta$ the 39M or 125M parameters. Evaluation uses a sentence-level log-perplexity to neutralise tokenizer differences across compared models:

Each FLORES sentence $s$ is cut in half by character count into $s_0$, $s_1$. The model gets $s_0$ as prompt and is scored on log-probability of $s_1$, rounded to the nearest subword boundary. [UNK] is handled at $1/|\mathcal{V}|$, neutralising the trivial trick where unseen scripts artificially compress perplexity. The setup matters: a naive sequence-level perplexity gives compactly-tokenised models a free advantage, and across 134-language comparisons that matters.

Data and Training Pipeline

Sources

Goldfish merges roughly 80 corpora, in three priority layers:

1. Chang et al. (2024a)’s 250-language collection: OSCAR, Wikipedia, NLLB seed, eBible, AfriBERTa, NusaX, AmericasNLI, and many more (~40 sources).
2. Glot500 (Imani et al., 2023): only for languages not already at 1GB after layer 1.
3. MADLAD-400 (Kudugunta et al., 2023) clean split: CommonCrawl-derived, added only when layers 1+2 still fall short. For languages with very short average line length (<25 tokens), MADLAD-400 contiguous sequences are appended so the LM gets useful context windows.

Preprocessing

• Deduplication at 100-UTF-8-byte chunks (Lee et al., 2022) — critical because the same language appears across multiple sources.
• Language code canonicalisation: ISO 639-3 + ISO 15924 (e.g. khm_khmr). Macrolanguages and individual languages are both kept (que_latn for Quechua broadly, quy_latn for Ayacucho, quz_latn for Cusco). Some redundant codes are dropped (ory_orya Odia merges into ori_orya).
• FLORES and AmericasNLI held out from training. To audit for contamination the authors check how often the first 10 tokens of each FLORES sentence appear in the per-language training set: 98% of languages have fewer than 10 contaminating sequences (out of 2000). The two exceptions are Samoan (7,155 hits) and Central Kanuri (371). Reported but not excluded.
• Bible-only languages dropped. A corpus that is only Bible translation is too domain-skewed to support LM training.

Final corpus statistics

• 547GB of text covering 523 language-script combinations (486 unique language codes, 32 unique script codes).
• The 350 Goldfish languages train 1,154 models total: 350 at 5MB, 288 at 10MB, 166 at 100MB, 83 at 1GB, 267 full-data.
• “1GB” is post byte-premium scaling, so the actual on-disk size ranges 0.7–5.0×. Burmese (mya_mymr) hits the 5.0 clip.

Hardware

A heterogeneous mix of older and newer NVIDIA cards (GTX TITAN X through RTX A6000), single GPU per run. ~15,600 A6000-equivalent hours for pretraining (about 651 GPU-days). Inference for FLORES + four downstream benchmarks adds ~250 A6000 hours, dominated by inference on the much larger comparison models rather than on Goldfish itself.

Experimental Results

FLORES log-perplexity

Headline numbers, on the 73 languages where Goldfish has a 1GB model:

• Goldfish 76.9 — Bigrams 112.3 — XGLM 4.5B 78.6 — MaLA-500 10B 84.7.

The 73 + 22 + 5 = 100 languages in Table 1 are the ones present in XGLM 4.5B, MaLA-500, and FLORES simultaneously, partitioned by Goldfish’s max dataset size. The 10/5MB row is the most striking — even with just 5MB of content-matched training data, Goldfish (130.5) beats XGLM 4.5B (183.1) by a wide margin. A 5MB monolingual model is a better next-token predictor than a 4.5B multilingual one on those languages.

Bigrams beat large multilingual LMs

This table carries the strongest message in the paper. Two signals:

1. Goldfish vs large multilingual. Goldfish wins 60/102 against XGLM 4.5B (59%) and 111/175 against MaLA-500 10B (63%). The single exception is XGLM 7.5B — but XGLM 7.5B only covers 30 high-resource languages, and Goldfish wins only 2/30 there.
2. Bigrams vs large multilingual. Plain backoff bigrams beat XGLM 4.5B on 24/102 (24%) and BLOOM 7.1B on 20/46 (43%). This means the large LMs aren’t even producing better-than-unigram next-token distributions for a substantial fraction of their nominal coverage.

XGLM 7.5B as outlier underscores the curse-of-multilinguality reading: concentrate on fewer languages and the large model wins; spread across many and it collapses.

Downstream benchmarks

• MultiBLiMP (grammaticality, 74 languages): Goldfish 124M = 78.8, the best of any model tested. Gemma 3 270M = 72.3, BLOOM 560M = 64.2, XGLM 564M = 66.6. Even the 1B Gemma 3 (78.0) and LLaMA 3 (77.4) sit slightly behind.
• Belebele (121-language reading comprehension): everything is at chance (chance = 25, all models 22.7–29.0).
• XCOPA / XStoryCloze: chance + ~5% for every sub-1B model.

Two readings. First, small monolingual gives outsized returns on grammatical knowledge — MultiBLiMP scores models on next-token log-probability ranking without fine-tuning, so it measures grammatical competence directly. Second, reasoning is scale-bound below 1B — neither XGLM 564M, Gemma 3 270M, Goldfish 124M, nor BLOOM 560M actually reasons on these tasks. The Goldfish setup just confirms it across 350 languages.

Analysis

Why a small monolingual model beats a 10B multilingual one

The paper doesn’t ablate this directly, but the contributing factors are clear:

1. Capacity allocation. XGLM 4.5B’s 4.5B parameters spread over 134 languages averages ~34M per language. Goldfish 124M dedicates all parameters to one language — about 4× more, applied monolingually.
2. Tokenizer fit. Each Goldfish vocab is 50K monolingual tokens. Multilingual tokenizers (100K–250K vocab over hundreds of languages) approach byte-level fallback on low-resource languages.
3. Data balancing. Byte-premium scaling gives Goldfish content-matched 1GB. Multilingual training inevitably gets pulled toward high-resource sampling even with temperature reweighting.

The 24–43% bigram-wins fraction is the striking number. For bigrams to beat a 7.1B LM, the LM’s next-token distribution must be worse than a unigram-smoothed distribution — i.e. the LM has essentially not learned the language.

Contamination analysis

If FLORES is leaked into training, the perplexity claims would be hollow. The authors search the training data for the first 10 tokens of every FLORES sentence and find <10 hits per 2000 sentences for 98% of languages. Only Samoan (7,155 hits) and Central Kanuri (371) show non-trivial contamination. The reported averages include those languages, but removing them is unlikely to change the conclusions — and the authors flag it explicitly.

What the reasoning ceiling means

The fact that no sub-1B model — Goldfish, XGLM 564M, BLOOM 560M, Gemma 3 270M — clears chance on Belebele/XCOPA/XStoryCloze suggests this is a property of base LMs without instruction tuning at this scale, not a Goldfish-specific weakness. Even Gemma 3 1B and LLaMA 3 1B sit at 55–60 (chance = 50). Goldfish extends that observation to 350 languages and confirms the ceiling is global.

Limitations and Critical Assessment

Authors’ stated limitations:

• Comparability and availability. 5MB isn’t available for every language; corpus quality and domain (news vs social vs religious) vary. Even with byte premium scaling, perfect comparability is unachievable.
• Monolinguality. A deliberate design choice but a real cost: similar-language transfer (Conneau et al., 2020b) is forfeited. Goldfish trades coverage for clarity.
• Sizes capped at 1GB / 125M. Reasonable for low-resource focus, but small relative to high-resource state-of-the-art.
• Downstream evaluation limited to perplexity + grammaticality. Reasoning evaluations are at chance and thus uninformative.
• No PII or content-safety guarantees beyond using publicly released corpora that themselves may not be audited.

Reviewer additions:

• Bigram fairness questions. Bigrams use the same monolingual tokenizer and the same training data, but still beat XGLM 4.5B on 24% of languages despite seeing no long-range context. This could mean XGLM doesn’t use long-range context for low-resource languages — or it could mean FLORES log-perplexity with a half-sentence prompt already truncates the useful context. The two interpretations have different policy consequences and the paper doesn’t disentangle them.
• Per-language compute framing. Total compute (1.65 × 10²⁰ FLOPs) is presented as 1/1900 of GPT-3. Per-language compute is 1.43 × 10¹⁷ FLOPs, about 1/2,200,000 of GPT-3. The “tiny model can do this” message depends on which framing you take.
• Monolingual-only utility. Low-resource speakers usually want their language paired with English or another high-resource language for downstream tasks. Goldfish is only the first half of that pipeline — fine-tuning or paired-model setups are still required.
• Long-term hosting. 1,154 HuggingFace repos is a lot to maintain. The paper doesn’t commit to long-term curation, model cards, or content updates, which matters if Goldfish is to become a community resource rather than a snapshot.

Takeaways

• Massively multilingual is not free. For low-resource languages a 4.5B multilingual model loses to a 39M monolingual model on basic perplexity. Future multilingual-model design needs to take per-language capacity allocation seriously rather than treating “more languages” as a strict win.
• The bigram result is genuinely shocking. Plain backoff bigrams beat XGLM 4.5B on 24% and BLOOM 7.1B on 43% of languages. If you build the next massively multilingual model, you should treat this number as a falsification target.
• Baseline coverage matters. 215 of these 350 languages had no monolingual LM at all before this release. That alone enables a wave of follow-up work — fine-tuning baselines, language ID validation, corpus quality measurement.
• Byte-premium scaling should be standard. Goldfish’s headline comparisons fail to hold without it. There’s no reason future multilingual evaluation papers shouldn’t normalise the same way.
• Reasoning isn’t going to come from sub-1B base LMs. Better perplexity doesn’t deliver Belebele/XCOPA. The future of low-resource NLP is probably small monolingual LM + something else — instruction-tuning, retrieval, or efficient transfer to a larger backbone.

Installation and Use

All models are hosted on Hugging Face under goldfish-models/<lang_script>_<size>.

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("goldfish-models/eng_latn_1000mb")
model = AutoModelForCausalLM.from_pretrained("goldfish-models/eng_latn_1000mb")

prompt = "Goldfish are"
ids = tokenizer(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=20, do_sample=False)
print(tokenizer.decode(out[0]))
# → "Goldfish are a species of fish that are found in the sea."

Swap the dataset suffix (_5mb, _10mb, _100mb, _full) and language code (khm_khmr, swa_latn, mya_mymr, …) as needed. Training corpora live at goldfish-models/fish-food, and evaluation code (FLORES log-perplexity, contamination check) is in the GitHub repo under evaluation/.

References

• Paper: https://arxiv.org/abs/2408.10441
• Code: https://github.com/tylerachang/goldfish
• Models (Hugging Face): https://huggingface.co/goldfish-models
• Datasets: https://huggingface.co/datasets/goldfish-models/fish-food

Further Reading

• When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages (Chang et al., EMNLP 2024) — the direct predecessor: systematic monolingual-vs-multilingual comparison across 250 languages, concluding small models + enough monolingual data favour monolingual.
• A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages (Arnett et al., SIGUL @ LREC-COLING 2024) — original paper for the byte premium tool. Measures and regresses byte premiums across 1,155 languages, the underpinning of Goldfish’s fair-comparison story.
• Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages (Imani et al., ACL 2023) — one of Goldfish’s main data sources, plus a 511-language continued-pretraining of XLM-R.
• MaLA-500: Massive Language Adaptation of Large Language Models (Lin et al., 2024) — Goldfish’s closest comparison: LLaMA 2 10B continued-pretrained on Glot500 for 534 languages.
• Few-shot Learning with Multilingual Language Models (XGLM) (Lin et al., EMNLP 2022) — the XGLM 4.5B / 7.5B baselines used throughout the paper.
• MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs (Jumelet et al., TACL 2026) — the grammaticality benchmark Goldfish wins. Auto-generated minimal pairs from Universal Dependencies + UniMorph, 101 languages, 128K pairs.