Results and reports

What works, and what remains hard

Every figure on this page comes from a dated report, listed at the foot of the page. Automatic measurements are replayed by a reproducible evaluation pipeline; human judgements always carry their reliability: “usable” when inter-validator agreement passes the threshold set in advance, “indicative” otherwise.

Automatic measurements

Base model against fine-tuned model, for the five models, with the external comparisons: FLORES+ for translation (Meta's published value is reproduced to within 0.24 points) and FLEURS for speech.

Base model against fine-tuned model (E01, E02, E09, E10)
Model	Measure	Base	Fine-tuned	Test set	Reliability
Whisper Small + QLoRA	WER / CER	—	44,17 % / 18,95 %	Leak-free V3.2 test, 1 797 segments, reading C	usable
MMS 1B + adaptateur	WER / CER	—	44,12 % / 16,26 %	Same test; gap with Whisper not established	usable
MMS 1B + adaptateur	Normalised WER, FLEURS	14,66 %	22,48 %	FLEURS ln_cd, 478 read sentences: domain specialisation	usable
Qwen2-VL 2B + QLoRA	Correct characters (CER)	—	88,7 % (11,3 %)	50 test examples; strict accuracy 28 %	usable
NLLB-Maloba-h1	BLEU, internal test	23,37	32,00	79 sentences; p ≤ 0,001	usable
NLLB-Maloba-h1	BLEU, FLORES+	15,49	16,39	1,012 sentences; p ≤ 0,001	usable
Llama 3.1 8B + QLoRA	ROUGE-L	0,1521	0,2379	964 test examples	usable

The base Whisper and Qwen2-VL models produce no usable Lingala: their measurement is not comparable. Whisper's WER measured in August (46.4 % on V2) is superseded by the E09 re-measurement on a leak-free test set.

Speech recognition: word error rate on the leak-free test setThe shorter the bar, the better. The thin line is the 95 % confidence interval, obtained by bootstrap over the 19 recordings in the test set: the intervals for fine-tuned Whisper and fine-tuned MMS overlap widely, so the gap between the two is not established.

Source : E09, normalised reading C · data

Show the chart data

Model	WER %	95 % interval
Whisper Small, base	197,46	182,35 – 211,17
MMS 1B, base	58,21	55,61 – 60,59
Whisper + QLoRA	44,17	40,71 – 47,78
MMS + adapter	44,12	41,32 – 47,04

Speech recognition: character error rateAt character level the MMS adapter does better than fine-tuned Whisper: its mistakes are lighter and its output easier to read.

Source : E09, normalised reading C · data

Show the chart data

Model	CER %	95 % interval
Whisper Small, base	104,32	94,84 – 113,24
MMS 1B, base	22,87	21,39 – 24,25
Whisper + QLoRA	18,95	16,90 – 20,77
MMS + adapter	16,26	14,91 – 17,59

What the spelling convention costsThe same transcription, judged three ways: against the original reference (A), against the harmonised reference (B), and with both harmonised (C). The gap between A and C is the share of error that comes not from the model but from spelling variation.

Whisper + QLoRA

MMS + adapter

MMS 1B, base

Source : E09 · data

Show the chart data

Model	A — original reference	B — harmonised reference	C — both harmonised
Whisper + QLoRA	47,45 %	47,11 %	44,17 %
MMS + adapter	47,35 %	44,84 %	44,12 %
MMS 1B, base	61,03 %	60,01 %	58,21 %

On read speech, the MMS base model stays aheadFLEURS is a read-speech set, far from the radio broadcasts and service desks the project recorded. The adapter specialised on field speech: this is domain specialisation, not a regression.

Source : E10, FLEURS ln_cd · data

Show the chart data

Model	Normalised WER %
Whisper Small, base	116,15
Whisper + QLoRA	39,39
MMS + adapter	22,48
MMS 1B, base	14,66

E02 — Statistical tests and external translation reference

Paired bootstrap of 1,000 draws, corrected for eight tests. Base → h1: +8.63 BLEU and +7.36 chrF on the internal test, +0.91 BLEU and +1.78 chrF on FLORES+ (p ≤ 0.001 throughout). h1 → h2 (4,700 machine pairs added): no gain on the internal test, a significant drop on FLORES+, so h2 is rejected. Meta reference on FLORES devtest fra → lin: chrF++ 46.2 published, 46.44 measured by CDS for the base, 48.31 for h1.

Translation: what training produced, epoch by epochh1, fine-tuned on 288 human-translated pairs, rises to epoch 4 then falls back: that is the point retained. h2, which adds 4,700 machine-made pairs, improves on its own validation but falls back on FLORES+, and was rejected.

Source : F04, courbes de validation · data

Show the chart data

Series	Point	Validation chrF
h1 — 288 paires humaines	époque 1	45,51
h1 — 288 paires humaines	époque 2	56,23
h1 — 288 paires humaines	époque 3	60,13
h1 — 288 paires humaines	époque 4	62,40
h1 — 288 paires humaines	époque 5	61,68
h2 — rejeté	pas 75	60,64
h2 — rejeté	pas 150	61,65
h2 — rejeté	pas 225	64,31
h2 — rejeté	pas 300	64,24
h2 — rejeté	pas 375	63,86

Translation: base model against fine-tuned modelThe gain is confirmed on an external reference, FLORES+, and not only on the project's own test set. Paired bootstrap of 1,000 resamples, corrected for eight tests.

Test interne — 79 phrases

FLORES+ devtest — 1 012 phrases

Source : E02, tests statistiques

Show the chart data

Test set	Measure	Base	Fine-tuned	Gap
Test interne — 79 phrases	BLEU	23,37	32,00	+8,63
Test interne — 79 phrases	chrF	54,38	61,74	+7,36
FLORES+ devtest — 1 012 phrases	BLEU	15,49	16,39	+0,90
FLORES+ devtest — 1 012 phrases	chrF	50,31	52,09	+1,78

Human review

Three Lingala-speaking validators, three questions, a counting rule fixed before the campaign. Majority verdict (2 out of 3); 95 % Wilson intervals; agreement measured by Krippendorff's alpha, threshold 0.60 set in advance. Responses kept by CDS under pseudonym. Validation methodology note (PDF) (opens in a new window)

What was judged (E03 to E08)
What was judged	Result	Reliability
Corpus (310 échantillons, 930 jugements)	Sens préservé dans 96,5 % des cas ; 3 % seulement d'erreurs de fond	indicative weak agreement on “correct” or “partial”
Lecture de documents (25 images)	32 % parfaites, 92 % récupérables	usable accord suffisant (α 0,665)
Traduction (38 phrases)	66 % correctes, 97 % récupérables	indicative α 0,19 ; paire V1–V2 0,73
Reconnaissance vocale (25 segments)	16 % parfaits, 100 % récupérables	indicative α 0,26

What three validators answered, model by modelMajority verdict, two validators out of three. Only document reading reaches the agreement threshold of 0.60 set before the campaigns: the other two results are published as indicative.

Source : E06, E07, E08

Show the chart data

What was judged	Sample	Correct	Partly correct	Incorrect	Agreement α
tradNllb	38 phrases	66 %	32 %	2 %	0,190
docQwen	25 images	32 %	60 %	8 %	0,665
voixWhisper	25 segments	16 %	84 %	0 %	0,258

Agreement between validators, campaign by campaignThe 0.60 threshold was set before the campaigns. Only one campaign passes it: document reading. Weak agreement does not mean the data is bad, but that the line between “correct” and “partly correct” is not the same for everyone — largely because Lingala spelling is not settled.

Source : E03 to E08

Show the chart data

Campaign	Krippendorff's alpha
ASR corpus (E03)	0,420
OCR corpus (E04)	0,530
LLM corpus (E05)	0,080
Speech (E06)	0,258
Documents (E07)	0,665
Translation (E08)	0,190

Documented

Limits at closure

58,1 h annotated and verified, below the revised target of 75 h and below the 80–100 h range of the terms of reference.
Lingala spelling is not settled: the variation accounts for about 5 % of WER on the references.
Disagreement between human transcribers reaches 12 % at character level: that is the measured floor.
Inter-validator agreement stays below the 0.60 threshold set before the campaigns: the rates are published as indicative.
The test sets hold 19 recordings, 50 images and 79 sentences.
The number of distinct speakers in corpus V3.2 is not established.
Human evaluation of Llama 3.1 was not carried out: the resources went to translation.

Documented

Resources gathered, outside the annotated corpus

Health Voice-a-thon recordings: 11 recordings, no duration of speech published.
About 150 h of Radio Rurale magnetic tape, not digitised.
SIL Congo written corpus: 22 documents, 1 008 pages, used for document reading and not released as running text.
1 282 harmonised OCR lines, kept outside the published training set.

Evaluationreports

Evaluations E01 to E10, frozen on September 29, 2026. Pseudonymised versions: no validator response carries a name. Index of evaluations (PDF) (opens in a new window)

Evaluations
Réf.	Evaluation	Set and size	Main result	Reliability	Report	Files
E01	Mesures automatiques des modèles Automatique · Définitif ; WER ASR complété par E09	Tests internes ; FLORES+	ROUGE-L 0,2379 ; CER OCR 11,3 % ; BLEU 32,00	exploitable	PDF (opens in a new window)	—
E02	Tests statistiques et référence externe de la traduction Automatique · Définitif	Test interne (79) ; FLORES+ devtest (1 012)	+8,63 BLEU et +0,91 sur FLORES+ (p ≤ 0,001) ; chrF++ de la base 46,44 contre 46,2 publié par Meta	exploitable	PDF (opens in a new window)	—
E03	Validation humaine du corpus ASR Humaine, volet 1 · Accord faible (α 0,25 à 0,42)	88 segments	43 à 44 % correct ; 97 à 100 % correct ou partiel	indicatif	PDF (opens in a new window)	1
E04	Validation humaine du corpus OCR Humaine, volet 1 · Accord faible (α 0,49 à 0,53)	91 images	Test : 84 % correct ; 96 % correct ou partiel	indicatif	PDF (opens in a new window)	1
E05	Validation humaine du corpus LLM Humaine, volet 1 · Accord faible (α 0,08)	131 exemples	Test : 58 % correct ; 98 % correct ou partiel	indicatif	PDF (opens in a new window)	1
E06	Évaluation humaine de Whisper affiné Humaine, volet 2 · Indicatif (α 0,26)	25 segments du test	16 % correct ; 100 % correct ou partiel	indicatif	PDF (opens in a new window)	1
E07	Évaluation humaine de Qwen2-VL affiné Humaine, volet 2 · Exploitable (α 0,665)	25 images du test	32 % correct ; 92 % correct ou partiel	exploitable	PDF (opens in a new window)	1
E08	Évaluation humaine de NLLB-Maloba-h1 Humaine, volet 2 · Indicatif (α 0,19 ; paire V1–V2 0,73)	38 phrases du test	66 % correct ; 97 % correct ou partiel	indicatif	PDF (opens in a new window)	1
E09	Remesure ASR sans fuite (V3.2) Automatique · Définitif	Test V3.2 (1 797 segments, 19 enregistrements)	WER 44,12 % (MMS affiné) et 44,17 % (Whisper affiné) ; CER 16,26 % et 18,95 %	exploitable	PDF (opens in a new window)	10
E10	MMS affiné : FLEURS et évaluation humaine Automatique et humaine · FLEURS définitif ; humain en cours	FLEURS (478 phrases) ; 25 segments × 2 modèles	FLEURS : WER 22,48 % (MMS d'origine 14,66 %) : spécialisation de domaine	indicatif	PDF (opens in a new window)	5

Trainingreports

Entraînements F01 à F05 : données, configuration complète, courbes, point retenu, résultat officiel, défauts connus. Index of training runs (PDF) (opens in a new window)

Training runs
Réf.	Model	Method	Data	Official result	Report	Files
F01	Llama 3.1 8B Instruct, QLoRA	QLoRA, rang 16	5 700 exemples Alpaca, 5 registres	ROUGE-L 0,2379 (base 0,1521)	PDF (opens in a new window)	2
F02	Whisper Small, QLoRA	QLoRA 4 bits NF4, rang 32	train V2, augmentation figée	WER 44,17 % sur V3.2 (E09)	PDF (opens in a new window)	3
F03	Qwen2-VL 2B Instruct, QLoRA	QLoRA, rang 32, 4 jetons ajoutés	417 exemples image–texte	88,7 % au caractère (CER 11,3 %)	PDF (opens in a new window)	2
F04	NLLB-200 distilled 1.3B, affinage complet	Affinage complet	288 paires humaines	BLEU 32,00 ; FLORES+ 16,39	PDF (opens in a new window)	14
F05	MMS 1B, adaptateur de langue	Adaptateur de langue, 0,23 % des paramètres	train V3.2, 47,79 h	WER 44,12 %, CER 16,26 % sur V3.2	PDF (opens in a new window)	4

The contractual deliverables (L3, L4, L5) are not released on this site.

Files and logs

What produced each result: run logs, raw scores, training curves, release records and validator responses. The responses are pseudonymised: no name and no contact details are released.

E03Validation humaine du corpus ASR1 pièce