In August, the site announced a WER of 0.534 for Whisper “at the best checkpoint”, a CER of 0.170 for document reading and “every annotation validated by the language lead”. In September, we took every measurement again from the start.
What changed
- A single evaluation suite, maloba_eval, with more than 1,400 automated tests, which replays each measurement from the published weights.
- A leak-free speech test set (V3.2): no recording in the test set was used for training.
- A reading rule fixed in advance: harmonised reference against harmonised output (reading C), with confidence intervals computed by recording and not by segment.
- An external reference for translation: Meta's published value on FLORES is reproduced to within 0.24 points.
What the new figures say
Document reading reaches 88.7 % correct characters, confirmed by human validators. Translation improves significantly. Speech recognition remains hard: 44 % word error on field speech, radio and service desk, with noise and language switching. We publish both.
Saying what works and what remains hard. That candour is what makes the rest credible.
By The Maloba team, September 28, 2026.