Maloba

Speech recognition

Whisper Small + QLoRA

Transcribe a radio broadcast, an exchange at a service desk, a consultation.

Merged weightsAdaptersTraining report F02

Result

WER 44,17 %, CER 18,95 % on leak-free V3.2 test, reading C. Source : E09. Voir la page Résultats.

Mesures automatiques (E09)
MesureValeur
WER (mots erronés)44,17 %
CER (caractères erronés)18,95 %

Jugement humain (trois validateurs) : 16 % de sorties correctes à la majorité, 100 % correctes ou récupérables ; accord α 0,26 : indicatif.

The measurements, as charts

Word error rate, leak-free V3.2 testThe shorter the bar, the better. The thin line is the 95 % confidence interval, bootstrapped over the 19 recordings in the test set.
057114171228Whisper Small, base197,46 %MMS 1B, base58,21 %Whisper + QLoRA44,17 %MMS + adapter44,12 %

Source : E09, lecture C normalisée · data

Show the chart data
ModelWER %95 % interval
Whisper Small, base197,46182,35 – 211,17
MMS 1B, base58,2155,61 – 60,59
Whisper + QLoRA44,1740,71 – 47,78
MMS + adapter44,1241,32 – 47,04
Character error rateAt character level, the MMS adapter beats fine-tuned Whisper: its errors are lighter.
0316192122Whisper Small, base104,32 %MMS 1B, base22,87 %Whisper + QLoRA18,95 %MMS + adapter16,26 %

Source : E09, lecture C normalisée · data

Show the chart data
ModelCER %95 % interval
Whisper Small, base104,3294,84 – 113,24
MMS 1B, base22,8721,39 – 24,25
Whisper + QLoRA18,9516,90 – 20,77
MMS + adapter16,2614,91 – 17,59
Public benchmark: FLEURS, read speechFLEURS is a public set of read speech. The MMS base model stays better on it than its adapter: the adapter specialised on the project's field speech.
0316394125Whisper Small, base116,15 %Whisper + QLoRA39,39 %MMS + adapter22,48 %MMS 1B, base14,66 %

Source : E10, FLEURS ln_cd, 478 phrases · data

Show the chart data
ModelNormalised WER %
Whisper Small, base116,15
Whisper + QLoRA39,39
MMS + adapter22,48
MMS 1B, base14,66

What it does well, what it does badly

What it does well

  • Field speech with noise and Lingala–French code-switching: the Whisper foundation model is robust there.
  • Every test transcription is judged recoverable by the human validators.
  • Full model and adapters published under Apache 2.0: usable commercially.

What it does badly

  • 44 % word error on the leak-free test set: one transcription in two needs proofreading.
  • Frequent spelling mistakes on ɔ and ɛ, which the model only learned indirectly.
  • Only 16 % of segments are judged perfect by a majority (weak agreement: indicative).

How it was built

Socle openai/whisper-small, adapté par qLoRA 4 bits, LoRA rang 32. Données : train V2, augmentation figée. Détail dans le rapport F02.

Licence

Apache-2.0 : usable commercially. Un modèle garde la licence de son modèle de base, quelle que soit celle des données d'entraînement. Gouvernance des données.