Maloba

Speech recognition

MMS 1B + adaptateur lingala

Transcribe the same field speech, with output that reads better at character level.

AdapterTraining report F05
Trained in 1 h 50 on 0.23 % of its parameters, the MMS adapter reaches the same word accuracy as Whisper, with lighter errors at character level. It learned to write ɔ and ɛ, two letters the original model did not know.

Result

WER 44,12 %, CER 16,26 % on leak-free V3.2 test, reading C. Source : E09. Voir la page Résultats.

Mesures automatiques (E09)
MesureValeur
WER (mots erronés)44,12 %
CER (caractères erronés)16,26 %

The measurements, as charts

Word error rate, leak-free V3.2 testThe shorter the bar, the better. The thin line is the 95 % confidence interval, bootstrapped over the 19 recordings in the test set.
057114171228Whisper Small, base197,46 %MMS 1B, base58,21 %Whisper + QLoRA44,17 %MMS + adapter44,12 %

Source : E09, lecture C normalisée · data

Show the chart data
ModelWER %95 % interval
Whisper Small, base197,46182,35 – 211,17
MMS 1B, base58,2155,61 – 60,59
Whisper + QLoRA44,1740,71 – 47,78
MMS + adapter44,1241,32 – 47,04
Character error rateAt character level, the MMS adapter beats fine-tuned Whisper: its errors are lighter.
0316192122Whisper Small, base104,32 %MMS 1B, base22,87 %Whisper + QLoRA18,95 %MMS + adapter16,26 %

Source : E09, lecture C normalisée · data

Show the chart data
ModelCER %95 % interval
Whisper Small, base104,3294,84 – 113,24
MMS 1B, base22,8721,39 – 24,25
Whisper + QLoRA18,9516,90 – 20,77
MMS + adapter16,2614,91 – 17,59
Public benchmark: FLEURS, read speechFLEURS is a public set of read speech. The MMS base model stays better on it than its adapter: the adapter specialised on the project's field speech.
0316394125Whisper Small, base116,15 %Whisper + QLoRA39,39 %MMS + adapter22,48 %MMS 1B, base14,66 %

Source : E10, FLEURS ln_cd, 478 phrases · data

Show the chart data
ModelNormalised WER %
Whisper Small, base116,15
Whisper + QLoRA39,39
MMS + adapter22,48
MMS 1B, base14,66

What it does well, what it does badly

What it does well

  • The same word accuracy as Whisper (44.12 % against 44.17 %), with a lower CER: 16.26 % against 18.95 %.
  • It learned to write ɔ and ɛ, two letters the original model did not know.
  • Trained in 1 h 50 on 0.23 % of its parameters: reproducible at low cost.

What it does badly

  • Specialised on field speech: on FLEURS (read speech) the original MMS stays ahead (14.66 % against 22.48 %).
  • CC-BY-NC 4.0 licence inherited from MMS: no commercial use.
  • Human evaluation still under way (E10).

How it was built

Socle facebook/mms-1b-all, adapté par adaptateur de langue seul (0,23 % des paramètres), alphabet étendu à ɔ et ɛ. Données : train V3.2, 47,79 h. Détail dans le rapport F05.

Licence

CC-BY-NC-4.0 : non-commercial. Un modèle garde la licence de son modèle de base, quelle que soit celle des données d'entraînement. Gouvernance des données.