Data Augmentation and Evaluation Quality in Low-Resource Kikuyu Speech Recognition: A Whisper Case Study

October 29, 2026 · Allan Karanja

Data Augmentation and Evaluation Quality in Low-Resource Kikuyu Speech Recognition: A Whisper Case Study

Abstract

Automatic speech recognition (ASR) for low-resource African languages faces challenges arising from limited transcribed speech, inconsistent annotation practices, and substantial variability across speakers and recording conditions. This study investigates audio augmentation and evaluation reliability for Kikuyu (Gĩkũyũ) ASR using OpenAI’s Whisper architecture.

We first conduct controlled augmentation experiments with Whisper-small, comparing an unaugmented baseline against speed perturbation, gain perturbation, SpecAugment, and their combinations. Under a fixed training-step budget, three-seed experiments show no consistent improvement from speed or gain perturbation, while SpecAugment increases mean development word error rate (WER) by approximately 0.99 percentage points relative to the baseline.

We fine-tune Whisper large-v3 on approximately 465.9 hours of Kikuyu speech using speaker-disjoint partitions. The original two-epoch model achieves a 26.78% development WER and a 25.01% corpus WER on a quality-screened test subset of 9,117 recordings. The same checkpoint yields 34.09% WER on the original unfiltered test set of 9,515 recordings. The screening excludes 398 recordings with atypical reference characters (å or ä), representing 4.18% of test examples, 16.52% of reference words and 38.75% of measured word errors.

These findings suggest that augmentation benefits are not automatic and that reference-transcription consistency can substantially influence reported ASR accuracy. The 25.01% headline figure describes a post-hoc screened subset.

Keywords: Kikuyu, Gĩkũyũ, automatic speech recognition, Whisper, low-resource languages, data augmentation, transcription quality, evaluation methodology.

1. Introduction

Automatic speech recognition has improved substantially through multilingual pretrained models, but many African languages remain underrepresented in available training corpora and evaluation benchmarks. Kikuyu, a Bantu language spoken primarily in Kenya, presents a useful case study of both the opportunities offered by transfer learning and the difficulties of evaluating speech recognition in settings with heterogeneous data quality.

Audio augmentation is frequently proposed as a method for improving generalization when labeled speech is scarce. However, performance gains depend on the model, the data distribution, and the training budget. Increasing variation in the input signal does not guarantee better recognition, particularly when a pretrained model already provides substantial acoustic robustness.

A second challenge arises from the reference transcripts used for evaluation. WER assumes that the reference is a reliable representation of the spoken words. Inconsistent orthography, unusual word segmentation, annotation artifacts, and mismatched speech–text pairs can increase measured error independently of recognition quality.

This study examines both issues through controlled Whisper-small augmentation experiments and a larger-scale Whisper large-v3 training and evaluation pipeline.

2. Dataset and Experimental Setup

2.1 Kikuyu speech corpus

The study uses speech–text records derived from the MCAA1-MSU/anv_data_ke dataset. After preparation, the training partition contains 180,086 recordings from 979 speakers, totaling approximately 465.9 hours.

The frozen development partition contains 10,330 recordings from 57 speakers, while the original test partition contains 9,515 recordings from 60 speakers. Speaker identities, example identifiers, and audio records were checked for overlap across partitions.

2.2 Controlled augmentation experiments

Whisper-small experiments use a 20,000-recording subset of the training corpus. The augmentation conditions include speed perturbation, gain perturbation, SpecAugment, pairwise combinations, and combined augmentation. Models are compared under a common 15,000-step training budget, with three independent random seeds for the baseline and individual augmentation conditions.

Evaluation uses corpus WER on the original development partition.

2.3 Whisper large-v3 fine-tuning

Whisper large-v3 is fine-tuned using the prepared training partition. The published model and held-out test predictions correspond to the selected checkpoint from the initial two-epoch run, chosen using evaluation loss on a fixed 512-example development subset. Saved predictions were reused for retrospective quality-screening analyses within their respective evaluation runs.

2.4 Reference-transcription quality audit

An audit of the frozen reference transcripts identifies recordings containing å or ä, characters that differ from the more common Kikuyu orthography observed in the training corpus.

Source-provenance checks establish that these characters are present in the original source transcripts rather than being introduced by the local preprocessing pipeline. Manual examination of a small collection of affected recordings raises additional concerns about transcript–audio correspondence and word segmentation.

A deterministic filtering analysis excludes all recordings whose normalized references contain these characters. The same rule is applied to development and test data. Original manifests and predictions remain preserved.

3. Results

3.1 Effect of augmentation

Augmentation	Mean development WER, 3 seeds
No augmentation	46.29%
Speed perturbation	46.42%
Gain perturbation	46.42%
SpecAugment	47.28%

Neither speed nor gain perturbation provides a consistent improvement over the unaugmented baseline. SpecAugment produces a mean increase of approximately 0.99 percentage points in WER. A three-seed paired analysis gives an exploratory Holm-adjusted p-value of approximately 0.047 for this comparison.

These findings apply to the tested Whisper-small training configuration and should not be generalized to other model sizes, augmentation intensities, or training schedules without further experiments.

3.2 Whisper large-v3 evaluation

The original two-epoch Whisper large-v3 checkpoint achieves 25.01% corpus WER on a retrospectively quality-screened, speaker-disjoint test subset of 9,117 recordings.

Evaluation	Recordings	Corpus WER
Two-epoch model: quality-screened test subset	9,117	25.01%
Two-epoch model: original unfiltered test	9,515	34.09%

Evaluation caveat: The original unfiltered held-out test yielded 34.09% WER. The rule excluding references with å or ä was selected after examining test errors. The resulting 25.01% WER is therefore a retrospective subset measurement, not an untouched independent test estimate. The character-based rule may exclude valid annotations and miss other transcription errors.

3.3 Concentration of errors in flagged references

The reference audit identifies 128 affected development recordings and 398 affected test recordings. The 398 test recordings contain 16.52% of all test reference words but contribute 38.75% of measured word errors.

The affected test subgroup has an original WER of approximately 79.98%, compared with 25.01% for recordings without the flagged characters. A hypothetical character-mapping analysis reduces test WER from 34.09% to 32.79%, suggesting that character differences alone do not account for the full discrepancy.

3.4 Evaluation-distribution sensitivity

An exploratory retrospective analysis standardizes the recorded development and test evaluation distributions to the training distribution across five metadata dimensions.

The recorded standardized development and test WER estimates are 28.57% and 28.41%, respectively, a numerical difference of −0.16 percentage points with a reported 95% speaker-bootstrap interval of [−5.00, 3.96] percentage points. This analysis illustrates sensitivity to evaluation composition; it does not establish a causal explanation for the difference. The reported interval also conditions on fitted weighting parameters.

4. Discussion

Two observations emerge.

First, straightforward audio augmentation does not necessarily improve a pretrained multilingual ASR model. Under the tested computational budget, the unaugmented Whisper-small baseline performs at least as well as speed and gain perturbation, while SpecAugment performs worse.

Second, evaluation-data quality materially affects reported model performance. The atypical transcription convention is rare in training but more prevalent in the original test set, where affected recordings contribute disproportionately to word errors.

These results reinforce earlier observations that transcription noise and dataset quality are central challenges in African-language ASR research.

5. Limitations

The principal limitation is that the quality-filtering criterion was developed retrospectively after inspecting test-set errors. Although the rule is deterministic and does not depend on model predictions during application, the decision to investigate it was influenced by observed errors.

Additionally, the presence of atypical characters is an imperfect proxy for transcription quality. Some excluded recordings may contain valid speech–text annotations, and some retained recordings may contain unrelated annotation problems. The manual audio review covers only a small sample.

Finally, the Whisper-small augmentation findings are based on a subset of the available training data and should not be interpreted as a direct controlled comparison with the larger Whisper large-v3 system.

6. Conclusion

This study presents controlled augmentation experiments and a data-quality-focused evaluation of Whisper-based Kikuyu speech recognition. Simple augmentations did not improve Whisper-small under the tested fixed-compute configuration, and SpecAugment consistently reduced accuracy across three seeds.

The original two-epoch Whisper large-v3 model achieves 25.01% WER on a quality-screened, speaker-disjoint test subset of 9,117 recordings.

References and further reading

1. Akera, B., Walukagga, P., Yiga, G., Quinn, J., & Mwebaze, E. (2026). How Much Speech Data is Necessary for ASR in African Languages? An Evaluation of Data Scaling in Kinyarwanda and Kikuyu. Proceedings of Machine Learning Research, 314, 41–49. https://proceedings.mlr.press/v314/akera26b.html

2. CLEAR Global (2026). Lessons from advancing African language AI and insights on data scaling for African Automatic Speech Recognition. https://clearglobal.org/lessons-from-advancing-african-language-ai-and-insights-on-data-scaling-for-african-automatic-speech-recognition/

3. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv:2212.04356. https://arxiv.org/abs/2212.04356

4. Microsoft Research (2026). Paza: Introducing automatic speech recognition benchmarks and models for low-resource languages. https://www.microsoft.com/en-us/research/blog/paza-introducing-automatic-speech-recognition-benchmarks-and-models-for-low-resource-languages/