A Zero-Shot Benchmark of Arabic Dialect Identification Models on Libyan Speech
DOI:
https://doi.org/10.51984/v08aa731Keywords:
Arabic dialect identification, Libyan Arabic, zero-shot evaluation, speech benchmark, low-resource languages, model generalisationAbstract
Spoken Arabic dialect identification (DID) has advanced quickly, yet Libyan Arabic remains almost invisible in public benchmarks: it is either absent from the label set or folded into a coarse “Maghrebi” class, so a model can appear to cover Libyan speech while never recognising it. No published study reports how existing spoken Arabic DID systems perform on Libyan speech. This study tests whether publicly available Arabic DID models recognise Libyan speech when they encounter it. We evaluate three off-the-shelf models spanning the three label granularities available in public checkpoints: a 5-class Massively Multilingual Speech (MMS) based regional identifier (badrex), an 18-class HuBERT classifier, and the 20-class ADI-20 system that won the spoken-DID subtask of the 2025 Nuanced Arabic Dialect Identification (NADI) evaluation, on 79,395 segments (135.4 hours, 664 speakers, 163 sources) of region-labelled Libyan Arabic, with no fine-tuning. Every evaluated speaker was reviewed by an annotator raised in Libya, and speakers judged to be speaking Modern Standard Arabic or a non-Libyan variety were removed before evaluation. Because every segment carries a single true label (Libyan), we report recall rather than accuracy and organise it hierarchically (exact-country, Maghrebi-region, and top-3). ADI-20 assigns the correct Libyan label to only 14.1% of segments (95% confidence interval 10.2–19.2%), routing 53.1% of Libyan speech to Tunisian and 18.1% to Modern Standard Arabic. The HuBERT model reaches 48.1% Libyan recall (43.9–52.7%) and 73.3% top-3 recall, exceeding ADI-20 by 34.0 points on exact Libyan recall (95% CI 31.2–37.2 points, McNemar p < 0.001). ADI-20 shows a significant regional collapse, dropping from 15.6% recall on the coastal regions to 6.3% on southern Fezzan, while HuBERT is stable across regions. Recall rises modestly with segment duration, and the differences we observe between male and female speakers are reported as descriptive, since they are confounded with source and speaker composition.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Sebha University Conference Proceedings

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.