A Zero-Shot Benchmark of Arabic Dialect Identification Models on Libyan Speech

Authors

  • Nour Zohdy Faculty of Information Technology, Sebha University, Sebha
  • Mansour Essgaer Faculty of Information Technology, Sebha University, Sebha

DOI:

https://doi.org/10.51984/v08aa731

Keywords:

Arabic dialect identification, Libyan Arabic, zero-shot evaluation, speech benchmark, low-resource languages, model generalisation

Abstract

Spoken Arabic dialect identification (DID) has advanced quickly, yet Libyan Arabic remains almost invisible in public benchmarks: it is either absent from the label set or folded into a coarse “Maghrebi” class, so a model can appear to cover Libyan speech while never recognising it. No published study reports how existing spoken Arabic DID systems perform on Libyan speech. This study tests whether publicly available Arabic DID models recognise Libyan speech when they encounter it. We evaluate three off-the-shelf models spanning the three label granularities available in public checkpoints: a 5-class Massively Multilingual Speech (MMS) based regional identifier (badrex), an 18-class HuBERT classifier, and the 20-class ADI-20 system that won the spoken-DID subtask of the 2025 Nuanced Arabic Dialect Identification (NADI) evaluation, on 79,395 segments (135.4 hours, 664 speakers, 163 sources) of region-labelled Libyan Arabic, with no fine-tuning. Every evaluated speaker was reviewed by an annotator raised in Libya, and speakers judged to be speaking Modern Standard Arabic or a non-Libyan variety were removed before evaluation. Because every segment carries a single true label (Libyan), we report recall rather than accuracy and organise it hierarchically (exact-country, Maghrebi-region, and top-3). ADI-20 assigns the correct Libyan label to only 14.1% of segments (95% confidence interval 10.2–19.2%), routing 53.1% of Libyan speech to Tunisian and 18.1% to Modern Standard Arabic. The HuBERT model reaches 48.1% Libyan recall (43.9–52.7%) and 73.3% top-3 recall, exceeding ADI-20 by 34.0 points on exact Libyan recall (95% CI 31.2–37.2 points, McNemar p < 0.001). ADI-20 shows a significant regional collapse, dropping from 15.6% recall on the coastal regions to 6.3% on southern Fezzan, while HuBERT is stable across regions. Recall rises modestly with segment duration, and the differences we observe between male and female speakers are reported as descriptive, since they are confounded with source and speaker composition. 

Downloads

Published

2026-11-11

How to Cite

A Zero-Shot Benchmark of Arabic Dialect Identification Models on Libyan Speech. (2026). Sebha University Conference Proceedings, 5(4), 235-241. https://doi.org/10.51984/v08aa731

Similar Articles

1-10 of 194

You may also start an advanced similarity search for this article.