An audit of presentation-form corruption in 341 public Arabic datasets
A measured sweep of 341 public Arabic-language datasets on the Hugging Face Hub for text corrupted by the widespread arabic_reshaper.reshape() + python-bidi get_display() pre-shaping recipe, which stores Arabic as Unicode presentation forms in visual order rather than as logical-order text. Method. Datasets were read t...