Bornil: An Open-Source Sign Language Data Crowdsourcing Platform for AI Enabled Dialect-Agnostic Communication

Abstract

The absence of annotated sign language datasets has hindered the development of sign language recognition and translation technologies. In this paper, we introduce Bornil; a crowdsource-friendly, multilingual sign language data collection, annotation, and validation platform. Bornil allows users to record sign language gestures and lets annotators perform sentence and gloss-level annotation. It also allows validators to make sure of the quality of both the recorded videos and the annotations through manual validation to develop high-quality datasets for deep learning-based Automatic Sign Language Recognition. To demonstrate the system's efficacy; we collected the largest sign language dataset for Bangladeshi Sign Language dialect, perform deep learning based Sign Language Recognition modeling, and report the benchmark performance. The Bornil platform, BornilDB v1.0 Dataset, and the codebases are available.

Data is the bottleneck

Motivation

Sign language recognition and translation need large, carefully annotated video datasets. For most of the world's sign languages, including Bangladeshi Sign Language, such data barely exists. Collecting it is slow because recording, labelling and quality control usually happen in separate tools and by separate people. Bornil puts the whole loop in one open-source, crowdsourcing-friendly platform.

Platform

Record, annotate, validate

1. RecordContributors sign a sentence shown on screen. Each recording stores metadata such as lighting, camera distance, background and subject view.
2. AnnotateAnnotators add sentence-level transcriptions and gloss-level labels aligned to the video timeline, and can mark noise.
3. ValidateValidators review both the recording and its annotations before they enter the dataset, keeping quality high.

BornilDB v1.0

A test group of three sign-language-literate people with congenital hearing loss used the platform to build BornilDB v1.0 in under three months.

• 73 hours of video, with 21,104 scripted samples and 50 spontaneous recordings prompted by a topic word.
• 138,586 words in total and 25,572 unique Bengali words, averaging 7.7 words and 13.4 seconds per recording.
• Released with MediaPipe and OpenPose keypoints, multi-view recordings, transcriptions and gloss annotations.

Benchmark

BornilDB test set · BLEU

A deep-learning model trained on the collected data gives a first benchmark. As expected for a new, low-resource language, scores are highest on short sentences and fall as sentences get longer.

BornilDB later served as one of the three benchmarks for SignFormer-GCN.

Citation

@article{dhruvo2023bornil,
  title   = {Bornil: An Open-Source Sign Language Data Crowdsourcing Platform for
             {AI} Enabled Dialect-Agnostic Communication},
  author  = {Dhruvo, Shahriar Elahi and Rahman, Mohammad Akhlaqur and Mandal, Manash Kumar and
             Shihab, Md. Istiak Hossain and Ansary, A. A. Noman and Shithi, Kaneez Fatema and
             Khanom, Sanjida and Akter, Rabeya and Arib, Safaeid Hossain and Ansary, M. N. and
             Mehnaz, Sazia and Sultana, Rezwana and Rahman, Sejuti and Chowdhury, Sayma Sultana and
             Chowdhury, Sabbir Ahmed and Sadeque, Farig and Sushmit, Asif},
  journal = {arXiv preprint arXiv:2308.15402},
  year    = {2023}
}