SEMANTICALLY ENHANCED MULTIMODAL NEURAL ARCHITECTURE FOR UZBEK SIGN LANGUAGE TRANSLATION

作者：Juraev D. B., Ochilov M. M., Abdullaeva M. I. · 发表于：Advanced Computing An International Journal · 年份：2025 · DOI：10.5121/acij.2025.16602

This study proposes a semantically enriched multimodal neural architecture for automatic translation of Uzbek Sign Language (UZSL). In the approach, first, the text stream is brought to semantic consistency through n-gram tokenization, lemmatization, and morphological normalization, and a high-information context is prepared for BERT-MLM. Then, hand, face, and body landmarks are extracted from video frames using MediaPipe Holistic, speech transcriptions are generated using Speech-to-Text, and they are annotated with gloss dictionary-based labels in the text→gloss→landmark relationship. During the translation process, the text is fed to BERT-MLM, and semantic candidates are generated using a Top-k masking strategy; these candidates are checked against a standard gloss corpus in the constraint layer and filtered using semantic similarity calculations based on Word2Vec. Finally, a re-ranking mechanism based on SBERT evaluates the cosine similarity between the context and glosses and selects the most appropriate gloss. This integrated pipeline provides a near-real-time, consistent, and scalable translation workflow for UZSL by reducing the morphological complexity of the text, extracting stable features from multimodal signals, and selecting glosses based on semantic criteria with dictionary constraints.