genomicBERT : A Light-weight Foundation Model for Genome Analysis using Unigram Tokenization and Specialized DNA Vocabulary

作者：Tyrone Chen, Naima Vahab, Navya Tyagi, Eleanor Cummins, Anton Y. Peleg, Sonika Tyagi · 发表于：bioRxiv (Cold Spring Harbor Laboratory) · 年份：2023 · DOI：10.1101/2023.05.31.542682 · 被引用次数：8 · 研究领域：Genomics and Phylogenetic Studies、Machine Learning in Bioinformatics、RNA and protein synthesis mechanisms

Abstract The genome, which serves as the inherent language directing the blueprint of life, offers significant analysis prospects by combining Natural Language Processing (NLP) and machine learning (ML). Integrating biological sequences with other digital healthcare information has potential to transform data-driven diagnostics. Large language models (LLMs) can be harnessed to decode the genomic language. This endeavor encounters three critical challenges: First, long biomolecular sequences require segmentation into smaller subunits, which is non-trivial since many biological “words” remain unknown. Second, the analysis of extended DNA sequences using LLMs demands a compute-intensive infrastructure. Third, ensuring reproducibility and reusability of modeling workflows remains an unresolved issue. To tackle these challenges, we introduce an empirical DNA tokenisation approach and a versatile, semantic-aware, genome language model — genomicBERT . The model is species-agnostic and operates seamlessly at the DNA or RNA levels. By introducing a reduced and specialized DNA vocabulary, our approach minimizes computational overhead and optimizes performance. Our benchmarking demonstrates that the genomicBERT matches or surpasses the performance of contemporary tools on the same datasets under different experimental conditions. To encourage collaboration and ease of access, we introduce genomicBERT as an integral component of the openly accessible conda package, genomeNLP . Validated ...