Biomedical Entity Recognition Using a Small Language Model
DOI:
https://doi.org/10.32870/recibe.v14i3.448Keywords:
Small Language Model, Selective State Space Models, Biomedical Named Entity RecognitionAbstract
This research proposes developing a computationally efficient Small Language Model (SLM) for Biomedical Named Entity Recognition (NER). By fine-tuning a general-domain pretrained SLM (based on selective state space models, SSMs) on specialized biomedical texts, we aim to achieve performance comparable to larger models. Our approach addresses limitations of Transformer-based models like BioBERT (110M+ parameters) by combining linear-time sequence processing with biomedical domain adaptation techniques.References
Brown et al., (2020). Language Models are Few-Shot Learners (No. arXiv:2005.14165). arXiv. https://doi.org/10.48550/arXiv.2005.14165
Gu, A., & Dao, T. (2024). Mamba: Linear-Time Sequence Modeling with Selective State Spaces (No. arXiv:2312.00752). arXiv. https://doi.org/10.48550/arXiv.2312.00752
Huang et al., (2020). ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission (No. arXiv:1904.05342). arXiv. https://doi.org/10.48550/arXiv.1904.05342
Lee et al., (2020). BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4), 1234–1240. https://doi.org/10.1093/bioinformatics/btz682
Marengo et al., (2024). Benchmarking LLMs and SLMs for patient reported outcomes. https://doi.org/10.48550/ARXIV.2412.16291
Yoshimura et al., (2024). MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba (No. arXiv:2411.03855). arXiv. https://doi.org/10.48550/arXiv.2411.03855
