Clinical analogy resolution performance for foundation language models

Simple item page
dc.catalogadorvzp
dc.contributor.authorVillena, Fabián
dc.contributor.authorQuiroga Curin, Tamara Nancy
dc.contributor.authorDunstan Escudero, Jocelyn Mariel
dc.date.accessioned2025-03-21T16:39:56Z
dc.date.available2025-03-21T16:39:56Z
dc.date.issued2024
dc.description.abstractUsing extensive data sources to create foundation language models has revolutionized the performance of deep learning-based architectures. This remarkable improvement has led to state-of-the-art results for various downstream NLP tasks, including clinical tasks. However, more research is needed to measure model performance intrinsically, especially in the clinical domain. We revisit the use of analogy questions as an effective method to measure the intrinsic performance of language models for the clinical domain in English. We tested multiple Transformers-based language models over analogy questions constructed from the Unified Medical Language System (UMLS), a massive knowledge graph of clinical concepts. Our results show that large language models are significantly more performant for analogy resolution than small language models. Similarly, domain-specific language models perform better than general domain language models. We also found a correlation between intrinsic and extrinsic performance, validated through PubMedQA extrinsic task. Creating clinical-specific and language-specific language models is essential for advancing biomedical and clinical NLP and will ensure a valid application in clinical practice. Finally, given that our proposed intrinsic test is based on a term graph available in multiple languages, the dataset can be built to measure the performance of models in languages other than English.
dc.format.extent13 páginas
dc.fuente.origenORCID
dc.identifier.doi10.1145/3709155
dc.identifier.eissn2637-8051
dc.identifier.urihttps://doi.org/10.1145/3709155
dc.identifier.urihttps://repositorio.uc.cl/handle/11534/102927
dc.information.autorucEscuela de Ingeniería; Quiroga Curin, Tamara Nancy; S/I; 1207385
dc.information.autorucEscuela de Ingeniería; Dunstan Escudero, Jocelyn Mariel; S/I; 1285723
dc.language.isoen
dc.nota.accesocontenido parcial
dc.revistaACM Transactions on Computing for Healthcare
dc.rightsacceso restringido
dc.subjectInformation systems
dc.subjectLanguage models
dc.subjectApplied computing
dc.subjectHealth informatics
dc.subjectComputing methodologies
dc.subjectNatural language processing
dc.subject.ddc610
dc.subject.deweyMedicina y saludes_ES
dc.subject.ods03 Good health and well-being
dc.subject.odspa03 Salud y bienestar
dc.titleClinical analogy resolution performance for foundation language models
dc.typeartículo
sipa.codpersvinculados1207385
sipa.codpersvinculados1285723
sipa.trazabilidadORCID;2025-03-03
Files
Collections
Artículos de revistas