Data Science Commons^™

Validation Of A Risk Score For Cancer-Associated Thrombosis Using Nationwide Ehr Data, Ang Li, Omid Jafari, Barbara D Lam, Jun Y Jiang, Rock Bum Kim, Shengling Ma, Emily Zhou, Joyce W Tiong, Elizabeth C Chiang, Justine Ryu, Christopher I Amos, Jennifer La, Nathanael R Fillmore

Faculty, Staff and Student Publications

Importance: Venous thromboembolism (VTE) is associated with increased mortality and morbidity in patients with cancer. Existing risk prediction models are typically validated within individual sites, a fragmented approach that limits clinical adoption.

Objective: To validate the electronic health record cancer-associated thrombosis (EHR-CAT) score compared with the benchmark Khorana score in a contemporary cohort of patients with cancer across the nation, before and after treatment, excluding those at high risk of bleeding.

Design, setting, and participants: This prognostic study included patients in a nationwide longitudinal EHR database from January 2018 to December 2023 with follow-up continuing to April 2025. Patients with …

Computerized Diagnostic Decision Support Systems-Isabel Pro Versus Chatgpt-4 Part Ii, Joe M Bridges, Xiaoqian Jiang, Michael Ige, Oluwatoniloba Toyobo

Faculty, Staff and Student Publications

Objective: Does a Tree-of-Thought prompt and reconsideration of Isabel Pro's differential improve ChatGPT-4's accuracy; does increasing expert panel size improve ChatGPT-4's accuracy; does ChatGPT-4 produce consistent outputs in sequential requests; what is the frequency of fabricated references?

Materials and methods: Isabel Pro, a computerized diagnostic decision support system, and ChatGPT-4, a large language model. Using 201 cases from the New England Journal of Medicine, each system produced a differential diagnosis ranked by likelihood. Statistics were Mean Reciprocal Rank, Recall at Rank, Average Rank, Number of Correct Diagnoses, and Rank Improvement. For reproducibility, the study compared the initial expert panel run …

Histone Methyltransferase Ash1l Primes Metastases And Metabolic Reprogramming Of Macrophages In The Bone Niche, Chenling Meng, Kevin Lin, Wei Shi, Hongqi Teng, Xinhai Wan, Anna Debruine, Yin Wang, Xin Liang, Javier Leo, Feiyu Chen, Qianlin Gu, Jie Zhang, Vivien Van, Kiersten L Maldonado, Boyi Gan, Li Ma, Yue Lu, Di Zhao

Faculty, Staff and Student Publications

Bone metastasis is a major cause of cancer death; however, the epigenetic determinants driving this process remain elusive. Here, we report that histone methyltransferase ASH1L is genetically amplified and is required for bone metastasis in men with prostate cancer. ASH1L rewires histone methylations and cooperates with HIF-1α to induce pro-metastatic transcriptome in invading cancer cells, resulting in monocyte differentiation into lipid-associated macrophage (LA-TAM) and enhancing their pro-tumoral phenotype in the metastatic bone niche. We identified IGF-2 as a direct target of ASH1L/HIF-1α and mediates LA-TAMs' differentiation and phenotypic changes by reprogramming oxidative phosphorylation. Pharmacologic inhibition of the ASH1L-HIF-1α-macrophages axis elicits …

Incorporating Latent Survival Trajectories And Covariate Heterogeneity In Time-To-Event Data Analysis: A Joint Mixture Model Approach, Fu-Wen Liang, Wenyaw Chan, Michael D Swartz, Bouthaina S Dabaja

Faculty, Staff and Student Publications

Background: Finite mixture models have been recently applied in time-to-event data to identify subgroups with distinct hazard functions, yet they often assume differing covariate effects on failure times across latent classes but homogeneous covariate distributions. This study aimed to develop a method for analyzing time-to-event data while accounting for unobserved heterogeneity within a mixture modeling framework.

Methods: A joint model was developed to incorporate latent survival trajectories and observed information for the joint analysis of time-to-event outcomes, correlated discrete and continuous covariates, and a latent class variable. It assumed covariate effects on survival times and covariate distributions vary across latent …

Student Expectations And Outcomes In Virtual Vs In-Person Interprofessional Simulations: A Qualitative Analysis, Padmavathy Ramaswamy, Abbey M Bachmann, Tiffany Champagne-Langabeer, Chasisty L Gilder, Samuel E Neher, Jennifer L Swails

Faculty, Staff and Student Publications

Background: Health-related programs frequently integrate interprofessional education (IPE) into their training. The COVID-19 pandemic transitioned many IPE programs online, making it essential to assess student expectations and perceived learning outcomes across virtual simulations and in-person settings.

Methods: This qualitative study compared student expectations and self-reported outcomes across in-person and virtual case scenarios at a Texas health science center. Responses to open-ended questions from two data collection periods were analyzed using inductive coding and thematic analysis.

Results: Students from nursing, medicine, dentistry, public health, and informatics participated in each group. Three major themes emerged from this study: communication, teamwork, and …

Precision Phenotyping For Curating Research Cohorts Of Patients With Unexplained Post-Acute Sequelae Of Covid-19, Alaleh Azhir, Jonas Hügel, Jiazi Tian, Jingya Cheng, Ingrid V Bassett, Douglas S Bell, Elmer V Bernstam, Maha R Farhat, Darren W Henderson, Emily S Lau, Michele Morris, Yevgeniy R Semenov, Virginia A Triant, Shyam Visweswaran, Zachary H Strasser, Jeffrey G Klann, Shawn N Murphy, Hossein Estiri

Faculty, Staff and Student Publications

BACKGROUND: Scalable identification of patients with post-acute sequelae of COVID-19 (PASC) is challenging due to a lack of reproducible precision phenotyping algorithms, which has led to suboptimal accuracy, demographic biases, and underestimation of the PASC.

METHODS: In a retrospective case-control study, we developed a precision phenotyping algorithm for identifying cohorts of patients with PASC. We used longitudinal electronic health records data from over 295,000 patients from 14 hospitals and 20 community health centers in Massachusetts. The algorithm employs an attention mechanism to simultaneously exclude sequelae that prior conditions can explain and include infection-associated chronic conditions. We performed independent chart reviews …

Evaluating The Meditation Practices And Barriers To Adopting Mindful Medicine Among Physicians, Tiffany Champagne-Langabeer, Chelsea G Ratcliff, Christine Bakos-Block, Francine Vega, Marylou Cardenas-Turanzas, Aila Malik, Radha Korupolu

Faculty, Staff and Student Publications

Background: Chronic pain affects over 25% of U.S. adults and is a leading cause of disability. Mindfulness meditation (MM) is a nonpharmacologic approach to manage pain and improve well-being. Despite mounting evidence supporting its efficacy, MM remains underutilized in medical practice. Understanding physicians' engagement with MM and the barriers they face can inform strategies for integration into clinical care. This study assessed physicians' attitudes toward MM, including barriers to practice and their likelihood of recommending it to patients.

Methods: A cross-sectional survey of U.S. physicians was conducted from April to July 2024. Participants provided information on demographics, health struggles, and …

Leveraging Large Language Models For Knowledge-Free Weak Supervision In Clinical Natural Language Processing, Enshuo Hsu, Kirk Roberts

Faculty, Staff and Student Publications

The performance of deep learning-based natural language processing systems is based on large amounts of labeled training data which, in the clinical domain, are not easily available or affordable. Weak supervision and in-context learning offer partial solutions to this issue, particularly using large language models (LLMs), but their performance still trails traditional supervised methods with moderate amounts of gold-standard data. In particular, inferencing with LLMs is computationally heavy. We propose an approach leveraging fine-tuning LLMs and weak supervision with virtually no domain knowledge that still achieves consistently dominant performance. Using a prompt-based approach, the LLM is used to generate weakly-labeled …

Proxy Panels Enable Privacy-Aware Outsourcing Of Genotype Imputation, Degui Zhi, Xiaoqian Jiang, Arif Harmanci

Faculty, Staff and Student Publications

One of the major challenges in genomic data sharing is protecting participants' privacy in collaborative studies and in cases when genomic data are outsourced to perform analysis tasks, for example, genotype imputation services and federated collaborations genomic analysis. Although numerous cryptographic methods have been developed, these methods may not yet be practical for population-scale tasks in terms of computational requirements, rely on high-level expertise in security, and require each algorithm to be implemented from scratch. In this study, we focus on outsourcing of genotype imputation, a fundamental task that utilizes population-level reference panels, and develop protocols that rely on using …

Classification Of Schizophrenia, Bipolar Disorder And Major Depressive Disorder With Comorbid Traits And Deep Learning Algorithms, Xiangning Chen, Yimei Lu, Joan Manuel Cue, Mira V Han, Vishwajit L Nimgaonkar, Daniel R Weinberger, Shizhong Han, Zhongming Zhao, Jingchun Chen

Faculty, Staff and Student Publications

Many psychiatric disorders share genetic liabilities, but whether these shared liabilities can be utilized to classify and differentiate psychiatric disorders remains unclear. In this study, we use polygenic risk scores (PRSs) of 42 traits comorbid with schizophrenia (SCZ), bipolar disorder (BIP), and major depressive disorder (MDD) to evaluate their utilities. We found that combining target specific PRS with PRSs of comorbid traits can improve the classification of the target disorders. Importantly, without inclusion of PRSs from targeted disorders, we can still classify SCZ (accuracy 0.710 ± 0.008, AUC 0.789 ± 0.011), BIP (accuracy 0.782 ± 0.006, AUC 0.852 ± 0.004), …

Scproatlas: An Atlas Of Multiplexed Single-Cell Spatial Proteomics Imaging In Human Tissues, Tiangang Wang, Xuanmin Chen, Yujuan Han, Jiahao Yi, Xi Liu, Pora Kim, Liyu Huang, Kexin Huang, Xiaobo Zhou

Faculty, Staff and Student Publications

Spatial proteomics can visualize and quantify protein expression profiles within tissues at single-cell resolution. Although spatial proteomics can only detect a limited number of proteins compared to spatial transcriptomics, it provides comprehensive spatial information with single-cell resolution. By studying the spatial distribution of cells, we can clearly obtain the spatial context within tissues at multiple scales. Spatial context includes the spatial composition of cell types, the distribution of functional structures, and the spatial communication between functional regions, all of which are crucial for the patterns of cellular distribution. Here, we constructed a comprehensive spatial proteomics functional annotation knowledgebase, scProAtlas (https://relab.xidian.edu.cn/scProAtlas/#/), …

Aspdb: An Integrative Knowledgebase Of Human Protein Isoforms From Experimental And Ai-Predicted Structures, Yuntao Yang, Himansu Kumar, Yuhan Xie, Zhao Li, Rongbin Li, Wenbo Chen, Chiamaka S Diala, Meer A Ali, Yi Xu, Albon Wu, Sayed-Rzgar Hosseini, Erfei Bi, Hongyu Zhao, Pora Kim, W Jim Zheng

Faculty, Staff and Student Publications

Alternative splicing is a crucial cellular process in eukaryotes, enabling the generation of multiple protein isoforms with diverse functions from a single gene. To better understand the impact of alternative splicing on protein structures, protein-protein interaction and human diseases, we developed ASpdb (https://biodataai.uth.edu/ASpdb/), a comprehensive database integrating experimentally determined structures and AlphaFold 2-predicted models for human protein isoforms. ASpdb includes over 3400 canonical isoforms, each represented by both experimentally resolved and predicted structures, and >7200 alternative isoforms with AlphaFold 2 predictions. In addition to detailed splicing events, 3D structures, sequence variations and functional annotations, ASpdb uniquely offers comparative analyses and …

Metsdb: A Knowledgebase Of Cancer Metastasis At Bulk, Single-Cell And Spatial Levels, Sijia Wu, Jiajin Zhang, Yanfei Wang, Xinyu Qin, Zhaocan Zhang, Zhennan Lu, Pora Kim, Xiaobo Zhou, Liyu Huang

Faculty, Staff and Student Publications

Cancer metastasis, the process by which tumour cells migrate and colonize distant organs from a primary site, is responsible for the majority of cancer-related deaths. Understanding the cellular and molecular mechanisms underlying this complex process is essential for developing effective metastasis prevention and therapy strategies. To this end, we systematically analysed 1786 bulk tissue samples from 13 cancer types, 988 463 single cells from 17 cancer types, and 40 252 spots from 45 spatial slides across 10 cancer types. The results of these analyses are compiled in the metsDB database, accessible at https://relab.xidian.edu.cn/metsDB/. This database provides insights into alterations in …

Crisprofft: Comprehensive Database Of Crispr/Cas Off-Targets, Grant Wang, Xiaona Liu, Aoqi Wang, Jianguo Wen, Pora Kim, Qianqian Song, Xiaona Liu, Xiaobo Zhou

Faculty, Staff and Student Publications

The CRISPR (clustered regularly interspaced short palindromic repeats)/Cas (CRISPR-associated protein) programmable nuclease system continues to evolve, with in vivo therapeutic gene editing increasingly applied in clinical settings. However, off-target effects remain a significant challenge, hindering its broader clinical application. To enhance the development of gene-editing therapies and the accuracy of prediction algorithms, we developed CRISPRoffT (https://ccsm.uth.edu/CRISPRoffT/). Users can access a comprehensive repository of off-target regions predicted and validated by a diverse range of technologies across various cell lines, Cas enzyme variants, engineered sgRNAs (single guide RNAs) and CRISPR editing systems. CRISPRoffT integrates results of off-target analysis from 74 studies, encompassing …

A Bayesian Deep Segmentation Framework For Glioblastoma Tumor Segmentation Using Follow-Up Mris, Tanjida Kabir, Kang-Lin Hsieh, Luis Nunez, Yu-Chun Hsu, Juan C Rodriguez Quintero, Octavio Arevalo, Kangyi Zhao, Jay-Jiguang Zhu, Roy F Riascos, Mahboubeh Madadi, Xiaoqian Jiang, Shayan Shams

Faculty, Staff and Student Publications

Background: Glioblastoma (GBM) is the most common malignant brain tumor with an abysmal prognosis. Since complete tumor cell removal is impossible due to the infiltrative nature of GBM, accurate measurement is paramount for GBM assessment. Preoperative magnetic resonance images (MRIs) are crucial for initial diagnosis and surgical planning, while follow-up MRIs are vital for evaluating treatment response. The structural changes in the brain caused by surgical and therapeutic measures create significant differences between preoperative and follow-up MRIs. In clinical research, advanced deep learning models trained on preoperative MRIs are often applied to assess follow-up scans, but their effectiveness in this …

Leveraging Gpt-4o For Automated Extraction Of Neural Projections From Scientific Literature, Rashmie Abeysinghe, Gorbachev Jowah, Licong Cui, Samden D Lhatoo, Guo-Qiang Zhang

Faculty, Staff and Student Publications

Sudden Unexpected Death in Epilepsy (SUDEP) is a major cause of death for epilepsy patients having uncontrolled seizures. Understanding the complex neural circuits within the central nervous system is crucial for understanding the mechanisms underlying cardiorespiratory regulation, particularly in the context of SUDEP. This study explores the potential of GPT-4o, a cutting-edge language model, to automate the extraction of neural projections from scientific literature. We developed prompts to extract neuroscientific structures, extract projections, and perform synonym harmonization. Applying the approach to four neuroscientific articles, the method extracted 205 projections. A random sample of 100 projections identified was handed over to …

Depletion Of Adipose Stroma-Like Cancer-Associated Fibroblasts Potentiates Pancreatic Cancer Immunotherapy, Joseph Rupert, Alexes Daquinag, Yongmei Yu, Yulin Dai, Zhongming Zhao, Mikhail G Kolonin

Faculty, Staff and Student Publications

This study shows that populations of CAFs have distinct effects on pancreatic cancer progression and shows that depletion of CAFs expressing adipose markers potentiates tumor/metastasis suppression effects of immune checkpoint blockade.

Gene Expression Changes In Human Cerebral Arteries Following Hemoglobin Exposure: Implications For Vascular Responses In Sah, Chathathayil M Shafeeque, Arif O Harmanci, Sithara Thomas, Ari C Dienel, Devin W Mcbride, Kumar T Peeyush, Spiros L Blackburn

Faculty, Staff and Student Publications

Subarachnoid hemorrhage (SAH), characterized by the presence of hemoglobin (Hb) in the subarachnoid space, significantly impacts cerebral vessels, leading to various pathological outcomes. The toxicity of cell-free Hb released from erythrocytes and its metabolites after SAH causes vasoconstriction and neuronal damage, and correlates with delayed ischemic neurological deficits (DIND). While animal models have provided substantial and invaluable data in the research of aneurysmal SAH, the specific effects of subarachnoid blood on cerebral arteries remain greatly understudied. Here, we describe the changes in the genetic profile of human cerebral arteries exposed to free Hb for 48 h. We performed an ex …

An Ontology-Based Approach For Understanding Appendicectomy Processes And Associated Resources, Nadeesha Pathiraja Rathnayaka Hitige, Ting Song, Steven J Craig, Kimberley J Davis, Xubing Hao, Licong Cui, Ping Yu

Faculty, Staff and Student Publications

Background: Traditional methods for analysing surgical processes often fall short in capturing the intricate interconnectedness between clinical procedures, their execution sequences, and associated resources such as hospital infrastructure, staff, and protocols.

Aim: This study addresses this gap by developing an ontology for appendicectomy, a computational model that comprehensively represents appendicectomy processes and their resource dependencies to support informed decision making and optimise appendicectomy healthcare delivery.

Methods: The ontology was developed using the NeON methodology, drawing knowledge from existing ontologies, scholarly literature, and de-identified patient data from local hospitals.

Results: The resulting ontology comprises 108 classes, including 11 top-level classes and …

Key Epigenetic And Signaling Factors In The Formation And Maintenance Of The Blood-Brain Barrier, Jayanarayanan Sadanandan, Sithara Thomas, Iny Elizabeth Mathew, Zhen Huang, Spiros L Blackburn, Nitin Tandon, Hrishikesh Lokhande, Pierre D Mccrea, Emery H Bresnick, Pramod K Dash, Devin W Mcbride, Arif Harmanci, Lalit K Ahirwar, Dania Jose, Ari C Dienel, Hussein A Zeineddine, Sungha Hong, Peeyush Kumar T

Faculty, Staff and Student Publications

The blood-brain barrier (BBB) controls the movement of molecules into and out of the central nervous system (CNS). Since a functional BBB forms by mouse embryonic day E15.5, we reasoned that gene cohorts expressed in CNS endothelial cells (EC) at E13.5 contribute to BBB formation. In contrast, adult gene signatures reflect BBB maintenance mechanisms. Supporting this hypothesis, transcriptomic analysis revealed distinct cohorts of EC genes involved in BBB formation and maintenance. Here, we demonstrate that epigenetic regulator's histone deacetylase 2 (HDAC2) and polycomb repressive complex 2 (PRC2) control EC gene expression for BBB development and prevent Wnt/β-catenin (Wnt) target genes …

Tumor Expression Of Cd83 Reduces Glioma Progression And Is Associated With Reduced Immunosuppression, Malcolm F Mcdonald, Rachel Naomi Curry, Isabella O'Reilly, Brittney Lozzi, Alexis Cervantes, Zhung-Fu Lee, Anna Rosenbaum, Peihao He, Carrie Mohila, Arif O Harmanci, Akdes Serin Harmanci, Benjamin Deneen, Ganesh Rao

Faculty, Staff and Student Publications

Immunosuppression in malignant glioma remains a barrier to therapeutic development. CD83 overexpression in human and mouse glioma increases survival. CD83+ tumor cells promote signatures related to cytotoxic T cells, enhanced activation of CD8+ T cells, and increased proinflammatory cytokines. These findings suggest that tumor-expressed CD83 could mediate tumor-immune communications.

Chatgpt Vs Expert-Guided Care Pathways For Postesophagectomy Symptom Management, Mohamad K Abou Chaar, Giovanna Grigsby-Rocca, Ming Huang, Shanda H Blackmon

Faculty, Staff and Student Publications

BACKGROUND: The objective of this study was to compare generative artificial intelligence-initiated care pathways, using ChatGPT, with expert-guided consensus-initiated care pathways from AskMayoExpert (AME) for symptom management of esophageal cancer patients after esophagectomy.

METHODS: A formal protocol for development of 9 AME care pathways was followed for specific patient-identified domains after esophagectomy for esophageal cancer. Domain scores were measured and assessed through the Upper Digestive Disease tool. These care pathways were developed by experts validated by a consensus-driven methodology. ChatGPT was used to answer specific questions similar to the AME care pathway on April 9, 2023, and March 28, 2024. …

Addressing Ethical Issues In Healthcare Artificial Intelligence Using A Lifecycle-Informed Process, Benjamin X Collins, Jean-Christophe Bélisle-Pipon, Barbara J Evans, Kadija Ferryman, Xiaoqian Jiang, Camille Nebeker, Laurie Novak, Kirk Roberts, Martin Were, Zhijun Yin, Vardit Ravitsky, Joseph Coco, Rachele Hendricks-Sturrup, Ishan Williams, Ellen W Clayton, Bradley A Malin, Bridge2ai Ethics And Trustworthy Ai Working Group

Faculty, Staff and Student Publications

OBJECTIVES: Artificial intelligence (AI) proceeds through an iterative and evaluative process of development, use, and refinement which may be characterized as a lifecycle. Within this context, stakeholders can vary in their interests and perceptions of the ethical issues associated with this rapidly evolving technology in ways that can fail to identify and avert adverse outcomes. Identifying issues throughout the AI lifecycle in a systematic manner can facilitate better-informed ethical deliberation.

MATERIALS AND METHODS: We analyzed existing lifecycles from within the current literature for ethical issues of AI in healthcare to identify themes, which we relied upon to create a lifecycle …

De-Identification Is Not Enough: A Comparison Between De-Identified And Synthetic Clinical Notes, Atiquer Rahman Sarkar, Yao-Shun Chuang, Noman Mohammed, Xiaoqian Jiang

Faculty, Staff and Student Publications

For sharing privacy-sensitive data, de-identification is commonly regarded as adequate for safeguarding privacy. Synthetic data is also being considered as a privacy-preserving alternative. Recent successes with numerical and tabular data generative models and the breakthroughs in large generative language models raise the question of whether synthetically generated clinical notes could be a viable alternative to real notes for research purposes. In this work, we demonstrated that (i) de-identification of real clinical notes does not protect records against a membership inference attack, (ii) proposed a novel approach to generate synthetic clinical notes using the current state-of-the-art large language models, (iii) evaluated …

Question Answering For Electronic Health Records: Scoping Review Of Datasets And Models, Jayetri Bardhan, Kirk Roberts, Daisy Zhe Wang

Faculty, Staff and Student Publications

Background: Question answering (QA) systems for patient-related data can assist both clinicians and patients. They can, for example, assist clinicians in decision-making and enable patients to have a better understanding of their medical history. Substantial amounts of patient data are stored in electronic health records (EHRs), making EHR QA an important research area. Because of the differences in data format and modality, this differs greatly from other medical QA tasks that use medical websites or scientific papers to retrieve answers, making it critical to research EHR QA.

Objective: This study aims to provide a methodological review of existing works on …

Characterizing The Progression From Mild Cognitive Impairment To Dementia: A Network Analysis Of Longitudinal Clinical Visits, Muskan Garg, Sara Hejazi, Sunyang Fu, Maria Vassilaki, Ronald C Petersen, Jennifer St Sauver, Sunghwan Sohn

Faculty, Staff and Student Publications

Background: With the recent surge in the utilization of electronic health records for cognitive decline, the research community has turned its attention to conducting fine-grained analyses of dementia onset using advanced techniques. Previous works have mostly focused on machine learning-based prediction of dementia, lacking the analysis of dementia progression and its associations with risk factors over time. The black box nature of machine learning models has also raised concerns regarding their uncertainty and safety in decision making, particularly in sensitive domains like healthcare.

Objective: We aimed to characterize the progression of health conditions, such as chronic diseases and neuropsychiatric symptoms, …

Meta-Analysis Of Censored Adverse Events, Xinyue Qi, Shouhao Zhou, Christine B Peterson, Yucai Wang, Xinying Fang, Michael L Wang, Chan Shen

Faculty, Staff and Student Publications

Meta-analysis is a powerful tool for assessing drug safety by combining treatment-related toxicological findings across multiple studies, as clinical trials are typically underpowered for detecting adverse drug effects. However, incomplete reporting of adverse events (AEs) in published clinical studies is frequently encountered, especially if the observed number of AEs is below a pre-specified study-dependent threshold. Ignoring the censored AE information, often found in lower frequency, can significantly bias the estimated incidence rate of AEs. Despite its importance, this prevalent issue in meta-analysis has received little statistical or analytic attention in the literature. To address this challenge, we propose a Bayesian …

A Framework For Human Evaluation Of Large Language Models In Healthcare Derived From Literature Review, Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R Mccarthy, Hunter Osterhoudt, Xizhi Wu, Shyam Visweswaran, Sunyang Fu, Piyush Mathur, Giovanni E Cacciamani, Cong Sun, Yifan Peng, Yanshan Wang

Faculty, Staff and Student Publications

With generative artificial intelligence (GenAI), particularly large language models (LLMs), continuing to make inroads in healthcare, assessing LLMs with human evaluations is essential to assuring safety and effectiveness. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare across various medical specialties and addresses factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Our literature review of 142 studies shows gaps in reliability, generalizability, and applicability of current human evaluation practices. To overcome such significant obstacles to healthcare LLM developments and deployments, we …

A Case Demonstration Of The Open Health Natural Language Processing Toolkit From The National Covid-19 Cohort Collaborative And The Researching Covid To Enhance Recovery Programs For A Natural Language Processing System For Covid-19 Or Postacute Sequelae Of Sars Cov-2 Infection: Algorithm Development And Validation, Andrew Wen, Liwei Wang, Huan He, Sunyang Fu, Sijia Liu, David A Hanauer, Daniel R Harris, Ramakanth Kavuluru, Rui Zhang, Karthik Natarajan, Nishanth P Pavinkurve, Janos Hajagos, Sritha Rajupet, Veena Lingam, Mary Saltz, Corey Elowsky, Richard A Moffitt, Farrukh M Koraishy, Matvey B Palchuk, Jordan Donovan, Lora Lingrey, Garo Stone-Derhagopian, Robert T Miller, Andrew E Williams, Peter J Leese, Paul I Kovach, Emily R Pfaff, Mikhail Zemmel, Robert D Pates, Nick Guthe, Melissa A Haendel, Christopher G Chute, Hongfang Liu, National Covid Cohort Collaborative, Recover Initiative

Faculty, Staff and Student Publications

BACKGROUND: A wealth of clinically relevant information is only obtainable within unstructured clinical narratives, leading to great interest in clinical natural language processing (NLP). While a multitude of approaches to NLP exist, current algorithm development approaches have limitations that can slow the development process. These limitations are exacerbated when the task is emergent, as is the case currently for NLP extraction of signs and symptoms of COVID-19 and postacute sequelae of SARS-CoV-2 infection (PASC).

OBJECTIVE: This study aims to highlight the current limitations of existing NLP algorithm development approaches that are exacerbated by NLP tasks surrounding emergent clinical concepts and …

Improving Large Language Models For Clinical Named Entity Recognition Via Prompt Engineering, Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, Kirk Roberts, Hua Xu

Faculty, Staff and Student Publications

IMPORTANCE: The study highlights the potential of large language models, specifically GPT-3.5 and GPT-4, in processing complex clinical data and extracting meaningful information with minimal training data. By developing and refining prompt-based strategies, we can significantly enhance the models' performance, making them viable tools for clinical NER tasks and possibly reducing the reliance on extensive annotated datasets.

OBJECTIVES: This study quantifies the capabilities of GPT-3.5 and GPT-4 for clinical named entity recognition (NER) tasks and proposes task-specific prompts to improve their performance.

MATERIALS AND METHODS: We evaluated these models on 2 clinical NER tasks: (1) to extract medical problems, treatments, …