Chemistry Commons^™

Inter-Rater Agreement For The Annotation Of Neurologic Signs And Symptoms In Electronic Health Records, Chelsea Oommen, Quentin Howlett-Prieto, Michael D. Carrithers, Daniel B. Hier

Chemistry Faculty Research & Creative Works

The extraction of patient signs and symptoms recorded as free text in electronic health records is critical for precision medicine. Once extracted, signs and symptoms can be made computable by mapping to signs and symptoms in an ontology. Extracting signs and symptoms from free text is tedious and time-consuming. Prior studies have suggested that inter-rater agreement for clinical concept extraction is low. We have examined inter-rater agreement for annotating neurologic concepts in clinical notes from electronic health records. After training on the annotation process, the annotation tool, and the supporting neuro-ontology, three raters annotated 15 clinical notes in three rounds. …