Large Scale Semi-Automated Labeling of Routine Free-Text Clinical Records for Deep Learning.
View the PDF.
Trivedi HM1, Panahiazar M2, Liang A3, Lituiev D2, Chang P4, Sohn JH4, Chen YY5, Franc BL4, Joe B4, Hadley D2.
- Department of Radiology and Biomedical Imaging, University of California, San Francisco, CA, USA. email@example.com.
- Institute for Computational Health Sciences, University of California, San Francisco, CA, USA.
- University of California School of Medicine, San Francisco, CA, USA.
- Department of Radiology and Biomedical Imaging, University of California, San Francisco, CA, USA.
- Department of Pathology, University of California, San Francisco, CA, USA.
Breast cancer is a leading cause of cancer death among women in the USA. Screening mammography is effective in reducing mortality, but has a high rate of unnecessary recalls and biopsies. While deep learning can be applied to mammography, large-scale labeled datasets, which are difficult to obtain, are required. We aim to remove many barriers of dataset development by automatically harvesting data from existing clinical records using a hybrid framework combining traditional NLP and IBM Watson. An expert reviewer manually annotated 3521 breast pathology reports with one of four outcomes: left positive, right positive, bilateral positive, negative. Traditional NLP techniques using seven different machine learning classifiers were compared to IBM Watson’s automated natural language classifier. Techniques were evaluated using precision, recall, and F-measure. Logistic regression outperformed all other traditional machine learning classifiers and was used for subsequent comparisons. Both traditional NLP and Watson’s NLC performed well for cases under 1024 characters with weighted average F-measures above 0.96 across all classes. Performance of traditional NLP was lower for cases over 1024 characters with an F-measure of 0.83. We demonstrate a hybrid framework using traditional NLP techniques combined with IBM Watson to annotate over 10,000 breast pathology reports for development of a large-scale database to be used for deep learning in mammography. Our work shows that traditional NLP and IBM Watson perform extremely well for cases under 1024 characters and can accelerate the rate of data annotation.