Transcription

Feasibility of the Partial Automation of DataAbstraction for the Quality and SafetyReview SystemPrepared for:Agency for Healthcare Research and Quality (AHRQ)U.S. Department of Health and Human Services5600 Fishers LaneRockville, MD 20857www.ahrq.govContract No. 290-2014-00008-IPrepared by:Clinovations Government HealthWashington, DCContributing Team:Clinovations GovHealthNicole Kemper, M.S.Laura McQueen, R.N., M.S.Fatema Rafiqi, Ph.D.Anita SamarthManan Trivedi, M.D., M.P.P.MedStarKatie AdamsAllan Fong, M.S.Daniel HoffmanRaj Ratwani, Ph.D.tie AdamsAllan Fong, MSAHRQ Publication No. 18-0034-EFAugust 2018

This project was funded under contract number 290-2014-00008-I from the Agency for Healthcare Researchand Quality (AHRQ), U.S. Department of Health and Human Services.The opinions expressed in this document are those of the authors and do not reflect the official position ofAHRQ or the U.S. Department of Health and Human Services. None of the investigators have any affiliationsor financial involvement that conflicts with the material presented in this report.This document may be used and reprinted without permission except those copyrighted materials that areclearly noted in the document. Further reproduction of those copyrighted materials is prohibited withoutthe express written permission of copyright holders.ii

Contents1 Executive Summary . 11.11.21.3Background .1Analysis .2Recommendations .42 Background and Approach . 62.12.2Background/Need .6Method/Approach .72.2.1 QSRS Analysis (Task 1) . 72.2.2 EHR Physician User Feedback (Task 1) . 82.2.3 Environmental Scan (Task 2) . 82.2.4 Stakeholder Interviews (Task 3). 82.2.5 Automation Feasibility Analysis (Task 4) . 83 QSRS Analysis and EHR Chart Review (Task 1) .103.13.23.3QSRS Analysis.103.1.1 Overview . 103.1.2 Findings . 10EHR Chart Review.113.2.1 Approach. 113.2.2 Findings . 11Certified EHR Technology Review.113.3.1 Approach. 113.3.2 Findings . 124 Environmental Scan (Task 2).134.14.24.34.44.54.6Methods .13Coding Framework .14Grey Literature Search .15Peer-Reviewed Literature Search .154.4.1 Overview . 154.4.2 Literature Search Results in the Context of QSRS Queries . 16“Grey” Literature Search.184.5.1 Overview . 184.5.2 Quality Measure Reporting. 194.5.3 Public Health Reporting and Surveillance . 194.5.4 NLP Technology. 20Discussion and Recommendations .214.6.1 Limitations With Current Studies. 214.6.2 Feasibility of Automating QSRS . 214.6.3 Applying Commercial Solutions . 214.6.4 Implications for Feasibility Study . 214.6.5 Additional Reference Information . 225 Stakeholder Findings (Task 3) .235.1Stakeholder Selection, Outreach, and Discussions .235.1.1 Providers/Health Systems. 24iii

5.25.1.2 EHR Vendors . 255.1.3 NLP/Analytics Vendors. 255.1.4 Process, Outreach & Discussions . 25Results Overview .265.2.1 Providers/Health Systems. 265.2.2 EHR Vendors . 295.2.3 NLP/Analytics Vendors. 306 Automation Feasibility Analysis (Task 4) .316.16.2Approach Detail .316.1.1 QSRS Algorithm/Question-by-Question Analysis . 316.1.2 Vendor Selection for Analysis . 35Results Overview .366.2.1 Clinical Analysis . 366.2.2 NLP Analysis . 396.2.3 EHR Vendor Data Availability . 466.2.4 Vendor Analytics Capabilities . 507 Recommendations .517.17.27.37.4AHRQ Strategies and Considerations.517.1.1 eMeasures Consideration for QSRS Reporting by Hospitals . 527.1.2 Prioritize Automation of QSRS Entry Questions . 537.1.3 Identify Lowest Difficulty/Highest Value QSRS Modules . 56Drive Real-World Implementation and Adoption.587.2.1 Leverage Commercial Technologies and Platforms . 587.2.2 Consider Human-Computer Automation Levels . 587.2.3 Sampling. 59Pilots and Projects .617.3.1 Proof-of-Concept Project: Test Rule-Based and NLP Approaches to Populate QSRS 617.3.2 Pilot Project: Test Partial Automation Interface-Level Guidance . 627.3.3 Comparison Study: Automation with One Vendor Versus Manual Abstraction . 637.3.4 Comparison Study: C-CDA and HL7 ADT Interfaces Versus Manual Abstraction . 647.3.5 Additional Research/Pre-Work . 64Conclusions .658 Appendixes . A-18.18.28.3Appendix A. Environmental Scan Keywords and Journals . A-1Appendix B. NLP Vendor Summary Findings . B-1Appendix C. References . C-1FiguresFigure 1. QSRS Source Data Analysis . 3Figure 2. QSRS Question Types in Order of Complexity With Associated ComputationalChallenges . 3Figure 3. Certified EHR Technology Common Clinical Data Set . 12Figure 4. The General Algorithm Development Process and Resulting Coding Framework. . 14Figure 5. Categories of Coded Articles . 15iv

Figure 6. QSRS Question Types in Order of Complexity With Associated ComputationalChallenges . 17Figure 7. Gartner Data Analytics Maturity Model . 33Figure 8. Mapping of Complexity Hierarchy to Low, Medium, and High Complexity Ratings . 34Figure 9. QSRS Source Data Analysis . 37Figure 10. QSRS Source Data Analysis . 38Figure 11. QSRS NLP Complexity Profile . 39Figure 12. QSRS NLP Difficulty . 41Figure 13. Unique Data Sources By QSRS Module . 42Figure 14. Data Sources for QSRS Questions. 43Figure 15. Data Source Mapping to Shortened Source Names . 45Figure 16. Prioritized Entry Questions. 54Figure 17. Entry Questions Prioritized for NLP Evaluation . 55Figure 18. Lower Levels of NLP Difficulty . 57Figure 19. Technology Readiness Level Approach Used by NASA/DoD . 58Figure 20. Human-Computer Automation Levels. 59Figure 21. Human-Computer Automation Levels. 60Figure 22. Proposed Process for Identifying Effective Interface Designs To Support QSRSAutomation .59Figure 23. NLP-Driven Human Guidance . 63v

1 Executive Summary1.1 BackgroundThe Agency for Healthcare Research and Quality (AHRQ) developed and maintains the Quality and SafetyReview System (QSRS), a patient safety surveillance system with clinical event data populated by humanabstractors who review medical records at the Centers for Medicare & Medicaid Services Clinical DataAbstraction Center. QSRS is being considered as a replacement for the current Medicare Patient SafetyMonitoring System, and was developed so that it could also be made available for other users (e.g.,hospitals). QSRS is a patient safety surveillance system designed by AHRQ to detect adverse events (AEs)from a sample of hospital and patient records and to provide reports on rates of AEs reviewed.The rapid adoption of electronic health records (EHRs) meeting industry standards required by the Officeof the National Coordinator for Health Information Technology (ONC) Certification Program provides anopportunity to determine the degree to which QSRS may be enhanced with automated download ofdiscrete data values in electronic medical records and the opportunity to apply natural language processing(NLP).As of 2014, nearly 97 percent of non-Federal acute care hospitals have implemented certified EHRtechnology (CEHRT), and over 75 percent of hospitals have adopted basic EHR functions. Given this highadoption rate, AHRQ’s Center for Quality Improvement and Patient Safety contracted ClinovationsGovernment Health (Clinovations GovHealth) and its subcontracting partner, MedStar Health NationalCenter for Human Factors in Healthcare (MedStar), to perform a feasibility study of full or partialautomation of the QSRS abstraction process using EHR data.This study consisted of five tasks: Task 1 - Study, Review, and Analyze: Perform a review of QSRS events, algorithms, and dataabstraction guidelines and perform an analysis of certified EHR technology.Task 2 - Environmental Scan: Conduct an environmental scan of peer-reviewed and greyliterature publications to identify existing capabilities for electronic tools and methods for1

detection and surveillance of adverse events in health care settings, and the adverse eventsspecified in QSRS.Task 3 - Analyze EHR and Other Source Information: Identify and review adverse eventdetection and surveillance capabilities from EHR vendors, reporting/analytics vendors, andhealth systems/customers.Task 4 - Feasibility Analyses and Review: Analyze EHR capabilities to support QSRS automation,identify opportunities for automation leveraging EHRs and health information technology (IT)capabilities, to discuss barriers and opportunities for automation.Task 5 - Final Report and Presentation: Develop a final presentation and final narrative reportcovering the findings from Tasks 1, 2, 3, and 4.1.2 AnalysisThe team developed an evaluation framework, applied to the 205 QSRS questions, to assess the feasibilityof automation at a question level and overall module level. QSRS abstraction guidelines provide multipledata sources versus a singular, primary data source, as guidance for addressing each question. As EHRdocumentation practices and culture vary across health systems, hospitals within the health system, andprovider types (e.g., gastroenterologist vs. hospitalist; medical/surgical nurse vs. intensive care unit nurse),multiple data sources should be considered to address questions. Certain questions may have a definitiveprimary data source that can negate the need to research additional sections of the patient chart and arewell-positioned for automation.CEHRT provides only for a small subset of functionality needed to support QSRS algorithms. CEHRTstandards and interoperability requirements do not support capture and exchange of the metadata neededto effectively address QSRS algorithms. The detailed clinical information needed to address surveillanceaccording to QSRS algorithms is contained within nursing assessments, flowsheets, and physician/providerprogress notes.Commercial EHR vendors are addressing adverse events (AEs). For a number of AEs that are a part of QSRS,EHRs offer clinical decision support rules/alerts and safety/population health reports that support bothprevention and reporting of AEs. Commercial EHRs rely on clinical documentation practices to chart inspecified manners or require health systems to setup mappings to address localized workflows.Figure 1 below depicts the source data for the 205 QSRS questions. This classification was based upon studyactivities with clinician users of multiple EHRs, stakeholder reviews, and direct analysis with EHR vendors.Questions that can be addressed using numeric values or structured and coded data are the easiest toautomate, and likely no not require advanced analytics, machine learning, or extensive natural languageprocessing (NLP) to analyze the EHR source data to make a QSRS question determination. Informationstored in free text or uncoded values that vary across organizations require NLP for automation.In addressing NLP feasibility, we developed the following framework and classified all questions against thisframework (Figure 2).2

Figure 1. QSRS Source Data AnalysisFigure 2. QSRS Question Types in Order of Complexity With Associated ComputationalChallengesType of QuestionsExampleChallengesPresence of aconcept/entityDid the patient have a urinary tract catheter insertedduring the stay? [GENERIC25]Negation and synonymsNumeric valueextractionDuring this hospitalization, did the patient have a PTTvalue greater than 100 seconds? [MEDIQ10]Part-of-speech taggingand word sensedisambiguationWhich secondary morbidities developed? [PUQ7]Co-referencingTemporal occurrenceof a concept andconcept referencingOn or within the first 24 hours of admission, was ahistory of allergies and/or sensitivities documented?[MEDIQ1]Defining temporalrelationsContingencyDid bleeding develop more than 24 hours afteradmission and within 1 day of [(‘PTT’ 100 seconds)OR (‘Protamine administration’) OR [MEDIQ35]?Co-referencing“Fuzzy” conceptsDid the patient undergo an unplanned transfer to ahigher level care area within the facility or to anotherfacility? [OTHERQ13]Reasoning andSubjectivityOpen-ended responsesIf at all, describe how the device harmed the patient.[DEVICEQ6]SummarizationMultiple conceptdetection3

We determined 58 percent of QSRS questions (118 out of 205) could be reasonably automated with the useof currently available software and applications either without NLP or “low complexity” NLP.Upon adding “medium complexity” NLP for consideration, an additional 40 questions can be automated,resulting in 77 percent of QSRS questions that are categorized as not requiring NLP or those as low andmedium relative NLP complexity. In summary, we believe that 58 percent of QSRS questions are relativelyeasy to automate and 77 percent of QSRS questions are feasible for automation using available capabilitiesin the market today.For the remaining 23 percent that require NLP and are classified as “high complexity,” in parallel with theapproaches and pilots recommended in this section (which best address the 77 percent of questions thatare feasible for automation), AHRQ should consider a review of these questions and determine whetherAHRQ may identify areas for introduction and engagement with standards development organizations andother standards and specifications bodies, as well as health IT certification.1.3 RecommendationsThree modules (Pressure Ulcers, HAI-CDI, and Blood/Blood Products) yielded NLP difficulty levels of onlylow or medium, making them ideal for pilots or proof-of-concept studies for automation. An additional areafor AHRQ focus are the Entry Questions, both generic and module specific. Automation of Entry Questionsenables the automation of processing the greatest number of charts where only those that are flaggedduring the Entry Question process would require manual chart review or abstraction.AHRQ may also want to consider mimicking the eMeasure development process to facilitate automation ofdata collection of EHRs, if QSRS is to be expanded for widespread use. Health system culture is interestedin focusing its health IT interventions on predictive analytics and decision support versus surveillance andleveraging of its EHR investments versus performing manual chart abstraction using limited coding orquality improvement/performance improvement resources and budgets.We recommend a series of next steps and pilots that support evaluation and testing of partial automationapproaches for QSRS. Given that much of the information needed to address QSRS algorithms is text-based,natural language processing (NLP) and advanced analytics capabilities provide promise for automatingcertain QSRS questions. These recommendations include— Proof-of-Concept Project: Test Rule-Based and NLP Approaches to Populate QSRS – Identifyeffective interface designs to support QSRS automation. Determine whether potentialautomated approaches will work by piloting on a sample of data from different EHRs.Pilot Projects/Competition: Test Partial Automation Interface-Level Guidance – Use NLP tofacilitate automatic identification of the correct information to human-in-the-loop processes likehighlighting where the appropriate information might be found.Comparison Study: Automation With One Vendor Versus Manual Abstraction – Single vendoror competition approach to automate as much of QSRS possible, prioritizing the modulesindicated in this report. Conduct a parallel effort using manual abstraction only for comparison.Comparison Study: C-CDA (Consolidated Clinical Document Architecture) and HL7 ADT (AdmitDischarge-Transfer) Interfaces Versus Manual Abstraction – Conduct a study to determine theefficacy of available electronic data using standards formats such as the C-CDA, HL7 ADT, andbilling/claims transactions to determine if the needed information can support QSRS functionsand needs. Conduct a parallel portion of the study where manual abstractors are reviewing thesame charts that are processed electronically.Additional Research/Pre-Work – Additional research using data sources external to the EHR:Mini-Automation Feasibility Assessment that considers non-EHR data sources: Scaled downenvironmental scan to ensure tools used by Performance Improvement, Quality Improvement,and Risk Management teams in hospitals are considered; Risk Management/Insurance Premium4

Opportunities for use of QSRS Surveillance Data; and Review of Hospital Resources and Tools forPredictive Versus Retrospective Chart Analysis.5

2 Background and Approach2.1 Background/NeedThe Agency for Healthcare Research and Quality (AHRQ) developed and maintains the Quality and SafetyReview System (QSRS), a patient safety surveillance system with clinical event data populated by humanabstractors who review medical records at the Centers for Medicare & Medicaid Services (CMS) ClinicalData Abstraction Center (CDAC). QSRS is being considered as a replacement for the current MedicarePatient Safety Monitoring System (MPSMS), and was developed so that it could also be made available forother users (e.g., hospitals). QSRS is a patient safety surveillance system AHRQ designed to detect adverseevents (AEs) from a sample of hospital and patient records and to provide reports on rates of AEs reviewed.Currently, data abstraction for QSRS is manually performed by clinical abstractors who follow a series ofquestions in the software, guided by algorithms seeking specific answers to questions determined bymedical experts. QSRS provides 19 modules for event descriptions within the hospital setting. CDACabstractors review and extract data from 20,000 to 40,000 charts identified by CMS each fiscal year for CMSreviews. Today, charts are requested and provided by hospitals as photocopies or in CD-ROM format forpatients age 18 and older, not restricted to CMS/Medicare beneficiaries. QSRS abstraction for a set ofrecords in a fiscal year takes approximately 9 months, with an abstractor spending about 1 hour and 15minutes per chart to perform chart review and complete the abstraction process. QSRS providesinformation on adverse event incidents that may have occurred but may not have been realized or notedwithin the medical record. It also has a way for abstractors to note a patient safety event that they sawwhile reviewing the chart, whether or not it was noted as an event by the software.The rapid adoption of electronic health records (EHRs) meeting industry standards required by the Officeof the National Coordinator for Health Information Technology (ONC) Certification Program provides anopportunity to determine the degree to which QSRS may be enhanced with automated download ofdiscrete data values in electronic medical records and the opportunity to apply natural language processing(NLP).As of 2014, nearly 97 percent of non-Federal acute care hospitals have implemented certified EHRtechnology, and over 75 percent of hospitals have adopted basic EHR functions 1. Given this high adoption6

rate, AHRQ’s Center for Quality Improvement and Patient Safety contracted Clinovations Government Health (Clinovations GovHealth) and its subcontracting partner, MedStar Health National Center for HumanFactors in Healthcare (MedStar), to perform a feasibility study of full or partial automation of the QSRSabstraction process using EHR data.2.2 Method/ApproachAn in-depth, multiphased approach was developed and implemented to address the complexity of thefeasibility assessment and to ensure that all facets of potential feasibility were explored.The initial task of work involved reviewing all relevant and available information on QSRS and thealgorithms, modules and questions that define it. During this task, an initial review of QSRS was completedby applying the algorithms and questions to EHR charts. The second task of work was the completion of anenvironmental scan and literature review to identify the current state of computational methods availableto detect adverse events in hospital settings from EHRs. Building off the first two tasks of work, the thirdand fourth tasks of work were completed in parallel. The third task of work involved identifying stakeholdergroups and conducting interviews with representatives from each group to obtain additional informationrelated to the focus of the environmental scan. The fourth and final task of work involved an extremelydetailed analysis of the QSRS system in the context of two identified EHR developers and one NLP/analyticsvendor.To support this multiphased approach, a multidisciplinary team was assembled to provide the necessaryskill sets. The team was comprised of practicing clinicians, clinical informaticists, research scientists, healthIT system implementers, and interoperability/standards experts. This diverse team ensured that theassessment was robust and inclusive of all relevant areas impacting the ultimate feasibility of automatingthe QSRS process using EHR data.2.2.1 QSRS Analysis (Task 1)To support subsequent tasks of work and the overall feasibility assessment it was essential that the projectteam obtained a solid foundational understanding of QSRS. This involved reviewing all documents andinformation provided by AHRQ.Objectives of this QSRS analysis included obtaining a working knowledge of— The QSRS Event Descriptions, Algorithms and QuestionsThe complexity and nuances of QSRSThe background and history of QSRS developmentThe current QSRS process and workflowsPossible challenges of implementing QSRSThe initial task of work involved obtaining baseline information on the QSRS system and ensuring that theproject team had all relevant information to support the other tasks. This work involved reviewingdocuments provided by AHRQ which included, but was not limited to— QSRS Manual: Abstraction instructions and guidelinesQSRS Event Descriptions and Algorithms: Flowcharts depicting decision logic for QSRS questionsby moduleQSRS User Guide: Guide for administration and use of QSRS for CMS CDAC usePublicly available information regarding QSRS and MPSMSDemonstration of QSRS functionality and discussion of current

Task 3 - Analyze EHR and Other Source Information: Identify and review adverse event detection and surveillance capabilities from EHR vendors, reporting/analytics vendors, and health systems/customers. Task 4 - Feasibility Analyses and Review: Analyze