Undertaking a Systematic Review:What You Need to KnowAlicia Livinski, MA, MPHDoug Joubert, MS, MLISNancy Terry, MLSOctober 2015NIH Library

Class Objectives Understand the importance of systematic reviews inresearch Distinguish between a narrative review & a systematicreview Identify the steps involved in selecting members for asystematic review team Outline the steps in developing the systematic reviewprotocol

Class Objectives Describe the steps for conducting the literature search Identify appropriate tools for managing data associatedwith a systematic review Understand the different types of bias associated with asystematic review Select the appropriate guidance document to write upyour systematic review for publication

Online companion to the classLink to Online Guide

Overview of a Systematic Review

What is a Systematic Review? “A systematic review attempts to collate all empiricalevidence that fits pre-specified eligibility criteria toanswer a specific research question. It uses explicit,systematic methods that are selected to minimize bias,thus providing reliable findings from which conclusionscan be drawn and decisions made”.Liberati, A., et al. (2009).

Levels of EvidenceAdapted from Strauss and Dartmouth Libraries, (2011).

Rationale for a Systematic Review Inform medical decision making Plan future research agendas Establish clinical or health policy Prevent unnecessary studies Possible use for comparative effectiveness research

Systematic Review Components Starts with a clearly articulated question Uses explicit, rigorous methods to identify, criticallyappraise, and synthesize relevant studies Appraises relevant published and unpublished evidencefor validity before combining and analyzing data Reports methodology, studies included in the review, andconclusions Should be reproducible

Systematic vs Narrative ReviewSystematic ReviewNarrative ReviewClear question to be answered orhypothesis to be testedMay also start with clear question butmore often involves general discussionof subject with no stated hypothesisLocates all relevant published andunpublished studies to limit impact ofpublication and other biasesDoes not usually attempt to locate allrelevant literatureInvolves explicit description of whattypes of studies are to be included tolimit selection biasUsually does not describe why certainstudies are included and othersexcludedExamines in systematic manner themethods used in primary studies;investigates potential biases in thosestudies and sources of heterogeneitybetween study resultsOften does not consider differences instudy methods or study qualityBases conclusions on those studieswhich are most methodologically soundOften does not differentiate betweenmethodologically sound and unsoundstudies

Organizations Cochrane Collaboration Produces and disseminates systematic reviewsof health care interventions through the onlineCochrane Library International source of high quality systematicreviews since 1993 Cochrane Library vis NIH Library Campbell Collaboration An international research network thatproduces systematic reviews of the effects ofsocial interventions

Guidelines for Conducting a SRCRD: Systematic Reviews: CRD's guidance for undertakingsystematic reviews in health careCochrane Collaboration: Cochrane Handbook forSystematic Reviews of InterventionsCochrane Collaboration: Methods newsletterInstitute of Medicine: Finding What Works in Health Care:Standards for Systematic ReviewsAHRQ: Methods Guide for Effectiveness and ComparativeEffectiveness Reviews

Guidance on Reporting SRs PRISMA (Preferred Reporting Items for SystematicReviews and Meta-Analyses) PRISMA-E (PRISMA health equity reporting) MOOSE (Meta-analysis of Observational Studies inEpidemiology) RAMESES publication standards: meta-narrativereviews EQUATOR Collects guidance documents on reporting SRs andother types of health research

PRISMA Checklist PRISMA%202009%20Checklist.pdf

PRISMA Flow Diagram PRISMA%202009%20Checklist.pdf

Steps in a Systematic Review1, 2, 3, 4, 5, 67, 8, 9, 10, 11, 12

The Systematic Review Process1. Assess need for a systematic review2. Assemble the systematic review team3. Develop a research question4. Define inclusion and exclusion criteria5. Develop the protocol for the systematic review6. Locate studies7. Title/abstract & full-text review

The Systematic Review Process8. Extract data9. Assess study quality10. Analyze results11. Write the systematic review12. Submit the review Update the review as needed

A Realistic SR TimelineMonthActivity1-2Prepare protocol3-8Search for published & unpublishedstudies2-3Pilot test eligibility criteria3-8Inclusion assessments33-103Pilot test of ‘Risk of Bias’ assessmentValidity assessmentsPilot test data collection3-10Data collection3-10Data entry5-11Follow up on missing information8-10Analysis1-11Preparation of review report12-Keep the review up-to-dateHiggins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The CochraneCollaboration, 2011. Available from

The Systematic Review Team Include individuals with expertise in: the pertinent clinical content areas systematic review methods - methodologist searching for relevant evidence - librarian/informationist quantitative methods – biostatistician You want to make sure you have enough members todesignate 2 reviewers & 1 tie breaker when reviewingrecords Also 1 administrative support person would be a good ideato include

Systematic Review Protocol Developed before starting the review to serve as roadmap for the review Publication of the protocol prior to beginning: Reduces impact of review authors’ biases Promotes transparency of methods and processes Reduces potential for duplication Allows for peer review of planned methods Registries Proprietary: Cochrane, Campbell Open: PROSPERO (Prospective Register of OngoingSystematic Reviews)Light, R. J. (1984).

The literature searchForm the TeamSearch

Developing the Research Question Confirm the need for the new review. Develop well-framed question(s) that will be answeredthrough the review. A formula for a structured approach that helps you toidentify terminology that captures the question you aretrying to answer is called PICO.

PICO P Population/Patient/Problem/Program How would you describe a group of patients similar toyours? I Intervention, Prognostic Factor, Exposure Which main intervention, prognostic factor, or exposureare you considering? C Comparison What is the main alternative to compare with theintervention? O Outcomes What can you hope to accomplish, measure, improve oraffect?

PICO (example)Question: Are sugar sweetened beverages associated with thedevelopment of dental caries in African-American and Hispanic childrenin the USA?PICOPopulationAfrican-American and Hispanic childrenIntervention/ExposureSugar sweetened beveragesComparison, if anyControl or Comparison GroupOutcomeDevelopment of dental caries

The Literature Search A comprehensive literature search cannot be dependenton a single database. Inclusion of multiple databases helps avoid publicationbias (geographic bias or bias against publication ofnegative results) in the systematic review. Cochrane recommends PubMed/MEDLINE, Embase,and the Cochrane Library Central Register of ControlledTrials (CENTRAL), at a minimum.

Developing Your Search Strategy Quality of the systematic review depends directly on thequality of the identified studies. Balance need for sensitivity (comprehensive) vs.specificity (precision) of retrieval. Strategies must take into account the unique structureand search functions of each database.

Fine Tuning Search Strategies Identify variant terminology/synonyms for specificconcepts. Use both database controlled vocabulary free textwords Run preliminary searches to test recall and retrieval. It is very important to save your search strategies. Infact, when you are doing a SR, save the exact strategiesyou used for each database!

Save Search Strategies - MyNCBI

Grey Literature Term for the mass of information that falls outside themainstream of published journal and monographliterature, not controlled by commercial publishers Often more current than published literature Less publication bias and more global in scope Grey literature includes: Unpublished or hard-to-find studies, reports, ordissertations Conference abstracts or papers Governmental research Clinical trials (ongoing or unpublished)

Grey Literature Sources Sources for Grey Literature: Library catalogs Conference Proceedings Clinical Trials databases, such as Dissertation Abstracts Government databases, such as NTIS, WHO reports Google Scholar HSR Information Central National Institute for Health and Care Excellence - UK Open Grey New York Academy of Medicine Grey Literature Report Reference lists from selected studies

Additional Sources National and Regional Databases – produced bycountries and regions that concentrate on the literatureproduced by those regions Examples: LILACS – Latin American and CaribbeanCenter on Health Sciences Information, Index Medicus forEastern Mediterranean and for Southeast Asia, AfricanIndex Medicus and an Australasian Index Medicus. Subject Specific Databases – concentrate on theliterature on a specific subject Examples: International Pharmaceutical Abstracts,PsycINFO, Sociological Abstracts

Additional Searching Tips Hand Searching Identify the most highly regarded journals in the field Examine journal Table of Contents for potentially relevantarticles Consultation with Experts Ask clinical team members for experts in the field Personal correspondence, etc.

Data organizationSearchDownload

Managing Your References Using software, such as DistillerSR, EndNote, Mendeley,or Zotero you can: Create and maintain a searchable database of records related to the SRCreate groups & group setsUse labels to annotate records with database detailsShare records (EndNote/EN Online, Mendeley, or Zotero)Organize PDFsCreate citations and bibliography when writing up theresults of the SR

DistillerSR (example) 2015 Systematic Review and Literature Review Software from Evidence Partners.

EndNote Library (example)

Mendeley (example)

Record Keeping Document the following: Lists of databases and vendor (e.g., MEDLINE/PubMed,MEDLINE/Ovid) Limits of the search (date ranges, type of study, languagerestrictions) Number of references retrieved Exact search strategies for each database Sources searched for gray literature Other search techniques (e.g., scanning bibliographies ofpertinent articles, contacting authors, hand-searching, etc.)Refer to PRISMA, the gold standard for conducting and reportingSR searches.Liberati, A., et al. (2009).

Record Keeping (example) Set 1: Racial or Ethnic Disparities terms. Set 2: Access Terms. Set 3: Mental Health Services. Set 4: Disorders: depressive disorders, anxietydisorders, schizophrenia, or bipolar disorder terms. APA PsycNET, Medline, and Scopus databases weresearched: APA PsycNET (n 102, number of records) Medline (n 448, number of records) Scopus (n 823, number of records)

Selecting studies for inclusionDownloadScreen

Adhering to PRISMA statement

Screening Articles Once the databases searches are complete, the nextstage in the systematic review is to identify and selectrelevant articles from those retrieved. Each article should be evaluated using the inclusion andexclusion criteria, define in the SR protocol. At least 2 reviewers should review each articleindependently, to minimize bias. Where uncertainly exists, a third review should make anindependent decision.Meade & Richardson. (1997).

Levels of Screening Study eligibility screening Title/abstract level(determining in or out) Full-text level (determiningin or out) Each level of screening isguided by theinclusion/exclusionscriteria defined in your SRprotocol.Wilson, (2013).

Screening Process (example) First stage of screening involved screening the title/abstract against the inclusion and exclusion criteria.Epi Info 7 was used to develop a screening form.4 screeners, worked in pairs.Applied inclusion and exclusion factors using screeningform.Intercoder reliability (concordance rate) was discussedduring weekly meetings.

Epi Info 7 (Slide 1)

Epi Info 7 (Slide 2)

Screening Alternatives

DistillerSR 2015 Systematic Review and Literature Review Software from Evidence Partners.

Inter-rater Reliability It is often recommended to formally assess inter-rater agreement across studies for each item on the selectionform.The simplest measure is to compute the percentage ofagreement between each reviewer.A kappa score (k-statistic) is often used to measureagreement, that is not due to chance.This can be used to create a concordance reportBackground article: Landis, J. R., et al. (1977). TheMeasurement of Observer Agreement for CategoricalData. Biometrics, 33(1), 159-174Meade & Richardson. (1997).

Concordance Reports

Screening counts (example)1373 Titles &Abstracts1285 excluded71 Full texts forinclusion6 in need of thirdscreener11 missing outputReasons for exclusion:- 1181 Not anintervention- 52 Not racial/ethnicminority- 52 Not concerningcommon mentaldisorders and mentalhealth service useoutcomes

After screening Once screening of all relevant full-text reports iscomplete, you will have: A set of studies eligible for coding (data extraction). An accounting of the ineligible studies and the reasons fortheir ineligibility. Campbell and Cochrane reviews often include a table ofineligible studies as an appendix. You are now prepared to move to the Data Extraction(coding) phaseWilson, (2013).

Data ExtractionScreenCoding

Adhering to PRISMSA statement

What data should you collect? Comprehensive data about each study: Participants/clients/sample Interventions Methods and potential sources of bias Outcomes effect sizes, and authors conclusions Sources of funding This data is required for: References Description of included studies Risk of bias assessment AnalysisHiggins, JPT, Deeks, JJ (2011).Wilson, (2013).

Data Coding Manual The coding manual explicitly outlines what you will belooking at, when extracting data from studies. Cochrane recommends that you pilot your dataextraction form, to ensure that all participating authorsare retrieving comparable results, and this should benoted in the protocol. Information on study characteristics should provideenough information to allow readers to assess theapplicability of the findings to their area of interest.Wilson, (2013).

Studies with no usable data Studies must be included in the review if they meet yourpre-defined criteria. Studies that do not report outcomes of interested mighthave still measured them, so these still need to beincluded in your review. Studies that “did not measure outcomes of interest mayonly be excluded if measured outcomes were included inyour predefined eligibility criteria.” Also, you must report excluded studies, and why thesestudies were excluded.Higgins, JPT, Deeks, JJ (2011).

Data extraction considerations Multiple studies on the same data (study versus areport). Especially in large studies.people will publish multiplearticles and you cannot treat these as separate studiessame sample, same experiment. Campbell recommends that you identify all "friend studies"and code them as a "block" Publication type and publication bias issues This is why searching grey literature is so important Publication date versus study date (sometimes hard tofind or determine)Wilson, (2013).

What is bias, and what is not bias? There are two elements that affect the validity of thefindings from a systematic review: External validity: how applicable are the sample results tothe population. Internal validity: how correct is your estimate of the effectyou are trying to measure. Bias is the systematic error or a deviation from the“truth.” Bias is directly related to the internal validity of a study. Bias is not the same as imprecision this is randomerror.Agency for Healthcare Research Quality, (2012).The Cochrane Collaboration, (2013).

Options for data extraction Variety of options for coding study methods: Cochrane Risk of Bias framework GRADE system Method quality checklist Direct coding of methodological characteristicsTanner-Smith, E., & Wilson, S. J., (2013).

Common types of biasType of bias DescriptionCollaboration’s ‘Risk of bias’ tooldomainSelectionbiasSystematic differences betweenbaseline characteristics of the groupsthat are compared. Sequence generation: due toinadequate generation of arandomized sequence. Allocation concealment: due toinadequate concealment ofallocations prior to assignment.PerformancebiasSomething other than the interventionaffects groups differently. Blinding of participants, personneland outcome assessors. Other potential threats to validity.DetectionbiasMethod of outcomes assessmentaffects group comparison. Blinding of participants, personneland outcome assessors. Other potential threats to validity.Attrition biasSystematic differences in the loss ofparticipants from the study and howthey were accounted for in the results. Incomplete outcome data. Blinding of participants, personneland outcome assessors.ReportingbiasSystematic differences betweenreported and unreported findings. Onlyreport outcomes of interest. Selective outcome reporting.Adapted from Cochrane Handbook for Systematic Reviews of Interventions, Chapter 8. (2011).

The GRADE SystemGuyatt et al. BMJ 2008;336:924

Method quality checklist More than 200 scales and checklists available, few if anyappropriate for systematic reviews Overall study quality scores have questionable reliabilityand validity (Joni et al., 2001): Conflate different methodological issues and study design/implementation features, which may have different impactson reliability/validity Preferable to examine potential influence of keycomponents of methodological quality individually Weighting results by study quality scores is not advised!Wilson, (2013).

Methodological Quality Assessmentof Studies for RCTs Jadad score (for RCTs) McMaster University Harms scale (McHarm) tool Physiotherapy Evidence Database (PEDro) scale

Methodological Quality Assessmentof Studies for non-RCTs AHRQ Medical Test Guidance. Cochrane Effective Practice and Organisation of Care(EPOC) Group Risk of Bias Tool. Quality Assessment of Diagnostic Accuracy Studies(QUADAS). McMaster University Harms scale (McHarm) tool. Newcastle - Ottawa Quality Assessment Scale (casecontrol/cohort) studies.

Assessment of study quality andrisk of bias (public health studies) Assessing the quality of public health and healthpromotion studies, and their resulting risk of bias, maybe difficult, partly due to the wide variety of studydesigns used (Cochrane, 2011). EPHPP Assessment Tool Critical Appraisal Skills Programme (CASP) AppraisalChecklist National Centre for Social Research (UK) Cochrane Public Health Group (CPHG)Adapted from Cochrane Handbook for Systematic Reviews of Interventions, Chapter 8. (2011).

Common mistakes in coding Too many coding items Coding items should be outlined in your SR protocol: beselective in the number of items you want to code(inclusion/exclusion criteria) Coding two reports from the same study as two differentstudies Coder drift You have started coding one way and then 50 studieslater you have drifted away from original coding method Failure to ask questions (checking in) If you are not comparing notes and asking questionsduring coding then you are doing something wrongWilson, (2013).

Synthesizing the EvidenceCodingWriting

Authorship ICJME authorship recommendationsDefining the Role of Authors and Contributors Why authorship matters Who is an author Contributions of non-authors Journal editorial and authorship instructions/policies Acceptance of sys reviews How to report a systematic review Which standard(s) to use etc. Governs acknowledgements too

Standards for Reporting The Institute of Medicine recommendations areorganized into the following categories: Systematic Reviews Published in Journals Recommended Standard for Preparing the Final Report Recommended Standard for Report Review Recommended Standard for Publishing the Final Report PRISMA Statement & Checklist* Cochrane Handbook* for Systematic Reviews ofInterventions Others mentioned earlierInstitute of Medicine, Finding What Works in Health Care: Standards for Systematic Reviews pg. 195 (2011).

Writing the Abstract PRISMA checklist outlines what information should beincluded in the various sections of your paper PRISMA for abstracts – use structured summaryCochrane Handbook standard on Abstract information List databases searched Language or publication status restrictions Dates of last search for each dbase or period searchedAdapted from PRISMA 2009 Checklist. (2009).

Example of SR Abstract Structured summary Includes detailed methods section 22552846

Writing the Methods PRISMA Checklist elements for including in Methodssection of paper

Writing the Results, Discussion etc. PRISMA Checklist Cochrane Handbook PRISMA Checklist Cochrane Handbook

Example Flow Chart

DistillerSR - Overview 2015 Systematic Review and Literature Review Software from Evidence Partners.

DistillerSR - Benefits DistillerSR’s Intuitive 5 Step Process Step 1: Load Your References Step 2: Create Your Forms Step 3: Lay Out Your Workflow and Assign Reviewers Step 4: Monitor and Tune Your Review Step 5: Export Your Results 2015 Systematic Review and Literature Review Software from Evidence Partners.

Following Up If you would like to discuss conducting a systematic reviewand how a librarian can assist, or how to use DistillerSR,please contact either: Alicia Livinski, Informationist, NIDCR Holly Thompson, Informationists, Point of Contact NIDCR

Thank you.NIH Library

Describe the steps for conducting the literature search Identify appropriate tools for managing data associated with a systematic review Understand the different types of bias associated with a systematic review Select the appropriate guidance do