A. Systematic Literature Reviews

A systematic literature review is a form of structured literature review that addresses one or more evidence questions (or key questions) that are formulated to be answered by analysis of evidence. Broadly, this involves:

  • Objective means of searching the literature
  • Applying predetermined inclusion and exclusion criteria to this literature
  • Critically appraising the relevant literature
  • Extraction and synthesis of data from evidence base to formulate answers to key questions

Depending on the purpose of the systematic review and the quality of the included studies, systematic reviews often include meta-analyses. A useful way to define the key questions used in a systematic review and to guide the literature search is the “PICOTS” format (see, e.g., Counsell 1997):

  • Population: e.g., condition, disease severity/stage, comorbidities, risk factors, demographics
  • Intervention: e.g., technology type, regimen/dosage/frequency, technique/method of administration
  • Comparator: e.g., placebo, usual/standard care, active control
  • Outcomes: e.g., morbidity, mortality, quality of life, adverse events
  • Timing: e.g., duration/intervals of follow-up
  • Setting: e.g., primary, inpatient, specialty, home care

Not all evidence questions use all of these elements; some use PICO only.

The main steps of a systematic review include the following (see, e.g., Buckley 2014; Rew 2011; Shea 2007; Sutton 1998):

  1. Specify purpose of the systematic review
  2. Specify evidence questions. Use appropriate structured format, e.g., PICOTS
  3. Specify review protocol that is explicit, unbiased, and reproducible, including:
    • Inclusion and exclusion criteria for studies to be reviewed, including type/status of publication (e.g., peer-reviewed publication vs. grey literature)
    • Bibliographic databases (and other sources, if applicable) to be searched
    • Search terms/logic for each database
    • Methods of review (e.g., number of independent parallel reviewers of each study)
    • Intention to conduct meta-analysis (if appropriate and feasible) and specification of methods to combine/pool data
    • Register or publish protocol, as appropriate
  4. Perform comprehensive literature search
    • Document all search sources and methods
  5. Review search results and compare to inclusion/exclusion criteria
    • Account for included and excluded studies (e.g., using a flow diagram)
    • Identify and exclude duplicate studies, as appropriate
    • Compile and provide lists of included studies and excluded studies (with reasons for exclusion)
    • Assess potential sources of publication bias
  6. Systematically extract data from each included study
    • Consistent with review protocol
    • Include PICOTS characteristics
    • Present extracted data in tabular form
  7. Assess quality of individual studies retrieved/reviewed
    • Document quality for each study
    • Account for potential conflicts of interest
  8. Perform meta-analysis (if specified in protocol and if methodologically feasible based on primary data characteristics)
  9. Assess quality (or strength) of cumulative body of evidence
    • Assess risks of bias, directness or relevance of evidence (patients, interventions, outcomes, etc.) to the evidence questions, consistency of findings across available evidence, and precision in reporting results
    • Assign grade to cumulative body of evidence
  10. Present results/findings
    • Link results/findings explicitly to evidence from included studies
    • Account for quality of the included studies
    • Present clearly to enable critical appraisal and replication of systematic review
  11. Conduct sensitivity analysis of review results
    • Examine the impact on review results of inclusion/exclusion criteria, publication bias, and plausible variations in assumptions and estimates of outcomes and other parameters
    • Also conduct analyses (e.g., subgroup analyses and meta-regression) for better understanding of heterogeneity of effects
  12. Describe limitations and actual/potential conflicts of interest and biases in the process
    • Account for body of included studies and the systematic review process
    • Describe evidence gaps and future research agenda, as appropriate
  13. Disseminate (e.g., publish) results

Assessing the quality of individual studies is described in chapter III. Assessing the quality (or strength) of a cumulative body of evidence is described later in chapter IV. Dissemination of results is described in chapter VIII.

In conducting literature searches for systematic reviews, a more comprehensive and unbiased identification of relevant clinical trials and other studies (consistent with the inclusion criteria) may be realized by expanding the search beyond the major biomedical bibliographic databases such PubMed and Embase. Additional in-scope studies may be identified via specialized databases and clinical trial registries, reference lists, hand-searching of journals, conference abstracts, contacting authors and trials sponsors (e.g., life sciences companies) to find unpublished trials, and Internet search engines. The studies identified in these ways should remain subject to the quality criteria used for the systematic review. The extent to which an expanded search has an impact on the findings of the systematic review varies (Savoie 2003). Systematic reviews of particular types of technologies may use different sets of databases and synthesis approaches for particular types of technologies, such as for imaging procedures and diagnostic tests (Bayliss 2008; Whiting 2008).

One of the tools developed to assess the quality of systematic reviews, the Assessment of Multiple Systematic Reviews (AMSTAR), which was derived using nominal group technique and factor analysis of previous instruments, has the following 11 items (Shea 2007; Shea 2009):

  1. Was an ‘a priori’ design provided?
  2. Was there duplicate study selection and data extraction?
  3. Was a comprehensive literature search performed?
  4. Was the status of publication ([e.g.,] grey literature) used as an inclusion criterion?
  5. Was a list of studies (included and excluded) provided?
  6. Were the characteristics of the included studies provided?
  7. Was the scientific quality of the included studies assessed and documented?
  8. Was the scientific quality of the included studies used appropriately in formulating conclusions?
  9. Were the methods used to combine the findings of studies appropriate?
  10. Was the likelihood of publication bias assessed?
  11. Was the conflict of interest stated?

In addition to those for assessing methodological quality of systematic reviews, there are instruments to assess the reporting of systematic reviews and meta-analyses, including the Preferred Reporting Items of Systematic reviews and Meta-Analyses instrument (PRISMA) (Moher 2009), as shown in Box 1. Various computer software packages are available to manage references and related bibliographic information for conducting systematic reviews; examples are EndNote, Reference Manager, and RefWorks (see, e.g., Hernandez 2008), though no particular recommendation is offered here.

Box 1. PRIMSA Checklist of Items to Include When Reporting a Systematic Review

Section/topic # Checklist item Reported on page #
TITLE
Title 1 Identify the report as a systematic review, meta-analysis, or both.
ABSTRACT
Structured summary 2 Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number.
INTRODUCTION
Rationale 3 Describe the rationale for the review in the context of what is already known.
Objectives 4 Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS).
METHODS
Protocol and registration 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number.
Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale.
Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched.
Search 8 Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated.
Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis).
Data collection process 10 Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators.
Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made.
Risk of bias in individual studies 12 Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis.
Summary measures 13 State the principal summary measures (e.g., risk ratio, difference in means).
Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I2) for each meta-analysis.

Source: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6(7):e1000097.

Box IV-1. PRIMSA Checklist of Items to Include When Reporting a Systematic Review (cont’d)

Section/topic # Checklist item Reported on page #
Risk of bias across studies 15 Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies).
Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified.
RESULTS
Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram.
Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations.
Risk of bias within studies 19 Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12).
Results of individual studies 20 For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot.
Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals and measures of consistency.
Risk of bias across studies 22 Present results of any assessment of risk of bias across studies (see Item 15).
Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]).
DISCUSSION
Summary of evidence 24 Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers).
Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias).
Conclusions 26 Provide a general interpretation of the results in the context of other evidence, and implications for future research.
FUNDING
Funding 27 Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review.

Source: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6(7):e1000097.

A resource for minimizing publishing bias in systematic reviews is PROSPERO, an international database of prospectively registered systematic reviews in health and social care. The main objectives of PROSPERO are to reduce unplanned duplication of systematic reviews and provide transparency in the review process with the aim of minimizing reporting/publication bias. By providing a comprehensive listing of systematic reviews and their protocols at their inception, PROSPERO is intended to help counter publication bias by providing a permanent record of the original protocol of each systematic review, whether or not it is published. As such, comparison of this record to any reported findings of the systematic reviews can reveal any differences between the methods and outcomes of the registered protocol with those that are ultimately reported. Also, registration of reviews should diminish instances of duplication of effort. Established in 2011, PROSPERO is managed by the Centre for Reviews and Dissemination and funded by the UK National Institute for Health Research (Booth 2012).

results matching ""

    No results matching ""