C. Publication Bias

As described in chapter III, various forms of bias can affect the validity of evidence used in HTA. One reason for careful planning and conduct of search strategies for HTA is to recognize and minimize the effects of publication bias. Studies of the composition of the biomedical research literature have found imbalances in the publication patterns of methodologically sound studies (Chalmers 1990), including in HTA (Song 2010). For instance, positive studies (i.e., that find statistically significant effects of a technology) are more likely than negative studies (i.e., that find no effects of a technology) to be published in peer-reviewed journals (Dickersin 1993, 1997). A study sponsored by a company or other organization with a financial or other interest in the results may be less likely to be submitted for publication if the findings are not favorable to the interests of that organization. RCTs and other clinical trials conducted for market approval (e.g., by the US FDA) often are not published for proprietary reasons (MacLean 2003). Some research indicates that, among studies of health technologies that are published, smaller studies tend to report positive results more frequently than larger ones (Agema 2002). Publication bias is not the same as reporting bias, which refers to differential (e.g., between treatment and control groups) or incomplete reporting by investigators of findings in individual studies. (See further discussion in chapter III.)

Multiple analyses have found that positive studies are more likely to be published in English-language journals, reported in multiple publications, reported for late-phase drug trials, and cited in other articles (Easterbrook 1991, Gøtzsche 1989; Hall 2007). These multiple appearances and citations increase the likelihood of being identified in literature searches and included in meta-analyses and other systematic reviews, which may introduce bias into the results of these syntheses as well (Sterne 2001). Certain forms of bias may be linked to the clinical area under study. A detailed analysis of the characteristics of clinical trials used in systematic reviews indicated that, compared to other clinical areas, trials in the fields of psychiatry, rheumatology, and orthopedics tend more often to be published in non-English languages and appear in sources not indexed in MEDLINE (Egger 2003).

Bias in selection of studies used in HTA may arise when a literature searching strategy (or protocol) excludes reports by language (i.e., language bias), when it draws only on MEDLINE or other major bibliographic databases, when it includes only reports in the peer-reviewed literature, or when it excludes studies of lesser (or poorly reported) methodological quality (Jüni 2002). The recent global trend toward publication in English-language journals may be reducing the impact of language bias (Galandi 2006; Higgins 2011).

Prospective, systematic literature searching protocols for HTA should seek to identify relevant evidence beyond traditional published sources where possible. This should include circumstances in which potential conflicts of interest might affect submission or publication of clinical trial reports or other forms of evidence.

Time lag bias occurs when the time from completion of a clinical trial to its publication is affected by the direction (positive vs. negative findings) and strength (statistical significance) of the trial results. This can have implications for the timing of conducting HTA and may be of particular importance when the number of relevant studies is small (Hopewell, Clarke 2007; Ioannidis 1998).

While the validity of an HTA is likely affected by the effort to include an unbiased sample of relevant studies, the size and direction of this relationship varies. There is a growing literature on the extent to which more or less restrictive inclusion criteria for meta-analyses affect their results. For example, some research indicates that systematic reviews limited to the English language literature that is accessible via the major bibliographic databases produces similar or same results to those based on less restricted reviews (Egger 2003). Even so, exclusion of non-English-language studies may result in higher risk of bias in some areas of research, e.g., complementary and alternative medicine (Song 2010). Lowering the standard of methodological quality for inclusion of published studies in an HTA may bias the findings if these included studies tend to report positive findings more often that higher-quality studies.

Recognition of publication bias in peer-reviewed literature may increase the acceptance of grey literature, to the extent that the grey literature can contribute to a more representative capture of the available relevant evidence pertaining to a given topic. An analysis of the impact of grey literature in meta-analyses of RCTs found that published trials tended to be larger and report greater treatment effects than RCTs reported in the grey literature (i.e., abstracts and unpublished data). The analysis also reported that there was limited evidence to show whether grey RCTs were of poorer methodological quality than published trials (Hopewell, McDonald 2007).

Publication bias can be diminished by prospective registration of clinical trials, as in ClinicalTrials.gov; adherence to guidelines for reporting clinical trials and other types of studies in journals; efforts to locate relevant unpublished studies and non-English language studies for literature searches; and certain statistical techniques (e.g., “funnel plots”) to detect potential publication bias (Song 2010; Sterne 2011).

The emergence, validation, and ongoing improvement of guidelines and checklists for reporting research is helping to strengthen the quality of the biomedical and health services research literature, including regarding aspects of publication bias, for HTA reports. Over the last decade in particular, such guidelines have been developed for published reports of diverse study types relevant to HTA. As detailed in chapter IV, some of these include CONSORT (Consolidated Standards of Reporting Trials), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), and STROBE (Strengthening the Reporting of OBservational Studies in Epidemiology). In addition to guiding the reporting of completed research, researchers’ knowledge of such guidelines and related standards is improving the design and conduct of research. An extensive list of these is maintained in Research Reporting Guidelines and Initiatives by NLM. See: http://www.nlm.nih.gov/services/research_report_guide.html.

In planning a literature search, assessors should weigh the anticipated quality of a search with time and resource constraints. Efforts to recognize and minimize bias may be further subject to such factors as the availability of studies by language and for particular clinical areas, and their accessibility via bibliographic databases.