F. Complementary Methods for Internal and External Validity
Those who conduct technology assessments should be as innovative in their evaluations as the technologies themselves ... The randomized trial is unlikely to be replaced, but it should be complemented by other designs that address questions about technology from different perspectives (Eisenberg 1999).
Given the range of impacts evaluated in HTA and its role in serving decision makers and policymakers with diverse responsibilities, HTA must consider the methodological validity and other attributes of various primary data methods. There is increasing recognition of the need for evidence generated by primary data methods with complementary attributes.
Although primary study investigators and assessors would prefer to have methods that achieve both internal and external validity, they often find that study design attributes that increase one type of validity jeopardize the other. As described above, a well-designed and conducted RCT is widely considered to be the best approach for ensuring internal validity. However, for the reasons that an RCT may have high internal validity, its external validity may be limited.
Findings of some large observational studies (e.g., from large cohort studies or registries) have external validity to the extent that they can provide insights into the types of outcomes that are experienced by different patient groups in different circumstances. However, these less rigorous designs are more subject to certain forms of bias and confounding that threaten internal validity of any observed relationship between an intervention (or other exposure) and outcomes. These studies are subject, for example, to selection bias on the part of patients, who have self-selected or otherwise influenced choice of an intervention, and investigators, who select which populations to study and compare. They are also subject to investigator detection bias. Interesting or promising findings from observational studies can generate hypotheses that can be tested using study designs with greater internal validity.
It is often not practical to conduct RCTs in all of the patient populations that might benefit from a particular intervention. Combinations of studies that, as a group, address internal validity and external validity may suffice. For example, RCTs demonstrating the safety and efficacy in a narrowly defined patient population can be complemented with continued follow-up of the original patient groups in those trials and by observational studies following more diverse groups of patients over time. These observational studies might include registries of larger numbers of more diverse patients who receive the intervention in various health care settings, studies of insurance claims data for patients with the relevant disease and intervention codes, studies using medical records, and postmarketing surveillance for adverse events in patients who received the intervention. Further, the RCT and observational data can provide inputs to computer-based simulations of the safety, effectiveness, and costs of using the intervention in various patient populations.
The methodological literature often contends that, due to their inherent lack of rigor, observational studies tend to report larger treatment effects than RCTs. However, certain well-designed observational studies can yield results that are similar to RCTs. An analysis published in 2000 that compared treatment effects reported from RCTs to those reported from observational studies for 19 treatments between 1985 and 1998 found that the estimates of treatment effects were similar for a large majority of the treatments (Benson 2000). Similarly, a comparison of the results of meta-analyses of RCTs and meta-analyses of observational studies (cohort or case control designs) for the same five clinical topics published between 1991 and 1995 found that the reported treatment effects (including point estimates and 95% confidence intervals) were similar (Concato 2000).
Similar to quality assessment tools for various types of studies, the GRACE (Good ReseArch for Comparative Effectiveness) principles were developed to evaluate the methodological quality of observational research studies of comparative effectiveness. The GRACE principles comprise a series of questions to guide the evaluation, including what belongs in a study plan, key elements for good conduct and reporting, and ways to assess the accuracy of comparative effectiveness inferences for a population of interest. Given the range of types of potentially relevant evidence and the need to weigh applicability for particular circumstances of routine care, GRACE has no scoring system (Dreyer 2010). The accompanying GRACE checklist is used to assess the quality and usefulness for decision making of observational studies of comparative effectiveness (Dreyer 2014).