E. Different Study Designs for Different Questions

RCTs are not the best study design for answering all evidence questions of potential relevance to an HTA. As noted in Box III-11, other study designs may be preferable for different questions. For example, the prognosis for a given disease or condition may be based on a follow-up studies of patient cohorts at uniform points in the clinical course of a disease. Case control studies, which are usually

Box III-11. RCTs Are Not the Best Study Design for All Evidence Questions

Other study designs may include the following (not a comprehensive list):

• Prevalence of a disease/disorder/trait? Random sample survey of relevant population
• Identification of risk factors for a disease/disorder/adverse event? Case control study (for rare outcome) or cohort study (for more common outcome)
• Prognosis? Patient cohort studies with follow-up at uniform points in clinical course of disease/disorder
• Accuracy and reliability of a diagnostic test? Cross-over study of index test (new test) vs. reference method (“gold standard”) in cohort of patients at risk of having disease/disorder
• Accuracy and reliability of a screening test? Cross-over study of index test vs. reference method (“gold standard”) in representative cross-section of asymptomatic population at-risk for trait/disorder/preclinical disease
• Efficacy/effectiveness (for health outcomes) of screening or diagnostic tests? RCT if time and resources allow; observational studies and RCTs rigorously linked for analytic validity, clinical validity, and clinical utility
• Efficacy/effectiveness (for health outcomes) of most therapies and preventive interventions? RCT
• Efficacy/effectiveness of interventions for otherwise fatal conditions? Non-randomized trials, case series
• Safety, effectiveness of incrementally modified technologies posing no known additional risk? Registries
• Safety, effectiveness of interventions in diverse populations in real-world settings? Registries, especially to complement findings of available RCTs, PCTs
• Rates of recall or procedures precipitated by false positive screening results? Cohort studies
• Complication rates from surgery, other procedures? Registries, case series
• Identification of a cause of a suspected iatrogenic (caused by a physician or therapy) disorder? Case-control studies
• Incidence of common adverse events potentially due to an intervention? RCTs, nested case-control studies, n-of-1 trial for particular patients, surveillance, registries
• Incidence of rare or delayed adverse events potentially due to an intervention? Surveillance; registries; n-of-1 trial for particular patients; large, long-term RCT if feasible

retrospective, are often used to identify risk factors for diseases, disorders, and adverse events. The accuracy of a new diagnostic test (though not its ultimate effect on health outcomes) may be determined by a cross-over study in which patients suspected of having a disease or disorder receive both the new (“index”) test and the “gold standard” test. Non-randomized trials or case series may be preferred for determining the effectiveness of interventions for otherwise fatal conditions, i.e., where little or nothing is to be gained by comparison to placebos or known ineffective treatments. Surveillance and registries are used to determine the incidence of rare or delayed adverse events that may be associated with an intervention. For incrementally modified technologies posing no known additional risk, registries may be appropriate for determining safety and effectiveness.

Although experimentation in the form of RCTs is regarded as the gold standard for deriving unbiased estimates of the causal effect of an intervention on health care outcomes, RCTs are not always necessary to reach the same convincing finding. Such instances arise when the size of the treatment effect is very large relative to the expected (well-established and predictable) prognosis of the disease and when this effect occurs quickly relative to the natural course of the disease, as may be discerned using historical controlled trials and certain well-designed case series and non-randomized cohort studies. Some examples include ether for anesthesia, insulin for diabetes, blood transfusion for severe hemorrhagic shock, penicillin for lobar pneumonia, ganciclovir for cytomegalovirus, imiglucerase for Gaucher’s disease, phototherapy for skin tuberculosis, and laser therapy for port wine stains (Glasziou 2007; Rawlins 2008).