A. Primary Data Studies: Diverse Attributes

Primary data methods involve collection of original data, ranging from more scientifically rigorous approaches for determining the causal effect of health technologies, such as randomized controlled trials (RCTs), to less rigorous ones, such as case series. These study designs can be described and categorized based on multiple attributes or dimensions, e.g.:

• Comparative vs. non-comparative
• Separate (i.e., external) control group vs. no separate (i.e., internal) control group
• Participants (study populations /groups) defined by a health outcome vs. by having been exposed to, or received or been assigned, an intervention
• Prospective vs. retrospective
• Interventional vs. observational
• Experimental vs. non-experimental
• Random assignment vs. non-random assignment of patients to treatment and control groups

All experimental studies are, by definition, interventional studies. Some non-experimental studies can be interventional, e.g., if investigators assign a technology to a patient population but without a control group or with a non-randomized control group, and then assess their outcomes. An interventional cross-sectional design can be used to assess the accuracy of a diagnostic test. Some study designs are better at rigorous demonstration of causality in well-defined circumstances, such as the RCT. Other study designs may be better for reflecting real-world practice, such as pragmatic clinical trials and some observational studies, such as cohort, cross-sectional, or case control studies using data from registries, surveillance, electronic health (or medical) records, and payment claims.

Box III-1. Examples of Experimental and Non-Experimental Study Designs

*A controlled trial in which participants are assigned to treatment and control groups using a method other than randomization, yet intended to form similar groups. Sometimes known as a “quasi-experimental” design.

Box III-1 categorizes various types of primary data studies as experimental and non-experimental. Researchers have developed various frameworks, schemes, and other tools for classifying study designs, such as for the purpose of conducting systematic reviews (Hartling 2010). Box III-2 and Box III-3 show algorithms for identifying study designs. Some of these study designs have alternative names, and some studies use diverse combinations of design attributes.

Box III-2. Study Design Algorithm, Guide to Community Preventive Services

Source: Briss PA, Zasa S, Pappaioanou M, Fielding J, et al. Developing an evidence-based Guide to Community Preventive Services--Am J Prev Med 2000;18(1S):35-43, Copyright © 2000) with permission from Elsevier.

Box III-3. Design Algorithm for Studies of Health Care Interventions*

*Developed, though no longer advocated by, the Cochrane Non-Randomised Studies Methods Group.

Source: Hartling L, et al. Developing and Testing a Tool for the Classification of Study Designs in Systematic Reviews of Interventions and Exposures. Agency for Healthcare Research and Quality; December 2010. Methods Research Report. AHRQ Publication No. 11-EHC-007.

Although the general type of a study design (e.g., RCT, prospective cohort study, case series) conveys certain attributes about the quality of a study (e.g., control group, random assignment), study design type alone is not a good proxy for study quality. More important are the attributes of study design and conduct that diminish sources of bias and random error, as described below.

New types of observational study designs are emerging in the form of patient-centered online registries and related research platforms. For example, PatientsLikeMe, a patient network, is set up for entry of member patient demographic information, treatment history, symptoms, outcome data, and evaluations of treatments, as well as production of individual longitudinal health profiles and aggregated reports. Such patient-centered registries can supplement clinical trials and provide useful postmarket data across heterogeneous patients and circumstances (Frost 2011, Nakamura 2012).

Most HTA programs rely on integrative methods (especially systematic reviews), particularly to formulate findings based on available evidence from primary data studies that are identified through systematic literature searches. Some HTA programs collect primary data, or are part of larger organizations that collect primary data. It is not always possible to conduct, or base an assessment on, the most rigorously designed studies. Indeed, policies and decisions often must be made in the absence, or before completion, of definitive studies. Given their varying assessment purposes, resource constraints, and other factors, HTA programs use evidence from various study designs, although they usually emphasize evidence based on the more rigorous and systematic methods of data collection.

The following sections describe concepts that affect the quality of primary data studies, particularly their ability to yield unbiased and precise estimates of treatment effects and other findings.