B. Setting Assessment Priorities

Some assessment programs have explicit procedures for setting priorities; others set priorities only in an informal or ad hoc manner. Given very limited resources for assessment and increasing accountability of assessment programs to their parent organizations and others who use or are affected by their assessments, it is important to articulate how assessment topics are chosen.

Most assessment programs have criteria for topic selection, although these criteria are not always explicit. For example, is it most important to focus on costly health problems and technologies? What about health problems that affect large numbers of people, or health problems that are life-threatening? What about technologies that cause great public controversy? Should an assessment be undertaken if it is unlikely that its findings will change current practice? Examples of selection criteria that are used in setting assessment priorities are shown in Box VI-2.

Box VI-2. Examples of HTA Selection Criteria Used in Setting Assessment Priorities

• High individual burden of morbidity, mortality, or disability
• High population burden of morbidity, mortality, or disability
• High unit/individual cost of a technology or health problem
• High aggregate/population cost of a technology or health problem
• Substantial variations in practice
• Unexpected adverse event reports
• Potential for HTA findings to have impact on practice
• Potential for HTA findings to have impact on patient outcomes or costs
• Available findings not well disseminated or adopted by practitioners
• Need to make regulatory decision
• Need to make payment decision (e.g., provide coverage or include in health benefits)
• Need to make a health program acquisition or implementation decision
• Recent or anticipated “breakthrough” scientific findings
• Sufficient research findings available upon which to base HTA
• Feasibility given resource constraints (funding, time, etc.) of the assessment program
• Public or political demand
• Scientific controversy or great interest among health professionals

The timing for undertaking an assessment may depend on the availability of evidence. For example, the results of a recently completed major RCT or meta-analysis may challenge current practice, and prompt an HTA to consolidate these results with other available evidence for informing clinical or payment decisions. Or, an assessment may be delayed pending the results of an ongoing study that has the potential to shift the weight of the body of evidence on that topic.

As noted in section II. Fundamental Concepts, the demand for HTA by health care decision makers has increasingly involved requests for faster responses to help inform emergent regulatory, payment, or acquisition decisions. The urgency of such a request may raise the priority of an assessment topic and prompt an HTA organization to designate it for a more focused, less-comprehensive “rapid HTA.” See discussion of rapid HTA in chapter X.

Systematic priority-setting processes typically include such steps as the following (Donaldson and Sox 1992; Lara and Goodman 1990).

1. Select criteria to be used in priority setting.
2. Assign relative weights to the criteria.
3. Identify candidate topics for assessment (e.g., as described above).
4. If the list of candidate topics is large, reduce it by eliminating those topics that would clearly not rank highly according to the priority setting criteria.
5. Obtain data for rating the topics according to the criteria.
6. For each topic, assign a score for each criterion.
7. Calculate a priority score for each topic.
8. Rank the topics according to their priority scores.
9. Review the priority topics to ensure that assessment of these would be consistent with the organizational purpose.

Processes for ranking assessment priorities range from being highly subjective (e.g., informal opinion of a small group of experts) to quantitative (e.g., using a mathematical formula) (Donaldson 1992; Eddy 1989; Phelps 1992). Box VI-3 shows a quantitative model for priority setting. The Cochrane Collaboration has used a more decentralized approach in which review groups use a range of different priority-setting systems (Clarke 2003; Nasser 2013). Starting with topics suggested by their members, many Cochrane Collaboration review groups have set priorities by considering burden of disease and other criteria, as well as input from discussions with key stakeholders and suggestions from consumers. These priorities have been offered to potential reviewers who might be interested in preparing and maintaining relevant reviews in these areas.

Box VI-3. A Quantitative Model for Priority Setting

• A 1992 report by the Institute of Medicine provided recommendations for priority setting to the Agency for Health Care Policy and Research (now AHRQ). Seven criteria were identified:
• Prevalence of a health condition
• Burden of illness
• Cost
• Variation in rates of use
• Potential of results to change health outcomes
• Potential of results to change costs
• Potential of results to inform ethical, legal, or social issues

The report offered the following formula for calculating a priority score for each candidate topic.

Proriority Score = W₁lnS₁+ W₂lnS₂+ ... W₇lnS₇

where:

  _W_ is the relative weight of each of seven priority-setting criteria

  _S_ is the score of a given candidate topic for a criterion

  _ln is the natural logarithm of the criterion scores_.

Candidate topics would then be ranked according to their priority score.

Source: Donaldson MS, Sox HC, Jr, eds. Setting Priorities for Health Technology Assessment: A Model Process. Washington, DC: National Academy Press; 1992. Reprinted with permission from the National Academy of Sciences, courtesy of the National Academies Press, Washington, DC.

There is no single correct way to set priorities. The great diversity of potential assessment topics, the urgency of some policymaking needs, and other factors may diminish the practical benefits of using highly systematic and quantitative approaches. On the other hand, ad hoc, inconsistent, or non-transparent processes are subject to challenges and skepticism of policymakers and other observers who are affected by HTA findings. Certainly, there is a gap between theory and application of priority setting. Many of the priority setting models are designed to support resource allocation that maximizes health gains, i.e., identify health interventions which, if properly assessed and appropriately used, could result in substantial health improvements at reasonable costs. However, some potential weaknesses of these approaches are that they tend to set priorities among interventions rather than the assessments that should be conducted, do not address priority setting in the context of a research portfolio, and do not adopt an incremental perspective (i.e., consideration of the net difference that conducting an assessment might accomplish) (Sassi 2003).

Reviewing the process by which an assessment program sets its priorities, including the implicit and explicit criteria it uses in determining whether or not to undertake an assessment, can help to ensure that the HTA program is fulfilling its purposes effectively and efficiently.