Weight Measures

Weighting refers to emphasizing some measures more than others when you combine individual results to create summary scores. It is not an issue if you are only reporting results for individual measures. In that case, you are essentially asking consumers to weight the measures themselves, which they may do by focusing on some pieces of information more than or to the exclusion of others.

This section offers the following information:

• How Weights Are Derived.
• An Example of Weighting.
• Why Use Unequal Weights?
• Why Weight Measures Equally?

How Weights Are Derived

You can use any of four approaches to determine how to weight measures within a category:

1. Expert judgment. Ask experts to assign weights based on their assessments of the relative importance of whatever the measure reflects. To determine importance, experts may consider the effect on outcomes or look at what the literature says about consumers and their concerns.

2. Consumer judgment. Ask consumers to assign weights that reflect their preferences. You can use a survey to collect information of this kind. An alternative is to let individuals create their own weighting schemes for each category through a Web-based decision support tool.

3. Population impact. Assign weights based on the number of people for whom the measure may be relevant. For example, many more people are affected by flu shots than by diabetic retinal exams.

4. Factor weighting. Use the empirically derived weights that result from a factor analysis, which looks at how each measure correlates with the summary score and with each other. (According to an analysis by RAND, the results of this statistical approach are only moderately interpretable, so opinion-based weights may be more reliable.)

An Example of Weighting

Let's say a category called "Staying Healthy" is composed of four measures. Rather than give each measure equal weight (25 percent each), the sponsor could decide to give different weights to different measures:

Select to access Text Version.

By letting the breast cancer screening rate have more impact than the other measures on the summary score, the sponsor is essentially saying that this rate is a more important element of the category "Staying Healthy." When the sponsor calculates a summary score, this weighting system would magnify any differences among plans in the last two rates, while downplaying differences in the first two rates.

Why Use Unequal Weights?

By weighting the measures unequally when you combine them, you are basically imposing your view of what is most important on your audience. This is one way of helping consumers interpret the information you are providing without having to do their own analysis of what's important. Since sponsors tend to be more knowledgeable about quality measures (that is, what they represent and how reliably they reflect an organization's quality), you may be in the best position to make those value judgments on behalf of your audience.

You may want to take this approach if you strongly believe that some measures are more important or more reliable indicators of a health care organization's overall quality. For example, sponsors may give greater weight to measures with the following characteristics:

• Measures that are relevant to a large proportion of the audience.
• Measures that represent processes that are closely related to improved health.
• Measures that address the avoidance of illness or death.
• Measures that show the greatest differences among health care organizations.
• Measures that have the most precise measurement.

You may also trust some measures more than others, especially those measures that are familiar to sponsors and health care organizations. Results for new measures often reflect problems with data collection on the part of plans or providers more so than problems with quality. (In fact, to give health plans a chance to gain experience with collecting new indicators, the NCQA's policy is not to report measures to the public in their first year.) Weighting offers a way to include new measures in a category while still giving more prominence to older, well-tested measures.

Also, if you give equal weights to each measure, you may be assigning some of them a level of importance that is not really appropriate. For example, some sponsors believe that measures that are only relevant to a subset of the population do not say as much about health plan quality as measures that pertain to everyone that uses the health care system.

Why Weight Measures Equally?

Most methodologies for calculating summary scores apply the same weight to all measures in a category; for instance, all items in each CAHPS® composite are weighted equally. There are several arguments for this approach:

• Weighting is not an evidence-based science but a subjective decision. Researchers cannot say with any certainty that a particular weighting model is more valid than others. As a result, equal weighting is the default choice.
• Consumers assume that the measures in a category are weighted equally. If you impose uneven weights, you face the question of whether and how to explain what you have done, and whether this explanation will be sufficient. Most sponsors do not offer an explanation, so weighting is a "black box" to consumers. This means their assumptions are not consistent with the reality of the information they are seeing.
• For some categories, e.g., one that captures consumers' assessments of access, sponsors may not have any basis for determining that some measures should be weighted more heavily than others.

Return to Document