Evaluating the Results of Your Project

Once you have finalized and distributed your information on quality, how will you find out whether your audience saw the material, understood it, and used it in the way in which it was intended? More importantly, how can you say whether your project has been helpful to consumers, enabling them to make better health care decisions?

To answer these questions, you need to conduct some kind of evaluation after the information has been disseminated. This evaluation should be designed to let you know what worked, what didn't work, and what could be done differently the next time to create a better process and a better outcome.

This section discusses the following topics:

• Reasons for Doing an Evaluation.
• Planning an Evaluation.
• Choosing Your Approach.
• Sharing Your Findings.

Reasons for Doing an Evaluation

Everyone reaches conclusions about the relative success or failure of quality information projects. A systematic evaluation is a way to ensure that those conclusions are based on objective data. If done correctly, it can tell you whether your quality information project is on track and how well it worked.

Essentially, an evaluation is a formal way of collecting and organizing information about the following aspects of your project:

• The process. (What did you do? How did each stage of the project go?)
• Its effect, particularly in the short term. (How well did you meet your objectives? Did you make progress towards longer-term goals? What results can you detect now?)
• Its efficiency. (What did the project cost? Which strategies were more cost-effective than others?)

This information is critical to the long-term success (and future funding) of your initiative, because you can use what you learn for many purposes:

• To improve current and future efforts.
• To justify a mid-course adjustment.
• To certify the degree of change that has occurred.
• To reassess or revise your goals.
• To identify programs or elements of programs that aren't working.
• To identify areas where additional effort is needed (e.g., with some subsets of the larger population).
• To identify particularly effective strategies or activities that you could expand.
• To compare costs and results of different activities.
• To reaffirm support for the program.
• To support a decision not to continue with an ineffective program.

Also, in addition to documenting the direct benefits of this project, an evaluation may offer indirect benefits by giving you ideas for improving other things you may be doing.

Planning an Evaluation

Early planning of your evaluation will spur you to specify what kind of feedback or evidence you will need to learn how well your materials are working. For each task, objective, and goal, make a note of what data you will need to collect, where you can get that data, who will get the data, by when, and how. If there is no way to measure what you are doing or whether you are moving towards your goals, you may want to revise or expand upon your project plan. 

In addition to helping you determine what you'll need to collect at the end of the project, up-front planning can also allow you to identify any data that you may want to collect right away or as you proceed with the project. For example, if your goal is to increase your audience's awareness of differences in quality among health plans, you may want to survey them to find out how aware they are before you distribute information comparing quality. When you're thinking about what data to track, keep in mind that the things that are easiest to count are not necessarily the most informative. For instance, the number of reports mailed out to enrollees doesn't tell you whether they read it, understood it, or used it.

One of the most important reasons to incorporate the evaluation step into your initial plan is to ensure that funding and resources are available when the time comes. Many sponsors are unable to evaluate their projects simply because they failed to set aside funding for this purpose. But this does not mean that an evaluation is expensive. There are many ways to evaluate a quality information project, so there is always something you can do that will fit your budget.

Choosing Your Approach

After the implementation stage of your project—that is, after everything you have planned is operational—you (as well as other interested parties) will have many questions about its effectiveness. You can sort these questions into three different categories:

• Process.
• Short-Term Outcomes.
• Long-Term Impact.

Your evaluation may focus on any of these concerns, alone or in combination.

Evaluate the Process

In an evaluation of the process, you are trying to assess how and how well the implementation of your project is working. An evaluation of the process is important because it can help you figure out why something worked (or didn't work). For example, you may find that your audience isn't aware of your information because of a breakdown in your distribution system. This happened a few years ago in Colorado, when a business coalition learned that some insurance brokers and employers were not giving out its quality reports to employees.

Basic process measures can also offer you tangible evidence of progress. For instance, if you keep close track of your inventory as well as the distribution process, you'll know for sure that people who need the information are getting it when they are supposed to get it. If you pay attention to the media coverage and other publicity you get, you'll get a sense of how successfully you've conveyed your message.

Here are some examples of questions you may have about the process:

• Were the materials tested with different segments within your audience?
• Were materials tested with secondary audiences?
• Did those responsible for content and design use feedback from testing to help shape the materials?
• What percentage of the targeted population were exposed to your message about quality? How often did they receive it?
• What percentage of the targeted audience got your quality information?
• Did the local media pay attention to and help publicize the report?
• Are you distributing more or less reports (or getting more or less hits on your Web site) than you had projected? If you printed reports, has the demand outstripped your supply?
• How many people are attending meetings where information is being distributed and explained?
• Are intermediaries aware of your report? Are they getting it?
• Are intermediaries using the report with your intended audience?
• Did people respond to the information in some way? For example, did they call for additional information or with questions?
• What kinds of questions do people have about your information?
• Are you, the sponsor, effectively promoting the use of your information for quality improvement purposes?

How to Assess the Process

To judge the implementation of your project, you want to look critically at the procedures and activities you've put in place. Is everything happening as it's supposed to? Is the system functioning properly? How does the reality compare to your expectations?

For many process questions, the best approach is to go directly to your audience and key informants (such as information intermediaries). Both of these groups can offer insights into what worked well and what didn't. They can also suggest ways to improve the materials as well as the process you used to get their feedback initially. Focus groups and surveys are the most common methods for obtaining this kind of input. For example, surveys can help you track how many people are seeing your information or whether intermediaries are using the information. A focus group with intermediaries can help you get more details about their positive and negative experiences explaining and distributing the materials.

For some process issues, it is useful to develop a mechanism that will allow you to quickly and easily monitor how things are going. For example, you could give intermediaries a simple check-off form so that they can keep track of the kinds of questions and concerns they hear from your audience. This will enable you to correct minor problems before they become big ones.

Evaluate Short-Term Outcomes

Ideally, you would want to assess the short-, medium-, and long-term effects of your quality report on behavior, knowledge, attitudes, beliefs, and health coverage decisions. But for nearly all report card projects, it is most practical and feasible to focus on the immediate effects of the project on the audience. This information provides valuable feedback that you can use to make the project more effective over time. In addition, a focus on immediate or intermediate effects—such as a gain in knowledge, a stronger sense of salience, or a better informed dialogue among patients, providers, plans, and purchasers—can help you identify useful measures of success.

The first step in evaluating short-term outcomes is to think about what's most likely. What do you expect will happen as a result of your information? Most sponsors focus on the endpoint; they want to know whether their information affected the choices that people made. But that degree of change takes time and can be hard to detect. You need to be realistic about what is possible in the short term. It's more reasonable to focus on proximal, or interim, outcomes by collecting data on the following:

• Changes in awareness, knowledge, and attitude: Is your audience aware that information on quality is available? Do they understand what it says (e.g., that there are differences in performance)? Do the understand the differences between their choices? Do they believe that this information is important for them and their families?
• Changes in the interests and stated intentions of your audience: Are people demonstrating an increased interest in quality issues (perhaps by calling their employers' human resources staff or by requesting additional information)? Do they say they would want to use this information when they make decisions about health care coverage or services?
• Changes in short-term or intermediate behavior: Do they ask their employers about the quality of available health plans? Do they find it easier to make decisions? Do they have more confidence in their decisions? Do they use the information to make or confirm their choices?
• New policies: Do intermediaries make a point of incorporating information on quality into other information they share with your audience? Do employee organizations (such as unions) request information on quality? Are health plans or providers responding in some way to the data, particularly by investigating problem areas and making efforts to improve care and services? Do employers consider the information when developing their purchasing strategies?

When you look for the answers to these questions, don't forget that information on quality is only one factor that influences behaviors and decisions. For instance, the success of a project does not depend on whether people use the information on quality to make a change in their plans or providers. You wouldn't expect such changes if your audience is happy where they are.

How to Assess Outcomes

Sponsors typically gather information on short-term outcomes by looking at awareness, attitudes, and behaviors before and after they distribute the quality information. There are two ways to do this:

• Self-reporting: You can get self-reported information by surveying or interviewing the audience. This is the easiest way to get feedback.

• Observation: Observation is useful because most people cannot report on how information may have affected their behavior. For example, they may not be able to say how they weighted different factors, but you could infer their weights by observing them as they consider different pieces of information.

When you seek information on short-term outcomes, be sure to gather data on the demographics of your respondents. You may find that your report has a different impact on people from different walks of life. Also, keep in mind that even short-term effects take time to reveal themselves, so you may need to provide quality information for several years before you detect any changes. 

Although it is common to focus on your audience for this kind of information, you can also interview or survey key informants and others who interact regularly with consumers, such as benefits managers. They can tell you what kinds of issues and questions are raised by the population they deal with. If you pursue this route, one-on-one interviews are a good technique for getting detailed feedback

Evaluate Long-Term Impact

Everyone who develops health care quality reports hopes that they will have a long-term impact on consumer behavior and the quality of care we all receive. However, this kind of effect is very hard to measure. A report card—like any other communications program—is only one of many factors that influences health care attitudes, behaviors, and decisions. Separating the effects of that one factor from those of all the others is a formidable task.

For a report card, measures of long-term impact might include:

• The number of health plans and providers that actually improve their performance.
• The proportion of consumers who perceive the improvements in performance.
• The proportion of consumers reporting that the information is having a significant influence on coverage decisions.
• The proportion of consumers who are satisfied that they have sufficient information to make a good decision.
• The proportion of consumers who are reporting greater confidence in their health care-related decisions.
• Changes in consumers' level of interest in health care quality issues.
• Changes in enrollment patterns that reflect a preference for higher value (i.e., consumers are choosing the best quality plan at a given price point, and the lowest cost plan among those with a given level of quality).

In addition to talking directly with consumers to learn about some of these outcomes (such as confidence and satisfaction), you can also use observation to identify longer-term changes in behavior and decisions. For example, you could observe greater interest in quality information by tracking requests for information or possibly media coverage. You could also observe how performance affects health plan enrollment. For details, go to Other Sources of Data.

Unfortunately, this is a tricky way to assess report cards because many factors could influence decisions to enroll or disenroll from a plan. One way to get around this problem is by having a control group (i.e., a group that does not receive the report card); by comparing your audience's actions to those of the control group, you can factor out any other events or circumstances that may affect their behavior or decisions. However, since most sponsors are reluctant to (or simply cannot) limit distribution of their information to certain people, this approach is difficult to implement.

Sharing Your Findings

Sharing Information Internally

Your initial audience for the findings of the evaluation should be the people who worked on the project. In addition to the staff that was primarily responsible for the work, you would also want to include members of any planning committees, human resources staff, and your partners in the project (including health plan and provider representatives). Rather than simply distribute an evaluation report, draw attention to the key findings by producing an executive summary and providing internal briefings that review the lessons learned and the implications for what you would do differently for the next report.

Sharing Information Externally

In the same way that this Web site allows you to benefit from what others have done, your experience may be invaluable to the next "generation" of quality report sponsors. Possible ways to share your results include letters to journals or magazines, poster presentations, and presentations to peers at conferences (such as CAHPS® user group meetings).

Select for more ideas for Places to Share Your Findings.

Why Bother with Documentation?

Reports on the findings of your evaluations will help you make critical decisions:

• Was the project successful? If so, what made it effective? If not, what went wrong?
• Should it be continued?
• What changes or improvements to the report card would you recommend? What insights can you apply to the next round? To other projects?
• Who should be responsible for making those changes?
• What other recommendations would you make based on your experience?

Also, you will be able to use this documentation for several purposes:

• To justify your project with your managers and/or partners.
• To provide evidence that the project requires additional funds or other resources.
• To increase institutional understanding of and support for the project.
• To encourage ongoing cooperative ventures with other organizations (i.e., to attract other partners).