FAQs about program evaluation

Why conduct a program evaluation?

We are often asked to develop an evaluation that will permit program staff to decide whether to (1) offer the program again (or continue to offer) because it is successful, (b) alter the program in some way to make it more useful to participants, or (c) drop the program because it fails to meet a need. We are asked to help program staff consider what evidence would indicate the program is successful, what would mean that it should be improved, and what would show that it should be ended. In other words, the premise behind program evaluation is to discover the value and worth of something.

What is the difference between empirical research and program evaluation?

The main difference lies in the purpose of the two. Original research about the creation of new knowledge. Its purpose is testing theory and producing generalizable findings to make a valuable contribution to the field related to the investigation. Research is about gathering empirical evidence to prove a theory or hypothesis and to generate new understandings.   

By contrast, a program evaluation is an assessment of an activity, program, strategy, etc. Program evaluators gather information to help organizations improve their effectiveness, to assist administrators in making program-level decisions, and to enable interested parties to examine program effectiveness. The purpose of a program evaluation is to determine the effectiveness of a specific program to determine the value or usefulness of a program and to improve effectiveness. Evaluation is about drawing evaluative conclusions about quality, merit, or worth, essentially to improve the existing program for the target population.

What are the steps to a program evaluation?

Evaluations can be rather complicated and time consuming. A lot goes into each step in the evaluation, but a basic list is provided here:

1. The first step is to engage stakeholders (the people or organizations that are invested in the program, are interested in the results of the evaluation, and/or have a stake in what will be done with the results of the evaluation. In this step, we have discussions with stakeholders to bring clarity to the main outcomes, activities, and evaluation questions that guide the evaluation. Here, we discuss resources for conducting the evaluation and how the results of the evaluation will be used.
2. In step 2, we describe the program. Typically, we develop a logic model (a graphic description that presents the relationship of resources, activities, outputs, and outcomes/impacts for the program). It depicts the relationship between the program’s activities and its intended effects—“If I do this activity, then I expect this outcome.” 
3. Next, we focus the evaluation. Using the logic model, we would think through the purpose of the evaluation, who will employ the evaluation findings, and identify specific components to be evaluated (activities, inputs, outcomes, etc.), describe the evaluation questions, determine the research design and data collection procedures that best match the evaluation questions, and we draft the evaluation plan.
4. Next, we gather credible evidence. We strive to collect information that will convey a well-rounded picture of the program. Using multiple procedures for gathering, analyzing, and interpreting the data is a crucial step. We analyze and interpret the data, draw conclusions, and make recommendations in this step.
5. Finally, we engage stakeholders by sharing our findings and lessons learned. This typically comes in the form of a written report shared with key organizational personnel (Program Director, Vice President, Program Coordinator, etc.). After receiving feedback on the initial draft, the reports will be revised based on that feedback, and a final report is prepared for additional stakeholders. Most of the time a PowerPoint presentation of findings is shared with stakeholders, using graphs and other visual displays to convey important information.   

How much does an evaluation cost?

The cost of a program evaluation will realistically depend on several variables. However, a good gauge is to estimate 5-10%, sometimes as much as 15%, of the total program budget. This includes the value of the time it takes to plan an evaluation, collect the data, analyze and interpret the findings, prepare reports, and present findings to stakeholders. Travel expenses, communication tools, printing and administering surveys, conducting interviews and/or focus groups, and other supplies are included in the cost of an evaluation as well. 

Important questions for organizations to ask related to evaluation costs include deciding to hire an evaluator or do the evaluation yourself. In making that decision, an organization should start with asking questions such as, “Does someone on your staff have the expertise to conduct an evaluation that meets your needs? “Can you devote enough time to do the evaluation?” “Can you get funding specifically for an evaluation (some funders provide grants for this purpose)?”

Your turn:

What other questions, besides those listed here, does your organization have related to program evaluation?