Don’t navigate blind — let the evaluation questions guide you

Why do we need good evaluation questions?

In this post, I’m going to tell you five ways in which evaluation questions can help evaluators.  

Like ship captains of yore, program evaluators rely on the stars to get where they’re going. Well, not stars, exactly, but a few key questions. Both serve pretty much the same purpose — they help you navigate, help you to get where you’re going. In an ocean of data, where you can find yourself submersed in too many choices, these key questions, commonly referred to as “evaluation questions,” are your lodestars.

Good evaluation questions will guide you in making decisions, ensuring that you are heading in the right direction. After over 15 years of conducting evaluations, this has become a truism I swear by. Now, to see how far we can stretch this metaphor before it snaps, imagine that the ship, the crew and the navigational instruments represent the resources and methods the evaluation team has to work with. The evaluation questions are what guide the team.

Who determines the evaluation questions?

With most evaluations, evaluators are hired to address a pre-determined set of queries. These are normally provided by the client and embody what the client wants to know about the program, the intervention, the project, the policy, or whatever needs examining.

When the work is done, the analysis conducted and report is submitted, what people will want to know is “What are the answers to the questions we gave you?” Even if you don’t like the questions, you need to find a way of answering them.

If the client hasn’t developed the evaluation questions already, then the evaluator can propose them, based on the client’s objectives. Sometimes the questions are not clearly thought-out, or maybe they are difficult to answer. That applies particularly to questions regarding a program’s sustainability. How can you answer that in circumstances when the program is far from completed?

Evaluation questions are not the same as interview questions

Evaluation questions are not the same as interview questions, which are what evaluators use when interviewing people, such as beneficiaries, key informants, program implementers and so on.  Interview questions, for example, might be those asked by a police officer investigating a murder.  The police might ask the suspect: Who is the murderer? Why did he do it? And, Where is the weapon?  Evaluators don’t ask the people involved in, or benefiting from, the agriculture project, “Was the project effective?” Interview questions are more specific, a way of collecting multiple data points which will inform the body of evidence. 

In contracts, evaluation questions tend to be more along the lines of: Are stakeholders satisfied with the program? Is it sustainable? How effective is it at achieving its objectives?

Nevertheless, for both cops and evaluators, it comes down to asking the right questions in order to collect the evidence they need.

Questions should drill down from the evaluation objectives

The evaluation questions should not only embody the evaluation objectives, they should drill down into those objectives. They need to be specific.

Evaluation objectives can be broad, and open-ended. They are useful for explaining why the evaluation needs to be conducted, but not as useful for developing a methodology. For example, if the evaluation objectives are “To assess the project’s effectiveness” or “To draw lessons about the project,” evaluation questions should be much more specific. They should ask, for example, “Are women farmers using the new technology as intended?” or “Do stakeholders consider the hands-on technical assistance they receive to be effective?” However the questions are formulated, evaluators almost always have an opportunity to review them and propose modifications. I recommend doing this as early as possible in the process.

How do evaluation questions help the evaluator?

There is plenty of material out there providing guidance on developing and selecting good evaluation questions. That is not the subject of this post. Instead, I’d like to point out that there are multiple ways in which questions, once decided upon, can be extremely useful to the evaluator.

Evaluation questions are crucial to keeping you on track and staying relevant to your topic. Sometimes there is a temptation, while in the field, to go off on a tangent. For example, the substantive or technical aspects of a project are often very interesting in and of themselves. You or your colleagues may get caught up in discussions on different types of irrigation water pumps, the political roots of the disparities between northern and southern regions, or some other issue. That’s all good to know for context, but such lines of inquiry shouldn’t distract from your main purpose. That’s not what is being asked of the evaluator.  

Regardless of how the questions are generated, once agreed upon, they become your guide, serving your effort in a number of valuable ways. The questions should help you with these five aspects of your evaluation:

• What issues to focus on
• What evaluation methods to use
• Where and whom to collect the data from
• What interview questions to ask people
• How to structure and draft the final report

So, keep the questions close at hand, and check in with them regularly. Use them to guide you and help make decisions. Conducting an evaluation is far from being a gentle boat ride down the river. (If it were that boring, I would have bailed out long ago). No, it is often difficult, sometimes treacherous, and (predictably) full of uncertainties. Almost inevitably there is someone or something — on the client side, among the program stakeholders, or even on your team — that will make your life a challenge. Don’t let that distract you. Factor those challenges into your work.  With your evaluation questions to help you navigate, you’ll know where to set your course and be able to focus from there.