In this post, I’m going to tell you five ways in which evaluation questions can help evaluators.
Like ship
captains of yore, program evaluators rely on the stars to get where they’re
going. Well, not stars, exactly, but a few key questions. Both serve pretty
much the same purpose — they help you navigate, help you to
get where you’re going. In an ocean of data, where you can find yourself
submersed in too many choices, these key questions, commonly referred to as
“evaluation questions,” are your lodestars.
Good evaluation
questions will guide you in making decisions, ensuring that you are heading in
the right direction. After over 15 years of conducting evaluations, this has
become a truism I swear by. Now, to see how far we can stretch this metaphor
before it snaps, imagine that the ship, the crew and the navigational
instruments represent the resources and methods the evaluation team has to work
with. The evaluation questions are what guide the team.
Who determines the evaluation questions?
With most
evaluations, evaluators are hired to address a pre-determined set of queries.
These are normally provided by the client and embody what the client wants to
know about the program, the intervention, the project, the policy, or whatever needs
examining.
When the
work is done, the analysis conducted and report is submitted, what people will
want to know is “What are the answers to the questions we gave you?” Even if
you don’t like the questions, you need to find a way of answering them.
If the
client hasn’t developed the evaluation questions already, then the evaluator
can propose them, based on the client’s objectives. Sometimes the questions are
not clearly thought-out, or maybe they are difficult to answer. That applies
particularly to questions regarding a program’s sustainability. How can you
answer that in circumstances when the program is far from completed?
Evaluation questions are not the same
as interview questions
Evaluation
questions are not the same as interview
questions, which are what evaluators use when interviewing people, such as
beneficiaries, key informants, program implementers and so on. Interview questions, for example, might be
those asked by a police officer investigating a murder. The police might ask the suspect: Who is the
murderer? Why did he do it? And, Where is the weapon? Evaluators don’t ask the people involved in,
or benefiting from, the agriculture project, “Was the project effective?”
Interview questions are more specific, a way of collecting multiple data points
which will inform the body of evidence.
In
contracts, evaluation questions tend to be more along the lines of: Are
stakeholders satisfied with the program? Is it sustainable? How effective is it
at achieving its objectives?
Nevertheless,
for both cops and evaluators, it comes down to asking the right questions in
order to collect the evidence they need.
Questions should drill down from the
evaluation objectives
The evaluation
questions should not only embody the evaluation
objectives, they should drill down into those objectives. They need to be specific.
Evaluation
objectives can be broad, and open-ended. They are useful for explaining why the
evaluation needs to be conducted, but not as useful for developing a
methodology. For example, if the evaluation objectives are “To assess the
project’s effectiveness” or “To draw lessons about the project,” evaluation
questions should be much more specific. They should ask, for example, “Are
women farmers using the new technology as intended?” or “Do stakeholders
consider the hands-on technical assistance they receive to be effective?” However
the questions are formulated, evaluators almost always have an opportunity to
review them and propose modifications. I recommend doing this as early as
possible in the process.
How do evaluation questions help the
evaluator?
There is
plenty of material out there providing guidance on developing and selecting
good evaluation questions. That is not the subject of this post. Instead, I’d
like to point out that there are multiple
ways in which questions, once decided upon, can be extremely useful to the evaluator.
Evaluation questions
are crucial to keeping you on track and staying relevant to your topic. Sometimes
there is a temptation, while in the field, to go off on a tangent. For example,
the substantive or technical aspects of a project are often very interesting in
and of themselves. You or your colleagues may get caught up in discussions on different
types of irrigation water pumps, the political roots of the disparities between
northern and southern regions, or some other issue. That’s all good to know for
context, but such lines of inquiry shouldn’t distract from your main purpose.
That’s not what is being asked of the evaluator.
Regardless
of how the questions are generated, once agreed upon, they become your guide, serving
your effort in a number of valuable ways. The questions should help you with these
five aspects of your evaluation:
What issues to focus on
What evaluation methods to use
Where and whom
to collect the data from
What interview questions to ask people
How to structure and draft the final report
So, keep the
questions close at hand, and check in with them regularly. Use them to guide
you and help make decisions.
Conducting an evaluation is far from being a gentle
boat ride down the river. (If it were that boring, I would have bailed out long
ago). No, it is often difficult, sometimes treacherous, and (predictably) full
of uncertainties. Almost inevitably there is someone or something — on the client side, among the program
stakeholders, or even on your team — that
will make your life a challenge. Don’t let that distract you. Factor those
challenges into your work. With your evaluation
questions to help you navigate, you’ll know where to set your course and be
able to focus from there.
Regardless of what you think of politicians who love to speak in these terms, it is fair to say that the world is full of winners and losers. Government policy may be active, e.g. raising tariffs or taxes to improve its fiscal situation, or it may be passive, e.g. standing by while the forces of globalization and technology sweep jobs from an economy. In either case, some people benefit, some don’t, and some lose out. In fact, just about every benefit comes with a cost, but those costs are not evenly distributed.
Despite great progress in many sectors of the economy, inequality has increased in the US and elsewhere. According to the Pew Research Center, while median real wages have barely budged since 1970, most of the gains since 2000 have gone to the top 10 percent of the population.
How do we know what the impacts will be?
It is extremely difficult to predict the exact impacts of a given policy. Many factors come into play, and there are direct and indirect impacts. However, you can make estimates, and that can be very useful. A big part of my work involves looking at the potential impacts of policy reforms in developing countries. Years ago, the World Bank developed an approach called Poverty and Social Impact Analysis , or PSIA. This is a type of evaluation. It deploys quantitative and qualitative research methods to muck around in people’s financial and economic cupboards (using surveys and focus groups, for example) to try and figure out how they might fare if the government enacts a new policy.
How does this work in practice? I often conduct studies on the effects of price increases on different population groups. Analysis allows us to predict what the effects will be in terms of affordability for different income groups. Quite simply, we assess the winners and losers of a given reform. If electricity rates go up 25%, but you also get more hours of electricity supply per day, what does that mean for you, as a poor household? The risk is that the losers, in the short-term, will be those for whom electricity bills are a serious chunk of their total expenditures.
The agony of asking others to delay their gratification
Of course, reforms typically aim for positive outcomes. But these don’t kick in immediately. In the short-term, most people may experience only costs. The long-term benefits are abstract and perhaps uncertain. This is especially the case if the government has a poor track record of following through on its plans.
The relative costs of a price hike will be higher for those least able to afford the good or service being reformed. Thus, raising water or electricity rates can be a delicate exercise. It can make people quite unhappy, and politicians quite nervous. But those tariffs do need to rise. Otherwise, what will ensure that a utility company can operate sustainably, invest in operations and maintenance, attract external financing, and expand its network?
Tariffs are not boring if you’re poor
If you think utility rates is a boring topic, think again. Such an attitude suggests you are comfortably middle class. Probably your eyes glaze over when you see news reports from some faraway country about the latest unrest over price rises. At most, you may be mildly irritated when your utility bill goes up.
In fact, it is not unusual in some countries that when prices rise, some people pay with their lives. Demonstrations and riots occur on a regular basis. People protest against government attempts to squeeze more money out of people (the demonstrator’s perspective). The argument that it will put a utility on a sustainable footing (the technocrat and economist’s perspective) is not accepted. Especially when the government is perceived as corrupt and unaccountable.
Demonstrations over prices can block policies…and bring down governments
Just last month, violent demonstrations broke out in Cote d’Ivoire against proposed electricity price increases. One person was shot dead by security forces. In Nigeria in 2012 cuts to fuel subsidies led to widespread, violent protests leading to two deaths and many injured. In Bulgaria in 2013 in response to a doubling of electricity rates, six people immolated themselves, and street protests brought down the government. In 2015, Armenians took to the streets to protest proposed electricity price hikes, and dozens were injured. There are many other such cases throughout the world.
The Cochabamba demonstrations in Bolivia, dramatized in the 2010 feature film Even the Rain starring Gael Garcia Bernal, is a great primer on this, although it shows just one side of the story: the poor protesting against the privatization of the city’s water supplier. (It’s also an excellent and evocative film which draws a parallel between oppression by elite outsiders over 500 years of history.) The other side of the story, not covered in the film, concerns the aftermath of the multinational water company’s departure. Sixteen years on, the cost of water has indeed stayed low, but most of the poor still don’t have access to it. In other words, it was Pyrrhic victory for the protesters
Put some effort into finding a way forward
Yet it doesn’t have to be either/or. Yes, tariffs do have to cover costs or the infrastructure will break down and a lot of people will be left without any water at all. But there is more than one way to skin a cat. As a government, you can raise prices – or reduce subsidies, effectively the same thing for the consumer – gradually. You can include protection measures for the poor. You can engage on the problem with people affected. You can be smart about communicating why you have to do this.
In 2003, I was part of a team that studied the impact on the poor of rising electricity rates in Moldova. We found that the poorest 20% of households were not cutting back on electricity use, but were even consuming more. This was because the poverty rate had been coming down, so people in general had more money in their pockets to spend. In another study, in Lebanon, we found that the real (inflation-adjusted) cost of public electricity tariffs had been falling for years, but reliance on expensive private generation has pushed up household electricity expenditures.
The type of approach mentioned above – poverty and social impact analysis – can be usefully applied to all sorts of different reforms, not just utility tariffs. By doing some investigating, and carefully analyzing the data, the road to reforms can be smoothed. Social unrest can be avoided.
Government wins by generating goodwill among the population. It shows cares about them and is trying to work out a solution. And the people win because their needs and circumstances are being taken into account, and painful up-front costs are being reduced.
