Beneficiary assessments: Questions, questions, questions

This blog post about beneficiaries is built around a series of questions. Why? Because whether you are a program designer, a program implementer or an evaluator, you spend much of your professional life trying to find answers to them.

What policymakers want to know

Policies are generally designed with the aim of allocating resources or creating opportunities to one or more segments of a given population.  A question policymakers often ask themselves is: Who will benefit if…?

Addressing issues related to program beneficiaries depends a lot on how the questions are framed. Typical questions might be: Who will benefit from a new policy? Who is benefiting from the status quo? What are my policy options? or What will be the effect of choosing policy A over policy B?

Existing conditions, and who is benefiting from them, are themselves influenced by prior policy decisions. (And don’t forget that not implementing a policy change is a policy in itself.)

Policy choices and questions like those above also concern actors in the foreign aid / development assistance sector. Governments in rich countries need to be able justify their assistance programs to their taxpayers. They want to be able to justify their foreign aid spending. One important way they do this is by demonstrating that their foreign aid program is making a positive difference in people’s lives.

Policymakers in recipient countries want to exert a positive impact for some or all of their constituents. And they generally want to be rewarded for it, for example by getting re-elected, or their bosses getting re-elected.

Design, deliver and evaluate

The task of program designers is to demonstrate how investments will deliver. How will the money flowing into one end of the foreign aid funnel be transformed into better lives at the other end? This has to be figured out. It is a program logic issue.

The task of program implementers is to deliver on the program’s design, to realize the goals and reach the targets that are set out. Implementers address the question of How do I use the resources I have to produce the results they want? Think of it as a form of foreign aid alchemy. “The base metal” of financing, resources, plans etc. is, if all goes well, turned into a golden opportunity. This is a management issue.

The task of program evaluators is to figure out whether the desired benefits, in fact, came through. Evaluators apply a set of methods to determine What happened? Why, or why not? This is an analytical issue.

When a beneficiary isn’t a beneficiary

Some organizations prefer to avoid using the term “beneficiary” as it has connotations of passivity. Other terms include client, customer, end-user, program participants, affected groups or program-affected-populations. Each has a slightly different connotation. Because “beneficiary” is a catch-all term, I’ll stick with it in this post.

The case of roads

Let’s use investments in roads as an example. (I am currently involved in three evaluations of road projects for the Millennium Challenge Corporation, and so find myself thinking quite a bit about this issue.) While road networks connect virtually everyone nowadays, roads deteriorate over time. They need maintenance, resurfacing, and complete rehabilitation. 

Roads facilitate the movement of people and goods. Similar to roads, canals (mostly for cargo), railways (people and cargo), sewerage systems (wastewater), pipelines (oil and gas), power lines (electricity) telephone lines (communication), and broadband cables (information), improve the flow of things people need and value.

By reducing resistance, these various channels save time and energy compared to alternatives for moving things or people from point A to point B.  When these channels are cut, blocked, or destroyed, havoc can occur. Access is impeded. Very quickly, everything becomes more difficult and costly. 

The beneficiary perspective

From an individual’s viewpoint, the impact will depend on, for example, where that road leads, how often they use it, what they use it for, and how easy it is to reach.  Is there a paved feeder road, a gravel road, or a path? Is there a mountain or river that must first be crossed?

Many other questions can be asked. Are you a farmer who sells produce at a weekly market along the road?  Is your village close by and connected to the road via a well-maintained access road?

Maybe the new road won’t make any difference to you financially, but it saves you time, and improves your quality of life. It’s much more pleasant to drive along a smooth asphalt road at 80 km per hour than a bumpy one with perilous potholes. You might not see money in your pocket, but you will feel more connected to the outside world if you live near a good road instead of a bad one.

It’s also possible that you don’t see the effect today, but will see it years later. If your parent has a stroke five years from now, the new road could make a big difference in how quickly can you get them to the hospital.

The evaluator perspective

Now consider the issue from the perspective of a researcher or evaluator. You’re looking not at individuals but at aggregate effects. These are less precise, but more useful. You don’t want to collect five hundred stories of how the improved infrastructure has changed, or not changed, lives. You want to measure trends and aggregate changes. You want to know the effect of the new road on the population as whole.

And there are questions about distance: What if an old road with a rough surface and full of potholes is rebuilt? Are you a beneficiary if you are living in a town 2 km from the newly rehabilitated road? What about if you live 5 km or 10 km away? Or only people who use vehicles, i.e. not pedestrians?

In the past, it was common for road evaluations to pick a distance on either side of the road in question, draw two imaginary lines, and consider anyone within this band, the “corridor of influence” was a beneficiary. The distance of 5 km was considered too wide, and 2 km came to be considered a more reasonable distance. An alternative, and more conservative way of estimating road beneficiaries is to only survey people directly using the road. This ignores indirect beneficiaries, but road users are easier to count, and gives more confidence in the result.

The data you collect will depend very much on how you draw the line, who you choose to include among your potential beneficiaries, and what assumptions you make.

Analytical tools

The previous questions in this blog post are just the tip of the iceberg. When thinking about how people are affected, we can ask many, many more.  For example, here are questions about (potential) beneficiaries that evaluators will ask:

• What is our population of interest?
• How many beneficiaries are there?
• Where do they live?
• How many are direct / indirect beneficiaries?
• How are they benefiting?
• By how much are they benefiting, in relative and absolute terms?
• Among project-affected persons, how many are benefiting?
• Why are some people benefiting and not others?
• How are the benefits distributed among income groups? Among different stakeholders?
• Are some people taking advantage of the benefits more than others?
• Are some people dis-benefiting, i.e. negatively affected as a result of the project?
• What methods should we use?
• What are the critical information sources?
• Who should we to talk to?
• How should we talk to them?
• How many people should we survey?
• How big should the survey sample be?
• Where (what population) should the sample be taken from?
• What types of groups should be sampled?
• What questions should be asked?

Although beyond the scope of this blog post, common measurement tools for addressing questions about road project beneficiaries road are origin-destination (O-D) surveys and traffic counts. A range of approaches and methods exist for each.

Your decision on what approach to take will be influenced by resources at your disposal – money, time, expertise, etc. Your decision will also be guided by what others have done before you.

But, perhaps more than anything else, your findings will be influenced by the questions you ask. Assuming your methods are sound, the findings may vary, but none will be entirely wrong.