In this post, I’m going to tell you five ways in which evaluation questions can help evaluators.
Like ship
captains of yore, program evaluators rely on the stars to get where they’re
going. Well, not stars, exactly, but a few key questions. Both serve pretty
much the same purpose — they help you navigate, help you to
get where you’re going. In an ocean of data, where you can find yourself
submersed in too many choices, these key questions, commonly referred to as
“evaluation questions,” are your lodestars.
Good evaluation
questions will guide you in making decisions, ensuring that you are heading in
the right direction. After over 15 years of conducting evaluations, this has
become a truism I swear by. Now, to see how far we can stretch this metaphor
before it snaps, imagine that the ship, the crew and the navigational
instruments represent the resources and methods the evaluation team has to work
with. The evaluation questions are what guide the team.
Who determines the evaluation questions?
With most
evaluations, evaluators are hired to address a pre-determined set of queries.
These are normally provided by the client and embody what the client wants to
know about the program, the intervention, the project, the policy, or whatever needs
examining.
When the
work is done, the analysis conducted and report is submitted, what people will
want to know is “What are the answers to the questions we gave you?” Even if
you don’t like the questions, you need to find a way of answering them.
If the
client hasn’t developed the evaluation questions already, then the evaluator
can propose them, based on the client’s objectives. Sometimes the questions are
not clearly thought-out, or maybe they are difficult to answer. That applies
particularly to questions regarding a program’s sustainability. How can you
answer that in circumstances when the program is far from completed?
Evaluation questions are not the same
as interview questions
Evaluation
questions are not the same as interview
questions, which are what evaluators use when interviewing people, such as
beneficiaries, key informants, program implementers and so on. Interview questions, for example, might be
those asked by a police officer investigating a murder. The police might ask the suspect: Who is the
murderer? Why did he do it? And, Where is the weapon? Evaluators don’t ask the people involved in,
or benefiting from, the agriculture project, “Was the project effective?”
Interview questions are more specific, a way of collecting multiple data points
which will inform the body of evidence.
In
contracts, evaluation questions tend to be more along the lines of: Are
stakeholders satisfied with the program? Is it sustainable? How effective is it
at achieving its objectives?
Nevertheless,
for both cops and evaluators, it comes down to asking the right questions in
order to collect the evidence they need.
Questions should drill down from the
evaluation objectives
The evaluation
questions should not only embody the evaluation
objectives, they should drill down into those objectives. They need to be specific.
Evaluation
objectives can be broad, and open-ended. They are useful for explaining why the
evaluation needs to be conducted, but not as useful for developing a
methodology. For example, if the evaluation objectives are “To assess the
project’s effectiveness” or “To draw lessons about the project,” evaluation
questions should be much more specific. They should ask, for example, “Are
women farmers using the new technology as intended?” or “Do stakeholders
consider the hands-on technical assistance they receive to be effective?” However
the questions are formulated, evaluators almost always have an opportunity to
review them and propose modifications. I recommend doing this as early as
possible in the process.
How do evaluation questions help the
evaluator?
There is
plenty of material out there providing guidance on developing and selecting
good evaluation questions. That is not the subject of this post. Instead, I’d
like to point out that there are multiple
ways in which questions, once decided upon, can be extremely useful to the evaluator.
Evaluation questions
are crucial to keeping you on track and staying relevant to your topic. Sometimes
there is a temptation, while in the field, to go off on a tangent. For example,
the substantive or technical aspects of a project are often very interesting in
and of themselves. You or your colleagues may get caught up in discussions on different
types of irrigation water pumps, the political roots of the disparities between
northern and southern regions, or some other issue. That’s all good to know for
context, but such lines of inquiry shouldn’t distract from your main purpose.
That’s not what is being asked of the evaluator.
Regardless
of how the questions are generated, once agreed upon, they become your guide, serving
your effort in a number of valuable ways. The questions should help you with these
five aspects of your evaluation:
What issues to focus on
What evaluation methods to use
Where and whom
to collect the data from
What interview questions to ask people
How to structure and draft the final report
So, keep the
questions close at hand, and check in with them regularly. Use them to guide
you and help make decisions.
Conducting an evaluation is far from being a gentle
boat ride down the river. (If it were that boring, I would have bailed out long
ago). No, it is often difficult, sometimes treacherous, and (predictably) full
of uncertainties. Almost inevitably there is someone or something — on the client side, among the program
stakeholders, or even on your team — that
will make your life a challenge. Don’t let that distract you. Factor those
challenges into your work. With your evaluation
questions to help you navigate, you’ll know where to set your course and be
able to focus from there.
The year 2018 is over (thank goodness) and we have a chance
for a fresh start. For many of us that means time for personal stocktaking.
What did you accomplish last year? What did you learn? How can you apply those
hard-won lessons to the coming year? Should you keep striving to outdo yourself,
or should you settle for what you’ve got and ease into the comfort of routine?
Rifling through the mental files in my “2018 evaluations” folder,
I’ve come up with a few of my own lessons. The one I’ll share today is this: One
thing you can count on is that you can’t always count on people. And you need to prepare for that.
As I’ve observed in an earlier post, we live in a world
where professional failure is more common than conventional wisdom would allow.
Failure is also less interesting than is portrayed by the media and in the
self-help industry. It can become a serious headache, however, when it is your
fellow team member who is doing the failing. I can use myself as a prime
example: I don’t always live up to my own professional expectations. It won’t
come as a shock to readers that people are not always up to the task. The
question is, how do you handle it?
First, let’s get a few obvious things out of the way. Humans
are complex, multi-faceted and, not infrequently, multi-talented. This is a
marvelous thing, accounting for some truly astounding cultural, engineering,
and intellectual feats that have enriched life on this planet. Indeed, in many professions,
it is assumed that employees bring multiple talents to the table. We are not like
robots, programmed to do only one or two tasks at a time. This truism applies
very much to the evaluation field, where evaluators are called upon to deploy a
range of both soft and hard skills.
The fun starts when you suddenly discover that key talents are
missing from a team member. While it is rare that a new team member is brilliant
across the board, most bring at least basic levels of competence to the table.
Most score at least a seven on a 10-point scale across the range of necessary
competencies. But every now and then, someone doesn’t. They’re a “one” or a “two”
in some important area. That’s the thing with being human. We may be
multi-talented, or at least multi-capable, but we also come with built-in
limitations, which sometimes leads to a giant team-implosion. Oops!
What competencies are we talking about? I would offer that, in the evaluation field, you must be able to:
communicate comfortably with others;
put together words, sentences and paragraphs in
a clear and logical manner;
analyze the information you have collected;
collaborate with others like a mature and
responsible adult;
be pleasant and respectful;
do what you say you will do; and
manage your time and priorities.
On top of these soft, but necessary skills, you may also be expected to be equipped with technical skills and experience in:
the sector being evaluated, i.e. agriculture, education, environment, gender, etc.;
qualitative or quantitative evaluation methods; and, if applicable;
effectively leading a team.
Nothing listed above is rocket science, that particular
field generally not falling within the scope of international development
projects. You still find yourself surprised, however, when a fellow team member
is —
how to put this delicately? — totally incompetent.
Of course, the safest solution is to only work with people you
have worked with before, and whom you can count on. For individual consultants,
however, that is a luxury. Instead, what is more typical is that you join a new
team on almost every new assignment. Every year, for example, I end up working
on maybe half a dozen different teams, the majority of which are composed of
folks I have never laid eyes on. On the one hand, it’s a great way to meet
people, make new friends, and learn from your peers. On the other hand, you can
end up in some frustrating and stressful scenarios.
I’ve had experiences where it soon became obvious that a team
member had pretty serious deficiencies in the interpersonal skills department.
For example, Team leader Mr. A, a very plausible stand-in for Ricky Gervais in
the TV comedy series The Office, would
spend the first 10 minutes of a meeting boasting about his own experience and
often end the meeting by insulting the people on the other side of the table. Other
times you get a bad case of weak ethics and poor writing skills, as with Dr. B,
a native English speaker, who couldn’t write proper English despite her
academic pedigree. When I came across passages that were surprisingly
well-written, a quick check on Google revealed she had been happily
plagiarizing them. (Always good to find out that kind of thing sooner rather
than later.) Or someone might impress you in person, but not on paper. Local team
member Ms. C knew the sector and country very well and asked the right questions
during stakeholder interviews, but couldn’t string two sentences together in a
logical way in a report. These were all setbacks which it fell to me to remedy,
through many hours — and sometimes days — of extra work.
I have to admit that I only had the pleasure of working with
one of these people in 2018; I’d worked with the others before that. But it was
last year that it finally hit home: I needed a coping strategy for the next
time this happened.
So, what to do on occasions when capabilities are missing? For
starters, if the shortcomings are yours,
it’s a good idea to reflect and take concrete actions to perform better. If the
shortcomings belong to others,
cursing under your breath or venting to your significant other can have a wonderfully
calming effect, but may not be enough to rectify the situation. Is it possible
to overcome such defects through mentoring or teaching? Unfortunately, I have found
that it is totally unrealistic to attempt to build the capacity of someone (even
if you are in a position to do so), over the course of a single assignment. In
any case, you’d first need to spell out their failings to them. That could be
pretty awkward, right? Furthermore, you don’t really have much time for
capacity building — you need to get the bloody job done.
What you need is a back-up plan, especially if you are
ultimately responsible for the work (if you’re the team leader) or because you
were asked to pick up the slack (by the team leader). Here are three
suggestions:
Build in a
time buffer: Provide enough slack in your schedule to take into account the
extra time that you might need to address the shortfall. For example, if you
think a task will take two weeks, try to allocate three weeks.
Build in a
human resource buffer. Identify persons, either on the team or not, who could
step into the breach. Maybe the organization that put the team together (if you
are subcontracted) has the resources to bring on extra help.
Build in a
mental buffer: Prepare yourself not be surprised or upset when colleague X lets
you down. Unless you’ve worked with them before, and therefore know their
strengths and weaknesses, assume that people have a least one weakness, and
that it will impact the work at hand.
In a word, contingencies!
Let 2019 be a year of contingency planning. The plan comes
with its own reward: if you have a
decent contingency plan, you will end up with more time, energy, and even
inspiration, to focus on the interesting and fun stuff.
One country after another is exiting the 2018 World Cup, packing their bags and leaving Russia. Only Croatia and France are left to play in the final on Sunday. Fans from around the world have had to face the fact that their country lost. My conclusion? Now is a good a time as any to reflect on failure.
Aside from sports, many, many other areas of human endeavor are some mix of success and failure. Superficially, and depending on how we measure it, I’d like to argue that failure is generally far more widespread than success. And that’s not only because there can only be one winner, as in sports championships like the World Cup. Let’s look at examples from completely different fields:
Every year, the US Congress introduces thousands of bills (proposed legislation), but only about 4 to 5 percent of those end up as laws. That’s a lot of effort – given that bills are hundreds and sometimes thousands of pages long – going into something that doesn’t succeed.
Now let’s take an example from the private sector. In the US, hundreds of thousands of new businesses are established every year. In 2015 that number was 679,072 according to the Bureau of Labor Statistics. Based on historic trends, 50% of all new businesses shut down within five years. Not always because they failed, of course, but probably most people start businesses that they hope will last much longer.
Returning to the subject of sports – in baseball, America’s pastime, the chances of making it to the Major Leagues from the minors, i.e. the lower professional leagues, are only about one in 10. Once they reach the Majors, there is more failure – the average batter fails to reach base by getting a hit (the primary, if not the only, reason for stepping up to the plate) almost three out of every four tries.
What about New Year’s resolutions? Studies have shown that less than 10 percent of people who make resolutions stick to them. Many fail to keep their resolutions for more than a few weeks. Let’s take a longer view. Many, if not most of us, don’t achieve the life goals we set ourselves when we were young. I certainly myself failed to earn a living as an actor (after two years of theater school in Russia and over 4 years of trying to break into the theater in New York), never ran a marathon as fast as I wanted, never wrote the novel I started (and tore up) umpteen times in my younger years. The list goes on. Even high-performing individuals and companies often don’t reach their goals, as evidenced by the sports examples from above or the losers in Presidential elections.
Do these examples, and the statistics, mean are we are simply doomed to failure in our greatest aspirations? Do they suggest that we humans are hopeless at everything we try? I would argue no, not at all.
First, what they indicate is just how much effort we put into trying. How much thinking, planning, time, and money are invested in reaching goals! Think about it: Tens of thousands of kids work hard to become professional athletes or reach the Olympics. Hundreds of thousands of people start businesses every year (even in the depths of the Great Recession over half a million new businesses were started in the US). Millions of us still make New Year’s resolutions.
I’m not saying that merely trying is good enough, or that we should excuse failure because it is the norm. But we should understand it in context – not getting all the way there is human and quite normal. And I’m not saying that we should lower the bar. You need to make a darn good effort to reach whatever it is you are aiming for. Falling short of a goal when you try is very different from not getting there because you didn’t bother. Even if most of us don’t get there, we still get something out of it, we land somewhere. And hopefully learn a few things and become a little smarter in the process.
And now for the evaluation angle: When evaluating the performance of a project, it is really important to ask why the goal was not achieved, as well as asking what was achieved. Don’t just focus on the binary succeeded/didn’t succeed parameter. Don’t pass judgement too quickly because a performance metric wasn’t achieved.
Naturally, it also comes down to how we define and measure achievement. It’s hard not to be impressed by people who start their own businesses, run for high office, train for the Olympics, or get selected to play on your country’s World Cup team. On a CV, all those things look pretty good. But there is a lot of sweat and time that goes into reaching those goals.
And remember, most people who are, or seem, successful, have gone through many failures before they got there. Failure, is after all, the norm. So buck up, and go out there and try one more time.
As the title suggests, in this post I am going to try to tackle three things: meetings, the human brain, and digital intelligence. Bear with me.
I go to meetings, like most of you. They make up a small, but significant part of my work, when I’m not doing background research, writing reports, and travelling. I actually like meetings. This is not just because, in my line of work as a freelance consultant, you develop a real appreciation for periodic human contact, but because meetings – when they focus on specific goals or have a clear agenda – can be extremely productive. In the case of interviews, which is the form a lot of my meetings take (the other types are team meetings and policy discussions), they are the probably most efficient way of obtaining the information you need when I’m evaluating a program, a project, a sector, or some topic.
What are the alternatives to these meetings, to learn new things? Mostly culling information from reports, books, and, of course, the internet, mostly via a search engine or social media. A large amount of Google’s search engine activity now uses artificial intelligence (AI). (I like to think of AI as just the latest manifestation of brainless intelligence, but that’s another blog topic.) Yes, the internet has made our lives a lot easier. But we’re fooling ourselves if we think everything that’s knowable can be found at the click of a mouse.
But before I get to this, yes, I am aware of the complaints. I’ve read a lot about how office meetings are unproductive, a waste of time and money, and brain cells. That may be so in the private sector or in management, areas where I am quite happy not to work. However, I find meeting with other people extremely valuable, for two reasons. First, it is a quick and efficient way to learn the most important thing about an issue. Second, it promotes cooperation, through relationship-building. And without cooperation, things tend to fall to pieces. (I’ll try to get to that in yet another blog post). For now, I’ll focus on why holding meetings is great for information gathering and, in some important ways, much better than Google. Why? It comes down to this – humans are exposed to, immersed in, and able to reflect upon a breathtakingly large amount of real world experiences, interactions, visual stimuli and sensations. We also feel and use our judgement. This is something computers and artificial intelligence can hardly do, despite the recent advances so breathlessly talked up in the media. In fact, search engines are limited to what they can find on servers.
I am not a fan of the reductive approach, e.g. reducing the human mind, or the soul, to biological impulses to be digitally mimicked. But I think there is a useful comparison to make. Strides in computing power and artificial intelligence notwithstanding, humans still have some serious comparative advantages. You can read online about how the human brain compares to a supercomputer, with some saying it has been surpassed, and others saying, not yet, not by a long shot. There are also some interesting comparisons and discussions regarding the human brain vis á vis search engines, especially Google.
You’ll see that, in a narrow sense and along quantitative parameters, search engines may be superior: processing power to retrieve keywords, access to data, speed, etc. But here is one parameter where computer search engines don’t perform anywhere near as well as humans – they are limited to the written, numerical and recorded, information they can find online. That misses out on a huge amount of information. What might that be? Well, everything that isn’t recorded: conversations, events, personal notes, observations of others, email exchanges (that Google doesn’t have access to), and so on. An expert or stakeholder who is engaged in the field you are studying – whether it be education policy in the Maldives, the Uzbekistan irrigation sector, energy efficiency in Ukraine – will be able to draw on a depth and breadth of information that the most powerful search engine in the world can only dream of (if androids could dream of electric sheep, that is). Although the information stored on all the world’s servers is vast and growing, it is still a fraction of all the information in the world and inside the heads of its population.
Ask your interlocutor, your key informant (the term used in evaluation) a question, and he or she will be able to draw on countless, non-digital, resources in order to answer you. Google is limited to giving you what it finds on the web. Certainly useful, but limited. Humans still have some value, it seems. That is why meeting them, if you ask good questions, is so invaluable. If you are an evaluator, an investigator, a journalist, or in a similar line of work, you quickly realize that you get more from holding a few meetings with key individuals than from plowing through hundreds or thousands of pages of documents.
Caveats, caveats. There are always caveats. So yes, it is true that not everyone’s memory functions at an optimal level. And some key informants, you quickly realize, don’t have much to say. Or maybe you find yourself talking to the wrong person. And, naturally, you still need to consult the thematic literature, the reports and journal articles and so on, to complement the meetings you hold. But overall – as a professional, doing my job, I’ll keep going to those meetings. And here’s a(n open) secret: most people actually like to talk about what they do and what they know. Most are happy to share. Also they don’t show you those annoying ads before answering your questions…
Sometimes, at the tail end of another 20 or 30-hour trip, say, to conduct fieldwork for an evaluation, and after crossing as many 11 time zones and spending two nights on airplanes, I ask myself – why? What is the point of traveling, 7,000 miles? Couldn’t we have just had a few phone calls? In this day and age of instant communication, skype and other video chat applications, is it really necessary to be physically present for a meeting? Just think about the time and money spent on business travel. The Global Business Travel Association estimates that global spending on business travel topped $1 trillion about 5 years ago and has been expanding ever since. That amount represents, give or take, 1% of the entire global economy. That is a lot – roughly the size of the Mexico’s GDP. Basically, you can think of it as the marginal value, to the public and private sectors, of face-to-face meetings vs. phone, skype or videoconferencing.
I was stimulated to write this blog post by the latest issue of New Directions in Evaluation (Winter 2017). The entire volume, edited by Randi Nelson and Denise Roseland is devoted to field visits – how and why they are a key element of conducting evaluations, among other things – a somewhat neglected issue in the evaluation literature. The discussions and articles have caused me to reflect a bit on the importance of going to the field (i.e. being physically present in the area where the program is being implemented) in my own work in international development.
There are two main aspects to field visits, which in the field of evaluation, are defined as going to the place where the program or project is being implemented. There might not even be any physical thing to see, such as a new school, equipment, newly installed energy efficient boilers, or what have you. You need to be there to meet with people in their own environment. And long trips and jet lag are actually the least of it. Field trips within countries can involve long car trips to remote rural areas along terrible roads in countries where road safety is somewhat of an afterthought, if not a downright inconvenience. (Auto fatalities pose a real risk to international development professionals. A few years ago I was involved in an incident in which the car I was driving in spun and flipped over on a country road in southern Malawi. I was lucky. I escaped with minor bruises. (Use that seat belt, folks.)
Yet, if you ask me whether I’d rather to do all my analysis from the comfort of my own home, based only on phone interviews or a desk review, I would respond…well, sure, but don’t expect the same depth of analysis or the same quality of information. It is much easier to get a sense of a project when meeting people – implementers, beneficiaries, other stakeholders. Doing a focus group by phone or video? Forget it. Take a look at the photo below, from a field trip I was on in Tajikistan in 2011. Imagine talking to those people from the village by video-conference.
There is also a very practical reason for preferring face to face meetings. When you can see the other people in the room, you can avoid the awkward pauses and people speaking over each other.
Conducting many types of analysis is not just about collecting information. A key factor in getting people to open up to you, and reveal their thoughts, is trust. And when you have never met the person, you can’t look them in the eye, you can’t read their body language. When people get a measure of you (the evaluator, researcher, entrepreneur, etc.) as a person, they tend to talk more. Until you see someone in the flesh, both of you are quite literally disembodied, and not quite real. Who hasn’t in the online dating scene hasn’t had such an experience? Before you met the person they may have seemed terrific, almost perfect, but once you meet them, you have a completely different (and often disappointing) impression? The digital world, the world of two dimensions still doesn’t hold a candle to the world of flesh and blood! Whether you’re evaluating a project or a potential mate, you gotta get yourself up off the sofa, and get out there. For whatever reason, being physically present is a big step in getting closer to that ever elusive truth.
The title of this blog is probably a bit abstruse, and not quite as dramatic as King Kong vs. Godzilla. Bear with me while I explain.
In writing, there is a temptation to focus on problems, on challenges on things that aren’t working. These are all conflicts of one sort or another and conflict is grist for the mill. Conflict grabs our attention. Why else do we exchange gossip, follow sports, watch the news, read history, go to the theater, and slow down to gawk at car accidents? Every one of these “leisure” activities revolves around conflict of one sort or another.
Fundamentally, conflict is one of the cornerstones of the dramatic arts – drama, tragedy, and yes, even comedy. “Conflict creates comedy”, as the HBO show Curb Your Enthusiasm’s Jeff Garlin summed it up. (On a side note, I spent my twenties pursuing an acting career. I moved on when the conflict between the dream and the realization that I wasn’t good enough became too severe. I have no regrets.) I would go so far as to argue that it is human nature to revel in conflicts, their escalation and resolution.
However, for most people conflict is something to avoid in your own life, the current US President being a stark exception. We prefer to watch from the sidelines, from the safety of our living rooms and mobile devices.
This theme can obviously be taken in all kinds of directions. In line with this blog’s main focus, I’m going to talk about evaluation. The ‘temptation of conflict’, by which I mean the temptation to focus on conflict, can in fact creep into evaluation work. There is, always potential for tension to arise between the evaluator and the evaluated. The former is looking for what works and what doesn’t, and there is always something that doesn’t in this complex world. It can take experience and diplomacy to put a client at ease, to avoid losing their cooperation, and help everybody get the most out of the evaluation.
The temptation to focus on conflict can also come into play during the writing process. In my own experience, writing up an evaluation report is generally easier when a project or program has gone badly rather than when it has mostly gone swimmingly. There is more to say, more to analyze, and more to recommend. If we consider the term conflict to encompass disconnects, tensions, misunderstandings, misinterpretations, then it not difficult to uncover conflicts between what was planned and what was implemented, between stakeholders, between donors and government, between expectations and reality and on and on.
Thus, I have found that when conducting a project evaluation , there is a natural tendency to zero in on what doesn’t work. If you are so unlucky as to review a project that is splendid in every way, and have nothing to say in your report, whoever commissioned the evaluation will think you didn’t do a thorough job.
While projects should indeed be analyzed and accompanied by well thought out recommendations, the final evaluation report needs to strike the right balance. For example, If the overall assessment of a project is, let’s say, moderately satisfactory, the substance and tone of the report should reflect that.
I was once on a team evaluating a US government development project. Multiple agencies were involved in providing technical assistance to countries in three different continents, and it the coordination they achieved was quite remarkable. We, the evaluation team, agreed that it was an example of an effective, well designed and well implemented project. We noted numerous accomplishments, along with a few weaknesses. However, the tenor of our report, which methodically addressed each question in turn, was rather negative (As is customary the client who commissions an evaluation provides the questions they’d like evaluators to answer). For each question, we were able to identify some weakness, even while acknowledging the achievements. No one is perfect, after all. Upon reading the report, the project implementers were somewhat bemused. They, justifiably, expressed concerns that we had made the project sound, well, kind of mediocre. This was not our intention. Simply put, there was a lot more to say about the problems (i.e. the conflicts), even if they were modest, than about what had gone well! There was more to analyze and there were more factors to unpack for the negatives than for the positives. We did cover the good stuff, but not at the same level of depth. There were fewer issues to dig into, and not many recommendations linked to them.
One lesson I’ve taken from this and other evaluations is that you must make an effort to balance positives and negatives. While problems (and remedies) should be highlighted, the overall tone of the report should reflect the overall assessment of the project. If you find a few twisted or dead trees, do not portray the whole forest as if it were damaged. Of course, you should not neglect or avoid detailing any negative aspects in your findings. But don’t give them more importance – or less, of course – than is warranted. Be objective, rely on evidence, and be fair.
It is possible to make a decent living as a consultant in international development. If you’re reading this, and work in the international development field, you are probably someone who did not choose a career because of the money. But we all need to earn a living. We all have to eat. Indeed, a lot of our work centers on helping people in low-income countries earn more, increase their job opportunities, or lower their living costs. Today’s blog post is a series of observations on money, mostly aimed at consultants working in international development, who are maybe trying to navigate the earnings map and find their place on it.
Most international consulting jobs pay by the day. Like many others in today’s economy, a lot of international development consultants rely on short-term assignments. They may prefer them, as I do. Short-term jobs give you a lot of flexibility and don’t involve extended overseas stays, which can last for months. I know some consultants who spend as much as a year abroad, leaving their family at home. But that’s not for me. Typically, the short-term jobs will be contracted based on a specific number of days at a given rate. I’ve had contracts for as little as 5 days and as much as 150 days, but most fall in the 30-60 day range.
You can negotiate your daily rate.* In most cases, you can negotiate your daily rate just as you would negotiate a salary when you are offered a full-time job. Once you have been selected for the assignment, you will be offered a rate. You can either accept it or ask for more. If the offer is less than you earned previously, you can share contracts you’ve had with other organizations, they can be used as a reference point. Maybe you have misgivings about negotiating for yourself in a field you entered because you want to serve the greater good? I suggest you still try to earn what you think you are worth, and give away the surplus to a worthy cause.
Most organizations have fee structures. A lot of firms and development organizations such as the World Bank, IMF (and the US Government) have a fee structure. The structure usually has broad bands and is not necessarily rigid. The fee structure may factor in your years of work experience, sector, position (team leader, specialist, researcher), professional background, and education. The World Bank has a four-tier system, based on years worked, and each tier encompasses a fairly wide range, which includes a minimum, maximum, and market reference point. The number of years of experience seems to trump everything else. Even in development work, financial sector assignments seem to pay more, reflecting the market for these skills. (Whether finance jobs should pay more is another topic. My suspicion is that it is merely the ability of finance people to organize their work around commissions, which are easily calculated as a percentage of a deal or an investment, unlike most professionals.)
Fees can vary widely by organization and assignment. Don’t count on earning the same daily rate for every assignment. Your daily fee may vary substantially between organizations, by as much as 50%. Some organizations have salary caps which apply to contractors (as is the case for USAID contractors, which topped out at about $655 in 2016 [update – in 2020 the maximum was set at 698.08]). Exchange rate fluctuations can make a big difference. The US dollar went from $1.59 to the euro in 2008 to below $1.05 in 2016, which meant that working for European firms became less attractive for Americans over this period, and vice versa.
How much can consultants earn? As an established consultant, you should be able to earn the same as, if not more than, full-time staff at the international development organizations you engage with. Many staff have posted their salaries on glassdoor. Of course, you won’t receive any standard benefits. The main benefits of this work come from the satisfaction of doing good work, making a difference in people’s lives (hopefully a positive one), meeting new people, and exploring new countries. That means you should make sure that you can provide your own benefits out of your fee. That applies especially for health insurance and retirement savings. However, remember that your work is not worth less than that of salaried employees. To increase your rate significantly with the same organization may involve moving to a different level (based on years of experience or responsibility).
In theory, your rate should broadly reflect what the market will bear. Keep in mind that the market is different in different countries, and obviously exchange rate fluctuations can have a big impact. As noted above, when the euro was strong, working for a European entity was a better deal than working for American organizations. Within the last few years, the situation has reversed. This is by way of pointing out that one can operate in different markets, and must take that into account. In some markets you will find you are worth more than others.
Expenses are normally accounted for separately. In international development work, expenses are almost always budgeted and paid for separately. You receive your fee as milestones are met, and then submit receipts for any travel and accommodation expenses, unless the organization pays for it directly. Almost always, there is a per diem for meals and incidentals for which you don’t need to produce receipts. You should never have to pay for your expenses out of your fees. Some contracts are lump sum. You cover your own travel, accommodation, meals and incidentals and don’t submit any receipts.
Your present determines your future. By this I mean that whatever rate you agree on today will be the basis for future rate, at this and other organizations. So even if it is just a 20-day contract, and that $20 difference isn’t so important, your next contract may be 100 days, and it will be based on your previous contract. And most organizations will take into account what you earn elsewhere.
Does your rate seem high? Don’t forget – as a freelance consultant, you have to cover all your own benefits, and any enforced vacations from work. Essentially you are trading off benefits for freedom. You need to take into account taxes and benefits when considering your rate. Depending on your nationality and who your contract is with, you may or may not need to pay taxes. In contrast to full-time employees, you will also need to cover your own benefits, especially health insurance and retirement. Plow part of your earnings every year into a low-fee 401k. Warren Buffet suggests Vanguard S&P index funds and I agree with his advice. If you have a partner with an employee health insurance plan, get on that.
You can work overtime. Obviously, if you take on more contracts and work weekends your earnings will rise. Some organizations allow you to bill for a 7-day work week others for a 6-day work week while on site (i.e. working in country) or ‘on mission’ as it referred to by some organizations, as if international development professionals were modern-day secular missionaries preaching the gospel of improved living standards. Just make sure you do top quality work. Don’t double bill by charging different organizations twice. Be results-oriented and be ethical.
Your annual earnings will fluctuate. Ups and downs in earnings are inevitable for non-salaried workers. Over time you will, however, see an upward trend in your income as you get more work, add years of experience, and become better at negotiating your rate.
The daily rate isn’t everything. Despite the above, you may find yourself willing to accept a lower rate for a variety of reasons. Maybe you are trying to get your foot in the door, or you like the assignment, or the work conditions. Perhaps you are willing to earn less on this assignment because you’ll be working in a country you’ve always wanted to go to.
Some resources to consult
Glassdoor. Review the salary ranges on their website, glassdoor.com, which aggregates salaries for different positions, by organization. The data is based on what people have posted themselves.
Fellow consultants. The best source or information on what you can expect to earn is probably other consultants who work, or have worked in the field. (Don’t necessarily ask them about their own rates, however. Talking about one’s salary in the US remains an even bigger taboo than talking about one’s sex life!)
Fee structure matrices. These may or may not be accessible to consultants for all organizations. But it never hurts to ask around. They will give you a benchmark.
Devex. This organization describes itself as the “media platform for the development community.” It caters to organizations, companies and individuals working in international development. There is a monthly individual membership fee (starting at $9.50 at the time of this writing) and you can post your CV and look at job postings. Members have access to articles, including “Foreign aid salary spotlights,” which describes international development jobs and rates in a selection of countries.
Recently, a senior manager at a multilateral development bank told me about problems she had been having with a lead consultant on a project. He had resigned without completing the assignment, and she needed to replace him. She said she was looking for someone ‘with common sense.’ She was even willing to hire a candidate with fewer technical qualifications, as long as he or she could deliver the work and facilitate the relationship with the government. Apparently, this particular consultant had been unreliable and unpredictable. For example, on one occasion he failed to show up at a workshop he was supposed to lead, having failed to notify his client in advance. She arrived at the workshop only to learn that he had left the country without warning!
While I wouldn’t say that this type of behavior is common, it is certainly something you come across when you work in the international development field. Indeed, I have seen stranger behavior, to wit: Exhibit A: a consultant team leader who spent the first 10 minutes of nearly every meeting with partners or government officials bragging about his qualifications, yet never actually delivered anything. On top of that, he threatened to sue counterparts from other organizations who disagreed with him, and essentially went about ruining relationships with almost every key partner. Exhibit B: a consultant- engineer who determined that an entire country’s gas infrastructure was substandard based on observing a single, rusty pipe; and whose understanding of sampling boiled down to “Let’s ask our local colleague to get in touch with his friends to see what they think.” Exhibit C: a senior manager at a bilateral organization whose capricious and unpredictable behavior played havoc with an evaluation, by, among other things, changing the scope of work well after the work started, and undermining her colleagues in front of the consultants. If you’ve spent any time in the development field, you will have stories of your own to tell.
These people may have had the technical skills and looked good on their CVs, but they were difficult to work with, and had a disruptive effect on the work. They lacked soft power. In some cases, they ended up squandering months of time and hundreds of thousands of dollars, not to mention putting at risk the usefulness of the work.
The point I want to make, though, is that in order to do good work, non-technical skills – such as common sense, the ability to communicate with others and present themselves well, plus a commitment to doing the work – are at least as important as the other kind – such as mastery of a subject, experience in a sector, or technical abilities.
Based on my 17 years working as a consultant in international development, I would boil down the qualities you need to the following:
Knowledge
Experience
Common sense
Communication skills
Commitment
The first two I consider hard skills, and the last three soft skills. Try to cultivate each of them. Let’s take a closer look.
1. Knowledge
By knowledge I refer to the technical understanding about a subject that is gained from learning, study, and doing. It comes from formal education at university or in technical schools, training, continuing education, and staying up to date on the literature in the field. It is a core element of the specialization needed to accomplish a task, e.g. analyzing data, developing models, designing surveys, assessing a particular sector. For virtually any field you work in, there will be many courses, webinars, software programs, and training sessions, ranging in price from free to very expensive (thousands of dollars). As you assess these learning opportunities you will need to weigh the costs and benefits in terms of money and time. But this is how you invest in yourself!
2. Experience
Experience is what you get as you act in, and are acted upon, by the world. Your knowledge and assumptions are tested, and you find you have to adjust your approach to doing things. Drawing on experience enables you to make better decisions and work more efficiently. It is the whetstone against which you sharpen your theoretical knowledge. In the beginning, you will be inexperienced. If you are starting out, this is not held against you. It is normal and expected. On a team, or a consultancy, there will be a mix of more senior and more junior members. All are needed, like the cogs in a wheel, and there is a role to play for everyone .
3. Common sense
It turns out that common sense, as I discussed above, is a valuable and perhaps underrated commodity. At its most basic level it means acting as a reasonable person. It involves critical thinking. More advanced concepts of common sense could include seeing through complexity to the essence of a matter, connecting the dots, and framing and organizing information in a useful manner. Much of the time it boils down to the simple ability to reflect upon something, see what is needed, and do what is necessary to get results. Maybe this is not a satisfying explanation, but if you don’t understand the concept of common sense, this may be a problem.
4. Communication skills
The basic ability to have a conversation and get along with your colleagues and counterparts can carry you a long way in any field. People want to get a sense of who you are, whether you are trustworthy, and whether they will be able to cooperate with you. Under communication skills I would include not only to the ability speak in public and engage with colleagues and clients, but the ability to write well. This means conveying information, no matter how complicated or specialized, clearly, succinctly, and grammatically. If you want to build up your public speaking skills, I highly recommend joining a Toastmasters club near you. I have been a member of Toastmasters since 2011 and find it highly useful.
5. Commitment
Commitment to doing the work is just what it sounds like. This is probably what people in the business world refer to as “passion”, which I myself consider a wholly inappropriate quality that should be kept for the opera or the bedroom. Lucy Kellaway of the Financial Times even argues that it is dangerous in the professional field. Emotional teammates are not the most reliable or stable. No, what you want to have is a commitment to doing top quality work, to meeting deadlines, to developing new skills, to learning new things, to being there when your colleagues need you. It is part of having a good work ethic.
The portfolio approach
These five qualities form the bulk of your portfolio of skills. Consider them the tools for building a career. Good luck!
How to launch a satisfying career in international development
I am often asked how to break into the field of international development. In this post I’m going to address this question. I’m going to assume that you are interested in doing interesting work and having a satisfying career as an independent consultant. In an effort to keep things short, I’m going to zero in on the three things I consider the most critical.
Get a graduate degree in a related field
Because of the vast number of issues covered in international development, a degree in practically any field, from economics to health to statistics to law to education, will be useful. However, degrees in some fields are likely to open up more opportunities than others. This is either because they are more broadly transferrable, or because there is a lot of work in this sector. In the first category I would include anything related to economics, statistics, sociology, public policy, or international relations. In the second category, I would include agriculture, health, education, finance, water and energy. However, the farther along you are in your career, the less the specific degree matters. When I work with team members who have at least 5 to 10 years of experience, what they studied all those years ago is often more of a conversation starter than anything else.
When you are just starting out, expect to have shorter contracts and do more work that involves research. If you don’t have a lot of experience yet, being hired to do a desk review may be a common entry point. This is not a bad thing at all, however. It enables you to become familiar with and build expertise in a particular subject, while demonstrating that you are reliable and have good writing skills. (This last is important). With time, you will start working as a specialist or team leader
Be willing to travel anywhere
Travel abroad is obviously a big part of a career in this field. For some assignments, it can mean a foreign posting, for others it may involve months on end away from home. However, in many cases trips last from one to several weeks.
The more willing you are to go to less desirable places and stay there for longer periods, the better your chances of finding work. Think of it as paying your dues. I have friends who, for family reasons, were unable to take on long-term assignments in challenging countries (like Iraq or Afghanistan) and were stymied in their career goals. I have others who spent a year or more in some of the world’s least desirable countries, and it opened doors for them.
Let people know you exist
Finally, people simply need to know that you exist. From a strictly mathematical perspective, the more people in the field who come across your CV or, better yet, meet you in person, the more likely you are to find work.
So put your CV in the public realm. You have various options here. For a monthly fee, you can post your CV on Devex.com and other websites, such as Assortis (for work in Europe). Create a LinkedIn profile. Many consulting firms also ask you to upload your CV and other data onto their company websites, which they then check when trying to find consultants. Applying to specific jobs is another way of getting noticed. Even if you don’t get the assignment – a fair number of job advertisements seem to be pro-forma, with a preferred candidate already selected – many organizations will keep your CV on file and may contact you about other opportunities in the future. Of course, if you have a blog or a book out there, or have won an award in your field, you have opened up a whole new platform on which to stand and let the world know who you are.
I have found that a highly effective way of addressing this existential issue is meeting people face to face. It may appear counterintuitive in our digital age, but making that personal connection remains important. Perhaps it is related to the fact that everyone is now adept at curating their online persona; meeting people in the flesh, where self-editing is more difficult, is a more reliable gauge of character. I have found work because I happened to meet someone at a party, because I shared an office with them for a few days, or because I was introduced to them at a dinner with friends. Of course, qualifications matter, but it seems that the crucial factor was first getting a sense of the person. This informality, and serendipity, still plays a large role in moving along the career path.
There are many ways of getting out and meeting people. Here are some:
Attend events, seminars, BBLs in the international development field.
Join a professional association (I belong to the American Evaluation Association and the Association for Public Policy Analysis and Management, among others) and attend their events and conferences
Send out emails to people working in your field (blind calls). Although many won’t respond, some will. And those that do will tend to be the kind ones who are interested in helping others.
Request informational meetings through people you know, and thus expand your circle of contacts.
Go to more parties and dinners!
If you need some tips on taking a a healthy approach to and getting the most out of networking, , I recommend the book Make Your Contacts Count.
In another post, I’ll discuss some of the keys to getting repeated work in the field.
