Evaluation, the Unpopular Profession

Popularity vs. accountability

If you want to be popular, probably best not to go into evaluation. Pick another role, another profession.

However, evaluation performs a necessary and valuable function. Like street cleaning and colonoscopies, someone has to do it.

Evaluators are paid (tolerated?) to deliver sometimes unpleasant truths or hard-to-swallow advice. The role has evolved, slightly, since Medieval times when the king’s fool, among other things, had to speak truth to power, presumably in some palatable way like mixing humor and self-deprecation. Mercifully, we evaluators don’t need to dress up in funny costumes and makes fools of ourselves anymore (although sometimes we inadvertently do a bit of the latter).

In all honesty, the point of this blog post is not to deter would-be evaluators from entering the field. Rather, it is to warn you that you may not make as many friends as you would if you were, say, working in sales or marketing.

Also, when I say “unpopular” I do not mean that demand for evaluators is low. To the contrary, there is (still) lots of evaluation and evaluation-type work out there.

The fact is, auditors, tax collectors, inspectors, evaluators and their ilk – in what are sometimes called the accountability professions – are not really meant to be liked. 

President Trump’s firing of six Inspectors General in the last few months notwithstanding, most people know that accountability is important. Like taking medicine or going to the gym, being evaluated or investigated can be disagreeable for the object of the evaluation, but there is also good chance it will make whatever or whoever is being evaluated better.

Why you need a thick skin

I have known of evaluators who have been threatened, fired, had their work trashed, or even been held against their will. Here are a couple of examples (one personal).

I once was leading an evaluation in Croatia on the impact of employee redundancies at the country’s shipyards. The data collection supervisor on our team, an intrepid young Croatian woman, was asked by a shipyard manager to turn over the list of (randomly selected) employees she was interviewing. When she refused, he locked her in the interview room and threatened not to let her out unless she complied with his request. She still refused, risking her safety and well-being in the name of professional integrity and respondent confidentiality. Luckily, she had a contact in the Ministry she was able to call, and the manager relented and unlocked the door. She was shaken, but able to continue with her interviews. The evaluation was completed and well received. The reforms, on the hand, did not happen. 

It happens to evaluators – it has happened to me – that your findings and conclusions are rejected, even when I thought the analysis was strong. 

Last year I conducted an evaluation-type study in an African country for the World Bank. It involved assessing the likely social impacts of a $100 million program. The manager who commissioned my work (in a division– referred to as “Global Practices” – at the Bank, with a focus on social issues) was happy with the analysis. However, the manager whose program I reviewed (from the Global Practice responsible for the assistance program) actually refused to speak with me. After reading my report and its conclusions, he rejected the analysis and brought in someone else to redo it. Not a happy experience, but you develop a thick skin in this line of work.

A popular way of rejecting evaluation findings is to question or attack the evaluation methods, or the evaluator’s qualifications. These are good ways of deflecting attention from the findings.

Often, you won’t know whether the client was unhappy with the quality or scope of the work, or if there were other internal politics at play.

These types of unpleasant experiences tend to be rarer, and less of an issue when the client commissioning the evaluation is not the one whose work is being evaluated. This may be the case with donors or US Government agencies such as USAID or Millennium Challenge Corporation evaluations. They are the agencies funding the work, while another organization implements it, so they are generally truly interested in whether the money is being well spent, e.g. is the program going according to plan and getting results.

You may not be popular, but you still have to be nice

With evaluators, when people are nice to you, it isn’t necessarily because they like you, or see you as a linchpin in their career progression.  Of course, hopefully you’re a decent person with a disarming personality! But quite possibly, their chumminess could reflect a, shall we say, slight bias. You are evaluating their programs after all, and they most likely prefer it that you see them at their very best. If people are quite nice to you at the beginning, but when they realize that you’re serious about your job, they start cooling to you, you’ll know that the amicability was more of a tactic than anything else.

It is your job to look past the surface, dig into the data, find what what’s really happening, and report fairly. Yet, you yourself need to adopt an attitude of goodwill, and cordiality toward others, regardless of what your finding are, no matter how useless your inept or corrupt the program is. (I honestly have evaluated very few programs that fall into that category, quite possibly because everyone knows that the evaluators will be showing up.)

Why is being nice to others who might not be nice to you important?

First, being nice is simply part of being professional.

Second, you want to build relationships. You need others trust you and share information and, if all goes well, accept your findings.

Third, maintaining a pleasant demeanor is a simply good default attitude to have to contain whatever feelings you may have about the program you’re evaluating. Whether you think it is amazing or terrible, you want to keep those feelings separate from the work.

The bottom line is that being decent to others is a soft skill you want in your toolbox.

The evaluator as outsider

Closely linked to being “unpopular” is being an outsider.

Professional independent evaluators are, by dint of their position, outsiders. (This is different from internal evaluators who work within an organization.) You need to accept and embrace that role, even while building trust with the client and stakeholders you meet. You need to obtain information from them, and want them to accept your findings.

However, it is a fine line. As an evaluator, you arrive in a new place, with its own professional or work culture, maybe in a new country. You start poking around, and asking questions. That’s the job. People will be on their guard.

The outsider status is beneficial in that it can shield you from certain biases that you might bring. These biases might include if you were part of the system you are evaluating, such as belonging to one or the other political parties, ethnic groups, clans or other groupings of which you are probably not aware.

A corollary of this is that foreign governments often value non-nationals because they are outsiders, independent, not connected to a particular faction. Sometimes this is justified, sometimes not.  Despite being an outsider, and thus often partially aware (or unaware) of the unwritten and unspoken codes and connections, there is value in standing outside, in not being part of the culture.

One can be less beholden, less biased, and face lower risk of consequences from producing unpopular findings, since everyone knows that when it’s over the evaluator will board the plane and leave the country.

The international consultant, very much an outsider position, also brings an international perspective to the table, based on evaluation experience in multiple countries and cultures.

The evaluator as friend?

I like the concept of critical friend, which I have found very useful in understanding and accepting my role as an evaluator. It implies that you are there to help through constructive criticism. One of the best descriptions comes from John MacBeath, a Cambridge University academic, in a 1998 article on improving school effectiveness:

The Critical Friend is a powerful idea, perhaps because it contains an inherent tension. Friends bring a high degree of unconditional positive regard. Critics are, at first sight at least, conditional, negative and intolerant of failure. Perhaps the critical friend comes closest to what might be regarded as ‘true friendship’ – a successful marrying of unconditional support and unconditional critique.

Good evaluators should be respected for their work. They are not going to be the most popular kid on the block and should not strive for that.

Sometimes evaluation findings are accepted, sometimes rejected, sometimes ignored. Sometimes you are hired again, sometimes you are not. As an independent consultant moving from one assignment to the next, one client to the next, you often never learn of the outcome of your work.  It comes with the territory and you should not be disheartened by this.

I began this post by noting how evaluation is not a popular profession. In this age of online calumny and fake news, where many suspect any criticism of being driven by ulterior motives, the notion of accountability is more important than ever.

Nonetheless, if you stick to your guns, maintain your integrity and deliver credible and useful advice, you may be the best critical friend, the people who hired you have ever had.