It’s New Year’s eve! Ready to take the plunge into 2018?
I find that the cold and inclement weather and long nights in the northern hemisphere are especially conducive to reflection and inner work. Many people use the holidays to mull over the past year. I like to set aside some time to review what it was all about (if anyone can explain what happened in 2017, please let me know), and figure out what to do differently, starting with the metaphorical clean slate we all get on January 1.
You see, around this time of year, many of us turn into amateur evaluators. When making New Year’s resolutions we are essentially evaluating ourselves, are we not? It’s safe to say that you don’t make a resolution unless you’re dissatisfied with something. So, we reflect on the past year: what went well, what didn’t go so well, what was achieved, what wasn’t. And based on this self-evaluation, we think about what we want to do better, or differently, next year. That’s similar to what evaluators do when they take on a new assignment. Only in this case, the focus is more personal. And you don’t get paid for it. Which may explain why so many of us fail.
But resolutions are difficult to stick with. According to surveys, while almost half of Americans make them, less than one in ten keep them. I freely admit that I often fall short. But now and then I succeed in sticking with something. This is what I have found works for me:
Self-monitoring is another way of changing. Evaluators rely on data, and time-series data is an excellent source of measuring change over time. To measure something is the first step. The management thinker Peter Drucker reportedly said that to improve something it must first be measured. I have found that just tracking something gets me interested in doing more of it (or less of it, if it’s a bad habit). Many people track their spending as a way of managing their family finances. I’ve kept a list of every book I’ve read since 1986. I also keep a log of how many hours I work. Monitoring equals awareness and awareness forces one to decide how much to value a particular activity, and whether to increase, decrease or change something about it.
Divide and conquer! If your resolution involves a major project, divide it into smaller tasks and take them down one by one. Otherwise, it can all seem overwhelming. The temptation to throw in the towel and say ‘the heck with it’ may be hard to resist. Take your big fat goal, and sit down and make a list of all the steps to reach it. This might involve collecting information (how to fix that leaky ceiling or furnace, where to take language classes), talking to people, getting the tools for the job, setting aside time. The steps should be manageable and not too difficult or time-consuming. Then, one by one, go through your list, checking off the tasks as you complete them. I do this when I have to write reports,
Seek help. It is easier to accomplish something if you have support. It can come from a mentor, a coach or teacher, even a friend or colleague if you share similar goals. I took up the piano in my 40s after a 20 year hiatus, and found I could still play and enjoy it. But I wanted to play better, so I found a piano teacher.
Reward yourself. Incentives, if they’re designed properly, tend to work. Now while it may be difficult to find someone to pay you for achieving your goals (and good for you if you can!) perhaps you have some guilty pleasures (assuming you’re not a robot) that can serve instead. Such as things you like to eat, buy or experience but feel you shouldn’t indulge in that often. Why not make a deal with yourself? A reward for certain milestones on the path to achieving your resolution. ,
Develop some low-effort habits. If your resolution is about changing something in your life – maybe learning about a new area, developing a new skill, or reducing your screen time – try making a small adjustment to your routine. For example, you might set aside 10 minutes in the morning for working on a new language, for meditation, for exercising. Or you might commit to not checking your email except at prescribed times during the day (recently recommended by the Tim Harford of the Financial Times). If the change is not an onerous one, very quickly it may become a habit. You have to figure out what the best conditions are. In 1997 I began keeping a journal. Over two decades later, I still spend 10-20 minutes writing every morning, after shaving and before eating breakfast. I found that the trick was to get the writing over in the morning, not in the evening, by which time my cognitive resources are depleted and I don’t want to have any more obligations.
None of the above may work for you – I myself probably fail more often than I succeed – but it doesn’t hurt to try. Something might stick. Remember, the bottom line is to take whatever resolution you come up with, cut it up and organize it into chunks that are small and easy to digest.
As to my own resolutions for 2018? Post more blogs, for one.
Happy New Year and good luck with achieving those goals!
Edited December 30, 2020
Nils Junge
December 31, 2017
The title of this blog is probably a bit abstruse, and not quite as dramatic as King Kong vs. Godzilla. Bear with me while I explain.
In writing, there is a temptation to focus on problems, on challenges on things that aren’t working. These are all conflicts of one sort or another and conflict is grist for the mill. Conflict grabs our attention. Why else do we exchange gossip, follow sports, watch the news, read history, go to the theater, and slow down to gawk at car accidents? Every one of these “leisure” activities revolves around conflict of one sort or another.
Fundamentally, conflict is one of the cornerstones of the dramatic arts – drama, tragedy, and yes, even comedy. “Conflict creates comedy”, as the HBO show Curb Your Enthusiasm’s Jeff Garlin summed it up. (On a side note, I spent my twenties pursuing an acting career. I moved on when the conflict between the dream and the realization that I wasn’t good enough became too severe. I have no regrets.) I would go so far as to argue that it is human nature to revel in conflicts, their escalation and resolution.
However, for most people conflict is something to avoid in your own life, the current US President being a stark exception. We prefer to watch from the sidelines, from the safety of our living rooms and mobile devices.
This theme can obviously be taken in all kinds of directions. In line with this blog’s main focus, I’m going to talk about evaluation. The ‘temptation of conflict’, by which I mean the temptation to focus on conflict, can in fact creep into evaluation work. There is, always potential for tension to arise between the evaluator and the evaluated. The former is looking for what works and what doesn’t, and there is always something that doesn’t in this complex world. It can take experience and diplomacy to put a client at ease, to avoid losing their cooperation, and help everybody get the most out of the evaluation.
The temptation to focus on conflict can also come into play during the writing process. In my own experience, writing up an evaluation report is generally easier when a project or program has gone badly rather than when it has mostly gone swimmingly. There is more to say, more to analyze, and more to recommend. If we consider the term conflict to encompass disconnects, tensions, misunderstandings, misinterpretations, then it not difficult to uncover conflicts between what was planned and what was implemented, between stakeholders, between donors and government, between expectations and reality and on and on.
Thus, I have found that when conducting a project evaluation , there is a natural tendency to zero in on what doesn’t work. If you are so unlucky as to review a project that is splendid in every way, and have nothing to say in your report, whoever commissioned the evaluation will think you didn’t do a thorough job.
While projects should indeed be analyzed and accompanied by well thought out recommendations, the final evaluation report needs to strike the right balance. For example, If the overall assessment of a project is, let’s say, moderately satisfactory, the substance and tone of the report should reflect that.
I was once on a team evaluating a US government development project. Multiple agencies were involved in providing technical assistance to countries in three different continents, and it the coordination they achieved was quite remarkable. We, the evaluation team, agreed that it was an example of an effective, well designed and well implemented project. We noted numerous accomplishments, along with a few weaknesses. However, the tenor of our report, which methodically addressed each question in turn, was rather negative (As is customary the client who commissions an evaluation provides the questions they’d like evaluators to answer). For each question, we were able to identify some weakness, even while acknowledging the achievements. No one is perfect, after all. Upon reading the report, the project implementers were somewhat bemused. They, justifiably, expressed concerns that we had made the project sound, well, kind of mediocre. This was not our intention. Simply put, there was a lot more to say about the problems (i.e. the conflicts), even if they were modest, than about what had gone well! There was more to analyze and there were more factors to unpack for the negatives than for the positives. We did cover the good stuff, but not at the same level of depth. There were fewer issues to dig into, and not many recommendations linked to them.
One lesson I’ve taken from this and other evaluations is that you must make an effort to balance positives and negatives. While problems (and remedies) should be highlighted, the overall tone of the report should reflect the overall assessment of the project. If you find a few twisted or dead trees, do not portray the whole forest as if it were damaged. Of course, you should not neglect or avoid detailing any negative aspects in your findings. But don’t give them more importance – or less, of course – than is warranted. Be objective, rely on evidence, and be fair.
Nils Junge
October 31, 2017
Summer is in full swing. Before it’s over, I’d like to make some book recommendations. These are for anyone interested in international development but who might find the subject a bit heavy to delve into while at the beach. I think that’s pretty much everyone, myself included! None of these books are new, but each is compelling and even entertaining.
The Idealist: Jeffrey Sachs and the Quest to End Poverty (2013) by Nina Munk will no doubt appeal to the cynics who believe development aid is a waste of money. Munk describes Jeffrey Sachs’ Millennium Villages initiative to inject funds into several villages in Africa. This is part of Sachs’ effort to demonstrate that the fundamental problem is one of investments, i.e. that there are not enough being made. Sachs is a well-known economist who has called for a doubling of aid and he should be commended for putting his money where his mouth is (well, not his money, but funds raised from various donors). The results are not, by and large, happy. For anyone, myself included, who thinks the problem with aid is not so much the lack of financing, but cultural and institutional challenges that arise when you try to give shovel money into a place which lacks the enabling environment. It isn’t the quantity of aid financing, but about the quality. This book supports that argument, while describing what, at the time of writing looked like a lost cause. Highly readable and thought provoking.
Behind the Beautiful Forevers: Life, Death, and Hope in a Mumbai Undercity (2012) by Katherine Boo is not, strictly speaking, about development aid. I include it here because of the richly detailed descriptions of life among the poorest of the poor, especially their interactions with, and dependence on the world outside their slum. Looming large is the Mumbai International airport, the very reason for the slum’s existence, and a direct or indirect source of livelihood for most of its residents. Although a work of journalism, it reads like fiction, as Boo uses a very close narrative voice to relate what is going on inside the heads of a number of characters. This level of intimacy was achieved by conducting interviews with 168 people over a period of several years.
Adventures in Dystopia (2013) by Daniel Sellen, is an all-too believable and highly enjoyable fictional account of six connected characters, three from the West, three from developing countries. The Westerners are in the business, so to speak, of helping the poor, for a variety of motives. On the other side, the three characters – from Columbia, India and Cameroon – each have their own agenda, which doesn’t necessarily coincide with what the aid workers have in mind. Sellen, a World Bank staff, clearly draws on his own extensive professional and personal experiences to depict a world alternating between comedy and tragedy, and the chemical reactions that can and do occur when people from rich and poor societies encroach on each other’s territory, so to speak.
Don’t be put off by the somewhat hyperbolic title or fooled into thinking that this is an instruction manual. In How to Change the World (2007) David Bornstein describes the work done by social entrepreneurs. Social entrepreneurship differs from straightforward entrepreneurship in that its goals that encompass social, environmental or other common good problems. Bornstein devotes each chapter to a different Ashoka fellows (as well as the founder of Ashoka, one of the original promoters of social entrepreneurship) who, in various ways, have changed the lives of many, many people through creatively tackling problems in their communities. Issues they have addressed include promoting reading, self-employment for women, etc. These are a far cry from Silicon Valley apps which seek to make life slightly more convenient for the urban middle class. They are inspiring stories of how a little assistance can go a long way, and a welcome antidote to the cynicism that can come from seeing insurmountable problems everywhere.
From different perspectives all four books highlight the cultural barriers between rich and poor, between the citizens of Western and developing countries. It turns out – surprise, surprise! – that people in developing countries have agency. They are not a mass of abject, aid-dependent sufferers, but many are hard-working and entrepreneurial with their own rich and complex inner lives. They inhabit worlds every bit as complicated, hierarchical, and full of joys and sorrows, as we in developed countries do. They are not passive recipients, not the ‘meek who will inherit the earth’. They will succeed (if they succeed) because they are entrepreneurial, on the lookout for an opportunity. They are likely to be as morally upright as they are devious or corrupt. In many ways they are not unlike everyone else on the socioeconomic ladder, from the poorest of the poor to the billionaire elites.
What hinders the majority of people in poorer countries from doing better are systemic issues. Country institutions often work against their people, who must deal with arbitrary justice, antagonistic rule of law, and entrenched networks which keep them from finding opportunities to thrive. In this respect, let us hope that developing countries converge with the West, and not the other way around!
(Revised May 5, 2020)
Wherever you stand on the issue of foreign aid, the White House budget proposal has put the topic back on the table for discussion. And it definitely deserves being discussed.
On the whole, Americans are, apparently, a generous people. The vast majority say they donate money – 83% according to a 2013 Gallup survey, which also found that 65% volunteer. It is probably safe to say that most Americans believe giving to those in need is a good thing. But should the government be involved in giving to others, especially to people living in other countries? That is a different matter, and the debate over ‘foreign aid’ or ‘development aid’ has been going on for decades now.
American attitudes on this matter are a bit fuzzy. Before we even get to the principle of foreign aid, there is already plenty of confusion about how much the US spends on it. (The short answer is – vastly less than most people think.) However, one group of people has a pretty clear idea on the matter, and it is those working for the current US Administration.
The new White House budget proposal for fiscal year 2018 would see foreign aid, along with State Department funding, cut by almost one third. This is a massive reduction. Remember the budget sequestration of 2013, which cut programs across the board, and how difficult that adjustment was? The current White House budget proposal, which also includes deep cuts to domestic programs, transfers virtually all of the $52 billion in savings to the Department of Defense, increasing the latter’s budget by about 10%. (Bear in mind that this is not the actual budget, which is up to Congress to draft and pass. But it signals the President’s priorities to Congress and it will exert influence over Congress if there is pushback. It is best thought of as a negotiating position.)
So there it is – the new Administration has signaled the approach it wants to take in order to build “A New Foundation For American Greatness,” the official name of the proposal. It will be guns, not butter.
How exactly will this budget help America become great? The explicit rationale is that the President will “place America first by returning more American dollars home and ensuring foreign aid supports American interests and values.” Those values, as per the Administration, are pretty much all about security and military prowess. In evaluation, this is called a ‘theory of change’ or ‘program logic’ – how specific action will lead to a desired outcome. Naturally, there is much debate about whether the budget the White House is proposing would have the desired effect.
The proposal argues that it will make the Department of State and USAID leaner, more efficient and more effective, although it does not specify how it would do this, aside from forcing the two agencies to operate with considerably fewer resources. Furthermore, it proposes redirecting foreign aid spending primarily to security issues.
What the President’s budget proposal does is open a Pandora’s box of questions. Whatever you think of it, these questions are legitimate and worth pondering.
Definitions. How is foreign aid defined? What is the money actually spent on? Who ends up benefiting and how?
Effectiveness. How effectively is aid spent? Is it getting the intended results? How do we know?
Amounts. How much does the US actually spend on foreign aid? Interestingly, the average American believes we spend far, far more than we do. About 20 times more. Not 20% more, 20 times more.
Rationale. What is the rationale behind foreign aid? How valid are the arguments made in its favor, specifically: national security, commercial interests and humanitarian concerns?
Investment vs. cost. What is the chance that “returning more American dollars home” by cutting foreign aid and diplomacy funding will make America great? Is there a cost to spending less overseas? Is aid an investment or just a cost, an expenditure?
Relativity. Is foreign aid spent less effectively than other budget items, such as defense spending, which is targeted to be the prime beneficiary of cuts to aid and many other sectors? Are there different standards of transparency and accountability?
Measurement. How is effectiveness measured? Are random-control trials, expensive experiments which measure the impacts against what-if scenarios, the best way to measure assistance? What about short-term vs. long-term effects of aid? What about the indirect and intangible effects?
Home vs. abroad. Why should one country give to another? Even if you believe there are benefits, how do you balance giving assistance to foreigners vs. giving assistance to your own citizens?
Altruism. How strong is the moral case for giving to strangers in a strange land? Should rich country’s citizens give much more, as long as people are dying of hunger and disease, simply because they can?
Geopolitics. By drawing down its foreign assistance, does the US abdicate its leadership role and reduce the geopolitical influence?
Where it goes. How much of foreign aid does actually end up in other countries compared to how much of it finds its way back to US citizens, US companies and the US budget via contract work and purchase of US goods, services and military hardware?
Typologies of aid. Aid comes in many shapes and forms, and with different objectives. What form of aid has proven to be the most beneficial? How does aid in the form of investment in infrastructure compare with the transfer and exchange of know-how (referred to as technical assistance)? Most aid is not simply transferred to poor people in developing countries – much of it is in the form of investment and technical knowledge transfer.
Potential harm. Does aid do more harm than good? Even leaving aside aid that goes to foreign militaries with questionable records, there is evidence, and arguments, that aid is not only ineffective, but that it can actually be damaging to individuals and society. What does the record show?
Government’s role. Should the government be in the business of giving? Perhaps aid should be left to philanthropic organizations, and be based on purely voluntary donations as opposed to taxes.
Externally provided assistance. Is there something intrinsically problematic with externally-provided assistance, i.e. efforts to exert change from outside a country?
In a series of upcoming blog posts I’m going to grapple with these thorny issues one by one, not necessarily in order. Let’s see where we end up.
Numbers are used a lot to make a point or serve as evidence. When you come across a number, in a report or on the news, always remember its meaning depends almost entirely on its context.
Certainly, numbers often seem inherently more reliable than descriptions. In the social sciences, there is a distinct preference for quantitative vs. qualitative findings. There is a bias, but perhaps it’s justified?
When you are presented with a number, there seems to be less intermediation. Numbers don’t lie, they say. But then the phrase “lies, damned lies and statistics,” often attributed to 19th century British Prime Minister Disraeli, is also pretty popular. It highlights inherent distrust toward statisticians and politicians. Because even if on the face of it numbers seem to be objective and verifiable, often there is quite a bit of intermediation behind their production.
While intermediation (statistical analysis) is acceptable, indeed necessary in order to collect and organize information and ensure that the results represent what they are supposed to, manipulation is not.
Forgive me for being blindingly obvious, but we need hard numbers. They are extremely useful. They allow us to have a common basis for discussion.
On a side note, numbers even serve different purposes. There are cardinal numbers (indicators of quantity, e.g. how many cups of flour in a bread recipe), there are ordinal numbers (indicating the order of things, who came in first place), and there are nominal numbers (for identifying things, like your seat number in the theatre). I’m going to focus on cardinal numbers for now.
Remember that a number is just a number. Its meaning (good or bad, important or trivial, expected or unexpected) is imposed externally. Its value is not inherent but rather depends on context. The context is what tells the story.
It is not enough to say that the unemployment rate was low. The description “low” can mean any number of things. Your first reaction should be “how low?” and “low compared to what?” The thing is, plugging in an actual number only gets you part of the way there. Let’s say unemployment was 7%. How does that compare to last year? It may be low if last year it was 9%, or it may be high if it was 5%. Or if in similar economies is above 10%, 7% looks pretty good, whereas if everywhere else it is below 6%, it looks less good.
Not only are comparisons across time and space important, but comparisons with purely fictitious numbers also matter. What I’m referring to by this is the number in your head, which doesn’t’ actually exist, right? This is the number you were expecting.
When unemployment numbers are predicted to fall, but they actually go up, the stock market may fall. When you expect to be offered a salary of $60,000 but then get an offer of $80,000, that will make you very happy. The age of newly elected French President Emanuel Macron’s wife, Brigitte (64) is not interesting – there are many women in their sixties in France – until you know that he is only 39. Their age difference is unexpected because of the gender markers. Its unexpectedness is in inverse relation to the age difference between Donald Trump and his wife, Melania. Who wouldn’t expect a billionaire property developer to have a wife a quarter century his junior?
Placing bets is an extreme example of what we can call the arbitrage between expected and actual numbers (or current and future numbers, if you will.) So a lot of money is often at stake.
You can apply this type of thinking to pretty much everything – from the price of a new refrigerator, to your annual salary to the number of people without access to drinking water. Context is everything. What that means is that reference points are critical. Numbers floating out there on their own don’t mean much.
Of course, you want to know how a number is derived. You should ask how they are collected, what they represent (often a matter of definition), and whether they truly represent what they are meant to? Obviously, in the case of the Brexit referendum and the last US election, the numbers which the vast majority of polls gave us did not accurately represent the people who actually voted.
Statistics can be presented misleadingly even when they are accurate. One of the most common issues perpetuated by the media and researchers themselves, involves comparing differences to a (seemingly arbitrary) base, or even not revealing the actual base. For example, findings reported in the American Journal of Epidemiology indicated that drinking more coffee reduced the chances of dying from oral/pharyngeal cancer by 49%. Sounds impressive, right? But you also need to know that your chances of dying from this type of cancer are about 0.09%, i.e. less than one in a 1,000. They weren’t high to begin with. And what about all the other cancers? Did they check those? And what about negative effects? Maybe those cancel out the positive ones.
Responsible reporting of statistics, whether in academic journals or the media, involves providing the reader or viewer with meaningful context. When you see a number, ask yourself the following questions:
- What does this actually represent? How is the number defined?
- How does it compare with alternatives, similar situations?
- How does it compare with the past?
- How does it compare with what was expected? Does the number fall in line with a trend or go against the trend?
- What are the possible interests of the person or organization in sharing this number? Journalists and evaluators have a code of ethics they are supposed to follow.
Ideology vs. evidence
The tension between belief and evidence has probably been around ever since humans could sense the world and formulate thoughts. It has certainly been documented for several thousand years. In the Bible, we have the original ‘doubting Thomas’, who would not accept that Jesus had come back to life until he could put his hand on the wound in Jesus’ side. Galileo was put on trial by the powerful Church for arguing, based on scientific evidence, that the Earth circled the Sun and not the other way around.
The issue goes beyond belief. A set of beliefs tend to form an ideology, which typically concern human nature, religion, politics, or the economy.
For various reasons, people hold certain theories of how the world works, or should work. People may be influenced by tradition, tribe (the community to which they belong) or an authority they revere (like their church, or their president). Belief systems can have enormous consequences. At the extreme end, people have died for their beliefs – consider the early Christians refusing to renounce their faith and being fed to the lions, or young men volunteering to go to Syria and join ISIS. But even relatively anodyne beliefs about the world can have tremendous social impacts. For example, the American belief that this is the land of opportunity and if you don’t succeed you only have yourself to blame. In fact, there is a strong counter set of beliefs: that it is mostly those lucky enough to be born into the right (privileged) circumstances who actually enjoy those boundless opportunities…There are plenty of studies finding that children born into poverty have far fewer chances in life.
With respect to the current tensions between beliefs and evidence, there are at least two things worth noting.
Overwhelmed by too much information?
First, we live in an age saturated with information, which would make you think that evidence is readily accessible, and that virtually anyone can find at out information with a little web-browsing. Yet the rise of conspiracy theories, fake news, and the bifurcation into ideological communities in many countries defy this. The almost infinite wealth of information out there is also good cast doubting on evidence and supplying hermetic communities with a tailor-made set of facts that align with their beliefs. It has become easy to engage in selective perception. Information that doesn’t conform with what you think you already know can be ignored, devalued, or diluted with misinformation.
A bipartisan issue
Second, disregard for evidence or facts seems to be prevalent at both ends of the political spectrum. Both rightwing and the leftwing elements in US society (and possibly elsewhere) have embraced ideology at the expense of evidence. Facts that don’t fit are belittled or disregarded.
Many on the right, for example, scorn the science and statistics which show that the earth is more than roughly 6,000 years old, despite the geological evidence; that there is a near consensus among scientists that global warming is an actual, man-made phenomenon, or that gun ownership significantly increases risks to one’s life and those nearby. On the left, there is a rise of intolerance toward free speech on university campuses, which muzzles the expression of opinions deemed contrary to prevailing norms. There seems to be a prevailing wisdom that the college campus should be a protected ‘safe space’ (as if it were kindergarten) and that students need to be sheltered from ideas they disagree with or find uncomfortable – hence all those trigger warnings. Both right and left extremes seem to have become biased against any evidence which goes against their prevailing orthodoxy.
Living with belief and paying attention to evidence
Almost all humans believe things, or believe in things. This doesn’t mean we have to let beliefs cloud our judgment. If you believe in God, you generally still don’t take stupid risks because you believe He loves you and won’t let anything bad happen. Most people try to pay attention to their personal safety – staying on the sidewalk, driving carefully, not handling poisonous snakes – whether they believe in guardian angels or not.
Even evaluators believe things. We’re not robots. But as professionals we’re essentially paid to put our personal preferences aside and use our brains to objectively process large amounts of information.
Try this at home
If you are an evaluator, or just someone interested in getting closer rather than farther from the truth, you need to be aware of your beliefs and not let them interfere with your professional judgment. You need to maintain a boundary.
Here are a few things that I try to do to stay ‘conceptually sober’, i.e. not under the influence of beliefs, while on the job:
- First, recognize your biases. Monitor yourself as you collect and analyze and present information. It is easy to skew findings to fit in with your hypothesis, or ignore them. Unfortunately academics do this all the time, by not publishing research findings that aren’t in line with their hypothesis, or simply interesting (publication bias)
- Consider alternative causal explanations. You may have a good story that makes intuitive sense. You may have evidence (you better have evidence if you’re doing research). But always think about and look for evidence that might offer different explanations.
- When you have findings that seemingly contradict each other, dig deeper to find an explanation. Maybe your data is inaccurate, or your sampling method was poor. But maybe different sources are revealing different things, or people you are interviewing are interpreting or defining events in different ways.
- Double check what you hear from people when you’re doing research. If you hear pretty much the same thing from a lot of different that should give you confidence. If I hear something new or unusual from someone, I’ll ask others to corroborate.
- Follow a guiding principle. It could be something like: “To present what happened as accurately and fairly as possible.”
- Be curious. Find out as much as you can about a topic. Invest yourself in finding out more. Don’t be satisfied with the most obvious answers.
The shocking United Airlines ejection of a passenger was an outlier
This week I’d like to talk about outliers. These are people, events, or data points that are so far from the norm that they attract unusual amounts of attention. Outliers make the news. Most of the news stories you read concern exceptions or unexpected events. They grab out attention. The disturbing United Airlines’ scene of a paying passenger being forcibly and violently ejected from the plane represents just such an outlier.
Last week, a passenger was brutally dragged off a United Airlines flight in Chicago by security guards who broke his teeth and nose, leaving him with blood streaming down his face and a concussion. He had refused to give up a seat he had paid for, and United decided that he and three other passengers (chosen randomly according to some algorithm) had to leave the plane. The airline had offered travel vouchers (of $800) but no one had taken them, apparently. So it decided – time to draw straws. And what for? So space could be made for four airline employees, arriving at the last minute for the flight bound for Louisville, Kentucky.
The scene of security personnel pulling the 69-year old Asian American (he was born in Vietnam) through the aisle to the horrified looks and gasps of other passengers was filmed on smartphones and duly posted to the web. It has generated a huge outcry and calls for a boycott of United. The company made things worse when CEO Oscar Munoz issued a terse non-apology, blaming the passenger, Dr. Dao, for being “disruptive” and “belligerent”. Munoz wrote: “I apologize for having to re-accommodate these customers,” which is about as far away from saying sorry as it gets before entering antonym territory.
Meanwhile, there are plenty of news articles predicting that the public will get over it, and United Airlines will weather the storm. This is because many companies have overcome this type of scandal in the past, and because many customers don’t have a lot of choice when it comes to airlines. Consolidation among the big legacy airlines, blessed by the regulators, has ensured that.
The shocking incident is highly unusual, which is why it generated so much attention. We are used to hearing about passengers being escorted from flights for misbehaving, but in this case it was the airline which misbehaved (even if it did outsource the job to airport security).
Outliers – good for the news, but not so good for research?
In the research world, outliers are not a news opportunity. Most of the time, they’re viewed as a problem, sticking out like a sore thumb. Outliers raise questions about data reliability and validity. They distort the mean, leaving an inaccurate impression, even if the data is 100% correct. (Remember the one about how Bill Gates walks into a bar and suddenly everyone is a billionaire, on average?) The solution? Outliers are typically ignored or dropped from datasets by researchers.
But outliers can mean different things. They can be symptomatic or anomalous. Maybe an outlier highlights a larger problem, represents the tip of the iceberg, a leading indicator, a canary in the coal mine, i.e. a symptom of some larger phenomenon or a trend that is about to break. Or maybe it represents an exception to the rule, a bad apple, and can justifiably be disregarded. It is also possible that the outlier is an error. Before deciding how to deal with them or react to them, we need to understand what they mean. If they’re signaling something, then even researchers need to take a closer look at them. Like a doctor, you should check your outlier in order to make a diagnosis.
In a 2002 article, Vijayendra Rao advocates “having tea with an outlier,” i.e. looking more closely at what they represent, and maybe even talking to them if they represent a survey respondent, to get a different perspective on the issues.
The outlier may have different intrinsic characteristics that sets it apart. I was once asked to find a poor person in a village in Kazakhstan in a region where I was conducting an evaluation for the Asian Development Bank. It wasn’t easy, because the people we talked to didn’t really consider themselves poor and had to scratch their heads when we asked them to point us toward a poor household. Finally, my team members and I were directed to the home of a single mother. We brought her some groceries, and knocked on her door. She invited us in and we sat down and talked to her. It turned out that she had emigrated from Ukraine (I can’t remember whether her husband had died or merely left her) and thus lacked a social support network. She had problems with her papers. There were also health issues. I don’t believe we actually had tea. She was an anomaly, an exception. She didn’t represent the typical inhabitants of the region. While we learned about what kind of factors might drive people into poverty, her case didn’t tell us much about poverty issues among the population as a whole.
The case for symptomatic
But if the outlier represents an extreme case of a phenomenon that is happening to a lesser degree elsewhere, then it takes on a different meaning. What do we have with United? I would argue that the case is symptomatic and not anomalous. Indeed, although it was well outside the norm (the chances of a passenger getting bumped from a flight remain 1 in 10,000, and the chance of losing your teeth in the process remain vanishingly small) the mood against United has been building for some time and helps explain the outrage. United certainly did not have a good customer service reputation prior, and the extreme mistreatment represents, in many people’s eyes, all that is wrong with the company. The frustration and anger over poor service boiled over. United’s reputation was already solidly second rate. It ranks 66 out of 100 global airlines according to one survey.
My own experience flying United is far from pleasant, and presumably widely shared. Anyone who has flown with a European, Gulf region or Asian airline will know that US carriers in general deliver poor service. While with the better international carriers you might feel as though you were their guest, on most American carriers you feel like a revenue source that, inconveniently, must be processed, takes up physical space, and requires (minimal) attention. They get away with it because of a lack of competition, and because they know passengers put up with it because of the relatively low prices.
The irony is that Americans on the whole don’t tend to be unfriendly; quite the opposite. But once hired by an airline, I can only assume that they are processed through some kind of training module which strips away as much of their humanity as possible (although you will occasionally interact with a friendly crew member or ground staff whom the system clearly failed to process).
Finally, CEO Munoz’s initial reaction to back his staff and essentially blame the passenger was very telling and fairly indicative of United Airlines’ attitudes in general. Based on his initial reaction for Munoz the incident was not such a big deal. In other words, it was within the bounds of normalcy. That suggests it was symptomatic, not anomalous. Granted, Munoz did issue a proper apology days later, but that was tainted by the strong suspicion that it was a reaction to the airline’s share price dropping a bit and not some sort of recognition that it’s approach to customers in general is woefully lacking in common decency.
So while the news articles which argue that United will survive this debacle may be correct, it doesn’t mean that this particular extreme behavior doesn’t mean anything. I believe the evidence supports the view that it reveals a lot. It is symptomatic of a much larger problem – in a word, disrespect toward passengers, those important but still annoying revenue streams.
(If you’re curious, I will do my best to avoid giving United my business in the future, even if it costs extra.)
Inadvertent airline humor
I leave you with some one liners. They are taken verbatim from a link on United Airlines’ own website called, without a trace of irony, Shared Purpose and Values:
- We Fly Right: On the ground and in the air, we hold ourselves to the highest standards in safety and reliability. We earn trust by doing things the right way and delivering on our commitments every day.
- We Fly Friendly: Warm and welcoming is who we are.
- We Fly Together: As a united United, we respect every voice, communicate openly and honestly, make decisions with facts and empathy, and celebrate our journey together.
- We Fly Above & Beyond: With an ambition to win, a commitment to excellence, and a passion for staying a step ahead, we are unmatched in our drive to be the best.
Even setting aside the passenger ejection incident, anyone who has ever flown on United – an airline with some of the most unhelpful and unfriendly employees in the world – will be forced to acknowledge that they do have a dry sense of humor.
Nils Junge
April 18, 2017
When interviewing people as part of an evaluation, at some point I like to put them on the spot.
The interviewees will be well-informed about the program under evaluation. That’s how they were selected. They might be policy makers, managers, program implementers, sector specialists or some other type of what we in the business call “key informants,” or “KIs.” The interviews are semi-structured, with a pre-determined set of questions or topics. That means the answers can be open-ended, in contrast to surveys where most responses need to be kept succinct. The open-ended format allows the interviewer to probe, follow-up or clarify particular points. It’s not so different from how a journalist interviews a subject, or police detective interviews a suspect. It’s a process of discovery as well as a matter of answering straightforward questions about program.
During the interview, as we go through the questions and the respondent shares her assessments and opinions (often in many shades of grey) I’ll press her to take a stand and defend her position. I’ll ask for a number. I want a number that summarizes, say, her subjective assessment of the program.
Sure, I like words. I’m a writer, after all. You can learn a lot from words, but there are just so darn many out there! In its ability to synthesize a story, a number can almost be poetic…
In the middle of a discussion about Program A, I’ll ask something like: “Now, based on everything you know about this energy development project, how would you rate its impact on a scale of 1 to 5, where 1 means you noticed no impact at all and 5 means you noticed a very strong impact?” The key informant will come up with a number, say a “4.” I make a note of it, and then follow up with something like “Please elaborate. Tell me why you rated it a 4?” And the interviewee builds a case, offering more details, providing a rationale.
Sometimes the response you get is a surprise. A key informant will be criticizing a program left and right, but then rate it 4 out of 5. Why the apparent disconnect? Apparently all those criticisms carried less weight for the respondent, in the grand scheme of things, than I had assumed. You see, if I hadn’t asked them to give me a number, I might well have walked away from the interview thinking, “Hmm, he thought that program sucked.” In fact, they thought the program was pretty good, but just had some caveats. You can find the opposite scenario as well, of course. A person speaks positively about a program, then rates it a 3. It could be they just had very high expectations.
These are scale questions, referred to as Likert-type scale survey questions after the psychologist who invented the concept, Rensis Likert. Of course, such rating questions are extremely common in surveys. Who hasn’t responded to an online or in-person interview which didn’t include a scale question? Online sites including Amazon, Netflix, TripAdvisor, and Yelp use a similar approach to get us to rate products or services.
The concept has something in common with the efficient market hypothesis, which states that share prices reflect all current available information. All the negatives, positives, and expectations are priced into that one number. Doctors use a pain scale to gauge a patient’s chronic pain. Therapists will ask their clients to rate their depression using a scale. Similarly, evaluators might use a scale to understand the degree of effectiveness of an intervention, or a number of issues.
Typically, the scale question is used for closed-ended interviews, as part of surveys. Responses can be analyzed and used to obtain an aggregate measure for all the 1,243 respondents, as well as different subgroups. For example, you might be find that, on average female participants rate the program’s ability to improve their lives at 4.3, while male participants rated it 3.8. Done well, this line of inquiry can be a valuable method for taking a population’s pulse on an issue.
The nice feature of open-ended interviews with key informants is that you are able to do a little digging after they’ve coughed up a number. In a survey, if you ask respondents to explain their answers, things can get complicated – those longish answers can’t be easily summarized numerically. (Of course, open-ended answers can be coded according to type, but then you lose a lot of that rich detail along the way.) You don’t face that constraint when conducting qualitative research. You face other constraints instead.
I find that rating questions are a good way of cutting through the dense jungle of information you can be pulled into when doing research. You’re walking along, making your observations of the flora and fauna, taking as much in as you can, using your machete to clear the way, trying to figure out whether you’re heading in the right direction (i.e. testing incoming information against your hypothesis, the path which may or may not lead you to the truth). And then you emerge onto a rocky outcropping…and all at once see the whole rainforest spread out below you. Aha! So that’s how the program looks.
Rather than being a reductive or sterile exercise, I’ve found that people being interviewed rather like this type of questioning. They appear to enjoy the exercise of mentally processing large amounts of information on a subject to generate out a single number. And they like explaining how they got there.
Essentially, this is a way to leverage people’s cognitive functions. You’re engaging them in a kind of meta-cognition exercise, in which they examine and explain their own thought processes.
Try it out on yourself. On a five-point scale, rate your satisfaction with, say the place you live; your own job performance…or how much of your life you spend online. Then justify that number in words. You will most likely find that your brain immediately begins sorting through a whole succession of factors, lining up the pros and cons, weighing them against each other.
It may be that I’ve spent far too much of my life evaluating stuff, but I honestly find this exercise quite revealing, stimulating even, in a cerebral sort of way.
Nils Junge
April 11, 2017
It is possible to make a decent living as a consultant in international development. If you’re reading this, and work in the international development field, you are probably someone who did not choose a career because of the money. But we all need to earn a living. We all have to eat. Indeed, a lot of our work centers on helping people in low-income countries earn more, increase their job opportunities, or lower their living costs. Today’s blog post is a series of observations on money, mostly aimed at consultants working in international development, who are maybe trying to navigate the earnings map and find their place on it.
Most international consulting jobs pay by the day. Like many others in today’s economy, a lot of international development consultants rely on short-term assignments. They may prefer them, as I do. Short-term jobs give you a lot of flexibility and don’t involve extended overseas stays, which can last for months. I know some consultants who spend as much as a year abroad, leaving their family at home. But that’s not for me. Typically, the short-term jobs will be contracted based on a specific number of days at a given rate. I’ve had contracts for as little as 5 days and as much as 150 days, but most fall in the 30-60 day range.
You can negotiate your daily rate.* In most cases, you can negotiate your daily rate just as you would negotiate a salary when you are offered a full-time job. Once you have been selected for the assignment, you will be offered a rate. You can either accept it or ask for more. If the offer is less than you earned previously, you can share contracts you’ve had with other organizations, they can be used as a reference point. Maybe you have misgivings about negotiating for yourself in a field you entered because you want to serve the greater good? I suggest you still try to earn what you think you are worth, and give away the surplus to a worthy cause.
Most organizations have fee structures. A lot of firms and development organizations such as the World Bank, IMF (and the US Government) have a fee structure. The structure usually has broad bands and is not necessarily rigid. The fee structure may factor in your years of work experience, sector, position (team leader, specialist, researcher), professional background, and education. The World Bank has a four-tier system, based on years worked, and each tier encompasses a fairly wide range, which includes a minimum, maximum, and market reference point. The number of years of experience seems to trump everything else. Even in development work, financial sector assignments seem to pay more, reflecting the market for these skills. (Whether finance jobs should pay more is another topic. My suspicion is that it is merely the ability of finance people to organize their work around commissions, which are easily calculated as a percentage of a deal or an investment, unlike most professionals.)
Fees can vary widely by organization and assignment. Don’t count on earning the same daily rate for every assignment. Your daily fee may vary substantially between organizations, by as much as 50%. Some organizations have salary caps which apply to contractors (as is the case for USAID contractors, which topped out at about $655 in 2016 [update – in 2020 the maximum was set at 698.08]). Exchange rate fluctuations can make a big difference. The US dollar went from $1.59 to the euro in 2008 to below $1.05 in 2016, which meant that working for European firms became less attractive for Americans over this period, and vice versa.
How much can consultants earn? As an established consultant, you should be able to earn the same as, if not more than, full-time staff at the international development organizations you engage with. Many staff have posted their salaries on glassdoor. Of course, you won’t receive any standard benefits. The main benefits of this work come from the satisfaction of doing good work, making a difference in people’s lives (hopefully a positive one), meeting new people, and exploring new countries. That means you should make sure that you can provide your own benefits out of your fee. That applies especially for health insurance and retirement savings. However, remember that your work is not worth less than that of salaried employees. To increase your rate significantly with the same organization may involve moving to a different level (based on years of experience or responsibility).
In theory, your rate should broadly reflect what the market will bear. Keep in mind that the market is different in different countries, and obviously exchange rate fluctuations can have a big impact. As noted above, when the euro was strong, working for a European entity was a better deal than working for American organizations. Within the last few years, the situation has reversed. This is by way of pointing out that one can operate in different markets, and must take that into account. In some markets you will find you are worth more than others.
Expenses are normally accounted for separately. In international development work, expenses are almost always budgeted and paid for separately. You receive your fee as milestones are met, and then submit receipts for any travel and accommodation expenses, unless the organization pays for it directly. Almost always, there is a per diem for meals and incidentals for which you don’t need to produce receipts. You should never have to pay for your expenses out of your fees. Some contracts are lump sum. You cover your own travel, accommodation, meals and incidentals and don’t submit any receipts.
Your present determines your future. By this I mean that whatever rate you agree on today will be the basis for future rate, at this and other organizations. So even if it is just a 20-day contract, and that $20 difference isn’t so important, your next contract may be 100 days, and it will be based on your previous contract. And most organizations will take into account what you earn elsewhere.
Does your rate seem high? Don’t forget – as a freelance consultant, you have to cover all your own benefits, and any enforced vacations from work. Essentially you are trading off benefits for freedom. You need to take into account taxes and benefits when considering your rate. Depending on your nationality and who your contract is with, you may or may not need to pay taxes. In contrast to full-time employees, you will also need to cover your own benefits, especially health insurance and retirement. Plow part of your earnings every year into a low-fee 401k. Warren Buffet suggests Vanguard S&P index funds and I agree with his advice. If you have a partner with an employee health insurance plan, get on that.
You can work overtime. Obviously, if you take on more contracts and work weekends your earnings will rise. Some organizations allow you to bill for a 7-day work week others for a 6-day work week while on site (i.e. working in country) or ‘on mission’ as it referred to by some organizations, as if international development professionals were modern-day secular missionaries preaching the gospel of improved living standards. Just make sure you do top quality work. Don’t double bill by charging different organizations twice. Be results-oriented and be ethical.
Your annual earnings will fluctuate. Ups and downs in earnings are inevitable for non-salaried workers. Over time you will, however, see an upward trend in your income as you get more work, add years of experience, and become better at negotiating your rate.
The daily rate isn’t everything. Despite the above, you may find yourself willing to accept a lower rate for a variety of reasons. Maybe you are trying to get your foot in the door, or you like the assignment, or the work conditions. Perhaps you are willing to earn less on this assignment because you’ll be working in a country you’ve always wanted to go to.
Some resources to consult
- Glassdoor. Review the salary ranges on their website, glassdoor.com, which aggregates salaries for different positions, by organization. The data is based on what people have posted themselves.
- Fellow consultants. The best source or information on what you can expect to earn is probably other consultants who work, or have worked in the field. (Don’t necessarily ask them about their own rates, however. Talking about one’s salary in the US remains an even bigger taboo than talking about one’s sex life!)
- Fee structure matrices. These may or may not be accessible to consultants for all organizations. But it never hurts to ask around. They will give you a benchmark.
- Devex. This organization describes itself as the “media platform for the development community.” It caters to organizations, companies and individuals working in international development. There is a monthly individual membership fee (starting at $9.50 at the time of this writing) and you can post your CV and look at job postings. Members have access to articles, including “Foreign aid salary spotlights,” which describes international development jobs and rates in a selection of countries.
* For more on negotiating your daily rate, see: How to negotiate your fee as a consultant: Figuring out what matters (footnote added July 30, 2019)
Nils Junge
March 29, 2017
Congressional Budget Office logo
The non-partisan Congressional Budget Office (CBO) last week released its cost estimate of the American Health Care Act, the Republicans’ plan to replace the Affordable Care Act, colloquially known as Obamacare.
The CBO looked at a range of impacts. The headline numbers from the CBO estimate are a reduction in the federal deficit between 2017 and 2026 by $337 billion and a total of 52 million uninsured by 2026 (with 14 million losing insurance next year). There’s something to like (deficit reduction) and something to dislike (loss of health insurance for millions), depending on where you stand on these issues. Without passing judgment on the significance of the potential effects of the new bill, let’s focus on the reaction of the bill’s backers, including the White House, to the CBO and its work.
Even before the CBO report was published on March 9, potshots were being taken at the normally highly respected office. Forbes characterized the them as a “pre-emptive, coordinated attack.” Joe Barton, a Republican former House Energy and Commerce Committee Chairman, had this to say about the CBO: “I don’t know what they do, they sit around and think great thoughts and everything on the issues…One of the things we need to do is reform the CBO folks.” And Gary Cohn, director of the White House National Economic Council said on Fox News that “in the past, the CBO score has really been meaningless.”
The reactions suggest that some supporters of “repeal and replace” already sensed that the new healthcare proposal would not follow Trump’s professed goal of providing all Americans with great healthcare at lower costs than Obamacare. It is also worth remembering that the CBO director, Keith Hall, was named to his post by Republicans. This doesn’t mean that the CBO always gets its numbers right. It doesn’t. But its analysis is transparent and explained in enough detail that one can understand how it reaches its conclusions.
As an evaluator, part of whose work involves estimating the impacts of policy reforms, I can sympathize with the CBO being targeted for attack. Conducting evaluations, which is essentially what the CBO has done in tallying the costs and benefits of replacing Obamacare, is a great way to lose friends and alienate people. Evaluators are never the most popular kids on the block. We don’t control pots of money, we aren’t trumpeting success stories, our job doesn’t involve being ingratiating in order to sell stuff. We dig around and find out what worked and what didn’t, who’s winning and who’s losing. It’s necessary (and hopefully useful) work, but it’s not a popularity contest. And evaluations always turn up shortcomings. Nobody’s perfect. As the messenger, you can expect to get (metaphorically) shot at.
At a minimum, people get a bit nervous when their organization or program is evaluated. Even if the client who commissions the evaluation outlines the questions they want answered, evaluators are still being allowed ‘inside’; they’re able to ask questions of pretty much anyone connected to or benefiting from the project. Good evaluators pry through reports, extract data from whatever sources they can get their hands on, and double check everything they hear. Sometimes, the evaluation can seem a lot like an investigation.
I’ve conducted evaluations all over the world, some of them under fairly hostile circumstances. Even if the main client wants to have evidence on the impacts of a reform, that doesn’t mean everyone wants to know. There are potential winners and losers who have a stake in the outcome of your evaluation. There are vested interests. Trade union representatives, for example, can be a tough bunch. I once worked on an evaluation of the potential impacts of a mine privatization in eastern Serbia. Layoffs were expected. When I conduct this type of work, it is my policy to meet with representatives of all the affected groups. In this case, everyone knew that the restructuring was going to lead to the loss of about 2,500 jobs. It was the task of my evaluation team to estimate what would happen to their income and job prospects afterwards. The concerns of the workers were legitimate and completely understandable from their perspective, even if the mine was dependent on tens of millions of dollars of budget support annually. My approach to dealing with the trade unions was to open a line of communication with them, and keep it open throughout the study preparation, fieldwork and reporting period. This involved meeting with them periodically, listening to their concerns, and explaining what we researchers were doing.
On a similar study, this one collecting evidence on the impact of downsizing Croatia’s shipbuilding industry, we had a very different experience. There was unfortunately not enough budget or time to meet with the trade union representatives more than once. The antagonism toward the evaluation was considerable. Fieldwork included conducting an employee survey in a room on the premises of the shipyards. In fact, our survey supervisor, a young Croatian woman, was asked by a shipyard manager to turn over the list of (randomly selected) employees she was interviewing. When she refused he locked the door to the room and threatened not to let her out unless she complied with his request. She resolutely stuck to her guns however, risking her safety and wellbeing in the name of evaluation ethics. Luckily, she was able to call someone in the Ministry from the locked room with her cell phone, and secure her own release. But it left her shaken. I have even heard of survey interviewers in some countries being detained and jailed for doing their work.
In some respects, evaluators are indeed like investigative reporters. That makes the work interesting, and occasionally risky. But the evaluator as an investigator is not really the ideal association you want to create. It can sound, well, threatening. Another, and more conducive analogy is that of evaluator as a “critical friend.” This concept was proposed a quarter century ago by Costa and Kallick in a 1993 article. They noted that critical friendship must begin with building trust, and go on to highlight the qualities that such a friend provides. That includes listening well, offering value judgments only when the learner (i.e. the client) asks for them, responding with integrity to the work, and advocating for the success of the work (p. 50). As an evaluator, you are not trying to establish guilt, or attack or push an agenda. You are there to help the organization or policy maker better understand the impacts of their programs or proposals, and improve them so that their goals can be attained.
Going back to the CBO’s report, it reads like a levelheaded, thoughtful piece of analysis. If its critics have a problem with it, you might expect them (at least in a less frenzied atmosphere) to respond by questioning its assumptions, or offering counter-evidence. When critical voices fail to do this, it is probably because they don’t have good answers.
This does not mean that, as evaluators, we can be smug. We live in a world where the idea of “evidence-based” does not have a strong hold on the public’s imagination, and is anathema to many politicians. We need to work harder, and use the evidence we have to tell a more compelling tale.
Nils Junge
March 21, 2017