The information pyramid

We are swimming in a sea of information

Like other forms of inquiry, evaluation involves sorting, filtering and distilling information in order to communicate something of importance. (Academics, journalists, attorneys, and private investigators do this, too.) When the work is done, you want to be able to present your findings in a clear, convincing, and attractive manner for easy consumption.

The problem is, there is a vast amount of information out there.  It can easily overwhelm. With the Internet entering its mature phase, we swim in an information glut. I leave for another time a discussion on the differences between data, information, knowledge, intelligence, and wisdom, except to say that (from what I can tell, anyway) there is a lot less wisdom than there is data in the world…

A big part of your job, if you are an evaluator, is to know what information you need, and where and how to find it. To do this effectively, you want to be able to zero in on the essential stuff, while still being open to any interesting findings you may not have considered.

Let the purpose of the evaluation be your guide. Keep the reason you are searching at the front of your mind. Perhaps you seek to understand how well a program has built the capacity of agronomists to introduce innovative irrigation techniques to farmers? In that case, keep your focus on factors that may have a direct bearing on capacity building efforts, while limiting the amount of time you spend on learning about other things. Take note of them, but try not to let them distract you from the main question.

As an evaluator, you do not have the luxury of time that you would if, say, you worked in academia, to produce a dissertation or journal article (often years!). Evaluation, which is often about collecting and applying evidence to problems (in programs, policies, etc.) is relatively fast-paced. So even though you must review the literature, reports and data relevant to your evaluation, you simply will never have the time to read every word. You need to prioritize what you read and develop the ability to scan a document for what is essential.

When it comes to writing, you again will need to be strict with yourself. Avoid padding your reports with unessential information. Have you ever read a report (or a section of report) where you don’t understand what the point is? Have you found yourself asking, why am I reading this? Don’t put your readers through that.

A hierarchy of information

Imagine now that information and its offspring, exist in the form of a pyramid.

At the bottom of the pyramid is, let’s say, all the information in the world. Every piece of observable and non-observable phenomenon, from myriad perspectives. This amounts to untold trillions of bits of data, which are constantly accumulating and constantly changing. Think of this as an ocean of information. For all intents and purposes, this ocean is infinite and growing, much like the universe. It cannot be encompassed. All you can do is dip a sieve into these waters and try to collect what is most suitable to your purposes.

At the next level up is all the available information and knowledge. This is any information that has been processed somehow, whether printed or digital, spoken or written. Much, but far from all of it, is searchable using an Internet search engine. It is still a huge, overwhelming and unwieldy amount. But it has at least been produced by someone. It must also have some meaning, which is why I combine it with the concept of knowledge.

Next comes all the topical information that is out there. Maybe you’re writing about the electricity sector, or artificial intelligence, or breast-feeding. Depending on the area you are looking at, you will still find a plenty to review, and many experts, authors and practitioners you could talk to. If you are writing a general overview or introduction to these topics, you would synthesize all of this. Generally, however, you will not have such a broad focus.

Next comes the information that addresses your subject. It will be quite narrow in focus. For example, what is the impact on the poor of rising electricity tariffs? What does the introduction of artificial intelligence mean for workers in the fast food industry? What is the correlation between breastfeeding and the immune system? Now we are closer to where we want to be. It is still more information than you need, but the amount is manageable. You will only draw on the research and practices and reports that exist, plus any new primary data that you have distilled.

The next level in our pyramid is all the data and information collected for the specific purpose of the evaluation. This is the material you have reviewed with the aim of understanding your subject and informing your audience. It may include a database with thousands of observations and a hundred variables or more. You may have a bibliography of dozens or hundreds of sources. You may have hundreds of hours of interview or focus group discussion recordings. This is your personal store of information and it should, ideally, all be somehow relevant to the purpose of your evaluation.

Still higher up and narrower in scope are the evaluation findings. This is where the rubber hits the road. The findings, which normally come with conclusions and recommendations, are the core information, which you have transformed into knowledge. This is what answers the evaluation questions and backs them up with evidence. In the evaluation world, reports should generally not be longer than 20-30 pages, excluding annexes. That is about the amount of detail which specialist readers who are interested in your subject can stomach.

The summary findings, which includes the executive summary of a report, and may also exist in a standalone short note or slide presentation, is how the essence of the report is presented. This is what most people will read or watch. If the findings are a distillation of all the information you have collected, the summary findings are a distillation your broader findings. As a rule, the length should be about 10 percent of the full report, from 2 to 5 written pages maximum, or no more than 10-30 slides.

Finally, the main story. This is the quick one minute story you tell your significant other or friends or colleagues who ask what you learned, without boring them with all the details. It could be in the form of a few paragraphs and bullet points that result in a one-page policy note that goes to the Minister of Energy, for example, as a policy brief. For example, “We found that most of the poor didn’t suffer as a result of electricity tariff increases because electricity expenditures fell as a share of their total expenditures. And all households now have 24 hour service;” or “We project that artificial intelligence will eliminate, on average, one job per restaurant, while customers have shown a preference for interacting with humans when they order fast food;” or “Breastfeeding was shown to reduce the incidence of illness in infants under 5 if they were weaned only after x months.”

And with that we have reached the pinnacle of our pyramid. Time for the next project.

Post edited July 1, 2019