Numbers are used a lot to make a point or serve as evidence. When you come across a number, in a report or on the news, always remember its meaning depends almost entirely on its context.
Certainly, numbers often seem inherently more reliable than descriptions. In the social sciences, there is a distinct preference for quantitative vs. qualitative findings. There is a bias, but perhaps it’s justified?
When you are presented with a number, there seems to be less intermediation. Numbers don’t lie, they say. But then the phrase “lies, damned lies and statistics,” often attributed to 19th century British Prime Minister Disraeli, is also pretty popular. It highlights inherent distrust toward statisticians and politicians. Because even if on the face of it numbers seem to be objective and verifiable, often there is quite a bit of intermediation behind their production.
While intermediation (statistical analysis) is acceptable, indeed necessary in order to collect and organize information and ensure that the results represent what they are supposed to, manipulation is not.
Forgive me for being blindingly obvious, but we need hard numbers. They are extremely useful. They allow us to have a common basis for discussion.
On a side note, numbers even serve different purposes. There are cardinal numbers (indicators of quantity, e.g. how many cups of flour in a bread recipe), there are ordinal numbers (indicating the order of things, who came in first place), and there are nominal numbers (for identifying things, like your seat number in the theatre). I’m going to focus on cardinal numbers for now.
Remember that a number is just a number. Its meaning (good or bad, important or trivial, expected or unexpected) is imposed externally. Its value is not inherent but rather depends on context. The context is what tells the story.
It is not enough to say that the unemployment rate was low. The description “low” can mean any number of things. Your first reaction should be “how low?” and “low compared to what?” The thing is, plugging in an actual number only gets you part of the way there. Let’s say unemployment was 7%. How does that compare to last year? It may be low if last year it was 9%, or it may be high if it was 5%. Or if in similar economies is above 10%, 7% looks pretty good, whereas if everywhere else it is below 6%, it looks less good.
Not only are comparisons across time and space important, but comparisons with purely fictitious numbers also matter. What I’m referring to by this is the number in your head, which doesn’t’ actually exist, right? This is the number you were expecting.
When unemployment numbers are predicted to fall, but they actually go up, the stock market may fall. When you expect to be offered a salary of $60,000 but then get an offer of $80,000, that will make you very happy. The age of newly elected French President Emanuel Macron’s wife, Brigitte (64) is not interesting – there are many women in their sixties in France – until you know that he is only 39. Their age difference is unexpected because of the gender markers. Its unexpectedness is in inverse relation to the age difference between Donald Trump and his wife, Melania. Who wouldn’t expect a billionaire property developer to have a wife a quarter century his junior?
Placing bets is an extreme example of what we can call the arbitrage between expected and actual numbers (or current and future numbers, if you will.) So a lot of money is often at stake.
You can apply this type of thinking to pretty much everything – from the price of a new refrigerator, to your annual salary to the number of people without access to drinking water. Context is everything. What that means is that reference points are critical. Numbers floating out there on their own don’t mean much.
Of course, you want to know how a number is derived. You should ask how they are collected, what they represent (often a matter of definition), and whether they truly represent what they are meant to? Obviously, in the case of the Brexit referendum and the last US election, the numbers which the vast majority of polls gave us did not accurately represent the people who actually voted.
Statistics can be presented misleadingly even when they are accurate. One of the most common issues perpetuated by the media and researchers themselves, involves comparing differences to a (seemingly arbitrary) base, or even not revealing the actual base. For example, findings reported in the American Journal of Epidemiology indicated that drinking more coffee reduced the chances of dying from oral/pharyngeal cancer by 49%. Sounds impressive, right? But you also need to know that your chances of dying from this type of cancer are about 0.09%, i.e. less than one in a 1,000. They weren’t high to begin with. And what about all the other cancers? Did they check those? And what about negative effects? Maybe those cancel out the positive ones.
Responsible reporting of statistics, whether in academic journals or the media, involves providing the reader or viewer with meaningful context. When you see a number, ask yourself the following questions:
- What does this actually represent? How is the number defined?
- How does it compare with alternatives, similar situations?
- How does it compare with the past?
- How does it compare with what was expected? Does the number fall in line with a trend or go against the trend?
- What are the possible interests of the person or organization in sharing this number? Journalists and evaluators have a code of ethics they are supposed to follow.
Ideology vs. evidence
The tension between belief and evidence has probably been around ever since humans could sense the world and formulate thoughts. It has certainly been documented for several thousand years. In the Bible, we have the original ‘doubting Thomas’, who would not accept that Jesus had come back to life until he could put his hand on the wound in Jesus’ side. Galileo was put on trial by the powerful Church for arguing, based on scientific evidence, that the Earth circled the Sun and not the other way around.
The issue goes beyond belief. A set of beliefs tend to form an ideology, which typically concern human nature, religion, politics, or the economy.
For various reasons, people hold certain theories of how the world works, or should work. People may be influenced by tradition, tribe (the community to which they belong) or an authority they revere (like their church, or their president). Belief systems can have enormous consequences. At the extreme end, people have died for their beliefs – consider the early Christians refusing to renounce their faith and being fed to the lions, or young men volunteering to go to Syria and join ISIS. But even relatively anodyne beliefs about the world can have tremendous social impacts. For example, the American belief that this is the land of opportunity and if you don’t succeed you only have yourself to blame. In fact, there is a strong counter set of beliefs: that it is mostly those lucky enough to be born into the right (privileged) circumstances who actually enjoy those boundless opportunities…There are plenty of studies finding that children born into poverty have far fewer chances in life.
With respect to the current tensions between beliefs and evidence, there are at least two things worth noting.
Overwhelmed by too much information?
First, we live in an age saturated with information, which would make you think that evidence is readily accessible, and that virtually anyone can find at out information with a little web-browsing. Yet the rise of conspiracy theories, fake news, and the bifurcation into ideological communities in many countries defy this. The almost infinite wealth of information out there is also good cast doubting on evidence and supplying hermetic communities with a tailor-made set of facts that align with their beliefs. It has become easy to engage in selective perception. Information that doesn’t conform with what you think you already know can be ignored, devalued, or diluted with misinformation.
A bipartisan issue
Second, disregard for evidence or facts seems to be prevalent at both ends of the political spectrum. Both rightwing and the leftwing elements in US society (and possibly elsewhere) have embraced ideology at the expense of evidence. Facts that don’t fit are belittled or disregarded.
Many on the right, for example, scorn the science and statistics which show that the earth is more than roughly 6,000 years old, despite the geological evidence; that there is a near consensus among scientists that global warming is an actual, man-made phenomenon, or that gun ownership significantly increases risks to one’s life and those nearby. On the left, there is a rise of intolerance toward free speech on university campuses, which muzzles the expression of opinions deemed contrary to prevailing norms. There seems to be a prevailing wisdom that the college campus should be a protected ‘safe space’ (as if it were kindergarten) and that students need to be sheltered from ideas they disagree with or find uncomfortable – hence all those trigger warnings. Both right and left extremes seem to have become biased against any evidence which goes against their prevailing orthodoxy.
Living with belief and paying attention to evidence
Almost all humans believe things, or believe in things. This doesn’t mean we have to let beliefs cloud our judgment. If you believe in God, you generally still don’t take stupid risks because you believe He loves you and won’t let anything bad happen. Most people try to pay attention to their personal safety – staying on the sidewalk, driving carefully, not handling poisonous snakes – whether they believe in guardian angels or not.
Even evaluators believe things. We’re not robots. But as professionals we’re essentially paid to put our personal preferences aside and use our brains to objectively process large amounts of information.
Try this at home
If you are an evaluator, or just someone interested in getting closer rather than farther from the truth, you need to be aware of your beliefs and not let them interfere with your professional judgment. You need to maintain a boundary.
Here are a few things that I try to do to stay ‘conceptually sober’, i.e. not under the influence of beliefs, while on the job:
- First, recognize your biases. Monitor yourself as you collect and analyze and present information. It is easy to skew findings to fit in with your hypothesis, or ignore them. Unfortunately academics do this all the time, by not publishing research findings that aren’t in line with their hypothesis, or simply interesting (publication bias)
- Consider alternative causal explanations. You may have a good story that makes intuitive sense. You may have evidence (you better have evidence if you’re doing research). But always think about and look for evidence that might offer different explanations.
- When you have findings that seemingly contradict each other, dig deeper to find an explanation. Maybe your data is inaccurate, or your sampling method was poor. But maybe different sources are revealing different things, or people you are interviewing are interpreting or defining events in different ways.
- Double check what you hear from people when you’re doing research. If you hear pretty much the same thing from a lot of different that should give you confidence. If I hear something new or unusual from someone, I’ll ask others to corroborate.
- Follow a guiding principle. It could be something like: “To present what happened as accurately and fairly as possible.”
- Be curious. Find out as much as you can about a topic. Invest yourself in finding out more. Don’t be satisfied with the most obvious answers.