An Illuminating Case: Mixed Methods and Reconciling Conflicting Findings

Warning – Contradictions Ahead!

I’m a big fan of using mixed methods in evaluation. That means combining qualitative and quantitative data, such as statistical data on household energy consumption and interview findings, where people reveal what it’s actually like to live with regular blackouts. This is not just because it’s always interesting to poke around at problems from different angles, but because the resulting analysis is generally much more layered and nuanced. Let me use a case study from a few years ago to illustrate what I’m talking about.

Applying different methods to the same problem is like stepping into the shoes of different blind men around an elephant, all trying to determine the nature of the creature. Instead of standing in front, holding the trunk and guessing it’s some kind of snake, or standing alongside, grabbing a leg and guessing it’s a tree, and so on, you test every dimension by shifting your position. A key advantage in using mixed methods is the ability to triangulate findings. Not only can you compare and cross-check your results – when done well, a more multi-layered, nuanced version of the underlying issues emerges. On the other hand, doing so can also complicate your life: there is always the risk of ending up with findings that don’t make a lot of sense.

Nonetheless, when findings derived from different methods not only highlight different attributes, but point in different directions, that is when the most interesting decision points and nuanced analysis can arise. Assuming the problem does not lie with the methodology or data collection itself (something which should always be checked), evaluators will need to do some probing to figure out why the data points in different directions.

We faced this issue as part of a 2004 World Bank-financed evaluation of the household impacts of electricity sector reforms in Moldova, using the ‘poverty and social impact analysis’ (PSIA) approach. The accompanying steep increase in electricity tariffs was seen by some as hurting the poor, and a reason to roll back reforms.

The data we analyzed showed a marked improvement in electricity consumption among the poorest 20% of households. However, the perception among poor households, gleaned through qualitative methods (focus groups) painted a less than rosy picture. We heard a lot of complaints from them. Based solely on the quantitative evidence, our report might have concluded that all was well, that concerns over hurting the poor with high prices were overblown. Conversely, if we had only used qualitative findings, we might have concluded that the reforms were indeed a negative for the poor.

Is raising electricity tariffs good or bad for the poor?

A bit of background: by 2003, with World Bank assistance, Moldova had privatized two-thirds of its electricity distribution network (the part of the grid which delivers electricity to customers). The aim was to improve sector performance by moving it to a commercial footing. Then, as now, Moldova was one of the poorest countries in Europe. A Spanish operator, Union Fenosa, had won the tender, with a pledge to invest $54 million in upgrading the infrastructure.

Prior to privatization, bills had gone unpaid, operating costs had not been covered, barter payments were common, and the network was severely degraded. Outside of Chisinau, the capital, most people could only count on a few hours of electricity per day. However, a Communist government had recently been elected and, with electricity tariffs rising steadily for years, some policy makers were now arguing that the reforms were having a punitive effect on poor households. There were fears among the donors that the privatization would be reversed, returning the distribution networks to state control. This is what triggered the study.

I was part of a team of Bank staff and consultants tasked with evaluating whether the poor were, in fact worse off now than before the reform.  As tariffs increased, were they consuming less electricity? Or were they perhaps cutting back on consumption in other areas? Either scenario would have pointed to a negative welfare effect.

The methods behind the madness

By matching billing data provided by the electricity with data on household expenditures (from the country’s Household Budget Survey), we were able to track consumption patterns over the 5-year period which coincided with tariff rises and privatization. Tariffs had begun climbing prior to privatization (to make the sector more attractive to potential bidders) and continued to afterwards. Over that period, they rose a whopping 300%, in nominal terms which translated into 26% in inflation-adjusted terms. This meant that the cost of electricity rose slightly faster than the average costs of other goods in the household consumption basket. That fact doesn’t provide a sufficient basis on which to base a conclusion about welfare.

With this in mind, we did not just analyze the data, we also talked to the population directly (thereby mixing our methods). A local research firm held 43 focus group discussions and 59 key informant interviews across the country. We basically wanted to hear about the users’, the electricity consumers’, perspective on these changes. Many focus group participants related how badly conditions had become during the, literally, dark years after independence.

“About two years ago a girl was raped, and now I don’t permit them to leave the house in the night,” reported a woman named Anea

“Everybody, children, the elderly, grown-ups, are affected by the darkness in the entrance hall and on the streets. It is very difficult to walk when we return home late in the evening. The roads are in a bad state, full of holes and one can easily fall down and break the neck,” Tamar said.

“In our entrance hall, the bag of an old woman was stolen, together with her pension for one month,” according to Elena

What did the data show? We found that over the period electricity consumption among the poorest 20% had risen by 14.6%, even as electricity tariffs rose more than 300% in nominal terms and 26% in real terms. Of course, people weren’t consuming more electricity because the cost went up; they were consuming more because their incomes were increasing. We also found that among all other households (which we designated the ‘non-poor’), consumption rose by 3.2% (i.e. the poor experienced bigger consumption gains). Perhaps more importantly, there were no more rolling blackouts. From having on average 4 hours of electricity per day, everyone now enjoyed electricity service 24/7. Based on the data, we could confidently say that the poor were not being hurt by privatization with the concomitant tariff increases. (We did not seek to determine whether the electricity reform made the poor better off, a rather more complex and somewhat subjective question.)

Data vs. perceptions

So was the government wrong? Were the poor pleased with the changes? Hardly.  Although many people acknowledged that things were better – they now had electricity 24/7 – the overriding message that emerged from focus group discussions was that many people were unsatisfied. They did not feel better off. They certainly did not express undying gratitude for the reforms. They complained about costs going up, about having to save, and about quality issues (e.g. voltage fluctuations). And they were unhappy about what they perceived as Union Fenosa’s excesses. The company had built itself a multi-story headquarters building, bought new company vehicles, sent out electricity bills using colored ink on high quality paper (replacing the cheap brown paper used previously, a Soviet-legacy. To top it off, it was seen to be spending money frivolously by paying for schoolchildren to go on outings to the circus and other measures aimed at burnishing its corporate image. All of this did not seem to impress the average Moldovan. In her view, the company should instead have kept its tariffs lower.

Returning to our quantitative findings in the light of public perceptions, one particular piece of information was of critical importance for setting the context: electricity consumption levels in Moldova at the time of privatization were extremely low. The average household consumed just 50 kWh per month, the equivalent of a couple of light bulbs and a TV. This was close to one third the Eastern European average, and a small fraction of developed country consumption levels. Many people (exactly half, of course) fall below the average increase in electricity consumption, and many had to reduce

According to Vasile: I think that the reform had a positive impact, the regular blackouts prior to the reform practically stopped the activity in many fields. However, there is the price issue.

“During winters I unplug the refrigerator, hence I pay by 35 lei less per month” a participated named Victoria said.

Did these findings invalidate our quantitative findings? I would say no, they were complementary. They helped explain why the government had been hearing mostly negative feedback about the privatization. While we found that the poor were not being hurt financially, it would have been a stretch to say that privatization had been a boon for them. The difference between 50 and 57 kWh per month is not exactly a game changer or cause for rejoicing if you’re a poor household.

This study illustrated the importance of using different methods of inquiry and, conversely, the risks of not doing so. If we had used one method only, we would have arrived at rather different conclusions – one too rosy (based on the consumption data) and one two bleak (based on people’s perceptions). Neither finding was right or wrong. They simply told different sides of the same story, and helped explain two very different but valid perspectives on the complex and charged (no pun intended) issue of privatization.

You can read the full report here: World Bank 2004 Moldova Electricity Reforms