Skip to content

The First Law of Development Stats: Whatever our Bizarre Methodology, We make Africa look Worse

UPDATE: Just received notice of drastic punishment for this post: invited to join HDR 2011 Advisory Panel

I’ve complained previously about how design of the UN Millennium Development Goals make sub-Saharan Africa look worse than it really is. Now I realize that UNDP’s new Human Development Report (HDR) does the same thing. Not alleging any conspiracy here, it seems unintentional, but is then not caught because … well we all know Africa is supposed to look terrible.[1]

My HDR education comes from Martin Ravallion, who has a great new paper on the new methodology of the Human Development Index (HDI). (Martin does not mention the Africa angle, but provides the necessary insights described below).

Of course, UNDP has an impregnable position: while their results get huge publicity, the methodology behind the results is interesting to approximately 3 people.  As an avid promoter of hopeless causes, here goes…

The biggest change in method was that the new HDI is a geometric average rather than a normal (additive) average. Geometric average means you multiply the separate indices (each ranging between 0 and 1) for income, life expectancy, and education together and then take the cube root (I know your pulse starts to race here…)

Now, students, please notice the following: if one of these indices is zero, then the new HDI will be zero, regardless of how great the other indices are. The same mostly applies if one of the indices is close to zero. The new HDI has a “you’re only as strong as your weakest link” property, and in practice the weakest link turns out to be very low income (and guess which region has very low income).

So, as Martin noted, the new HDI relative to the old HDI penalizes countries with very low income compared to decent numbers on life expectancy and education. One reason I think this is unintentional is that these are exactly the cases that the HDR used to celebrate! The biggest losers here are Zimbabwe, Liberia, DR Congo, Burundi, Madagascar, Malawi, Niger, and Togo.

Martin makes the “decent life expectancy doesn’t help you if you have low income” point in a different way: the new HDI has vastly different numbers for the value of life between poor and rich countries. Martin had previously made this criticism of the old HDI in a paper published in 1997, which Aid Watch covered in a previous post. The HDR addressed this criticism by making the problem much worse. Previously we were all whining about differences in the value of life of 70 times between rich and poor – now it’s a differential of 17,000 to one. Sorry, Zimbabweans, UNDP thinks your lives are worth 50 cents.

But wait, Africa has another GREAT chance to perform well —  the HDR also gives mucho publicity to the “top movers” in HDI over 1970-2010, ranked in order of percentage increase. My old MDG paper mentioned above said Africa would look better on percentage increases in health or education indicators.  Indeed, Ethiopia, Burkina Faso, Niger, Mali, and Burundia all had more than a 300% increase in educational enrollments (using the UNDP’s own data) from 1970 to 2010.

So naturally, among the champion improvers are … Oman and Nepal … and no sub-Saharan African countries in the top 10. What happened?

In yet another twist, the HDR ranked the top improvers measured as deviations from the average growth in HDI of others at similar initial HDI in 1970. Since almost all of the bottom ranks of the HDI are sub-Saharan African (exacerbated by the above “weakest link” methodology), Africa will only do well if it does better on average than – Africa.

If we forget the deviations thing, and just rank by growth in the HDI from 1970 to 2010, then sub-Saharan Africa would get 6 out of the top 10 improvers.

If you have read this far, you get a medal. So what’s the lesson of all this mumbo-jumbo about methodology? Maybe you could make a case for the new methodology, but at the very least it’s clear that obscure choices of method make a big difference in who you celebrate – and who you make look bad. And way too often, Africa winds up in the latter category.

Postscript: we want to thank UNDP for generously making all their data and methodology available to us even though they knew we were critical, because they also generously gave reactions to a preliminary draft based on an earlier dataset we downloaded from the HDR website. They did not change our minds and the new dataset confirmed our earlier results. But we give them great credit for constructive engagement. The paper that describes their methodology is here.


[1] This post uses the words “Africa” and “sub-Saharan Africa” interchangeably, following common development-speak. North Africa is in a very different situation from that described here.

Related posts:
The First Law of Development Stats: Whatever our Bizarre Methodology, We make Africa look Worse
What the New HDI tells us about Africa
Human Development Index Debate Round 2: UNDP, you’re still wrong

This entry was posted in Data and statistics and tagged . Bookmark the permalink. Follow any comments here with the RSS feed for this post. Both comments and trackbacks are currently closed.

19 Comments

  1. Adam Baker wrote:

    Away with all indices! If policy-makers aren’t smart enough to understand multi-variate data, maybe they shouldn’t be making policy.

    Posted December 2, 2010 at 1:20 am | Permalink
  2. This is the best Aid Watch post in months. Great stuff.

    Posted December 2, 2010 at 3:44 am | Permalink
  3. geckonomist wrote:

    Mr. Ravallion, that was the one who found it all important to exactly count the poor, such that he could tell them they’re not alone in their misery, wasn’t it?

    I am glad he found a new mission in life to justify his World Bank director salary: discussing the calculation of a number between 0 and 1 that no poor person will ever care about.

    A lifetime achievement award should be awarded here.

    I am also happy the discussion of this non-issue justifies the existence of Prof. Easterly’s talking shop.

    Posted December 2, 2010 at 3:44 am | Permalink
  4. Matthias wrote:

    I guess any index of development is going to produce a partial, incomplete picture of what success and failure are. Any such index is open to criticism.

    The HDI emerged as a response to the primacy of GDP p/ capita which, of course, also purported to tell winners from losers, and did it pretty badly. I have Ravaillon’s paper but didn’t really get around to reading it yet. However, there are two thoughts that come to my mind regarding Easterly’s rendering of the criticism:

    1) indexes aren’t built to show us that Africa — or any other region — is doing better; so if HDI, or GDP or the old HDI show that Africa is not doing well, that’s not a useful critique of the index.

    2) I would guess that the new HDI should be assessed in terms of how it responds to valid criticism of the older version. Easterly complains, for instance, that the index is now geometric rather than additive. Why has that come to pass? What criticism of additivity spurred the change? It is indeed obscene that ‘life is valued less’ in Zimbabwe. But wasn’t that the case before? Wasn’t it he case that one could ‘exchange’ losses in life expectancy with increases in GDP to retain the same HDI? The issue here seems to be one of degree…

    Is there anything gained by the new methodology? We wouldn’t know by reading Easterly or, apparently, Ravaillon.

    Finally, should we use any index as an overall assessment of development and progress, Bill? If yes, which one?

    Posted December 2, 2010 at 5:10 am | Permalink
  5. David Zetland wrote:

    In conclusion, UNDP offers yet another reason why it should be shut down…

    Posted December 2, 2010 at 9:22 am | Permalink
  6. V K Madhavan wrote:

    If the purpose of the HDR is to recognise performance and provide incentives for those countries that display significant improvements, then the methodology would have done a grave injustice to countries in Africa. At the risk of sounding cynical am not sure whether the HDR actually has any impact on the allocation of resources even by UN agencies. Worse, I think countries showing improvements are often abandoned too early thereby penalising them for real attempts at change.

    Posted December 2, 2010 at 10:46 am | Permalink
  7. William Easterly wrote:

    Matthias, Sorry I thought I WAS clear on this — the new HDI not only fails to respond to the old Ravallion criticism about different valuations of life, it made it MUCH worse – which part of 17,000 to 1 vs. the old 70 to 1 was I not clear on? I’m happy to make it clearer if you can explain what’s fuzzy.

    On the primacy of income, the new HDI has effectively reversed course and restored the primacy of income at least for sub-Saharan Africa, so it is really unclear what the reason was for the switch in method.

    Thanks for the comment and any further guidance you can provide. Bill

    Posted December 2, 2010 at 10:48 am | Permalink
  8. Bernard Lowther wrote:

    This has parallels with the World Bank/IFC ‘s ranking of ‘Ease of Doing Business’ (which is not an index because it doesn’t have any cardinal values, only rankings). Although there are tons of problems with EODB’s data-gathering, assumptions and explanatory claims, the ranking process per se is easy to understand – first they rank all countries on 10 specific indicators, and then they take an average of the 10 rankings to create the overall ranking. Problem is, this ranking is quite stable year over year, especially at the top, and you can’t make headlines with “Singapore is top rated for the fifth year in a row”. So, like HDI, to jazz it up they identify the countries that supposedly “improved the most” since the previous year, involving some baroque combination of ‘depth’ (big improvements) and ‘breadth’ (improvements in several different indicators.) This allows them to call different countries up onto the stage each year for a round of applause, which is the main point of the exercise.
    There is no one perfect method for creating these indicators and indices — it all depends what you are trying to accomplish.
    So for HDI, I’m wondering what is the underlying institutional incentive for UNDP to create a methodology that delivers these particular results.

    Posted December 2, 2010 at 11:54 am | Permalink
  9. Alanna wrote:

    Okay, I didn’t take the tough econ course in grad school. Just wanted to get that out there. And I don’t understand the 17,000 to 1 point.

    That being said, I am curious about a couple of things. What was UNDP’s justification for the new methodology? Is there an explanation? Secondly, on a gut level – do the new high performers make sense as high performers? Do they “deserve” the ranking?

    Posted December 2, 2010 at 12:04 pm | Permalink
  10. Alanna wrote:

    Okay, one more point. Central Asia and the former USSR in general tend to rank too high on the HDI because of impressive literacy rates. It seems to be that switching to a geometric average might help to ameliorate that.

    Posted December 2, 2010 at 12:06 pm | Permalink
  11. William Easterly wrote:

    Alanna, let me try again on the 17,000 point. It comes back to the income index between 0 and 1 multiplying the life expectancy index. The effect on the HDI of higher life expectancy therefore depends on income. If the income index was literally zero, then an increase in life expectancy would have no effect on the HDI (see the “weakest link” argument again above). So in such a case, the UNDP is implicitly valuing life at zero. If income was close to one, on the other hand, life expectancy would have a big effect on the HDI so UNDP is valuing life a lot more in a rich country (actually INFINITELY more than in a zero income index country). If we compare the richest country (close to 1 or at 1) and the poorest country (Zimbabwe, very close to zero), the difference in valuing a life turns out to be a ratio of about 17,000. thanks for your comment Alanna. Bill

    Posted December 2, 2010 at 12:34 pm | Permalink
  12. Moussa P. Blimpo wrote:

    It is great to read this post because this was exercise 1 of the last homework that I gave to my undergrad Development Econ students. They did pretty well evaluating the new HDI.

    There are other challenges/issues with the index, such as the fact that if a country grows wealthier than the maximum income per capita ever observed, then the HDI of everyone else goes down due to that.

    But I think the most important of all this is to continue to improve what we think would eventually define the level of development.

    I am sure that many Development economists don’t really bother with the methodology behind this index even though it is heavily quoted, and because it is simple, everyone think they understand it. That is why I assigned it to my students and they loved it.

    Posted December 2, 2010 at 6:12 pm | Permalink
  13. MJ wrote:

    Leaving aside the unfortunate implication about the different value of life, it strikes me that there are times when a geometric mean could be v useful in constructing an index such as this. For instance, a business weighing up an investment decision will be most likely to be swayed by your lowest rating rather than the highest, e.g. high speed internet doesn’t count for much if you cannot reliably keep the lights on. Educated workers or healthy workers (with healthy families)? Both please!

    Should an index such as this be the last word in any determination of success in poverty alleviation or economic development? No, clearly not, but it could still fulfil a useful role, perhaps as a counterpoint to another index which emphasises the positives (invert all variables, take the geometric mean, then invert again).

    Posted December 3, 2010 at 4:30 am | Permalink
  14. Matthias wrote:

    Bill, you were very clear as to the fact that the new HDI aggravated the ‘substitutability’ issue already existent in the old HDI. No fuzziness there. What I had tried to point out was that the difference is of degree, not of kind.

    I recall reading, a couple of years ago, that one rationale for multiplying component indicators was to penalize underperformance (alternatively, to only consider high performance a situation in which performance is good in all dimensions). It was an understandable argument because there had been tremendous convergence in HDI since 1990 (as this gapminder graph shows there has been plenty of convergence in HDI, even in Africa, but movement on GDP has been much less pronounced). The idea of multiplying was then to underscore areas where states were lagging, rather than have the composite index only reflect the strongest indicator (which often wasn’t GDP, because the ‘wealth dimension’ was seriously discounted after a given cutting point). (HDR2010 on imperfect substitutability, see also section 2 of this)

    You state that [o]n the primacy of income, the new HDI has effectively reversed course and restored the primacy of income at least for sub-Saharan Africa which again seems to imply that the main design criterion for a multidimensional indicator should be ‘how good it makes sub-Saharan Africa look’. The HDI is supposed to rank most countries, it is not region-specific. Had it been region-specific (or income-group specific, for the matter), it would have been designed quite differently, and would — even with identical design — had rendered a totally different picture (as the HDI uses world-wide minimum and maximum values).

    It is debatable whether the best way to highlight deprivation is to include it in the design of the HDI. In the pastn the HDRO had used other indexes, such as Capability poverty, HPI-1 and -2, etc. Maybe that was a better approach, I don’t really have a strong opinion either way.

    But the broader question remains. We all know that no single index, however sophisticated, will give us a complete picture of the state of development of a country, region or municipality. At the same time, indicators are necessary to assess progress (however we define it), and as long as we have multiple indicators, some form of aggregation or ‘possible-world ranking’ mechanism will be necessary. So what should we use?

    Posted December 3, 2010 at 9:47 am | Permalink
  15. Matthias wrote:

    By the way, thank you for answering!

    Posted December 3, 2010 at 9:49 am | Permalink
  16. Amanda wrote:

    A note on SSA countries looking good on education and health indicators when looking at improvements over time, particularly with health:

    When looking only at increase in the percent of a population covered by an intervention (for simplicity, immunization), countries that start at lower levels (like many in SSA; for example, going from 20% coverage to 80% coverage) have more space to move than those starting higher (from 70% to 90%). Looking at the change in percent not covered (as the MDG folks do for immunization) deals with this issue a bit.

    Thanks for a great post – the detail is much appreciated.

    Posted December 3, 2010 at 11:45 am | Permalink
  17. William Easterly wrote:

    Matthias, Great dialogue.

    I think you are misunderstanding my concern to “not artificially make Africa look bad” as “try to make Africa look good.” The point is that some arbitrary changes were made for which there is no obvious justification (or a bad choice like using deviations from average growth at each HDI level rather than just growth of HDI) that had the unintended consequence of making Africa look worse FOR NO GOOD REASON.

    just for the record, there is a paper by Chakravarty 2003 (cited in the Ravallion paper) that makes a convincing argument that an HDI with reasonable properties has to be additive, not geometric.

    Amanda, your point is a good one. I discussed it in the Africa MDG paper cited as one of the first links in the blog post above.

    thanks, Bill

    Posted December 3, 2010 at 4:43 pm | Permalink
  18. Thomas wrote:

    Dear Mr. Easterly,

    Could you please let us know your thoughts on the new Inequality-adjusted HDI in this year’s HDR?

    Thank you.

    Posted December 4, 2010 at 9:56 am | Permalink
  19. Alex O wrote:

    It is a little worrying you feel the need to thank UNDP for giving you access to the data; isn’t the data publicly funded? isn’t everything the UN does publicly funded? Hence the need for graciousness is only a symptom of poor (no) UN(DP) accountability.

    Posted December 13, 2010 at 5:43 pm | Permalink

11 Trackbacks

  1. [...] This post was mentioned on Twitter by Robert Went and OpenEye Group, Conduit Journal. Conduit Journal said: The First Law of Development Stats: Whatever our Bizarre Methodology, We make Africa look Worse http://bit.ly/fCd7zg [...]

  2. [...] Some methodological complaints about the U.N. Human Development Index. [...]

  3. By Sunday paper | Osmosis on December 2, 2010 at 6:38 pm

    [...] The First Law of Development Stats: Whatever our Bizarre Methodology, We make Africa look Worse – Combined with the research behind the Daily chart: Money and happiness | The Economist, I wonder whether we should not revert to the simple “ one issue” statistics. If happiness is just function of a log(income) we can raise income or distribute it to raise happiness. On a different note, we continue to diminish child mortality, corruption etc. It is just not right to let the bureaucrats decide how many years of education is one dead child worth. [...]

  4. [...] 2) How the U.N. Development Program’s Human Development Report makes sub-Saharan Africa look worse than it really is. [...]

  5. By In praise of… « Waylaid Dialectic on December 4, 2010 at 3:37 am

    [...] it’s not. It’s simply a really useful education about development. Bill and Laura, thanks for your post on the HDR. Leave a Comment LikeBe the first to like this [...]

  6. [...] The First Law of Development Stats: Whatever our Bizarre … This entry was posted in News. Bookmark the permalink. ← Previous Post [...]

  7. By What the New HDI tells us about Africa on December 8, 2010 at 3:10 pm

    [...] 8, 2010by Francisco Rodríguez, Head of Research at the Human Development Report OfficeIn a post published last Thursday, Bill Easterly and Laura Freschi criticize the new formula for the Human Development Index (HDI) [...]

  8. By New HDI | Resource Curse on December 8, 2010 at 7:16 pm

    [...] blog recently had a terrific post explaining some of the downsides of the new HDI [...]

  9. [...] on this here and here. For a summary of this see [...]

  10. [...] Continue reading on Aid Watch [...]

  11. [...] Group at the World Bank Francisco Rodriguez has defended the HDI against recent criticisms by Bill Easterly and Laura Freschi, who drew in part on my new paper, “Troubling Tradeoffs in the Human Development [...]