Skip to content

“Proofiness:” trashing back on FAO hunger numbers

Just before the big UN meetings here in New York around the Millennium Development Goals, the FAO released new world hunger numbers, and Aid Watch listed reasons to worry that these numbers were “made up.”

A blog post from Oxfam GB’s Duncan Green called our post “lazy and supercilious,” with the amusing headline “Easterly trashed.”  The accusation that I am “lazy” struck a raw nerve, and so I have responded forcefully by asking Laura to do more work.

A closer look at the FAO’s documents, along with information provided by smart Aid Watch commenters as well as the FAO’s own senior economist David Dawe validates, rather than “trashes,” many of the concerns Aid Watch raised.

For one, the methodology for the FAO survey numbers does not actually directly measure malnutrition but tries to estimate it indirectly based on a model of human calorie requirements and food availability and distribution:

From the total calories available, total calories needed for a given population, and the distribution of calories, one can calculate the number of people who are below the minimum energy requirement, and this is the number of undernourished people.

A modeled number is NOT the same as directly measuring malnutrition (as the WDI anthropometric numbers cited in the previous post attempt to do). Is the model correct? How did they test it? A model has many assumptions and parameters, which are inevitably less than 100 percent reliable. All of these make the modeled numbers subject to a LOT of uncertainty. Has FAO made any attempt to quantify the uncertainty? Have they tried comparing their estimates to the anthropometric measures in WDI?

Second, according to the FAO’s downloadable data charts, this exercise was last carried out in 2005-2007. These survey numbers are available for every country in the database (176 in all). The data tables tell us that while there is no country-level data for Iraq, Afghanistan, Somalia or Papua New Guinea, they are included in regional estimates, and there are country-level entries for places like Sudan, Zimbabwe and Libya for each three-year data collection period going back to 1990.

Neither of the most recent FAO State of Food Insecurity reports, from 2009 or 2008, includes discussions of the methodology for the 2005-2007 surveys. And neither explains how the 2008 figures were obtained. The 2009 report’s tables list as sources UN population data from 2006, and “FAO estimates” for undernourishment.

Third, the estimates for 2009 and 2010 are not only based on very indirect and noisy links between capital flows, imports, terms of trade and food availability, but the numbers for the former are not real numbers but based on USDA projected scenarios using IMF estimates for quantities that are notoriously difficult to estimate or project.

The comments from FAO economist David Dawe suggest (quite logically) that the economists, statisticians and policy-makers responsible for the FAO numbers are well aware of the drawbacks of the methodology they’ve chosen to produce both the survey year data and the estimates for years with no surveys at all; entire conferences and volumes are devoted to debating how to measure food deprivation.

Of course, none of this ambiguity and caution makes it into the papers. The New York Times reported simply that the UN said Tuesday, September 14 that the “the number of hungry people fell to 925 million from the record high of 1.02 billion in 2009,” but that “the level remains higher than before the 2008 food crisis.”

An alternative narrative based on the above would be something like: “the UN attempted on Tuesday to provide some projections for 2010 of the number of hungry people in the world compared to previous projections for 2009, all of which are in turn based on a combination of remarkably shaky links to other projections of impossible-to-project factors like capital flows, unverified and uncertain models of hunger and food availability, an unexplained estimate for 2008, and a survey of uncertain coverage and usefulness last conducted in 2005-2007.”

A new best-selling book called Proofiness opens with a quote that we are “vulnerable to the belief that any alleged knowledge which can be expressed in figures is in fact as final and exact as the figures in which it is expressed,” then the rest of the book explains why this “proofiness” is really “mathematical deception.”

Aid Watch will continue its lazy and supercilious attacks on proofiness.

Be Sociable, Share!
This entry was posted in Data and statistics and tagged , , . Bookmark the permalink. Follow any comments here with the RSS feed for this post. Both comments and trackbacks are currently closed.

21 Comments

  1. David Zetland wrote:

    estimates based on simulations calibrated with models that impute missing variables are so much more sexy than “I made this up while I flying to Geneva,” especially when you get paid $$$ to do it.

    Oh, and WHAT’S the POINT? As if these numbers are going to result in fewer hungry people?

    Send these statisticians into the market, to buy potatoes. Observe frictions and reduce. Repeat until hunger abates.

    Posted September 30, 2010 at 1:54 am | Permalink
  2. Yi-An wrote:

    It would be helpful if estimates were required to come with confidence intervals. A more useful alternative narrative might be, “The number of hungry people fell to 925 million, plus or minus 200 million. This represents a statistically insignificant difference from last year, when the number of hungry people was estimated at 1.02 billion, plus or minus 200 million.”

    Regardless of the exact number though, I do wonder whether you believe the number of hungry people has fallen, or if we just don’t know? In other words, is the indirect methodology trustworthy enough to say that a directional change represents less hungry people, or is the error so high that we’re just in the dark?

    And perhaps the deeper question: Does it really matter what the global number of hungry people is?

    Would be interested in what people think.

    Posted September 30, 2010 at 4:46 am | Permalink
  3. joe wrote:

    I’m no statistician but it seems a bit rich to question conclusions by suggesting the data is a bit fragmented and uncertain. It seems to me this is why we pay statisticians – they come up with conclusions despite the noise of the data. At least that is why my wife is paid – she is an academic statistician.

    What is the difference between a development economist and a development statistician? One has a reputation for lies, mistruths and generalisations and the other is a statistician.

    Posted September 30, 2010 at 5:26 am | Permalink
  4. Scott wrote:

    How about this: it was lazy to start by calling the numbers made up, then wait for people to explain to you how the numbers were arrived at, and then evaluate the quality of the data and method. That seems at least irresponsible, if not lazy. Your critique in the above post amounts to this: modeling food deprivation is hard. You have neither shown that there is a better method, nor that the current method should be rejected. The fact that the NYT doesn’t get the nuances probably has more to do with the NYT than the FAO, no?

    Posted September 30, 2010 at 5:56 am | Permalink
  5. Till Bruckner wrote:

    Any global figure is inherently suspect.. I worked in Ghor province of Afghanistan during 2005. Various population estimates for Ghor ranged from 170,000 to 500,000.

    Q: How many hungry people are there in Ghor?
    A: Many. Especially in spring.

    Echoing Yi-An’s question (see above):
    What purpose does compiling these global figures serve? Are they just advocacy tools, or do policy-makers really use them to make decisions? Will a reported increase of 50 million hungry people lead to different policy outcomes than a decrease of 50 million?

    I don’t know the answers. Can anyone better informed please share their thoughts?

    Posted September 30, 2010 at 7:07 am | Permalink
  6. Nick Gogerty wrote:

    Another germain question, even with the “correct” answer, would it make policy more effective? It seems global numbers at best only generate emotive responses. Only properly measured numbers indicate outcomes and impacts. Global figures tend to be factoids. Although I must confess, I prefer my factoids to be true as bogus factoids undermine the institutions producing them, cheapen the underlying issue and ultimately turn farce into more tragedy.

    Posted September 30, 2010 at 9:46 am | Permalink
  7. zulusafari wrote:

    Case in point. It’s long been estimated that the famous Kibera slum in Nairobi had 1 million people. Last year’s census (recently released) now tells us there is less than 200k. 20%!!!!! of the previously estimated population. You start taking calories/person numbers and that changes drastically. So while the world though everyone was going hungry there, they actually have a large surplus of calories (hypothetically).

    Just a specific example I thought I’d share.

    Posted September 30, 2010 at 10:20 am | Permalink
  8. Andy wrote:

    I suspect these sort of figures are mainly important for the internal political economy of large aid organizations.

    Those working on food security will use such figures to justify lobbying for a budget increase to their sector/department, while in a few months time another study will come out further exalting the link between transport infrastructure and growth, and a similar internal lobbying process will occur.

    Ad nauseum.

    Posted September 30, 2010 at 10:22 am | Permalink
  9. William Easterly wrote:

    comment on Twitter from @gentlemandad: someone needs to explain statistics to @bill_easterly, broken data doth not make statistics invalid

    Posted September 30, 2010 at 1:16 pm | Permalink
  10. William Easterly wrote:

    Joe and @gentlemandad:

    Thanks for the crash course in statistics. Most empirical economists do actually know a little about statistics. The “modeled numbers” situation does not lend itself easily to classical statistical analysis, and requires a lot more information from the modelers (currently nowhere in sight), explaining how they estimated the parameters of their model and what the standard errors are, just for a start, not to mention testing the whole model against alternative hypotheses.

    Posted September 30, 2010 at 1:19 pm | Permalink
  11. William Easterly wrote:

    Scott, your “lazy” accusation seems to have a very top-down idea of a blog, in which Laura and I research all the questions very thoroughly and then announce “the answers” to our passive audience.

    I have an alternative model in which the audience includes many smart and well-informed people, and answers will emerge in a decentralized way from dialogue and debate on the blog.

    And I think it has worked well in this case. If you or others disagree, feel free to say more.

    Posted September 30, 2010 at 1:23 pm | Permalink
  12. joe wrote:

    So… you’re saying you don’t have to understand the methodology and robustness of something before trashing it in a blog, Bill?

    Given the numbers are big, please tell us how you would give a more reliable estimate of the numbers of malnourished people.

    Btw I am both Joe and @gentlemandad

    Posted September 30, 2010 at 2:54 pm | Permalink
  13. William Easterly wrote:

    Joe, I said in the blog what were the questions raised, admitted what I knew and what I didn’t, and asked the readers to contribute their knowledge. I don’t usually do posts this open-ended, and it turned out well in this case. But I am open to the criticism that this was too open-ended, I would like to hear from others.

    I am NOT open to the criticism “you have to make your own numbers out of thin air before you criticize how others make up numbers out of thin air.” Sometimes global numbers are just not possible.

    Posted September 30, 2010 at 4:17 pm | Permalink
  14. Scott wrote:

    Bill, I agree that you should and do use the wisdom of your audience so we can all learn together. But your initial post said “spot the made up hunger numbers”. So you level a very serious charge before the group learning occurs. The normative evaluation preceded the learning. I think you have to recognize that as a widely read blog (perhaps the most widely read development blog?) you have significant influence. A better approach, and in my mind much more in line with the dialogue you intend to conduct, would have been to write “Help us understand the latest FAO numbers” and then had a discussion, rather than judging said figures before understanding them.
    Best,
    Scott

    Posted September 30, 2010 at 4:20 pm | Permalink
  15. vnemana wrote:

    Joe,

    I’m not entirely sure how you gather that Prof. Easterly’s saying that “you don’t have to understand the methodology and robustness of something before trashing it in a blog.” Feel like Aid Watch raised valid criticisms that were backed by both Oxfam’s Richard King and FAO Senior Economist David Dawe, and explained specifically why the methodology was not so robust. A follow-up post once the methodology was further looked-into was always in the cards, if you read the last line of the first FAO numbers post.

    Not to repeat what’s already been said, but it turned out that the FAO methodology is just not very good — something acknowledged by FAO economists. I don’t see why you can’t do the same. And the methodology is not clearly explained in the reports, which give no indication about certain data and are sometimes contradictory. I know this because I helped looked through the reports for Aid Watch (FD: I work here). If you’re so inquisitive, you should do the same!

    What’s more is that these latest numbers weren’t even entirely based on their usual methodology but, in order to be released in time for the MDG summit, used previous USDA projections — another (several) dimension(s) of tenuousness.

    And the fact that all of this is filtered and condensed into simplistic headlines that influence public perceptions is, I’m sure you’d agree with me, a problem. I am with Scott (and Easterly) in that this onus should also fall on the newspapers. I’m pretty sure that readers of the NYT or FT can handle reading a little about the ambiguous methodology.

    Also, the post did suggest a better way to carry out hunger surveys: the WDI anthropometric numbers. These involve actually going into the field and measuring malnutrition within sample populations. But if you’re looking for a comprehensive new methodology from this site…well, that’ll happen when movie critics start directing and releasing new versions of the movies they critique and restaurant critiques start opening new restaurants to show how “they would do it better.”

    Sorry for the point-by-point reaction, but I felt compelled to respond. So could you please elaborate on what the problem is? Thank you.

    Posted September 30, 2010 at 4:35 pm | Permalink
  16. Popa eugen wrote:

    There are many things promoted globally as being “fact”.As such i do not believe in things promoted globally anymore because all is done in manipulative pourposes.Regarding the FAO methodology and the algorithm that give the final result…. i think it is way to messy, has to many variables in it that can lead to huge variations, some variables do not belong there and on top of that the basic principle seems kind of wrong…measuring malnutrition by calory requirements.

    Posted September 30, 2010 at 9:20 pm | Permalink
  17. joe wrote:

    Vnemana, if I recall correctly, the first flame in this war was Bill suggesting the number was wide of the mark. In the above post he says that the FAO knows there is inherent problems in collecting the data and as a result there are academic conferences set up to debate it. But I reiterate the point, academic statisticians are a) very well trained in the valid use of pretty dirty data and b) constantly debating the methods and robustness of what they do. The numbers are not made up, they are offered and debated at academic conferences, as they should be.

    So the point that Bill seems to take exception to is the way that the conclusions are presented in condensed form by the FAO and in a national newspaper.

    Have you or Bill or anyone else in this fight actually given a paper at the academic conference referred to above? How do you know that your sampling methodology referred to above would give any more accurate results when extrapolated to large populations?

    I’d suggest you’d do well to invite a statistician – and preferably one that actually handled the data – to comment rather than assuming the economists know better than they do. Given there are so many academics giving so much time to study and debate the data, I’d think there is a good case to think that the comparison between years is a valid one.

    Or maybe it is all humbug and they’re all wasting their time anyway. Why are we talking about this again?

    Posted October 1, 2010 at 4:12 am | Permalink
  18. I might be in the wrong forum but: It seems to me the key issue is that you mix your accusations of the FAO stats.

    On the one hand, it would appear to me that the underlying criticism is about methodology and the thick-skulled application of the ‘results’ taken at face value.

    However, once you get down to your argument, you solely go after method (apparently without bothering to check it first – but …).

    No?

    Posted October 1, 2010 at 4:38 am | Permalink
  19. Prof. Easterly, Laura,

    Thank you for continuing to challenge, question and sometimes provoke the aid industry. I’m often baffled by development statistics, particularly when they are supposedly “global”.
    It probably doesn’t matter whether there are 1.2 billion or 925 million hungry people in the world – there is no solution to “world hunger”. Saying that there are more or less 1 billion people suffering from hunger is only useful insofar as it influences public opinion and MIGHT create some political space to dedicate more resources to strengthening global food security, though I actually doubt it can achieve that.
    These statistics are numbing, in my opinion, for the general public. I think people are more touched when they are that one person or five people in a particular place are hungry, and feel powerless and despondent when they hear that a billion people all over the world are.
    As for policy makers and practitioners, I’m pretty sure that no plan is ever conceived on the basis of global numbers that are essentially meaningless.
    I understand that, to some extent, we need numbers to support the very existence of some institutions and their important work. But it also raises some more questions, such as should there not be enough world hunger, does the FAO become obsolete? What is that level? Does institutional survival matter?
    I’m definitely one to rely on global development statistics, and I wish they were more reliable and simply more real. I have hope that we will come up with better and more accurate methods to measure global phenomena (hunger, poverty) to produce numbers that can actually serve a useful purpose.

    Posted October 1, 2010 at 5:28 pm | Permalink
  20. Numbers can help to inform a decision, of course, but they are simply not value-free and neutral-and critical research on the World Bank and its procedures have confirmed this time and again, e.g. the publications of Ben Fine or Robin Broad.

    [T]his focus on data, on abstract numbers and seemingly neutral, objective statistics is by no means about to change development research or even ‘democratising development’. (…) The underlying argument is that any aid organisation bases its programmes and decisions solely on ‘objective’ data; First, this ignores the ‘social science’ that is involved in collecting and interpreting data. If you have worked with development economist and/or large datasets you may be aware how working with a household survey from country X can produce astonishing results based on your baseline data, averages etc.

    That is Tobias Denskus on Zoellick’s recent speech on democratising development discourse.

    Posted October 4, 2010 at 5:27 am | Permalink
  21. …and here‘s the direct link to the short blog post.

    Posted October 4, 2010 at 5:40 am | Permalink

2 Trackbacks

  1. [...] This post was mentioned on Twitter by William Easterly, William Easterly, TMS Ruge, Piali Roy, Kit Cody and others. Kit Cody said: touché! RT @bill_easterly: "Proofiness:" the problem with the lazy attack on my lazy attack on FAO hunger numbers. http://bit.ly/aUsWz6 [...]

  2. [...] “Proofiness:” Trashing back on FAO Hunger Numbers [...]

  • About Aid Watch

    The Aid Watch blog is a project of New York University's Development Research Institute (DRI). This blog is principally written by William Easterly, author of "The Elusive Quest for Growth: Economists' Adventures and Misadventures in the Tropics" and "The White Man's Burden: Why the West's Efforts to Aid the Rest Have Done So Much Ill and So Little Good," and Professor of Economics at NYU. It is co-written by Laura Freschi and by occasional guest bloggers. Our work is based on the idea that more aid will reach the poor the more people are watching aid.

    "Conscience is the inner voice that warns us somebody may be looking." - H.L. Mencken

  • Recent Comments

  • Archives