Skip to content

If an evaluation is released on the internet and no one comments, does it make a sound?

The release of the Millennium Villages Project mid-point evaluation has so far been met with no discernable public response.

Strange, since the release is billed as the “first major scientific report on progress after three years of MVP activity.” Doubly strange, since the MVP is an ambitious project that reaches into nearly all areas of its 500,000 recipients’ lives, and proposes, in scaled-up version, to completely change the architecture and delivery of aid to Africa.

So why the silence? Two possible reasons come to mind. Perhaps:

  1. The evaluation doesn’t contain much that is unexpected or useful, and/or
  2. No one really cares about evaluation.

We knew that the report would give the mid-point results of a longitudinal study comparing data from 300 Millennium Village families collected when the project began and again three, and five years later. (Although this is no longer the midpoint of anything, as the project has since expanded from 5 to 10 years.)

The new data give a picture of encouraging results across all sectors compared to the baseline. In Mwandama, Malawi, for example, bednet use for children under five increased from 14 percent to 60 percent and malaria prevalence for all age groups fell from 19 percent to 15 percent. Maize yields increased dramatically from .8 tons per hectare to 4.5 tons per hectare.

Such short-term results are positive in the sense that they describe real, immediate changes in the lives of thousands of very poor people. But they are not surprising given what we know about the level of resources and intensive technical expertise invested in these villages: the project doubles the size of the local economy—it is roughly equivalent to a 100 percent increase of per capita income per year (see here for calculations from Michael Clemens).

Unfortunately the results are also not that useful: Three years is too short a period to know how to interpret this dramatic increase in maize yields, for example. Is this consistent with normal variation in crop yields? Was 2006 an unusually good or bad year for maize? We don’t know.

The results also don’t help us determine whether current and future resources should be shifted away from other existing or even yet-to-be invented approaches, towards the MVP template. Will those short-term gains last beyond the timeline of the project? Can the project become self-sustaining?

Again, we don’t know, in part because not enough time has passed. Consider this anecdote from a New York Times blog series by Jeff Marlow on the Millennium Village of  Koraro, Ethiopia:

In 2005, all fertilizer was given away, leading to a significant increase in food production. Fertilizer subsidies were then progressively rolled back; by last year, only 50% of the cost was covered. For the 2009 growing season, the project tried something new: farmers were given loans for fertilizer, but they are expected to pay back the full cost plus interest when the harvest comes.

For many Koraro farmers, this is a daunting challenge. “The project used to help us with fertilizer,” says Brhana Syum…“But now it’s very expensive, and there’s no way to pay for it all.” Many farmers facing similar constraints have chosen to scale back their farms, thereby requiring less fertilizer, rather than face enormous debts…

So this particular push towards sustainability has come up against some obstacles. It may yet succeed, or it may fail. We don’t know the end of the story.

Supporters of the project argue that the individual interventions have already been proven: for example, we know that using better seeds and adding fertilizer will increase crop yield. But what the MVP says it is proving with this evaluation is the “value and feasibility of integrated community-based investments”—that is, the whole package of interventions, as well as the management systems used to deliver them. And this is precisely what the MVP does not have the data to demonstrate.

This evaluation repeats the call to scale up the project within existing project countries and expand to new ones, as quickly as possible. But the MVP as a whole remains an untested and unproven intervention, while the lives of Millennium Villagers—their habits, beliefs, livelihoods, and sources of authority—are  inevitably being changed in profound ways. This evaluation does nothing to change the argument of my previous post that the MVP should live up to their promise to be a ‘proof of concept:’ to be seriously and independently evaluated, and proven to work—beyond immediate short-term effects—before it is scaled up.

If you were sick and someone offered you a drug that hadn’t been tested, would you take it? And even if you would, would you want hundreds of millions of people whose lives depended on it to forego other types of treatment and take that drug too?

This entry was posted in Aid policies and approaches, Grand plans and aid targets, Metrics and evaluation and tagged , . Bookmark the permalink. Follow any comments here with the RSS feed for this post. Both comments and trackbacks are currently closed.


  1. geckonomist wrote:

    Again and again the same crop yield/production fallacy…
    Crop yields say bugger all about the income of a farmer!!!!! That’s economics 101.

    Actual volume traded x actual farmgate price might say a little more, but even that is suspicious, esp. in this artificial MV environment.

    I am very sorry if MV Project managers, and apparently Mrs. Fresschi as well, do not understand that.

    Even if the MVP guarantee their farmers a bottom price for their corn, the day that that price moves above the farmgate market spot price (i.e. in a bumber harvest year like 2009 in East Africa), each and every one of those lucky MVP farmers turns into a middle man/woman who’s buying @market rate and selling @MVP price.

    You don’t need to be an agricultural engineer to understand that this makes yield “jumps” from 0.8T/hectare -> 4,5T/hectare very likely, regardless of whether they threw the fertiliser in the pit latrine or not.

    Posted June 4, 2010 at 3:48 am | Permalink
  2. case wrote:

    You ask, “So why the silence?” …

    a) Collective inaction. Who exactly do they expect comment from? I’d be interested to know how the outcomes of this eval are being communicated to the MVP beneficiaries.
    b) It’s quite a big ask. Who really has time to read a 100 page long PDF with no particular indication of what, if anything, would be the point in commenting?
    c) The format sucks. If they wanted sincere comment, they should have chosen a different approach; perhaps a well-publicised open debate event, a public web-forum, a live webchat or something. Or even a way to annotate the document live and collaoboratively. But not a 100 page PDF.

    Posted June 4, 2010 at 3:54 am | Permalink
  3. Jim Harries wrote:

    Hi, The report leaves so much to be desired that it is hard to know where to start … Has the MVP not realised that its ‘subjects’ are 1. primarily oral and 2. not native English speakers, so that to find out what is going on one has to have people on the ground who are 1. familiar with local tongues. 2. ready to talk to people. 3. not identified with massive projects that have ‘bought up’ everybody by offering them (directly or indirectly) vast amounts of money?

    Posted June 4, 2010 at 5:27 am | Permalink
  4. Jess wrote:

    Just another round of same-old, same-old.

    Millenium blah blah

    Posted June 4, 2010 at 11:57 am | Permalink
  5. Amy B. wrote:

    The Amyloidosis Foundation estimates that approximately 3,000 people are diagnosed with amyloidosis each year in North America and that blood cancers overall have increased more than 40% in the last decade.

    Posted June 4, 2010 at 6:33 pm | Permalink
  6. Jonathan Seib wrote:

    @geckonomist: I’m wondering about your criticism of Mrs. Freschi’s observation of the increased crop yeild. I understand that the numbers say little of a farmer’s ABSOLUTE level of income but to say that a potential quadrupling (caveates considered) of crop may occur as a result of the process suggest the potential of this aid format in comparison to the counter-factual.

    I understand that there’s problems with the artificiality of the MV environment, however, there is still clear grounds for optimism for these farmers RELATIVE to other farmers if we assume that, coming out of the MV environment, their yields are higher.

    I’m new in this arena of study and to the concept of the MVP itself. I could very well be missing something.

    Posted June 5, 2010 at 2:43 am | Permalink
  7. Stephen Jones wrote:

    suggest the potential of this aid format in comparison to the counter-factual.

    Nothing to do with millennium aid. That year the Malawian government provided free fertilizer to all farmers.,

    The policy is of course unsustainable in the long run.

    Posted June 6, 2010 at 7:18 pm | Permalink
  8. Jessica L. wrote:

    A few comments re:

    1. If you were sick and someone offered you a drug that hadn’t been tested, would you take it? – If I were sick and there were no other proven options available and this one seemed like it would work, YES. This is the case with MVP. Do you know something that works better? If so, give it to me. If not, I’ll try it.

    2. To Jim H. – Obviously MVP realizes its subjects are non-native english speakers. That’s why evaluations are conducted by people from the area, fluent in the local language.

    3. Of course, the evaluations aren’t perfect, but they are a start. I would imagine MVP would be happy to be “seriously and independently evaluated..” but who is going to do it and who is going to pay for it?

    4. The silence is particularly surprising given all of the outcries demanding the publishing of the evaluations….

    Posted June 7, 2010 at 2:10 am | Permalink
  9. Raphael wrote:

    @geckonomist. See page 65 of report. MVP HHs in various places did see an increase in incomes through group formation, bulking, and value chains for cash crops- all the usual stuff. However, this info is interspersed in the report and income is not even one of the primary measures according to the index. (Caveat, I hate measuring income!)

    But my first question is where is the research methods section? And is the survey data public? Are the statistical results somewhere in an unpublished annex? A bit of transparency would help.

    Now let me print out this behemoth and try to read it….

    Posted June 7, 2010 at 5:05 pm | Permalink
  10. D. Watson wrote:

    @geckonomist – Given the high rates of subsidy the farmers are getting, it’s a good bet incomes went up. The question is if farm incomes increased by more than the subsidy.

    Posted June 8, 2010 at 2:47 pm | Permalink

2 Trackbacks

  1. [...] Visit link: If an evaluation is released on the internet and no one comments … [...]

  2. [...] This post was mentioned on Twitter by William Easterly, @mikegechter's RSS. @mikegechter's RSS said: If an evaluation is released on the internet and no one comments, does it make a sound?: The release of the Millen… [...]