Skip to content

Can We Push for Higher Expectations in Evaluation? The Millennium Villages Project, continued

There’s been some good discussion—here in the comments of yesterday’s post and on other blogs—on the Millennium Villages and what sort of evaluation standard they can (realistically) and should (ideally) be held to.

Yesterday on Aid Thoughts, Matt was distressed that over 70 percent of the student body at Carleton University voted in a tuition hike—$6 per student, per year, every year—to fund the Millennium Villages. The students apparently concluded that the MV program “offers the most effective approach for providing people with the tools they need to lift themselves out of extreme poverty. It is also the only approach that, if scaled, would see countless lives saved, an African Green Revolution and, amongst other things, every child in primary school.”

How is it that students are coming away with that glowing impression from a project that—as Matt points out—has yet provided little evidence that its benefits are scalable, sustainable, persistent or transferable?

Focusing in on results published on one specific MVP intervention, blogger Naman Shah pointed us to his analysis of the MVP scientific team’s early malaria results.  The project’s claims to have reduced malaria prevalence were “disproportionate to the evidence (especially given the bias of self-evaluation)” and suffered from some “bad science” like a failure to discuss pre-intervention malaria trends.

Chris Blattman stepped into the wider debate to offer an evaluator’s reality check, questioning whether rigorous evaluation of the MVP is feasible.  Chris said:

[T]here are other paths to learning what works and why. I’m willing to bet there is a lot of trial-and-error learning in the MVs that could be shared. If they’re writing up these findings, I haven’t seen them. I suspect they could do a much better job, and I suspect they agree. But we shouldn’t hold them to evaluation goals that, from the outset, are bound to fail.

But if the industry standard of best practice is moving towards funding interventions that are measurable and proven to work, why is the MVP encouraging the international community to shift limited aid resources towards a highly-visible project apparently designed so that it can’t be rigorously evaluated?

Fact is, none of us know exactly what kind of evaluation the Millennium Villages Project is doing, or the reasoning behind why they’re doing what they’re doing, since they haven’t yet shared it in any detail.  Perhaps someone at the MVP will respond to our request to weigh in.

Be Sociable, Share!
This entry was posted in Aid policies and approaches, Metrics and evaluation and tagged . Bookmark the permalink. Follow any comments here with the RSS feed for this post. Both comments and trackbacks are currently closed.

12 Comments

  1. geckonomist wrote:

    About the students and their vote: When i was a student, I would also probably have voted yes to such proposal. It sounds good and who doesn’t want to do some good with 6$/year.
    I can understand that outcome of the vote.

    However, years later and having seen and followed up (as a neutral, non-involved party) many aid projects, talked to many aid workers and “poverty experts”, after living, working and investing for years in an aid sector’s darling country, I would vote no.
    But i don’t expect that students of Carleton University have the same perspective, and even if they did, they might disagree with the conclusion.

    It is clear that the MV project does not want to evaluate their efforts. They are part of a belief, and believers are usually not the most rational, self criticising group.
    But it is damn difficult to evaluate such a project and include the externalities and unintended consequences.

    Suppose you have a corn/maize flour market in a county with 2 villages, isolated from each other. One is MV the other one not.
    Each village maize output was 50tons last year.
    In the MV village there are plenty of new inputs, and the corn/maize production suddenly rises to 200tons.
    The other villages produces the same as the year before, 50 tons.

    There is now 150tons more on the market, 250 tons in total.

    But 150 tons more than there is demand for it!!

    Therefore, the maize price/ton halves in the county, the traders go to the MV and buy at once whatever they can sell, 100 tons.
    The income of the MV does not fall since they sell 100tons instead of 50.
    The other village gets no money and 50tons moulding flour. If they are lucky, a trader comes to take it away for free. No money means no school fees, no medicine, no wedding this year…

    Hard to honestly evaluate the results, if you ask me.

    It is the problem of most aid projects in agriculture/manufacture: they know very well how to increase supply, but they are clueless how to increase demand.
    They simply don’t understand that demand/consumption is the incentive for production, not the other way round.
    That’s why all of them fail.

    As a commodity trader in a developing country, I am always very surprised when hearing how ignorant aid workers, “fair” traders and academics are about prices, incentives and how they always grossly underestimate the intelligence of farmers.
    And of course, the traders are seen as evil and exploiting (although they pump daily millions of dollars straight in the pockets of the poorest farmers!)
    whereas the aid workers are perceived as “making a difference” with their 5000$ projects every other year…

    Therefore, when i hear the MV people talk about “increase the food security by providing expertise”, i know they haven’t learned a thing in the last 100 years, and they probably won’t in the next 100 years.

    Posted October 16, 2009 at 8:02 am | Permalink
  2. It’s unfortunate that the author of this post chose to publish such an uninformed blog on the Millennium Village Project’s monitoring and evaluation activities. She and William Easterly at Aid Watch were invited to meet with our scientists and discuss the science and research behind the Villages and the details of the MVP monitoring and evaluation process before publishing any commentaries. Instead the author hastily chose to publish without talking with MVP researchers. The inaccuracy of the blogpost is a reflection of the lack of rigor and objectivity with which the Aid Watch authors approach this subject time and again.

    For readers interested in reading factually accurate information about the Millennium Villages project and its monitoring and evaluation strategy, please see: http://www.millenniumvillages.org/progress/monitoring_evaluation.htm

    Erin Trowbridge
    Director of Communications
    The Earth Institute

    Posted October 16, 2009 at 8:29 am | Permalink
  3. Skeptic wrote:

    Wow. Strong pushback from Jeff’s PR people. I suspect that most communications professionals would say this inappropriately harsh tone makes them sound overly defensive and that the comments page on a semi-academic blog is not the place for the ad hominem attacks. But I guess not too surprising.

    There’s a brilliant May 2007 Harpers article called “Pebranding foreign aid” about Sachs and a troupe of donors visiting an MVP in Kenya.

    Posted October 16, 2009 at 10:24 am | Permalink
  4. E Aboyeji wrote:

    “The students apparently concluded that the MV program “offers the ….”

    Knowing how Canadian student government works, I can assume that is incorrect. Half the time student government is just a euphemism for liberal leaning dictatorial co-option.
    I can bet you a million that this “referendum” was not discussed outside of the student council, it has an opt-out option (which many will use), far less than 30% of the student population voted in this referendum and that people who voted yes simply did so to look good (actually that’s pretty obvious). Considering this is the same school that stripped out funding for cerebral fibrosis arguing it was a “whiteman’s” disease, I can see why they would need to come up with a grand alternative–and who will better fit the description than another coterie of poor emanciated and disenfranchised black Africans…..just call it white guilt

    Posted October 16, 2009 at 10:48 am | Permalink
  5. Andrew wrote:

    The problem with their methodology described in the link provided by Erin is not one of scientific rigor, but that it lacks transparency and independence. It’s fine to use propensity score matching to form an artificial control group – I’ve done exactly that for an evaluation of a roads program where randomization was infeasible, as it may well have been with the MVs. However, I know from experience that forming articial control groups is an extremely complicated and delicate exercise and the quality of the whole evaluation is basically placed at the mercy of both the data sources available ex-ante and the capacity of the algorithm to replicate the procedure used to allocate treatment. The limited data resources likely available to the evaluators on the universe of potential controls and the likely politicization of the treatment allocation procedure makes me initially skeptical of the capacity to yield rigorous inferences over program impact. I could be convinced otherwise, but not unless the MVP starts posting a more detailed description of its methodology and submitting it to some sort of review by professional academics. Which in turn raises another question, which is why the evaluation of the whole expensive exercise was never properly outsourced to an independent entity, such as J-PAL. It would appear from Erin’s link that the whole evaluation is being managed by the MVP, with limited capacity for those involved in the evaluation to independently disseminate their results. All in all, the combination of such a complicated – and manipulatable – methodology with a systematic lack of independence of the research team and its data collection procedures from the program’s management would make me very skeptical of the integrity of the results. In all honesty, I think this could have – and should have – been done a lot better, if we were indeed interested in learning more about the efficacy of interventions in improving development outcomes. It’s quite a shame, actually, especially when there are multiple rigorous and independent evaluations of similar village-based multi-pronged interventions presently being conducted around the world.

    Posted October 16, 2009 at 12:14 pm | Permalink
  6. Khadija wrote:

    Wow. Some of the comments on here are clearly from individuals who either consider themselves intelligent enough to make blanket statements with no concrete backup or are just plain bored. Taking a critical stance on a topic is ok, but usually people validate their arguments with concrete facts and not just hear say don’t they? I, on the other hand, agree wholeheartedly with Erin’s comment. And I’m sorry Aboyeji but did you know the true intentions of the “30%” you say voted to look good? I mean how would you possibly know that? But judging from the tone of your very passionate statement, once could say that your arguments is lacking and solid foundation. Yes, the millennium villages may not be perfect, please tell me which ‘development’ initiative is 1000% effective and perfect, there will and should be (i would hope) difficulties and obstacles that may hinder success, but individuals learn from their mistakes and we are all human here. With time, and effort, I believe the Millennium Villages have the ability to be extremely successful and will have saved the lives of countless individuals who would have otherwise died. Many deaths are preventable in the developing world and I believe there are many people who know enough to want to support an initiative like this. So out of that “30%”, you referred to, I’m pretty sure a great majority weren’t voting yes to appease their conscience. They actually knew what they were doing. I voted yes, and would do so again in a heartbeat. Don’t pretend to know about something you don’t have sufficient knowledge of, and if you are going to do that, at least try and act like you actually KNOW what you are talking about.

    Posted October 16, 2009 at 1:03 pm | Permalink
  7. Confused wrote:

    Erin Trowbridge wrote:
    It’s unfortunate that the author of this post chose to publish such an
    uninformed blog on the Millennium Village Project’s monitoring and
    evaluation activities. She and William Easterly at Aid Watch were
    invited to meet with our scientists and discuss the science and
    research behind the Villages and the details of the MVP monitoring and
    evaluation process before publishing any commentaries. Instead the
    author hastily chose to publish without talking with MVP researchers.
    The inaccuracy of the blogpost is a reflection of the lack of rigor
    and objectivity with which the Aid Watch authors approach this subject
    time and again.

    For readers interested in reading factually accurate information about
    the Millennium Villages project and its monitoring and evaluation
    strategy, please see:
    http://www.millenniumvillages.org/progress/monitoring_evaluation.htm

    Erin Trowbridge
    Director of Communications
    The Earth Institute

    When was the “monitoring and evaluation” webpage published? 13 days ago that category did not even exist on the “charting progress” page.

    Posted October 16, 2009 at 1:11 pm | Permalink
  8. Andrew wrote:

    “With time, and effort, I believe the Millennium Villages have the ability to be extremely successful and will have saved the lives of countless individuals who would have otherwise died.”

    Perhaps, but it seems we will probably *never* know because those leading the respective effort consciously chose not to build a rigorous, independent evaluation into the structure of the program. That is disappointing, to say the least.

    Posted October 16, 2009 at 1:15 pm | Permalink
  9. Nadir Q wrote:

    Just based on the discussion here, and MVP’s response, I have to wonder if these villages are actually more harmful for the wider area than people realize. How can the goods of other non-MVPs possibly compete against something that gets such enormous subsidies?

    Does anyone care to explain? Is there any evidence of this actually hurting nearby villages?

    Posted October 18, 2009 at 9:26 pm | Permalink
  10. charles wrote:

    The only thing I can conclude with any reliability from the exchanges here is that, sadly, describing this blog as “academic” is really false advertising. In equal proportion, the “two sides” on this debate are driven by ideology — though I would say the MV people less so (anyone familiar with development will recognize that they are actually using best practices as designed, yes, by economists with many others over the last 10 years). Bill Easterly’s posts are really quite insulting to the intelligence. Can he really argue with a straight face that letting free markets lift people out of poverty is the right approach? If you think the free markets in the West are fair and transparent (does anyone?), take a look at markets in the poorer parts of the world. They’re really just a playground for the greedy elite in those countries and international investors outside of them.

    Easterly’s argument appears to be to let poor people fend for themselves and the market will reward their competitive behavior. Just where and when has that worked? Government intervention, which is to say development really, has always been critical for that. Otherwise you get monopolies and cartels and things like that. And guess where you see those concentrated and sucking up ridiculous profits? In poor countries! They’re riddled with monopolies and cartels. I know – because I work with them.

    Finally, given that the average Carleton student is going to be spending about $6 A DAY on beer and other important things, why is there such hand-wringing over the $6 per term donation to MV? How many people at this blog carry around “world is ending” placards on the streets of NYC? My guess is probably all!

    Posted October 21, 2009 at 9:37 am | Permalink
  11. Tracy W wrote:

    Easterly’s argument appears to be to let poor people fend for themselves and the market will reward their competitive behavior. Just where and when has that worked?

    Singapore and Hong Kong are the most famous examples.

    Government intervention, which is to say development really, has always been critical for that. Otherwise you get monopolies and cartels and things like that.

    Actually governments easily fall into regulatory capture. They start off intending to protect the broader public good, but the people whose business it is to sell whatever is being regulated have more of an immediate interest in their regulation (as most people tend to sell only a few things relative to the number of things we buy), so the regulated have a lot of reason to lobby and make arguments and generally bend the government towards protecting them at the expense of the broader society. See this Wikipedia article for more detail: http://en.wikipedia.org/wiki/Regulatory_capture

    And guess where you see those concentrated and sucking up ridiculous profits? In poor countries! They’re riddled with monopolies and cartels. I know – because I work with them.

    And it’s poor countries that have less economic freedom. Look at The Heritage Foundation’s Index of Economic Freedom. The countries that are high on this list are amongst the wealthier countries.

    Finally, given that the average Carleton student is going to be spending about $6 A DAY on beer and other important things, why is there such hand-wringing over the $6 per term donation to MV?

    Umm, because the money is intended to help the poor, and it’s worthwhile questioning whether it will or not?
    Plus, if the results from MVP are properly evaluated, the benefits of the increased knowledge about what works and doesn’t work should be much more widely available. This is what makes it important.

    Posted October 22, 2009 at 8:55 am | Permalink
  12. Robert Tulip wrote:

    Geckonomics: “traders are seen as evil and exploiting (although they pump daily millions of dollars straight in the pockets of the poorest farmers!) whereas the aid workers are perceived as “making a difference””. Great point. The ‘big push’ is sublimely indifferent to market distortions because it is more about guilt and ideology than results. If aid was really about development it would focus on these “evil traders” and make life easier for them in order to increase effective demand. These “evil traders” provide jobs, goods, services, models, customers, stability and growth for the poor. For aid donors to work directly with the poor in the absence of clear value chain analysis is a recipe for failure. From first principles, the MVP model is socialist in inspiration and unsustainable by definition.

    Reading the MDG ideology in the MVP Handbook almost makes me puke. You would think the Berlin Wall never fell.

    Posted October 30, 2009 at 2:14 am | Permalink
  • About Aid Watch

    The Aid Watch blog is a project of New York University's Development Research Institute (DRI). This blog is principally written by William Easterly, author of "The Elusive Quest for Growth: Economists' Adventures and Misadventures in the Tropics" and "The White Man's Burden: Why the West's Efforts to Aid the Rest Have Done So Much Ill and So Little Good," and Professor of Economics at NYU. It is co-written by Laura Freschi and by occasional guest bloggers. Our work is based on the idea that more aid will reach the poor the more people are watching aid.

    "Conscience is the inner voice that warns us somebody may be looking." - H.L. Mencken

  • Recent Comments

  • Archives