Skip to content

Do Millennium Villages work? We may never know

Jeffrey Sachs’ Millennium Villages Project has to date unleashed an array of life-saving interventions in health, education, agriculture, and infrastructure in 80 villages throughout ten African countries.

The goal of this project is nothing less than to “show what success looks like.” With a five-year budget of $120 million, the MVP is billed as a development experiment on a grand scale, a giant pilot project that could revolutionize the way development aid is done.

But are they a success? To address that question, we need to know: What kind of data is being collected? What kinds of questions are being asked? Three years into the start of one of the highest-profile development experiments ever, who’s watching the MVPs?

The most comprehensive evaluation of the project published so far is a review by the Overseas Development Institute, a large UK-based think tank. The review covered two out of four sectors, in four out of ten countries, with data collected in the MVs only, not in control villages. The report’s authors cautioned that “the review team was not tasked and not well placed to assess rigorously the effectiveness and efficiency of individual interventions as it was premature and beyond the means of the review.”

Despite this, a Millennium Villages blog entry on Mali says, “With existing villages showing ‘remarkable results,’ several countries have developed bold plans to scale up the successful interventions to the national level.” Millennium Promise CEO John McArthur described Sachs’ recent testimony to the Senate Foreign Relations Committee: “Sachs noted the success of the Millennium Villages throughout Africa and the tremendous development gains seen in the project over the past three years.”

The Evaluation that Isn’t?

In contrast, evaluation experts have expressed disappointment in the results they’ve seen from the Millennium Villages Project to date. This isn’t because the MVPs fail to produce impressive outcomes, like a 350 percent increase in maize production in one year (in Mwandama, Malawi), or a 51 percent reduction in malaria cases (in Koraro, Ethiopia). Rather, it has to do with what is—and is not—being measured.

“Given that they’re getting aid on the order of 100 percent of village-level income per capita,” said the Center for Global Development’s Michael Clemens in an email, “we should not be surprised to see a big effect on them right away. I am sure that any analysis would reveal short-term effects of various kinds, on various development indicators in the Millennium Village.” The more important test would be to see if those effects are still there—compared with non-Millennium Villages—a few years after the project is over.

Ted Miguel, head of the Center of Evaluation for Global Action at Berkeley, also said he would “hope to see a randomized impact evaluation, as the obvious, most scientifically rigorous approach, and one that is by now a standard part of the toolkit of most development economists. At a minimum I would have liked to see some sort of comparison group of nearby villages not directly affected by MVP but still subject to any relevant local economic/political ‘shocks,’ or use in a difference-in-differences analysis.” Miguel said: “It is particularly disappointing because such strong claims have been made in the press about the ’success’ of the MVP model even though they haven’t generated the rigorous evidence needed to really assess if this is in fact the case.”

An MVP spokesperson told me that they are running a multi-stage household study building on detailed baseline data, the first results from which will be published in 2010. The sample size is 300 households from each of the 14 MV “clusters” of villages (which comprise about 30,000-60,000 people each.) She also said that their evaluation “uses a pair-matched community intervention trial design” and “comparison villages for 10 MV sites.”

But Jeff Sachs noted in a 2006 speech that they were not doing detailed surveying in non-MV sites because—he said— “it’s almost impossible—and ethically not possible—to do an intensive intervention of measurement without interventions of actual process.” A paper the following year went on to explain that not only is there no selection of control villages (randomized or otherwise), there is also no attempt to select interventions for each village randomly in order to isolate the effects of specific interventions, or of certain sequences or combinations of interventions.

CEO John McArthur declined to comment on this apparent contradiction. The MVP spokesperson could say only that the evaluation strategy has evolved, and promised a thorough review of their monitoring and evaluation practices in 2010.

Comparison villages could be selected retroactively, but the MVP has failed to satisfactorily explain how they chose the MVs, saying in documents and in response to our questions only that they were “impoverished hunger hotspots” chosen “in consultation with the national and local governments.” If there was no consistent method used in selecting the original villages (if politics played a role, or if villages were chosen because they were considered more likely to succeed), it would be difficult to choose meaningful comparison villages.

Living in a Resource-Limited World

Imagine that you are a policymaker in a developing country, with limited resources at your disposal. What can you learn from the Millennium Villages? So far, not very much. Evaluations from the MVP give us a picture of how life has changed for the people living in the Millennium Villages, and information about how to best manage and implement the MVP.

Sandra Sequeira, an evaluation expert at London School of Economics, sums up the quandary neatly. “Their premise is that more is always better, i.e. more schools, more clinics, more immunizations, more bed nets. But we don’t live in a world of unlimited resources. So the questions we really need to answer are: How much more? Given that we have to make choices, more of what?”

These are tough questions that the Millennium Villages Project will leave unanswered. For a huge pilot project with so much money and support behind it, and one that specifically aims to be exemplary (to “show what success looks like”), this is a disappointment, and a wasted opportunity.

Be Sociable, Share!
This entry was posted in Aid policies and approaches, Metrics and evaluation and tagged , , , , , . Bookmark the permalink. Follow any comments here with the RSS feed for this post. Both comments and trackbacks are currently closed.

18 Comments

  1. Lukas wrote:

    “Given that we have to make choices, more of what?” – now I’m not a huge fan of MV, and their monitoring and evaluation does seem to be lacking, but this is clearly missing the point. The point is not to see which of the tested interventions are most effective. The point is that a multi-pronged approach is more effective than the sum of its parts – to show what breaking out of the development trap (when improvements in one area are overwhelmed by pathologies in other areas) looks like.

    Posted October 15, 2009 at 1:14 am | Permalink
  2. naman wrote:

    Overall evaluations aside, even stated outcomes such as “a 51 percent reduction in malaria cases (in Koraro, Ethiopia)” are grounded in poor methods. While a reduction of malaria may have occurred, one can’t infer that from the data presented. This is surprising given the premise of MVP drawing upon the “integrated scientific expertise” available at the Earth Institute and elsewhere…. More specifics on their malaria data here.

    Posted October 15, 2009 at 10:05 am | Permalink
  3. zulusafari wrote:

    Lukas, I think you make a good point. To add further to that though, the AidWatch team still makes a good point, there’s nothing to compare the MV against. How do you know if it’s made any progress compared to the save village not having had Millions in aid poured onto it. Better yet, why is ANYONE looking at it now? The point is, can it stand on it’s own two legs down the road after the aid has left. Of course it’s doing well, it’s got funding and aid of all sorts pumping into it. Of course malaria has been halved, free nets were likely handed out and with some sort of incentive to boot. But what about when they are not free and there is no incentive. Will the people have learned the value of it and continued it despite the new cost and lack of direct and immediate incentive?

    Posted October 15, 2009 at 11:15 am | Permalink
  4. Roberto wrote:

    I still remember Victoria Schlesinger´s report on MVP in Harper´s Magazine May 2007. So, evidences seem to increase on the new failure of useless millions wasted on “saving the poor”.

    Posted October 15, 2009 at 3:21 pm | Permalink
  5. E Aboyeji wrote:

    Ok.
    So I am not exactly a fan of Mr Sachs but I must say that it seems like you are knocking him for refusing to do RCTs to evaluate the success of MV’s because they are unethical-and this is true.
    That’s particularly strange especially since, except I am mistaken, it was on this same blog that condemned the ethics behind RCT’s.

    Fairer criticism my friends…fairer criticism

    Posted October 15, 2009 at 4:04 pm | Permalink
  6. Anonymous wrote:

    Absolute effects are not that important. Yes, if you pump millions of dollar into villages, you’d expect some absolute improvement.

    But what about cost effectiveness? Surely that is the relevant quantity if the approach is to be replicated elsewhere.

    For this we typically need finer measurements and RCTs.

    If the conclusion of this project is to be that pumping tons of money has an impact on poor villages you could have saved the money. We all agree.

    As for sustainability… We’ll have to wait several years, by which point all kinds of confounding could have taken place. My prior is that none of the reforms will be sustainable. I hope I am wrong.

    Posted October 15, 2009 at 4:28 pm | Permalink
  7. E Aboyeji

    This blog was by Laura, and the RCT blog was by me. This blog encourages each contributor to do their own research and their own analysis. It invites guest bloggers with other points of view, and it invites anyone whose ideas one of us criticizes to respond. So the blog itself does not “have a viewpoint,” it depends on the author.

    regards, Bill

    Posted October 15, 2009 at 5:28 pm | Permalink
  8. Alli wrote:

    The main question that I think this brings up is the one on the banner of this blog…’just asking that aid benefits the poor.’ If nothing else MVP represents a $120 million going right into 14 communities that need it and need it now.

    In a resource limited world, I would rather see money going towards education in a poor (albeit not randomly selected) village then to an economist looking for an evaluation project.

    For the claims and scale I do see why the lack of certain evaluation measures is troubling, but I also do not think it is a claim that can be made until the evaluations come out and we know for sure what has been done throughout the entire evolution of the project.

    Posted October 16, 2009 at 12:09 am | Permalink
  9. E Aboyeji wrote:

    Ok. I guess that clears things up a little.

    I certainly agree that the MV’s should find an evaluation model-except that I hope its not those datardly rat experiments for the poor someone has taken the liberty to euphemistically name RCT’s.

    I understand the frustration behind such evasiveness especially since right now I am tearing my head out that no concrete evaluation work has been done on the HIPC program. Bill’s impressive work from 2002 is very obviously outdated since its post- Jubillee. But oh well, I’ll keep looking

    Posted October 16, 2009 at 10:19 am | Permalink
  10. Jeff Barnes wrote:

    The MVP approach is old wine in a new bottle. When I started in development 20 years ago with an NGO, we called it integrated rural development. In the absence of randomized control evaluation, one can look at the result of previous experience. We were able to produce short term improvements, but they were completely dependent on donor subsidies. There was no “break through” or “take off.” There is nothing about this approach that will solve the problems of bad transport, overvalued currencies or poor health systems. Let’s not delude ourselves into thinking that the MVP’s will tell us anything new.

    Posted October 16, 2009 at 7:08 pm | Permalink
  11. Jim Bob wrote:

    “I hope its not those datardly rat experiments for the poor someone has taken the liberty to euphemistically name RCT’s” . . . This is an ignorant and offensive statement. It is caluminous mischaracterizations like this that serve to thwart the implementation of rigorous evaluations and ultimately the generation of solid evidence that can actually be used to reduce poverty and alleviate associated hardships.

    Posted October 17, 2009 at 9:43 am | Permalink
  12. E Aboyeji wrote:

    Considering how long we have spent trying to find solid evidence of how to reduce poverty and the limited success we have had at this effort, perhaps we should just concentrate on empowering the poor to do these themselves.
    A typical RCT costs between 30,000 to 150,000 US dollars. First, that’s a waste of money that could be doing better. Besides that, considering that they almost always REQUIRE that some other deserving human being is deprived of possibly life changing/saving help, I can’t understand why anyone would see it as an ethical practice. That some privileged scientist thinks they have the right or some feigned responsibility to walk into my community and “test” us to see what “works” seems to me a patriarchal idea that should have died with colonialism.

    My 0.02

    Posted October 19, 2009 at 12:23 am | Permalink
  13. Jim Bob wrote:

    “A typical RCT costs between 30,000 to 150,000 US dollars. First, that’s a waste of money that could be doing better.”

    No. First, it’s not a waste because RCTs aren’t necessarily more expensive than other forms of evaluation which most responsible organizations are fiduciarally mandated to undertake. Secondly, it’s certainly not a waste of money if your time horizon stretches over more than a couple of months and the RCT generates evidence which allows future program / allocation decisions to be better targeted, thereby increasing the efficiency of aid disbursements.

    “Besides that, considering that they almost always REQUIRE that some other deserving human being is deprived of possibly life changing/saving help, I can’t understand why anyone would see it as an ethical practice.”

    Because if it wasn’t for these “dastardly RCTs”, all of those deserving of the program / intervention (and note that RCTs target programs and often components of program, rather than obviously “life-saving” or otherwise critical forms of assistance, such as medical care, emergency food distributions etc.), would have received it, right? Of course not. There isn’t a program in the developing world I can think of that is so well funded as to be able to be disbursed to all deserving beneficiaries. In the real world, all programs have to be rationed through some mechanism. And, in most cases, RCTs actually can improve the allocation mechanism, by introducing some transparency into the process and preventing targeting mechanisms from being contaminated by political favoritism etc.

    “That some privileged scientist thinks they have the right or some feigned responsibility to walk into my community and “test” us to see what “works” seems to me a patriarchal idea that should have died with colonialism.”

    Ugggh. Please go and actually read something about the purpose and functioning of RCTs before you say such offensive and ignorant things.

    Posted October 19, 2009 at 12:27 pm | Permalink
  14. E Aboyeji wrote:

    I don’t have a lot of time for rejoinders now otherwise I would post up a more nuanced counter-argument.
    Nevertheless, I thought the whole point of “aid disbursements ” was to enable poor people make their own choices. If planners are looking at aid from a strictly RCT perspective, I wonder where the truth in that perspective is. As far as I know RCT’s do not equate poor people making their own choices.

    It shocks me as a blatant redistribution scheme sourced from data that has basis that might not be as sound as many assume them to be. This idea that development is a function of reaching the poor with finite resources is outdated. Rather we should facilitate the innovative efforts of the poor to create many more resources from the little they are given.

    Economic growth and development will only happpen when these central planning techniques stop and the poor are actually considered entreprenuers who can innovate their way out of poverty.

    Posted October 20, 2009 at 1:27 am | Permalink
  15. A Brown wrote:

    I’m dissapointed that in the context of “pushing for higher expectations in evaluation” the only possibilities being suggested are randomised control trials and the use of control/comparison groups.

    I’m also dissapointed that the two “evaluation experts” selected to quote are those that support these types of evaluation. They hold them up as a gold standard and suggest they are the only types capable of “scientific rigour” and achieving “solid results”.

    As pointed out by MVP spokespeople and some commentators on the blog, the use of “control groups/communities” in evaluation of aid and development interventions (in evaluation of changes in peoples lives) have serious and very real ethical and practical issues.

    If your point in this blog is that the MVPs should be thoughtfully, seriously and systematically evaluated, then perhaps you could have said that in far fewer words and we could have all agreed with you.

    Instead, you’ve limited the debate and side steped a more interesting, informed and sophisticated discussion about what a balanced, ethical, empowering and effective monitoring and evaluation system for the MVPs might be. One that contributes to development outcomes rather than just testing for them. This could include some discussion about what they already have in place – merits and challenges.

    Now, as an evaluation practitioner, rather than an evaluation theorist, that’s a debate I’m interested in. It’s also a debate that many evaluation practitioners around the world could contribute to (and already are).

    Posted October 20, 2009 at 3:20 am | Permalink
  16. Jim Bob wrote:

    “Rather we should facilitate the innovative efforts of the poor to create many more resources from the little they are given. Economic growth and development will only happpen when these central planning techniques stop and the poor are actually considered entreprenuers who can innovate their way out of poverty.”

    I agree whole-heartedly. But it is very, very difficult to design programs, innovations, or other uses of development assistance which effectively facilitate such without experimentation. After all, if we knew exactly what interventions facilitate entrepreneurship, stimulate growth, and reduce poverty, this blog wouldn’t exist and we wouldn’t being having this discussion. And, for much the same reason that new vaccines and pharmaceuticals have to be subjected to RCTs (I don’t really see how this argument about RCTs in development being unethical does not apply to the medical profession, which of course has made dramatic strides over the past century thanks mainly to RCTs) and also because development agencies cannot allocate the universe of all possible programs to the deserving population simultaneously, it is highly desirable for experimentation in new program designs to be effective, they must be subjected to RCT evaluation. That way, solid knowledge is generated, better programs can be funded, and poverty can be steadily reduced. I really can’t see where the problem is . . . Of course, if your complaint is more with the “central planning” that is inextricable from the practice of development assistance and ODA disbursement, then that’s an entirely different argument, but taking the development complex as a constant for the time being, RCTs are a vast improvement over what has been the state-of-play for the past half century.

    Posted October 20, 2009 at 3:23 am | Permalink
  17. Tracy W wrote:

    except that I hope its not those datardly rat experiments for the poor someone has taken the liberty to euphemistically name RCT’s.

    RCTs are applied to rich people as well, in medical trials of new drugs. RCTS were in fact developed in rich countries for use on their own citizens.

    As pointed out by MVP spokespeople and some commentators on the blog, the use of “control groups/communities” in evaluation of aid and development interventions (in evaluation of changes in peoples lives) have serious and very real ethical and practical issues.

    Despite this, the reason that people use RCTs is that there are very serious and very real ethical and practical issues with not using RCTs. If you don’t use an RCT, it is far easier to be fooled about whether an intervention does more harm than good. It is entirely possible to spend millions on interventions that are actually harmful.

    The opponents to RCTs are effectively arguing that the citizens of poor countries should get poorer-quality interventions than the citizens of rich ones (at least in rich citizens medical lives). What is ethical about that?
    See http://www.ukskeptics.com/cms/randomised-placebo-controlled-double-blind-trials/ for some description of why RCTs are used in medicine.

    Posted October 22, 2009 at 6:03 am | Permalink
  18. E Aboyeji wrote:

    “If you don’t use an RCT, it is far easier to be fooled about whether an intervention does more harm than good. It is entirely possible to spend millions on interventions that are actually harmful.”

    Its funny the lengths to which people that have little or no clue about the lives the poor lead beyond the obvious would go to “intervene” in their lives-with or without their consent.

    How about you just ask us? And if you don’t feel comfortable doing so, keep your ruinious dollars to yourselves.

    Posted October 25, 2009 at 2:00 am | Permalink

3 Trackbacks

  1. By The headless heart « Aid Thoughts on October 15, 2009 at 7:10 am

    […] Fresci at Aid Watch has also posted today about the problems with discerning the MVP’s impact. […]

  2. […] at your disposal. What can you learn from the Millennium Villages? So far, not very much.’ Laura Freschi on the Aid Watch blog laments the lack of proper monitoring in the Jeff Sachs flagship project (and starts another row – […]

  3. […] Millenium Villages Controversy By evidon Aid Watch argues we may never know if they […]

  • About Aid Watch

    The Aid Watch blog is a project of New York University's Development Research Institute (DRI). This blog is principally written by William Easterly, author of "The Elusive Quest for Growth: Economists' Adventures and Misadventures in the Tropics" and "The White Man's Burden: Why the West's Efforts to Aid the Rest Have Done So Much Ill and So Little Good," and Professor of Economics at NYU. It is co-written by Laura Freschi and by occasional guest bloggers. Our work is based on the idea that more aid will reach the poor the more people are watching aid.

    "Conscience is the inner voice that warns us somebody may be looking." - H.L. Mencken

  • Recent Comments

  • Archives