Esther Duflo is having a good month, first the John Bates Clark medal for best economist under 40, and now a new profile in the New Yorker. It’s great to see development economists appearing in the New Yorker (link to abstract, full article alas requires subscription).
Esther is very deserving of this recognition. Anyone who gets hundreds of other academics and researchers approaching things in a new way (“randomized controlled trials” to measure the impact of development projects by comparing treatment and control groups) deserves tremendous credit.
A couple of things I liked about the New Yorker article:
Her childhood view of the poor, Duflo said, was shaped by “Protestant left-wing Sunday School.”
In Duflo’s view, both sides of the Sachs-Easterly argument reflect an unrealistic public desire “for an expert discourse, which is going to be able to tell you: This is going to be the end of poverty.” Duflo…argues that “there is not going to be le grand soir – one day, the big revolution, and the whole world is suddenly not corrupt. But maybe you create a small little virtuous group here and something else there. All these things are incremental.”
What I really did NOT like about the New Yorker article:
It gives short shrift to criticisms of the limits of Randomized Trials, quoting Lant Pritchett and Angus Deaton so briefly that I doubt the general reader will even get the criticism. The article spends more time on a completely inappropriate attack on the tone and alleged “elitism” of these critics than explaining why MANY in economics today feel discomfort with the claims made for Randomized Trials.
15 Comments
Bill, can you post links to the Lant Pritchett and Angus Deaton references.
Thanks,
Nelson
I love this insightful blog post on RCT and the innate entrepreneurialism found among the BoP (or whatever we’re calling them these days)
http://thecomingprosperity.blogspot.com/2010/02/randomized-out-of-control-trials.html
Here’s my favourite snippet:
I can see it now… A well-meaning RCT acolyte popping up on NPR to talk about the Aravind Eye Hospital and saying
Well, yes, it is true that they have cured two million people of blindness in thirty years. But we don’t have any real evidence of their effectiveness. How many of the people treated at Aravind would have ceased to be blind without the surgery they received? Was the Aravind approach really better than alternatives? And did it really have an impact on economic growth in the regions in question? The only thing I can say without proper evaluation is that this is a nice retirement project. But I can’t tell you that it is effective development.
Am I making this up?
Oh I just love the Emporer’s New Clothes !
That would probably be the Deaton piece:
http://weblamp.princeton.edu/chw/papers/Instruments_of_Development.pdf
It is not only the development field that has suffered from overzealousness with respect to RCTs. Even though interventions related to health are typically more susceptible to RCTs than ours, many are NOT susceptible, but have heaps of other kinds of evidence. And, just like us, some in the field have tried to draw attention to the problems you can generate when the only knowledge that is given credence comes from RCTs. I heartily commend to all this 2003 article by Gordon Smith and Jill Pell “Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials”
BMJ 2003;327:1459-1461 (20 December), doi:10.1136/bmj.327.7429.1459
Abstract
Objectives To determine whether parachutes are effective in preventing major trauma related to gravitational challenge.
Design Systematic review of randomised controlled trials.
Data sources: Medline, Web of Science, Embase, and the Cochrane Library databases; appropriate internet sites and citation lists.
Study selection: Studies showing the effects of using a parachute during free fall.
Main outcome measure Death or major trauma, defined as an injury severity score > 15.
Results We were unable to identify any randomised controlled trials of parachute intervention.
Conclusions As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute
Note: the article is accessible with registration.
The deliberate ignorance and ridiculous reductio being displayed in the comments would be infuriating if it weren’t so pathetic.
The randomistas have shown that ideas can be held to account. And those who would rather hide behind theory without getting their hands dirty and doing the hard work to figure out whether they’re actually doing good, are running scared. Please keep running and don’t ever come back.
@niti bhan: yes you are making this up. Please show me a respected proponent of RCTs that has ever suggested something like an RCT of Aravind to figure out if it helps people. Or one that has proposed testing the use of parachutes.
RCTs can be used in vastly more situations than most give them credit for–this is one of the major reasons that Duflo (and Banerjee, Kremer, Karlan, Mullainathan, etc.) have attracted such attention. They’ve been able to creatively find ways to test ideas that many believed were untestable. What they don’t do is waste time and resources on RCTs of Aravind.
What they would do is suggest tests of how to get more people access to Aravind’s services–as they’ve done testing access to vaccines, fertilizer, bed nets, etc. Damn them for attempting to find out how we might actually get useful things in the hands of the people who need them at lowest cost and maximum effectiveness.
Where there are “out-of-control-trials” the fault lies not with the randomistas but with the structure of academia. Grad students and post-docs have to publish and running or reviewing RCTs is still novel and therefore likely to get published. Don’t blame the randomistas for the broken academic publication system.
And don’t attempt to disguise your fear or justify your failure by parroting mindless critiques of one of the most useful tools development economics has ever come up with.
Why, hello everybody!!
)
Thanks Tim! I was on my way running to the Hilton for safe shelter when I realized that there is nothing to fear!
Listen up all you aid workers! There are a lot of academics and editors out there, that unlike us lazy fools are getting their hands dirty! Little by little, they’re changing the world by typing articles and blog posts and we all know keyboards are full of yucky germs so let’s salute them for really digging in and getting those hands dirty!! Why some of them have even been to the places where we work for up to three weeks at a time!! Just kidding! But once I did know someone who stayed an entire…are you ready?? …SIX MONTHS to a YEAR! Wow. That’s a lot of days.
Anyway, hardy-har, I’ve rethunk it and it seems to me that RCTs are a grrrEAT thing! Looks like the bar has been set pretty low for us (bed net distribution and immunization – yippee!! I thought they were gonna throw something bigger at us)…so I’m gonna ride this wave and suggest you do too! It’ll be fun! We’ll be able to get more money to build into projects, we’ll get to give jobs to more consultants-friends, and we’ll get to go to more conferences! Now…that doesn’t sound too different from anything else does it? And in ten years when Results are still questionable – we’ll say we triiiiiied!
So everyone – put those cocktails down, get off your butts and spray on some deet – the Randomistas are coming! the Randomistas are coming! Let’s dance!
Sundance Kid,
here’s a useful exercise for you:
Explain how those who use the ridiculous attacks on RCTs in these comments are materially different from those who espouse Intelligent Design.
Good to see some lively debate about RCTs!
Of course the randomistas will have thought of this, but whether one thinks that RCT is great or is somewhat closer emperor’s new clothes, depends on what the comparison is with.
Compared to much of mainstream economics, RCT probably is a breakthrough, and if it helps to make mainstream economics more useful then it’s very welcome. Compared to most of the rest of social science, RCT is somewhat less exciting. That ideas can be held to account by evidence is – in most fields – hardly revolutionary.
I was very pleased to see Bill mention the Pritchett and Deaton pieces that raise doubts about RCT and that were quoted in the NYT article. The Deaton one posted by Vincent looks great, although I don’t know if it’s the one that was mentioned in the NYT. Bill – looking forward to you posting the links.
That is the correct Deaton paper in the above comment. For Lant, see my edited volume for Brookings with randomista Jessica Cohen, Thinking Big vs. thinking small, which also has many other prominent academics giving pro and con arguments on randomized trials.
Alan Hudson hit the nail on the head for me. I suppose those of us from social science backgrounds assume that economists (esp dev econs) would be familiar with the vast bodies of research available already on ‘what works’…or ‘what works as best as it can’. But perhaps this is a everyone-in-their-own-Silos thing.
RCTs work best when you have controlled environments and stable variables – not how the real world operates with its multi-causalities and constantly changing environment. At best it is a good guide re what does NOT work.
I have one curiosity about this debate based upon a benchmarking adaptation which we used to do in consulting – border comparisons. We would take two communities sharing very similar environmental factors then tick off the differences in governance and argue about how those differences may account for the different rates of economic success.
The same technique can be applied on a macro-scale with the two Koreas, the former East and West Germanys and surely in Africa, where I have never been, but which has a multitude of borders cutting tribal areas and traditional economic units. I found the Berger study in Nigeria, posted on this site, especially fascinating for this reason.
Is there any significant different between the proposed RCT models and something like Multivariate Testing / multivariate analysis / A/B testing?
I’m basically wondering if there is a softer, more sophisticated way to determine effectiveness using statistics without running into the objections that RCT brings up.
As a physicist I think that it is a good idea that some people in the “dismal science” actually use the scientific method.
In physics for example without the inversion of the modern scientific method by Galileo and the careful measurements of Tycho Brahe there would not have been Newton and all his equations.
In the spirit of comparing RCTs and other evaluation approaches. I’ve found the working paper from 3ie to be really useful as it forces three ‘M&E’ experts to say how they would conduct an evaluation of various types of projects…
2 Trackbacks
[...] This post was mentioned on Twitter by William Easterly, 모아 . 모아 said: RT @bill_easterly: Esther-mania! Randomized experiment pioneer Duflo is profiled by New Yorker: what I liked and NOT. http://bit.ly/c4swPo [...]
[...] of results may be over-optimistic, and I have some doubts about whether impact evaluation and Randomized Controlled Trials really are the breakthroughs that many development economists seem to think they are). It will also [...]