The modest aim of an initiative like Aid Watch is to be one more small voice holding aid agencies and foundations accountable for doing good things for poor people. The aim of more accountability is to induce improved behavior by those guys, so that aid will work better.
The Aid Watch blog already has had its first small test on trying to induce accountability. This post took Bill and Melinda Gates to task for claiming in the Financial Times that foreign aid had big victories over malaria in countries like Rwanda and Ethiopia, because the WHO country data they based it on was made up and later contradicted by the WHO itself.
The Gates Foundation did respond to this criticism, to their great credit (not directly, but that’s OK, it was visible enough in a response to the Chronicle of Philanthropy’s coverage of this controversy.)
What was their response to criticism for using invalid country data? Oops, they offered more invalid country data. The Gates Foundation spokesman offered the country data on Rwanda and Ethiopia from this journal article as defense for the Gateses’ claims on those countries’ victories over malaria.
What does the cited article actually say? “Districts and health facilities were not randomly selected, but constituted a (stratified) convenience sample, selecting those sites where intervention scale-up had been relatively rapid and successful … Therefore, estimated impacts cannot be extrapolated to the countries nation-wide.”
Still, the Gates Foundation was a tad more responsive than the WHO, whose malaria chief first led astray the Gateses and the New York Times with false reports of victories over malaria based on made up country data, then the WHO issued totally different data in its official 2008 Malaria report a few months later, without ever retracting the New York Times story.
When Aid Watch’s intrepid investigator Laura Freschi approached the WHO for comment, she got the following response from the WHO Project Leader for Information Management & Communications, Epidemic and Pandemic Alert and Response (EPR):
“Hello. I have received your emails and phone call. However, WHO does not participate in blog discussions.
Thank you.”
It may seem obsessive to insist on good data, but bad data costs lives. The sad thing is that there have been SOME victories against malaria, and that solid data on WHAT is working WHERE is vital to guide the campaign against this tragic disease. Would Americans put up with the CDC using made up data to respond to a salmonella outbreak?
I guess Aid Watch is going to have to work a LOT harder to do our part to get a bit more accountability.
13 Comments
I guess WHO does not participate in email discussions either! This is a great example of ‘transparency’.
I’m less interested in your ongoing game of ‘Gotcha!’ with B&M Gates than I am in your views on specific interventions against malaria such as those reported in that journal article, which even if they are not nationally representative still appear to be succesful at the local or regional scale. I seem to remember in your book you derided the free distribution of ITNs but isn’t there more evidence now that concerted free distribution programs such as that in Ethiopia seem to be quite effective? It would be a shame if proper analysis of interventions got lost in the rush to slag off Bill Gates, Jeffrey Sachs, and so on.
Regarding the Gates response from criticism:
I don’t agree with Mr. Easterly that the sampling methodology makes the data cited by Gates invalid in this case.
It is important to note that the convenience sample described in the article was guided not by the success of the malaria intervention itself (i.e. it is not intervention effectiveness as “success”), but by the “success” of whether or not the clinics in question were able to actually distribute and rapidly administer the mosquito nets and ACTs.
This is a crucial difference in evaluating the validity of the data gathered. It is acceptable to restrict the sample to clinics that actually carried out the interventions in the manner foreseen by the project. Including other clinics that were understaffed and/or unreliable in regards to data collection or intervention speed would produce false or unreliable data.
The data would also be distinctly invalid if the study only considered certain types of malaria or restricted the sample demographic. The study in question did neither of these; the convenience sample probably should have included a separate evaluation examining why some clinics were not able to scale-up and deploy the intervention quickly, but that does not affect the data concerning the “success” or “failure” of the anti-malarial intervention itself. By employing convenience sampling, in this study, the conclusion is more accurately addressing the benefits of the intervention to the patient because the data does not include the clinics and health posts which were unable to perform the intervention in its entirety.
The stratification of the sample does not make the findings altogether invalid: although a national program should not be extrapolated on the basis of these findings, they do serve as valid trial data for expanded project consideration, together with an impact process evaluation.
I find it acceptable for the Gates Foundation to cite this evidence, as it is clear that the effectiveness of the intervention itself is not an aspect of the study that should be considered invalid.
Dear Jim,
This is not a game. How would feedback and accountability in aid be meaningful unless it was very specific on mistakes that undermine aid efforts, and then vigilant on whether the responsible party corrected these mistakes?
On the issue of free distribution of ITNs, the short answer is that the evidence varies across different circumstances, and both free and subsidized-price distribution continue to suffer from a problem of lack of utilization of the nets. I will have a longer blog on the whole issue of evidence from randomized controlled trials sometime soon (feel free to hold me accountable on that!)
All the best, Bill
Dear SK:
Thanks for your comment. I think the authors had a very good reason for saying they could not draw nation-wide conclusions. In general, any kind of selection renders statistical results suspect. First, the outcome is likely to be correlated with the clinic selection criteria, which biases the treatment effects. Second, there is a second kind of selection going on that the authors mention but I refrained from including in the post. In their words:
“A more general limitation of health facility data is that they cover only the cases and deaths of patients who accessed the (public) health care system…For this reason, it is difficult to extrapolate the observed health facility impacts to effects for the full populations living in the districts sampled.”
So the authors say not even local impact effects can be estimated.
This is not mindless quarreling — both the authors and I agree that local and national effects are not estimated by this study, and hence the original Gates’ claim continues to be based on invalid data.
best regards, Bill
Friends -
I would like to humbly suggest a kind of exercise that I know I would find very useful in this discussion, and that I believe others might find beneficial as well.
Most readers of this weblog, I presume, are at least interested in questioning the role and means of international aid agencies in effectively promoting development and poverty alleviation in poor/developing countries. Probably many readers have deep skepticism around the idea that outside agencies can even do this, even though as we have seen, there have been victories.
What I suggest we do is something like what Plato did with the city-state in the Republic: if we could scrap the whole ODA industry and start again from the ground up, what kind of institution would we build and why? How would it be structured? Who would control it? How would change happen within the organization? How would it be funded? How would it select and execute projects? How would it evaluate projects and change its procedures? And so on. Obviously the guiding principle would be having reasons and evidence to support one’s normative claims about how ODA agencies should be.
I certainly know I would learn a lot from such a discussion, and I think it might help give us an idea of where we want to go with these international development institutions as we become more and more clear about what has failed in the past.
Just an idea! Perhaps such a blog exists already? If so, where? If not, is this something we could explore?
Thanks for the posts and thoughtful comments, btw.
Brendan
Bill–
Great to follow up with WHO. Have you thought about teaming up with Michael Moore? You could get video clips of aid executives awkwardly explaining what they do and how they are spending their money and where they get their numbers.
What a missed opportunity for the WHO. They should be happy to explain themselves, to engage in a debate about their data, which speaks of their relevancy. This excuse about blogs shows a deep misunderstanding of the new media. Who cares if a serious author, like William Easterly, expresses his views in a blog or a newspaper? The idea is not any less important, as are the readers. I feel dismissed by the WHO.
SK -
There is a serious problem with using a convenience sampling involving areas where the two key interventions (net distn and ACTs dispensing) were more intensely implemented. The challenge for malaria control is precisely this: getting the technologies broadly available and used. There is no debate on the impact of high coverage of effective treatment and prevention (nets or IRS) – there is no doubt that you can diminish even eliminate malaria by doing this. What the global community is still searching for is a way to predictably turn funds committed to control malaria into high and sustained access to and use of the technologies in the context of sub-Saharan Africa (and some Asian countries too).
So, interpreting success from achieving high coverage of the interventions, to other regions is very misleading. There are many other countries and studies out there where malaria control activities were implemented, and coverage increased only slightly (a multitude of net studies) or even decreased. In fact, access to effective medicines has gone down in many places even while on-going efforts were in the field to increase access. Despite the fact that funding to malaria went up by 10 times since 1999 – access to medicine has diminished in 10 of the 13 countries where trends can be measured (see WHO Global Malaria Report 2008).
So, convenience sampling where, for whatever reason, the problems related to access/ distribution were less – gives a falsely positive impression about the most important challenge for malaria programs – achieving the higher coverage of interventions.
Nice post April, you beat me to it. I would be interested if anyone has any figures quantifying “use of nets (or medicines) by recipients as intended” vs. “coverage of distribution network”. I am interested in the validity of using coverage as a proxy for impact in these studies.
Steve
Anyone who questions Mr. Easterly’s skepticism regarding the Bill and Melinda Gates Foundation’s unjustified claims about the success of whatever treatment they funded needs to learn what ‘selection’ means. It is not a minor issue. Why on earth do these people want to throw away their own and others’ money on efforts whose returns they can’t even observe? And how can Bill Gates actually hate criticism so much that he can’t acknowledge the obvious logic in Mr. Easterly’s post regarding the (astounding) lack of evidence? How can Bill Gates go on as if actually having good data is not really that important? Is it actually easier for him to make a zillion dollars than it is to give it away effectively?
I work on tuberculosis control issues, and the data from the WHO is pretty much made up there, too. They rely on national governments, which rely on local governments, to report most TB statistics. So the WHO will raise, say, target treatment success rates, and in some places the local officials will magically find that their treatment success rates have risen at the same time. The reported data make one wonder if we have a TB problem at all: who ya gonna believe, me or your lying eyes?
I’m very happy to see some attention being paid to the data issue in a reasonably high-profile place, because bad data can do incredible harm. In TB, the lack of good data has stifled any discussion over the true efficacy of the DOTS protocol in practice and whether the best approach to TB is to continue to push for more funding for DOTS, or to support experimentation with alternative approaches.
I completely agree with you that these donors should consider funding real data collection efforts. It would do enormous good – right now we’re basically flying blind.
For an example of how contradictory malaria data can be, see this Global Dashboard post (http://www.globaldashboard.org/2008/11/05/who-knows/) on malaria cases and deaths in Nigeria, where World Health Organisation data contradicts both Nigerian government data and the data that WHO itself publishes in other documents. In one of its reports, indeed, WHO can’t make up its mind whether the disease kills 10,000 or 300,000 each year.