I’m very happy to see some attention being paid to the data issue in a reasonably high-profile place, because bad data can do incredible harm. In TB, the lack of good data has stifled any discussion over the true efficacy of the DOTS protocol in practice and whether the best approach to TB is to continue to push for more funding for DOTS, or to support experimentation with alternative approaches.

I completely agree with you that these donors should consider funding real data collection efforts. It would do enormous good – right now we’re basically flying blind.

Steve

There is a serious problem with using a convenience sampling involving areas where the two key interventions (net distn and ACTs dispensing) were more intensely implemented. The challenge for malaria control is precisely this: getting the technologies broadly available and used. There is no debate on the impact of high coverage of effective treatment and prevention (nets or IRS) – there is no doubt that you can diminish even eliminate malaria by doing this. What the global community is still searching for is a way to predictably turn funds committed to control malaria into high and sustained access to and use of the technologies in the context of sub-Saharan Africa (and some Asian countries too).

So, interpreting success from achieving high coverage of the interventions, to other regions is very misleading. There are many other countries and studies out there where malaria control activities were implemented, and coverage increased only slightly (a multitude of net studies) or even decreased. In fact, access to effective medicines has gone down in many places even while on-going efforts were in the field to increase access. Despite the fact that funding to malaria went up by 10 times since 1999 – access to medicine has diminished in 10 of the 13 countries where trends can be measured (see WHO Global Malaria Report 2008).

So, convenience sampling where, for whatever reason, the problems related to access/ distribution were less – gives a falsely positive impression about the most important challenge for malaria programs – achieving the higher coverage of interventions.

Great to follow up with WHO. Have you thought about teaming up with Michael Moore? You could get video clips of aid executives awkwardly explaining what they do and how they are spending their money and where they get their numbers.

I would like to humbly suggest a kind of exercise that I know I would find very useful in this discussion, and that I believe others might find beneficial as well.

Most readers of this weblog, I presume, are at least interested in questioning the role and means of international aid agencies in effectively promoting development and poverty alleviation in poor/developing countries. Probably many readers have deep skepticism around the idea that outside agencies can even do this, even though as we have seen, there have been victories.

What I suggest we do is something like what Plato did with the city-state in the Republic: if we could scrap the whole ODA industry and start again from the ground up, what kind of institution would we build and why? How would it be structured? Who would control it? How would change happen within the organization? How would it be funded? How would it select and execute projects? How would it evaluate projects and change its procedures? And so on. Obviously the guiding principle would be having reasons and evidence to support one’s normative claims about how ODA agencies should be.

I certainly know I would learn a lot from such a discussion, and I think it might help give us an idea of where we want to go with these international development institutions as we become more and more clear about what has failed in the past.

Just an idea! Perhaps such a blog exists already? If so, where? If not, is this something we could explore?

Thanks for the posts and thoughtful comments, btw.

Brendan

Thanks for your comment. I think the authors had a very good reason for saying they could not draw nation-wide conclusions. In general, any kind of selection renders statistical results suspect. First, the outcome is likely to be correlated with the clinic selection criteria, which biases the treatment effects. Second, there is a second kind of selection going on that the authors mention but I refrained from including in the post. In their words:

“A more general limitation of health facility data is that they cover only the cases and deaths of patients who accessed the (public) health care system…For this reason, it is difficult to extrapolate the observed health facility impacts to effects for the full populations living in the districts sampled.”

So the authors say not even local impact effects can be estimated.

This is not mindless quarreling — both the authors and I agree that local and national effects are not estimated by this study, and hence the original Gates’ claim continues to be based on invalid data.

best regards, Bill

This is not a game. How would feedback and accountability in aid be meaningful unless it was very specific on mistakes that undermine aid efforts, and then vigilant on whether the responsible party corrected these mistakes?

On the issue of free distribution of ITNs, the short answer is that the evidence varies across different circumstances, and both free and subsidized-price distribution continue to suffer from a problem of lack of utilization of the nets. I will have a longer blog on the whole issue of evidence from randomized controlled trials sometime soon (feel free to hold me accountable on that!)

All the best, Bill