Tuesday’s Financial Times printed a Martin Wolf interview with the Gateses from Davos, available as a video on the FT web site.
A sample quote from the interview:
We’re trying to make sure that people understand this: aid is effective…So, for instance, malaria incidence is down in countries such as Zambia, Ethiopia, and Rwanda. It’s down in some countries by over 50 percent and some by 60 percent…[if we and other donors] come in and distribute mosquito nets – 60m to date – that is how we have achieved these declines. So we are able to say, “Look, aid is making a huge difference, we are literally saving people’s lives.”
Real victories against malaria would be great, but false victories can mislead and distract critical malaria efforts. Alas, Mr. and Mrs. Gates are repeating numbers that have already been discredited. This story of irresponsible claims goes back to a big New York Times headline on February 1, 2008: “Nets and New Drug Make Inroads Against Malaria,” which quoted Dr. Arata Kochi, chief of malaria for the WHO, as reporting 50-60 percent reductions in deaths of children in Zambia, Ethiopia, and Rwanda, and so celebrated the victories of the anti-malaria campaign. Alas, Dr. Kochi had rushed to the press a dubious report. The report was never finalized by WHO, it promptly disappeared, and its specific claims were contradicted by WHO’s own September 2008 World Malaria Report, by which time Dr. Kochi was no longer WHO chief of malaria.
(There was never a retraction in the New York Times, so perhaps Mr. and Mrs. Gates can be forgiven for being confused – although with most of the world’s public health professionals on Mr. and Mrs. Gates’ payroll you would think their briefers would have access to the most accurate information.)
The September 2008 WHO Malaria Report keeps Rwanda as a success story (along with some other new success stories – not mentioned in the New York Times – like Sao Tome & Principe and Zanzibar), but Zambia and Ethiopia are gone: the effects of malaria control in Zambia were “less clear,” and in Ethiopia, “the expected effects” of malaria control are “not yet visible.”
Digging deeper into the WHO Malaria Report, the standards for data on malaria are set so low, it is even more striking how the Kochi numbers – those numbers that fueled a February 2008 New York Times story and a February 2009 Gates claim – failed to meet even these low standards. The WHO says (in a small print footnote): “in most countries of Africa, where 86% cases occur, reliable data on malaria are scarce. In these countries estimates were developed based on local climate conditions, which correlate with malaria risk, and the average rate at which people become ill with the disease in the area.” Another stab at explanation of their malaria numbers was: “From an empirical relationship between measures of malaria transmission risk and case incidence; this procedure was used for countries in the African Region where a convincing estimate from reported cases could not be made.” (Possible translation: we make the numbers up.)
The shakiness of the numbers is visible when you look at them by country in the WHO Malaria Report. For the “success story” of Rwanda, there is an estimate of 3.3 million malaria cases in 2006, with an upper bound of 4.1 million and a lower bound of 2.5 million. But wait – another way to estimate cases, which is the one used to estimate trends, shows 1.4 million cases in 2006 (and this was an increase over the 2001-2003 average). Estimates of child malaria deaths in Rwanda are similarly all over the place – they do show a drop from 2001 to 2006, but the change is dwarfed by the vast imprecision conveyed by the lower and upper bounds.
In another WHO success, Zanzibar (which, to be fair, Mrs. Gates also mentioned as a success by in the interview), there seems to be more consensus on success from a combination campaign featuring indoor spraying of homes, insecticide-treated bed nets, and treatment of malaria patients with advanced drugs. It seems to be easier to make inroads into malaria on small islands. The American Journal of Tropical Medical Hygeine has published two articles suggesting there was success of malaria control in Sao Tome (also an island) and a corridor in South Africa, Mozambique, and Swaziland, apparently using more rigorous data methods.
As far as the country claims by the WHO and Mr. and Mrs. Gates, however, there seems to be mass confusion, and data that ranges from phony to made-up to shaky, about what interventions are responsible for what trends where. The WHO Malaria Report offers this ringing conclusion in its “Key Points” summary on how to control malaria:
In general, however, the links between interventions and trends remain ambiguous, and more careful investigations of the effects of control are needed in most countries.
Maybe the Gates Foundation should be funding more rigorous data collection. With all this effort to fight the tragedy of malaria, it’s even more tragic that the malaria warriors can’t even get accurate reports of who is sick and dying when and where.
15 Comments
The problem of malaria – and countless other public health issues in the developing world – is infrastructural.
After Bill Gates’ TED talk a short while ago, I jotted down a few quick notes.
Here’s the crux of it:
The average mosquito only covers a 50 metre radius in its lifetime. If there are no malaria carriers in that radius, anopheles or not, the mosquito will present no risk of malaria. In short, you only need comprehensive treatment of the disease and a good prevention regime to reduce the incidence rate below its capacity to propagate. That’s hard to do in the developing world, but it’s hardly impossible.
Nets and medicine. Reduce the duration of the infection and mitigate its spread, and you can make malaria numbers drop substantially.
That kind of development isn’t sexy or headline-making. It’s boring and tedious as housework. Which, I suspect, is why we’re having so much trouble with it.
Dr Easterly,
thanks for the post – somebody in a comment asked me about the report early last year. I think my response strikes similar chords (though without your firebrand approach).
http://topnaman.com/advocacy/goodluck-to-the-un-malaria-envoy/
“Danika,
I’ve haven’t read the report in great detail but my impressions are as follows – it deals largely with available data only 1 year post-intervention. I think that’s simply too short of a follow-up time to draw any strong conclusions. There are no controls provided in terms of other countries or communities within those countries, and thus we could be looking at a decrease due to some confounding factor – like less rainfall. Finally, the results are based partially on convenience samples, populations the researchers had access to, and ones which are not necessarily representative making it difficult to extrapolate. In defense of the authors, it is very difficult to measure impact in an operational environment not designed for doing such research, but that also means they have a responsibility to make the limitations of their conclusions very clear.”
Nice job Bill.
My favorite was the WHO claiming credit for malaria reductions in the
LSDI region (Mozambique, Sout Africa, Swaziland etc.) due to bed net distribution when it was due to indoor spraying. I and others asked for a
correction from WHO, none was forthcoming.
You bring up a very important point: lack of good quality data.
The issue is pervasive. You could have written the same article re infant mortality, HIV (there was a similar incident recently), you name it.
One problem is that, in their effort to write complete reports and show success, a lot of data is imputed. Often, the imputation process, and the related uncertainty bounds, are not adequately reported, only the point estimates are, which severely underestimates the uncertainty around any research using these data.
Since most researchers do specification searches until they just about get significance (e.g. t stas typically between 2-3), adding the data uncertainty would take away all the significant results. [see Gerber, Green and Nickerson "Testing for Publication Bias in Political Science " ]
So think about that for a minute. If you live and die by significance tests (you should’t) you might as well throw away 90% of published articles to date. Their standard errors are all, quite probably, grossly deflated. And since published results are, typically, just about significant, then it follows that inflating the errors ever so slightly to account for data uncertainty will push most published results to insignificance.
Some suggestions:
1. Separate completely the roles of data collection and analysis. Build Chinese walls between these functions.
2. Publish the raw data — all of it — and not just the “clean” imputed data sets. Let each researcher do their own imputation.
3. Don’t just pick on the WHO. Almost all researchers take these datasets at face value. We need to focus a lot more on the data and less on estimation models. Unfortunately many journals, managers, etc. find data stuff pedestrian. “Bill gates provides new and improved data set on Malaria” just doesn’t cut it as a flashy headline.
Rwanda is apparently using such information to eliminate use of IPTp to protect pregnant women from malaria. This is a volatile region where guerrillas and mosquitoes do not worry about passports and visas.
A more important issue is that efforts to control malaria in the high burden countries like Nigeria and DRC are lagging so far behind that progress in Rwanda, real or not, is just a drop in the bucket.
A follow up to Roger Bate’s post. WHO AFRO still has the press release up (claiming nets account for the South Africa/Mozambique corridor success, when it was really indoor spraying). They moved it from their press releases section to Notes for the Press – http://www.afro.who.int/note_press/index.html
The offending article is here – http://www.afro.who.int/note_press/2008/pr20081905.html
Roger and I and others protested to the WHO; we never had any response to these letters, e.g. – http://fightingmalaria.org/pdfs/WHOAFRO_Press_Release.pdf
Thanks
Richard
Bill-
Great post, great blog. I just got back from Kenya where there has been massive net distribution and similar claims of reduction of millions of malaria cases saved. Unfortunately, I discovered that providers almost never actually test for malaria and lab capacity is abysmal. Probably 90% of fevers are reported as a malaria case. It would take a massive survey involving blood sampling to get a reliable estimate of the reduction of malaria and considerable investment would have to be made in building the countries diagnostic capacity to do it.
Bill,
You’ve hit a very important nail on the head.
The information about program implementation and impact which is collected and used in malaria programs is terrible: almost all of it comes from program reports of drugs or nets distributed, and/ or similar numbers from the information systems for public facilities. These numbers themselves are not very reliable, and we know that a bednet handed out is a long way from a bednet being slept under when mosquitoes are out. Ditto for drugs. Yet – the global community plows onward blindly. My friend Benjamin Loevinsohn asked me recently if we’d ever get tired of “bowling in the dark”. It occurs to me that we may prefer it, because then we can always just say that we are succeeding…with no info to contradict us.
Two actions which the Gates’ and other donors could contribute to, which would really make a difference are:
1) to move to doing household surveys every 2 years, rather than every 5. This would be frequent enough to give us much more insight into which progam interventions are really making a difference; and,
2) START (for the first time, believe it or not) conducting yearly facility surveys to find out what is happening at that level.
These initiatives would cost a fraction of the money that is wasted by continuing to use program approaches that aren’t working.
Thanks again for drawing attention to this critical, and all too often overlooked, issue.
What I am curious to know about is whether the Gates foundation or any other organization “fighting” against the malaria has put some money towards studying what Africans did to treat malaria before modern medicines and mosquito nets.
I know someone who was treated as a child by an older remedy in one bout with malaria (believe it or not, many people get malaria several times and don’t die! I myself have survived malaria as a child). This person felt better the same day and was well the next day – after having been practically bedridden. The knowledge of the herbs/etc used for the treatment is lost, though. From my experience in West Africa, I saw generic malaria meds everywhere – from the big cities to the villages. I didn’t see/hear of people taking them regularly.
http://femmeautonome.blogspot.com/search/label/malaria
Felix Salmon asks Maleria Matters, and gets pointed to research suggesting Gates’ numbers might be reasonable:
http://www.portfolio.com/views/blogs/market-movers/2009/02/12/malaria-easterly-vs-gates?tid=true
April, I think your point:
“These numbers themselves are not very reliable, and we know that a bednet handed out is a long way from a bednet being slept under when mosquitoes are out. Ditto for drugs. Yet – the global community plows onward blindly.”
…is key. Anyone who has lived in Sub-Saharan Africa knows that the battle is not supplying adequate nets, but rather ensuring that nets are used correctly. Too often, bednets are turned into fishnets, wedding dresses (seriously – wedding dresses), etc.
It always amazes me to hear activists and fundraisers in the US imply that malaria problems could be resolved if only development actors would supply enough nets. Perhaps because measuring net distribution is easier than measuring the accompanying behavior change or (as detailed above at length) measuring impact on incidence rates?
The WHO in recent years has recommended the use of interior spraying with DDT claiming reductions in transmission rate of as much as 90%. Are these numbers reliable and why is the WHO reluctant to mention spraying efforts?
This study seems to indicate the promise of a combination of interior spraying and treatment: Impact of DDT re-introduction on malaria transmission in KwaZulu-Natal
Chronicle of Philanthropy reports a comment on this post by the Gates Foundation:
http://philanthropy.com/giveandtake/index.php?id=915
I realise this may continue over at the ‘Chronicle of Philanthropy’ blog, but in case any is still reading down here I would just like to add my thoughts.
(i) Please can we spend 10% of budgets implementing, sustaining, reading and acting upon valid M&E.
(ii) Can our valid M&E be based on outcomes and impact, and not coverage of drugs/bednets supplied.
(iii) Ditto April’s points above.
Steve.
http://zambiaconservation.blogspot.com/2007/10/mossy-nets-and-feeling-good-about.html