Results-based management (RBM) is where you come up with indicators of results and try to get civil servants (national or international) to meet targets for these indicators. The emphasis on results would be welcome except for the ability of wily bureaucrats to manipulate the indicators in ways that do not improve performance. In development, RBM has already achieved results – another acronym to replace RBM: MfDR – Managing for Development Results.
The US already has RBM in the form of the No Child Left Behind Act, which rewards public schoolteachers when their students score high on standardized tests. Some of the pitfalls were revealed when I interviewed one of the customers –a 12 year old rising 8th grader in an average public school — about how this was working out. She said “Teachers remind us everyday about the test, and they spend more time teaching us how to phrase answers to test questions than actually teaching us facts.” Finally, the nightmare was over: “And then after the tests were over and taken, they stopped teaching, and the rest of the year we watched stupid movies.” (Less anecdotally, academic evaluations find this Act to have had some payoffs for the worst schools, but note the idiocy of applying it universally to previously well-performing or even average schools.)
In development RBM, i.e. MfDR, the manipulation of indicators for performance is even more brazen and distant from the well-being of any real person. MfDR meetings agree on performance indicators that include – more MfDR meetings. Success on this indicator includes the Monterrey Consensus (2002), Rome Declaration (2003), Washington: First Roundtable on Development Results (2002), Marrakech: Second Roundtable on Development Results (2004), Paris Declaration (2005), Hanoi: Third Roundtable on Managing for Development Results (2007), Accra: Third High Level Forum on Aid Effectiveness (2008).
So, for example,
“The third day of the Third International Roundtable on Managing for Development Results provided a setting for each of 35 county delegations to meet as a team to sketch out initial thoughts on how to mobilize resources within their own countries to initiate and further implement some steps discussed in the context of each of the major themes discussed during the event. Each country team started a Country Action Planning Process, the results of which will help the donor community at the country level target resources at overcoming current gaps in capacity and also to provide input into discussions on managing for results and aid effectiveness at the Ghana high level forum (February, 2008).”
The World Bank has recently published the 3rd edition of its Sourcebook, “Emerging Good Practice in Managing for Development Results:
“Emerging best practice” included the World Food Program in Guinea Bissau and DR Congo:
“Toolkits with clear guidance on managing for development results, with a focus on results-based M&E, are available for WFP and implementing partners’ staff in DRC (Results-based M&E Toolkit, version 1, June 2004) and in Guinea-Bissau (Results-based M&E Toolkit, Version 1, June 2006). These toolkits reflect up-to-date MfDR concepts, principles, and good practices, and draw on results based M&E toolkits of other international organizations, including within WFP. The toolkit produced for WFP Guinea-Bissau is also available on the WFP website (www.wfp.org) as one of the few reference documents for other WFP offices worldwide. A copy of these toolkits is available upon request.”
A much more positive result of MfDR was the City Council of Nairobi setting itself a goal to increase immunization rates from 75 to 85 percent, and then they actually reached 88 percent. Alas, examples like this of actual people getting better off through MfDR are few in the Sourcebook. (Immunization may be one area less subject to manipulation and bureaucratization, where MfDR could have a positive impact.)
Elsewhere, MfDR seems like an exercise for bureaucracies to monitor themselves on — being bureaucratic. Am I missing something?
11 Comments
You bring up a topic worth many blog posts I think.
I completely concur with your point about how a certain results based management (or more commonly in health programs, results based funding) may work to improve performance for the lagging performers, while generating performance undermining “teaching to the test” for the better performers. This is so obviously true, it’s strange I have never come across a results based funding initiative that was aimed at dealing with the underperformers only. As seems so common in development, we simply must have “one size” or one strategy to fit all.
It may (MAY) be the case that immunization coverage is easier to measure, and therefore more amenable to enhancing program effectiveness via results based funding. But even in immunization programs, there is some evidence that some governments who receive an additional payment for each fully immunized child are manipulating the data to receive more revenue. See this paper by Stephen Lim et al in The Lancet Dec 2008 http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(08)61869-3/abstract
One problem with results based funding pointed out in this paper: it is likely to undermine the quality of the data on programs, by creating incentives to mis-report.
I wonder if ensuring standards exist and funding based on standards – at least in education is such a bad thing. While there are certainly schools that will attempt to gain the system, what precisely is the alternative? Would those schools truly have attempted to perform ‘with excellence’ if such standards didn’t exist?
The same goes for using investment/markets and profits to incentivize development – yes the system can be gamed but those who seek to consistently perform, isn’t it a lot easier to seek excellence than it is to consistently game short term tests?
Here’s a thought provoking example of one charter school in California that is rather unapologetic to teaching to the tests – but that’s of course not all they do: http://www.latimes.com/news/local/la-me-charter31-2009may31,0,7064053.story
World Bank borrowers include many weak performers, so in theory, RBM should be useful.
Yet, a common problem with RBM I see in World Bank evaluations is that CASs and projects set out specific target outcomes without baselines (when the target is an increase or decrease) and/or that can’t be measured because of weak capacity, lack of interest, or changes in priorities by statistics gathering groups in country.
Another issue is that projects and CASs have perhaps too many target outcomes, e.g. 43 in the case of one CAS I’m looking at. With so many, it seems inevitable that many won’t be achieved as conditions and priorities change over a 3 year period. Thus RBM doesn’t help as much as it could to judge whether the overall aid program was worthwhile.
The PART system developed by OMB for US federal agencies is a good RBM model to look at.
I recently evaluated a large civil service reform program in an East Asian country–where the objective (expected results) was to reduce the size of civil service by 25 percent.
Nobody knew what the current size of the civil service was!
Apt comments. Results, of course, are a rather good thing. But the bureaucratic prose that has crept amoeba-like (or is it more like molasses?) through too much development literature deserves this kind of pillory though. The true questions are and shoud be, as you suggest, focused both on what results and how we can get there.
Naaah. Surely aid projects/programs should deliver something specific. Surely the taxpayers that ultmately finance aid have the right to know exactly what was achieved. It’s true that aid officials can manipulate indicators, and that indicators can induce perverse incentives (as in the No Child Left Behind Act).
It seems that in far to many cases there is a hidden agenda to programs and a PR agenda.
For NCLB the hidden agenda was (is) to dismantle the teacher’s unions and eliminate tenure. To do this the law was set up in such a way as to ensure that all schools would eventually “fail”. This is accomplished by requiring schools to show annual progress, regardless of the level at which the school is already performing. This is, of course, mathematically impossible. Where do you go when your school is in the 99% percentile? The fallacy was made obvious when one of the best HS in NYC “flunked” on the cities own equivalent measure.
The proponents don’t care a fig about improving the quality of inner city schools, they just want to transfer money to private, parochial and/or for-profit replacements. Discrediting the public school system is part of the intent.
In the case of foreign aid the PR is always framed in terms of some version of the “White Man’s Burden” as our host has so aptly shown. What needs to be discussed in depth is what the hidden objectives are.
It seems to me they may fall into several areas:
1. Corrupt officials supporting their profligate lifestyles.
2. Multi-nationals gain access to markets or resources under the umbrella of the aid program.
3. Feeding aid money back to the giving nation by restrictions on how it can be spent. For example, buying US agricultural commodities with the aid money and shipping them to the recipient country. Much military aid works this way as well.
4. Bureaucrats at many levels working to keep their cushy jobs. Since there work “product” is paperwork, that’s what gets produced by themselves and those they monitor.
5. Buying influence with foreign governments through bribery or extortion as part global power politics. Every time we buy an African leader we make it harder for the Chinese of Russians to do this instead.
Nowhere in this list will you find any mention of helping the poor or helping development. Any such success would lead to the dismantling of the aid system and all those who benefit from the status quo. It’s like the modern medicines which treat “diseases” that never end (like high cholesterol) and thus require use for the rest of one’s life – a drug firm’s gold mine.
Somebody pointed out to me that the Education Millennium Development Goal may have similar or worse effects than No Child Left Behind. (The somebody wishes to remain anonymous because the Millennium Development Goals were approved by God Himself http://blogs.nyu.edu/fas/dri/aidwatch/2009/04/does_god_believe_in_jeff_sachs.html)
By stressing a quantity target for primary enrollment, the Education MDG may cause kids to be herded into classrooms and quality to go to the pits. Bill Easterly
When I read articles like this I want to hurl.
Somehow Results Based Management is worse than what came before it? Which is: We know nothing about what a teacher or bureaucrat is doing. They hide like germs unexposed to the sunlight.
While measurement can surely result in unattended consequences like teaching for the test AT LEAST it starts the conversation about what is actually going on. Furthermore, the metrics can be tweaked and experimented with as necessary.
Without measurements bureaucrats and teachers can continue to hide in the darkness of anonymity.
Lastly, everyone likes to bash No Child Left Behind, but where was the outrage about NOT measuring how well schools and teachers did before NCLB? As a child of the 80’s who went to an Urban highschool, I can tell you I wish we were taught to take tests! At least I would have been more prepared for my SATs!
Both RBM and MfDR are admirable in their intent. Who wouldn’t want public sector and development spending (in particular) to be cast within a framework that demands:
So useful are these precepts, it is not surprising that broadly similar approaches have been developed outside the remit of the international NGO system (e.g. the Outcome Mapping system developed in Canada at about the same time MfDR was being invented). So the question should be not are such frameworks a good idea – but rather has MfDR anything specific to add beyond what is already available?. Rather than poking fun at acronyms, why not dig a bit further and find out if there really are any differences between RBM and MfDR?
As far as I can tell, MfDR simply wraps RBM up in another layer of process – encouraging those using it to embed RBM in a more complete process of objective setting and reporting. In this respect I think it is probably at least harmless, if not helpful. But where I do agree with your sentiments is with respect to the tendency of NGOs and correspondent public servants (and the pervasive network of ‘aid consultancies’) to bury such good ideas in comprehensive obfuscatory waffle. A great pity (and a huge waste of resource). But such obfuscation is not unique to performance management, it is simply how the public sector gets through the day.
Performance Management – the generic business of making sure (in as far as you can) that the resources you control are focused on delivering the results that you want, and taking actions to redeploy or redirect these activities if required, is a common activity across all management. To be simple about it, there is nothing controversial about wanting to set expectations for teachers, and then checking some time later to see if these expectations have been met. Such observations need data (in this case SATs). There is evidence that SATs have lead to goal displacement (at least in the UK – where SATs were explicitly introduced to measure teacher and school effectiveness, but have become simply part of the mechanism to evaluate students). Evaluations carried out with poor data can be harmful to all concerned, as can sensible processes perverted for obscure ends (vis your unintended consequence example). But where the data (and data collection processes) are less complex, there is less evidence of such displacement – one of the reasons medical programmes are often reported on in such literature is that it is pretty easy to measure if a vaccination has been given, and the incidence of an associated disease (at least compared to measuring the effectiveness of a teaching system). Because you can more easily measure the action and the associated desired outcome, ‘managing’ this activity is easier.
But is the correct response to these issues to stop trying to manage by results, or to find better ways to find out what is going on? Should anyone involved in delivering a ‘complex’ service be given a bye when it comes to being held accountable for what they do? After all, Presidents and medical practitioners are fair game for being accountable – why not teachers?
The first line of this post makes the point that “results-based management (RBM) is where you come up with indicators of results and try to get civil servants (national or international) to meet targets for these indicators.”
As someone who works primarily on the micro-end of the RBM curve, where the project field workers try desperately to explain their results in the face of often unintelligible RBM requirements of both bilateral donors and multilateral agencies, my view is that indicators will rarely be useful descriptors of results, if they are imposed on the people in the field.
My experience working with development practitioners in Asia, together trying to simplify results reporting in education, health, environment and governance projects suggests that, to be useful, indicators need to be realistic, field-based, related to the experience of the professionals who have to deliver programming, and to the people who are supposed to benefit from it. This means that those practitioners and service providers, and the people who work as managers with them, have to be involved in defining the indicators, and most important, in testing them against reality. Unrealistic or irrelevant indicators, of the kind often developed by policy makers with no connection to the realities of the field, significantly undermine the relevance of RBM to the real world, and its credibility among the people who are supposed to make it work.
I agree with much of what Gavin Lawrie writes. My experience suggests that complex services can still be explained by focusing on simple, core changes, rather than on complex and ambiguous terminology that many of the aid agencies use to “measure” results.
Making RBM simple (not simplistic) is not that difficult if we can get away from the jargon. But it is necessary if it is going to be used effectively.