As someone who works primarily on the micro-end of the RBM curve, where the project field workers try desperately to explain their results in the face of often unintelligible RBM requirements of both bilateral donors and multilateral agencies, my view is that indicators will rarely be useful descriptors of results, if they are imposed on the people in the field.

My experience working with development practitioners in Asia, together trying to simplify results reporting in education, health, environment and governance projects suggests that, to be useful, indicators need to be realistic, field-based, related to the experience of the professionals who have to deliver programming, and to the people who are supposed to benefit from it. This means that those practitioners and service providers, and the people who work as managers with them, have to be involved in defining the indicators, and most important, in testing them against reality. Unrealistic or irrelevant indicators, of the kind often developed by policy makers with no connection to the realities of the field, significantly undermine the relevance of RBM to the real world, and its credibility among the people who are supposed to make it work.

I agree with much of what Gavin Lawrie writes. My experience suggests that complex services can still be explained by focusing on simple, core changes, rather than on complex and ambiguous terminology that many of the aid agencies use to “measure” results.

Making RBM simple (not simplistic) is not that difficult if we can get away from the jargon. But it is necessary if it is going to be used effectively.

• that the purpose of an initiative be declared before it starts,
• that owners of the initiative track progress towards these stated goals
• that transparency and public accountability is a core element of the activity?

So useful are these precepts, it is not surprising that broadly similar approaches have been developed outside the remit of the international NGO system (e.g. the Outcome Mapping system developed in Canada at about the same time MfDR was being invented). So the question should be not are such frameworks a good idea – but rather has MfDR anything specific to add beyond what is already available?. Rather than poking fun at acronyms, why not dig a bit further and find out if there really are any differences between RBM and MfDR?

As far as I can tell, MfDR simply wraps RBM up in another layer of process – encouraging those using it to embed RBM in a more complete process of objective setting and reporting. In this respect I think it is probably at least harmless, if not helpful. But where I do agree with your sentiments is with respect to the tendency of NGOs and correspondent public servants (and the pervasive network of ‘aid consultancies’) to bury such good ideas in comprehensive obfuscatory waffle. A great pity (and a huge waste of resource). But such obfuscation is not unique to performance management, it is simply how the public sector gets through the day.

Performance Management – the generic business of making sure (in as far as you can) that the resources you control are focused on delivering the results that you want, and taking actions to redeploy or redirect these activities if required, is a common activity across all management. To be simple about it, there is nothing controversial about wanting to set expectations for teachers, and then checking some time later to see if these expectations have been met. Such observations need data (in this case SATs). There is evidence that SATs have lead to goal displacement (at least in the UK – where SATs were explicitly introduced to measure teacher and school effectiveness, but have become simply part of the mechanism to evaluate students). Evaluations carried out with poor data can be harmful to all concerned, as can sensible processes perverted for obscure ends (vis your unintended consequence example). But where the data (and data collection processes) are less complex, there is less evidence of such displacement – one of the reasons medical programmes are often reported on in such literature is that it is pretty easy to measure if a vaccination has been given, and the incidence of an associated disease (at least compared to measuring the effectiveness of a teaching system). Because you can more easily measure the action and the associated desired outcome, ‘managing’ this activity is easier.

But is the correct response to these issues to stop trying to manage by results, or to find better ways to find out what is going on? Should anyone involved in delivering a ‘complex’ service be given a bye when it comes to being held accountable for what they do? After all, Presidents and medical practitioners are fair game for being accountable – why not teachers?

Somehow Results Based Management is worse than what came before it? Which is: We know nothing about what a teacher or bureaucrat is doing. They hide like germs unexposed to the sunlight.

While measurement can surely result in unattended consequences like teaching for the test AT LEAST it starts the conversation about what is actually going on. Furthermore, the metrics can be tweaked and experimented with as necessary.

Without measurements bureaucrats and teachers can continue to hide in the darkness of anonymity.

Lastly, everyone likes to bash No Child Left Behind, but where was the outrage about NOT measuring how well schools and teachers did before NCLB? As a child of the 80’s who went to an Urban highschool, I can tell you I wish we were taught to take tests! At least I would have been more prepared for my SATs!

By stressing a quantity target for primary enrollment, the Education MDG may cause kids to be herded into classrooms and quality to go to the pits. Bill Easterly

For NCLB the hidden agenda was (is) to dismantle the teacher’s unions and eliminate tenure. To do this the law was set up in such a way as to ensure that all schools would eventually “fail”. This is accomplished by requiring schools to show annual progress, regardless of the level at which the school is already performing. This is, of course, mathematically impossible. Where do you go when your school is in the 99% percentile? The fallacy was made obvious when one of the best HS in NYC “flunked” on the cities own equivalent measure.

The proponents don’t care a fig about improving the quality of inner city schools, they just want to transfer money to private, parochial and/or for-profit replacements. Discrediting the public school system is part of the intent.

In the case of foreign aid the PR is always framed in terms of some version of the “White Man’s Burden” as our host has so aptly shown. What needs to be discussed in depth is what the hidden objectives are.

It seems to me they may fall into several areas:

1. Corrupt officials supporting their profligate lifestyles.

2. Multi-nationals gain access to markets or resources under the umbrella of the aid program.

3. Feeding aid money back to the giving nation by restrictions on how it can be spent. For example, buying US agricultural commodities with the aid money and shipping them to the recipient country. Much military aid works this way as well.

4. Bureaucrats at many levels working to keep their cushy jobs. Since there work “product” is paperwork, that’s what gets produced by themselves and those they monitor.

5. Buying influence with foreign governments through bribery or extortion as part global power politics. Every time we buy an African leader we make it harder for the Chinese of Russians to do this instead.

Nowhere in this list will you find any mention of helping the poor or helping development. Any such success would lead to the dismantling of the aid system and all those who benefit from the status quo. It’s like the modern medicines which treat “diseases” that never end (like high cholesterol) and thus require use for the rest of one’s life – a drug firm’s gold mine.

Nobody knew what the current size of the civil service was!

Yet, a common problem with RBM I see in World Bank evaluations is that CASs and projects set out specific target outcomes without baselines (when the target is an increase or decrease) and/or that can’t be measured because of weak capacity, lack of interest, or changes in priorities by statistics gathering groups in country.

Another issue is that projects and CASs have perhaps too many target outcomes, e.g. 43 in the case of one CAS I’m looking at. With so many, it seems inevitable that many won’t be achieved as conditions and priorities change over a 3 year period. Thus RBM doesn’t help as much as it could to judge whether the overall aid program was worthwhile.

The PART system developed by OMB for US federal agencies is a good RBM model to look at.

The same goes for using investment/markets and profits to incentivize development – yes the system can be gamed but those who seek to consistently perform, isn’t it a lot easier to seek excellence than it is to consistently game short term tests?

Here’s a thought provoking example of one charter school in California that is rather unapologetic to teaching to the tests – but that’s of course not all they do: http://www.latimes.com/news/local/la-me-charter31-2009may31,0,7064053.story