Skip to content

More Tales of Two Tails

The following post is by Dennis Whittle, co-founder of GlobalGiving. Dennis blogs at Pulling for the Underdog.

An eloquent 3 year-old would have been better asking “What the dickens are you talking about?  Who is defining success?  Who says failure is bad, anyway?” – Joe

Earlier I blogged about aid cheerleaders and critics. Each camp argues about the mean outcome of aid rather than the distribution of impact among projects. Both camps agree that some projects have positive results and others negative. So why not try to figure out which projects work and focus our resources on them?

I got some great and insightful comments and a few nice aid distribution graphs from readers.  Here are some key themes:

  1. The mean *does* matter if the distribution is random. In other words, if we can’t predict in advance what types of projects will succeed, we should only spend more resources if the mean outcome is positive.
  2. Many people believe that on average the biggest positive returns come from investment in health projects.
  3. We should also look at the distribution of impact even within successful projects, because even projects that are successful on average can have negative impacts on poorer or more vulnerable people.
  4. Given the difficulty in predicting ex-ante what will work, a lot of experimentation is necessary.  But do we believe that existing evaluation systems provide the feedback loops necessary to shift aid resources toward successful initiatives?
  5. “Joe,” the commenter above, argues that in any case traditional evaluators (aid experts) are not in the best position to decide what works and what doesn’t.

From reader Steve White: "Here is my graph based on two stylized facts about aid projects: 1) most projects have very marginal impacts (agricultural tools to villages, microcredit, school construction, textbooks, scholarships, deworming...) and 2) some health projects have HUGE impacts (vaccinations, DDT, bednets)." The two bars represent impacts between -1 and 0, and between 0 and 1

From reader Daniel Kyba: "Those which do a good job are the ones with defined and observable measures - profit/loss; live/die and so on. These measures provide a form of a feedback mechanism at the project level to which the aid provider can respond. As you move towards the world of fuzzy concepts and measures that is where the ineffectiveness occurs, due to the lack of feedback mechanisms and because there is less definition of success/failure."

Petr Jansky sent a paper he is working on with colleagues at Oxford about cocoa farmers in Ghana.  The local trade association was upset that they could not get pervasive adoption of a new package of fertilizer and other inputs designed to increase yields.  According to their models, the benefits to farmers should be very high.  The study found that – on average – that was true, but that the package of inputs has negative returns to farmers with certain types of soil or other constraints.  Farmers with zero or negative returns were simply opting out.

At first glance, these findings seem obvious and trivial.  But they are profound, in at least two ways.  First, retention rates are an implicit and easily observable proxy for net returns to farmers.  We don’t need expensive outside evaluations to tell us whether the overall project is working or not.  And second, permitting farmers to decide acknowledges differential impacts on different people even within a single project.

What other ways could we design aid projects to allow the beneficiaries themselves to evaluate the impact and opt in or out depending on the impact for them personally?  And how would it change the life of aid workers if their projects were evaluated not by outside experts and formal analyses but by beneficiaries themselves speaking through the proxy of adoption?

Be Sociable, Share!
This entry was posted in Aid debates, Metrics and evaluation and tagged , , , , , , . Bookmark the permalink. Follow any comments here with the RSS feed for this post. Both comments and trackbacks are currently closed.

6 Comments

  1. Alex Jacobs wrote:

    That’s got to be a bullseye about the importance of direct feedback from local people. I couldn’t agree more. Listening to what they say could revolutionise how aid projects work. Some great examples are emerging from NGOs round the world – eg see http://ngoperformance.org/related-initiatives/feedback-examples

    Posted April 28, 2011 at 6:31 am | Permalink
  2. Cripes. This talk of feedback from locals as a priority for aid evaluation has been going on forever. I think most people generally agree it is a good thing. So, why doesn’t it happen?

    If the aid ‘community’ really wanted to put local development into the hands of locals, they’d start by handing out 100 dollar bills, or something, instead of the current social engineering model.

    Posted April 28, 2011 at 1:31 pm | Permalink
  3. Jeff wrote:

    Yes, the lack of efficient feedback loops that weed out the failures is THE problem in development. Agreed that having customers opt in is the best way to judge success, but not all necessary development interventions have such mechanisms. It works well with product social marketing, but not in policy reform or governance. Policy reform is often very important, but many of the clients (government officials) for policy reform decline to implement it because it doesn’t serve their personal interests even if it might serve the interest of the customer (citizens).

    Your example from Ghana shows the other problem in development, what I call the myth of the magic bullet. Even if there is a useful innovation that achieves good results for 60% of the customers, there is a perception of failure because 100% did not adopt it. Usually such cases are followed by more investment devoted to fixing the “failure” of only reaching 60% of the customers needs instead of looking at better investments where early adopters can be found more effectively.

    Posted April 28, 2011 at 4:35 pm | Permalink
  4. Chris wrote:

    Jeff put back the discussion where it should be. I want to go back even further before the ‘policy or governance’, before the introduction of development aid and evaluation back to the indigenous markets and the cash crop markets like cocoa that propped-up during Colonial times and regimes that formed around them since independence.

    The entire policy or governance and the introduction of developmental aid are still implemented around the same institutions at present as they where before. The indigenous markets are still a throwback for the same commodities as before. The ‘policy or governance’ is also built around the same commodities and institutions.

    What puzzles me in understanding the development debate; does a given economy exist to experiment the aid model and measuring its impact or to independently develop and sustain on its own to produce and service the indigenous population and what ever is profitable? If the later is true, then why is noting resembling developing the indigenous market has not been done by any development aid. And then, what is the point of measuring the impact of entitlements that can not sustain without continuous aid.

    Can any one enlighten me why no one seems to out the issue in front burner?

    Posted April 28, 2011 at 8:33 pm | Permalink
  5. James Moore wrote:

    Chris I agree aid programs can not simply prop up existing institutions and hope that leads to their success. Indigenous economies must develop to be self supporting or the entire goal of development aid is bogus. The important thing to remember is that countries and economies both develop and evolve naturally over periods of time. Simple financial support or aid programs could boost an economy temporarily, but until the people living in the country begin innovating in their own way that is all it will be, temporary. It is easy to forget the psychological changes nations have gone through and must continue to go through to achieve a prosperity.

    Posted May 2, 2011 at 8:25 pm | Permalink
  6. Chris wrote:

    Moore, you rightly said ‘Countries and economies both develop and evolve naturally over periods of time’. Then wouldn’t you agree institutions that makes the natural evolution possible are missing by unnatural interference, partly sustained by development aid?

    Then, the question is why they are missing or not forming? The record shows the institutions under regimes in power and development aid that sustain them are in direct contradiction with the evolution or hinder the formation of these institutions.

    Could it be these divergence of interest; the former to sustain the status qua and the later to unleash the ‘natural evolution’ is what kept economies stagnant?

    Finally, what does monopoly of public information by a regime plays in the lack of evolution?

    Posted May 5, 2011 at 11:24 am | Permalink

2 Trackbacks

  1. By More Tales of Two Tails | Global Health Hub on April 28, 2011 at 7:44 am

    […] the article here: More Tales of Two Tails AKPC_IDS += "18375,";Weight: […]

  2. […] Link to the original site Filed in Aid by Mark Oppenneer SHARE THIS Twitter Facebook Delicious StumbleUpon E-mail « Can the World Bank Pay for Results or Will Critics Make It Impose Conditions? » Global Chocolate and Cocoa Organizations Pledge Humanitarian Support for the Ivory Coast No Comments Yet […]

  • About Aid Watch

    The Aid Watch blog is a project of New York University's Development Research Institute (DRI). This blog is principally written by William Easterly, author of "The Elusive Quest for Growth: Economists' Adventures and Misadventures in the Tropics" and "The White Man's Burden: Why the West's Efforts to Aid the Rest Have Done So Much Ill and So Little Good," and Professor of Economics at NYU. It is co-written by Laura Freschi and by occasional guest bloggers. Our work is based on the idea that more aid will reach the poor the more people are watching aid.

    "Conscience is the inner voice that warns us somebody may be looking." - H.L. Mencken

  • Recent Comments

  • Archives