Today’s topic was spurred by some rather unusual college rankings by the Wall Street Journal, in which Texas Tech has a higher rank than Harvard. This has been among the most popular articles on the Online Journal for 3 straight days now. Of course, also very popular are the US News and World Report College Rankings (which give the rather opposite results in which Harvard does slightly better relative to Texas Tech.)
We all love rankings.
Talking about ranking methodology, not so much.
(Turns out WSJ rankings were biased towards larger colleges. Texas Tech had undergraduate enrollment of over 22 thousand, compared to less than 7 thousand at Harvard.)
In development, we have all kinds of ranked beauty contests: the Human Development Index (HDI), Doing Business, Governance Indicators, Corruption, etc. etc. I’ve even been participating in one ranking exercise myself, on, of course, aid agencies.
Just like wacky college rankings, many development rankings are based on methodologies that could use a lot more scrutiny than they usually get. Take the much-loved uber-publicized Human Development Index (HDI). The HDI has a hidden implication pointed out by the World Bank’s Martin Ravallion as long ago as 1997. Imagine the reaction to this hidden implication if they publicly announced it:
The UNDP announced today that they consider a human life in the USA to be 70 times more valuable than human life in the Democratic Republic of the Congo.
The other consequence of this hidden implication is that rich countries will get A LOT of credit for higher life expectancy (and not much for income). Scandinavia does very well on the HDI because its life expectancy is higher by 1 or 2 years than the US, even though US income is higher. Bryan Caplan suggests:
Scandinavia comes out on top according to the HDI because the HDI is basically a measure of how Scandinavian your country is.
There’s an alternative to constructing murky indexes with dubious assumptions. This would be to be use people’s actual choices to infer which places are better. If a college recruiter (the basis of the WSJ rankings) had to choose between otherwise equivalent Texas Tech and Harvard graduates, which one would they pick? Instead of the US News convoluted rankings, why not just ask a student admitted to both Texas Tech and Harvard which one they would pick (or DID pick). These are called “revealed preference rankings” and have actually been done for colleges.
Instead of Human Development Indexes, why not just ask migrants where they would choose to live (or do actually choose to live): US or Iceland? Iceland has a much higher HDI, but actually has out-migration, whereas I think there are a few people trying pretty hard to get into the US.
Instead of Doing Business index-based rankings, why not use the actual choices of businesses where to invest? These also have the added attraction of not being subject to artificial manipulation (which happens with both college index rankings and Doing Business).
Indexes should not be discarded altogether. Sometimes there are no choices to guide rankings (like aid agencies, sadly). They can also summarize information for those making the choices.
But please double check your methodology.
16 Comments
I’m not sure asking migrants where they would choose/did choose to live is a very telling indicator of the quality of life for given country. Wouldn’t you need to account for asymmetric information in this approach, i.e. poor people may know nothing about Iceland so they chose America instead?
You seem to be presuming equal knowledge (or access to knowledge) among migrants, which I think is a questionable assumption. No offense to Icelanders, but their country isn’t exactly a giant on the world stage. By contrast America has a huge political and cultural impact on other nations, even remote and isolated regions. This affects knowledge and perceptions. Not to mention America’s larger population, so poor migrants (with limited information) may rely more on kinship connections or 2nd hand info in deciding where to go, factors both of which may be more applicable to a very large nation with a strong history of immigration (America) than a smaller one. Also, geographic access points may influence migration choices, regardless of perceived quality of life (America is big with a lot of ways to get in, legal or otherwise).
On another note, the QS World University Rankings and Times Higher Education World University Rankings both recently came out, also with different results. I haven’t looked at your WSJ uni rankings link yet but I’m guessing they’re different also.
Maybe I should just tell my kids to pick a school using the dartboard method? Seems whatever they hit, it’s bound to end up on someone’s “best of” list!
Hmmm, revealed preferences are incredibly important, but are they always informative about what the ideal should be? Should we stop producing nutrition tables and BMI indices because revealed preference tells us that Americans prefer to be fat, fat, fat?
I’m curious about how your revealed preference mechanism would work on aid agencies. If you use the aid-givers revealed preference, probably you’d end up with a ranking topped by all the agencies that do all the bad practices that we are trying to avoid.
So, your point is that the act of ranking presupposes some kind of predefined (string) scale?!
Or, that popular opinion, without a defined scale, is the superior ranking?!
I’m disappointed! I wanted to know what your top 25 rankings were… bummer.
good points. My favorite index story is on the TI’s Corruptions Perception Index. It’s composed of several data sets, but the datasets and their weightings change every year.
Nice game, that.
Thanks Bill, excellent point.
Living here in Rwanda, the World’s Number One Business Reformer according to the World Bank, feels like a bit of a joke. We rank higher than China, but businesses are closing here rather than opening. If a young guy decides one morning to pile up some mangos and sell them on the roadside, he’ll be in prison by the evening.
You have to ask yourself what the hell these World Bank people are doing. The answer of course, they review the “regulatory environment”, which means that all you need to do to jump above 100 countries is introduce a new law that the WB are happy with. I.e. a piece of paper that need have no relation to reality.
By a revealed preference criteria all toxic financial assets must be valuable, otherwise, the bank portfolios would not be stuffed with them. I guess a purpose of the index is to support more informed choices about where to live or what to buy. Given that AAA credit rating is an index, it does not work well, either.
It is hard to swim across the river on the US Mexican border, it is nearly impossible to navigate icy, stormy waters to Iceland. But if you could fly in legally, why would you chose US over Scandinavia?
On the Corruption Perception Index the bias is the other way round of course. If you’re a businessman and asked about how big an impediment corruption is, ceteris paribus the higher the growth in the country, the more likely you are to view it as “relatively uncorrupt”.
Countries with low growth due to other exogenous factors than corruption see their rank experience a strong corruption “perception” bias.
Methodology should also be informed by what you want to use your rankings for. If you are attempting to produce rankings for purposes of comparison (but not decision making), using revealed preferences in your methodology makes sense. That doesn’t work, however, if you are trying to rank a specific characteristic that you particularly care about. As an example, I wouldn’t want to move to Florida because they have flying cockroaches there and they gross me out (seriously, ew). Here, an index of “# of flying cockroach species known to inhabit a state” would be very useful to my decision making. An index of “net population inflows by state” would not.
As a child of two Texas Tech graduates who was born at the campus hospital, I feel like there’s something I need to defend here. But it’s not bad methodology. I’m just not convinced that revealed preferences always tell us what we think they do, which is why I’m not using them in a current project about the quality of social service provision in the eastern DRC. Just is often the case with kids from farms and oil fields in West Texas, who, accepted to Harvard or not, will probably not be able to go and will therefore choose Tech, Congolese mothers don’t often have much of a choice about where to take their children for health care. They use the place that is closest because they can’t afford to get to another, or a bureaucracy constrains their choices, or it’s too far to walk.
Laura, Oh I knew I was going to offend someone, I’m sorry it was you! I am myself a BA from a not-big-name undergrad university and I don’t want to imply any condescension or sense of superiority towards those schools.
Let me try this revised wording: the WSJ gave a high ranking to schools not usually considered to be high ranking, even though I myself am a strong admirer of {insert name of your school here}.
Bill
Bill, I wasn’t offended in the least. Clearly any metric that attributes value to research, an intellectual environment, a great library, etc. would rank Harvard higher. Football is another matter…
Seriously, I found this post very interesting because it points out something that’s not always obvious to the general public: how you measure something really matters. And I’m spending most of my time these days thinking about a troublesome measurement instrument.
Curiously, the World Bank produces both the Doing Business indicators, which rank countries based on an admittedly imperfect set of measurements of the regulatory environment, and the Enterprise Surveys (http://www.enterprisesurveys.org/), which ask what businesses actually do in practice. The Doing Business indicators seem to get all the attention — the world seems to get the data that the world wants.
However, there is a limitation with revealed preference in looking at how businesses invest. You can only get data from businesses that actually exist — i.e., those that have managed to adapt themselves to the existing business environment, which may be horrendous. You might get the idea that Rwanda (to use an example of another commenter) is a great place to invest in the mango retail business. In fact, it may simply be that the existing businesses have no better alternatives. (A similar point to what Laura was saying re: social service provision.) Because of this, it’s useful to provide policymakers a rough idea of what a decent business environment looks like, which only a ranking like Doing Business can do.
Also to Joe’s point–Rwanda is not ranked #1 overall in the Rankings. That privilege goes to Singapore. Rwanda is ranked as the top reformer, meaning it has moved the most places in the rankings. In economic terms, Rwanda has also done rather well in recent years, growing at over 7% per annum in real terms (at least before the financial crisis). I wouldn’t pretend to claim causality, but Rwanda doesn’t seem to bear out your claim that the reality is so different from the reformer status.
Ryan
Two points – one, don’t necessarily believe the growth figures, I know how they’re calculated and it ain’t all that robust. Two, I said top ‘Reformer’, but they are still ranked above China and the point still holds.
On the enterprise surveys, I agree they are a much more important and interesting source of data.
Gallup has such index of revealed preferences http://www.gallup.com/poll/124193/Potential-Net-Migration-Change-Developed-Nations.aspx
3 Trackbacks
[...] This post was mentioned on Twitter by Mark Thoma, Robert Went. Robert Went said: Top 25 Rankings of All Time – William Easterly Aid Watch – http://icio.us/3p5kwc [...]
[...] Top 25 Rankings of All Time [...]
[...] like Bryan Caplan’s suggestion (quoted by Bill Easterly) that: Scandinavia comes out on top according to the HDI because the HDI is basically a measure of [...]