This essay first appeared in my newsletter. Sign up here if interested in F’ing up the status quo and fixing education.

In the grand theater of American higher education, the most anticipated and controversial performance is the annual unveiling of the U.S. News & World Report college rankings. 

Think of these as a high-stakes beauty pageant for academia (less the swimsuits) that influence the decisions of millions of students, parents, and even the institutions they aim to evaluate. 

Underneath the glossy veneer of rankings that suggest numerical precision and accuracy lies a system full of flaws, manipulations and unintended consequences.

I propose a better way below but first, it’s worth digging into why the US News Best Colleges rankings so useless.

The Myth of Objectivity

Peter Bernstein, a former managing editor at U.S. News, offered a shockingly candid view behind the curtain when he describes the rankings’ creation as “full of value judgments, full of problems, challenges, dangers.”  

The rankings began as a “promotional gimmick” in the 1980s and now have morphed into what Bernstein accidentally dubbed a “mythology” (rather than a methodology) in a conversation with Malcolm Gladwell.

That’s quite the slip-up.

What the rankings have masterfully done is provide an illusion of objectivity.

By assigning precise numerical scores to institutions, U.S. News College Rankings have created a belief in their scientific accuracy. But as Malcolm Gladwell and others have pointed out, this precision is artificial. 

In fact, the rankings heavily weight subjective factors like “peer assessment” which amount to little more than a popularity contest among college administrators.

A Chorus of Criticism

Critiques of the U.S. News rankings are not new and span decades and disciplines, with numerous researchers and writers highlighting their flaws.

The studies collectively highlight how the rankings:

• distort institutional priorities
• overemphasize subjective measures
• incentivize manipulation
• focus on inputs over outcomes

A very tiny selection of these studies and their conclusions is provided below:

“Reputation Versus Reality: Ranking Major Colleges and Universities“

By James Fallows (1997) 

Findings: U.S. News rankings relies on subjective surveys that reflect reputation rather than actual educational quality. He highlights manipulative tactics like how inflated alumni giving rates to boost rankings.

“The Order of Things” 

By Malcolm Gladwell (2011)

Findings: Rankings rely excessively on peer assessments which are subjective and often are more reflective of an institution’s historical reputation rather than its current educational quality and outcomes. He argued that the rankings emphasize wealth and exclusivity.

“Is There Life After Rankings?”

By Colin Diver (2005) 

Findings: Details how colleges massage data like class sizes to inflate rankings and how the “urge to improve one’s ranking creates an irresistible pressure toward homogeneity”.

“College Rankings Reformed”

By Kevin Carey (2006)

Findings: Rankings push colleges to invest in the non-academic as they’re primarily driven by fame, wealth, and exclusivity. Rankings often measure inputs like spending instead of outcomes, i.e. what students gain educationally. 

Gaming the System

If the rankings were merely flawed, that would be bad.

But the fact that they can also be gamed is where the real trouble begins. 

Rowan University saw a massive rise in its rankings (from 193rd to 163rd in a single year) and it is a case study in how to play the U.S. News game.

I will say upfront that their strategy and President are ingenious for finding this hack to boost its peer assessment score which accounts for 20% of the overall ranking.

Their moves blended marketing savvy with good old free food.

Rowan sent bottles of its own branded “Houshmand’s Hazardous Hot Sauce” to provosts and presidents of other universities.

Named after Rowan’s president, Ali Houshmand, who actually makes the hot sauce as a hobby, this promotional item served multiple purposes:

1. Increased Name Recognition: By sending a memorable and pretty unusual item, Rowan helped fortify its name with other university leaders come time to fill out those important peer assessment surveys.
2. Positive Association: The hot sauce is undoubtedly unique. It’s also a personal touch from the university’s president which creates a positive association with Rowan in the minds of survey respondents.
3. Conversation Starter: The hot sauce bottles became a talking point, increasing the likelihood that other university leaders would discuss Rowan, further enhancing its visibility and reputation.
4. Demonstrating Innovation: The hot sauce campaign highlighted Rowan as creative and ‘outside-the-box’ thus impressing peers with the university’s innovative thinking.

This hot sauce strategy, while seemingly trivial and totally unrelated to education quality, exemplifies the lengths to which institutions will go to game the rankings system. 

It highlights how factors that have zero to do with educational quality can influence a school’s perceived prestige and its resulting ranking.

While incredibly clever, Rowan underscores the fundamental issues with the ranking system: it rewards marketing prowess and flashy resource allocation strategies over genuine improvements in educational quality and student outcomes.

The Collateral Damage

The consequences of this rankings-obsessed culture are wide and detrimental. 

Even Robert Morse, US News’ Chief Data Strategist, whose team is in charge of the mythology methodology is surprised by the significance they’ve taken.

“In the early years, the thing that’s happening now would not have been imaginable. This idea of using the rankings as a benchmark, college presidents setting a goal of ‘We’re going to rise in the U.S. News ranking,’ as proof of their management, or as proof that they’re a better school, that they’re a good president. That wasn’t on anybody’s radar. It was just for consumers.”

– Robert Morse, US News & World Report

For students and parents who often treat the rankings as gospel, they may make life-altering decisions based on flawed data and mythology. 

For colleges and universities, the impact is equally troubling as highlighted above in the Rowan case study. 

Resources get diverted from genuine educational improvements to ranking-friendly investments and tactics. The pressure to climb the rankings ladder distorts institutional priorities. 

In essence, the tail is wagging the dog, with educational quality and true outcomes taking a backseat to performative ranking performance.

Reimagining Rankings

So what would a better system look like? 

An alternative ranking system based on data would focus on outcomes rather than inputs. 

Instead of measuring an institution’s wealth or selectivity, it could track metrics like:

1. Post-graduation success
   • Employment Rates: Track the percentage of graduates employed in their field of study within six months to a year after graduation.
   • Earnings Data: Use data on median salaries adjusted for regional cost of living and industry to assess economic outcomes.
   • Graduate School Acceptance Rates: Measure the proportion of graduates who pursue and are accepted into advanced degree programs. (adjusted for field of study so acceptance into medical school > masters in art history)

2. Affordability and financial health
   • Net Price: Examine the average cost to students after financial aid, focusing on affordability for low- and middle-income families.
   • Student Debt Levels: Consider average debt at graduation and loan default & repayment rates. This might also help reign in educational debt factories.

3. Alumni satisfaction and career progression over time
   • National Survey of Student Engagement (NSSE): Incorporate data on student participation in educational practices associated with high learning and development levels.
   • Alumni Surveys: Collect feedback on graduates’ satisfaction with their education and its relevance to their careers.

4. Social mobility contribution
   • Economic Mobility Index: Calculate how effectively institutions enroll students from low-income backgrounds and graduate them into higher-income brackets.
   • Pell Grant Student Graduation Rates: Assess the success rates of low-income students receiving federal aid. (note: would have to normalize somehow for absurd grade inflation going on at present)

This system would be superior because it aligns incentives with actual educational value and not all the other things the current rankings prioritize. 

It would, most importantly, encourage institutions to focus on what matters most: preparing students for successful, fulfilling lives post-graduation.

Notes:

• All of this data is not publicly available although a good amount is.
• Doing this would require some creativity and likely the support of forward-thinking and confident colleges & universities to share their data and/or some forward-thinking regulator or policy-maker to push for this level of transparency.
• It may also be that this data is available (or proxies are) and my sleuthing did not uncover it. If so, definitely reach out to me, and I will update this essay.

Conclusion

The U.S. News & World Report college rankings are a prime example of Goodhart’s Law in action: “When a measure becomes a target, it ceases to be a good measure.” 

What began as an attempt to provide useful information has become a self-perpetuating system that ends up distorting the very thing it aims to measure.

As consumers of information, both students and parents have to approach these rankings with a critical eye and an understanding of just limited, biased, un-scientific and gameable these rankings are.

Although it is hard, we should also push for more meaningful ways to evaluate and improve our institutions. 

The goal of higher education should be to create an environment where students leave smarter and more capable to tackle the challenges of the real world. 

And so any ranking system that doesn’t directly serve this goal is, at best, a distraction, and at worst, actively harmful. 

We can start to address this with a better methodology (and less mythology).

—

If you have other metrics you believe should be used in a reimagined US News & World Report college rankings, let me know.

If you have an interest in and the capability to build this or are a policy maker who can help push for this as well, please reach out as well.

I’m at anands (at) gmail

If you read this far, some related essays you’ll also like:

1. Degrees of deception: How America’s universities became debt factories
2. Bread, circuses and education
3. The School of Entrepreneuring
4. The perverse incentives driving America’s government schools
5. The endless ladder
6. Solving the wrong problems
7. Your kids grades are bullshit
8. Harvard: The Birkin Bag of Education