On the relevance of university rankings

What are some of the most popular rankings schemes worldwide? Is the number of citations an adequate marker of a university’s research excellence? What are the concerns regarding conflicts of interest and data security issues?

SThe story so far: ince the first appearance of global university ranking systems around two decades ago, rankings have come to dominate the attention in higher education ecosystems around the world. Today, many countries including China, Japan, and Russia have committed substantial resources to elevate the statuses of their universities to “world class” as defined by these rankings, allowing them notinconsiderable political heft as well. Of late, however, some universities worldwide have pulled out of being ranked, over concerns about the incentives the systems set up and their compatibility with the universities’ own aspirations. But the din of these controversies have also drowned out two crucial aspects pertaining to the conduct of the companies behind some of the ranking systems. They are conflicts of interest and data rights.

What do ranking systems do?

At present, the Times Higher Education (THE), the Quacquarelli Symonds (QS), the Academic Ranking of World Universities (also known as the ‘Shanghai Ranking’), and the U.S. News & World Report are the most popular rankings schemes worldwide and hold significant weight and influence in shaping educational policies and priorities in the higher education sector in many countries.

A ranking system orders the higher education institutes in a place (country, region, etc.) by their accomplishments on various fronts — including teaching, research, reputation, industryfocused research, and collaborative efforts. Each of these activities is complex, multifaceted, and highly contextual, but for the purposes of the ranking, an institute’s performance on each one is translated into a few composite indicators, which are then combined to create a consolidated score.

Are ranking systems perfect?

In 2021, Elizabeth Gadd, a research officer at Loughborough University in the U.K., published a critique in which she reported that universities’ quests for higher ranking mirrors the flawed pursuit of Gross Domestic Product (GDP) as the sole measure of a country’s prosperity.

For example, in their 2010 book Mis-Measuring Our Lives: Why GDP Doesn’t Add Up, eminent economists Joseph Stiglitz, Amartya Sen, and JeanPaul Fitoussi contended that the use of a single indicator to capture the economic and social progress of a country will inevitably overlook the environmental impact of its growth and measures of its inequality, among other crucial issues. According to Dr. Gadd, university rankings, like GDP, distill complex roles that universities play in society into a single, unidimensional score.

Experts have noticed that the highest ranked universities in various ranking systems are old, large, wealthy, researchintensive, sciencefocused, Englishspeaking, and in the Global

North. Studies have also shown that higher scores in research excellence in rankings are influenced to a great degree by two factors: citations and reputation. For example Bielefeld University leaped from 250th to 166th in the 2020 THE rankings. The jump has been attributed to a single scholar’s work, who published 10 papers, coauthored with hundreds of other researchers, most of them in The Lancet, contributing to 20% of the university’s total citations over two years. (These citations are not spurious but overrepresented.)

Arbitrary measures of research excellence like citations can dramatically alter an entire university’s performance in the rankings. For example, in 2023, Science reported the case of Saveetha Dental College in Chennai rocketing up the ranking ladder allegedly by manipulating citations.

In two analyses in 2016, Richard Holmes, an expert in ranking systems and who has been running the ‘University Ranking Watch’ initiative since 2006, wrote that THE’s regional rankings appeared to favour universities that hosted an important THE summit. According to Mr. Holmes, these changes in favour were effected by, among other things, tweaking the way the ranking system counted citations. There are many similar instances, incentivised by the value accorded to ranking schemes and the riches that universities that rank highly reap.

What are the concerns over conflicts of interest?

Most entities that compile and publish rankings are private enterprises, and there have been instances of these entities consulting with universities to help the latter achieve better ranks in their own systems.

For example, in a 2021 paper, Igor Chirikov of the University of California, Berkeley, reported that “universities with frequent QSrelated contracts had an increase of 0.75 standard deviations (~140 positions) in QS World University Rankings and an increase of 0.9 standard deviations in reported QS facultystudent ratio scores over five years, regardless of changes in the institutional quality.” His study was based on the ranks of 28 universities in Russia between 2016 and 2021. Likewise, THE offers membership to an elite group it runs called “World 100 Reputation Network”. Dr. Gadd wrote that it’s intended “for institutions ranked in the top 200 of one of the big four global rankings to … share strategies for retaining their ranking topping status”.

Since these problems started to become more apparent, several prominent institutions have denounced traditional ranking systems. In 2022, Harvard and Yale Universities led a boycott against the U.S. News & World Report’s ranking over what they said was a conflict between the careers they wished their law students to have after graduation and the careers the ranking incentivised. Utrecht University in the Netherlands withdrew from the THE world rankings in 2023 for similar reasons. In India, several IITs have boycotted the same rankings.

What about data security?

Like conflicts of interest, another issue deserves similar examination: by participating in ranking exercises, universities and institutes provide ranking agencies free reign over their data, compromising their data security.

For example, to use the THE platform, website, and related services, universities are required to agree to an additional set of terms and conditions in addition to the general set. According to point 6 of the former: “You grant THE a royaltyfree, perpetual, irrevocable, nonexclusive and fully sublicensable right and license to use, reproduce, modify, adapt, publish, translate, create derivative works from, distribute, perform and display such data (in whole or part) worldwide and/or to incorporate it in other works in any form, media, or technology now known or later developed. With respect to all Data you post to the Website, you hereby waive any moral rights you have in the Data. You agree to perform all further acts necessary to perfect any of the above rights granted by you to THE, including the execution of deeds and documents, at our request”.

That is, universities are to give THE a free and permanent right over their data without having to seek their permission in the future. Such data includes details of institutional, industry, and research incomes, and of patents. There is no reason why universities, especially public universities, should agree to such a wideranging grant of rights in order to participate in the ranking exercise.

As the UN University’s statement on ‘Global University Rankings’ reads:

“While rankings may have incentivised some improvement in the quality of some universities, there is growing recognition that they also incentivise a number of perverse and harmful behaviours and produce systemic longterm negative effects.”

Moumita Koley is an STI Policy Researcher, DST-CPR, IISc, and consultant, International Science Council.