3 Flaws in the Journal Impact Factor

by | Jun 27, 2019




Since 1961, the citations of scientific journals around the world have been added up and quantified in order to formulate a journal impact factor (JIF). These findings are calculated by Thomson Reuters and published yearly in the Journal of Citation Reports, according to the article The Journal Impact Factor : Moving Towards an Alternative and Scientometric Approach. However, the JIF has come under growing criticism for its various shortcomings. These include its ineffectiveness as a tool for predicting an article’s impact, the opaque methods used to calculate it, and its potential for misuse.

Lack of Effectiveness

In the Science article Hate Journal Impact Factors? New Study Gives You One More Reason, Lucas Carey, a cell biologist at Barcelona’s Pomepeu Fabra University, notes that one of the biggest problems with the JIF is that it is “meaningless as a predictive measure.” Even if a given paper is published in a journal with a high impact rating, this does not mean that this individual article is more likely to be cited.

In the article Why The Impact Factors of Journals Should Not Be Used for Evaluating Research, published in the British Medical Journal (BMJ), authors note that the JIF is a reasonable tool for predicting impact only if the citation rates of all the individual articles fall close to the value of the JIF itself. However, studies of biochemical journals, for example, showed a wide distribution of article citation rates in a given journal. In this study, only a few articles had a citation rate whose value was close to the JIF.

Opaque Calculation Methods

Another shortcoming of the JIF is that, according to Science magazine, the methods by which it is calculated are opaque. The magazine notes that this opacity comes largely from Thomson Reuters. This company makes money by selling subscriptions to its extremely comprehensive database — the Web of Science — that allows for comprehensive citation searching. However, Thomson Reuters does not make the citation data it curates available to the public.

Because Thomson Reuters does not provide the raw data upon which it bases its work, JIF numbers are often not able to be reproduced.

Potential for Misuse

Yet another reason why the JIF comes under fire from its critics is due to the large potential for misuse. According to the article in Nature magazine Beat It, Impact Factor! Publishing Elite Turns Against Controversial Metric, Heidi Siegal, a spokesperson for Thomson Reuter, calls the JIF a “broad-brush indicator of a journal’s output” and notes that it should not be used to determine the quality of the individual authors or articles within that journal. However, the article also mentions Stephen Curry, a biologist at the Imperial College in London, who reports that many researchers domisuse the JIF as an evaluation tool for the articles within a journal — and this, he notes, can influence important decisions like university hiring or funding for research.

The JIF has been around for decades, and many have considered it to be a useful tool for assessing the quality of a journal and the articles published within it. However, this system has come under heavy criticism for many serious shortcomings, including its lack of effectiveness in predicting the impact of individual articles, the opaque means by which the JIF is calculated, and its significant potential for misuse.



Medmeme Blog Footer banner
© Pharmaspectra Group Limited, 2020
All Rights Reserved
2 Eastbourne Terrace
London, W2 6LG