When economists talk about research quality, they often talk about journals. But how confident can we really be about journal rankings? In our new paper, Confidence Intervals for Recursive Journal Impact Factors (Tinbergen Institute Discussion Paper 22-038), we address a simple but long-overlooked question: How much uncertainty is hidden in the numbers behind journal metrics?
Recursive impact factors adjust for the prestige of the citing journals. This makes them particularly attractive in economics, where citations from top journals are weighted more heavily than others. But while the concept is widely used, little attention has been paid to the uncertainty around these indicators.
We propose a bootstrap method to compute confidence intervals for both the recursive impact factors and the resulting journal ranks. Applying this to a set of 319 economics journals reveals striking results:
- The Quarterly Journal of Economics stands clearly at the top.
- Based on the simple bootstrap, the remainder of the “Top 5” journals are in the top 6 together with the Journal of Finance
- Many mid-ranked journals have wide intervals, with rank uncertainty spanning 25 to 100 places depending on the method.
- Most apparent differences in journal quality are, in fact, mostly insignificant.
The figure below illustrates the range of uncertainty:
Figure 1: 95% confidence intervals of the recursive impact factor, arithmetic scale (left axis) and logarithmic scale (right axis)
Our findings suggest that journal rankings should be interpreted far more cautiously than is often the case. Especially for journals in the middle of the distribution, apparent differences in prestige may not be statistically meaningful. This has implications for how researchers, institutions, and funders use rankings in decision-making.
The full paper is available here:
Dr. Johannes König 