Sir

The assessment of research quality is increasingly based on impact factors and citation analyses of published work. But there are possible biases in these measures1,2,3,4. Is the fact that Italian, French, German and Japanese publications received a less than average share of citations due to a lack of quality3 or to systematic bias1?

Further analysis of the data in ref.1 on papers by Italian scientists reveals a strong positive relationship between journal impact factor class (IFC) and proportion of publications undercited (Fig. 1). Furthermore, the proportion of publications undercited in the highest IFC is significantly greater than the corresponding proportion in the lowest IFC (χ21 = 4.89, P = 0.027). These effects strengthen the negative implications of underciting for Italian scientists: not only is their work undercited in general but, as the quality of their work increases, it is less recognized! These results clearly support Paris et al.'s conclusions1 that undercitation does not result from substandard publications3.

Figure 1: Relationship between the proportion of Italian publications undercited and the journal impact factor class (Spearman's r = 0.964, N = 7, P < 0.001).
figure 1

The numbers above the bars indicate the number of publications in each class (data from ref. 1).

Full size image

What drives the undercitation? I suggest that incorrect citation may be one factor. Price2 has analysed the citations of three highly cited publications, with astonishing results: the publications were cited incorrectly in more than 200 different ways. An analysis of the data for the G7 group of countries from Table 1 in ref. 3 reveals that all non-English-speaking countries receive less than their share of citations for their share of publications, whereas the English-speaking countries all receive an equal or greater share of citations for their share of publications. This difference is statistically significant (Fisher's exact P = 0.029). Given the high incidence of incorrect citations overall2, there may also be a general tendency towards making more mistakes when handling citations with unfamiliar names from another language.

When this tendency is coupled with the fact that almost half of all scientific publications come from English-speaking countries3, the hazard of incorrect citation may not be random, but instead biased towards non-English-speaking countries. Because incorrect citations do not appear in citation indexes2, undercitation may be a result of incorrect citation rather than an indicator of poor-quality research in non-English-speaking countries.

Nevertheless, the reason behind the positive relationship between impact factor and undercitation (Fig.1) remains a mystery that urgently requires further attention.