Sir

The choice of performance indicators sends a powerful message to those being evaluated, and when those measures are linked to the distribution of research funds, academics are quick to respond. Our analysis of Australian university publications shows clearly how the sector has reacted to funding formulae that reward quantity rather than quality.

A large part of the government funds that support the research activities of Australian universities is allocated on the basis of formulae that comprise three elements: research income, postgraduate students and publications1. Data on the third element have been collected annually since 1993. When this element was incorporated into the funding formulae in 1995, universities and researchers were quick to calculate the 'value' of a publication.

Between 1995 and 2000, this figure varied from A$761 (US$415) to A$1,089, influenced by the publication types included and the total funds allocated. After a review of higher-education research in 1999, the amount to be distributed via formulae increased significantly, to more than half of the funding specifically targeted to research and research training through the Education, Science and Training portfolio. As a result, a published paper is now 'worth' more than A$3,000 to a university. The value to individuals or departments can be appreciably higher.2 Some universities distribute these funds internally using the same formula, but giving more weight to publications — up to three times the sector value.

We have quantified the apparent effect of this policy. We have distributed all journals from the Institute for Scientific Information's (ISI) Science Citation Index (SCI) into quartiles, using journal impacts calculated on the basis of five-year citation means, and have tracked the presence of Australian universities in these four journal sets over time (Figure 1).

Figure 1
figure 1

Australian universities' share of publications in the SCI, by journal impact quartile: five-year windows, 1981–1985 to 1996–2000.

Full size image

The response of the academic community was predictable and clear. Until the 1989–93 period, there had been virtually no movement in the institutions' presence in the SCI journal sets. Subsequently, university output jumped noticeably, even though funds remain extremely tight and academic staff numbers stable. The most striking feature of that increase has been its lack of uniformity. The sector's share of publications in the top two quartiles rose by around 20%, but in the third quartile its share increased by over 50%, and it doubled in the bottom quartile.

With no differentiation between the quality or impact of the publications, there is little incentive to strive for placement in a prestigious journal. Whether a publication is a groundbreaking piece in Nature, or a pedestrian piece in a low-impact journal, the rewards are identical. And with the recent trebling in the 'value' of a publication unit, the pressure to focus on this will not diminish.

Concerns that this component of the funding formula was not measuring the characteristic that it was designed for — quality — were raised soon after its introduction. However, not all universities were keen to see it removed or replaced. For smaller institutions, this particular element was more rewarding, and easier to improve, than the others.

These concerns are now re-surfacing in the context of the latest review of the Australian higher education system3. A number of submissions to recent ministerial discussion papers have suggested the removal or modification of the publications component. The difficulty is that suggested alternatives are as problematic as the one they seek to replace. It is to be hoped that time will be taken to analyse the likely effects of any alternative measures before they are introduced.