Scientific research is not entirely about the objective, technical production and use of quantitative data. In an editorial last year1, we discussed the growing importance of citizen science, in which non-experts contribute to the data-gathering process, thus allowing it to be massively scaled up. However, a complementary trend in science is the generation of results based on the opinions of experts, opinions that are based on but are one step removed from raw objective data. Approaches based on expert opinion can be used to achieve synthesis or to scale up where obtaining direct observational data would be intractable. In this way, both expert-assessment and citizen-science approaches provide useful alternatives to traditional data collection.

Expert opinion has long played a role in science and its application. Taxonomists provide other researchers with identifications based on their recognized expertise; and science policy is often based on the informed opinions of select groups of advisors. The IUCN Red List data rely on expert opinion to assess the extent of threat a species is under within its range (see ref. 2, for a recent application of such data). More recently, however, attempts have been made to make the use of expert opinion more quantitative and scalable. There is now substantial methodological literature for conducting expert elicitations3, which illustrates that such an approach should not be considered a quick or soft option, and that anyone conducting such research should ensure they have the correct expertise in their authorship team to do so.

Some expert elicitation studies make use of surveys, conducted to reach as wide a range of experts as possible and minimize reliance on any individual expert. An example of such an approach is the study by Savary et al.4, which presents a global estimate of the burden on crops from pests and pathogens. This work required specific local knowledge from experts that would have been near impossible for an individual or small group to obtain; the authors leveraged the network of the International Society for Plant Pathology to contact the experts.

Another expert-driven approach is horizon scanning, used to identify upcoming trends in a scientific topic rather than answer a focused question. It has been applied in ecology and conservation biology both on a whole-of-field level (for example, the annual scan of global conservation issues5) and for more specific topics such as the potential impacts of pollinator decline (covered by an Article in this issue6) or robotics7. Horizon scanning typically uses a modified Delphi process8, which is an anonymized, iterative process designed to identify the priority topics of interest.

Bias is a concern when considering expert assessments. Considerable effort has gone into techniques for developing consensus among a group, and for tackling cognitive biases (for example, ref. 9). However, the diversity of participants or lack thereof also presents a potential source of bias. Many authors appreciate this and have gone to considerable effort to maximize diversity across several axes. For example, the authors of two expert elicitations on wind power in 201610 and 202111 recognized the geographical bias in their earlier study and made more proactive efforts to recruit experts from Asia in their second study. Likewise, Dicks et al.6 ensured a minimum representation from each global region and reported the gender balance of their experts.

However, compared with the methods for assessing the expert opinions themselves, the measures for ensuring and monitoring expert diversity are in their infancy. Fundamentally, they still depend on the desire and ability of the lead authors to reach out, and are limited by the extent of the authors’ own extended networks. These efforts tend to be ad hoc and not systematic, and it is still difficult to reach people and topics that the authors themselves do not already have on their radars. A more rigorous way of ensuring the inclusion of relevant experts across geographies, types of expertise, language, access to different information sources and more, would be enormously welcome. One possibility might be for institutional review boards or ethics committees — and note that a 2018 study12 found only 5% of expert elicitations they reviewed included any ethics information — to directly comment on the composition of expert panels in their assessments.

Of course, this problem is not unique to the use of expert elicitation methods. It applies when researchers recruit and collaborate on other forms of research, and when editors commission authors and reviewers. We all need to strive for more meaningful and representative diversity, using all the tools we can. Individual effort will always play a part, but we need to develop more systematic tools that support better embedding of diversity as a fundamental element of research processes, one worthy of the attention, time and critique already given to other elements of study design.