Introduction

Annual sales of pesticides in the European Union (EU) amounted to almost 360,000 tonnes, with a 46% share of fungicides as the most sold group1. Even with the deployment of resistant cultivars and integrated control strategies, fungicides still contribute heavily to plant disease control in conventional farming2. Even organic systems, although promoted for their environmental benefits, also depend on fungicides. In these systems, the amounts applied are sometimes high to compensate for lower efficacy3. Recently, new fungal plant diseases have emerged worldwide associated with the globalisation of trade and environmental change4, thus further increasing farmers’ dependency on fungicides. Nevertheless, their use in agriculture has been associated with growing environmental5 and public health6 concerns. In addition to their negative environmental impacts (e.g., on biodiversity7), some fungicides have been associated with increased risk to human health, particularly among farmers but also among people living in the vicinity of agricultural areas 8,9,10.

To promote more sustainable agricultural systems, EU Directive 2009/128/EC established several key principles to reduce pesticide use, fostering the adoption of prevention measures, non-chemical control methods, and chemical compounds with lower environmental impacts. Importantly, according to this Directive, any control intervention should in principle be based on field monitoring and trigger thresholds in order to reduce doses and treatment frequencies, thus limiting the risk of the development of pathogen resistance. The willingness to reduce the use of pesticides and especially fungicides was again highlighted in the ‘from farm to fork’ strategy of the European Green Deal, which targets a reduction in the use of chemical pesticides by half by 203011. Nevertheless, despite this regulatory framework, the amount of fungicides sold annually in the EU increased by up to 11% in the period 2011–20181.

Fungicide use in agriculture can be slightly reduced with improved spray application methods12, but to achieve a more substantial reduction a drastic decrease in the number of applications is essential. Decision support systems (DSSs) have been put forward as tools to substantially lower pesticide application frequency. In contrast to calendar-based fungicide programs, DSSs allow farmers to schedule fungicide applications based on an observed or a predicted risk of disease and thus spray only when necessary13. Numerous field experimental studies have been carried out to assess the performances of DSSs for different crops, diseases, and regions. However, to date, the whole set of data obtained in these experiments has not been compiled and subjected to rigorous statistical analysis to quantify the benefits resulting from the use of DSSs.

Our meta-analysis of 80 independent experiments conducted worldwide indicated that, for a given fixed number of fungicide sprays, DSSs were as effective as calendar-based programs (or more so) in reducing disease incidence for a wide range of crop species, fungal pathogens, types of fungicide and regions. When the number of sprays was halved, the resulting increase in disease incidence was greatly mitigated with a strategy based on DSSs rather than on calendars.

Our analysis thus shows that DSSs are essential tools for reducing fungicide use while limiting plant health risk and may help achieve the goals of the European Green Deal11. In addition to reducing the economic cost and environmental impact of disease control, the reduction in the number of sprays resulting from the use of DSSs also decreases the risk of developing resistance, thereby prolonging the effective life of the fungicides14. Ensuring the credibility of DSSs is essential to overcome producers’ aversion to perceived risks and thus make their application more widespread15,16.

Results

Data description

Our dataset includes the results of 80 independent experiments reported in 22 articles published from 1982–2015 that compare the efficacy of calendar-based and DSS-based strategies in reducing fungal disease incidence with fungicide treatments. Each experiment included data collected for at least one calendar-based strategy, one DSS-based strategy, and an untreated control plot. The dataset includes a total of 328 disease incidence data items collected for 80 untreated controls, 99 calendar-based strategies, and 149 DSS-based strategies (Supplementary Table S1).

The dataset included a total of 16 core DSSs, with additional modifications and different action thresholds. The DSSs comprised both empirical (correlative) and mechanistic (process-based) modelling approaches. Empirical models were mostly built using disease and weather observations in the field whereas mechanistic models were developed from controlled experiments to quantify the effects of environmental factors on the different components of the disease cycle. The environmental variables used to predict disease risk in the core DSSs were air temperature (14 out of 16), leaf wetness duration (11 out of 16), rainfall (7 out of 16), relative humidity (5 out of 16), solar radiation (2 out of 16) and wind speed (1 out of 16). The DSSs included a median of 2 environmental variables, with a minimum of 1 and a maximum of 5. Two DSSs included crop phenology and one also inoculum levels (Supplementary Table S2).

The experiments were located in South America (Brazil, n = 7 experiments), North America (Canada and USA, n = 51), and Europe (Italy, Lithuania, and Spain, n = 19) (Supplementary Table S3 and Supplementary Fig. S1a). The locations of the experiments covered 6 Köpen-Geiger climate classes17,18: Aw (equatorial with dry winter), Cfa (warm temperate, fully humid, hot summer), Csa (warm temperate, dry and hot summer), Csb (warm temperate, dry and warm summer), Dfa (snow, fully humid, hot summer) and Dfb (snow, fully humid, warm summer) (Supplementary Data 1). The experiments targeted different crops, pathogens, and fungicides. A total of n = 44 experiments were conducted on non-woody crops, including wheat, asparagus, lettuce, strawberry, and tomato. The remaining n = 36 experiments were conducted on woody crops, including apple, pear, grape, and mandarin (Supplementary Table S3 and Supplementary Fig. S1b).

The dataset covered both fungal and fungal-like pathogens from the classes of the Dothideomycetes (Alternaria alternata, A. solani, Stemphylium vesicarium, Cercospora asparagii, Schizothyrium jamaicense, Peltaster fructicola), Sordariomycetes (Fusarium graminearum, Colletotrichum coccodes), Leotiomycetes (Botrytis cinerea), Agaricomycetes (Rhizoctonia solani) and Oomycetes (Bremia lactucae, Plasmopara viticola). The Dothideomycetes was the most represented fungal class with n = 38 experiments and the Agaricomycetes was the least represented, with n = 3. These pathogens were causing the following diseases: Alternaria brown spot of mandarin, an early blight of tomato, brown spot of pear, Cercospora blight of asparagus, flyspeck and a sooty blotch of apple, Fusarium head blight of wheat, anthracnose of tomato, a grey mould of strawberry, Rhizoctonia fruit rot of tomato, downy mildew of lettuce and downy mildew of grape (Supplementary Table S3 and Supplementary Fig. S1c).

The most common spray programs in the experiments were those with non-systemic fungicides (n = 42) while only n = 7 experiments included programs with systemic fungicides alone. In the remaining n = 31 experiments, programs combined non-systemic and systemic fungicides. The non-systemic group included phthalimides (captan), dithio-carbamates (mancozeb, maneb, thiram), inorganics (copper hydroxide), chloronitriles (chlorothalonil), and phenylpyrroles (fludioxonil). The systemic group included demethylation inhibitors (tebuconazole), methyl benzimidazole carbamates (thiophanate-methyl), the quinone outside inhibitors (kresoxim-methyl, pyraclostrobin, trifloxystrobin), phosphonates (fosetyl-Al), phenylamides (metalaxyl), dicarboximides (procymidone), succinate-dehydrogenase inhibitors (boscalid), and anilino-pyrimidines (cyprodinil) (Supplementary Table S3). These fungicides covered all FRAC19 categories with regard to resistance risk: low risk (FRAC codes M01, M03, M04, M05, and P07), low to medium risk (FRAC code 12), medium risk (FRAC codes 3 and 9), medium to high risk (FRAC codes 2 and 7) and high risk (FRAC codes 1, 4 and 11) (Supplementary Data 1).

The spread of the disease incidence data was very large, ranging from 1.5% to 100% in the untreated controls (Fig. 1a), revealing the presence of experiments with very low, moderate, and very high disease pressure in the dataset. Disease incidences were lower in calendar-based and DSS-based strategies than in their untreated counterparts in the majority of the experiments (Fig. 1a, b). This observation remained valid when considering different subgroups of experiments corresponding to different locations, crops, pathogens, or types of fungicide (Supplementary Figs. S2S5). The number of sprays ranged from 0–25 across the whole set of experiments and were generally higher in calendar-based strategies than in DSS-based strategies (Fig. 1c). In contrast, the differences in disease incidence between DSS-based and calendar-based strategies were generally small and often very close to 0 (Fig. 1d).

Fig. 1: Observed disease incidences and numbers of sprays in the 80 experiments.
figure 1

(a) Individual (per experiment) and overall distribution of disease incidence data for untreated (Unt), calendar-based (Cal) and DSS-based strategies (DSS); (b) individual and overall distribution of the measured differences in incidence between Cal vs. Unt (Cal-Unt), DSS vs. Unt (DSS-Unt) and DSS vs. Cal (DSS-Cal) strategies; (c) individual and overall distribution of the numbers of sprays with Cal and DSS strategies and observed reduction in the number of sprays between DSS vs. Cal strategies; (d) observed difference in disease incidence vs. reduction in the number of sprays for DSS vs. Cal strategies. Each point represents a pair of plots (DSS, Cal) within the same experiment. Box-and-whisker plot elements for (a)–(d): central line represents the median value; box limitis represents the first (Q1) and the third (Q3) quantiles; upper whisker represents min(max(x),Q3 + 1.5 IQR; lower whisker represents max(min(x), Q1 − 1.5 IQR; IQR = Q3 − Q1; outliers are represented by dots.

Meta-analysis of disease incidences under different control strategies

According to our baseline statistical model MI0 (see Methods, Eq. (3)), the estimated average levels of reduction in disease incidence compared to untreated controls did not differ by more than a few percent between the calendar-based and DSS-based strategies. Specifically, the calendar-based strategy led to a reduction in disease incidence of−33.3%, 95% CrI = [−42.8%, −24.1%] compared to the untreated controls, while the DSS-based strategy led to a reduction of − 30.9%, 95% CrI = [− 39.9%, −22.2%], on average over all experiments (Fig. 2a). The average disease incidence was higher by +2.3%, 95% CrI = [+0.8%, +4.1%] with DSSs vs. calendars (Fig. 2a), revealing a slightly higher disease incidence with the DSSs. These results were confirmed when disease incidence ratios were considered instead of differences in disease incidence (Supplementary Fig. S7a). Because of a large between-experiment variability (see Methods), the predictive intervals were much larger than the credible intervals (Fig. 2b). Indeed, predictive intervals reflect the plausible range of values that could be obtained in a new, not yet conducted, experiment and their sizes depend on the extent of the between-experiment variability. Here, the predictive intervals of the levels of reduction in disease incidence (differences calendar vs. untreated and DSS vs. untreated) were 95% PI = [−80.4%, +6.0%] and 95% PI = [−75.0%, +6.3%] for the calendar-based and DSS-based strategies, respectively. Although large, these intervals do not differ much between the two strategies considered, thus confirming the similarity of the levels of disease incidence that could be expected with these two types of strategy across a large range of conditions. Alternative meta-analytic models that evaluated the effect of the moderator variables (location, crop, pathogen, or fungicide categories, see Methods and Supplementary Tables S3 and S5) were fitted and compared for their adequacy to the data (Supplementary Fig. S6). None of them led to a substantial change in the results (Supplementary Table S6).

Fig. 2: Probability distributions of differences in disease incidence between calendar-based strategies (Cal), DSS-based strategies (DSS) and untreated controls (Unt).
figure 2

Differences were computed for Cal vs. Unt (Cal-Unt), DSS vs. Unt (DSS-Unt) and DSS vs. Cal (DSS-Cal) strategies. While (a) describes the distributions of the expected differences across the experiments included in the dataset, (b) describes the plausible difference values predicted for a new, not yet conducted, experiment. Both types of the probability distribution are summarised by their medians, 95% probability intervals and probability of the differences being positive. CrI and PI mean credibility and predictive intervals, respectively. P(>0) indicates the probabilities of the differences being positive. Results show that the levels of reduction in disease incidence achieved with DSS and Cal compared to Unt are very similar (median reductions of −33 and −31% with Cal and DSS, respectively), although the reduction is slightly higher with Cal.

Disease incidence as a function of the number of sprays

The joint effect of the number of sprays and the fungicide strategy was evaluated considering the baseline disease incidence–number of sprays model (MIS0) (Eq. (5)), after checking that the adequacy and performance of the models were satisfactory (Supplementary Fig. S8) and that the moderator variables (i.e., location, crop, pathogen and fungicide) were not influential (Supplementary Tables S3 and S8). The medians of the numbers of sprays observed with the DSS-based and calendar-based strategies across the 80 experiments were equal to 4 and 7, respectively. The difference between these two median values corresponds to a 43% reduction in the number of sprays with the DSS-based strategy compared to the calendar-based strategy. For these median numbers of sprays, the estimated average difference in disease incidence between DSS and calendar was equal to +2%, 95% CrI = [−1%, +6%] and was not substantially different from zero (Fig. 3a). The same conclusion was obtained when using the first (Q1) and third (Q3) quartiles of the observed distributions of the number of sprays for DSS-based (3 and 6 sprays) and calendar-based (4 and 10 sprays) strategies (Fig. 3a) instead of the medians. These findings were further confirmed using the disease incidence ratio data instead of the differences (Supplementary Fig. S9). All these results concur to indicate that the use of DSSs allows for a meaningful reduction in the number of sprays without any noticeable impact on the disease incidence. The predictive intervals showed wider amplitude than credible intervals (Fig. 3b and Supplementary Fig. S9b), although they resulted in similar levels of disease incidence with a reduction of up to 43% in the number of sprays in the calendar-based strategies with respect to those based on DSS.

Fig. 3: Probability distributions of differences in disease incidence between DSS-based (DSS) and calendar-based (Cal) strategies considering different observed numbers of sprays.
figure 3

While (a) describes the distributions of the expected differences across the experiments included in the dataset, (b) describes the plausible difference values predicted for a new, not yet conducted, experiment. In each case, three distributions are reported, corresponding to the first (Q1), second (median) (Q2) and third (Q3) quartiles of the observed numbers of sprays for DSS (3, 4 and 6) and Cal (4, 7, and 10) strategies in the 80 experiments. Each probability distribution is summarised by its median, 95% probability interval and probability of the differences being positive. CrI and PI mean credibility and predictive intervals, respectively. P(>0) indicates the probabilities of the differences being positive. All credibility probability intervals include zero, thus revealing that the use of lower numbers of sprays resulting from the DSS strategies does not lead to a substantial increase in disease incidence.

We further compared the DSS-based and calendar-based strategies considering a hypothetical scenario in which both strategies were implemented using exactly the same number of sprays. This number of sprays was set equal to different values ranging from 0 to 25 successively and the difference in disease incidence obtained with the two strategies (DSS vs. calendar) was estimated with our statistical model for each value considered. The estimated difference in incidence DSS vs. calendar was negative from 0 to 18 sprays, reaching its maximum (−5.4%, 95% CrI = [−10.2%, −1.4%]) with 3 sprays (Fig. 4a), and then became not greatly different from zero when the number of sprays exceeded 18. These results reveal that there is an intrinsic advantage in using DSSs and that, for a given number of sprays (in the range 0–21), a lower level of disease incidence could be achieved by adopting a DSS-based strategy instead of a calendar-based strategy.

Fig. 4: Effects of the number of sprays on disease incidences for calendar-based strategy (Cal), DSS-based (DSS) strategy and untreated controls (Unt).
figure 4

(a) Differences in estimated disease incidence for Cal vs. Unt (Cal-Unt), DSS vs. Unt (DSS-Unt) and DSS vs. Cal (DSS-Cal) strategies for numbers of sprays ranging from 0–25; (b) Differences in estimated disease incidence resulting from the application of DSS and Cal strategies with a number of sprays reduced by 50% compared to total number recommended by the original Cal strategy. In (a), the same number of sprays is considered for both DSS and Cal, and the green curve reveals that DSS leads to lower levels of disease incidence than Cal when the same number of sprays are applied with both strategies. In (b), the curves DSS50%-Cal and Cal50%-Cal indicate the extent of the increase in disease incidence resulting from a reduction of 50% of the total number of sprays recommended by the original calendar strategy. The disease incidence is increased with both DSS50% and Cal50% compared to Cal, but the level of increase is lower with the former than with the latter. The different curves describe the posterior medians (solid line) and the 95% credible intervals (dotted lines) (95% CrI).

Finally, we assessed the consequences of halving the number of fungicide sprays recommended by the calendar-based strategies. To do so, we considered two scenarios. In the first one, we assumed that the 50% reduction was achieved by adopting a DSS-based strategy (scenario DSS50%) instead of a calendar-based strategy. In the second one, we assumed that the 50% reduction was achieved by using a calendar-based strategy (CAL50%). In the first scenario, the resulting estimated average increase in disease incidence never exceeded +5.1% (95% CrI = [+1.2%, +9.7%]) (Fig. 4b). This worst-case was obtained when the 50% reduction resulted in a decrease of 3 sprays (corresponding to an initial number of sprays equal to 6). When the decrease in spray number was lower or higher than 3, the resulting increase in disease incidence was lower than 5%. In the second scenario (CAL50%), the increase in disease incidence resulting from a 50% reduction in the number of sprays was always higher than in the first scenario (DSS50%) and could reach more than +10% of disease incidence (Fig. 4b). Taken together, these results indicate that it is preferable to reduce the number of sprays by adopting the DSS-based strategy.

Discussion

Based on a comprehensive dataset covering major producing areas and a large number of crops and pathogens, our results show that DSSs can play an important role in reducing the use of fungicides while maintaining a high level of crop protection. Across the 80 selected experiments, the median number of fungicide sprays applied with DSSs was 43% lower compared to standard calendar-based strategies. Moreover, for a given number of sprays, DSS-based fungicide programs were equally and even more effective (by up to 5.5%) for disease control. More specifically, a higher efficacy was observed for DSSs when the number of spray applications was relatively low (<4). When the number of sprays increased, both DSS-based and calendar-based strategies showed similar disease control efficacy (Fig. 4a). The good performances of DSSs can be explained by the fact that, with DSS, spray timing is based on the observed or predicted risk of disease, allowing farmers to apply fungicides when they are most effective during the growing season. In contrast, with calendar-based strategies, spray timing is preset without considering the changes in disease dynamics, leading to suboptimal treatments. In the case of low numbers of sprays, DSSs can target the optimal application periods better to halt disease progress, while some risk periods may be missed with calendar-based programs. When the number of sprays increases, calendar-based programs may then also cover all the risk periods, but at a cost of applying unnecessary high numbers of sprays.

Our study shows that the goal of a 50% reduction in the number of fungicides (as envisioned by the ‘from farm to fork’ strategy of the European Green Deal11) is not a utopia. When the number of sprays was reduced by 50% with DSSs compared to recommended calendar strategies, the increase in disease incidence never exceeded +5%. Considering the major economic savings and reduced environmental impacts obtained by halving the number of sprays, this increase in disease incidence can be considered bearable. With the calendar-based programs, a 50% reduction in the number of sprays resulted in a higher increase in disease incidence of about +10%. Although twice as high as the level obtained with DSS, this effect also remains relatively small, i.e., much smaller than the level of 50% that would have been reached in the case of a one-to-one relationship between spray number and disease incidence. This result can be explained by the fact that the recommended calendar programs are probably overdosing fungicides. Importantly, we found our results robust to several important factors, such as the geographic location of the experiments, the taxonomic groups (necrotrophic or biotrophic lifestyles, dispersed by wind and water, causing monocyclic, polycyclic and polyetic diseases with rather different epidemiological traits), the type of host (woody and non-woody hosts) and the types of fungicide (Supplementary Tables S3 and S8).

Despite substantial progress in spray application methods with dose adjustment and reduced spray volumes to maximise coverage while minimising drift12, the amount of fungicides sold annually in the EU increased by up to 11% over the last decade1. Substantial reductions in the amount of fungicides applied can therefore only be achieved by constraining the number of applications, which also results in greater economic savings than by just reducing spray volumes. Our study clearly shows that this is a realistic strategy. The use of fungicides in agriculture may be further reduced if DSSs are integrated with other disease management methods, now facilitated by the advent of precision farming. For instance, agronomic practices such as canopy management, crop sequences and timing can reduce disease pressure and thus the need for fungicide sprays. In the short term, the dependency of fungicides can be greatly reduced with the use of resistant cultivars. Nevertheless, those cultivars are typically bred for monoculture systems where pathogens are under intense selection pressure. Host resistance can be also exploited to design diversified farming systems with cultivar mixtures and intercrops, resulting in more durable plant resistance and fungicide efficacy20.

The reduction in the use of fungicides is not only an issue in conventional agriculture, but also in organic production. Control of airborne diseases by means of fungicides can be even more demanding in organic farming because the plant protection products allowed are often less effective3. Of the 164,345 tonnes of fungicides sold in 2018 in the EU, 86,231 tonnes (52%) were inorganic fungicides1 (i.e., copper and sulphur), which are allowed in organic production. The ‘from farm to fork’ strategy of the European Green Deal11 aims to boost the amount of agricultural land under organic farming in the EU from 7.5–25% by 2030. Under this scenario, DSSs will become even more important for optimising treatments against fungal diseases as applications of fungicides need to be timed as precisely as possible on organic farms due to the relatively low efficacy of the products available3.

Disease prediction models and action thresholds are essential components of DSSs for plant disease control. Empirical and mechanistic models are often evaluated (i.e., validated) by comparing predictions against independent disease observations15. Model evaluation can be performed for instance by monitoring disease progress or exposing trap plants. Proper evaluation is an essential step to assess the reliability and generalisation of the disease model under different situations. However, the evaluation of DSSs should not be restricted to the evaluation of disease models and should also consider the assessment of the action thresholds determining appropriate deployment of disease management measures21. The evaluation of DSSs also integrates factors related to data availability and communication, fungicide and spray performance, among others, which in certain situations may be more important than the disease prediction model itself. DSSs may sometimes be released onto the market without proper evaluation, resulting in inefficient disease management actions undermining their trustworthiness and rate of adoption15. Proper evaluation of DSSs is typically performed by comparing disease intensity (i.e., incidence or severity) of a DSS-driven fungicide spray schedule with that of a routine calendar program and an untreated control21, as was the case in all the experiments included in our meta-analysis.

The fungicides and modes of action included in our meta-analysis represented all FRAC19 categories in relation to the risk of developing resistance. The database included mainly programs involving non-systemic fungicides (n = 42) and combinations of non-systemic and systemic products (n = 31). Only n = 7 experiments included programs with systemic fungicides alone. These fungicides generally act against single biochemical targets and are thus considered of medium or high risk for the development of resistance19. The application of fungicides with more than one mode of action, either in mixtures or in alternation, is recommended for resistance management. Typically, single-site systemic fungicides were combined with multi-site non-systemic ones with a low risk of resistance2. However, due to their associated non-target effects, multi-site fungicides often present higher ecotoxicity than single-site compounds and so they are being progressively withdrawn.

With the increasing use of single-site fungicides, fungal resistance development and the subsequent loss of efficacy of fungicides are of increasing concern2. In addition to reduced application dose and the combination of different modes of action, a limitation in the number of applications is also essential for the effective management of fungicide resistance. In fact, for some groups of fungicides, the maximum number of applications per season is already strictly limited in order to slow down the build-up of resistance. The reduction in the number of sprays minimises the exposure time and the overall selection for fungicide resistance14. Therefore, in addition to lessening the environmental and economic costs of disease control, a reduction in the number of sprays based on DSSs could substantially diminish the risk of developing resistance, thereby prolonging the effective life of the fungicides, with a limited increase in disease risk.

A number of the selected publications with experiments comparing an untreated control with the calendar and DSSs fungicide programs were not included in our meta-analysis because the sample size was not reported (n = 45) (Supplementary Fig. S10), thus precluding the weighting of the individual studies. Sample sizes in the experiments included in our meta-analysis were relatively large, ranging from 80–1500 with a median of 500 (Supplementary Information). Our database covered practically the full range of disease incidence values (from 1.5–100%), representing different disease pressure scenarios. Studies reporting disease severity or derived metrics, such as the area under the disease progress curve (AUDPC), were not included (n = 28) (Supplementary Fig. S10). While disease incidence is the number of diseased plant organs in relation to the total number evaluated, disease severity is the proportion or the actual host area affected. Disease severity is typically evaluated using standard area diagrams, disease scales or ordinal rating scales22. However, severity measurements are not standardised. Depending on the study, the intervals, ranges and ratings used differ greatly among diseases and even for the same disease. Moreover, they seldom represent equal gradations of the underlying continuous disease severity scale. This leads to serious statistical constraints when, as in our meta-analysis, experiments using different disease severity assessment methods should be combined. In contrast, analysing disease incidence is relatively straightforward and robust statistical methods can be applied based on generalised linear mixed models. Moreover, in the case of fruit and vegetables, disease incidence is more informative than severity in relation to loss in marketable yield, since the presence of just a few lesions makes this produce out of grade23.

After more than a decade with Directive 2009/128/EC in force, official reports noted the limited implementation of the measures to achieve a more sustainable use of pesticides in the EU24. Action thresholds and reduced application frequencies were among the measures with a relatively low level of adoption25. Growers’ aversion to risks has been pointed out as one of the main reasons for the limited implementation of DSSs26. Adoption of DSSs is even more restricted in intensive high-input crops, because the consequences of a disease outbreak by missing a spray (i.e., false-negative case) sometimes exceed the economic benefits of reducing the number of fungicides applications13,27. Indeed, the use of fungicides is highly dependent on the crop, with dosages ranging from 0.2 kg ha−1 in arable crops to 11.3 kg ha−1 in fruit and vegetables in the EU28. Consequently, different degrees of adoption of DSSs depending on the crop may have a considerable effect on the overall reduction in fungicide use. Nevertheless, our study indicated that crop or pathogen types (Supplementary Table S8 and Supplementary Fig. S8) did not have a substantial impact on the risk of disease when halving the number of sprays, suggesting that perceived rather than actual risks are likely driving growers’ cautiousness regarding the adoption of DSSs. Those perceived uncertainties in the timing of fungicide applications can be narrowed down by increasing the amount of timely and spatially explicit data on the weather and the onset of disease outbreaks available to growers16. Furthermore, it is essential to deploy DSSs with a high degree of credibility, after proper calibration and evaluation. The involvement of growers in the process of DSS development through a participatory approach is also an interesting and promising avenue15. Finally, the new European Green Deal11, through its major role in reshaping the EU common agricultural policy, will certainly set the scene for a much wider adoption of DSSs and more sustainable disease management.

Methods

Data collection

A comprehensive dataset was constructed to synthesise and quantify the effects of DSS-based fungicide programs as compared to calendar-based programs in terms of the number of spray applications and disease incidence. Relevant studies were identified in August 2019 by searching i) Web of Science (WoS) and ii) Fungicide and Nematicide Tests (F&N Tests and Plant Disease Management Reports) by the American Phytopathological Society. Multiple search strings based on different combinations of keywords were used for the literature search (Supplementary Table S4). Additionally, our search also included relevant studies found in the reference lists of the selected studies. A PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses29) diagram (Supplementary Fig. S10) was included to present the details of the selection procedure.

To be included in the database, the published experiments had to satisfy the following criteria: (i) at least one untreated control (Unt), one calendar-based strategy (Cal) and a DSS-based strategy (DSS) were tested; (ii) disease incidence (i.e., the proportion of diseased organs); (iii) sample size (i.e., the total number of organs evaluated) and (iv) number of fungicide spray applications were provided for the untreated control and for each fungicide strategy. The final database included data on 80 independent experiments reported in 22 published articles that evaluated 80 untreated controls, 99 calendar-based strategies and 149 DSS-based strategies. The number of independent experiments in each publication varied from 1 to 11 and the number of plots within an experiment ranged from 3 to 7 (Supplementary Table S1). A unique identifier was assigned to each experiment and the following data were also extracted: location (country, state, locality), year, crop, pathogen, disease, fungicide, plant organ evaluated, experimental design, number of replicates and other relevant characteristics (Supplementary Data 1). However, as described below, only country, crop, pathogen and fungicide type were used as moderator variables in the meta-analysis.

Data preparation

Four moderator variables at the experiment level were defined based on the data provided by (i) location, (ii) crop, (iii) pathogen and, (iv) fungicide (Supplementary Table S3). Based on experiment locations, three continents were distinguished: Europe, North America and South America. Two crop types were considered, namely woody and non-woody crops. Finally, five classes of pathogens were distinguished: Dothideomycetes, Sordariomycetes, Leotiomycetes, Agaricomycetes and Oomycetes. Fungicide programs were allocated to three categories2: non-systemic, systemic and non-systemic/systemic. The last category included programs combining non-systemic and systemic fungicides.

Meta-analyses

We performed two independent meta-analyses: (i) a meta-analysis of disease incidences under the different control strategies (MI), which was conducted to quantify, synthesise and compare disease incidences for DSS-based, calendar-based strategies and untreated controls and (ii) a meta-analysis relating disease incidences to the number of sprays under the different control strategies (MIS), which was conducted to quantify, synthesise and compare the effect of the number of sprays on disease incidences between DSS-based and calendar-based strategies.

Based on Lázaro et al.30, where frequentist and Bayesian models were compared, here both meta-analyses (MI and MIS) were specified considering beta-binomial mixed-effect regression modelling framework31. Beta-binomial models are more adequate than the binomial generalised linear models used by Lázaro et al.30 to deal with overdispersed observations32,33,34, which result in discrepancies between the theoretical and empirical variances. Both MI and MIS were based on the following beta-binomial distribution:

$${Y}_{ij} \sim \,{{\mbox{Bin}}}\,({n}_{ij},\ {\theta }_{ij}),$$
(1)
$${\theta }_{ij} \sim \,{{\mbox{Beta}}}\,({\mu }_{ij}\ \phi ,(1-{\mu }_{ij})\ \phi),$$
(2)

where Yij denotes the number of diseased organs in the plot j in the experiment i out of a total of nij organs evaluated. Yij is assumed to follow a binomial distribution with a probability of disease incidence θij, which in turn is assumed to follow a beta distribution with mean and precision parameters μij and ϕ, respectively. The expression of μij was different in MI and MIS, as shown below.

Meta-analysis of disease incidence under different control strategies (MI)

The MI baseline model (MI0) was defined as,

$$\,{{\mbox{logit}}}\, ({\mu }_{ij})\, =\, {{\mbox{log}}}\,\left(\frac{{\mu }_{ij}}{1-{\mu }_{ij}}\right)\, =\, ({\beta }_{0}+{b}_{{{{{{{{\rm{0(i)}}}}}}}}})+({\beta }_{{{{{{{{\rm{cal}}}}}}}}}+{b}_{{{{{{{{\rm{cal(i)}}}}}}}}})\, {I}_{{{{{{{{\rm{cal(ij)}}}}}}}}}\\ +({\beta }_{{{{{{{{\rm{dss}}}}}}}}}+{b}_{{{{{{{{\rm{dss(i)}}}}}}}}})\, {I}_{{{{{{{{\rm{dss(ij)}}}}}}}}},$$
(3)

in which the logit of the mean of the disease incidence (μij) is expressed as a linear function of two dummy variables, Ical(ij) and Idss(ij), equal to one if plot j in experiment i corresponds to a calendar-based or DSS-based strategy, respectively, and to zero otherwise. The fixed parameters, βcal and βdss, define the population fungicide treatment effects of the calendar and DSS strategies in relation to the untreated control, β0. The model takes into account the between-experiment variability of the effects of the treatment strategies and control through three random parameters bcal(i), bdss(i), and b0(i) assumed to be multivariate normally distributed with zero means and a 3 x 3 variance-covariance matrix Σ defined as,

$$\left[\begin{array}{l}{b}_{{{{0\,{{{{\rm{(i)}}}}}}}}}\\ {b}_{{{{{{{{\rm{cal}}}}}}\,{{{{{\rm{(i)}}}}}}}}}\\ {b}_{{{{{{{{\rm{dss}}}}}}\,{{{{{\rm{(i)}}}}}}}}}\end{array}\right] \sim {{{\mbox{N}}}}_{3}\left\{\left[\begin{array}{l}0\\ 0\\ 0\end{array}\right],\left[\begin{array}{lll}{\sigma }_{0}^{2}&{\sigma }_{{{{{{{{\rm{0,cal}}}}}}}}}&{\sigma }_{{{{{{{{\rm{0,dss}}}}}}}}}\\ {\sigma }_{{{{{{{{\rm{0,cal}}}}}}}}}&{\sigma }_{{{{{{{{\rm{cal}}}}}}}}}^{2}&{\sigma }_{{{{{{{{\rm{cal,dss}}}}}}}}}\\ {\sigma }_{{{{{{{{\rm{0,dss}}}}}}}}}&{\sigma }_{{{{{{{{\rm{cal,dss}}}}}}}}}&{\sigma }_{{{{{{{{\rm{dss}}}}}}}}}^{2}\end{array}\right]\right\}.$$
(4)

The variances, \({\sigma }_{0}^{2}\), \({\sigma }_{{{{{{{{\rm{cal}}}}}}}}}^{2}\) and \({\sigma }_{{{{{{{{\rm{dss}}}}}}}}}^{2}\) capture the extent of variability between experiments, while σ0,cal, σ0,dss, σcal,dss define the covariances between plots clustered within the same experiment.

Disease incidence as a function of the number of sprays (MIS)

The MIS baseline model (MIS0) was defined as,

$${{\mbox{logit}}}\, ({\mu }_{ij}) = \,{{\mbox{log}}}\,\left(\frac{{\mu }_{ij}}{1-{\mu }_{ij}}\right)= \;({\beta }_{0}+{b}_{{{{{{{{\rm{0(i)}}}}}}}}})+({\beta }_{{{{{{{{\rm{nspcal}}}}}}}}}+{b}_{{{{{{{{\rm{cal(i)}}}}}}}}})\ {{{{{{{\rm{nspcal(ij)}}}}}}}}\ {I}_{{{{{{{{\rm{cal(ij)}}}}}}}}} \\ +({\beta }_{{{{{{{{\rm{nspdss}}}}}}}}}+{b}_{{{{{{{{\rm{dss(i)}}}}}}}}})\ {{{{{{{\rm{nspdss(ij)}}}}}}}}\ {I}_{{{{{{{{\rm{dss(ij)}}}}}}}}},$$
(5)

in which the logit of the mean of the disease incidence (μij) is expressed as a linear function of nspcal (ij) Ical(ij) and nspdss (ij) Idss(ij), which represent the number of sprays when plot j of the experiment i is under calendar-based (Ical(ij) = 1) or DSS-based (Idss(ij) = 1) strategies, respectively. The fixed effects, βnspcal and βnspdss, capture the population effect of the number of sprays for calendar and DSS strategies relative to the untreated control, β0. As for MI, the model takes into account the between-experiment variability of the effects of the treatment strategies and control through three random parameters bcal(i), bdss(i) and b0(i). These random parameters are assumed to be multivariate normally distributed with zero means and a 3 × 3 variance-covariance matrix Σ defined as,

$$\left[\begin{array}{l}{b}_{{{{{{{{\rm{0}}}}}}\,({{{{{\rm{i}}}}}})}}}\\ {b}_{{{{{{{{\rm{cal}}}}}} \,({{{{{\rm{i}}}}}})}}}\\ {b}_{{{{{{{{\rm{dss}}}}}} \,({{{{{\rm{i}}}}}})}}}\end{array}\right] \sim {{{\mbox{N}}}}_{3}\left\{\left[\begin{array}{l}0\\ 0\\ 0\end{array}\right],\left[\begin{array}{lll}{\sigma }_{0}^{2}&0&0\\ 0&{\sigma }_{{{{{{{{\rm{cal}}}}}}}}}^{2}&0\\ 0&0&{\sigma }_{{{{{{{{\rm{dss}}}}}}}}}^{2}\end{array}\right]\right\}.$$
(6)

Here, the variance-covariance matrix of the random effects is diagonal to ensure convergence of the fitting algorithm. Note that despite using the similar notation in MI0 and MIS0 (i.e., as a general rule we used β for fixed effects and b for random effects), those parameters do not capture the same effect in the two models.

Several variants of the models MI0 and MIS0 were fitted in order to investigate the possible effects of the four moderator variables (i.e., location, crop, pathogen and fungicide). Specifically, for each moderator variable, two independent models (without and with the interaction between treatment fixed effects) were built from Eqs. (3) and (5), respectively. All the meta-analytic models including moderator variables are described in Supplementary Table S5 for those built from MI0 and in Supplementary Table S7 for those built from MIS0.

The inference was conducted using Bayesian procedures. Model parameters were approximated by means of Hamiltonian Monte Carlo (HMC) simulation methods using the programming language Stan35 via its R interface, RStan (version 2.19.2)36 and the R package brms (version 2.13.0)37,38. All inference processes were set under weakly independent prior scenarios considering the default prior specifications of the brms package. For each model, the HMC algorithm was run with four Markov chains each including 10,000 iterations after a warm-up of 5000 iterations to ensure a proper convergence35.

Model evaluation and model selection

Model evaluation, that is, the checking of the adequacy and performance of all meta-analytic models (Supplementary Tables S5 and S7) was carried out considering posterior predictive checks. We evaluated whether the meta-analytic models proposed were able to generate a database that resembles the observed incidence. Thus, we compared observed disease incidence vs. the predicted disease incidence by plotting them around the regression line 1:1. (Supplementary Fig. S6 for MI models and Supplementary Fig. S8 for MIS models).

Model selection between the baseline models (MI0 and MIS0) and the corresponding models including moderator variables was accomplished by means of K-fold cross-validation procedures (K-fold-CV)39. K-fold-CV evaluates the predictive performance of the model. The principle is to re-fit each model to a subset of the original database. This approach was implemented with K = 5. With this setting, all the meta-analytic models were re-fitted 5 times randomly leaving out 20% of the experimental observations of the original data at each iteration. The K-fold-CV difference was computed by the combination of the kfold() and the loo_compare() functions of the R packages brms and loo40, respectively. The K-fold-CV difference estimates were used to compare the predictive accuracy between the baseline model and each of the moderator meta-analytic models and were quantified through a point estimate (Δkfold) as well as its corresponding standard error (SEΔ kfold). Models with the highest K-fold-CV value (i.e., the highest accuracy) were preferred but standard errors associated to the difference were also considered39. Thus, differences not higher than 4 (i.e., \(\left|{{\Delta }}\ \,{{\mbox{kfold}}}\,\right|\leqslant 4\)) were considered negligible regardless of the value of the corresponding standard error. But differences in the K-fold-CV higher than 4 were considered not meaningful if these differences were smaller than four times the standard error associated to this difference (i.e., \(\left|{{\Delta }}\ \,{{\mbox{kfold}}}\right|\setminus {{{\mbox{SE}}}}_{{{\Delta }}{{\mbox{kfold}}}}\leqslant 4\)). If no meaningful differences were observed, the simplest model (i.e., MI0 or MIS0) was selected based on the parsimony principle (Supplementary Table S6 for the incidence models and Supplementary Table S8 for the incidence-number of sprays model). The results of the model evaluation and selection are available in Supplementary Fig. S6 for MI models and Supplementary Fig. S8 for MIS models. None of the moderator-based models showed improved behaviour compared to the baseline models (i.e., MI0 and MIS0) according to the model evaluation assessment (Supplementary Figs. S6 and S8) and the model selection criterion (Supplementary Tables S6 and S8). The two baseline models were thus further considered for the meta-analyses.

Effect sizes

For both meta-analyses (i.e., MI and MIS) we considered the disease incidence difference (DID) and disease incidence ratio (DIR) as the effect sizes to assess the efficacy of calendar-based and DSS-based strategies compared to untreated controls. The fitted models were used to compute effect sizes with two different approaches: i) by including only the fixed-parameter estimates in order to compute the expected disease incidence across the experiments included in the dataset (i.e., expected effect size values) 30,41, and ii) by including both the fixed-parameter estimates and the random effect estimates in order to predict the disease incidence for a new experiment, not yet conducted (i.e., the predicted effect size values)42,43.

Meta-analysis of disease incidence under different control strategies

Considering the baseline MI model (MI0), the expected disease incidences (in probability scale) associated with the three fungicide strategies were computed according to Eq. (3), as follows:

$${\mu }_{Unt}\ =\ \frac{\,{{\mbox{exp}}}\,\{{\beta }_{0}\}}{1+\,{{\mbox{exp}}}\,\{{\beta }_{0}\}},\qquad \\ {\mu }_{Cal}\ =\ \frac{\,{{\mbox{exp}}}\,\{{\beta }_{0}+{\beta }_{{{{{{{{\rm{cal}}}}}}}}}\}}{1+\,{{\mbox{exp}}}\,\{{\beta }_{0}+{\beta }_{{{{{{{{\rm{cal}}}}}}}}}\}},\qquad \\ {\mu }_{DSS}\ =\ \frac{\,{{\mbox{exp}}}\,\{{\beta }_{0}+{\beta }_{{{{{{{{\rm{dss}}}}}}}}}\}}{1+\,{{\mbox{exp}}}\,\{{\beta }_{0}+{\beta }_{{{{{{{{\rm{dss}}}}}}}}}\}}.$$
(7)

DID were then derived as,

$${{{\mbox{DID}}}}_{Cal-Unt}\ =\ {\mu }_{Cal}\ -\ {\mu }_{Unt},\qquad \\ {{{\mbox{DID}}}}_{DSS-Unt}\ =\ {\mu }_{DSS}\ -\ {\mu }_{Unt},\qquad \\ {{{\mbox{DID}}}}_{DSS-Cal}\ =\ {\mu }_{DSS}\ -\ {\mu }_{Cal},$$
(8)

and DIR as,

$${{{\mbox{DIR}}}}_{Cal/Unt}\ =\ \frac{{\mu }_{Cal}}{{\mu }_{Unt}},\qquad \\ {{{\mbox{DIR}}}}_{DSS/Unt}\ =\ \frac{{\mu }_{DSS}}{{\mu }_{Unt}},\qquad \\ {{{\mbox{DIR}}}}_{DSS/Cal}\ =\ \frac{{\mu }_{DSS}}{{\mu }_{Cal}}.$$
(9)

The values of DID and DIR defined above were computed from the posterior distributions of the fixed parameters and summarised by their medians, 95% credible intervals (95% CrI) and probabilities of being positive (P(DID>0)) or of being higher than 1 (P(DIR>1)).

The predicted values (i.e., for a new experiment) were computed according to Eq. (5), as follows:

$${\mu }_{Unt}^{P}\, =\, \frac{{\exp}\,\{{\beta }_{0}+{b}_{{{0,{{{{{\rm{new}}}}}}}}}\}}{1+\,{\exp }\,\{{\beta }_{0}+{b}_{{{{0,{{{{\rm{new}}}}}}}}}\}},\\ {\mu }_{Cal}^{P}\, =\, \frac{{\exp }\,\{{\beta }_{0}+{b}_{{{0,{{{{{\rm{new}}}}}}}}}+{\beta }_{{{{{{{{\rm{cal}}}}}}}}}+{b}_{{{{{{{{\rm{cal}}}}}},{{{{{\rm{new}}}}}}}}}\}}{1+{\exp }\,\{\beta_{0} + b_{{0,{{{{{\rm{new}}}}}}}}+{\beta }_{{{{{{{{\rm{cal}}}}}}}}}+{b}_{{{{{{{{\rm{cal,new}}}}}}}}}\}},\\ {\mu }_{DSS}^{P}\, =\, \frac{{\exp }\,\{\beta_{0} + {b}_{{0,{{{{{\rm{new}}}}}}}} + {\beta }_{{{{{{\rm{dss}}}}}}} + {b}_{{{{{{{{\rm{dss}}}}}},{{{{{\rm{new}}}}}}}}}\}}{1+\,{\exp }\,\{{\beta }_{0}+{b}_{{{0},{{{{{\rm{new}}}}}}}}+{\beta }_{{{{{{{{\rm{dss}}}}}}}}}+{b}_{{{{{{{{\rm{dss}}}}}},{{{{{\rm{new}}}}}}}}}\}}.$$
(10)

where b0,new, bcal,new, bdss,new are the effects for a new experiment computed from Eq. (3)30,44. Likewise, DID and DIR were computed according to Eqs. (8) and (9) but based on disease incidence estimates from Eq. (10). They were summarised by their medians, 95% prediction intervals (95% PI) and probabilities of being positive (P(DID>0)) or of being higher than 1 (P(DIR>1)).

Disease incidence as a function of the number of sprays

Based on the baseline MIS model (MIS0), the expected disease incidences in probability scale) were estimated as a function of the number of sprays for calendar-based (nspcal) and DSS-based (nspdss) strategies as follows:

$$\, \mu_{Unt}\ =\ \frac{{{{{{\mathrm{exp}}}}}} \,\{\beta_{0}\}}{1+\,{{{{{\mathrm{exp}}}}}}\,\{\beta_{0}\}},\qquad \\ \, \mu_{Cal}\ =\ \frac{{{{{{\mathrm{exp}}}}}} \,\{\beta_{0} + \beta_{{{{{{\rm{nspcal}}}}}}}\ {{{{{\rm{nspcal}}}}}}\}}{1+\,{{{{{\mathrm{exp}}}}}}\,\{\beta_{0}+{\beta }_{{{{{{\rm{nspcal}}}}}}}\ {{{{{\rm{nspcal}}}}}}\}},\qquad \\ \, {\mu }_{DSS}\ =\ \frac{{{{{{\mathrm{exp}}}}}} \,\{\beta_{0} + \beta_{{{{{{\rm{dss}}}}}}}\ {{{{{\rm{nspdss}}}}}}\}}{1+\,{{{{{\mathrm{exp}}}}}}\,\{\beta_{0} + \beta_{{{{{{\rm{dss}}}}}}}\ {{{{{\rm{nspdss}}}}}}\}}.$$
(11)

Similarly, the predicted distributions were estimated as:

$${\mu }_{Unt}^{P}\, =\, \frac{{{{{{\mathrm{exp}}}}}}\,\{\beta_{0} + {b}_{{0, {{{{{\rm{new}}}}}}}}\}}{1+\,{{{{{\mathrm{exp}}}}}}\,\{\beta_{0} + {b}_{{0, {{{{{\rm{new}}}}}}}}\}},\\ {\mu }_{Cal}^{P}\, =\, \frac{{{{{{\mathrm{exp}}}}}}\,\{(\beta_{0} + b_{{0, {{{{{\rm{new}}}}}}}})+(\beta_{{{{{{\rm{nspcal}}}}}}} + b_{{{{{{{\rm{cal}}}}}},{{{{{\rm{new}}}}}}}})\ {{{{{\rm{nspcal}}}}}}\}}{1+\,{{{{{\mathrm{exp}}}}}}\,\{(\beta_{0}) + (b_{{0, {{{{{\rm{new}}}}}}}}) + (\beta_{{{{{{\rm{nspcal}}}}}}} + b_{{{{{{{\rm{cal}}}}}},{{{{{\rm{new}}}}}}}})\, {{{{{\rm{nspcal}}}}}}\}},\\ {\mu }_{DSS}^{P}\, =\, \frac{{{{{{\mathrm{exp}}}}}}\,\{(\beta_{0} + {b}_{{0, {{{{{\rm{new}}}}}}}}) + (\beta_{{{{{{\rm{dss}}}}}}} + b_{{{{{{{\rm{dss}}}}}},{{{{{\rm{new}}}}}}}})\, {{{{{\rm{nspdss}}}}}}\}}{1+\,{{{{{\mathrm{exp}}}}}}\,\{(\beta_{0} + {b}_{{0,{{{{{\rm{new}}}}}}}}) + ({\beta }_{{{{{{\rm{dss}}}}}}} + {b}_{{{{{{{\rm{dss}}}}}}, {{{{{\rm{new}}}}}}}})\, {{{{{\rm{nspdss}}}}}}\}}.$$
(12)

The model MIS0 was used to assess the impacts of several scenarios of reduction in pesticide use (i.e., reduction of nspcal and nspdss):

  • Disease incidences were obtained for the first (Q1), second (median) (Q2) and third (Q3) quartiles of the observed numbers of sprays for the DSS (3, 4 and 6) and calendar (4, 7, and 10) strategies in the observed experiments. The difference between the two medians corresponds to a 43% reduction in the number of sprays with the DSS strategy compared to the calendar strategy.

  • Disease incidences obtained across the whole range of values of the number of sprays observed in the database (from 0 to 25 sprays) with the DSS and calendar strategies.

  • Disease incidences with two 50% reduction scenarios: a DSS-based (DSS50%) and a calendar-based (CAL50%) strategy compared to a 100% calendar strategy. Specifically, we considered nspdss50% = nspdcal50% = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) compared to a 100% calendar-based nspdcal = (2, 4, 6, 8, 10, 12, 14, 16, 18, 20). The difference in the number of sprays for both hypothetical scenarios corresponds to a 50% reduction compared to the 100% calendar-based strategy.

The estimated and predicted values of DID and DIR were computed adapting Eqs. (8) and (9) to the disease incidence estimates from Eqs. (11) and (12) for each of the comparisons addressed in each of the scenarios. Posterior distributions of DID and DIR were summarised by their medians, 95% CrI and 95% PI. Posterior distributions of DID and DIR were used to compute P(DID>0) and P(DIR>1), respectively.

Sensitivity analysis

A sensitivity analysis45 was conducted to evaluate the behaviour of the Bayesian inferential methods in relation to the setting of prior specification30,41. The MI models (Supplementary Table S5) as well as the MIS models (Supplementary Table S7) were also fitted using a frequentist method by maximum likelihood through Laplace approximation using the glmmTMB() function of the package glmmTMB46 implemented in the R software47. Model parameter estimates between the Bayesian and the frequentist approaches were compared (Supplementary Fig. S11 for the MI models and Supplementary Fig. S12 for the MIS models). The sensitivity analysis revealed that the results were robust.