Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Matters Arising
  • Published:

Big-team science does not guarantee generalizability

Nature Human Behaviour volume 8, pages 1053–1056 (2024)Cite this article

Subjects

arising from K. Ruggeri et al. Nature Human Behaviour https://doi.org/10.1038/s41562-022-01392-w (2022)

A new era of global ‘big-team science’ studies has transformed human behaviour research. These innovative studies rely on a large, distributed network of participants from different parts of the world and represent a substantial advancement over the average study in psychology that rarely goes beyond a single demographic population (for example, North American undergraduates)1. Here we examine one such big-team science project that claimed the ‘globalizability’ of temporal discounting, the phenomenon in which the subjective value of deferred rewards is smaller than that of immediate rewards2. We argue that, although this study represents a substantial advance over the typical psychology study in its sampling approach, claims of global generalizability are overstated given the samples collected. Although the project recruited 171 researchers from 109 institutions, and 13,629 research participants speaking 40 languages across 61 countries, relying solely on the typical big-team methodology created an illusion of generalizability, leading authors to overestimate the extent to which research findings can be applied globally. Across the low-and-middle-income countries (LMICs) and high-income countries (HICs) included in Ruggeri et al.2, we found that the samples were all similarly young, well educated, urban and digitally connected. This homogeneity belies the heterogeneity present within each country3,4. To avoid this illusion of generalizability, we argue that researchers should carefully consider three dimensions of diversity: sample, author and methodological diversity.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Results comparing Nigerian, Chinese and US samples and census data for each country.
Fig. 2: Results from comparing the authors’ institution affiliations.

Data availability

The data for reproducing the figures can be found at https://github.com/hcp4715/NHB_Globalization_Revisit (ref. 16).

Code availability

The code to reproduce the analyses reported in this commentary can be found at https://github.com/hcp4715/NHB_Globalization_Revisit (ref. 16).

References

  1. Forscher, P. S. et al. The benefits, barriers, and risks of big-team science. Perspect. Psychol. Sci. 18, 607–623 (2023).

    Article  PubMed  Google Scholar 

  2. Ruggeri, K. et al. The globalizability of temporal discounting. Nat. Hum. Behav. 6, 1386–1397 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  3. Ghai, S. It’s time to reimagine sample diversity and retire the WEIRD dichotomy. Nat. Hum. Behav. 5, 971–972 (2021).

    Article  PubMed  Google Scholar 

  4. Tiokhin, L., Hackman, J., Munira, S., Jesmin, K. & Hruschka, D. Generalizability is not optional: insights from a cross-cultural study of social discounting. R. Soc. Open Sci. 6, 181386 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  5. Yue, L., Zuo, X.-N. & Chuan-Peng, H. The ‘weird’ problem in a ‘non-WEIRD’ context: a meta-research on the representativeness of human subjects in Chinese psychological research. OSF https://doi.org/10.17605/OSF.IO/EHW54 (2021).

  6. Schimmelpfennig, R. et al. The moderating role of culture in the generalizability of psychological phenomena. Adv. Methods Pract. Psychol. Sci. 7, 1–6 (2024).

    Google Scholar 

  7. United Nations, Department of Economic and Social Affairs, Population Division. World Urbanization Prospects: The 2018 Revision (United Nations, 2019).

  8. Yarkoni, T. The generalizability crisis. Behav. Brain Sci. 45, e1 (2022).

    Article  Google Scholar 

  9. Hoekstra, R. & Vazire, S. Aspiring to greater intellectual humility in science. Nat. Hum. Behav. 5, 1602–1607 (2021).

    Article  PubMed  Google Scholar 

  10. Coles, N. A., DeBruine, L. M., Azevedo, F., Baumgartner, H. A. & Frank, M. C. ‘Big team’ science challenges us to reconsider authorship. Nat. Hum. Behav. 7, 665–667 (2023).

    Article  PubMed  Google Scholar 

  11. Ghai, S., de-Wit, L. & Mak, Y. How we investigated the diversity of our undergraduate curriculum. Nature https://doi.org/10.1038/d41586-023-00614-z (2023).

  12. Adetula, A., Forscher, P. S., Basnight-Brown, D., Azouaghe, S. & IJzerman, H. Psychology should generalize from—not just to—Africa. Nat. Rev. Psychol. 1, 370–371 (2022).

    Article  Google Scholar 

  13. Mughogho, W., Adhiambo, J. & Forscher, P. S. African researchers must be full participants in behavioural science research. Nat. Hum. Behav. 7, 297–299 (2023).

    Article  PubMed  Google Scholar 

  14. Carvajal-Velez, L. et al. Measurement of mental health among adolescents at the population level: a multicountry protocol for adaptation and validation of mental health measures. J. Adolesc. Health 72, S27–S33 (2023).

    Article  PubMed  Google Scholar 

  15. Clarke, B., Schiavone, S. & Vazire, S. What limitations are reported in short articles in social and personality psychology? J. Pers. Soc. Psychol. 125, 874–901 (2023).

    Article  PubMed  Google Scholar 

  16. Chuan-Peng, H. et al. NHB_Globalization_Revisit. GitHub https://github.com/hcp4715/NHB_Globalization_Revisit (2024).

Download references

Author information

Authors and Affiliations

  1. Oxford Internet Institute, University of Oxford, Oxford, UK

    Sakshi Ghai

  2. Busara Center for Behavioral Economics, Nairobi, Kenya

    Patrick S. Forscher

  3. School of Psychology, Nanjing Normal University, Nanjing, China

    Hu Chuan-Peng

Authors
  1. Sakshi Ghai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrick S. Forscher
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hu Chuan-Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

S.G., H.C.-P. and P.S.F. drafted the manuscript. S.G. and H.C.-P. analysed the data.

Corresponding author

Correspondence to Hu Chuan-Peng.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghai, S., Forscher, P.S. & Chuan-Peng, H. Big-team science does not guarantee generalizability. Nat Hum Behav 8, 1053–1056 (2024). https://doi.org/10.1038/s41562-024-01902-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-024-01902-y

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing