Abstract
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.
| Original language | English |
|---|---|
| Pages (from-to) | 84-88 |
| Number of pages | 5 |
| Journal | Nature |
| Volume | 582 |
| Issue number | 7810 |
| DOIs | |
| Publication status | Published - 2020 |
Keywords
- Brain/diagnostic imaging
- Data Analysis
- Data Science/methods
- Datasets as Topic/statistics & numerical data
- Female
- Functional Neuroimaging
- Humans
- Logistic Models
- Magnetic Resonance Imaging
- Male
- Meta-Analysis as Topic
- Models, Neurological
- Reproducibility of Results
- Research Personnel/organization & administration
- Software
Fingerprint
Dive into the research topics of 'Variability in the analysis of a single neuroimaging dataset by many teams'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver
}
Variability in the analysis of a single neuroimaging dataset by many teams. / Botvinik-Nezer, Rotem; Holzmeister, Felix; Camerer, Colin F; Dreber, Anna; Huber, Juergen; Johannesson, Magnus; Kirchler, Michael; Iwanir, Roni; Mumford, Jeanette A; Adcock, R Alison; Avesani, Paolo; Baczkowski, Blazej M; Bajracharya, Aahana; Bakst, Leah; Ball, Sheryl; Barilari, Marco; Bault, Nadège; Beaton, Derek; Beitner, Julia; Benoit, Roland G; Berkers, Ruud M W J; Bhanji, Jamil P; Biswal, Bharat B; Bobadilla-Suarez, Sebastian; Bortolini, Tiago; Bottenhorn, Katherine L; Bowring, Alexander; Braem, Senne; Brooks, Hayley R; Brudner, Emily G; Calderon, Cristian B; Camilleri, Julia A; Castrellon, Jaime J; Cecchetti, Luca; Cieslik, Edna C; Cole, Zachary J; Collignon, Olivier; Cox, Robert W; Cunningham, William A; Czoschke, Stefan; Dadi, Kamalaker; Davis, Charles P; Luca, Alberto De; Delgado, Mauricio R; Demetriou, Lysia; Dennison, Jeffrey B; Di, Xin; Dickie, Erin W; Dobryakova, Ekaterina; Donnat, Claire L; Dukart, Juergen; Duncan, Niall W; Durnez, Joke; Eed, Amr; Eickhoff, Simon B; Erhart, Andrew; Fontanesi, Laura; Fricke, G Matthew; Fu, Shiguang; Galván, Adriana; Gau, Remi; Genon, Sarah; Glatard, Tristan; Glerean, Enrico; Goeman, Jelle J; Golowin, Sergej A E; González-García, Carlos; Gorgolewski, Krzysztof J; Grady, Cheryl L; Green, Mikella A; Guassi Moreira, João F; Guest, Olivia; Hakimi, Shabnam; Hamilton, J Paul; Hancock, Roeland; Handjaras, Giacomo; Harry, Bronson B; Hawco, Colin; Herholz, Peer; Herman, Gabrielle; Heunis, Stephan; Hoffstaedter, Felix; Hogeveen, Jeremy; Holmes, Susan; Hu, Chuan-Peng; Huettel, Scott A; Hughes, Matthew E; Iacovella, Vittorio; Iordan, Alexandru D; Isager, Peder M; Isik, Ayse I; Jahn, Andrew; Johnson, Matthew R; Johnstone, Tom; Joseph, Michael J E; Juliano, Anthony C; Kable, Joseph W; Kassinopoulos, Michalis; Koba, Cemal; Kong, Xiang-Zhen; Koscik, Timothy R; Kucukboyaci, Nuri Erkut; Kuhl, Brice A; Kupek, Sebastian; Laird, Angela R; Lamm, Claus; Langner, Robert; Lauharatanahirun, Nina; Lee, Hongmi; Lee, Sangil; Leemans, Alexander; Leo, Andrea; Lesage, Elise; Li, Flora; Li, Monica Y C; Lim, Phui Cheng; Lintz, Evan N; Liphardt, Schuyler W; Losecaat Vermeer, Annabel B; Love, Bradley C; Mack, Michael L; Malpica, Norberto; Marins, Theo; Maumet, Camille; McDonald, Kelsey; McGuire, Joseph T; Melero, Helena; Méndez Leal, Adriana S; Meyer, Benjamin; Meyer, Kristin N; Mihai, Glad; Mitsis, Georgios D; Moll, Jorge; Nielson, Dylan M; Nilsonne, Gustav; Notter, Michael P; Olivetti, Emanuele; Onicas, Adrian I; Papale, Paolo; Peelle, Jonathan E; Pérez, Alexandre; Pischedda, Doris; Poline, Jean-Baptiste; Prystauka, Yanina; Ray, Shruti; Reuter-Lorenz, Patricia A; Reynolds, Richard C; Ricciardi, Emiliano; Rieck, Jenny R; Rodriguez-Thompson, Anais M; Romyn, Anthony; Salo, Taylor; Samanez-Larkin, Gregory R; Sanz-Morales, Emilio; Schlichting, Margaret L; Schultz, Douglas H; Shen, Qiang; Sheridan, Margaret A; Silvers, Jennifer A; Skagerlund, Kenny; Smith, Alec; Smith, David V; Sokol-Hessner, Peter; Steinkamp, Simon R; Tashjian, Sarah M; Thirion, Bertrand; Thorp, John N; Tinghög, Gustav; Tisdall, Loreen; Tompson, Steven H; Toro-Serey, Claudio; Torre Tresols, Juan Jesus; Tozzi, Leonardo; Truong, Vuong; Turella, Luca; van 't Veer, Anna E; Verguts, Tom; Vettel, Jean M; Vijayarajah, Sagana; Vo, Khoi; Wall, Matthew B; Weeda, Wouter D; Weis, Susanne; White, David J; Wisniewski, David; Xifra-Porxas, Alba; Yearling, Emily A; Yoon, Sangsuk; Yuan, Rui; Yuen, Kenneth S L; Zhang, Lei; Zhang, Xu; Zosky, Joshua E; Nichols, Thomas E; Poldrack, Russell A; Schonberg, Tom.
In: Nature, Vol. 582, No. 7810, 2020, p. 84-88.Research output: Contribution to journal/periodical › Article › Scientific › peer-review
TY - JOUR
T1 - Variability in the analysis of a single neuroimaging dataset by many teams
AU - Botvinik-Nezer, Rotem
AU - Holzmeister, Felix
AU - Camerer, Colin F
AU - Dreber, Anna
AU - Huber, Juergen
AU - Johannesson, Magnus
AU - Kirchler, Michael
AU - Iwanir, Roni
AU - Mumford, Jeanette A
AU - Adcock, R Alison
AU - Avesani, Paolo
AU - Baczkowski, Blazej M
AU - Bajracharya, Aahana
AU - Bakst, Leah
AU - Ball, Sheryl
AU - Barilari, Marco
AU - Bault, Nadège
AU - Beaton, Derek
AU - Beitner, Julia
AU - Benoit, Roland G
AU - Berkers, Ruud M W J
AU - Bhanji, Jamil P
AU - Biswal, Bharat B
AU - Bobadilla-Suarez, Sebastian
AU - Bortolini, Tiago
AU - Bottenhorn, Katherine L
AU - Bowring, Alexander
AU - Braem, Senne
AU - Brooks, Hayley R
AU - Brudner, Emily G
AU - Calderon, Cristian B
AU - Camilleri, Julia A
AU - Castrellon, Jaime J
AU - Cecchetti, Luca
AU - Cieslik, Edna C
AU - Cole, Zachary J
AU - Collignon, Olivier
AU - Cox, Robert W
AU - Cunningham, William A
AU - Czoschke, Stefan
AU - Dadi, Kamalaker
AU - Davis, Charles P
AU - Luca, Alberto De
AU - Delgado, Mauricio R
AU - Demetriou, Lysia
AU - Dennison, Jeffrey B
AU - Di, Xin
AU - Dickie, Erin W
AU - Dobryakova, Ekaterina
AU - Donnat, Claire L
AU - Dukart, Juergen
AU - Duncan, Niall W
AU - Durnez, Joke
AU - Eed, Amr
AU - Eickhoff, Simon B
AU - Erhart, Andrew
AU - Fontanesi, Laura
AU - Fricke, G Matthew
AU - Fu, Shiguang
AU - Galván, Adriana
AU - Gau, Remi
AU - Genon, Sarah
AU - Glatard, Tristan
AU - Glerean, Enrico
AU - Goeman, Jelle J
AU - Golowin, Sergej A E
AU - González-García, Carlos
AU - Gorgolewski, Krzysztof J
AU - Grady, Cheryl L
AU - Green, Mikella A
AU - Guassi Moreira, João F
AU - Guest, Olivia
AU - Hakimi, Shabnam
AU - Hamilton, J Paul
AU - Hancock, Roeland
AU - Handjaras, Giacomo
AU - Harry, Bronson B
AU - Hawco, Colin
AU - Herholz, Peer
AU - Herman, Gabrielle
AU - Heunis, Stephan
AU - Hoffstaedter, Felix
AU - Hogeveen, Jeremy
AU - Holmes, Susan
AU - Hu, Chuan-Peng
AU - Huettel, Scott A
AU - Hughes, Matthew E
AU - Iacovella, Vittorio
AU - Iordan, Alexandru D
AU - Isager, Peder M
AU - Isik, Ayse I
AU - Jahn, Andrew
AU - Johnson, Matthew R
AU - Johnstone, Tom
AU - Joseph, Michael J E
AU - Juliano, Anthony C
AU - Kable, Joseph W
AU - Kassinopoulos, Michalis
AU - Koba, Cemal
AU - Kong, Xiang-Zhen
AU - Koscik, Timothy R
AU - Kucukboyaci, Nuri Erkut
AU - Kuhl, Brice A
AU - Kupek, Sebastian
AU - Laird, Angela R
AU - Lamm, Claus
AU - Langner, Robert
AU - Lauharatanahirun, Nina
AU - Lee, Hongmi
AU - Lee, Sangil
AU - Leemans, Alexander
AU - Leo, Andrea
AU - Lesage, Elise
AU - Li, Flora
AU - Li, Monica Y C
AU - Lim, Phui Cheng
AU - Lintz, Evan N
AU - Liphardt, Schuyler W
AU - Losecaat Vermeer, Annabel B
AU - Love, Bradley C
AU - Mack, Michael L
AU - Malpica, Norberto
AU - Marins, Theo
AU - Maumet, Camille
AU - McDonald, Kelsey
AU - McGuire, Joseph T
AU - Melero, Helena
AU - Méndez Leal, Adriana S
AU - Meyer, Benjamin
AU - Meyer, Kristin N
AU - Mihai, Glad
AU - Mitsis, Georgios D
AU - Moll, Jorge
AU - Nielson, Dylan M
AU - Nilsonne, Gustav
AU - Notter, Michael P
AU - Olivetti, Emanuele
AU - Onicas, Adrian I
AU - Papale, Paolo
AU - Peelle, Jonathan E
AU - Pérez, Alexandre
AU - Pischedda, Doris
AU - Poline, Jean-Baptiste
AU - Prystauka, Yanina
AU - Ray, Shruti
AU - Reuter-Lorenz, Patricia A
AU - Reynolds, Richard C
AU - Ricciardi, Emiliano
AU - Rieck, Jenny R
AU - Rodriguez-Thompson, Anais M
AU - Romyn, Anthony
AU - Salo, Taylor
AU - Samanez-Larkin, Gregory R
AU - Sanz-Morales, Emilio
AU - Schlichting, Margaret L
AU - Schultz, Douglas H
AU - Shen, Qiang
AU - Sheridan, Margaret A
AU - Silvers, Jennifer A
AU - Skagerlund, Kenny
AU - Smith, Alec
AU - Smith, David V
AU - Sokol-Hessner, Peter
AU - Steinkamp, Simon R
AU - Tashjian, Sarah M
AU - Thirion, Bertrand
AU - Thorp, John N
AU - Tinghög, Gustav
AU - Tisdall, Loreen
AU - Tompson, Steven H
AU - Toro-Serey, Claudio
AU - Torre Tresols, Juan Jesus
AU - Tozzi, Leonardo
AU - Truong, Vuong
AU - Turella, Luca
AU - van 't Veer, Anna E
AU - Verguts, Tom
AU - Vettel, Jean M
AU - Vijayarajah, Sagana
AU - Vo, Khoi
AU - Wall, Matthew B
AU - Weeda, Wouter D
AU - Weis, Susanne
AU - White, David J
AU - Wisniewski, David
AU - Xifra-Porxas, Alba
AU - Yearling, Emily A
AU - Yoon, Sangsuk
AU - Yuan, Rui
AU - Yuen, Kenneth S L
AU - Zhang, Lei
AU - Zhang, Xu
AU - Zosky, Joshua E
AU - Nichols, Thomas E
AU - Poldrack, Russell A
AU - Schonberg, Tom
PY - 2020
Y1 - 2020
N2 - Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.
AB - Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.
KW - Brain/diagnostic imaging
KW - Data Analysis
KW - Data Science/methods
KW - Datasets as Topic/statistics & numerical data
KW - Female
KW - Functional Neuroimaging
KW - Humans
KW - Logistic Models
KW - Magnetic Resonance Imaging
KW - Male
KW - Meta-Analysis as Topic
KW - Models, Neurological
KW - Reproducibility of Results
KW - Research Personnel/organization & administration
KW - Software
U2 - 10.1038/s41586-020-2314-9
DO - 10.1038/s41586-020-2314-9
M3 - Article
C2 - 32483374
VL - 582
SP - 84
EP - 88
JO - Nature
JF - Nature
SN - 0028-0836
IS - 7810
ER -