How accurately does output reflect the nature and design of transdisciplinary research programmes?

TY - JOUR

T1 - How accurately does output reflect the nature and design of transdisciplinary research programmes?

AU - Koier, Elizabeth

AU - Horlings, E.

PY - 2014

Y1 - 2014

N2 - Many of today’s societal problems are wicked problems that require a new, transdisciplinary approach in which knowledge of scientists and stakeholders from different disciplines is integrated. The evaluation of transdisciplinary science requires a multi-method approach. Bibliometric analysis is consistently among the methods in multi-method evaluations. We analyse the accuracy of bibliometric evidence for the evaluation of transdisciplinary research by examining two large climate adaptation research programmes in the Netherlands. The assessment of accuracy involves a comparison of different approaches to defining and measuring involvement, output, and quality. We draw three conclusions with regard to accuracy. First, scientific output covers a fairly high amount of the scientific activities of the programmes, though information on funding agencies is not yet sufficiently accurate to reconstruct a programme’s output through the Web of Science (WoS). Second, scientific output does not accurately reflect the nature and design of the programmes. The WoS appears to underestimate locally oriented and practically oriented research, non-academic actors rarely co-author scientific publications, and the contributions of non-academic organizations to projects could not be recognized from author affiliations. Third, our exploration of two alternative reproducible metrics (non-scientific output and download statistics) shows that it is too early to introduce such metrics into evaluation practices. The research agenda for transdisciplinary output metrics should focus on the development of a common definition of transdisciplinary research output and a typology of non-scientific outputs, as well as a discussion and assessment of the relative value of such outputs for the integration of knowledge.

AB - Many of today’s societal problems are wicked problems that require a new, transdisciplinary approach in which knowledge of scientists and stakeholders from different disciplines is integrated. The evaluation of transdisciplinary science requires a multi-method approach. Bibliometric analysis is consistently among the methods in multi-method evaluations. We analyse the accuracy of bibliometric evidence for the evaluation of transdisciplinary research by examining two large climate adaptation research programmes in the Netherlands. The assessment of accuracy involves a comparison of different approaches to defining and measuring involvement, output, and quality. We draw three conclusions with regard to accuracy. First, scientific output covers a fairly high amount of the scientific activities of the programmes, though information on funding agencies is not yet sufficiently accurate to reconstruct a programme’s output through the Web of Science (WoS). Second, scientific output does not accurately reflect the nature and design of the programmes. The WoS appears to underestimate locally oriented and practically oriented research, non-academic actors rarely co-author scientific publications, and the contributions of non-academic organizations to projects could not be recognized from author affiliations. Third, our exploration of two alternative reproducible metrics (non-scientific output and download statistics) shows that it is too early to introduce such metrics into evaluation practices. The research agenda for transdisciplinary output metrics should focus on the development of a common definition of transdisciplinary research output and a typology of non-scientific outputs, as well as a discussion and assessment of the relative value of such outputs for the integration of knowledge.

KW - transdisciplinary research

KW - evaluation

KW - bibliometrics

KW - large research programmes

KW - Web of science

U2 - 10.1093/reseval/rvu027

DO - 10.1093/reseval/rvu027

M3 - Article

SN - 0958-2029

VL - 24

SP - 37

EP - 50

JO - Research Evaluation

JF - Research Evaluation

IS - 1

ER -