How can we measure the 'interdisciplinarity' of research?
05 Dec 2016
A report by Digital Science published today, raises questions important to the production of better indicators of research activity.
The study, sponsored by the Research Councils, and funded by the MRC, set out to compare the consistency of indicators often assumed to explain ‘interdisciplinarity’. One aim was to recommend a methodology to measure research interdisciplinarity which could be used to track this characteristic over time.
Digital Science, in collaboration with Science-Metrix, tested a batch of potential indicators with data from a common set of disciplines and countries. The results reveal that choice of data, methodology and indicators can produce inconsistent results.
Indicators used in the study, which was steered by an advisory group of evaluation experts, included: measures derived from analysis of the text of grant applications; the text of research papers; publications cited in these papers; and the departmental affiliations of the authors of these papers.
Previous analyses have used these criteria as indicators of the extent to which research includes ideas or expertise from different disciplines, but such analyses have tended to focus on a single type of research data or indicator in isolation.
Dr Ian Viney, MRC’s Director of Evaluation commented: “We expected that some indicators might prove better for suggesting whether work was more or less interdisciplinary, but it was a surprise that some of the indicators gave such conflicting results.”
The report concludes that common assumptions made about the connection between research metadata and research activity may sometimes be flawed.
Dr Viney added “There is interest in finding better quantitative indicators to support research assessment. However, we want to use metrics responsibly, which means carefully testing assumptions about what it is you are measuring.”
This study highlights to users of research metrics the importance of clarifying the link between any proxy indicator and the assumed policy target, even if more research is required to explore this link. It is suggested that to develop indexes for complex research qualities, such as interdisciplinarity, a set of indicators used in combination may be required, but that currently there is no single satisfactory indicator.
The contribution of this study to our understanding of interdisciplinarity will be raised in a joint RCUK/HEFCE/British Academy conference on “Interdisciplinary research: policy and practice” to be held on the 8th of December.