Access Keys:
08/12/2016
A report by Digital Science published today, raises questions important to the production of better indicators of research activity.
The Research Councils sponsored study, funded by the MRC, steered by an advisory group of evaluation experts and representatives from HEFCE, to compare the consistency of indicators often assumed to explain ‘interdisciplinarity’, and to identify a preferred methodology.
Indicators used included measures derived from analysis of the text of grant applications, the text of research papers, publications cited in these papers, and the departmental affiliations of the authors of these papers.
Previous analyses have used these properties as indicators of the extent to which research includes ideas or expertise from different disciplines, but such analyses have tended to focus on a single type of research data or indicator in isolation.
In this new study Digital Science, in collaboration with Science-Metrix, tested a batch of potential indicators with data from a common set of disciplines and countries. The results reveal that choice of data, methodology and indicators can produce seriously inconsistent results.
Dr Ian Viney, MRC’s Director of Evaluation commented “We expected that some indicators might prove better for suggesting whether work was more or less interdisciplinary, but it was a surprise that some of the indicators gave entirely conflicting results.”
The report concludes that common assumptions made about the connection between research metadata and research activity may sometimes be flawed. Unless we better understand what it is that we are measuring, these indicators may not be appropriate at all.
Dr Viney added “There is interest in finding better quantitative indicators to support research assessment. However, we want to use metrics responsibly, which means carefully testing assumptions about what it is you are measuring.”
This study highlights to users of research metrics the importance of clarifying the link between any proxy indicator and the assumed policy target, even if more research is required to explore this link. It is suggested that to develop indexes for complex research qualities, such as disciplinarity, a set of indicators used in combination may be required, but that currently there is no single satisfactory indicator.
The contribution of this study to our understanding of interdisciplinarity will be raised in a joint RCUK/HEFCE/British Academy conference on “Interdisciplinary research, policy and practice” to be held on the 8th of December.
- ends -
Contact:
RCUK Communications Team
Tel: 01793 444593 or email: communications@rcuk.ac.uk
Download a commentary on the report here.