Report on the pilot exercise to develop bibliometric indicators for the Research Excellence Framework

September 2009 | ref: 2009/39

To:

Heads of HEFCE-funded higher education institutions, Heads of universities in Northern Ireland

Of interest to those responsible for:

Research, Planning
This report presents the work undertaken to develop the bibliometrics element of the Research Excellence Framework, the key part of which has been a pilot exercise to develop bibliometric indicators.

Contents

  • Executive summary
  • Introduction
  • Aims and scope of the pilot
  • Compilation of databases
  • Normalisation
  • Indicators and outcomes
  • Interpretation
  • Further analysis
  • Work in progress
  • List of abbreviations
  • Annex A Pilot data specification
  • Annex B Compilation of the Scopus databases
  • Annex C Field categories and normalisation
  • Annex D Other variants of staff-based models
  • Annex E Main outcomes graphs
  • Annex F Outcomes tables
  • Annex G Discussion of bibliometrics by the Expert Advisory Groups
  • Annex H Availability of citation data by subject
  • Annex I Development of indicators
  • Annex J Stability of indicators
  • Annex K Normalisation factors for Web of Science and Scopus
  • Annex L Conference proceedings in Scopus

Executive summary

Purpose

1.   This report presents the work undertaken to develop the bibliometrics element of the Research Excellence Framework (REF), the key part of which has been a pilot exercise to develop bibliometric indicators.

2.   The report forms part of the evidence base for our consultation on the REF (HEFCE 2009/38).

Content

3.   The report outlines the aims and process of the pilot and summarises the outcomes and feedback received from participating HEIs and our REF Expert Advisory Groups. We also present the results of further bibliometric analysis that arose out of the pilot exercise and discuss analysis planned for the future.

Coverage

4.   The pilot exercise covered 35 Units of Assessment (UOAs) from the 2008 Research Assessment Exercise (RAE 2008) where coverage of research outputs was 40 per cent or greater in the commercial bibliometric databases. Twenty-two higher education institutions (HEIs) were selected to participate in the exercise. They covered a range of HEI types and sizes.

5.   Publications between 2001 and 2006 inclusive were analysed; including citations up to 2007. We used the two main commercially available citation databases for the pilot exercise: Web of Science (WoS) and Scopus.

Main models

6.   We tested three models in the pilot exercise:

  1. Model 1, based on institutional address. Data were harvested from the citation databases from publications recorded between 2001 and 2006 according to the inclusion of an address associated with a participating HEI. These data were then mapped to UOAs.
  2. Model 2, based on staff, all papers. Data sets were constructed to include all publications produced by staff submitted by the HEI to RAE 2008 between 2001 and 2006.
  3. Model 3, based on staff, selected papers. Data sets were constructed to include the six most highly cited papers of staff submitted by HEIs to RAE 2008.

Outcomes

7.   The main outcomes are presented graphically and in tabular form in the annexes to this report. We took extensive advice on the interpretation of the outcomes from the REF Expert Advisory Groups and participating pilot institutions.

Key points

8.   Bibliometrics are not sufficiently robust at this stage to be used formulaically or to replace expert review in the REF. However there is considerable scope for citation information to be used to inform expert review.

9.   The robustness of the bibliometrics varies across the fields of research covered by the pilot, lower levels of coverage decreasing the representativeness of the citation information. In areas where publication in journals is the main method of scholarly communication, bibliometrics are more representative of the research undertaken.

Action required

10.   No action is required in response to this document.

Download:

Annex A: Pilot data specification

Download the Download Annex A: Pilot data specification as Adobe PDF 12K as PDF (18 KB) | Download the Download Annex A: Pilot data specification as MS Word 66K as MS Word (98 KB)

Annex B: Compilation of the Scopus databases

Download the Download Annex B: Compilation of the Scopus databases as Adobe PDF 33K as PDF (32 KB) | Download the Download Annex B: Compilation of the Scopus databases as MS Word 68K as MS Word (67 KB)

Annex C: Field Categories and normalisation

Download the Download Annex C: Field Categories and normalisation as Adobe PDF 489K as PDF (488 KB) | Download the Download Annex C: Field Categories and normalisation as MS Word 687K as MS Word (687 KB)

Annex D: Other variants of staff-based models

Download the Download Annex D: Other variants of staff-based models as Adobe PDF 48K as PDF (47 KB) | Download the Download Annex D: Other variants of staff-based models as MS Word 96K as MS Word (96 KB)

Annex E: Main outcomes graphs

Download the Download Annex E: Main outcomes graphs as Adobe PDF 1.2Mb as PDF (1,272 KB)

Annex F: Outcomes tables

Download the Download Annex F: Outcomes tables as Adobe PDF 848K as PDF (847 KB)

Annex G: Discussion of bibliometrics by the Expert Advisory Groups

Download the Download Annex G: Discussion of bibliometrics by the Expert Advisory Groups as Adobe PDF 21K as PDF (20 KB)  Download the Download Annex G: Discussion of bibliometrics by the Expert Advisory Groups as MS Word 59K as MS Word (58 KB)

Annex H: Availability of citation data by subject

Download the Download Annex H: Availability of citation data by subject as Adobe PDF 21K as PDF (54 KB)  Download the Download Annex H: Availability of citation data by subject as MS Word 59K as MS Word (138 KB)

Annex I: Development of Indicators

Download the Download Annex I: Development of Indicators as Adobe PDF 164K as PDF (163 KB)  Download the Download Annex I: Development of Indicators as MS Word 119K as MS Word (231 KB)

Annex J: Stability of indicators

Download the Download Annex J: Stability of indicators as Adobe PDF 102K as PDF (101 KB)  Download the Download Annex J: Stability of indicators as MS Word 119K as MS Word (119 KB)

Annex K: Normalisation factors

Download the 09_39k.xls as MS Excel Spreadsheet (252 KB)

Annex L: Conference proceedings in Scopus

Download the Download Annex L: Conference proceedings in Scopus as Adobe PDF 42K as PDF (42 KB)  Download the Download Annex L: Conference proceedings in Scopus as MS Word 173K as MS Word (158 KB) 

Enquiries should be directed to:Pam Macpherson Barrett, tel 0117 9317471, e-mail p.macpherson-barrett@hefce.ac.uk

Page last updated 18 January 2012

Share this: