You are in :
HEFCE

August 2006/32 (web only)
Issues paper

This report is for information


Selection of staff for inclusion in RAE2001

This study investigates how disability, age, sex and ethnicity are related to selection of staff for inclusion in the 2001 Research Assessment Exercise (RAE2001), and the possible reasons for the differences in selection rates found. It examines the question of whether the process of selecting staff was fair, or if some staff were disadvantaged.


To: Heads of HEFCE-funded higher education institutions
Of interest to those responsible for: Equality and diversity management, Human resources management, Institutional strategic planning, Research management
Reference: 2006/32
Publication date: Equality and diversity management, Human resources management, Institutional strategic planning, Research management
Enquiries about HEFCE research and RAE policy Paul Hubbard
tel 0117 931 7334
e-mail p.hubbard@hefce.ac.uk
Enquiries about HEFCE equality policies Yasir Mirza
tel 0117 931 7316
e-mail y.mirza@hefce.ac.uk
Technical enquiries relating to data, modelling, etc Mark Gittoes
tel 0117 931 7052
e-mail m.gittoes@hefce.ac.uk

Contents and executive summary (read on-line)



Contents

  • Summary
  • Introduction
  • Determining rates of selection
  • Selection rates for staff with and without recorded disabilities
  • Selection rates for men and women
  • Selection rates by ethnicity
  • Bibliometrics measures of research outputs of groups of selected staff
  • Are there biases in selection? Some provisional conclusions
  • The 2008 RAE and beyond
  • References
  •  
  • Annex A   Terminology and abbreviations
  • Annex B   HESA data definitions and quality checks
  • Annex C   Model of staff selection
  • Annex D   Bibliometrics data extraction and linking to RAE data
  • Annex E   Bibliometrics linking to HESA data
  • Annex F   Bibliometrics estimation of within-submission differences
  • Annex G   Bibliometrics full results

Executive summary

Purpose

1.   This study investigates how disability, age, sex and ethnicity are related to the selection of staff for inclusion in the 2001 Research Assessment Exercise (RAE2001), and the possible reasons for the differences in selection rates found.

Key points

Background and scope

2.   The RAE assesses the quality of research in higher education institutions (HEIs) in the UK through a process of peer review. This study is based on an analysis of data from the last completed exercise in 2001. Institutions were able to select the staff whose research outputs were to be included in this assessment from their 'eligible staff', that is those academic staff employed by the institution, who were not employed to carry out another individual's research programme.

3.   There have been concerns about the possible impact of the RAE on equal opportunity policies. Is the process of selecting staff fair, or are some staff disadvantaged? The scope of our analysis for this study is limited to answering this question. We have not, for example, attempted to ascertain whether RAE panels assessed the work of different groups of academics fairly, or whether the process of accepting or rejecting an article is fair, or whether the research process as a whole is biased or not.

Methodology

4.   To assess whether there were disadvantages for certain groups of academics in the process of being selected for inclusion in RAE2001, we first considered to what extent these groups of academics were associated with a 'department' that their HEI had decided to submit for assessment. We then took just those staff who were associated with these 'submitting departments' and, by using statistical models, tried to compare staff on a 'like for like' basis.

5.   This statistical modelling is not straightforward, for two reasons:

  1. To give a definitive answer as to whether the selection of staff was fair, we would need a measure of research output quality, and we do not have this. Our only alternative is to rely on proxies for such a measure.
  2. Ambiguities are introduced into the analysis by including grade and other measures of employment status in the modelling. This is because the proportions of staff in different grades are also an equal opportunities issue. To help with the interpretation of the results we therefore present two sets of analysis: 'full' models, in which all the available variables are included and 'restricted' models which exclude variables relating to employment status.

We also used bibliometric data to assess the relative research strength of the men and women, and of staff from ethnic minorities, whose work was submitted to RAE2001.

Results and conclusions

6.   The selection rate for staff with recorded disabilities was 58 per cent, slightly lower than for staff not so identified (59 per cent). When other attributes were taken into account, the analysis showed disability was not a significant factor in the propensity to be selected.

7.   Both simple and 'like for like' comparisons showed that the proportion of staff selected varied significantly by age. The general pattern was that staff aged over 30 were more likely to be selected. This result suggested either that institutions were unwilling to select staff who had not had long enough to build up a substantial research record, or that, in general, researchers in the early parts of their careers might not produce outputs of the highest possible research quality.

8.   There was a large difference in the selection rates for men and women, which were 64 per cent and 46 per cent respectively. These overall figures hid differences by age. When other factors are taken into account, including grade and other employment related variables, the like for like comparisons still showed that men had significantly higher selection rates than women over the middle age range between 30 and 47. However, bibliometric measures of research strength of the selected staff showed no great differences between men and women if the most cited staff are excluded. Though not conclusive, these results were consistent with an explanation of the lower selection rate of women being due to a lower proportion of women having a research record that leads them to be selected, rather than bias in the selection process.

9.   The simple unadjusted comparisons showed selection rates of around 58 per cent to 60 per cent for staff from different ethnic groups, apart from staff from Black ethnic groups, who had a lower rate of 37 per cent. This lower rate was partly the result of a higher proportion of these staff being employed in departments which did not make an RAE2001 submission. But even when non-submitting departments were excluded, the selection rate for staff from Black ethnic groups was much lower than for others. The results from the two models used in the analysis gave different results when comparing staff by ethnic group. The 'full' model showed no significant difference in the selection rates of ethnic groupings when compared on a 'like for like' basis, while the 'restricted' model, which excluded employment variables showed that selection rates for 'Black', 'Asian' and 'Other' ethnic groupings were all lower than expected. We could not be certain whether this result from the restricted model reflected an unmeasured difference in quality of research, or whether it was evidence of an unjustifiable bias. However, evidence from the bibliometric analysis suggested that it was the former, the weakness of the proxies for research quality, rather than bias in selection. The bibliometric measures of research strength of the selected staff showed no great differences between researchers from different ethnic groups.

10.   We have put in place comprehensive guidelines to ensure that, as far as possible, the next RAE in 2008 is fair, and seen to be fair, to all academic staff. However, these measures will not directly address any underlying factors which result in different groups of staff being less likely to produce research outputs of the standing expected for selection to the RAE.

11.   To address these wider concerns we should look to other equal opportunities policies as the instruments for change. We, working with the Equality Challenge Unit and the Leadership Foundation for Higher Education, have an ongoing programme to support institutions in ensuring there are equal opportunities through a range of measures, for example, the introduction of more flexible working arrangements. A consequence of these initiatives should be to reduce the impact of career breaks on individual academics' research programmes. These policies are being introduced in the context of a trend for increasing proportions of women and academics from ethnic minority groups to be in the more senior academic positions, including professorial posts.

12.   The differences in rates of selection for submission to the RAE are highly visible. It would be unfortunate if this transparency led to the conclusion that the RAE was the cause of inequalities in research careers. Without the relatively stable public funding through block grant that the RAE underpins, it would, in our view, be more difficult for HEIs to improve the career opportunities for academic staff.

Action required

13.   No response to HEFCE is required in relation to this document.