You are in :
HEFCE

April 2008/14
Issues paper

This report is for information only


Counting what is measured or measuring what counts?
League tables and their impact on higher education institutions in England

This report was commissioned by HEFCE to investigate league tables and their impact on higher education institutions in England. It presents findings from an analysis of five league tables, and an investigation of how higher education institutions respond to league tables generally and the extent to which they influence institutional decision-making and actions.

Report to HEFCE by the Centre for Higher Education Research and Information (CHERI), Open University, and Hobsons Research.


Table of contents and foreword (read online)


Appendices

Appendix A - Research methodologies

[ MS Word 140K | Zipped Word 26K | Adobe PDF 86K | Zipped PDF 63K ]

Appendix B - Standard statistical concepts, methods and processes used in the compilation and analysis of league tables

[ MS Word 150K | Zipped Word 47K | Adobe PDF 302K | Zipped PDF 130K ]

Appendix C - Detailed findings of the analyses of the five league tables

[ MS Word 1.2Mb | Zipped Word 233K | Adobe PDF 959K | Zipped PDF 680K ]

Appendix D - Detailed findings of the survey of higher education institutions

[ MS Word 2.4Mb | Zipped Word 312K | Adobe PDF 1.8Mb | Zipped PDF 1.2Mb ]

Appendix E - Detailed findings from the institutional case studies

[ MS Word 331K | Zipped Word 83K | Adobe PDF 341K | Zipped PDF 300K ]

Appendix F - The National Student Survey: A brief description

[ MS Word 31K | Zipped Word 6K | Adobe PDF 15K | Zipped PDF 8K ]

Appendix G - Bibliography and relevant web-sites

[ MS Word 122K | Zipped Word 29K | Adobe PDF 78K | Zipped PDF 59K ]


Table of contents

  • Acknowledgements, steering group and research team
  • Foreword by David Eastwood, Chief Executive, HEFCE
  • Executive summary
  • 1   Introduction
    • 1.1 Analysis of the league tables
    • 1.2 Impact on higher education institutions
    • 1.3 Issues arising
  • 2   The debate about league tables and their impact
    • 2.1 The case for league tables
    • 2.2 The case against league tables
    • 2.3 Who uses league tables and why? What is the evidence?
  • 3   League tables: how they are compiled and the results they produce
    • 3.1 General comparison of five league tables
    • 3.2 How the league tables are compiled
    • 3.3 A critique of the five league tables
    • 3.4 Findings from the statistical analysis of the tables
    • 3.5 Summary of key findings on the five league tables
  • 4   The impact of league tables on institutions
    • 4.1 Survey of higher education institutions
    • 4.2 Institutional case studies
    • 4.3 Common themes
  • 5   Alternative approaches and principles of good practice
    • 5.1 The CHE rankings
    • 5.2 Bringing league tables up-to-date
    • 5.3 The Berlin Principles
    • 5.4 What can compilers learn from these approaches?
  • 6   Discussion and conclusions
    • 6.1 What has been confirmed?
    • 6.2 New research findings
    • 6.3 Implications and challenges for key parties
    • 6.4 Where do we go from here?
  • 7   References
  • Appendices
    1. Research methodologies
    2. Standard statistical concepts, methods and processes used in the compilation and analysis of league tables
    3. Detailed findings of the analyses of the five league tables
    4. Detailed findings of the survey of higher education institutions
    5. Detailed findings from the institutional case studies
    6. The National Student Survey: A brief description
    7. Bibliography and relevant web-sites

Foreword

League tables are part of the higher education landscape and the newspaper calendar. They are one of the sources to which prospective students refer when making choices, and bring attention to important issues such as 'the student experience', employability and retention.

The league tables also have a much wider impact - for example, on institutions' reputations and potentially on the behaviour of academics, businesses and potential benefactors. Governing bodies take an interest in them as a means of assessing institutional performance, sometimes seizing on them in default of other, more sensitive indicators of institutional performance.

There clearly is a demand for league tables, but there are also questions about their quality, impact and possible perverse incentives. Concerns have been raised about the compilers' choice of indicators, the validity of the methodologies which are employed, the transparency of the processes and the robustness of the rankings.

As a funder of higher education, we have an interest in ensuring that the sector is accurately presented to prospective students, policy-makers and others with a stake in the quality of higher education; and that the relative strengths of particular institutions are appropriately recognised and reflected. We also have an interest in how governors and managers use league tables, and whether this helps them in pursuing and refining their institution's mission or deflects them from these and other key responsibilities. The prominence of research performance and entry qualifications are two issues that have been examined. We are interested in the extent to which league tables support policy objectives - for instance, by making higher education institutions more sensitive to student demands, and any impacts on objectives such as widening participation.

Our purpose in commissioning this research is to stimulate informed debate about league tables across the higher education sector; not to endorse any particular approach. We certainly do not intend to introduce an official published ranking, as some have suggested. We will continue to support the Unistats web-site, which enables users to compare subjects and institutions in a way that recognises the diversity of user needs.

This research throws a considerable amount of light on the approaches and limitations of different league tables and the way universities and colleges respond to them. We hope the debate will lead to improvements to league table methodologies; enable users to better understand the complexities of the league tables, and avoid misunderstanding them; and to help higher education institutions develop approaches that help them satisfy the legitimate information needs of their stakeholders.

I am grateful to all those who have contributed to this research project: to the compilers who were willing to speak frankly to the researchers; to the many institutions who responded in detail to the online survey; and the case study institutions who were so generous with their time. I look forward to the debate!

Professor David Eastwood
Chief Executive, HEFCE