Counting what is measured or measuring what counts? - League tables and their impact on higher education institutions in England

April 2008 | ref: 2008/14

This report was commissioned by HEFCE to investigate league tables and their impact on higher education institutions in England. It presents findings from an analysis of five league tables, and an investigation of how higher education institutions respond to league tables generally and the extent to which they influence institutional decision-making and actions.

Report to HEFCE by the Centre for Higher Education Research and Information (CHERI), Open University, and Hobsons Research.

Table of contents

  • Acknowledgements, steering group and research team
  • Foreword by David Eastwood, Chief Executive, HEFCE
  • Executive summary
  • 1   Introduction
    • 1.1 Analysis of the league tables
    • 1.2 Impact on higher education institutions
    • 1.3 Issues arising
  • 2   The debate about league tables and their impact
    • 2.1 The case for league tables
    • 2.2 The case against league tables
    • 2.3 Who uses league tables and why? What is the evidence?
  • 3   League tables: how they are compiled and the results they produce
    • 3.1 General comparison of five league tables
    • 3.2 How the league tables are compiled
    • 3.3 A critique of the five league tables
    • 3.4 Findings from the statistical analysis of the tables
    • 3.5 Summary of key findings on the five league tables
  • 4   The impact of league tables on institutions
    • 4.1 Survey of higher education institutions
    • 4.2 Institutional case studies
    • 4.3 Common themes
  • 5   Alternative approaches and principles of good practice
    • 5.1 The CHE rankings
    • 5.2 Bringing league tables up-to-date
    • 5.3 The Berlin Principles
    • 5.4 What can compilers learn from these approaches?
  • 6   Discussion and conclusions
    • 6.1 What has been confirmed?
    • 6.2 New research findings
    • 6.3 Implications and challenges for key parties
    • 6.4 Where do we go from here?
  • 7   References
  • Appendices
    1. Research methodologies
    2. Standard statistical concepts, methods and processes used in the compilation and analysis of league tables
    3. Detailed findings of the analyses of the five league tables
    4. Detailed findings of the survey of higher education institutions
    5. Detailed findings from the institutional case studies
    6. The National Student Survey: A brief description
    7. Bibliography and relevant web-sites

Foreword

League tables are part of the higher education landscape and the newspaper calendar. They are one of the sources to which prospective students refer when making choices, and bring attention to important issues such as 'the student experience', employability and retention.

The league tables also have a much wider impact - for example, on institutions' reputations and potentially on the behaviour of academics, businesses and potential benefactors. Governing bodies take an interest in them as a means of assessing institutional performance, sometimes seizing on them in default of other, more sensitive indicators of institutional performance.

There clearly is a demand for league tables, but there are also questions about their quality, impact and possible perverse incentives. Concerns have been raised about the compilers' choice of indicators, the validity of the methodologies which are employed, the transparency of the processes and the robustness of the rankings.

As a funder of higher education, we have an interest in ensuring that the sector is accurately presented to prospective students, policy-makers and others with a stake in the quality of higher education; and that the relative strengths of particular institutions are appropriately recognised and reflected. We also have an interest in how governors and managers use league tables, and whether this helps them in pursuing and refining their institution's mission or deflects them from these and other key responsibilities. The prominence of research performance and entry qualifications are two issues that have been examined. We are interested in the extent to which league tables support policy objectives - for instance, by making higher education institutions more sensitive to student demands, and any impacts on objectives such as widening participation.

Our purpose in commissioning this research is to stimulate informed debate about league tables across the higher education sector; not to endorse any particular approach. We certainly do not intend to introduce an official published ranking, as some have suggested. We will continue to support the Unistats web-site, which enables users to compare subjects and institutions in a way that recognises the diversity of user needs.

This research throws a considerable amount of light on the approaches and limitations of different league tables and the way universities and colleges respond to them. We hope the debate will lead to improvements to league table methodologies; enable users to better understand the complexities of the league tables, and avoid misunderstanding them; and to help higher education institutions develop approaches that help them satisfy the legitimate information needs of their stakeholders.

I am grateful to all those who have contributed to this research project: to the compilers who were willing to speak frankly to the researchers; to the many institutions who responded in detail to the online survey; and the case study institutions who were so generous with their time. I look forward to the debate!

Professor David Eastwood
Chief Executive, HEFCE 

Executive summary

This report investigates league tables and their impact on higher education institutions (HEIs) in England. It presents findings from two strands of research:

  • an analysis of five league tables selected for the study, their methodologies and the underlying data employed
  • an investigation of how higher education institutions respond to league tables generally and the extent to which they influence institutional decision-making and actions.

The five league tables analysed are:

  • national
    • Sunday Times University Guide
    • The Times Good University Guide
    • The Guardian University Guide
  • world rankings
    • Academic Ranking of World Universities (Shanghai Jiao Tong University Institute of Higher Education)
    • THES-QS World University Rankings.

The purpose of the research is to stimulate informed debate about the approaches and limitations of the various league tables, and greater understanding among the users and stakeholders of the implications of making decisions based on these sources of information.

Analysis of the five league tables

Below are the main findings about the league tables themselves.

  • The five league tables do not provide a complete picture of the sector. Their focus is on full-time, undergraduate provision and institutional, rather than subject-based, rankings. This emphasis results in the exclusion of a wide range of specialist, postgraduate, small or predominantly part-time institutions from the published rankings. The lack of availability of certain types of published data results in some higher education provision by further education institutions also being excluded. Not including the full range of higher education provision that would be of interest to the target users of league tables is a significant limitation on their usefulness.
  • Some of the measures included are poor proxies for the qualities identified. The measures used by the compilers are largely determined by the data available rather than by clear and coherent concepts of, for example, ‘excellence’ or ‘a world class university’. Also the weightings applied do not always seem to have the desired effect on the overall scores for institutions. This brings into question the validity of the overall tables. More attention should be given to developing methodologies that reflect the qualities of institutions identified as desirable by the publishers.
  • There is insufficient transparency about the way the league tables are compiled. Methods for calculating the scores for each institution are not always made clear, and some appear to be non-standard or, at least, produce non-standardised results. Some publishers even warn readers that it is not possible to replicate the overall scores from the published indicators.
  • The resulting rankings largely reflect reputational factors and not necessarily the quality or performance of institutions. In the national league tables, entry qualifications, good degrees and Research Assessment Exercise grades are more highly correlated with the total scores than are other measures. The total scores of institutions are less highly correlated with indicators based on the National Student Survey (NSS) results. This suggests that important elements of course and institutional quality, such as educational processes, do not contribute as much as intended to ranking outcomes. In the world rankings the picture is more mixed, although articles published and cited seem to have a considerable influence on the ranking positions.
  • The format and content of league tables could be brought up to date. They could be made more easily accessible and interactive. For example users could be given the facility to select the indicators which are important to them, and the weightings applied to these. League tables could also reflect recent developments in higher education, such as online learning, and current issues of concern to users, such as social responsibility and environmental impact.

Impacts on higher education institutions

Below are the main findings about how league tables impact on institutions’ actions and decision-making.

  • Institutions are strongly influenced by league tables. League tables and the individual indicators used to compile them appear to be having a significant influence on institutions’ actions and decision-making, although HEIs themselves are reluctant to acknowledge this. League tables are being used by many institutions as key performance indicators and, in some cases, strategic targets. They are being used by some senior management teams and governing bodies as one of several drivers for internal change. While it is understandable that an institution values its public image as represented in league tables, each needs to manage the tensions between league table performance and institutional and governmental policies and priorities. Some institutions expressed the belief that league tables will become more influential as higher education becomes more competitive.

There is a challenge for institutions, sector bodies and policy makers to ensure the accessibility of accurate, relevant and comprehensive information about higher education institutions to prospective students, their advisers and other users of league tables.

  • Institutions do not feel they have sufficient influence on the compilers and the methodologies used in the rankings. Many favour the inclusion of more measures of value added, and for league tables to reflect broader characteristics than just reputation and research. The lack of transparency about how league tables are compiled is a concern to many HEIs.
  • Institutions are responding to the National Student Survey. Increasing importance is being attached to the results of the National Student Survey, and their inclusion in league tables may be contributing to this. There is widespread evidence of institutional actions and initiatives arising from NSS results.
  • League tables have resulted in better data collection. League tables have prompted many institutions to review their data collection and submissions to HESA and other bodies. They are now seeking to provide higher quality returns.
  • Staff are affected by league tables. Despite widespread scepticism about league tables and their methodologies within HEIs, rankings affect staff morale. However, they do not appear to influence academic recruitment significantly except, perhaps, for some individuals considering a move to the UK. Nevertheless, it is thought to be unlikely that academics will move to a lower-ranked institution than their current one unless there is a pocket of excellence or other overriding reason.
  • League tables may conflict with other priorities. There is perceived tension between league table performance and institutional and governmental policies and concerns (e.g. on academic standards, widening participation, community engagement and the provision of socially-valued subjects). Institutions are having to manage such tensions with great care.

Institutions’ perceptions of the impacts of league tables on users

Institutions were asked what impact they thought league tables have on users. Below is a summary of their perceptions.

  • ‘Traditional’ prospective students are more likely to use league tables. For younger HE applicants of higher academic achievement and social class, league tables may be influential, but only part of the complex decision making process and often used to confirm a decision already made. Factors such as subject and location still appear to play a greater part in decision-making. Applicants who are mature, locally recruited, more vocationally orientated and/or from less advantaged backgrounds are not as likely to use them. To make the most of league tables, prospective students and their advisers could be better informed about which table(s) or indicators best reflect the higher education experience they are looking for. Subject-based rather than institutional rankings are likely to give a better indication of ‘performance’ (at least in the way that this is assessed in a league table) compared with overall institutional rankings.
  • Internationally, league tables influence students, academics and governments. International students seem to be increasingly using league tables in selecting which higher education institution in the UK to apply to. Foreign governments and scholarship bodies are using them to inform decisions about support for students and which institutions in the UK to partner with. League tables appear to influence international academics from some countries in deciding which UK institution to come to, and more so than academics moving institution within the UK.

General conclusions

The influence of league tables is increasing both nationally and internationally, and cannot be ignored despite serious methodological limitations. They are being used for a broader range of purposes than originally intended, and being bestowed with more meaning than the data alone may bear. It is possible that the influence of league tables will increase further if the cap on tuition fees for full-time undergraduate courses is raised or lifted altogether. It is possible that ranking position will affect an institution’s ability to charge the highest fees across all its courses. The world rankings are growing in influence due to internationalisation and are likely to continue to do so if bibliometric indicators are introduced to assess research quality in the UK.

Given this increasing influence, there is an onus on policy makers and institutions themselves to promote greater public understanding of league tables and alternative sources of information about higher education. There is also an argument for codifying good practice in the compilation of rankings as a reference point for both compilers and users of league tables. With the increasing influence of world rankings originating from outside the UK and their use by overseas students, academics and governments, this may be best achieved at an international level as part of an inter-governmental initiative.

 There are a number of areas that would benefit from further research, in particular, into users’ perspectives, including:

  • prospective (including international) students’ use of league tables
  • the use of new sources of information on higher education institutions, e.g. social networking internet sites and Unistats
  • the influence of league tables on foreign governments, scholarship bodies, employers and individual academics.

Download:

Appendix A - Research methodologies

MS Word 140K | Zipped Word 26K | Adobe PDF 86K | Zipped PDF 63K

Appendix B - Standard statistical concepts, methods and processes used in the compilation and analysis of league tables

MS Word 150K | Zipped Word 47K | Adobe PDF 302K | Zipped PDF 130K

Appendix C - Detailed findings of the analyses of the five league tables

MS Word 1.2Mb | Zipped Word 233K | Adobe PDF 959K | Zipped PDF 680K

Appendix D - Detailed findings of the survey of higher education institutions

MS Word 2.4Mb | Zipped Word 312K | Adobe PDF 1.8Mb | Zipped PDF 1.2Mb

Appendix E - Detailed findings from the institutional case studies

MS Word 331K | Zipped Word 83K | Adobe PDF 341K | Zipped PDF 300K

Appendix F - The National Student Survey: A brief description

MS Word 31K | Zipped Word 6K | Adobe PDF 15K | Zipped PDF 8K

Appendix G - Bibliography and relevant web-sites

MS Word 122K | Zipped Word 29K | Adobe PDF 78K | Zipped PDF 59K


Page last updated 10 January 2012

Share this: