Executive summary
Purpose
1. This document presents the results and trends from an analysis of the first nine years of data from the National Student Survey (NSS). The two main areas of investigation are the robustness of the NSS results, which is considered by investigating response patterns, and the difference between the NSS results split by a variety of student and course characteristics such as ethnicity, gender and subject.
Key points
Robustness of the results
The NSS question scales are valid across all years of the NSS
2. The National Student Survey (NSS) originally contained 22 questions which were split into six sets of questions, also known as NSS question scales. These six scales were:
- Teaching and learning
- Assessment and feedback
- Academic support
- Organisation and management
- Learning resources
- Personal development.
3. There is an additional standalone question regarding overall satisfaction.
4. Statistical methods can be used to determine whether the individual NSS questions can be grouped into related sets of questions. This analysis of the responses indicates that the individual questions cluster into the same groups as the NSS question scales listed in paragraph 2. Hence the way in which the questions were originally grouped formed a solid structure for the survey which has remained consistent over the nine years of the NSS.
When presenting results, ‘definitely agree’ and ‘mostly agree’ can be combined as a measure of satisfaction. However, ‘definitely disagree’ and ‘mostly disagree’ should not be combined as a measure of dissatisfaction
5. Response patterns of related questions differ more depending on whether ‘definitely disagree’ or ‘mostly disagree’ is the chosen response, compared with those who choose ‘definitely agree’ or ‘mostly agree’. This suggests that the results for ‘neither agree nor disagree’, ‘mostly disagree’ and ‘definitely disagree’ should be presented as separate percentages.
The questions have become more similarly answered over time
6. The relationship in the responses between individual questions in the NSS has become stronger over time. This means that the responses to individual questions are more similar in 2013 than they were in 2005.
7. There are close relationships between the response to the overall satisfaction question and the responses to the individual questions. These relationships suggest that all the responses to the questions reflect the overall satisfaction of students.
Over 5 per cent of respondents tick the same answer for every question
8. The act of choosing the same answer to every question is known as acquiescence bias or yea-saying. In 2005 only 1.0 per cent of respondents were found to be yea-saying, but by 2013 this had risen to 5.4 per cent. This is much higher than would be expected, even after accounting for the very high levels of satisfaction and strong relationship between the answers to different questions.
9. The vast majority of yea-saying occurs with respondents choosing the ‘definitely agree’ response category.
10. There is no indication of any link between the incentives or prizes offered by institutions to their students for completing the survey and the proportion of yea-saying. There is also no relationship between the size of an institution and the proportion of yea-saying.
11. We have tested the impact of yea-saying on the NSS results by removing all responses with identical answers to every question from the analysis, and have found no material difference in the sector results. However, if the proportion of yea-saying continues to rise, there is a possibility that it could affect the robustness of the NSS results.
Online responses are the most similar
12. When the responses are split by the three survey completion methods (online, postal and phone), the strongest relationship between responses to the questions is found in the online responses. In other words, the responses of any given individual are most similar when responding online.
13. Yea-saying is present in responses amongst all completion methods, but most predominant in online responses. When yea-sayers are removed from the data, the results of the analysis in paragraph 13 still hold true.
Student satisfaction
Students have become more satisfied over time
14. The percentage of students who agree with each of the six NSS question scales and the overall satisfaction question has increased over time.
Black Caribbean students are less satisfied than the average, but Black African students are more satisfied
15. The ‘overall satisfaction’ of Black Caribbean students has always been lower compared with White students. An expected ‘overall satisfaction’ score for different student characteristics can be generated by modelling the NSS results. The largest difference between this and the actual score for ‘overall satisfaction’ from Black Caribbean students was seen in 2011, when the satisfaction score for Black Caribbean students was 73.6 per cent compared with the expected satisfaction score of 80.1 per cent. Black Caribbean students were therefore 6.5 percentage points less satisfied than expected in 2011.
16. Conversely, the proportion of Black African students who agree with the overall satisfaction question is higher compared with White students. This difference peaked in 2010, when Black African students had an expected ‘overall satisfaction’ score of 79.7 and an actual ‘overall satisfaction’ score of 83.3. Hence, in 2010, Black African students were 3.6 percentage points more satisfied than expected.
The largest variation in satisfaction scores is observed when considering subject of study
17. By far the biggest variation in satisfaction is between subjects of study. The results of modelling 2013 ‘overall satisfaction’ are spread across a range of more than 15 percentage points when split by subject.
Students who declare a disability are less satisfied than those with no known disability
18. Across the five years of modelled overall satisfaction results, those with a declared disability have been consistently less satisfied than those with no known disability.
Action required
19. In addition to this report, there is an opportunity to explore the NSS data further using an online NSS tool. The online tool contains data from all nine years of the NSS, and allows investigation of the results split by a variety of student characteristics (such as age, gender and ethnicity).
20. Over the course of the NSS we have had over 2 million responses which have been linked back to the student characteristics of individual respondents.