Measuring the benefits of higher education is complex and challenging. In the past, this has led to a focus on graduate earnings, as something we can easily measure. There is now a danger there will be too big a focus on another single measure: highly skilled employment.
The government itself has produced a lot of evidence for the many personal, societal, economic and non-economic benefits of higher education. In recent years policymakers have also recognised the limits of just focussing on graduate earnings. However, regulatory processes have not caught up with the evidence, and remain focused on measures that are limited and absolute.
Measuring the value delivered through higher education only through graduate earnings or employment ignores the meaningful benefits that universities provide for students, employers and their local and national economies.
Our framework for programme reviews for English institutions applies a broader idea of value to how universities assess their courses. We want to see this wider view of value used in regulation and funding for higher education.
Employment outcomes are important to students. Getting on the career ladder is the top reason students cite for going to university. Young students and applicants report that getting a career they are passionate about is the most important goal in their life.
Widening opportunities for employment is a core part of universities’ social mobility missions.
Employment and job characteristics also influence various aspects of our health. Workers in higher skilled occupations often report higher life satisfaction than those in other forms of work.
Graduates deliver substantial value to their employers and the economy more widely. Supporting a skilled workforce that meets the needs of national and local employers is a key purpose of higher education.
Highly skilled employment is often defined as jobs that fall within one of three groups in the Standard Occupation Classification (SOC) system:
This is the classification that the Higher Education Statistics Agency (HESA) use in its Graduate Outcomes survey (GOS).
Measures of employment or highly skilled employment already feature heavily in assessments of value, including:
Highly skilled employment also features in the proposed measures for regulating student outcomes in England and the revised Teaching Excellence and Students Outcomes Framework (TEF). Both of these proposals are currently under consultation. We plan to publish responses to these consultations when we have fully reviewed the proposals.
Measures of ‘positive’ graduate outcomes sometimes include both graduates in highly skilled employment and those in further study.
We define how useful measures of value in higher education are by how useful they are as tools for regulation, funding. They should be useful to universities in reviewing courses and providing information for prospective students.
To see how well the GOS supports these areas, we’ve looked at how it meets three requirements of a good measure for policymaking:
Accuracy of a measure is how well it measures what we expect it to.
Reviews of the Graduate Outcomes Survey (GOS) have found there is little reason to doubt the accuracy of classifying jobs as highly skilled. However, there are still reasons to be cautious when using these statistics, particularly in ways which place a disproportionate weight on outcomes.
Response rates for the survey have been relatively low, at 53% full responses from UK domicile graduates compared with the 70% target. Research found that the response rates had no major impact on the accuracy of statistics drawn from the data. However, there is no guarantee that this will be the case for future cohorts.
Statistics from the GOS have been rated positively by the Office for Statistical Regulation (OSR) in an initial review against the Code of Practice for Statistics which looks at the trustworthiness, quality and value of statistics.
However, the statistics are still experimental until there is a fuller assessment by the OSR in a few years' time. Until then, the highly skilled employment measure shouldn’t be used for large-scale decision-making without wider context. This is both as a measure by itself, or as part of a composite measure (a combination of measures that make up one measure).
A measure for decision-making must be relevant to what we’re trying to achieve with a policy. The best measure (or set of measures) tells us information about the different ways higher education affects graduates and society while minimising ‘noise’ from irrelevant sources. Bad use of measures can lead to problematic and unintended consequences, some of which we’ve outlined below.
Although good outcomes are important for students, students have different ideas of what a ‘good outcome’ is. This is because:
Progression into highly skilled employment only captures outcomes during one week at one point in time (15 months after graduation) This has its limits, for example:
The SOC grouping system was not designed to tell the difference between graduate and non-graduate jobs.
The SOC approach uses job titles to determine whether the role is ‘professional’ or not using a pre-existing list, but it’s questionable how much holding a professional occupation tells us about graduate progress.
Job titles alone do not tell us whether the job is suitable for a graduate, especially as employers benefit from the skills graduate workers bring to all roles.
Classification of the skill levels of occupations isn’t updated regularly. It therefore falls behind changes in the labour market. In particular:
Students on ‘one-year interim study’ are studying for one year after they graduate and before they complete the survey. These graduates make up around 8% of respondents with known outcomes. Courses with high numbers of graduates on one-year interim study are likely to have inflated unemployment and non-highly skilled
employment figures.
This is because:
Whether a graduate is in a ‘highly skilled job’ is less relevant than if they are in a job that matches their skill level.
Graduates who enter jobs that are typically held by those with lower-level qualifications suffer from low job satisfaction and high indicators of poor mental health. Even when compared to their colleagues with lower qualifications.
However, the negative effects of ‘underemployment’ are not seen as clearly when this is identified using job titles and SOC codes. We get a better picture by looking at the skills someone says they are using.
A good measure can tell us something meaningful about higher education and allow us to make practical changes. There are several ways in which putting too much focus on the highly skilled employment measure fails this.
Highly skilled employment measures are only based only on historical data and are unlikely to reflect recent changes in the labour market. This means:
Economic recovery and growth are not just dependent on the level of skills in the workforce, but also how they are used to meet current employer demand.
The World Economics Forum estimates that 97 million new jobs will emerge between 2020 and 2025 and that the demand for different skills delivered through higher education, such as critical thinking and problem-solving, will only increase during this period.
Current measures of highly skilled employment cannot tell us how well graduates fill these skills gaps.
High skilled employment does not help us understand how graduates address local and regional skills needs.
A far more useful measure would show which courses have graduates who take skilled jobs that support growth in parts of the country that would benefit from levelling up.
Such a measure might use data about the location of graduates, as well as up-to-date information about vacancies and skills shortages across different parts of the country.
Graduates engaging in start-ups and other entrepreneurship may be valuable, though access to some relevant details will depend on HESA’s upcoming review of its Higher Education Business and Community Interaction survey.
If used inappropriately, this measure tells us nothing about the value added by higher education.
The likelihood of a graduate being employed in a highly skilled role is influenced by multiple factors outside of a university’s control. This includes where they live, ethnicity, and socioeconomic background.
Without accounting for this, either by benchmarking or using contextual information, there is a risk of negatively affecting providers with high numbers of students who are less likely to access highly skilled employment straight after graduating, even with degrees.
It’s crucial that the level at which graduate employment is assessed ensures students are supported in their career aspirations even where students’ individual circumstances mean they are less likely to gain immediate entry to ‘highly skilled employment’.
Using measures of employment inappropriately, such as a heavy focus on the proportion of graduates in ‘highly skilled employment’, can have negative implications for policymaking.
Not being sensitive to changing labour needs will lead to missed opportunities to support courses that support emerging local skills needs. Policy based on measures of highly skilled employment alone will not create meaningful growth in a rapidly changing economy.
By only focussing on highly skilled employment, providers who are working successfully to widen participation in higher education will be penalised. This includes universities that teach students from demographics that are less likely to be hired in professional roles because of systemic challenges that aren’t to do with their employability. This includes students who:
As the OfS reviews its Access and Participation strategy, the English higher education system has an opportunity to provide for students and graduates who have been disrupted by the pandemic. However, any target-based approach to measuring student outcomes must support and not undermine the efforts of providers to further diversify their student intake and deliver opportunities locally.
Skilled employment outcomes are important to how we understand some of the value of higher education. However, there are several ways we can improve how we use them.
The OfS has released a series of experimental statistics to explore how highly skilled employment rates differ by graduate location. It is critical for policymakers and everyone using the data to use the indicators the OfS suggests, or any others that account for regional differences. This will support the efforts universities are making to local employment and widening participation.
Employers should be involved in developing skills and employment measures that reflect their needs. These could inform changing needs across different sectors and different parts of the country.
Apply benchmarking or other techniques that consider how opportunities vary across different student groups. The methods should be evidenced and updated to include new knowledge about external factors of graduate employment and new data.
Data is available that can build more nuanced measures of good employment outcomes.
For example, measures can be developed based on survey responses to:
HESA have used some of the data to develop an experimental composite measure of ‘fair work’. Early evidence indicates that, while there are regional differences in job quality, these do not favour areas such as London and South East England that typically perform well in earnings and highly skilled employment rates because of their labour markets.
It is still important for these measures to account for contextual factors such as student demographics – something that will require our understanding and practices to be developed.
We have to look at the many ways that higher education delivers value. Focusing on one measure of one type of benefit will always be limited and have negative effects.
Using qualitative approaches – such as gathering insights from students, employers, and academic staff – is necessary to fill in the gaps and limitations of data. While it can be time-consuming, it can shed light on the causes behind findings in the data.
Universities have a duty to deliver valuable courses. By moving beyond graduate earnings and a select type of employment to assess the value of higher education, universities will be able to design courses that deliver on the aims of universities, students, employers and taxpayers.
Universities have a duty to deliver valuable courses. By moving beyond graduate earnings and a select type of employment to assess the value of higher education, universities will be able to design courses that deliver on the aims of universities, students, employers and taxpayers.
