A Cautionary Note on Measuring the Pupil Premium Attainment Gap in England

This exploratory paper uses figures from the National Pupil Database for England to assess the known characteristics of three categories of pupils – those never eligible for free school meals, those who have been eligible but are not now, and those eligible now. It shows that these groups display a clear gradient in terms of special education needs, English as an additional language, and formal qualifications at age 16. The group currently eligible for free schools meals is geographically stratified, faces on average more educational challenges, and gains worse results than the group that had once been eligible but is not now. This shows that we cannot expect the same results from schools with more permanently poor pupils as from schools with many pupils on the threshold of poverty or who move in and out of poverty during their school careers. These findings could be crucial for the rules on how the pupil premium is allocated to schools, and to current policies based on assessing the pupil premium gap in schools, including the work of OFSTED, RAISE, the National pupil premium Champion, and various school awards. Many of the calculations underlying such policies will be unintentionally misleading, and unfair to certain regions and individual pupils.


INTRODUCTION 1.1 The Pupil Premium Gap
According to the Department for Education (DfE), the pupil premium (PP) is additional funding given to publicly funded schools in England to raise the attainment of disadvantaged pupils and close the gap between them and their peers [1].The pupil premium policy was announced by the Coalition government in 2010, and the amount of extra funding per school rose to £1,300 per annum for specified primary school pupils, and £935 for secondary pupils.The funding is received by schools for every pupil who has been entitled to receive free school meals (FSM) over the previous six years.FSM, or its equivalent in other countries, is a widely used and convenient administrative proxy for a pupil from a disadvantaged background, who is more likely than average to struggle at school [2].Additional PP funding is available for children who are or have been living in care.The money must be spent on activities primarily intended to raise the attainment of these potentially disadvantaged pupils [3].The idea of PP is well-meaning and, once schools are clear on suitable evidenceinformed approaches to raising attainment for this group, the policy is likely to have considerable and beneficial impact [4].
Since 2010, PP has become embedded in schools policy in England.Its use is assessed by the school inspection regime OFSTED when inspecting schools, and a pupil premium achievement gap has been formalised on their tracking system 'Reporting and Analysis for Improvement through school Self-Evaluation' [5].This gap is the simple difference in percentage points in each school between the percentage of PP and non-PP pupils attaining five GCSEs at grade A*-C or their equivalent, including English and maths [6].The GCSE is the most common traditional public examination at age 16.This gap is used routinely by schools themselves, their local authorities and sponsor chains, and by the government-appointed National pupil premium Champion [7], to monitor progress in improving attainment for PP pupils.It is even used to justify giving annual awards to schools with small or narrowing PP gaps [8].
There are several, perhaps relatively minor, problems with calculating and using attainment gaps in this way.There are objections that such an approach disguises and so diverts attention from the issues of disadvantage that help generate it, and that it emphasises standardised attainment over other educational outcomes [9].Nevertheless, such gaps are calculated and used in practice, despite having considerable relevant data missing.Around 4% of pupils in statefunded schools have unknown FSM-eligibility status, for example [10].It is also not clear that a simple percentage point difference adequately expresses the gap, because it takes no account of the figures from which the difference emerges [11].This means that a school with none of its FSM-eligible pupils, but 15% of its other pupils, attaining the level 2 GCSE indicator of five or more GCSEs graded A*-C would be said by OFSTED and others to have the same gap as one where 85% of FSM-eligible pupils and 100% of the rest attained five such 'good' GCSEs.It is not clear that this is correct.And some small schools or schools with low levels of disadvantage would naturally have gaps subject to considerable volatility, because of the way small numbers behave in practice.There are also concerns that, once other benefits are taken into account, FSM-eligible pupils are no longer from the very lowest income families in England [12].
However, there is a more fundamental problem which this paper explores.FSM-eligibility is not a constant characteristic of an individual pupil, in the same way that sex or ethnicity usually are.Levels of FSM-eligibility are linked to the economy [13].They are also linked to family circumstances, meaning that pupils might move in and out of FSM-eligibility over their school careers.Those pupils previously eligible for FSM but not subsequently are termed a 'hidden poor' by Noden and West (p.4) [14], no longer entitled to some benefits but potentially still suffering the impacts of earlier disadvantage.Partly for this kind of reason, the DfE now produces a measure 'EverFSM6' which includes pupils both currently and previously eligible for FSM (over the previous six years of schooling).Treadaway [15] considers that even this may not be enough.EverFSM6 still ignores pupils in secondary school who had been eligible more than six years previously, during their formative primary school years.
Since FSM-eligibility is a threshold characteristic, this means that there will be variation within FSM-eligibility.Put simply, some FSM-eligible pupils will be poorer than others and eligible for assistance every year, and some may be at or near the threshold and so moving in and out of FSM-eligibility over time.All will trigger receipt of the pupil premium by their schools, but their absolute level of deprivation may vary considerably in a way that is subsequently linked to their attainment.If so, this would make the PP gap calculation intrinsically unfair, by favouring those schools or regions with more pupils near the threshold and fewer who are FSM-eligible year after year.
Based on existing data for all maintained mainstream secondary schools in England for one year, this brief exploratory paper addresses the following issues: • What proportion of pupils is in the three possible groups -never eligible for FSM, previously eligible, or currently eligible?• Are there discernible differences between these three groups in terms of their known characteristics and attainment at age 16, and their distribution between areas of England? • What are the potential implications of these patterns for policy-makers and practitioners?

METHODS
The analysis used to test and illustrate the issues outlined above is based on the National Pupil Database (NPD) for England, Key Stage 4, 2013.This contains a record for every young person in the 15-year-old cohort attending a statemaintained educational establishment -a total of 643,139 cases.
The two key 'independent' variables used here are both flags -zero or one -representing whether a pupil is currently eligible for free school meals (FSM) and whether they have ever been eligible in the past six years (Ever FSM, which is a variable appearing in NPD only recently, and so making this analysis possible).These two flags were used to generate a new variable with the following three values: 'Never FSM' -if FSM and Ever FSM are both zero; 'Previously FSM' -if FSM is zero and Ever FSM is one; 'FSM now' -if FSM is one (and Ever FSM is one).
There is a category unavailable here -where pupils had previously been not eligible for FSM but are currently -and which it is not possible to code from one year of data alone (but which will be assessed as part of a larger study funded by the ESRC -ES/N012046/1).This means that the category 'Previously FSM' underestimates the number of pupils at the threshold of being FSM-eligible or going in and out of FSM eligibility over their secondary school careers.Nevertheless, this newly created variable offers the opportunity for a finer graded consideration of the link between FSM and attainment at school.
The frequencies of the three FSM groups are calculated and converted to percentages of the total cohort (after missing cases are accounted for), for England as a whole and for three local authorities chosen to be illustrative of variation in geography, local prosperity, and the proportions of these three FSM groups.
The 'dependent' outcome variable is the DfE points score for each pupil's best eight GCSE results or equivalent.The points score is used by DfE and others to assist comparability between GCSE results and less common qualifications such as NVQs and BTECs, and it assigns 16 points to a grade G GCSE, increasing in steps of 6, to 58 for a grade A*.The average KS4 points score per pupil is calculated for each of the three FSM groups, and compared in terms of a simple effect size (the difference between two averages divided by their overall standard deviation).
The pupil background variables used are whether a pupil is currently listed as having any form of special educational need, whether they have a statement of special educational need, and whether English is their first language.These variables are all categorical and are crosstabulated in terms of the three FSM groups, and the results are converted to percentages within each category.

National Figures
Around 11.1% of the relevant pupils in England do not have a value in the NPD for whether they are eligible for FSM or not, or for whether they have ever been eligible.Of these, the majority (7%) are in private fee-paying schools which are not required to provide this information, and which anyway would have relatively few FSMeligible pupils.The remaining 4% of pupils in state-funded provision who do not have a value for FSM-eligibility have been shown previously to be a kind of super-deprived group, including those in special schools or recently moving between schools, with higher levels of special needs and lower attainment even than those known to be eligible for FSM [10].This is also true for the 2013 cohort used here.These 'missing FSM' pupils are not used for the most of the remainder of the analysis because so many of them are also missing other key information such as their first language.
For those with valid figures, the clear majority of secondary pupils have never been eligible for FSM in the last six years (Table 1).A high proportion of pupils has previously been eligible, but is not now (11.7%).This group is used for the rest of this paper as indicative of at least some of those pupils from families on the threshold of poverty.However, as noted above, this will be an underestimate of the families moving in and out of poverty over time, because there will also be pupils in the 'FSM now' group who not previously been eligible.Nevertheless, if there are families with permanently very low incomes they will, by definition, be in the last group only (15%).Are they similar in all other respects to the 'FSM previously' group?What this new analysis shows is that pupils who had been eligible but are not now form a group between these two, on all other available indicators as well (Table 2).'FSM now' pupils are more likely than the 'FSM previously' group to be listed as having a special educational need, to have a statement of need, and to speak a language other than English at home, for example.This means that we might expect this new analytical group to have lower attainment at school, on average, than the other two groups.
The three groups do indeed have different levels of attainment at age 16, and in the order envisaged.This fine 'poverty gradient' appears in all measures of assessment, and is illustrated here in terms of the best 8 mean GCSE (and equivalent) point scores (Table 3).The gap between the two new analytical groups themselves is smaller than that between the two groups combined and NeverFSM, but it is still considerable (as also noted by Crawford et al.) [16].
Pupils never eligible for FSM do best, followed by those who had been but are no longer eligible, and finally by those currently eligible.As an 'effect' size, the gap between the last two groups is -0.21.This is smaller than the difference between FSM and not FSM-eligible, but it is a solid figure, based on all relevant pupils in an entire national cohort.It is large enough to make a difference to a pupil examination grade, and easily large enough to make a difference to the overall results for a school or region with a higher proportion of one FSM group than another.The group that contains all of the pupils who are permanently FSM-eligible does considerably worse at school, on average, than the group that contains all of the pupils who move in and out of FSM-eligibility over time.This difference matters.

Local Examples
The difference that this could make to the pupil premium attainment gap is illustrated using three local authorities.Birmingham, Kensington and Chelsea, and Middlesbrough are all urban areas, in the midlands, south east and north of England respectively.These three all have around the same proportion of pupils who have never been eligible for FSM, which means that they all receive comparable pupil premium payments (Table 4).However, all of these areas are different in terms of the proportions of the kind of FSM-eligible pupils they contain.
In the London Borough of Kensington and Chelsea, the clear majority of pupils who have even been FSM-eligible are not now.They probably include, therefore, a proportion who are near the threshold of FSM rather than among the poorest in the country.This could affect the level of qualifications obtained.In fact, over 36% of pupils Kensington and Chelsea are missing any data on FSM-eligibility, confirming that a large number of residents use private fee-paying schools.This might remove some of the highestattaining or richest pupils from attendance at local state-maintained schools.Because of the well-established correlation between socioeconomic status and attainment, this would then tend to reduce the overall level of attainment in local state-funded schools.But it would also reduce the likely gap between the poorest and the majority of those pupils remaining in state-funded schools.This is the kind of factor never considered by those promoting the apparent success of the London Challenge [17].
Any assessment of the pupil premium attainment gap must take these two factors into account.In Kensington and Chelsea most pupils receiving the pupil premium are not currently FSM-eligible, and a large proportion of pupils go to school outside the state system and are not included in the figures here.On average the pupil premium attainment gap is lower in Kensington and Chelsea than in England overall (Table 5).This is to be expected because some of the highest attaining pupils are missing (not in maintained schools), and more importantly because it has fewer permanently deprived pupils than the other areas.Curiously, and in opposition to the national picture, the 'FSM now' pupils do somewhat better than the 'FSM previously' ones.
The situation in the deprived authority of Middlesbrough is very different.Here only 4.7% of pupils are missing data on FSM eligibility, which is around the same as the national average of those genuinely missing data.This confirms that few pupils attend private fee-paying schools.Almost all pupils are in the state-funded system and so contributing to the pupil premium attainment gap there.Unlike in Kensington and Chelsea the clear majority of pupils who have ever been FSM-eligible still are (Table 4).They are likely to include many of those from families permanently receiving other benefits or on low incomes.And it should be expected that these two factors would both tend to increase the pupil premium attainment gap (irrespective of what actually goes on in schools or how the PP is used).
This is what the figures show (Table 6).The pupil premium gap in Middlesbrough is larger than that for England overall.As with the national figures, The picture in Birmingham is slightly different again.Like Middlesbrough, Birmingham has a majority of pupils who have ever been FSMeligible who are currently eligible (Table 4).Around 11.6% of pupils are missing FSM data, which is about the same as the national average, suggesting that attendance at private fee-paying schools is also around average (and so higher than Middlesbrough, but much less than in Kensington and Chelsea).All other things being equal this suggests that the pupil premium attainment gap should be lower than Middlesbrough, but higher than Kensington and Chelsea.And again this is what the figures show (Table 7) above.The finer-graded poverty gradient in results, between the local FSM groups proposed by this paper, is there again.And it would be expected to be there, regardless of how well local schools are using their PP funding.

CONCLUSION: IMPLICATIONS FOR POLICY
The results in this paper raise the possibility that the threshold nature of eligibility for FSM is disguising an important distinction between those who move in and out of eligibility and might be close to the threshold for benefits, and those from even poorer families living in relative poverty during the child's whole school career.
And it must be recalled that while it is not possible with these one-year figures to say anything about pupils who have only recently moved into FSM-eligibility, the figures presented here are likely to be an under-estimate of the pool of pupils who are volatile in terms of FSMeligibility.
The potential implications for policies and practices based on calculating a pupil premium attainment gap are substantial.The findings mean that when policy-makers, advocates of the success of the London Challenge, OFSTED, RAISE, the pupil premium Champion, awards committees and others use the pupil premium gap as a measure of success they are probably and unwittingly being very unfair.It has already been suggested that there is a problem for all such calculations caused by missing data, and because they take no account of the proportion of local residents using private schools, both currently ignored in the calculation of any pupil premium attainment gap (and, as shown above, both influencing the calculation by their absence).What this paper shows more importantly is that they are unfair because they do not take account of the threshold nature of FSM-eligibility.They are ignoring the variation within that category.
As the analysis reveals, this variation within FSM-eligibility is stratified by prior educational challenges like SEN and EAL, and then again by the qualification outcomes used to calculate the gap.Almost as importantly, the analysis shows that different areas have different proportions of the three FSM pupil groups.Heavily disadvantaged areas are likely to have more of the always FSM-eligible pupils, and this makes any comparison with other areas based on the pupil premium gap intrinsically invalid.This is in no way an argument against the pupil premium policy itself, but it does suggest that the impact of the policy needs a rather more robust evaluation than simply measuring changes in the pupil premium attainment gap.
It also means that the PP attainment gap should not be used by OFSTED to pre-determine any aspect of the outcomes of school inspections.
Perhaps just as importantly, the paper has implications for the delivery of the pupil premium itself.Currently these extra resources are given to schools on the basis of the number of pupils in that school who have ever been eligible for free school meals (for the previous six year).This means that schools not only miss out the extra money when data is missing, but that those schools taking the most disadvantaged pupils (likely to attain the lowest at KS4) get the same per capita as those who take the pupils moving in and out of eligibility.Currently, until all else is resolved it would make more sense to allocate the pupil premium primarily on the basis of pupils eligible for FSM at the time of allocation, and then to update this every year throughout their school career.This would mean money going to the schools of those most in need, while they are most in need.