The Independent Schools Council received a survey on parents’ responses to VAT on private school fees. It was statistically meaningless, and the authors of the report say this was clear in the report. The ISC then gave the survey results to the Daily Mail without this vital context, and the result was a highly misleading headline suggesting that 40% of children would leave private schools. The ISC should be ashamed.
UPDATE 5 June 2024: Baines Cutler have published a statement saying that their research was misused and wasn’t representative of the sector, and distancing themselves from the 42% figure. Coverage in ipaper here.
We’ve written before about interest groups generating headlines using dodgy statistics. There was a particularly bad example last week in The Daily Mail.
If you tried to find the report in question, from education consultancy Baines Cutler, you won’t have succeeded – it’s not published anywhere.
The methodology
Baines Cutler kindly provided us with background on the report. They sent a full copy to the Independent Schools Council1, who sent the Mail a short summary and these two charts:
We should immediately be sceptical of this result. Would a 15% fee increase2 really cause 40% of private school pupils to leave, when significant historic increases haven’t had measurable effects? And how reliable a guide is this kind of question to what people will actually do?3 And doesn’t the second chart contradict the first?
But the more fundamental issue is that the survey was not statistically representative. Here’s what Baines Cutler told me about their methodology:
“The data is from parental surveys which represent 30,000 parents and 35,000 pupils.”
In other words, they sent a survey to parents, and then just collated the responses and published the results. Baines Cutler applied no statistical controls of any kind. It’s no more reliable than a Twitter poll.
The problem is that, whether for systematic reasons4 or sheer chance, those responding to surveys will usually be unrepresentative of those who don’t respond. There are two ways of dealing with this.
- Traditional opinion polling surveys a random sample of the population (e.g. by calling randomly selected numbers).
- The newer approach, pioneered by YouGov and others, has a panel of registered users, and then sends surveys to a statistically representative sample of that panel.
In both cases, the results are statistically adjusted (“weighted”) so they are representative of the population.
The fallacy that a large survey will be accurate was most famously illustrated by the Literary Digest, who surveyed 2,376,523 readers for their poll of the 1936 US Presidential election, and got it spectacularly wrong.5
The importance of random sampling is literally GCSE-level maths.
The professional view
I have studied advanced statistics, and am reasonably proficient – but I would never claim to be an expert. Matt Singh of Number Cruncher Politics very much is. Here’s his take on the presentation of the Baines Cutler report:
Dan Neidle of Tax Policy Associates drew my attention to a report in the Daily Mail claiming that 4 in 10 private school pupils could be “driven out” by VAT on fees. Dan dug into the background to this research and the consultancy that did it, and was told only that it had an impressive sounding sample size and response rate.
Regular readers will know that those things are, on their own, meaningless– the sample has to be scientific for you to generalise to people that haven’t taken part from those that have. And based on the information provided, this survey appears to be unscientific, not being a quota or random sample, and therefore cannot be generalised.
Additionally, even with a representative sample, this would still be difficult to poll. For one thing, people are not good at predicting their behaviour in the future, and for another, people may “preference signal” by exaggerating their likelihood of taking an action, in order to emphasise their view (in this case on charging VAT).
I doubt this will be the last time something like this pops up during the campaign. My advice to all is to be on your toes and exercise appropriate scepticism.
This is from Matt’s latest newsletter – you can subscribe to it here.
The Baines Cutler and ISC response
I asked Baines Cutler about this. They told me:
“The full report makes it clear that 30,000 parents is not statistically representative of the entire sector “
and:
“The entire point of this data in our report was to give schools “something to model” – because there is such lack of clarity from Labour’s actual plans.
It was never designed to be grossed up to the entire population of pupils like the Daily Mail have done, and the 224k number has never been published anywhere in our reports and is in our eyes too high for many reasons.”
This is pretty astonishing, because it implies that the ISC received a report that said it wasn’t representative, but then press-released a summary without this caveat.
I asked the Independent Schools Council if this was true. They denied that the report said it was unrepresentative but refused to go into more detail. I put to them that anyone with any knowledge of statistics would know a survey of this kind was meaningless – they didn’t respond.
There are really only two possible conclusions here.
- If we believe Baines Cutler, then the ISC cynically presented their report as meaningful when they knew it was not.
- If we believe the ISC, then they didn’t know what they were doing, and would fail GCSE maths.
Either way, it seems clear that nobody should trust any statistics from the ISC. And private schools should speak to their Year 11 maths classes before they use any of this data themselves.
What’s the correct figure?
I have no view on this question, as it requires expertise in econometrics and education policy which we do not have.
We wrote about the difficulties of coming up with an estimate here. The only serious attempt to come up with an estimate is this from the IFS. The analysis is, as the authors note, subject to numerous uncertainties, but it takes a rigorous approach.
There’s also a report from the Adam Smith Institute. It contains some valid criticisms of the IFS approach, but is then fatally undermined by using Baines Cutler figures employing the same worthless methodology as those discussed above.
Many thanks to polling expert Matt Singh for his comments. And a quick plug for How to Lie with Statistics, which is brilliant.
Daily Mail front page © Associated Newspapers Limited, and reproduced here for purposes of review/criticism.
Footnotes
The original version of this article said that the ISC commissioned the report. The ISC tells me that is not correct. ↩︎
It’s generally agreed, by the ISC and individual schools, that VAT recovery means the net cost of VAT will be 15% not 20%. Baines Cutler surveyed the effect of a 20% increase and then reduced that by 1/4 to reflect the expected 15% increase. ↩︎
As the pollster Matt Singh put it to us, “people are not good at predicting their behaviour in the future… and may “preference signal” by exaggerating their likelihood of taking an action, in order to emphasise their view”. ↩︎
Most obviously: more engaged people, not representative of the population, are more likely to return the survey ↩︎
Many thanks to Matt Singh for this. A mistake in the first draft read “Readers Digest” – that was my error, and (again) shows the danger of writing anything from memory without checking it first. ↩︎
Leave a Reply to Dan Cancel reply