It’s been one hell of a week for everyone involved with exams – especially for the thousands of students receiving A levels. Earlier this week I posted my thoughts about the system: Adjusting CAGs: no politics, no disgrace, no injustice. It’s just technical. Well – that headline didn’t turn out so well did it! The blog was my attempt to lay the ground for what ought to have been a sound technical process, adjusting moderating centre-assessed grades. I have a bias to assume technical wisdom and neutrality and was reacting to a few problems that I have seen in the discourse around assessment:
- The tendency to over-characterise technical decisions as political ones designed by ‘people who are against us’ – even when this is far from the reality. (It’s nearly always cock-up, not conspiracy)
- The tendency to see technical concepts such as normal distributions and grade boundaries and algorithms as inherently unfair. (Lots of commentators really don’t understand the concept of standards and how grades are arrived at and maintained in normal circumstances.)
- A poor understanding of the intrinsic need for moderation of different people’s judgements of standards – which has nothing whatsoever to do with trusting people. (Amazing to me that some people’s response to all this is to suggest that we need even more teacher assessment – please no!)
- A lack of concern about grade inflation – people can be quick to wave this away but, in my view, there are risks to the value of everyone’s qualifications if the level of inflation is too high. (eg If schools hadn’t been urged to keep results ‘realistic’ – A*/A % could easily have been well over 40%. Is that ok? Not if you want some sense of long-term value to any of the grades)
- The tendency to offer critique without alternatives that can be shown to be better. (Even the alternatives for the moderation approach that have since been shared with me were not, to my knowledge, discussed and circulated widely enough. They’re quite technical. Jon Coles from United Learning tweeted an excellent long thread explaining his alternative. https://twitter.com/JonColes01/status/1293233928384520192?s=20 His principle that CAGs should be used unless there was a reason not to was interesting – I wish we’d all been discussing it in April. Generating detailed solutions and getting them known about is not a strength in our community. Easier to critique and make general claims that ‘there are lots of other solutions’ without offering any specifics. )
However, I now realise I was making some assumptions about the model that didn’t play out. Mainly I assumed, just from common sense in terms of easing the pain of disappointment in a crisis recovery model, that there would be a built-in inflation much higher than we’ve seen. I’d have gone for something closer to 10%, not the 2% we’ve had at A level. I thought that introducing a controlled but generous level of inflation was obvious. Secondly, I imagined the adjustment process would entail shifting and smoothing CAGs distributions to fit a wider national pattern via some factors that had some origin in the cohort baseline and progress data as well as past school results- not forcing rank orders to shoe horn into quirky small-cohort distributions in the crude way we’ve seen. People are shocked by how crude the algorithm appears to be. For example, nothing I had imagined would allow rank order neighbours with the same CAG to be then separated by two or more grades following the adjustment based on rank order. It also did’t occur to me that people would get downgraded to U – which is just not ‘the grade below E’; it means something else altogether.
The scale of the wave of outrage right now is remarkable – driven by people who have had to counsel distressed students on the receiving end of indefensible outcomes. The technical process has been executed poorly; even if the concept is not unjust in principle – for sure it has delivered a great many injustices. To me it just looks like they weren’t prepared to get into the school-level detail enough. They stopped short once the overall patterns seems ‘close enough’. Given what’s at stake, you have to listen when so many people are this angry about something; the feelings are legitimate. That said, I do think arguments are lost when details become sloganised and people revert to cringey assessment-illiterate wishful thinking. This thread from Sam Freedman is excellent https://twitter.com/Samfr/status/1293979033458417668?s=20 The issue around disadvantage is there in the data – a small net gain overall, but slightly larger adjustment on CAGs. Making it a simple case of rich vs poor winners and losers just undermines the argument – to me.
As Laura McInerney expresses brilliantly in the Guardian, nothing Gavin Williamson has said or done has made it any better and it’s hard to understand what he’s spent the last five months doing on this issue. When it’s time for heads to roll, his will be one, that’s for sure.
With GCSEs next week – and the A level appeals issue to resolve – I would suggest that, as a minimum, various things could be done: (for me, there is zero value in doing a ‘what they should have done in March is X…’ – because, now it’s literally pointless)
- Examine and respond to the ’rounding down’ issue suggested by Alex Weatherall here: https://twitter.com/A_Weatherall/status/1294012623776817158?s=20 Let’s see what effect rounding up would have.
- Upgrade all Us to E as a minimum (callous not to)
- Remove all adjustments of -3 grades (callous not to)
- Systematically review all adjustments of -2 grades to spot obvious anomalies and cohort discontinuities.
- For GCSEs, see if a 10% inflation limit at 4+ could be established and reset the grades with that built-in – at least in the full cohort entry subjects (Others would say 5% but I’d go higher)
There’s now a party political tussle around whether to follow Scotland and just use CAGs. On one hand – it makes life simpler. It releases the pressure on appeals; lots of short-term gains for thousands of students. The question then is whether the net long-term devaluation and unstandardised nature of those grades matters enough. They’ll always be the ‘much easier 2020 grades’ – and do we care enough about that happening, mixed in with all the variation between schools? Maybe, it’s ok… given where we are and how messed up it all is. I’d favour easing the moderation over a full-blown ‘revert to CAGs’ approach. I wonder, now Labour has come out in favour of CAGs, whether the government will double down even harder and just focus on their embarrassing triple lock mantra. We shall see.
One think I’m more sure of – the ideas for a National Bacc would address a lot of these issues, putting exams into a wider context of achievements, accumulating credits over time, not just at the end.. maintaining breadth, removing the cliff edge intensity at 16… more to come on that soon.
On a lighter note… I think my Mum wrote the tweet of the week:
Thanks for this, Tom. Please also see the concerns I raise this morning at https://constantinides.net/2020/08/15/a-levels-and-gcses-in-2020/ for some further suggestions of things to be done / looked at urgently.
LikeLiked by 1 person
This is a commendable post: it’s good to see that you’ve revised your view rather than digging in. And your suggestions are pretty sensible, too.
(There were, in fact, lots of entirely reasonable objections before Thursday to Ofqual’s approach. I saw lots of careful technical analysis, and no “emotive outrage-warriors” at all. Perhaps you hang out with a particularly emotive crowd of people.)
LikeLiked by 1 person
There is a vagueness about grade inflation; why is a little bit deemed acceptable? A slower rate does not prevent the total direction, similar to the economic nonsense of perpetual growth.
Exams have always been unpredictable.
LikeLiked by 1 person
Tom, I agree with the gist of your article, and, in particular, your point that “I’d have gone for something closer to 10%, not the 2% we’ve had at A level. I thought that introducing a controlled but generous level of inflation was obvious.”
The public and the government should have had the chance to discuss questions such as, using hypothetical numbers, “Should we aim to keep grade inflation down to 2% at the risk of 25% of downgrading the wrong students, or should we allow 10% grade inflation if that reduce the risk of downgrading the wrong students down to 5%.”
Unfortunately, Ofqual has denied the public this chance. I see that as a failure for a public servant.
The is also no information in the public domain that it has provided this kind of information to the government to help the latter decide either.
LikeLiked by 1 person
A level results for my subjects at our centre vary massively every year depending upon the abilities of the students who take the subject. So centre assessed grades being compared to previous years are not valid. The only answer is a degree of trust after moderation?
This isn’t the case for GCSE, but we really must start calling them centre assessed grades, and get this into the media.
Shifting that ‘bell curve’ using some empathy for pupils should have been fairly simple, but doesn’t seem to be what happened. I wonder what proportion of entries each year do actually end up as Us? But this year that ‘non grade’ should not have been awarded to anyone. I am also disappointed and angry with the lack of flexibility of Cambridge university, where excellent pupils missing their offer by one grade not being A* are not being given their place. This is where the rank order came into effect, had their schools had a very slightly different order they would have got their place. It can be so difficult to order pupils, where was that understanding from the admissions tutors who are not even willing to listen to that reasoning.
CAGs had to be moderated, and the fact that more pupils got their first choices at uni his year and more disadvantaged pupils got a uni place is a plus, but I completely feel for those who are disappointed.
The 37% of CAGs being downgraded as a headline figure is difficult, if that meant a pupil doing 3 A levels had just one going down by one grade I could probably live with that. But it is just an average, many had no changes, which means others had much bigger losses of grades. I agree with your suggestion that a grade inflation closer to 10% for one year would have been acceptable, and not have devalued the grades achieved by the pupils.
Should the algorithm used have been different for independent schools?
I hope there is more thought and flexibility around the ‘borderline’ pupils with the GCSE grades, where a ‘pass’ grade will be key to many moving to a college or onto a particular course of study.
Given that Ofqual’s own testing data, released after A-level results, indicate that for cohort size 10-24 the final grades awarded this year has only less than 60% chance of being correct, and cohort size 25-49 the final grades awarded this year has only less than 70% chance of being correct, keeping grade inflation down to 2% doesn’t actually do very much for maintaining the integrity of grades, but it has been achieved at enormous human cost. It is arguably a misguided policy. Given that kind of inaccuracy in the final grade anyway, it would have been better to allow higher grade inflation, if not 10% then, say, 8%, to lower the human cost. But you cannot expect Williamson to work that out for himself, and Ofqual is employed to do it and advise him, but the taylors who for months have been saying to the public “We are refining the details of his clothes” have left the emperor exposed.
[…] The grades fiasco […]