(Hindsight note: I was badly wrong! They stuffed up the algorithm big-time and I totally underestimated the legitimate sense of injustice people would feel after CAG adjustments of any kind. But this is what I thought before reality happened)
In the wake of the Scottish exam results being published, Nicola Sturgeon’s apology for ‘getting it wrong’ is adding fuel to the bonfire of sanity around this year’s exam results. One of the things that winds me up is that nobody is presenting an alternative that:
- provides a more fair or more accurate outcome.
- doesn’t require a time machine to go back to March to re-do the whole thing
- doesn’t make it worse
I fear that a lot of the emotive kicking and screaming is founded on a pretty shallow understanding of how grades and standards work in national examinations and a lack of understanding of how variable teacher assessments are in the absence of rigorous moderation processes.
It’s obviously a nightmare situation: back in March, with weeks to go, we had the decision to close schools and cancel exams. (We could argue about whether that was right – but what’s the point.) Then what? How do we award qualifications with some value? The decision was to use Centre Assessed Grades, alongside rank order in lieu of exams. (No better ideas were being proposed at this time.)
The key challenge here was to take school grades and ranks to form a national data-set from which grades with some kind of parity could be awarded. One part of this is the parity from year to year – and we could debate the issue of grade inflation. But another important part is the parity between all the centres just this year. There is a lot of very woolly emotive guff out there about ‘trusting teachers’ – as if the best solution would be to take the CAGs and just give them out.
Why is that a problem?
It’s a problem because, teachers don’t just ‘know’ the standards for grades by some kind of professional wizardry. Even if teachers within a school have a good shared understanding of what they mean by a 4, a 5 , a 6, a 7 – and the A*-Es for A level – there is no way to know that these standards are shared across schools. In the absence of extensive moderation through work sampling or common assessments, there is no reference point for teachers to use to set grade boundaries in common.
Luckily we have Ofqual – a non-political technocratic organisation staffed with nerdy number crunchers. There are no dark forces here. (I’m really not interested in people thinking it’s all some conspiracy. Giant yawn.). They have data sets for previous years and baseline data that can give us some idea of the typical spread of results that schools yield, relative to the national scale that determines the grades. In the absence of any new exam data – this data allows some kind of re-alignment of all the thousands of rankings and CAG estimates. Of course it’s an approximation – of course it’s not without flaws – but it’s not a scandal. It makes sense.
In the diagram, showing three schools with comprehensive cohorts – School A has assessed grades that are close to the national spread. This matches what is typical for that school’s cohorts. They don’t need much adjustment. School B’s grades are set higher than their typical cohorts – so they need to be adjusted downwards to align with the national pattern. The rank order is intact – this references students’ efforts and work; the adjustment is needed to give the school’s overall grade pattern parity with other school’s grades. That is entirely fair in these circumstances- regardless of the level of deprivation in the area where School A and B are. (It’s the deprivation that’s unfair – not the statistical adjustments.)
School C’s grades are below the typical pattern – so, theoretically they would need an upwards adjustment. Of course there are probably almost no school Cs in reality. Why? Because teachers have a strong natural tendency to give the benefit of the doubt. Obviously. So, in the absence of school Cs, we get national grade inflation unless grades are adjusted. That might not be too big a problem if the inflation was consistent between schools. But if we want to assume students’ grades nationally are meaningful and broadly comparable, this national adjustment was necessary anyway.
(It turns out that for large cohort entries, the actual individual CAGs don’t help to set the position of the school’s scale against the national scale – the internal ranking sets the grade positions once the overall school scale has been adjusted, given a large, continuous grade spread.. Again – no scandal ; just a technical feature of the process – albeit not well-explained or publicised.)
The ‘scandal’ stories about 40% of grades needing to be adjusted….. well, it’s just not a scandal. That’s just how big a change is needed to create the parity we need to retain a set of outcomes with internal integrity. And what does this tell you about the scale of the drift that unmoderated teacher assessment creates? It’s huge. Maybe that’s the scandal – except it’s not; it’s just how assessment works. That’s why we have exams. That’s why Speaking and Listening assessments and a lot of coursework components were eventually corrupted beyond meaning.
Imagine the stats staff now dealing with Nicola Sturgeon’s political directive. What are they going to do? Bump the grades up a bit in the deprived areas to dress it up a bit? Shift everything up by 10%? Just give the teacher assessments? None of this is more fair, more objective…it’s just gerrymandering. Political.
I despair at the woeful levels of understanding of exam grading – this ludicrous obsession with ‘norm referencing’ being some kind of enemy. (And anyone who suggests ‘criterion referencing’ would be better or wants full blown teacher assessment instead of exams can’t really understand assessment – in my view. )
And still – all the emotive outrage-warriors are not offering a cogent valid alternative. Just outrage.
*Addendum: Something I forgot to add originally: My overall suggestion is that schools – and maybe even exam boards, perhaps on request – issue certificates with both CAGs and standardised grades. That could become the currency for Cohort 2020. Grades would be recognised as SA (CAG). e.g. 6 ( 7). That would be something people would put on their CVs etc. It just gives the factual information for people to interpret as they need: This is what the system gave me (this is what my school gave me).