In my recent talk at ResearchEd in Birmingham (and more recently at Blackpool) , I explored some ideas about assessment and critiqued various responses to the challenge of getting assessment right.  Sensibly enough, given all the limitations and flaws in grading systems, lots of schools are trying to make their assessment systems meaningful, rooted in the detail of what our students actually know, not simply relying on macro descriptors.

I have seen a lot of schools move towards an approach based on ‘mastery statements’ or ‘can do’ statements.  They sound sensible, look reasonable on paper and certainly offer a plausible solution to one of our central challenges:  how do we capture and track meaningful assessment information that helps students and teachers to know where they are and where to go next?

😊😐😟

In theory, a mastery statement, expressed in studentspeak as an ‘I can’ statement, does the job of looking at the detail, free from arbitrary grade descriptors.  However, in reality  I think this approach is a big mistake for three reasons:

• The illusion of absolutes.
• Implausible scale of meaningful tracking.
• Persistent creation of false numerical scales.

In my talk I use some images of Can Do statements that I found via a google search.  However, I have no desire to focus my critique on specific schools – everyone involved means well and is trying to come up with solutions.  So, let me just share some of the statements I’ve found – they are typical of many others.

Example 1: Computing.

Each of these statements is placed in a grid with tick box where students can self-assess using the self-reflection emoji scale – that staple of reliable school assessment! 😊😐😟

I can explain the term ‘computer system’.  This sounds ok surely? Well, not really.  It depends on what the requirements are for the explanation.  How much depth is required? For example, is this a one mark or four mark answer? We can easily imagine a range of answers some of which would be more detailed and accurate than others.  This statement doesn’t allow anyone, teacher or student, to gauge how good their explanation is or what knowledge they have. And how does 😊😐😟 help?

I know at least three functions of the CPU. This is more like it.  It is precise – three functions.  But, even here this is a heap of ambiguity. Surely some functions are more important to know than others?  Which functions are known and which are not? Hmmm  😊😐😟

I can explain the importance of ethical, environmental and legal considerations when creating computer systems.  Note: this is on the same list as the statements above – ie given the exact same status.  You could probably find a PhD thesis on this topic.   It’s not too cynical to suggest that a simple 😊😐😟 evaluation isn’t really going to get to the heart of our assessment on this rather complex compound Can Do statement; but there it is.

To fully assess this, you would need to state the specific components of the explanation required to demonstrate understanding of the issues at GCSE level and then test whether each students knew them as isolated facts and could then assemble them into a coherent explanation – orally or in writing.  That would require some form of assessment process – probably generating a mark of some kind.  A student’s mark compared to the range of marks scored by the class would tell you more about the degree of ‘can do’ they’ve mastered, on a specific assessment.  Do you need to record it?  Not really – not for everything.  The record is there on the test, in their book, on their assessment.

Example 2: Science 

The example I found is basically a spreadsheet with multiple tabs for each science topic.  There are literally hundreds of statements.   Here are some examples.

I can state that sound is produced by vibrations.  Straightforward enough? Not really. How meaningful is it exactly?  Imagine your student stating “Sound is produced by vibrations”. There you go. Tick. Smiley face.  What does it mean? Do they understand it?  Can they use this information in a context? What about stating ‘Quantum theory introduces probabilistic elements to our understanding of nature’?  I’m not convinced that recording that you can state anything amounts to knowing it.  It’s just such a weird way to record a student’s knowledge.

I can state that sound cannot travel in vacuum.  Imagine a tick against every child’s name on your giant tracking sheet except maybe Michael, so you ask him.   ‘Michael – can sound travel in a vacuum?’ No. ‘Well done, be sure to tick your Can Do statement’. Full house for me.  Move on.  Boy… is this laborious!?  And that’s just one of the hundreds of statements….. And we still don’t really know if Michael understands the nature of sound waves.

In both of these examples, it seems to be that taking an entire syllabus and turning all the factoids into Can Do statements is a false move.  There are already plenty of resources that capture the content – revision guides, the specification itself, text books.  They are packed with content. Why not simply tick and cross these if you want a checklist.  But to add ‘I can’ at the start of a sentence really doesn’t tell you enough to be worth the massive time investment even if it tells you anything at all.

Example 3: History 

I’ve found an example of Can Do statements appearing in a document used to track progress at GCSE against some very generic descriptors relating to London:

• I can demonstrate that I am developing knowledge of London across the time period.  (KS4 Target Direction 4)
• I can demonstrate secure knowledge of London across the time period. (KS4 Target Direction 6)
• I can demonstrate advanced knowledge of London across the time period (KS4 Target Direction 8/9)

How helpful is this? How would any typical GCSE student know if they are ‘developing knowledge’, have ‘secure knowledge’ or ‘advanced knowledge’?  I find myself screaming – Which Knowledge??  Of course, the exact knowledge required to constitute advanced, secure or developing will be stated elsewhere  – so this Can Do exercise is weirdly, unhelpfully removed from anything that informs the assessment.  And as for the link to GCSE grades – that seems monumentally simplistic and vague.  Knowing more = doing better.  Who knew?

Sure enough, this example includes advice for improvement. If you are at the level called Developing, here’s the advice given: You need to add detail; you need to extend your analysis.  Again, this just doesn’t take anyone forward.  In the absence of any actual information about specific knowledge elements, it’s just saying ‘to do better, you need to do better’.

Example 4: Massive Mastery Statement Banks. 

There are lots of systems being developed in the marketplace where various concepts such as flightpaths and minimum target grades are used.  However, in a post-levels world, they are now looking towards statement banks to generate the data.  In some systems I’ve seen, teachers evaluate each statement for every student against a multi-point scale such as Emerging, Developing, Secure, Mastered.  I don’t know what criteria are used by teachers to judge which applies but you can imagine the level of moderation needed to get any kind of consensus on any given statement.  But then multiply that up by hundreds – because the lists of statements are massive.

Here’s some examples from the Art curriculum at KS3:

• Pupils should be taught to use a range of techniques to record their observations in sketchbooks as a basis for exploring their ideas.  
• Pupils should be taught to increase their proficiency in the handling of different materials.

Hmmmm.  😊😐😟 ??  Isn’t the first one just a case of getting students to use a sketchbook? That seems to be more about the teacher than the students.  And how do you condense ‘proficiency in handling different materials’ into a selection of ‘developing, secure’ or whatever.?  The teacher evaluation will be so subjective.

And yet…here’s where I really lose the plot with this….. these systems turns the average of the subjective Emerging, Developing, Secure, Mastered evaluations into a number for each student.  Sometimes the numbers are in the form 4.47, 3.6, 5.92….  Yes, THREE SIG FIGS!! I kid you not.  We’ve taken a bunch of Can Do statements, applied the teachers’ “hmmm, roughly, kind of, probably, let’s say Secure” guesstimates and averaged it up to 4.47?  This will look lovely on the system; there will be graphs.  Coloured graphs.  But it’s just so majorly spurious. It’s data porn – seductive, maybe,  but so so wrong.

And still, even if we swallow the Can Do data pill, we have the question of ‘so what’?

And that was the other part of my talk:

We know Amy is struggling. The question is what we do about it.