As most people recognise, when evaluating the degree of success in a set of GCSE outcomes we need some sense of the baseline before we start making like-for-like comparisons.   At the risk of stating the obvious, how well we’ve done all depends on what our results look like relative to the prior attainment of the students. This applies to comparing whole schools, year-on-year comparisons for one school, subjects within schools and teaching groups – especially if they are set by ability.

Given how long various data tools have existed and how commonly they are used, it still surprises me when school leaders, subject leaders and teachers refer to raw outcomes but don’t automatically engage with a banded analysis.   Using national transition matrices is a really good way to give a nuanced, stratified analysis of outcomes compared to national trends in all subjects – and I’m sure many of you will do this as a matter of routine. However,  my feeling is that all teachers would benefit from engaging with data at this level, not just have it presented to them by data managers; it’s not a mysterious process and it pays to get a feel for it.  It can be a powerful exercise to engage with in a departmental meeting where teachers look at their data collectively.

I found this excellent post from @DataEducator today, http://dataeducator.wordpress.com/2014/03/23/interacting-with-the-interactive-transition-matrices/ and that has led to messages from @KristianStill who discusses the issue on his blog: http://www.kristianstill.co.uk/wordpress/2014/03/22/u-turn-on-interactive-transition-matrices/ . Evidently ASCL has also produced a sophisticated transition matrix tool to help schools although it appears to require membership login to access it.

In this post, I am going to outline my approach. The origin of the data is the national data set mapping KS2 sublevels to GCSE outcomes.  Of course, all of the data has error attached – you need to approach this process intelligently, recognising the limits of its validity.  (Sublevels carry a data delusion health warning but we have to work with what we’re given). You are looking at an average pattern for the whole country and then lining up your very specific cohort of students against it.  It’s not an absolute indicator – it is simply a tool to identify possible issues or successes based on the trend observed when one bell curve is lined up against another, with five years in between.

Transition Matrices for 40 subjects are now provided on the OfSTED RaiseOnline site:

At first glance it can be overwhelming but if you select your particular subject and start to think about your own cohort or even your own class it starts to make sense. I have chosen Geography to illustrate the process.

The national data maps average KS2 sub-levels in English, Maths and Science to GCSE grades for all students who took Geography.  The details fluctuate from year to year but not by much – because of the bell-curve effect.  This information can be used to work out how well your Geography students have done compared to the national trend for all Geography students, based on their prior attainment at KS2.

I have produced this spreadsheet to help with the analysis.  It focuses on two outcomes that I am interested in: grades A*-C and grades A*-A.  As we move away from A*-C thinking, with P8 and A8 on the way, it’s really important to consider a range of outcomes.

You can download it here to use as you wish: GCSE TM Analysis Template

The only work that needs to be done extracting the data subject-by-subject is to add up the percentages for the outcomes you want to focus on. I am interested in A*-C but also A-A* so I have compiled that data from the master matrix. That allows me to look at top end achievement from all groups alongside core achievement. The spreadsheet calculates expected outcomes – essentially the grades expected if every student performed in line with national trends. I deliberately keep in the decimal points to highlight the fact that this is a calculated outcome – you don’t get 4.8 students of course.

Once the national data is in place for a given subject, you can enter your school data, for a class or whole cohort.  By comparing actual with expected grades overall and for each band you get a measure of how well a class or cohort has done relative to the national trend. There’s no use getting excited about lots of As and A*s if that is simply what was expected. Similarly, this method shows up any major achievements with lower prior attaining intakes where the raw outcomes may be less dazzling at first glance. Obviously not everyone can be above average but any outcomes above what is expected is a sign of adding value.   At the same time, you need to be  careful not to overload expectations. If only 4% of a 4b KS2 band gets an A/A* at GCSE it means it is possible but relatively rare. Leaders need to be careful not to turn this into a false expectation. (Let’s be intelligent and realistic as well as aspirational.)

I’ve always found that this analysis is helpful in getting an idea of the structure of a set of outcomes, way in advance of the publication of the RaiseOnline analysis.  It shows you if you’re doing well with the top end, the bottom end or simply keeping in line with the overall national trend – which is no disgrace.  It’s important to be careful with small cohorts; the patterns are much more meaningful and secure with larger numbers, although it’s still worth having a look to see what the analysis shows.

As well as undertaking a retrospective analysis, arguably it is even more important as a tool for setting expectations for current groups.

In this example I have collated the data for a hypothetical class of 28 Geography students.  It suggests that 15 students should be gaining at least a C and 4 should be gaining A/A*, on average.  Any teacher ought to be able to do this for their own class.  It helps to put projected outcomes into perspective, compared with national trends. It could serve as a way to set minimum goals for a class.  Of course, there is the possibility that many more top grades could be achieved so it’s important for this analysis not to ever limit expectations; it’s simply a way to establish the national trend line as it applies to a specific group.

If you are interested in looking at whole-school outcomes in terms of 5A*-C EM, while that remains the key performance measure, there is a transition matrix for that too:

You can use this data to calculate the expected 5A*-C inc EM for your school either in a predictive way or retrospectively.  After a period of turmoil, as we’ve had this summer, the exact details may vary a little more so it may be worth waiting for the 2014 data to be published. However, that can take many months so the 2013 data could be used to give a reasonable rough estimate.

I’m going to suggest that we explore some of this thinking at my school.  Once teachers have met their classes and can reference prior attainment data against their own assessments, an exercise like this can be very useful to prompt discussion about potential outcomes across the ability range within any given year group or teaching group.  When levels finally disappear from KS2 tests, new transition matrices will emerge using whatever assessment outcomes are generated by the new tests. The same principles will apply.

UPDATE December 2014:  The new matrices are now available:

https://www.raiseonline.org/documentlibrary/ViewDocumentLibrary.aspx

The 2014 TM for 5A*-CEM is much more generous than 2013 – at least it is for my school.  It may be worth recalibrating your analysis.