Capping Ambition – The Return to Norm Referencing

One of the side effects of the English grading fiasco has been to bring to wider attention Ofqual’s doctrine of comparable outcomes. In effect this is a slightly more sophisticated return to norm referenced exams – in other words we can expect to see roughly the same exam results every year.

In Ofqual’s own words, written before this year’s disaster:

“We want comparable outcomes when:

  • the cohort for the subject is similar, in terms of ability, to previous years
  • the syllabus and the exams and other assessments are fit for purpose
  • the purpose, requirements and nature of the qualification is the same
  • there has been no substantial improvement (or drop) in teaching and learning at a national level
  • where previous grade standards were appropriate. “

The emphasis is very clearly on stability in results. All the ministerial rhetoric about maintaining standards makes it clear that this is the brief Ofqual have been given.

The implications of this change are far reaching and have had very little public debate as yet. Ofqual itself has set out some of the issues that arise in a letter to Michael Gove. Perhaps the most breathtaking statement in the letter is that

For future years, we will explore whether there is scope to develop the approach so that genuine increases in performance can be more easily demonstrated, where there is evidence for that.”

So it would seem that Ofqual is admitting here that it has no adequate way of telling whether overall students’ performance has improved, stayed the same or (presumably) got worse. So it would seem that there is really nothing we can say about how standards have changed because regulators don’t know how to measure it. Can this really be true?

The implications for schools and for the whole school improvement industry are equally dramatic. Gove has decided to increase the floor target at GCSE this year to 40% and ultimately to 50%. But if results are going to stay the same overall, how can this possibly be achieved? If one school’s results go up, someone else’s will have to go down. If the average of 5 A to C with English and maths stays in the high 50’s, it is arithmetically obvious that a great many schools will be below a 50% floor target.

Or as Ofqual has told Michael Gove:

“Our approach means that whilst some schools will see improvement in their exam results, due to comparable outcomes the overall results will not show significant increases. So, it will be difficult to secure system-level improvements in exam results which you have said you want to see.”

We await a response.

So what is to be done? Obviously the first thing will be to make them all academies. But simple maths says that this won’t make any difference. The whole premise of the last 20 years school improvement activity has been that low performing schools can catch up with the rest. If the examiners are going to make that impossible, where do we go from there?

More importantly though, we – that is pupils, teachers, employers, colleges – will have real problems knowing how to interpret results. We will have a league table of pupils – we’ll know who did best and who didn’t. That of course is one purpose of exams and is not to be dismissed. But will we still know what any given grade really means? A college or an employer won’t really know what an applicant can knows and can do – just where they come in the pecking order.

We got rid of norm referencing because it was seen to be a cap on ambition. As Keith Joseph said when GCSE was introduced , it “would be fairer because pupils would be judged by what they could do and not how they compared to someone else.”

The objective set by Keith Joseph was to get the majority of pupils up to the standard then achieved by only a minority. That still needs to be our aim. But we have now fallen into a situation where we’ve effectively capped the number getting Grade C and have no way of knowing whether or not we are getting nearer to the objective we set all those years ago.

To get out of this muddle is going to require some hard work and a degree of intellectual honesty that has been sadly lacking over many years. We have to find ways of knowing how well pupils and schools are actually doing against real criteria as well as against each other. This will require a much more sophisticated approach to assessment than – reverting to simplistic end of course exams are unlikely to do the job.

Advertisements

4 Comments on “Capping Ambition – The Return to Norm Referencing”

  1. David Pavett says:

    Sections in quotes are from the above article.

    “One of the side effects of the English grading fiasco has been to bring to wider attention Ofqual’s doctrine of comparable outcomes.”

    A study of examination systems by Harvey Goldstein in the 1980s (if I remember rightly) showed that this was the general practice of the examination boards. He also showed that most systems combined both criterion and norm referencing. It is not an innovation by Ofqual. There was not a golden past where this was not happening.

    Nevertheless I agree that the Ofqual statement makes the norm-referencing aspect very explicit, in particular in its reference to comparability with “previous years”. Of course this is a criterion that has not been applied in that there has been a year on year improvement in results. But then this is not an area of public discourse where one should expect much in the way of consistency.

    “So it would seem that there is really nothing we can say about how standards have changed because regulators don’t know how to measure it. Can this really be true?”

    To be fair to Ofqual I think that we should admit that this is not always an easy matter. Syllabuses change as do the style of question papers. How then do you compare? It is not simple.

    But, that said, it is outrageous that a system of general education should be thought of on competitive/selective lines. This means that kids are being told that it is not enough to reach a certain standard. If others do better than this can determine that you will fail. It is a very nasty concept of general educational qualifications – although not one that we should kid ourselves is entirely new.

    The important thing is to set standards and the criteria by which to judge them so that these are understood by all. The issue of year by year consistency is a makeshift solution. It does not even allow for general national improvement.

    “The whole premise of the last 20 years school improvement activity has been that low performing schools can catch up with the rest. If the examiners are going to make that impossible, where do we go from there?”

    Good point.

    “We got rid of norm referencing because it was seen to be a cap on ambition.”

    My question is “Did we?”. What is the evidence for that. Examination boards work in mysterious ways, most of which are not open to easy public examination. Are we really so sure that we got rid of norm referencing?

    The objective set by Keith Joseph was to get the majority of pupils up to the standard then achieved by only a minority.

    It is both startling and very sad that we should be referring to Keith Joseph (the “mad monk” of Thatcherism) as the first education minister to talk about objective standards. It is nevertheless true and does not say a lot for Labour’s educational leaders.

    “To get out of this muddle is going to require some hard work and a degree of intellectual honesty that has been sadly lacking over many years.”

    We do indeed. Unfortunately on current showing the likelihood of getting that from any of our major political parties is very slim indeed.

  2. Richard Pring says:

    Dear John

    Thank you for this and every other of yur excellent pieces. I treasure them.

    But was not one solution to comparability of standards over time that proposed by the Assessment of Performance Unit, based on the NAEP work in the USA? (I was sent by the gov’t of the time to Denver , Colorado, to see the work of NAEP and my report is buried somewhere in the bowels of the Department. The APU was ended by Baker despite its excellent work based on research.

    You may be interested in the chapter on testig and exam in my book just published which I have attached

    Best wishes

    Richard

    ________________________________

    • trevor fisher says:

      what is now needed is a Royal Commission into the exam system. I actually agreed with Keith Joseph, we needed criterion referencing. There is no problem in exams being comparable year on year, it is the performance that cannot be comparable. If every year the same standards are attained this means that there is a diversion from reality.

      Norm referencing was indeed got rid of. That is why pass rates started climbing from the middle 1980s, more so with A Level than GCSE. The question is whether this meant real achievement was being attained.

      If Gove wants to keep the pass rate constant this would indeed be a return to norm referencing, but this is not clear. He wants the percentage achieving grade C and above at GCSE to go up, but that is not the same thing,.

      A Royal Commission would focus the debate and allow us all to contribute. WIthout it, we have ad hoc initiatives that make no difference. And there is only one exam system, not GCSE and A LEvel. Not a lot of people know that any more.

      Time to tell them

      trevor fisher.

  3. David Pavett says:

    There are a number of issues here. One is norm versus criterion referencing. I share the view that, insofar as it can be achieved, the latter is what we should use in general education.

    But a second issue is whether there was a time in which norm referencing was actually replaced by criterion referencing. John and Trevor assert that this was so but I would like to know what the evidence for that is. It is not enough to cite rising performance levels since that could have other causes (e.g. political and commercial). Did grade boundaries cease to be changed in the period when norm referencing disappeared? If so, on what basis were they changed?

    I am not well informed about this subject so this is a genuine request for information. As I said in my earlier post, I remember reading a study by Harvey Goldstein a long time ago which convinced me that it was much more difficult to separate norm and criterion referencing than is normally imagined. He also showed that systems that claimed to be one or the other were rarely, if ever, so.

    So, what is the evidence that we had a period in which norm referencing ceased to operate?