The Independent Review of Qualifications and Assessment will shape Scottish secondary schooling for decades to come. The Review was Commissioned by the Scottish government last summer, and reported last week. The Review’s chair, Professor Louise Hayward, has ensured its durability by embedding its conclusions in a complex network of consultations with all the influential interest groups in Scottish education. Few politicians would dare to challenge the might of this Scottish educational establishment.
Yet the Review ought to be challenged, rigorously and radically, because it is deeply disappointing. Its methods were flawed, and its recommendations vapid. It has a few good ideas, but they are not worked out in any detail and their practicability is doubtful. Implementing what it proposes would perpetuate the harm already inflicted by the implementation of Curriculum for Excellence, that two-decade-old reform which the present Review extols as admirable.
First, the methods. Previous formal reviews of assessment have started with systematic evidence of how the current system was working. In the past, that has required voluminous statistical investigation of attainment and progression. It has included specially commissioned research on social inequalities in these measures. Sometimes, as in the 1990s, it has also included statistical evaluation of the educational standards that are embodied in the current system, and whether these have declined since the previous reform. All of these rigorous methods were adopted by the reforms which led to the current Highers in 2000, to Standard Grade in the 1980s, to new vocational qualifications at that same time, and to Ordinary Grade in 1962.
Yet here we have nothing of the kind. Behind this review, there seems to have been no actual research into the status quo. The report’s account of what it calls the ‘current context’ is brief and superficial, one page out of 93 pages of main text. The research that is cited mostly relates to various mission statements and similar exhortatory exercises by campaigning groups and international organisations. The research cited specifically on Scottish education comes from members of the Review’s advisory group (which is fair enough), but with very little beyond that. Its two main research sources on Scotland are the deeply flawed OECD review of 2021, and a dismayingly ill-informed report on the history of Scottish assessment which the OECD also commissioned in that year.
The present review did, however, stimulate lots of consultation, the results of which it presents in support of almost all its recommendations. Consultation ought indeed to be part of the evidence-gathering that might inform recommendations. Embedding recommendations in what the system would be willing to endorse is a wise way of ensuring that a Review’s conclusions will lead to action. But the opinions expressed in a consultation are not themselves evidence of a need for change. Nor are they a reliable map for reform. These opinions are valuable as guides to policy only if they, in turn, are based on systematic analysis of what is wrong and of what other options might work.
Undoubtedly some of the consulted views will indeed have been informed in these necessary ways. But the present Review does not report on that rational basis, quoting only the summary conclusions of the people and organisations consulted. In effect, then, the main body of evidence cited by this Review is a series of large focus groups. It is as if a review of diagnostic procedures in the NHS were to be based primarily on the opinions of clinicians and patients. Important though these are, they are not the same – or as important – as properly scientific evaluation of what works.
It is then not very surprising to find that, on its main topics, the Review is not convincing, retreating too often into vague aspiration. For example, it notes that not all subjects in the curriculum are amenable to the same kinds of assessment. That is true, though trite. What is needed is some detailed analysis of this point. Before getting to that, there would have to be an honest appraisal of the strengths and weaknesses of all kinds of assessment – essays, projects, investigations, invigilated examinations. Yet there is no such discussion here. Everything except exams is presented as unproblematically desirable. Exams are damned with very faint praise, as being sometimes appropriate, but are mostly condemned because existing exams in Scotland encourage rote learning. They certainly do, which is why they badly need to be reformed, but there is no advice in this Review on what exams might be good at, nor, therefore, on how they might be designed to be more educationally effective.
One technical way of putting this point is to say that the Review pays no more than cursory attention to the two ideas which have been central – in all countries – to the development of assessment for a century: validity and reliability. On the latter, there is nothing at all. So there is no consideration in the Review of whether its preferred modes of assessment might give only transient results which would be different if they were given at a different time, or on a different topic. Validity is at least mentioned, though there is no explanation of why, for example, an essay might give a more valid indication of a student’s understanding of a complex subject than a short exam. The Review thus comes across as hectoring – asserting that essays and so on are effective, not arguing the case in the careful detail that is required.
Perhaps most surprising of all, in connection with validity, is the almost complete absence from the Review of any analysis of social inequalities of opportunity, progression and attainment. This is really rather shocking. There is a large body of evidence on this, some of the most impressive indeed produced by the few among the Review’s consultees who do this kind of thing. Yet none of it is cited, not even the impact on inequality which was caused by the Covid disruption (which the Review mentions briefly). The sections on artificial intelligence tell us nothing new, and barely even acknowledge that it is likely to become a new dimension of inequality because access to the best equipment will be rationed by affordability. On the inequality of home circumstances more generally, the Review rests entirely on faith in teachers’ being able to spot when a pupil’s work is not their own. In the face of the research evidence that such inequalities abound, this is rather naïve.
All these weaknesses of the Review’s methods then vitiate most of its proposals. The headline-grabbing proposal to end invigilated exams in school fourth year, and to reduce their role in later years, suffers from the absence of prior attention to the role of exams in maintaining standards. It also suffers from the Review’s neglect of the biases that inevitably arise when any of us marks our own students’ work.
The flagship of the Review’s proposals is a Scottish Diploma of Achievement, which would be based on assessed coursework, on an assessed project, and on ‘personal pathways’. The last of these would include a wide range of extra-curricular activities that, while commendable, do not lend themselves to assessment or even to anything other than bland summary, like the mostly uninformative headteachers’ reports that form part of students’ application to university.
The need for reform to the assessment of individual courses is where this Review had its origins, but it provides nothing other than a reiteration of that starting point. Its gives no guidance on how the balance of exams and non-exam assessment might be specified in different subjects. That vagueness is an inevitable consequence of the Review’s methods, because of the absence of any systematic analysis of what is wrong with present assessment, nor of the strengths and weaknesses of all feasible forms of assessment. To be told ‘where appropriate, retain external examination’ is to be told nothing at all in the absence of guidance on how to judge appropriateness. (This slogan also illustrates a recurrent problem in this badly written Review – the conflation of external assessment with exams, and school-based assessment with other kinds of assessment.) So the work of revising the assessment of individual subjects is no further forward.
The Review’s idea that each student would work on an individually chosen project, which might span several years, is the most interesting suggestion in the whole report. Bringing together disparate bodies of learning could be very effective. But, to work, far more attention than is given here would have to be devoted to how the different levels of the various contributing subjects might be made to work harmoniously. This requires a sequencing of subjects that might be unique to each student – in principle desirable, but in practice a nightmare for a school to manage. Moreover, the Review does not even contemplate the possibility that some students might thrive better doing a specialised project in a particular subject than combining their learning from several. That possibility of specialising is, after all, one of the reasons why students choose particular Advanced Highers.
This Review thus raises far more questions than it answers. Perhaps that’s good, and it certainly offers plenty of opportunities for policy-makers to cherry-pick the bits of the report that they find most congenial. But what can certainly be said is that the Review takes us no closer to a system of assessment that is educationally sound.
Lindsay Paterson is emeritus professor of education policy at Edinburgh University.