How do we make judgments about how well our schooling system is performing? Domestically, the most common methods use NCEA and National Standards results. In addition, the Education Review Office(ERO) reports on school performance. For international benchmarking the current favourite method is the Programme for International Testing (PISA).

All of these methods are problematic and have significant negative and unintended consequences. Together they also represent a largely quantitative paradigm of thinking about, and measuring school performance which needs to be challenged.

NCEA data

Our Minister of Education has set a national pass rate target for NCEA at Level 2 as a measure of the performance of the schooling system. It is entirely possible that she will achieve her target since NCEA results are tracking upwards, with some schools achieving spectacular improvements in pass rates.

But we should be clear that many schools, particularly lower decile schools, are now using more internal assessments (which have higher pass rates than external assessment) and developing more vocational programmes (which provide more options for less academic students), and it is these changes in programmes and assessment techniques which are the main drivers of improved NCEA pass rates.

Although this is good news, we cannot easily use these data to argue that the performance of the New Zealand school system has improved over time since we are using different tools (more internal assessment and more vocationally based assessment), to measure performance.

For the same reason, comparing the NCEA performance of individual schools which have different policies regarding the amount of internal assessment to be used, and the range of vocational programmes to be provided, is also problematic.

The increased focus on achieving better NCEA pass rates has also created an assessment driven curriculum. This has resulted in schools ignoring large parts of the New Zealand Curriculum which are not easily assessable -though critically important to improving learning. The longer term consequences of this could be dire.

National Standards

We now have national reading, writing and mathematics standards for students in years 1-8, and use them to measure performance in these learning areas across the country and in individual schools. To avoid the narrowing effects of “pen and paper” national testing, teacher professional judgement is seen to be critically important in deciding whether a student has met the standard.

The problem of course is that judgements made by thousands of individual teachers working in a target based environment, in the absence of a common national test (with all its considerable drawbacks), are inconsistent and even “ropey”. Given this, they are of limited value if they are to be used to reliably and validly measure performance of the system or individual schools over time.

And it gets worse: the focus on measuring performance in reading, writing and mathematics in primary schools also has negative effects on the attention schools give to other important curriculum areas (in the same way as has happened with the NCEA), which of course go unmeasured and unreported.

Education Review Office (ERO) reports

ERO regularly reviews various parts of the New Zealand education system and individual schools. It has made some progress in reviewing schools in a more nuanced manner. For example, leadership, curriculum delivery, teaching quality and ability to self review are all reported on.

However, hard achievement data remains central to ERO’s methodology. All these “inputs” are considered important only in that they contribute to improved learning and student achievement “outputs”.

The problem, as we have noted above, is that since ERO has limited means of measuring student learning achievement “outputs” reliably and validly, as it is supposed to do, its findings are also of limited value.

Programme for International Student Assessment (PISA)

The PISA is the most commonly used international measure of performance. It tests 500,000 students across 65 OECD counties with a focus on reading, maths and science. The latest 2012 results point to a long term decline in New Zealand’s educational performance. Fixing this has become a central part of education policy development in New Zealand.

However, the PISA “horse race” approach to measuring performance is questionable to say the least:

• It uses “paper and pen” and some computer based testing only, which necessarily reflects a narrow focus for measuring educational performance. As Hugh Morrison comments: “There are very few things you can summarise with a number and yet Pisa claims to be able to capture a country’s entire education system in just three of them.” (Reading, Mathematics and Science) “It can’t be possible. It is madness.”

• It is not based on any common national or international curriculum, so what is taught and the way it is taught is not necessarily captured in the testing process.

• It may be culturally biased. India pulled out of the testing in 2009 claiming that the questions were particularly difficult for its rural students.

• Its methodology is suspect. For example, results for significant numbers of students who have not actually sat particular tests are created as if they had sat these tests by using a (well established) scaling methodology inappropriately.

The way forward

If our politicians require us to maintain this quantitative measurement-based paradigm, we must try to do the job better! This probably means we will need to construct a proper Value Added Measure (VAM) of school performance which would address some of the problems related to NCEA and National Standards achievement data discussed above. This will be a technically daunting task to say the least, and may create many negative unintended consequences. It may however represent an honest attempt to address a real issue.

A far more radical and attractive option would be to accept that our current quantitative measurement paradigm, even it includes a VAM, will never work well in an educational setting. Such an acceptance will encourage us to develop a new more nuanced and complex paradigm of school performance which requires us to:

• Understand that home background and socio-economic status are huge contributors to the school experience for most children, and outside the control of schools
• Stop publishing NCEA and National Standards results (since they are less than robust as performance measures), and encourage all teachers to deliver to the intent of the New Zealand Curriculum.
• Focus more on school “inputs”– particularly the recruitment, development and retention of high quality teachers and school leaders, based on professional judgements made by experienced practitioners.
• Establish better mechanisms through which teachers and leaders can collaborate to solve learning problems, instead of compete for better results.
• Establish strong regional and school based support structures for schools through which struggling schools and teachers are identified, supported and counselled by experienced practitioners. Such a process could look at a comprehensive range of qualitative and quantitative indicators such as community feedback, staff morale, quality of leadership, staff turnover, retention rates, truancy patterns, as well as NCEA and National Standards data.
• Focus on monitoring the effectiveness of the schooling system as a whole, and not individual schools. This would mean developing a sampling assessment process across the country in all curriculum areas which focuses on the explicit requirements of the New Zealand Curriculum through to year 13, and not on the NCEA or National Standards. Such a process (building on work already done through Otago University) will require a range of assessment methods and will not be attributable to particular schools. Its purpose would be to help make judgements on the performance of the system as a whole.
• Place far less emphasis on PISA testing results for national educational policy development

Conclusion

Current performance measures of schools and the schooling system are not useful, and dangerous, particularly because politicians and many school leaders are basing their decision making on the data they produce.

We need to change the paradigm.