Today, ACARA has released results of research which provides additional evidence that automated essay scoring is a viable solution for marking NAPLAN online writing tasks.
"The research, which began in 2012, has found four separate and independent automated essay scoring systems were able to mark NAPLAN persuasive writing tasks as reliably as human markers.ACARA CEO, Robert Randall, welcomed the findings, which form a part of a comprehensive research and development program designed to prepare for the transition to NAPLAN online.
“Automated essay scoring of the writing component of NAPLAN will result in parents and teachers receiving their children and students’ results within two weeks of taking NAPLAN,” Mr Randall said.
“The precision and earlier provision of the results will help teachers tailor their teaching to student needs.
“Teachers and other markers will continue to be involved in the process – by training the automated essay scoring system (via 1,000 human scored essays), marking a sample of essays to check and validate what the system is doing; and by marking essays that the system might have difficulty marking (because the essays are different from what the system has been trained to mark).
“The research results show that automated essay scoring works for NAPLAN-type writing, but we will continue with our research to refine the system and to gather more evidence, which we will use to assure parents and teachers of the viability of automated essay scoring and to make a final decision about proceeding. If need be, we could double mark samples of student essays, until everyone is comfortable with automated essay scoring.”
Four separate and experienced vendors were engaged to score a sample of NAPLAN persuasive essays in 2012, using the current NAPLAN writing rubric. The vendors represented a cross-section of different approaches and methods for automated assessment of writing. The vendors were provided with 1,014 essays, along with scores provided by human markers, to train and validate their automated essay scoring systems. After training and validating the systems, the vendors used them to mark 339 tests. On overall scores and each writing criteria assessed, the four automated essay scoring systems achieved levels of agreement comparable with the human markers.
“What is exciting about this research is that, although the four vendors had different automated essay scoring systems, they were all able to mark the essays as well as the human markers,” Mr Randall said.
“This is not the end of the research into automated essay scoring,” Mr Randall explained. “We intend to expand on this research in 2016 to include a larger sample of students and multiple prompts within and across writing genres (persuasive and narrative) before making a final decision about the approach to be used in 2017.”
View our infographic that illustrates the automated essay scoring research (PDF 1.3 kb).
About NAPLAN online
Federal, state and territory education ministers have agreed that NAPLAN will move online from 2017, over a two–three year period. NAPLAN online will provide better assessment, more precise results and faster turnaround of information. Significant planning, development, research and trialling are going on behind the scenes to make sure we are all ready to move NAPLAN online."http://www.acara.edu.au/news_media/acara_news/acara_news_2015_11.html#20151130
