Computer-grading of essays raises concerns

Computers have taken the place of humans in grading student essays on Ohio’s standardized tests, which the state says will save time and make scoring more equitable. But some school officials are questioning whether the practice has been properly vetted.

The Ohio Department of Education has been transitioning to automated scoring for all of its statewide exams since 2015, and the English language arts tests are the latest to make the switch.

“As you can imagine, even extensive training cannot eliminate all differences in the way humans score tests,” said department spokeswoman Brittany Halpin in an emailed statement. “Machine scoring applies the scoring rules in the exact same way across all students.”

For quality control, she said, 25 percent of essay scores are reviewed by humans.

The first time that artificial intelligence graded Ohio student essays was this past fall, for the English language arts test for third-graders.

In Columbus City Schools, nearly half of third-graders earned a zero on the writing portion, even some students who scored high enough on the rest of the test to be considered “accelerated.”

So Machelle Kline, Columbus schools’ chief accountability officer, compared this year’s results with last year’s. This past fall, 48.04 percent of thirdgraders scored a zero on their writing; the fall before, 36.3 percent received zeroes — a difference of nearly 12 percentage points.

“To see such a very large jump without changing our curriculum, that would be highly suspect,” Kline said.

Then she learned that if a student copies the wording of the question into his answer, the computer interprets that as plagiarism, earning a zero. That copying is something they’re often taught to do, she said.

Kline wasn’t the only school official who noticed the discrepancies over the past month or so: Akron schools saw more zeroes. So did Toledo, Shaker Heights and Copley-Fairlawn — all of these districts sought to have tests re-graded, and wondered whether they could get the $25 fee waived.

To date, Halpin said, the education department has received requests from 27 districts to re-score about 1,000 students’ tests. Of them, only one score was changed.

Rep. Andy Brenner, chairman of the House Education Committee, repeating that statistic, said, “If it’s an issue, I would think there would be a major re-scoring,” Brenner said. “Do we need to monitor it? Yes. But the system seems to be working as it should.”

Ohio uses standardized tests from the American Institutes for Research. Jon Cohen, AIR’s president of assessment, said at least six states are using AIR’s computer essay scoring and have been for several years. Interest has been rapidly expanding to other states, and the technology has advanced dramatically.

“Ohio, in particular, is taking a responsible approach,” Cohen said.

Cohen said that what often happens is a student copies large chunks of the provided reading passages into their essays. If a majority of the essay is cut-and-paste work, the computer doesn’t have enough original writing to grade. The technology is better at detecting that than a human grader.

“For each essay prompt, the engine is individually trained,” Cohen said. The machine is given a huge number of essays that have previously been scored by humans and then compared to a portion that have never been scored before.

Automated essay grading has had its critics over the years. That includes a former Massachusetts Institute of Technology professor, Les Perelman, who hated it so much that he and some MIT and Harvard students invented essay-writing software to trick the machines, called the Basic Automatic B.S. Essay Language Generator, or BABEL: https:// babel-generator.herokuapp. com.

Kline said she was concerned that the guinea pigs were the third-graders, for whom the stakes are higher than for other students. As part of the state’s “thirdgrade reading guarantee,” they must pass the English language arts test to be promoted to fourth grade.

Officials at the education department have offered Columbus schools additional resources and professional development to try to get those writing scores up. Kline said she welcomes the help.

“I need to know how to help the children,” she said. “We already have enough things stacked up against us.”