Did DPI ‘rejigger’ school scores?
Formulas, goal posts altered amid pandemic
After two years of pandemic schooling with few measures of how students were performing, the release of state assessments of Wisconsin schools in November was to be the first comprehensive picture of academic slide.
Surprisingly, despite standardized test scores dropping statewide, the state Department of Public Instruction gave passing marks to the same percentage of schools as it did before the pandemic in 2019.
That raised some red flags.
DPI officials had skipped producing their report cards of schools in 2020 and had wanted to do the same in 2021, as the pandemic caused many students to miss standardized tests and complicated metrics. Lawmakers didn’t grant the exemption for 2021.
When DPI released the report cards, they cautioned against making any comparisons between the years, as they’d changed the formula for calculating scores, along with the goal posts for each performance category. They did not take questions on a press call at the time.
In response, Republican lawmakers accused DPI of obscuring the impact of the pandemic on academic progress.
“We certainly should have never rejiggered the test scores to make it look like we did better because we can’t admit our failure at making a bad decision in the pandemic,” Assembly Speaker Robin Vos, R-Rochester, said during a legislative hearing.
Republican lawmakers are now pushing a bill that would require DPI to undo their changes to the formula and prevent DPI from making any changes without legislative approval. It has already passed the Assembly.
The Journal Sentinel spoke with DPI officials — including Tom McCarthy, executive director of the Office of the Superintendent; Sam Bohrod, assistant director of DPI’s Office of Educational Accountability that produces the report cards; and Patrick Chambers, a consultant for that office — about what really changed.
Who changed the report card formulas?
Discussions about changing the formula started in 2019 under the previous state superintendent, Carolyn Stanford Taylor. She was appointed by Gov. Tony Evers when he left the role to take the governorship, and she did not run for re-election.
By the time Jill Underly was elected to the superintendent position in 2021, it was functionally too late to change course, DPI officials said, while also noting that they trusted the process that had taken place.
The changes were determined by DPI after 10 meetings throughout 2020 with a group of 33 stakeholders, including representatives from 24 school districts, four choice school operators, a charter school authorizer and several Cooperative Educational Service Agencies.
Why did they make changes?
The report cards produced by DPI have graded schools on four areas: achievement on standardized tests; growth in performance on those tests over time; attendance and graduation; and the performance of students with the most challenges.
DPI officials said stakeholders were frustrated in particular by that last area of the report cards. In 2019, that area focused on “closing gaps” for students facing challenges based on race, ethnicity, income, disability and learning English.
The problem, DPI officials said, was that the category was highly erratic from year to year. Demographic groups of students were only considered as part of the category if there were at least 20 students in that group at the school.
For example, if a school had 20 English language learners one year, and 19 the next year, those students were bumped out of the category. If those students were particularly high performers, the school’s “closing gaps” score could suffer dramatically by losing that group.
As a result, schools could sometimes see their scores zoom up or down in a given year based not on performance but on a change in the size of a certain demographic group, DPI officials said.
To address the volatility, DPI officials said, they decided to remove the focus on those demographic groups and instead focus on a quartile of the school that had scored the lowest on standardized tests the previous year.
Was anything else changed?
DPI also changed another of the four areas of the report card: the one focused on attendance and graduation.
Previously, schools received a hardline five-point deduction if at least 13% of students were chronically absent or if the dropout rate was at least 6%. If schools were slightly below the threshold, their scores were not affected. If they were far over it, their scores weren’t affected beyond those five points.
School representatives said they wanted a more fluid measurement that would more closely measure their dropout and absentee rates, DPI officials said.
So, the five-point deduction was tossed. Instead, the exact rate of chronic absenteeism is factored into the overall score for the area of attendance and graduation. Depending on the rates, schools can end up losing less or more than five points in that area.
DPI also made a change to the way the chronic absenteeism rate is calculated, now matching the way the federal government calculates it. Previously, students qualified as chronically absent for an attendance rate below 84%. Now, it includes students with attendance rates up to 89%.
Did DPI try to make schools look better?
To see how the new formula would change schools’ report card scores, DPI officials ran the new formula on the old 2019 data, which had already been run through the old formula for the 2019 report cards.
After running the numbers, DPI saw that the new formula resulted in lower scores statewide compared with the old formula, with the same 2019 data.
To offset that change, DPI officials changed the numeric goal posts for their standards, so that the same number of schools would meet expectations under both formulas using the same 2019 data.
That way, when people compared a school’s performance over the years, the formula change wouldn’t be as disruptive, DPI said.
The goal posts, also known as cutscores, were lowered for most performance categories:
● Significantly exceeds expectations: stayed the same
● Exceeds expectations: moved from 73-82.9 to 70-82.9
● Meets expectations: moved from 63-72.9 to 58-69.9
● Meets few expectations: moved from 53-62.9 to 48-57.9
● Fails: moved from under 52.9 to under 47.9
DPI officials denied any bias in the process. They said they based their changes to the cut-scores based on running the 2019 data through both formulas, before looking at how it would impact 2021 data.
When they ran the 2021 data, it so happened that the same percentage of schools, about 87%, met or exceeded expectations.
What do the report cards actually tell us?
While statewide trends are hard to discern, viewers can still find trends by looking more carefully at report cards for individual districts and schools in various years.
Each report card breaks down a district or school’s standardized test scores for different populations and subject areas, as well as attendance rates and graduation rates.
Report cards can be found at dpi.wi.gov/accountability/report-cards. More details about schools’ performance can be found at wisedash .dpi.wi.gov/Dashboard.