Data inherently flawed, school officials say
After what his school district has been through the past two years, the challenges that were thrust on his students and staff, Dr. Joseph Macharola isn’t too concerned about what a handful of test scores say.
The Muhlenberg School District superintendent said he has more important things on his plate. He has more urgent things to worry about than the results of the 2020-21 Pennsylvania State System of Assessment and Keystone exams that were released publicly by the state Friday.
He’s trying to keep people safe, to make sure there’s not a loss of life from COVID in his district. He’s doing his best to address the growing, dire mental health needs of students and staff.
He’s working to make sure kids have something healthy to eat, even if a parent’s loss of a job has made that difficult at home.
Standardized tests like the PSSAs and Keystones simply aren’t a priority for him right now.
Those tests were taken last year in the midst of a pandemic that has wreaked havoc on communities, and the idea that students would be able to focus and perform their best on them doesn’t make sense, Macharola said.
“It’s ridiculous to expect good results,” he said. “I don’t believe the tests should have been given. This is not authentic assessment.”
Macharola isn’t alone in his sentiment. Many members of Pennsylvania’s education community have expressed similar thoughts, questioning the usefulness of the new data that shows big drops in scores across the state.
In fact, even the state Department of Education issued a warning about reading too much into the numbers they released Friday.
“Historically, standardized assessment results have been an important part of understanding school performance and our work to close achievement and opportunity gaps,” Deputy Secretary for Elementary and Secondary Education Sherri Smith said in a press release announcing the posting of the data on the Future Ready PA Index website. “But this year’s results are anything but standard. We recognize that the global COVID-19 pandemic brought tremendous challenges to the school year, impacting students, teachers and staff alike, as we worked to protect the public health and safety of everyone in our classrooms.”
Smith stressed that administering the tests is something that is required by the federal government. That requirement was suspended for the 2019-20 school year, since the exams are typically given in March and April.
That was when the first wave of the COVID pandemic was sweeping across the U.S.
But they were back last school year, with a few changes. Students were able to take the exams at the typical time in the spring or wait until school started again in the fall — a move that pushed back the release of scores from its normal spot in October.
Some students chose to do neither.
Smith said participation rates in exams saw a sharp drop, with many students opting to simply not show up for some. Some districts — like Brandywine Heights — told parents to consider the exams optional.
Because of all the variables at play, Smith said, the state will not be using the 2020-21 test results to evaluate schools.
“Given these circumstances, the results should not be viewed as a complete, representative sample of all students in the commonwealth, nor should a single assessment during an atypical school year be considered a true metric of student performance,” she said. “We will continue to work closely with schools to assist in the planning and implementation of evidence-based programs.”
An incomplete picture
Even in a normal year school districts don’t usually put a huge focus on standardized test scores. Most school officials say they are simply one of many tools at their disposal.
“The data that we are able to get from the PSSA/ Keystone scores is valuable, but I don’t think standardized test scores paint the full picture,” said Dr. Andrew Netznik, Tulpehocken School District superintendent. “English language arts, math and science are important parts of a student’s education, but so are art, music, history and civics, foreign language and wellness. Standardized tests fail to encompass what a child is truly capable of.”
The exams also fail to capture what is going on in a student’s life. And over the past two years, that has been quite a bit.
The pandemic closed schools. It forced students to learn virtually. It has exacerbated shortages of teachers and staff and bus drivers.
“We had to pivot on a dime to address an entirely different mode of instruction,” Macharola said. “Our students and staff were put into a situation they’ve never been in.”
And that’s just the school environment. Macharola said many of his kids were dealing with even tougher issues at home.
A survey done in the Muhlenberg School District in December showed that half of students had experienced depression for two or more weeks since the start of the pandemic, Macharola said.
A quarter of students at one point or another was worried about where their next meal would come from. About 20% knew someone who died of COVID, and 20% had a family member lose a job.
Most concerning, Macharola said, 15% attempted suicide.
“Our kids are getting pushed through the cracks. They’re not falling through,” he said.
Macharola said any data that comes from exams students have to take when dealing with all of that is worthless.
“They’re putting these numbers out there. They’re meaningless,” he said. “They mean nothing.”
Bill McKay, Gov. Mifflin superintendent, said he isn’t really sure what to make of the new data.
“The past two years have been so disruptive and overall assessment participation is much lower than any other year’s, it’s difficult to have an informed opinion,” he said when asked what the test scores mean. “We’ve never experienced anything like this in regards to assessment and data.”
Taking what they can get
Other local educators agreed the new standardized test results are flawed, with the pandemic looming large over them.
Some said they will still try to get something out of the data, looking for ways to glean whatever information they can.
“Our scores are lower than normal, but I think a lot of that can be attributed to the pandemic,” Netznik said. “When looking at the published scores, we have to look at them keeping in mind that some students were not present in a classroom for over a year. We will certainly study our testing data, but we put more stock in the real-time data we are currently collecting than data from last year.
“I believe the testing that takes place this spring will be a better indication for how our students are performing post-pandemic.”
Dr. Brett A. Cooper, Daniel Boone School District superintendent, said he plans to use the scores as a way to pinpoint areas where learning loss due to the pandemic occurred.
“Using this data as a learning baseline for what our students have learned, what they currently know, and what they are currently able to do is how we can best support identified growth opportunities through a multitiered system of support moving forward,” he said.
Christian Temchatin, Kutztown School District superintendent, said he thinks any achievement data point can be valuable.
But, he added, low participation rates in the 2020-21 exams lessen that value. In Kutztown, he said, participation was down 10% to 15%.
“This is significant enough to disrupt the longitudinal uses for the system evaluation,” he said. “Fortunately, our evaluation of student growth and success is calculated using multiple measures.”
Temchatin said that no matter what measure is used, the disruptions to education the pandemic caused is evident. But, he added, he has full faith his district has the people and systems in place to meet the additional needs of students.