Incomplete data often at heart of school rankings

The recent U.S. rankings from education website Niche featured 12 Bay Area schools in the top 100 “best private high schools.”

East Bay and Peninsula private schools fared particularly well in the ranking, with eight of the 12 in those areas and five of them landing in the top 20 — including the top-ranked Bay Area school, College Preparatory in Oakland, at No. 8. Meanwhile, there were only two San Francisco schools in the entire top 100: University High School at No. 68, and Lick-Wilmerding at No. 72.

High school rankings don't typically play as large a role in parents' decision-making as college rankings, experts said — factors like geographic closeness to home and reviews from friends and family tend to have more of an impact on high school choice.

However, they can appear to show how schools stack up against each other, whether on academics, college preparation, diversity, teacher quality or student experience. And that implied prestige can carry some weight, experts said, particularly for parents considering Bay Area private schools — which are among the most expensive in the country, with tuition at some topping $50,000 per year.

But those rankings rarely offer a complete picture of a school's offerings, experts said. The data collection itself, as well as which data points are deemed more or less important by the rankings website, is not uniform, but it significantly affects how the schools are rated on school information websites like Niche, they said.

Niche's methodology, for example, relies most heavily on self-reported data about test scores, academics and college admissions that comes from Niche users — parents, students and alumni — especially for private schools, which it notes are not required to share that data with the Department of Education.

Natalie Tsay, a spokesperson for Niche, said in an email that for self-reported data, the website has a minimum of five to 20 responses, depending on what data is being looked at, and significant statistical analysis is performed to ensure that the data is evaluated fairly across schools.

Still, it's important to look closely at where the information comes from when looking at rankings of any kind — including the Niche list, which draws from multiple data sources, said Trinette Marquis, executive director for the California School Public Relations Association.

“People need to look at the quality of the data,” she said. If it's based on very few respondents, “how much can you depend on that?”

On top of that, how the website decides to weigh these factors gives some schools an advantage. Rankings that evaluate schools largely on test scores, or the colleges their graduates go on to attend, represent only a sliver of what can make a school “good,” said Carol Kocivar, former California State PTA president, adding that they are more reflective of community education levels and income than student growth.

In the Bay Area, private schools ranked more highly on average than public schools did on their respective lists. But that's likely because the schools that rank highly often have fewer English learners, fewer students with special needs and fewer students from low-income backgrounds than state averages, Kocivar said. That means students are often entering elite, selective schools with some advantages already in place, and the rankings don't reflect how much the school helped their students improve.

“Schools that have parents that are relatively affluent and are well-educated are going to have higher results,” she said.

Marquis echoed this, saying that it's important to keep in mind whether the rankings are a matter of correlation versus causation.

For a school in a wealthy area or a costly private school, for example, students will probably do well on standardized tests and get into good colleges no matter the quality of the school, because they'll have access to the resources they need.

“People need to keep in mind what we do know actually impacts student success,” she said, adding that simply being in a place where other students perform well on standardized tests, a measure that rankings tend to rely on, isn't it.

What's more important, she said, is teacher quality — which parents can judge best by simply visiting prospective schools.

“If students like a teacher, if they are inspired by a teacher, they are so much more motivated to work hard for them, no matter their background,” she said.

Tsay said that Niche rankings try to evaluate schools on a variety of topics that “are meant to capture the overall quality of the school,” as well as whether it meaningfully helps its students. Part of what allows them to do that, the company's website says, is combining numbers from places like the U.S. Department of Education and Common Core data with the surveys it administers to K-12 students and parents.

She also noted that differences in ranking can come down to a very small margin.

In the Niche lists, for example, private high schools in the North Bay and on the Peninsula slightly outperformed schools in San Francisco in user-generated information about standardized test scores and where students end up going to college, which meant that they ended up outranking the San

Francisco high schools.

But for data pulled from outside reports, including factors like student-teacher ratio and proficiency statistics, the schools were largely the same, Tsay wrote.

On top of that, all 12 of the Bay Area private schools that made the top 100 in the nation still had an A+ overall score from Niche, which is only given to the top 2.5% of schools and means that they're performing at “an elite level,” she said.

Paula Molligan, a Bay Area school consultant and co-author of “The Private High Schools of the San Francisco Bay Area,” said that such limitations mean that her consultancy firm never uses these kinds of rankings when helping parents decide where to send their children. It's more helpful, she said, to visit schools and talk with current parents and alumni to figure out the best fit.

“We don't believe there is any one best school,” she said. “There's a lot you have to take into consideration. Statistics are so limited.”

Tsay agreed.

“Niche grades and rankings are just a starting point,” she wrote. “We encourage parents to spend time doing a deep dive on all the content and reviews Niche has available on schools of interest to them, but also to visit the school's website or contact the school directly for more information before making final decisions.”