WHAT’S IN A NUMBER?
Yelp’s new health scores don’t mean what you think they do
Redwood City’s HoM Korean Kitchen is a trendy fast-casual eatery where guests can build their own rice-banchan-protein bowls using fresh, local ingredients. If all you’re looking at is the restaurant’s 55 out of 100 health score on Yelp, you might not visit. But that Yelp score doesn’t mean what you think it does.
During a routine county health inspection in March, owner Konan Pi got dinged for insufficient availability of hot and cold water due to a wonky water heater, which he promptly fixed. It’s considered a minor infraction by San Mateo County officials, one they’ll check at the next routine inspection, along with five other minor violations. If it had been a major violation, the county says, an inspector would have returned within three days to ensure that the matter had been resolved.
But the original violation endures on the county’s website and, more critically, on Yelp, which works with a third-party vendor that pulls health inspection records to extrapolate a health score for Bay Area restaurants.
Now, Pi, who has two other HoM locations in the South Bay and recently opened a new poke spot in Santa Cruz, will have to live with a misleading, algorithm-generated score emblazoned on the Yelp page until HoM is up for its annual inspection next year. To any former high school student, that 55 sounds a lot like a failing grade, even though San Mateo’s website clearly states that HoM passed.
“It’s very concerning,” Pi said. “Yelp generates a huge following for us. I’m hoping it doesn’t affect people’s decision in com-
"In some cases, the numerical scores posted to Yelp in no way reflect the observations staff have documented during the last inspection. This has led to confusion from the public, frustrated restaurant operators and calls to our office..."
— Diana Rohini LaVigne, chief communications officer for San Mateo County Health System
ing here.”
Pi is hardly alone. When Yelp rolled out its health rankings Bay Area-wide on July 24, things got bumpy for many local restaurants. Unlike counties such as San Francisco, whose health department issues numerical scores, San Mateo, Santa Clara, Contra Costa and Alameda scores are delivered as a simple color-coded pass/fail. So HDScores, the tech company working with Yelp, uses a proprietary algorithm to create a score.
Glynne Thompson, chief marketing officer of HDScores, would not disclose the methodology used, calling it “proprietary and quite complex,” but explained that each restaurant inspection is assigned a score based on the number of demerits. The lower the number of demerits, the cleaner the restaurant.
A critical violation, such as inadequate food storage temperatures, might receive three demerits, while a noncritical violation, such as poorly constructed or dilapidated walls, would receive one. The scoring is adjusted according to how jurisdictions apply the federal
food guidelines. A score of 83 in a city with a more lenient code, for example, might be a 50 in a city with a stricter one.
A simple Yelp search cross-referenced with restaurant health inspection records reveals other restaurants in the same situation as Pi’s, from Daly City’s Bangkok Thai Garden Cuisine to San Mateo’s The Pantry and Berkeley’s La Mediterranee.
Yelp considers the program, dubbed LIVES, or Local Inspector Value-Entry Specification, a boon for public health and a nudge for jurisdictions to update their public records in a timely fashion. “The target of Yelp’s sunlight is not only restaurants, but also city governments,” said Luther Lowe, Yelp’s head of public policy. “Why is the county of San Mateo taking nearly a year to reinspect a restaurant?”
The county’s response? When a private technology company mines raw inspection data and converts it to a numerical score, it often leads to misinterpretation, said Diana Rohini LaVigne, chief communications officer for San Mateo County Health System.
“San Mateo County has no control over what is posted on Yelp,” she said.
“In some cases, the numerical scores posted to Yelp in no way reflect the observations staff have documented during the last inspection. This has led to confusion from the public, frustrated restaurant operators and calls to our office to complain about the scores posted to Yelp.”
A disclaimer on the county’s website states, “Violations posted here and noted on the inspection form represent conditions found during the actual inspection. The facility’s present condition may be substantially different.” And when you click on a violation to read the actual description, an error message pops up.
LaVigne said the error message is the result of a recent software update; the county is currently reviewing options for refreshing the online restaurant inspection interface. “This is a multistep process, requiring numerous interconnected updates, which we have been undergoing for over a year, and look forward to increasing our ability to best serve our residents and businesses.”
In San Francisco, where the Yelp health scores rolled out in 2013, the city’s Golden Gate Restaurant Association is no stranger to the issues facing these restaurants.
But while Yelp is blaming San Mateo County health inspectors for not updating their website, there’s a more fundamental issue here, said GGRA executive director Gwyneth Borden.
“Here’s the problem: We know the pace government works,” Borden said. “What Yelp hasn’t taken into consideration is what the purpose of those scores are.”
Health inspections are designed to protect the public, of course, by examining restaurant facilities for major and minor infractions, warning restaurants about specific issues that need immediate
correction — and returning swiftly to make sure anything major has been resolved.
“You need to update this, you need to fix that,” Borden said. “I often say the health score is a pop quiz, not a final exam. It’s a snapshot in time. A bathroom without soap on the day of inspection? Clearly your restaurant has soap in the storage room, but if the health inspector walks in and the bathroom has no soap, you’re dinged.”
What diners need to realize, Borden said, is that if they’re reading about a violation months after the fact, it probably isn’t outstanding now — and it may not have been outstanding even a week after it was first flagged. The system was never set up to update scores after a reinspection. It was set up to trigger compliance.
“You don’t get issued a new score,” Borden said. “The things you got in trouble for, you fix. You have a reinspection within one week. You have to live with the bad score.”
Now, she said, Yelp’s data management company “is basically creating their own scoring system without telling you how they arrive at those scores. Businesses are being penalized by a (Yelp) system that is not transparent.”
The economic fallout can be severe. In 2015, Yelp added a health alert box to the pages of San Francisco restaurants with major violations that landed them in the bottom 5 percent. Business for those restaurants fell by 15 percent, Borden said. Yelp is continuing to utilize the warning boxes for San Francisco, for now.
Bottom line for Yelpers? Some of those brand-new health scores are long out of date.