Fiddling with hospital ratings
System that measures facilities is costly, secretive and open to political manipulation
One hundred and three years ago, around the same time shipbuilders were putting the finishing touches on the Titanic, a religious nursing order known as the Daughters of Wisdom was overseeing the final stages of a new hospital for Castor, Alta.
The three-storey facility that gradually rose from the eastern Alberta prairie was a handsome structure, built of red brick and adorned with a prominent wooden veranda. Given the name Our Lady of the Rosary, it was considered such a crucial need that the town provided a $3,000 grant toward the facility’s $12,000 construction and exempted it from taxation.
“On Nov. 1, (1911) the hospital was finished far enough to receive ... six patients, and by Nov. 6, the number rose to 15!” says a history of the hospital published by the Catholic Health Association of Alberta. “One room was completely finished; so, to accommodate the workmen, the sisters moved patients from room to room and floor to floor. For the first two months there was no running water.”
Today, that building still stands, registered as a Canadian historic site, but also remaining in service as part of a working hospital with several infrastructure concerns. Among other issues, the last evaluation of the site done five years ago cited poor insulation, inadequate fire protection, deteriorated sanitary pipes and an ancient steam heating system. Next door, the 1962 addition to the hospital — where most of the patient care now takes place — was found to have its own problems, including numerous design flaws ranging from temperature issues and cramped spaces to a lack of ventilation and insufficient patient privacy.
Though it may seem a simple matter, asking whether Our Lady of the Rosary deserves to be rated in government reports as in Good, Fair or Poor condition has produced a variety of answers.
In 2009, the private-sector consultant who evaluated the hospital for Alberta Infrastructure estimated it needed almost $4 million in maintenance work, a calculation that, according to the government’s infrastructure rating system, placed the facility in Fair condition.
However, about two years later, modifications to the ratings system meant the hospital’s score improved, almost pushing it into the Good category. A short time after that, things changed again, when the facility’s rating was discreetly and without explanation altered to Poor.
“I am confused all right,” Castor Mayor Garry Devloo admits when told of the sequence of events. “If you are an architect, maybe you see it differently, but if you ask me, I don’t think there is anything wrong with the place. It’s an older hospital, but it’s well taken care of and run by good, friendly people.”
Devloo’s puzzlement is hardly an anomaly.
A five-month investigation by the Journal found the province’s infrastructure rating system is convoluted, lacks transparency and is subject to manipulation behind the scenes.
Instead of providing objective measures of facility need that can be shared with the public and used to determine funding priorities, the system appears influenced by frequent changes in methodology, inconsistent practices and political whim — most of it applied outside of public view.
HOW ARE OUR HOSPITALS RATED?
Alberta Infrastructure’s annual reports lay out the basic formula for determining the rating of a health facility. Two things are required: a calculation of the maintenance a facility will need over the next five years, and a calculation of how much it would cost to replace the facility.
The first number is divided against the second number to get a Facility Condition Index score. Take for example a facility that would cost $50 million to replace. If it has a maintenance bill of $5 million — amounting to 10 per cent of its replacement cost — its FCI score would be 10. That building would get a Good rating, as would any building with an FCI under 15. Facilities with an FCI between 15 and 40 are considered Fair, while buildings over 40 are considered Poor.
The data used to determine the scores and ratings is obtained from assessments of the building, both by an external consultant once every five years, and by local facility managers on a more frequent basis.
Alberta Infrastructure began hiring the private-sector engineering and architectural consultants in 2008; the government says the consultants are chosen through an open competition, and are paid an average of $1.60 per square metre of the facility they are evaluating. That means an assessment of a small hospital like the one in Hanna might cost around $14,000, while a large structure such as the Rockyview Hospital in Calgary could cost up to $175,000.
Since Alberta has more than 3.2 million square metres of health infrastructure, it would cost more than $5 million to have all of it evaluated — although facilities less than 10 years old are not assessed as they are assumed to be in Good condition.
The company that has received the most work to review hospitals in the last five years is Golder Associates, a large, multinational construction and consulting firm that has previously given several donations to the Progressive Conservatives.
At each site, evaluators go through a checklist of all the facility’s components.
They then submit a report that highlights needed repairs, set a timeline for future maintenance, and provide an FCI score for the facility. Once a report is finished, it is edited by Alberta Infrastructure and posted on the ministry’s website.
These postings, which the government describes as a “snapshot” of a facility’s condition at a point in time, are about the only part of the rating system that is made public. Yet the way the reports are handled raises doubts about how valuable they are.
According to Alberta Infrastructure, each health facility is supposed to be evaluated once every five years on a rotating schedule, which means the first cycle of assessments should have been completed by 2013. While some critics suggest a five-year gap is already too long between evaluations, the province has failed to even keep up with this schedule and admits the first cycle won’t be complete until 2015.
The Journal found that of the 90 or so assessments of hospitals currently posted online, 44 are now more than five years old — providing a lot of very outdated snapshots for the public.
“The political consequence of those choices is pretty obvious. They help hide the problem longer and longer,” says Regan Boychuk, former research director at the Parkland Institute who has studied the manipulations of the rating system. “It’s supposed to be five years but it hasn’t come close to that target, and the result is further obscuring the problem of deferred maintenance and aging infrastructure.”
Of additional concern, there is often a long delay between when an evaluation report is done and the time it is posted.
For instance, the last review of the Leduc Hospital was conducted on March 19, 2010, but was not made public until more than two years later on April 11, 2012.
Also troubling is that in between assessments, local facility managers can update the database on their own with information they have gathered from their sites. These alterations are not made public, even though they often change a facility’s FCI score and rating. It is not clear how often these kinds of changes are made and whether there is a standard, documented procedure for making them.
Finally, instead of publishing updated FCI scores and ratings for individual facilities, Alberta Infrastructure provides a brief summary each year in its annual report that aggregates the condition of all its health facilities into one set of numbers.
The last report, from 201314, said 75 per cent of Alberta’s health infrastructure was in Good shape, 21 per cent in Fair shape and four per cent in Poor condition.
Those latest percentages are among the best the ministry has posted in recent years, although that isn’t because the government has done a lot of repair work. It just changed the way it adds things up.
METHODOLOGY CHANGES
In recent years, Alberta Infrastructure has introduced several different “improvements to methodology” to calculate the ratings. The effect of the changes has been twofold: The percentages reported to the public appear better, and it’s nearly impossible to make direct year-to-year comparisons.
One of the new methodology wrinkles, introduced two years ago, has added building size as a factor in the final percentages.
For example, in mid-2012, Alberta Infrastructure listed 166 health facilities in Good condition, 95 in Fair condition and 27 in Poor condition. In percentage terms, this works out to 58 per cent Good, 33 per cent Fair and nine per cent Poor.
But instead of using those figures, the ministry incorporated the actual size of the facilities into the equation.
Since the facilities in Good condition tended to be bigger in size than those with Fair or Poor ratings, that improved the percentages. The new calculation resulted in Alberta’s health infrastructure being rated 72 per cent Good, 24 per cent Fair and four per cent Poor — which were the numbers the ministry reported to public that year.
Another change to the methodology involved using the location of a facility to adjust its replacement value. In a statement to the Journal about the change, the ministry argued that the cost of rebuilding a hospital in a rural centre is more expensive than in Calgary or Edmonton, due to the higher labour and material costs in remote communities.
As a result, many hospitals outside the two big cities had their replacement values adjusted to be higher, which improved many of their FCI scores. And that, critics say, means the public isn’t getting the true story about the government’s failure to keep up with maintenance.
“The purpose of the metrics is accountability ... so that there is some way for taxpayers to know how all this stuff is being managed,” says Boychuk. “When they fudge the numbers and come up with dramatically different results without the required investment, they are avoiding accountability for their mismanagement.”
However, that’s not the only manipulation going on: the Journal also found evidence the consultants’ assessments, on which the FCI scores are based, can be secretly disregarded by the province and Alberta Health Services if they don’t agree with the findings — even though the assessments are still posted for the public.
RATINGS ALTERED
In the summer of 2012, then-health minister Fred Horne requested data on the physical condition of Alberta’s health facilities. Alberta Infrastructure responded with a memo, which the Journal obtained through an access to information request.
The memo provided a summary of the rating system and noted that 153 health facilities had so far been evaluated by a consultant. It then added an important proviso.
“Not all ... evaluations were used to determine the physical condition of a facility. INFRA and AHS review each evaluation to reach a consensus on the latest condition of a facility.”
The memo then noted that only 119 of the 153 consultant evaluations had been accepted.
Alberta Infrastructure’s annual reports make no mention of this part of the rating process, that consultant assessments can be dismissed in favour of the views of AHS and ministry staff.
Asked for an explanation as to why that information is missing from the annual reports, Alberta Infrastructure said the memo is part of its detailed business plan, which is a larger, more comprehensive document than the annual reports, which are more of a “brief synopsis.”
The data attached to the memo indicated 23 hospitals had their condition ratings altered in a way that deviated from their FCI score. Among them was Our Lady of the Rosary in Castor, which had its rating changed to Poor despite having an FCI of 15.1.
A further analysis of these changes revealed an interesting pattern. Of the 23 alterations, just two hospitals had their ratings improved, while 21 had their ratings downgraded to Fair or Poor.
In two cases, the Northern Lights hospital in Fort McMurray, with an FCI of 11.65, and the Beaverlodge hospital, with an FCI of 7.83, the ratings were reduced all the way from Good to Poor.
In a statement to the Journal, Alberta Infrastructure said the changes were made after meetings with AHS maintenance and engineering staff who raised concerns “the FCI did not accurately reflect the current condition of their facilities.”
Asked why only the condition ratings were changed and not the hospitals’ FCI scores, the ministry said that question had been the subject of a review. Asked what the result of the review was, the ministry said it was determined that administrators had enough information about the facilities to change the ratings, but did not have enough information to set new FCI scores.
“When the facilities are reviewed again, the FCI will be adjusted accordingly,” the statement said.
At a rate of $1.60 per square metre, the consultants who reviewed the 23 hospitals with altered ratings were paid a combined $390,000, even though their findings were dismissed.
The Journal’s analysis of the alterations also found that of the 18 rural hospitals to receive a lowered rating, 14 are in ridings represented by a Progressive Conservative MLA. The remaining four are in ridings with a Wildrose MLA, all of whom won a close race (less than 1,000 votes) in the 2012 election.
The political divisions are of interest because a potential impact of lowering a hospital’s rating is that it may create a higher perception of need, and thereby help to move that hospital up the list for a renovation or rebuild.
There is some evidence this may have happened, since several of the facilities that were downgraded are currently the subject of a major capital project, or are being considered for construction funding. A further analysis of this will appear Thursday in the Journal.
Infrastructure Minister Manmeet Bhullar says he is not aware of anything untoward going on with the rating system, including consultants’ reports being disregarded or facility ratings changed.
“This is the first I’ve ever heard of something like that. That is a very serious issue. So I would ask for some detail so I can get to the bottom of that. I presume there could be a variety of reasons for that,” he says.
“I can make that commitment to you, that I believe that if there is information we are using that is different than what is made available, we should be making that information available.” kgerein@edmontonjournal.com twitter.com/keithgerein