Auditor dissects Phoenix failure
No matter how many times you revisit the origins of the Phoenix Pay debacle, the utter failure of management and lack of government oversight remains stunning.
Auditor general Michael Ferguson and his team scoured the eight-year long decisionmaking that led to the launch of the federal government’s new pay system, Phoenix, on Feb. 24, 2016.
The gist of his report, published Tuesday, is that a small management clique within Public Services and Procurement Canada failed to seek independent advice, paid little attention to the concerns of federal departments that would use Phoenix, ignored the mounting project risks and, when faced with growing costs, elected to strip key software features from the pay system rather than ask for more money to do the job properly.
“The building and implementation of Phoenix was an incomprehensible failure of project management and oversight,” the report says.
Accordingly, Ferguson offers a series of recommendation aimed at strengthening the management of large government projects. The key ones: that Public Services create an oversight mechanism approved by the Treasury Board, and that government-wide projects should be compelled to carry out fully independent reviews before they proceed. The government has accepted these are worth doing.
Much of the story about Phoenix’s botched rollout has already been told, including in this newspaper.
However, Ferguson’s observations are worth recounting because, as he notes in the report, it’s crucial that “government learn from the mistakes.”
One of the most profound errors in this case involved a false economy. It occurred in the spring of 2012, just as the former Conservative administration of Stephen Harper was getting aggressive about cutting federal spending.
Public Services had hired IBM as a subcontractor to help design and customize the software Phoenix relies on. Ferguson notes that during the planning stages, IBM estimated it would require $274 million to include the nearly 1,000 pay functions demanded by Public Services. Trouble is, the budget for Phoenix had been set at $155 million when approved in 2009.
So what did Public Services do? It cut costs to make sure Phoenix fit the initial budget. This was no small matter. Ferguson points out the Phoenix project office cut staff, eliminated certain pay-processing functions, such as automating pay transactions for people in acting (temporary) positions, reduced the testing schedule and compressed the time frame for introducing Phoenix.
Nor, he adds, did the Phoenix project reassess how all of these changes affected the overall risk of the rollout. There was no test of the system as a whole and a pilot test scheduled for 2015 was cancelled because the project’s managers did not want to delay Phoenix’s launch.
All of these contributed to a rapid escalation of errors in calculating pay early in 2016. In his November 2017 report on the pay project, Ferguson calculated that half of the 290,000 government employees who rely on Phoenix had outstanding pay requests in mid-2017. That number has since widened to more than 600,000, albeit under a wider definition of problematic pay transactions.
A common theme in Ferguson’s most recent report is the deafness of the senior management group. Ferguson doesn’t name them but the three top executives at the time of the launch were assistant deputy minister Brigitte Fortin, associate assistant deputy minister Rosanna Di Paola and associate deputy minister Gavin Liddy.
Ferguson points out the Phoenix project team over-estimated the ability of new hires at the government’s pay centre in Miramichi to handle the new system, then failed to heed warnings that this would be a problem. Nor did Public Services management pay much attention to the concerns of the 101 departments and agencies that were being asked to move to the Phoenix system.
Public Services did take the trouble to commission an assessment of the pay system’s readiness from S.i. Systems, a staffing services company based in Alberta.
However, Ferguson says there were several serious problems with the report. First, it’s difficult to assess a project’s readiness while it’s still under development, as was the case here. Nor was S.i. Systems’ review particularly independent. Phoenix executives not only helped to develop the questionnaire used by S.i. Systems, but also directed the company to question only Phoenix project staff. Other federal departments were not consulted, Ferguson adds.
The time of the report was also problematic: S.i. Systems completed its assessment just
weeks before Phoenix was due to be launched.
There was a second independent report — by Gartner Group — that did warn of significant risks and urged the project to implement Phoenix more carefully, in stages, starting with the least complicated departments. This report was commissioned by the Treasury Board, which forwarded a draft to the Phoenix project office on Jan. 29, 2016.
Gavin Liddy later testified before a House of Commons committee that he did not show the Gartner report to then public services minister Judy Foote until the summer of 2016, after Phoenix’s many problems had become public.
“We thought we had addressed all of the concerns in the Gartner report when we went live (on Feb. 24),” Liddy told members of Parliament, “and we thought we had a more systematic thirdparty review (S.i. Systems) which provided us assurances that we were making the right decision to move forward.”
Ferguson’s purpose with Tuesday’s report was to make sure nothing like this happens again in federal government.
As yet, Ferguson hasn’t committed to a future assessment on how Public Services is managing the profoundly difficult $1-billion job of repairing Phoenix. At this point, it’s probably fair to say government workers care very little about lessons learned. They just want Phoenix fixed.
We thought we had addressed all of the concerns in the Gartner report when we went live.