CAN IT RISE FROM THE ASHES?
Phoenix debacle after two years
The warning signs were there all along, but perhaps none was as clear as this.
On Jan. 13, 2016 — barely two months into the new Liberal government of Justin Trudeau — Treasury Board officials prepared a concise memo for their boss and comptroller general, Bill Matthews.
The note, obtained under accessto-information laws, was headlined “readiness assessment of Phoenix” in reference to a pay system that was scheduled to launch in a matter of weeks.
It had been a massive undertaking. Nearly 300,000 federal government employees from 101 federal departments and agencies were about to exchange a clunky, labour-intensive pay apparatus with an automated system that promised quick and accurate results. About 190,000 of these workers were to be served by newly hired pay administrators based out of Miramichi, N.B.
The Treasury Board had summarized the major worries conveyed to Matthews in an earlier meeting with top human resources officials from across government. Taken together, their concerns offer a disturbing glimpse of a project that appeared nowhere near ready for prime time despite years of development.
Officials at Employment and Social Development Canada, for instance, had warned Matthews that Phoenix’s “readiness is questionable. Out of 25 outstanding defects, 10 are still critical and not fixed.”
The feedback from Correctional Service Canada suggested there had been “no end-to-end testing,” and that error rates topped 50 per cent for pay transactions involving shift workers — a particularly egregious failing given that shift employees made up 40 per cent of that department’s workforce.
The RCMP and Public Safety department complained about escalating error rates; at Public Safety, error rates had reached 30 per cent.
Canada Revenue Agency’s concerns included a pointed question about Phoenix: “What is the contingency plan if it does not work in February?”
While the Public Services and Procurement Canada department had developed the governmentwide pay system, the individual departments were vitally important.
It was their job to forward information about promotions, transfers, maternity leave and other changes involving personnel to the Phoenix system for processing. If data wasn’t entered properly, or was incorrect, the system would spit out errors at a prodigious rate.
The departments’ human resources officials were telling Matthews there was a high risk of this occurring, in part because their pay administrators weren’t getting the kind of training they needed, but also because Phoenix was, well, complicated.
Yet, just 16 days later, a Treasury Board-led committee of 45 top bureaucrats — deputy ministers and equivalent — determined such concerns would not stand in the way of Phoenix’s launch.
Today we all know how much turned on this decision.
THE TWO-YEAR TIRE FIRE
Since the launch of Phoenix, federal government employees have faced a seemingly endless parade of errors.
As recently as Jan. 24 of this year, more than 630,000 transactions were in a queue for processing at the Miramichi pay centre. The vast majority were requests to rectify incorrect pay — employees had received either too much, too little or none at all. A minority had to do with administrative issues — such as requests for forms, information about benefits or explanations about payroll deductions.
The knock-on effects are reverberating with particular force through the capital region, where federal government employees make up nearly one-fifth of the workforce.
When something as basic as pay goes awry, the stresses accumulate.
In the public service, long perceived as a bastion of workplace and career stability, workers have been deferring promotions, refusing overtime and even delaying retirement because they don’t trust Phoenix to deposit the proper amounts into their accounts. Multiple technology projects throughout government have been postponed until Phoenix is fixed because managers don’t trust the linkages to the pay system.
The burden on taxpayers has escalated sharply. Canadians are now on the hook for at least $500 million in extra costs, the current estimate for repairing the pay system and rehiring hundreds of compensation advisers. Nor does this include the $310 million it cost to build the Phoenix system and centralize much of its operations in Miramichi in the first place.
Among the officials charged with getting this thing fixed — none of them associated with the original project — there’s concern the Phoenix system may be beyond repair.
Dozens of smaller projects have been launched to isolate the worst problem areas and devise appropriate fixes. The government, with the help of private contractors, is experimenting with SWAT teams of experts to redesign how work is done at the pay centre.
Public Services Minister Carla Qualtrough framed the challenge this way last fall: “The definition of ‘fixing ’ is one that we’ve struggled with a lot. If by ‘fixing ’ you mean at what point we will have a stable systems that pays everybody on time and accurately, it will not take years. If by ‘fixing ’ you mean a state-of-the-art, integrated human resources-to-pay process policy system, that will most likely take years.”
For the moment, the goal remains limited to trying to stabilize Phoenix by year-end. But in case that doesn’t work, Qualtrough and her cabinet colleagues have said they will simultaneously develop plans for building a full replacement for Phoenix down the road — a remarkable prospect.
Such is the current state of Phoenix the government has adopted something akin to a war footing. The prime minister last April created a working group of senior cabinet ministers to keep close watch on fixes being applied to Phoenix. The key ministers are Qualtrough, whose department is directly responsible for pay systems, and Treasury Board president Scott Brison, who has oversight of government spending and management. Public Services and Treasury Board are jointly responsible for the newly created Pay Stabilization Project, which is directing the effort to fix Phoenix.
Starting last summer, the government established no fewer than three bureaucracies — deputy ministers, assistant deputy ministers and directors-general — to provide oversight on matters of pay across all government departments and agencies.
And, not least, federal government unions are being consulted regularly by senior government bureaucrats.
No one knows yet whether this unprecedented attempt at repair will succeed or merely result in thousands of hours wasted in meetings. What is clear is that the Liberals are increasingly anxious to put the Phoenix debacle behind them. The last thing they want to see in advance of next year’s federal election is a retreat to the worstcase scenario — having to rebuild Phoenix from scratch. BEST OF INTENTIONS
In retrospect, the Phoenix pay failure has acquired the air of inevitability.
Yet the system was built, mostly, on solid concepts. Certainly an upgrade was needed. The pay system Phoenix replaced was more than four decades old. As with any piece of aging technology, administrators responsible for maintaining it were spending too much time on patches, workarounds and other repairs. Pay errors were increasing and weren’t being tracked on a consistent basis.
The old system had grown up willy-nilly. Some departments used human resources software from PeopleSoft, a tech pioneer based in California’s Silicon Valley. Others relied on products built by SAP, a German software specialist. Most departments and agencies adapted the software for particular requirements.
To work, this setup required a generation of specialists with expertise in near-obsolete versions of software and deep knowledge of thousands of pay rules that conformed to federal legislation and dozens of collective agreements. Many of these same workers were getting ready to retire.
The need to replace the system was obvious as early as 1989, when the Conservative government of Brian Mulroney began studying options. Four years later, during what would prove to be the tail end of the Tories’ electoral mandate, Accenture won a $45-million contract to automate much of the work handled by 750 pay and pension administrators.
Sadly, that effort proved a mess. Two difficulties emerged: First, the Accenture effort was a custombuilt system — much more difficult technically than the Phoenix pay project. (Phoenix is based on tweaking PeopleSoft’s standard technology platform.) Accenture was also modernizing two systems in one go — pay and pensions.
Second, the government had awarded a fixed-price contract, which meant that when glitches emerged in development, Accenture was forced into difficult tradeoffs: To stay within budget, it had to either eliminate certain software features or try to make the case for a revised contract.
Indeed, soon after development of the new pay system began, Public Services and Accenture argued over the proper scope of the work. In 1995, Accenture submitted multiple alternatives — involving different combinations of contract extensions, increased prices and fewer features. Accenture was confident it could successfully build the combined compensation systems but it required an estimated $10 million more to do it and additional time.
But the negotiations with Public Services came at the worst possible time. In late February 1995, then Liberal finance minister Paul Martin delivered his seminal deficitbusting budget that called for dramatic cuts in the budgets of most federal departments.
On April 20 that year, the government terminated the contract with Accenture, triggering a legal battle that would not be settled until 2003. Terms were never disclosed but the cost to the government likely topped $20 million, when measured in today ’s dollars.
The upshot: The government’s 1960s pay system had continued to age and was no closer to renewal.
A TEMPLATE FOR PHOENIX
When the Conservatives returned to power in 2006 under Stephen Harper, they took a fresh look at government operations. They examined in particular the approach adopted by an obscure unit within Public Services called the accounting, banking and compensation branch. The unit, which was directly responsible for government employees’ pension services as well as the pay system, had, several years earlier, begun revamping the obsolete pension system that served half a million government employees and retirees. The department had determined it would be more manageable to try this in a separate project, rather than combine the pay and pension systems as attempted in the early 1990s.
This pension project would become a template for Phoenix.
On the surface, the two projects had striking similarities. To modernize the federal government’s pension plans, Public Services invested $230 million in technology and new processes, reduced the number of pension administrators and created a new bureaucracy in Shediac, N.B., about 90 minutes up the road from Miramichi.
After 2013, when the initial pension project was complete, the government planned to book savings of nearly $30 million per year — thanks in large part to a reduced head count.
It’s not clear whether those savings were actually achieved, but the project is considered a success. By early 2017, the Canadian Forces and RCMP had also joined the pension plan, which now administers 854,000 accounts. The Shediac pension centre consistently meets targeted service standards, ranging from making initial pension payments to providing estimates for annuities.
Indeed, it’s little wonder that Public Services officials cited this project as evidence that they knew what they were doing.
However, the Phoenix project was a different beast.
There was just one governmentwide pension system to deal with, not dozens of departmental pay systems, as was the case with Phoenix.
The timing was also much better for the pension system, which had been largely rebuilt by 2011. That’s when the Conservatives began seeking budget cuts from most departments and agencies.
Indeed, the Phoenix pay project was hit twice. First, some of its capital budget was trimmed during the fiscal year ended March 31, 2013, as part of a governmentwide cost-cutting exercise. But, perhaps more importantly, the Conservatives encouraged Phoenix project managers to make achieving savings a priority for the new system.
In 2012, Public Services began shedding the first of 1,200 pay administrators who were to be replaced by 550 new hires at the central pay office in Miramichi. The annual salary savings and greater efficiencies were predicted to save $70 million a year once the new Phoenix system was up and running.
This meant the government was looking to recoup the upfront investment costs of the new pay system in less than four years — or twice as fast as was the case with the pension modernization project.
Such aggressive payback would contribute heavily to Phoenix’s undoing. Public Services also appeared to neglect a crucial lesson from the shift of pension administrators from across the country into Shediac nearly four decades ago: it had taken three to four years to fully transfer all the arcane knowledge, according to a retired Public Services manager who was involved with the original pension modernization project. The workers hired from 2012 to 2015 to do pay administration in Miramichi were faced with an even higher learning curve given the complexities of the Phoenix system.
UNDER PRESSURE
At the Place du Portage headquarters of the Phoenix project in downtown Gatineau, the tightness of the timelines would have important consequences.
There were two keys to making the new pay system work.
First, Public Services project managers had to reshape and standardize roughly 80,000 pay rules — definitions of what constitutes overtime or shift work, when employees are eligible for various types of leave, bilingual bonuses, job classifications, travelling time, among many others. Often, these definitions vary by collective agreement or department. Second, government pay administrators — whether in the departments or Miramichi — were required to enter this data into the Phoenix computer and software system developed by IBM Canada, a subcontractor hired in 2011. The Phoenix system calculates what will appear on employees’ pay stubs.
The initial design of Phoenix was finished in June 2014. A summary of tests of the new pay system — dated Jan. 19, 2016 — shows the meticulous interplay between the pay rules and the Phoenix calculator. The summary, obtained through access to information, shows more than 100 errors.
What kind of errors? They can be found in the minutiae of the summary.
For instance, the Phoenix calculation program “does not refund properly” if a pay administrator generates a benefit request though a general deduction, rather than one specific to the type of deduction.In another instance, Phoenix was calculating a pension plan contribution “though no contributions should be taken while on paid leave of absence.” One other example is more administrative: pay advisers can’t eliminate an ‘acting’ title for an employee unless the date he or she ceases to have this title “falls on the beginning date of next pay period.”
Originally, 101 federal departments and agencies were to shift to the Phoenix system in three waves, starting in July 2015. But Public Services and IBM reckoned the schedule wouldn’t allow them to finish and test the system. They delayed the rollout by three months in part by compressing it into just two waves, starting in October 2015 and ending in December of the same year.
It wasn’t enough. Public Services and IBM made two further adjustments, which proved fateful.
In mid- September, a little more than a month before the Liberals would win a majority mandate, the project managers delayed the scheduled launch of Phoenix a second time. Now the first wave would go out in February 2016. The second and final wave would finish two months later.
This gave the project more breathing room to fix software issues and system errors that were still cropping up.
But the delay also represented a complication.
Most federal departments had already hired extra staff to assist with the transition from the old pay system to Phoenix. Top bureaucrats were therefore reluctant to consider a third delay in Phoenix’s launch beyond February because the additional salary costs related to the transition were coming out of their departmental budgets.
The sooner Phoenix got rolling, the better, as far as budgets were concerned. Equally important, Public Services officials were reassuring colleagues in other departments that the errors emerging from tests of the Phoenix system would be dealt with before the February 2016 launch.
The second major adjustment to the project that summer involved software.
A November 2015 briefing note informed new Public Services Minister Judy Foote that IBM Canada “has encountered some delays due to the complexities of the federal pay system but are working diligently to ensure successful delivery.”
An industry official familiar with the project said the source of the delay had been the government’s insistence on adding new features to Phoenix. Just which ones isn’t clear. However, in order to simplify the project in preparation for launch, Public Services agreed to delay several key software enhancements. These included pay requests involving retroactive transactions — particularly when these involved ‘acting ’ positions — temporary promotions, in other words — or provisions in a collective agreement.
These were just some of the 200 software patches IBM had developed to adapt federal government pay rules to Phoenix, but their omission would prove crucial. It would, in fact, plant a time bomb within Phoenix.
“Acting pay” issues would haunt the system. The auditor general of Canada determined late last year that roughly 25 per cent of outstanding pay requests that accumulated after Phoenix went live were related to acting pay. Even after IBM added this software feature in March 2017, 40 per cent of requests to put through acting pay still had to be processed manually, Auditor General Michael Ferguson estimated in his report late in 2017.
Finally, there was also an urgency to the Phoenix project, a momentum that seemingly could not be denied.
Before the launch of Phoenix in 2016, even before the Liberals won the election in October 2015, the program to consolidate pay administration in Miramichi was nearly complete: Just 550 employees were to take over the work of 1,400 experienced pay administrators who had been eased out of government in the name of economy.
With the human resources already trimmed and allocated, there was no going back because of Phoenix.
There was no Plan B.
EVERYTHING WILL BE FINE. NO, REALLY
Certainly the bureaucrats most closely associated with Phoenix were confident it would all work out. But why were they so confident? Rosanna Di Paola, the associate assistant deputy minister at Public Services, had been a member of the project team off and on since 2009, when she co-authored the business case study that launched Phoenix. Di Paola was in charge of Phoenix during the launch phase.
Her direct boss, assistant deputy minister Brigitte Fortin, advised Foote in December that Phoenix was ready.
Going up the chain of accountability, associate deputy minister Gavin Liddy, cited the results of a study commissioned by Public Services to assess the project’s readiness. The consulting firm S.i. Systems delivered a draft report on Jan. 18, 2016, that concluded the Phoenix project team was to be commended on a job well done. “S.i. Systems recommended that on balance we should proceed,” Liddy told a Commons committee Nov. 29, 2016.
Other departments were still doing due diligence. Treasury Board, for instance, hired Gartner, an independent consultant, Dec. 21, 2015, to provide perspective on the potential risks facing Phoenix. Gartner, too, produced a draft report in late January but it was much less sanguine than that of S.i. Systems.
“End-to-end testing has not been performed by any department that Gartner has interviewed,” the consultants noted. Gartner was also concerned that the federal departments had yet to put in place training programs — an omission that could lead to “unanticipated consequences such as an incorrect pay calculation.”
To mitigate these and other risks, Gartner suggested dividing the two main waves of employees into multiple waves during the crossover to Phoenix, and to begin with the least complicated departments — ones with relatively few seasonal or shift workers, for instance.
Treasury Board forwarded the Gartner conclusions to Liddy on Jan. 29.
Liddy would later testify before a House of Commons committee that he did not show the Gartner report to his boss Judy Foote until the summer.
“We thought we had addressed all of the concerns in the Gartner report when we went live (on Feb. 24),” Liddy told members of Parliament, “and we thought we had a more systematic third-party review (S.i. Systems) which provided us assurances that we were making the right decision to move forward.”
Liddy, who declined an interview request from this newspaper, explained to members of Parliament that when he read the Gartner report he saw that it had been based on interviews with officials in federal departments outside Public Services and Procurement Canada. He said those departments, “weren’t aware of what we had actually done. We ran 16,000 tests. When we went live, we had 124 pay and pension defects remaining, none of which were critical.”
A report later commissioned to isolate what lessons could be learned from Phoenix pointed to a tendency of some Public Services officials to downplay bad news.
“There was a culture at Public Services in particular but also to some extent within the government more broadly that is not open to risk and does not reward speaking truth to power,” concluded a 2017 report by consultants at Goss Gilroy — which had been hired by Treasury Board. “This practice of not providing briefings that contained bad news was exacerbated by a tendency to accord a great deal of leeway to managers with a good track record of managing projects,” the consultants added.
Certainly the top bureaucrats in Public Services’ accounting, banking and compensation branch — the group directly responsible for Phoenix — had an excellent reputation, in large part thanks to their successful piloting of the pension modernization project.
That may have been why they were completely unprepared for the firestorm that awaited them after they pushed the go-live button on Phoenix on Feb. 24, 2016.
TOO SLOW TO REACT
At first, they thought it was successful.
In the first wave, 34 federal departments switched from their old pay systems to Phoenix for 120,000 employees and began dealing with the Miramichi pay centre. The second wave, involving 67 departments and agencies and another 170,000 employees, crossed over on April 21.
The top bureaucrat in Public Services — deputy minister George Da Pont — retired before the second phase of pay system’s rollout began.
“The number of complaints we were getting — and we were monitoring this — was actually very low,” Da Pont’s successor Marie Lemay testified before the Commons committee in July 2016. “We expected to see some (complaints) and that we’d be able to address them.”
Lemay added that after they had launched the first wave, it would have taken something extraordinary to prompt a halt before the second wave. “You’re managing two systems, trying to understand and learn one while dealing with the other one,” she added.
The limitations of Phoenix became manifestly more apparent during the second phase of the rollout. Auditor General Michael Ferguson calculated in his most recent report that, prior to Phoenix’s launch, about 36,000 employees had nearly 96,000 outstanding pay queries, which could include anything from a request to update banking information to a demand to fix a pay error.
These essentially doubled during the second wave. By mid-2017, about 150,000 employees were awaiting resolution of some half a million pay requests.
Briefing notes for Brigitte Fortin written in advance of conference calls held April 28 and May 4, 2016 offer a view of the bureaucrats’ state of mind immediately following the rollout.
“The Phoenix pay system itself is fine,” the briefing note asserted. “With any major information technology change time is required for managers and employees to become familiar with it. We have now ironed out many processes, answered many questions and addressed many issues,” the briefing continued.
Ferguson, the auditor general, found this perspective troubling.
“Public Services did not have a full understanding of the extent and causes of pay problems,” the auditor general said last fall, “Until a year after Phoenix was launched, the department was still responding to pay problems as they arose.”
Indeed, Ferguson concluded that Phoenix’s project managers blamed departmental managers and pay advisers for “not understanding how and when to enter information into Phoenix.”
This is one of several factors that contributed to the failure of Phoenix. Public Services, the monopoly supplier of pay services within government, failed to make sure its clients — the departments — were happy with the system before rolling it out.
A second factor had to do with politics, which were aligned with priorities that had little to do with building a successful pay system.
The Conservatives insisted on the false economy of extracting savings from a project before it was clear it would work. At the same time, the bureaucrats in charge of Phoenix downplayed many of the red flags that kept cropping up during the testing and analysis phases of the project during the months leading to launch.
While IBM had significant experience with building pay systems, Public Services — or, more specifically, its accounting, banking and compensation branch (ABC) — was managing the overall project. This meant the ABC branch gave IBM specific tasks to do such as software patches and modules designed to make the government’s pay information flow smoothly and accurately through the Phoenix calculation engine. To date, IBM has been awarded more than $200 million for its efforts. Qualtrough believes the government should have mapped out the end result and left it up to the subcontractors to figure out how to get there.
“With respect to the IBM contract,” Qualtrough said last fall, “as we modernize procurement, we need to focus on outcomes ... instead of asking people to do a series of things that will hopefully end up where we want it to go.” THE ROAD FORWARD
Can the backlog be reduced on current plans?
The latest data isn’t encouraging. Public Services revealed earlier this month the number of pay transactions in the queue at Miramichi had reached an all-time high of 633,000 in late January — affecting more than half the departments’ workforce.
It’s not clear yet whether this epic information technology disaster will trigger real reform in how government manages its big contracts.
A common failing in federal government IT projects has been an unhealthy emphasis on achieving savings over that of building systems that actually work. Certainly that was a principal cause of the flawed launch in 2011 of Shared Services Canada, the central agency in charge of building and running the government’s electronic backbone.
Qualtrough has been refreshingly blunt discussing the Phoenix file. “While we didn’t create this problem, it’s ours to fix,” she told members of a Commons committee in late November. “Once launched, Phoenix’s problems ran so deep that it took time to understand what was wrong and to identify solutions to stabilize the system.”
One of the lingering questions in the Phoenix debacle has inevitably been about blame. With a project so crucial to the everyday functioning of government going off the rails, who should bear responsibility?
The answer is, many share the blame — the Conservative government for booking savings before these were realized, the Liberals for not paying close enough attention to all the warning signs in advance of Phoenix’s launch, the bureaucracy for a failure to properly exercise oversight, for taking at face value the assurances of the Public Services managers who had been pushing Phoenix for years.
There’s a popular saying in the tech industry. “Good information technology costs a lot but bad IT costs a fortune.” That’s because fixing a broken system is so much more difficult than doing it right the first time.
That hard-learned insight could be what pushes the Liberals later this year to rebuild Phoenix from scratch.