NEW PS EMAIL GLITCH A SIGN OF AGING SYSTEM
A sign of the times. A government official recently sent the following email:
“Treasury Board has been experiencing significant email issues since October 6, 2016,” the note to other federal departments read. “We recommend that you contact employees by phone until further notice. We will update you when the issues are resolved.”
Taken on its own, this was not a big deal. A few hundred Treasury Board employees went without email service off and on for two weeks. The workers also lost email history and calendar appointments, which were being restored over the weekend using backups.
“Shared Services Canada has been working 24 hours a day to resolve the initial and any subsequent email issues,” noted Stephanie Richardson, a spokesperson for Shared Services, the federal department responsible for computer services, including email and data centres.
But there was an unsettling message in this latest email glitch because it involved existing infrastructure. Most of the headlines involving government email over the past two years have focused on problems with the new system ordered from Bell Canada in 2013 but with delivery nowhere in sight. Now the reasonably reliable older systems are showing their age and may require additional investment.
Until the Bell system is up and running, most of the government’s 600,000-plus email boxes must be hosted in aging data centres, most of them located in the National Capital Region. This means government email service is increasingly vulnerable to breakdowns in data centres generally or the parts specific to the delivery of email.
It’s not clear yet what triggered the Treasury Board email issues. “The root cause of the incident is still under investigation,” Richardson explained in an email.
The email breakdown followed routine maintenance at one of the government’s older data centres. When power was restored, email servers failed. Richardson confirmed that a “corrupted database” was involved. Regardless, this is not good news for Shared Services, the five-year-old computer services agency.
Early in its mandate, Shared Services launched a $400-million project to consolidate 63 email systems across government into one. Bell Canada and CGI were to have completed the switch to canada.ca email addresses by March 31, 2015, but the contractors have been wrestling with a revised design, faulty hardware and other technical difficulties. While hardware issues, according to Shared Services, have been fixed, an overall solution has proved elusive.
Shared Services a year ago halted the migration of the new system to federal employees until Bell could resolve the difficulties. Not quite 70,000 government email boxes — less than 15 per cent of the total — currently use the new addresses.
In the meantime, of course the volume of data being sent on email is significantly higher each year, putting further strain on the legacy infrastructure.
The Conservative government estimated the new system designed by Bell would save taxpayers $50 million annually by taking advantage of economies of scale. Those savings have not been realized.
To date, most of the cost of the delay has been borne by Bell, which under its contract will not be paid until it delivers the goods. Bell declined to comment for this article.
Richardson noted in her email: “There are outstanding contracted, functional issues that need to be resolved before new migrations can resume.” Shared Services did not elaborate or provide a revised deadline.
While Bell is on the hook for fixing the new email system, Shared Services bears responsibility for keeping the older email systems running — and may soon have to invest significant sums.
Shared Services faces a similar issue with another of its responsibilities involving the consolidation of more than 500 data centres into a handful of modern ones. Because Shared Services was late to begin the streamlining process, it is diverting significant sums into keeping legacy data centres running even as it’s investing in new data centres in Barrie and Borden.
The Liberals early this year said they would kick in an extra $384 million over two years to keep aging electronic infrastructure going. It’s not known what percentage of this amount, if any, was earmarked specifically for email service — or if the Liberals anticipated continuing long delays in the new email system.
This much is clear: Attempting simultaneously to modernize large computer networks and keep the old ones running appears to be a hugely difficult job in government. Federal workers may have to get used to using the telephone in the months ahead.