Signal failure: Who's to blame for Spark debacle
Finger pointed at Akamai but pundit says NZ telco should have managed woes
The finger has been pointed at content delivery giant Akamai for Saturday night’s Spark Sport streaming fail — which saw thousands receive substandard video, and the decision made to simulcast the All BlacksSouth Africa game free on TVNZ.
But a leading industry expert says it looks more as if Spark should carry the can on all levels, from technical to management.
Spark chief executive Jolie Hodson yesterday told the Herald: “We identified . . . a config issue of the video stream coming in from the US, which we rectified over the 24 hours that followed.
“An international partner that we work with, the way they were configuring streams to come into New Zealand — it was getting congested in a particular channel.
“In simple form, if you think of it as lanes on the motorway. They had too much going down one particular lane.”
Hodson did not want to name the international partner concerned. But another person close to the situation bluntly identified it as Akamai Technologies: the US-based content delivery giant whose former clients include Netflix, and whose current roster includes the likes of NBC and Fox Sports.
And based on a July 8 media briefing by Spark Sport head Jeff Latch, it could only be Akamai.
Latch detailed the live Rugby World Cup feed would come down from Japan via satellite or fibre to Spark’s broadcast partner TVNZ.
TVNZ would then add its production elements, including ads, then send the feed up to Spark’s US-based streaming partner, iStream planet, which encoded the video feed for a multitude of different devices.
iStream would hand off the feed to Akamai, which would deliver the Spark stream to a never-disclosed number of points around NZ.
A person close to the situation said Akamai had failed to optimise its delivery for a typical Saturday night, where Kiwis were loading up Netflix.
Regardless, shouldn’t Spark have been monitoring everything and ensuring Akamai was following the plan for what was always going to be a huge Saturday night, and very likely Spark Sport’s busiest day (given the All Blacks face soft competition in the rest of their pool games and likely quarter-final, and the semis and final will be free on TVNZ)?
The Herald asked Hodson if Spark was passing the buck. Wasn’t it up to Spark to manage everything?
“It wasn’t a management issue. It was our partner and how they had configured the channels,” she said.
However, the problem seems to be not as simple as an international partner dropping the ball.
A manager at an internet service provider told the Herald it appeared Akamai had “broken peering protocol” and taken an unusual approach to configuring its servers. He said there had poor communication.
“Spark need to be more upfront on the technical side with the rest of us.”
He forwarded the Herald the transcript of a chat between ISP owners and managers to prove his points.
He added, “There is also a lot of contempt towards Spark because they had the rest of the industry invest tonnes of money and then they did a ‘classic Telecom’ and screwed everyone on the wholesale deal.”
In the build-up to the World Cup, Spark met with Vodafone, 2degrees, Vocus (owner of Slingshot and Orcon) and other ISPs, who agreed to accelerate network upgrades to facilitate better RWC streaming in return for a keenly priced wholesale deal to resell Spark Sport Tournament passes.
But after Spark launched its earlybird pricing before its putative partners had a wholesale deal on the table, the spirit of co-operation evaporated. Bad blood emerged.
And industry expert John Humphrey says it just doesn’t wash to blame Akamai.
Humphrey, now an international consultant, used to be a Telecom network and business strategy manager and a co-founder of the Pacific Fibre bid to build a new transpacific cable. “Blaming Akamai sounds far too easy,” Humphrey said. “Spark’s bandwidth-provisioning planning and network monitoring and responsiveness let them down. They just underestimated the network demand or ran into bandwidth restrictions that they could not handle at the time.”
Humphrey said while there could have been an international issue, “It’s Spark bandwidth to NZ, not Akamai’s. Akamai just runs the CDN [content delivery network] capacity which is easily incremented on the fly. It was up to Spark to manage this . . . Southern Cross and Hawaii have a lot of spare bandwidth available.
“Capacity from the game to the undersea cable head-ends was possibly managed by Akamai and this is what Spark may have been referring to. If that was the case, Spark should have been checking Akamai and overseeing what they were doing.”
Humphrey added, “Part of the challenge for Spark is that this was feed by unicast IP video streams from a CDN — somewhere in NZ — to each subscriber. A lot of simultaneous unicast streams occupying a lot of bandwidth. This is why it is so easy for Sky to do this and so hard for Spark. Sky can use multicast from the satellite to each home with no stress.
“The true power of satellite is in its one-to-many usage.
“This is a common problem in many countries. The US has an OTT streaming challenge. Up to 70 per cent of their domestic internet usage in the evening is OTT IP Unicast video streams and if you are in an area with poor mid- and last-mile connections, you get buffering, lag, stop/start etc that we don’t often experience in NZ.
“NZ has one of the best terrestrial networks in the world now but even it can get overloaded.
“It was not quite good enough and they are learning a hard PR lesson as a result,” Humphrey said.” The next All Black games will show whether they have sorted this out or not.”
Asked if Spark would be seeking redress, Hodson said her company had worked constructively with its international partner to implement a fix by Sunday afternoon.
She was confident the problem would not be repeated, and Spark was returning to streaming-only coverage for tonight’s Wales-Georgia clash.
Akamai declined to comment.
It is important for New Zealanders, our economy, and the New Zealand Rugby Union to give Spark a break in the early stages of this new Rugby World Cup online service, with its fixable teething problems.
There is so much to gain from changing away from Sky dominance, which had captured the sports bodies and the public, to their disadvantage. There’s a lot more at stake than just Saturday night footie.
Internationally, sports bodies historically gave the broadcast rights to single providers, such as Sky in New Zealand. There’s a big trend though to the sport controlling and producing the content and providing it to more content carriers. This is a change from content providers competing for the market (to be appointed as the single provider) to competing in the market (that is, competing against other providers).
That spreads the risk, but, more importantly, the competition between providers, vying for viewers, will improve quality and options for Kiwis, and lead to additional features such as multiple parallel channels showing different views of the game, interactive viewing, and so on. That’s what competition in the market does.
I spent a few months acting for parties successfully opposing Vodafone and Sky merging. As is the position internationally, a telco (here, Vodafone) controlling live premium sport (here, live tests) can distort the telco market in its favour, away from competitors, thereby leading to market failure and poor outcomes for New Zealanders. That’s the main reason the Commerce Commission did not allow the merger. Spark getting the sole longer-term rights over live premium sport such as All Blacks tests can have the same effect, for the reasons Spark itself argued when opposing that merger.
Having the telcos competing with their own live premium rugby packages is especially important in terms of encouraging uptake of new fast broadband services, including in rural areas.
At first sight, the idea that live rugby tests could have such a dramatic effect on telecommunications, and the broader NZ economy, seems a stretch. But that is the international experience, reflected in the decision to stop the Vodafone/Sky merger.
The good news is the NZRU can win out too if it is smart. It has the monopoly over the live AB tests. By being stuck with Sky, the NZRU monopoly power was essentially given to Sky, reducing NZRU revenues. NZRU has the monopoly, so use it, don’t give it away!
With live premium sports rights up for renewal, this is a defining moment. If the NZRU doesn’t go for multiple carriers and paths now, they’ll be regulated within four years, as night follows day.