Bleak moments in Black Friday frenzy
System problems stymied shoppers and blocked sales
Online retailers were caught with their pants down by the shopping frenzy that gripped South Africa on Black Friday last week.
Falling on the day after the US holiday of Thanksgiving, and marking the beginning of Christmas shopping there, the idea has rapidly caught on in South Africa.
Most local online retailers now offer markdowns on a wide range of products and have seen it become their biggest shopping day of the year.
However, just minutes after midnight on Friday last week, South African shoppers found they couldn’t get into websites such as CNA, takealot.com and DionWired.
Many payments made through Nedbank could not be processed, and gifting site NetFlorist saw a slowdown on its site as traffic intensified.
Takealot.com co-CEO and chief technology officer Willem van Biljon explained: “Our website relies on a technology called Kubernetes to manage network traffic and hardware scaling . . . The enormous increase in traffic exposed an unknown bug in the component of Kubernetes that is responsible for routing network traffic to the right parts of the system. This caused very slow response times on internal services . . .
“We tested our systems, anticipating the huge demand, but despite months of diligent work, this unknown latent bug in our system presented itself — exposed by the traffic volumes that our testing couldn’t have anticipated.”
Fibre damage
Nedbank, too, had an unforeseen event. Fred Swanepoel, Nedbank group chief information officer, said the outage was not related to the increase in traffic. “Our outage, which happened at 8.32am, was a direct result of partial damage to a fibre line between our two data centres. The repair took 89 minutes and did not affect the full spectrum of our transactions. Only some transactions were affected, but we were fully operational from 10am and overall volumes for the day were up on last year.”
Takealot.com recorded R87-million in sales on the day, 55% up on the same day last year and its biggest single day of sales yet.
However, a week earlier CEO Kim Reid had forecast sales of between R80-million and R130-million, which suggests that the outage had a significant impact.
Takealot said it was also bedevilled by problems with the SABC TV-licensing validation needed to conclude television sales.
Samsung TV sets were the initial bestseller on the day, with 162 units purchased in the first hour, before site problems and licensing issues choked sales.
Julie-Anne Walsh, takealot.com’s chief marketing officer, said: “Additionally, retailers across South Africa were affected by various banks declining an unusually high percentage of credit-card transactions.”
An error message on the DionWired website indicated flaws in implementation, although DionWired marketing manager Ahmed Gangat blamed high traffic. He said the chain’s parent company, Massmart, would invest heavily in new systems that would prevent a repeat of Black Friday’s problems.
“Massmart is investing in worldclass e-commerce infrastructure, including SAP’s Hybris Commerce, which will be implemented in the coming months at DionWired.
“With the new e-commerce platform and server capacity, customers can expect an improved shopping experience,” said Gangat.
NetFlorist CEO Ryan Bacher said its site remained up throughout the day although heavy traffic slowed it down. There were no specials on offer, so there was no spike in purchases. Black Friday contributed only about 1.5% of annual sales; the site peaks on Valentine’s Day, which makes up as much as 8% of annual turnover.
“We were unintentionally and very fortunately ready for the idea of Black Friday, whereas most other online retailers hadn’t needed to manage that kind of scale over just one or two days,” Bacher said.
Cloud services
NetFlorist uses cloud hosting, via Amazon Web Services, which allows it to scale up or down very quickly.
“This allows us to predict the traffic and get ready the day before, but then to up our capacity quickly if we underestimated the size of the increase, or decrease if we overestimated, thereby saving costs,” Bacher said.
Industry observers have all pointed out that appropriate cloud services are readily available to major websites to prevent heavy traffic from bringing the sites crashing down.
The example often cited is Ama- zon’s self-styled annual Prime Day, June 11, which this year grew by 60% over the previous year and saw 3.5 million toys sold. All without a noticeable impact on site performance.
Amazon uses the same cloud services, from its subsidiary, Amazon Web Services, that are available to any retailer anywhere in the world.
Retailers were affected by various banks declining credit-card transactions Julie-Anne Walsh Takealot.com chief marketing officer