AMAZON TROUNCES RIVALS IN BATTLE OF THE SHOPPING ‘BOTS’

them. “Nothing has changed recently in how we manage bots on our site,” she said. Still, she said, “we prioritise humans over bots as needed.”

Bots can slow down a website, a big motivator for retailers to block them. Reuters interviewed 21 people familiar with bots and how they are deployed, including current and former Wal-Mart employees, former Amazon employees and outside specialists. Many spoke only on condition of anonymity because they were not authorised to discuss the issues publicly. The company’s technological edge has been good for its profit margin, and it’s proving a winning formula for investors. Shares of the internet powerhouse have risen about 15-fold since the market’s bottom in March 2009, while the S&P 500 has more than tripled in value. Amazon hit $100 billion in annual sales in 2015 — faster than any company in history, it said.

Brave new world

Bot-driven pricing has represented a massive change for the retail industry since Amazon helped pioneer the practice more than a decade ago. Traditionally, brick-and-mortar stores changed prices no more than weekly because of the time and expense needed to swap labels by hand. In the world of e-commerce, though, retailers update prices with ease, sometimes multiple times a day, helped by algorithms that consider inventory levels, sales forecasts and rivals’ pricing data. To stay in the game, companies such as online wholesaler Boxed, based in New York, depend on a variety of methods including bots to ensure they do not lag others’ price moves for even 20 minutes. “That’s like a lifetime during Christmas,” said Chief Executive Chieh Huang, whose company sells bulk staples like toilet paper and pet food. “If we’re not decently priced, we’ll see it almost immediately” in sales declines.

Disguised as humans

Using bots to view massive amounts of data on public websites — a pro- cess known as crawling or scraping — has many purposes. Alphabet’s Google, for example, constantly crawls the Web to gather information for its search engine results and to sell ads. In e-commerce, though, the use of bots has developed into a cat-and-mouse game. Companies try to thwart the practice on their own websites while aiming to penetrate their competitors’ defences. Third-party services abound to help less-savvy retailers. To protect data from rivals, some retail websites use what’s known as a “CAPTCHA” — typically a distorted string of letters and numbers that humans can read but most bots can’t. Amazon shies away from the practice because it annoys some customers.

For merchants seeking to evade such defences, disguising their computer programs as real shoppers is key. Some pricing technology experts have programmed computer cursors to meander through a Web page in the way a person might, instead of going directly to the prized data. Another technique is to use multiple computer addresses so that retailers cannot track a barrage of clicks to a single source. “It is an arms race,” said Keith Anderson, a senior vice president at e-commerce analytics firm Profitero, based in Ireland. “Every week or every month, there’s some new approach from both sides.”

Amazon’s maneuver that halted Wal-Mart in January took aim at a specialised Web browser called PhantomJS. Unlike, say, Internet Explorer, this browser is designed specifically for programmers — a telltale clue that its users are not typical shoppers. Amazon put up a digital curtain to hide its listings from PhantomJS users, according to three people familiar with the situation. It was unclear how the move, which was not aimed at Wal-Mart in particular, affected other companies.

Tests conducted in recent weeks for Reuters show that among major US retail chains, Amazon had by far the most sophisticated bot detection in place, both for its home page and for two popular items selected by

Swarming with bots

Despite Amazon’s capabilities, the sheer volume of crawling on its site is staggering. At times, as many as 80 per cent of the clicks on Amazon product listings have been from bots, people familiar with the matter say, compared with just a third or more of the traffic on other large sites.

In addition to rivals seeking price data, that traffic includes bots from university researchers studying competition, search engines, advertising services and even fraudsters trying to break into Amazon accounts.

For Wal-Mart, a small group in Silicon Valley directs its automated pricing strategy while dozens of engineers in India and around the world handle the code, current and former Wal-Mart employees said. Amazon had about 40 engineers who would covertly extract and organise rivals’ data with bots as of several years ago, one of the people interviewed said. Amazon did not discuss the size or structure of its teams working with bots. According to one US patent application, Amazon is working on encryption technology that would force bots, but not humans, to solve a complicated algorithm to gain access to its Web pages.

“Amazon has both the competency to detect bot traffic and the wherewithal to do something about it,” said Scott Jacobson, a former Amazon manager and now managing director of Madrona Venture Group. That “isn’t the case for most retailers.”