YEARLY “THE SKY IS FALLING” EXERCISES GENERATE DETAILED ACTION PLANS IN CASE THE MARKET DRIES UP OR THE ECONOMY SLOWS DOWN. WHEN COVID-19 STRUCK, THESE CONTINGENCY PLANS HELPED DATABRICKS MANAGE EXTREME TURBULENCE AS YEARS OF DIGITAL TRANSFORMATION WERE C

Moving from one seedy Stockholm neighborhood to another, Ghodsi and his younger sister constantly had to change schools and make new friends. He credits the wide range of human interactions he encountered for his social deftness today.

The first glimmers of his engineering genius also came early. Ghodsi’s parents could ill afford to buy their kids new presents. For Ali, they found a deal on a used Commodore 64, a home computer with a cassette player that could load video games, but that was so cheap precisely because the cassette deck was irreparably broken. Curious, the fourth grader began reading manuals and soon figured out how to code his own games. “I was one of those geeks that got super sucked into tech,” Ghodsi says with a smile.

That obsession continued into college at Mid Sweden University, in the quiet industrial town of Sundsvall, where Ghodsi stayed an extra year to get master’s degrees in computer engineering and business administration. He then earned a place at KTH Royal Institute of Technology, the Swedish equivalent of MIT or CalTech, where he received a Ph.D. in computer science in 2006.

In 2009, the 30-year-old Ghodsi came to the United States as a visiting scholar at UC Berkeley, where he got his first glimpse of Silicon Valley. Despite the collapse of the dot-com bubble nine years prior and the ongoing financial crisis, innovation was at a peak. Facebook was only five years old and not yet public. Airbnb and Uber were in their first year of existence. And a few upstart companies were just beginning to boast that their technology was able to beat humans at narrow tasks.

“It turns out that if you dust off the neural network algorithms from the ’70s, but you use way more data than ever before and modern hardware, the results start becoming superhuman,” Ghodsi says.

Unlike many foreign-born future entrepreneurs (see “Unwelcoming Shores,” page 100), Ghodsi was able to stay in America on a series of “extraordinary ability” visas. Once at Berkeley, he joined forces with Matei Zaharia, then a 24-yearold Ph.D. student, on a project to build a software engine used for data processing that they dubbed Spark. The duo wanted to replicate what the big tech companies were doing with neural networks, without the complex interface.

“Our group was one of the first to look at how to make it easy to work with very large data sets for people whose main interest in life is not software engineering,” Zaharia says.

Spark turned out to be good— very good. It set a world record for speed of data sorting in 2014 and won Zaharia an award for the year’s best computer science dissertation. Eager for companies to use their tool, they released the code for free, but soon realized it wasn’t gaining any significant traction.

Over a series of meetings at cheap hole-in-the-wall Indian restaurants beginning in 2012, a core group of seven academics agreed to start Databricks. Entrepreneurial wisdom came from the Romania-born Zaharia’s thesis advisors, Scott Shenker and fellow Romanian Ion Stoica, two well-respected academics. Stoica was an exec at $300 million video streaming startup Conviva, while Shenker had been the first CEO of Nicira, a networking firm sold in 2012 to VMware for about $1.3 billion. Stoica would be CEO, and Zaharia the chief technologist. Shenker, who joined the board rather than working full-time for the company, arranged the initial meeting between Ben Horowitz, an early Nicira investor, and the researchers, who nearly balked at the idea.

“We thought to ourselves and said, ‘We don’t want to take his money because he’s not a researcher,’ ” Ghodsi says. “We’d wanted to get some seed funding, maybe raise a couple hundred thousand dollars and then just code away for a year and see what we could get.”

On a summer day at their new office space one block off Cal’s campus, the founders sat idly in their conference room, pondering how much money would be too much to turn down. An hour after their scheduled meeting time, Horowitz arrived. “Traffic is brutal to this Berkeley place,” he said, before cutting to the chase: “I’m not going to negotiate with you guys; I’m just going to give you an offer, so take it or leave it.” The offer: $14 million in capital at close to a $50 million valuation. It was too much to turn down.

“These kinds of ideas have a time limit on them,” Horowitz explains. “For most people, starting with seed money is the right thing to do, but not for these guys.”

The following Friday afternoon, the ragtag group of geeks—between ages 24 and 48 at the time—were again in their office. But this time one of them was mashing the “F5” key to refresh the webpage showing Databricks’ bank-account balance. “Then, at some point, it said ‘14-zero

“IF YOU DUST OFF THE AI ALGORITHMS FROM THE ’70S BUT YOU USE WAY MORE DATA THAN EVER AND MODERN HARDWARE, THE RESULTS START BECOMING SUPERHUMAN.”

zero-zero-zero-zero-zero,’ ” Ghodsi recalls. “We were totally blown away. I was making $58,000 or $57,000, so this was a lot of money.”

STOICA

quickly brought on NEA partner Sonsini, himself a Cal alumnus, as the company’s second venture investor, thanks to a connection dating back to Stoica’s time at Conviva. Sonsini’s firm was Conviva’s largest shareholder, and the investor bought into Databricks— close to zero revenue in 2014—on potential alone. (“I was fully planning on leading the first financing too, but Horowitz just took it right from under my nose,” he says.) The $33 million investment boosted the startup to a $250 million valuation, just 13 months after it began.

Says Ghodsi: “2015 was the year when Spark was the hottest thing since apple pie.” In anticipation of accelerated growth, Databricks moved its headquarters from its modest Berkeley office to the 13th floor of a skyscraper in San Francisco’s Financial District. The team didn’t care about the unlucky floor number. “We got it for a cheaper price, maybe for that reason, and we thought, ‘That’s great,’” Ghodsi says. And yet, within months, bad fortune appeared to be manifesting itself.

“We were taking too long to figure out go to market,” Horowitz says. Bigger fish like Amazon Web Services and Cloudera were bypassing Databricks and integrating Spark into their own products. “All of our competitors started talking about how they loved Spark,” Ghodsi says. “But we had almost no revenue.”

Ghodsi immediately enacted three measures when he took over in January 2016. First: Bulk up the sales force with people who knew how to pitch to enterprise chief information officers. Second: Build out Databricks’ C-suite with “people who have done it before.” Third: Create proprietary portions of the software so those hotshot salespeople would have something to sell. At the time, the technology was too open-sourced. “We didn’t have anything that special because [other companies] had all of Spark for free,” Ghodsi says.

Within a year, the executive team was entirely new, filled with tech veterans who had helped steer successful exits at companies like AppDynamics and Alteryx. Ghodsi offered old executives the chance to stay on if they were willing to report to their replacement. “If people were smart enough, they put their egos aside,” he says. Only two of seven quit.

The new Databricks platform proved popular because it harnessed the core Spark engine better than the copycats did. “The others barely understood Spark,” Ghodsi says. And since the founders were the creators of Spark, they were building and incorporating new features into Databricks long before they were released to the public: “We’re always a year or two ahead of everyone else.”

Sales picked up rapidly, reaching $12 million in 2016. “The first year was so spectacular that it was obvious Ali should be CEO after that,” Horowitz says. Confidence restored, the high-profile investor sent a recommendation letter to Microsoft CEO Satya Nadella, proclaiming Databricks to be at the vanguard of a revolution in AI and big data. Nadella responded instantly. “He cc’d a bunch of these super high-up Microsoft employees, and suddenly they were extremely eager to do a close partnership with us,” says Ghodsi, who had tried in vain to get in touch with the Microsoft chief for years. The two companies collaborated to integrate Databricks directly into Azure, Microsoft’s $59.5 billion (calendar 2020 sales) cloud service. Microsoft’s sales force now touts “Azure Databricks” when pitching to prospective clients, and in 2019 the Redmond giant invested in Ghodsi’s company.

Ghodsi says there’s little mystery about how Databricks works: Simply feed massive amounts of data into algorithms to train AI models on how to analyze and make predictions with the data. “It’s not like a deep secret sauce that no one knows about.”

But competitors, slower to start, are often forced play catch-up on either data processing or artificial intelligence tools. “As academics, we were just thinking big: ‘Where does the future go?’ It was almost like sci-fi,” Ghodsi says.

All the while, Databricks has been busy expanding well past Spark. In 2018, it released MLflow to manage machine learning projects, and a year later announced Delta Lake, which turns existing data lakes into lakehouses, so that companies don’t have to start from scratch. Both have proven to be hits. Spark, Ghodsi says, is only 5% of the reason customers use Databricks.

“Every other open-source company is still whatever open-source [product] they started with. Databricks is so far beyond Spark,” says Horowitz, whose early investment in the company helped him place at No. 38 on Forbes’ 2021 Midas List of top tech investors. Assuming Andreessen Horowitz has held onto its full stake, its initial $14 million investment is now worth $8.9 billion.

IN

February, Databricks raised $1 billion to cement its position as one of the world’s most valuable startups. The fresh funds give it a massive war chest as it competes to win contracts from the planet’s biggest companies. No competitor looms larger than Snowflake, the newly public, best-in-class data warehousing provider, which as recently as three years ago maintained a business partnership with Databricks. Even today, 70% of Databricks users are also Snowflake customers, according to Piper Sandler tech analyst Brent Bracelin. But the two are starting to throw haymakers.

“Snowflake is obviously an unbelievable company in a great position, but they’ve got a professional CEO,” Horowitz says. “How much longer is he going to be there? Probably not much longer.” With a founding team that’s still fully engaged, “nobody in enterprise software is going to out-innovate Databricks.”

“Every single thing that [Databricks has] done that I think is a good architectural choice in the last three or four years, Snowflake did it eight years ago,” retorts Christian Kleinerman, senior vice president of product for Snowflake, throwing shade at Databricks’ new warehousing features. Still, he admits Snowflake’s next act, a hub where users can feed their data into AI tools, will be used in “very similar” ways as Databricks.

In any case, as Ghodsi sees it, Snowflake is only one of four main competitors. The others are the cloud’s Big Three: Amazon, Microsoft and Google. It makes for a tricky situation, as all three are Databricks investors. But they all have long been constructing their own data analytics suites.

Ghodsi is cognizant of threats from both the established tech giants and new disruptors alike. “I think most people who know me will tell you I’m the most paranoid CEO they’ve ever met,” he says, paying homage to longtime Intel chief Andy Grove’s mantra.

“It comes natural for me, because I grew up in a war. If you’re seeing people die on the streets as a kid, you know anything can change at any given time.” Ghodsi puts his employees through yearly “the sky is falling” exercises—creating detailed action plans in case the market dries up or the economy slows down.

When Covid-19 struck, those contingency plans helped Databricks manage extreme turbulence as the pandemic compressed years of digital transformation into mere months. It’s opening offices and building an army of techies and salespeople across the globe, from Australia and India to Japan and Sweden.

Back in the Bay Area, Ghodsi is preoccupied with something more immediate: his son’s kidney cancer. After a late-night visit to the emergency room, Ghodsi reflects on the present. Technology and data have already advanced to the point of helping Ghodsi and his wife discover a genetic predisposition to the disease in their son before tumors appeared. Firms like Databricks are helping pharmaceutical and health-care companies with the next step: using AI to speed the discovery of new treatments.

“If this would have happened 10 or 15 years ago, he would have died. You wouldn’t have found it until he’s vomiting and the cancer’s spread everywhere,” Ghodsi says. “This kind of technology can help.”