NUCLEAR REACTION!
How the bomb led to the rise of supercomputers
As anyone who’s ever tried to work out a restaurant bill, including drinks, taxes, and tip, already knows, some math is difficult. Expand that by several orders of magnitude, and suddenly you’re simulating the effects of a nuclear bomb, or protein folding, or calculating how many oil rigs to send up the Bering Strait before winter, and your needs go beyond mere computers. You need a supercomputer.
Established in the 1960s, supercomputers initially relied on vector processors before changing into the massively parallel machines we see today in the form of Japan’s Fugaku (7,630,848 ARM processor cores producing 442 petaflops) and IBM’s Summit (202,752 POWER9 CPU cores, plus 27,648 Nvidia Tesla V100 GPUs, producing 200 petaflops).
But how did we get to these monsters? And what are we using them for? The answers to that used to lie in physics, especially the explodey kind that can level a city. More recently,
however, things like organic chemistry and climate modeling have taken precedence. The computers themselves are on a knife-edge, as the last drops of performance are squeezed out of traditional architectures and materials, and the search begins for new ones.
This, then, is the story of the supercomputer, and its contribution to human civilization.
DEFINE SUPER
What exactly is a supercomputer? Apple tried to market its G4 line as ‘personal supercomputers’ at around the turn of the millennium, but there’s more to it than merely having multiple cores (although that certainly helps). Supercomputers are defined as being large, expensive, and with performance that hugely outstrips the mainstream.
Apple’s claim starts to make more sense when you compare the 20 gigaflops of performance reached by the hottest, most expensive, dual-processor, GPU-equipped G4 PowerMac to the four gigaflops of the average early-2000s Pentium 4. For context, Control Data’s CDC Cyber supercomputer ran at 16 gigaflops in 1981, a figure reached by ARMv8 chips in today’s high-end cell phones.
Before supercomputers there were simply computers, though some of them were definitely super. After World War II, many countries found ways to automate code-breaking and other intensive mathematical tasks, such as those involved in building nuclear weapons. So let’s begin in 1945, and the ENIAC.
This programmable mass of valves and relays was designed to compute artillery trajectories, and it could do a calculation in 30 seconds that would take a human 20 hours. Its first test run, however, was commandeered by John von Neumann of the Los Alamos National Laboratory and consisted of calculations for producing a hydrogen bomb. ENIAC was programmed, and provided its output, using punch cards, and a single Los Alamos run used a million cards.
ENIAC was upgraded throughout its life, and when finally switched off in 1956 (having run continuously since 1947, pausing only to replace the tubes that blew approximately every two days) it contained 18,000 vacuum tubes, 7,200 crystal diodes, 1,500 relays, 70,000 resistors, 10,000 capacitors, and around half a million joints, all of them soldered by hand. It weighed 27 tons, and took up 1,800 sq ft, while sucking down 150kW of power. Its computational cycle took 200 microseconds to complete, during which time it could write a number to a register, read a number from a register, or add/ subtract two numbers. Multiplication took the number of digits plus four cycles, so multiplying 10-digit numbers took 14 cycles, or 357 per second.
Early computers owed much to the design of the ENIAC and the British Colossus. Breaking enemy codes was still a high priority, as was finding ever more efficient ways to blow things up with both high explosives and pieces of uranium. It’s around the early 1960s, though, that things such as processors and memory became recognizable. Take the UNIVAC LARC, or Livermore Advanced Research Computer, a dual-CPU design delivered in 1960 to help make nuclear bombs, and the fastest computer in the world until 1961. The LARC weighed 52 tons and could add two numbers in four microseconds.
TECH EXPLOSIONS
There was a burst of computer development in the early 1950s. IBM had been in the game since WWII, its Harvard Mark I electromechanical machine coming online in 1944, with one of its first programs, again run by Von Neumann
to aid the Manhattan Project. In 1961, Big Blue would release the IBM 7030, known as Stretch, the first transistorized supercomputer and the fastest computer in the world until 1964 (a customized version of Stretch, known as Harvest, was used by the NSA for cryptanalysis from 1962 until 1976, when mechanical parts in its magnetic tape system wore out).
Von Neumann was behind another computer, sometimes named for him, at the Institute for Advanced Study in Princeton, which was in operation until 1958. This computer was the basis for a new generation, including the IBM 701, the ILLIAC I at the University of Illinois, and Los Alamos’ alarmingly named MANIAC I, which became the first computer to beat a human at a chess-like game (on a 6x6 board with no bishops to suit the limitations of the machine).
The ILLIAC line would become highly influential, with ILLIAC II completed before Stretch and the open-source nature of its design leading to suspicions of ‘borrowing’. Certainly, the two computers are early examples of a pipelined design, and feature heavy use of transistors instead of vacuum tubes. When faculty member Donald B Gillies programmed ILLIAC II to search for Mersenne prime numbers, it found three new ones.
The first massively parallel computer design was the ILLIAC 4 (the MkIII machine was designed to detect nuclear particles in bubble chambers, and was destroyed in a fire). Originally meant to have 256 floating-point units, and four CPUs, it could run at a billion operations a second, but due to budget constraints, only one CPU and 64 FPUs were completed in 1972. Even so, this quarter-computer still managed 50 Mflops, making it the fastest in the world. ILLIAC 4 was also the first networked supercomputer, being connected to the ARPAnet in 1975, a year before the Cray-1.
THE GENIUS OF CRAY
There’s a name that shook the world of supercomputing. The whole story would be nothing without the influence of Seymour Cray, who joined Engineering Research Associates in 1951 to build code-breaking machines, but left in 1957 to co-found Control Data, and would later found Cray Research, Inc (now part of HP) in 1972. Cray was gifted, some might say eccentric (he dug a tunnel under his home, and attributed his successes to the advice of ‘elves’ who visited him there), and was given to spending hours in deep concentration to solve a problem. The reason early Cray machines are circular or C-shaped, for example, is so that every electrical interconnect can be the same length, so an electrical signal always takes the same amount of time to travel down each one.
In his book The Super men: Seymour C ray and the Technical Wizards Behind the Supercomputer, author Charles J Murray relates this anecdote: “After a rare speech at the National Center for Atmospheric Research in Boulder, Colorado, in 1976, programmers in the
audience had suddenly fallen silent when Cray offered to answer questions. He stood there for several minutes, waiting for their queries, but none came. When he left, the head of NCAR’s computing division chided the programmers: ‘Why didn’t someone raise a hand?’ After a few moments, one programmer replied, ‘How do you talk to God?’”
John Levesque, the Director of Cray’s Supercomputer Center of Excellence based at Los Alamos National Laboratory (home of the Manhattan Project and much US nuclear research since), remembers him: “He was very shy. It was extremely difficult to get him to give a talk. But when he did, it was outstanding. When I met him, all I did was shake his hand. He didn’t say anything.”
Levesque has worked on all the greats of early supercomputing and remains in the sector today. He began his career working on the ILLIAC 4: “The UK had a machine called the DAP [International Computers Limited’s Distributed Array Processor, the first commercially available parallel computer, delivered to its first customer in 1979] at the same time as the ILLIAC, and similar to it, but it didn’t have the support that the ILLIAC did. I know there are a lot of people who felt it was a complete failure, because it never went into production, and they only really developed a quarter of the machine.”
This wasn’t Levesque’s first brush with supercomputers, however. “I started working at Sandia National Laboratories in 1968. And I was working for the Underground Physics Department, and then I went to work at the Air Force weapons lab in Albuquerque for three years. Then I worked for a government contractor in 1972 in southern California called R&D Associates, and while I was there, I got a contract from DARPA [Defense Advanced Research Projects Agency, the branch of the DoD interested in new materials and ideas] to monitor ILLIAC 4 code development efforts.
“In 1976, Cray gave a computer to Los Alamos—it was serial number one— with the intent of convincing them that the machine would be extremely good for their applications. So because of the experience that my team had on the ILLIAC 4, and back to this point we probably had a team of five or six people, Los Alamos hired us to port and optimize their principal application to that Cray and so in 1977, we started working on that. We had a cross-compiler but, initially, there was no vectorizing compiler, we were using what’s known as Cray vector primitives to load registers, perform operations, and store results. When Cray came out with a vectorizing compiler, we wrote Fortran do-loops that could be optimized and used vector instructions on the Cray-1.”
“Los Alamos was primarily using CDC 7600s [from Control Data] which were also designed by Seymour Cray”, Levesque says. Peaking at 36 Mflops, the 7600 had a clock cycle of 27.5ns, for a speed of 36.4MHz on its strange 60-bit processor, and its base configuration sold for $5 million in 1967. It wasn’t Cray’s first successful design, the Control Data 6000 series had outperformed IBM’s Stretch by a factor of three, and the 6600 was the fastest computer in the world from 1964 to 1969, when it lost the crown to the 7600.
The last of his Control Data designs was the 8600, essentially four 7600s welded together running at a faster cycle speed of 8ns; it was never released, and problems with its design and budget prompted Cray to leave the company in 1972.
He didn’t go far, setting up Cray Research in the same Wisconsin town as Control Data. The Cray-1, announced in 1975, was 5.5 tons of C-shaped genius running at 80MHz, with a cycle speed of 12.5ns on the faster, inside edge of the C (where interconnects could be shorter), it was slower on paper than the bruteforce power of the 8600 but made up for it with cunning design, full 64bit processing and limited parallelism for a peak output of 160 Mflops. The National Center for Atmospheric Research (NCAR) estimates it was 4.5 times faster than the 7600.
Levesque says Cray ‘gave’ the machine to Los Alamos, but there was a bidding war between the New Mexico lab and its rival, the Lawrence Livermore National Laboratory at the University of California, Berkeley. Los Alamos won and received the machine for a six-month trial. NCAR got one in 1977 (which it used until 1989), the NSA had one for code-breaking, possibly even before Los Alamos—a machine with serial number zero ended up in the British Atomic Weapons Establishment. In total, 80 Cray-1 machines were sold, for up to $8 million each.
VECTOR PROCESSING
The key to the Cray-1’s success was its combination of all-round high performance with vector processing. Traditional scientific code was written as loops in Fortran, where the operands (the objects of a mathematical operation) were processed one at a time. Vector processing opened up the abilities of parallel computers by carrying out the same operation on multiple pairs of operands at once, making it much more efficient. The trick was to convert the Fortran loops to vectors, which could be done automatically.
“The whole idea is that the user is writing in Fortran,” says Levesque. “And so the compiler has to identify where it can use a ray operation. And the logical place is in a Fortran do-loop. The compiler
has to determine if all of the operations are independent of one another. The main thing is that each add, or multiply, or multiply add, needs to be able to be done for the full extent of the do-loop. So the compiler determines that it can do that, and it generates vector instructions. There are things like NCAR’s code, which had ambiguous subscripts and was extremely difficult to vectorize because of the way they wrote their loops. Cray told them, ‘You have to rewrite your loops’. And NCAR said, ‘We don’t have enough manpower to rewrite our code’. And so, Cray came up with the very first compiler directive, which is a common line to any other compiler. It was DIR $ I V E P, and it stood for ‘ignore vector dependencies’. Once this directive was placed in front of the loops of compiler-generated vector code, it ran fast. NCAR bought a Cray.”
“Then, since we had experience on the Cray, we got numerous contracts to help people with moving their code to the Cray. We struck oil in the early ‘80s. At that time, there was a company by the name of ARCO that had been bought by BP. And there was a fella who gave a talk at a conference who said that his solution technique could not be vectorized. And so I got the code, vectorized it, and showed him, and ARCO ended up giving us a contract to port all of their reservoir simulators to the Cray.
It was kinda interesting because one time, the guy called me up and said, ‘It’s very important that you have this one code
running fast’. And I asked why, and he said, ‘Oh, we have to figure out how many oil drilling rigs to move up through the Bering Strait before it freezes over’. And we were successful. We even gave Cray training courses on the Cray.”
COOLING COMPONENTS
The Cray-1 generated a lot of heat, so its cooling system was almost as lovingly designed as its integrated circuits. Circuit boards were placed back to back, with a sheet of copper in between. This spread heat to its edges, where it met stainless steel pipes containing liquid Freon, which carried the heat away to a cooling system mounted under the C-shaped main unit.
There were two updated versions of the Cray-1, the 1S, and 1M. These had larger memories, faster cycle times, the addition of MOS RAM and even solid-state storage. The Cray itself was supervised by a second computer, Data General Supernova or Eclipse models, which fed them their operating systems (Cray OS at first, then later a version of UNIX) at boot time and could act as a front-end—these changed through the years as well. Further Cray machines, developed by a different team under designer Steve Chen (“Outgoing and personable,” according to Levesque), were released, each taking their turn as the fastest machine in the world, but it wasn’t until the Cray-2 in 1985 that Cray himself returned to the top.
The first Cray design with multiple CPUs, four custom vector processors, the Cray-2 used novel 3D wiring techniques and a ‘waterfall’ cooler that’s practically a work of art, but the new design had trouble beating 1982’s Cray X-MP, developed from the Cray-1, and its successor the Y-MP. Sales were poor. The Cray-3, meant to be 12 times as powerful as the Cray-2, saw Seymour Cray and his company part ways once again, with Cray research continuing to work on the Cray C90 (a development of Y-MP tech that ran at 244MHz/4.1ns in 1991) and a spin-off, the Cray Computer Company, taking the Cray-3 tech and its single customer, the Lawrence Livermore Laboratory, with it. The laboratory would later cancel its order. Cray lent the sole Cray-3 built to NCAR as a demonstrator, but bankruptcy followed.
This didn’t stop Cray, whose Cray-4 scaled from four to 64 processors each running at 1GHz. A 16-processor system came with 8GB of memory, provided 32 Gflops, and cost $11 million. Nobody was buying. The company stopped work in 1994, and Cray died following a car crash in 1996, aged 71. “It seems impossible to exaggerate the effect he had on the industry,” said Joel Birnham, former CTO of Hewlett Packard, in tribute to Cray. “Many of the things that high-performance computers now do routinely were at the farthest edge of credibility when Seymour envisioned them.”
His company would pass through a number of hands, including those of Silicon Graphics, Sun Microsystems, and Tera (which renamed itself Cray, Inc after the acquisition). Hewlett Packard Enterprise (HPE) acquired the company for $1.3 billion in 2019, and today is building the LUMI supercomputer in Finland with a theoretical maximum performance of 550 petaflops, slotting into the top five fastest computers in the world.
ONE HORSE RACE?
While Cray was dominating, others hadn’t been sitting around. MIT’s Connection Machine, the result of work into alternative computer architectures that strayed from the orthodoxy put together by Von Neumann, started with 1985’s CM-1 which had up to 65,536 individual processors, each extremely simple, processing one bit at a time, and a striking visual design for its casing that led to many programs being written just to blink its many LEDs. The CM range ended up, in the form of the CM-5, on top of the world computer speed rankings in 1993, with a 1,024-processor machine putting out 131.0 Gflops. One even appears in the Jurassic Park movie (though it’s a Cray in the novel). Also topping the rankings in 1993 was Intel, cramming up to 4,000 i860 RISC processors into Paragon for up to 143.4 Gflops. Fujitsu’s Numerical Wind Tunnel supercomputer used 166 vector processors to gain the top spot in 1994 with a peak speed of 170 Gflops.
IBM, in particular, had been releasing mainframes and minicomputers all through the 1950s to the ‘80s, as well as its PC line from the 1981 launch of the 5150 through the XT, AT, and more. A lot of what are thought of as its brands are actually made by Lenovo, coincidentally also the maker of 184 of the
top 500 supercomputers, the company to which IBM sold its PC business in 2005. Mainframes, being suited for bulk data processing, are not the same as supercomputers, which tend to concentrate on one extremely complex task at a time.
Blue Gene, and its predecessor the Scalable POWERparallel (SP) series, changed all that in the 1990s. An example of cluster computing, computers that work together so closely they can be considered a single unit, they mark the emergence of IBM’s Power architecture (known to Mac owners as PowerPC G3, G4, and G5, beloved of Xbox 360 and Nintendo Wii gamers, and also trundling about on Mars in the Curiosity and Perseverance rovers).
ASCI
Nuclear weapons rear their head too at around this time, with the creation of the Advanced Simulation and Computing Initiative (ASCI), a supercomputing program to extend the lifetime of the US’s aging stockpile of nukes by simulating the way a nuclear weapon will react under different conditions. Essentially, if we leave these things in the cupboard for 50 more years, will they still go bang if we want them to?
“A big change in computing came around in the early 1990s, and that was due to the Nuclear Test Ban Treaty,” says Jim Sexton, an IBM Fellow and Director of Data Centric Systems at the company. “In the 90s, it became clear that computing would allow them to manage the stockpile of nuclear weapons with simulations rather than having to go and actually explode the bombs. When you have a mission of that order, and that complexity, people get very focused on developing computing systems to support your work. Up until then, the way people had been designing computers was just playing around, trying different things.
“ASCI kicked off,” Sexton adds, “and has had a sequence of supercomputers developed ever since. There were a
number of players in the early days: IBM was providing some of the computers, Cray was providing some of the computers, a couple of other names too, and what quickly happened is that there was an insatiable demand for computing power. And there was a limit on how much power you could deliver to a laboratory to run a computer. We’re up to 40 or 50 megawatts today, the biggest ones in the world are our own national labs in the US.”
This hunger for power has led to a new consideration in supercomputer design— efficiency. Born out of a five-year plan to build a massively parallel computer to address protein folding and other biomolecular phenomena, the project had a secondary aim of exploring new ideas in parallel computing architecture, the problem with supercomputers being that new ones are hard to simulate on existing hardware. In the early 2000s, there was a bit of a plateau in computer speeds. NEC’s Earth Simulator had been at the top of the rankings from March 2002 until November 2004, its 35.86 teraflops pushing out ASCI White’s 7.226 teraflops that had been top for a year previously.
“Blue Gene was designed to be powerefficient,” says Sexton of a machine that forced its way to the top with 70.72 teraflops before losing the title to another IBM machine, Roadrunner at Los Alamos, in June 2008. “We were building systems with 100,000 CPU cores in the early 2000s, and that was unheard of at the time, to be able to get that many computer cores to work coherently together.
“And then towards the end of that time, what emerged was GPU acceleration. It turns out that GPUs are actually quite power-efficient, and a significant strain on the design of computers is your ability to manage power. A lot of systems design is about how to get more and more computing power into a system, but staying within a given power. The Blue Gene architecture used incredibly simple, lightweight, low power cores, but by the time we got to around 2014 or 2015, it became clear we would not be able to continue to improve performance with that technology.”
MODERN DAY
Trading power consumption for speed, Blue Gene/L (originally ‘Blue Light’) contains two PowerPC 440 cores running at 700MHz with floating-point accelerators in each node, with 1,024 nodes in each 19in rack, up to a minimum of 64 racks (65,536 nodes). A lightweight Linux OS further pares back the overhead. Blue Gene/P further developed this, with its PowerPC 450 cores running at 850MHz and with twice the chip-to-chip bandwidth of the L model, but with four cores per node and 4,096 cores per rack.
Blue Gene/Q dispenses with PowerPC, using instead IBM’s A2 open-source architecture. This means 18-core chips with four threads per core and a speed of 1.6GHz. Each rack contains 1,024 compute nodes, 16,384 user cores (16 are available, the 17th runs the OS, and the 18th is either a redundant spare or there to increase manufacturing yields) and 16 TB RAM.
Blue Gene is also notable for sprouting Deep Blue, the computer that beat Grandmaster Gary Kasparov at chess in 1996. With its roots reaching back to 1985, and earlier attempts to teach a computer to play chess, Deep Blue is more of an ASIC (Application Specific Integrated Circuit) as it uses specialized chips alongside IBM Power2 processors to brute-force solutions to chess problems rather than displaying modern artificial intelligence.
“We are at the limit of what it is physically possible to program and manage, so then we did a switch to work with Nvidia and built the fastest computers in the US National Labs: Summit and Sierra. They have tightly coupled CPUs and GPUs, and are at 200 petaflops today.”
Planned machines will produce more than an exaflop of computing power, comparable to that put out by Folding@ home during the great push to discover vulnerabilities in the protein-coding of the SARS-COV2 virus. “We’re at a disruptive time in computing,” says Sexton. “We’re running out of technology, and yet the demands of computing continue to grow. The ability for AI to do things, to get you more access to knowledge, it requires more and more compute. If you think of a self-driving car, something a lot of people talk about, getting a computer to drive a car is actually quite a lot, so you’re seeing huge amounts of concrete everywhere.
“The history from 1995 to today and into the next few years has been driven primarily by the mission activities that have supported nuclear weapons,” he continues, “and all the science that goes around understanding materials, understanding climate, understanding weather, understanding biological systems, so we’re now at the point where you can actually do serious simulation. Every five years, we were seeing a tenfold increase in peak performance. We’re now at an exaflop, so we’ve gone three orders of magnitude, in terms of performance, in 15 years. It’s absolutely amazing that that’s possible, but hard to see how it’s going to continue.
“On the other hand, the computing that we do is changing drastically now. It’s less and less about trying to focus on understanding basic physics and more and more about AI, and data analytics. It’s been a very interesting ride, but I don’t think the future of computing is going to be around nuclear weapons. I think we’ve reached the limits of traditional computing, but we have all these new capabilities coming too.”
FUGAKU KING
This may explain how Fugaku, a system made of an ungodly number of ARM-based CPU cores and no GPUs, was able to overtake Summit in June 2020 to take the top spot on the big list of really fast computers. Competition between nations for the prestige of hosting the world’s greatest supercomputer may still be a factor, but we may have to look elsewhere, such as distributed computing systems, to catch the really big numbers in future.
Or we could change the way we design our computers. A resurgence in analog computing, maybe? “We as humans, we don’t really work as precise ones and zeros,” says Sexton. “Our brain computes analog, and it does a pretty good job. When you look at a lot of the domain out there, do I need a computer of ones and zeroes to drive a car? Would it be better to go to an interesting, analog, future?
“An analog computer deals with a range of values,” he continues. “It doesn’t give you a precise binary value for something. And then there’s a different way of calculating, and you do that by having different materials analyzing the different programs or occasional model. In-memory computing is another one. More and more challenging computing is moving away from the old Von Neumann model, where you have your data in memory and move it to a computer to compute. We’re looking for ways around that because a significant amount of the power in the computer is going into moving data.”
Whatever happens, as quantum computers approach from left-field or new materials other than silicon become more common in microchip manufacture, we may never leave the age of the supercomputer behind. They may just fade into the background of our society, giant machines that help when great scientific problems need solving and spend the rest of their time predicting the weather. But that’s another story.