The surgeon who wants to connect

you to the Internet with a brain implant If you can conceive it, you can tweet it

“What I think is so interesting is that the future is always flying cars,” Leuthardt says, handing the resident his felt-tipped marker and picking up a scalpel. “They captured the dystopian component: they talk about biology, the replicants. But they missed big chunks of the future. Where were the neural prosthetics?”

It’s a topic that Leuthardt, a 44-yearold scientist and brain surgeon, has spent a lot of time imagining. In addition to his duties as a neurosurgeon at Washington University in St Louis in the USA, he has published two novels and written an award-winning play aimed at “preparing society for the changes ahead”. In his first novel, a techno-thriller called Reddevil 4, 90 per cent of human beings have elected to get computer hardware implanted directly into their brains. This allows a seamless connection between people and computers, and a wide array of sensory experiences without leaving home. Leuthardt believes that in the next several decades, such implants will be like plastic surgery or tattoos, undertaken with hardly a second thought.

“I cut people open for a job,” he notes. “So it’s not hard to imagine.”

But Leuthardt has done far more than just imagine this future. He specialises in operating on patients with intractable epilepsy, all of whom must spend several days before their main surgery with electrodes implanted on their cortex as computers aggregate information about the neural firing patterns that precede their seizures. During this period, they are confined to a hospital bed and are often extremely bored. About 15 years ago, Leuthardt had an epiphany: why not recruit them to serve as experimental subjects? It would both ease their tedium and help bring his dreams closer to reality.

Leuthardt began designing tasks for them to do. Then he analysed their brain signals to see what he might learn about how the brain encodes our thoughts and intentions and how such signals might be used to control external devices. Was the data he had access to sufficiently robust to describe intended movement? Could he listen in on a person’s internal verbal monologues? Is it possible to decode cognition itself?

Though the answers to some of these questions were far from conclusive, they were encouraging. Encouraging enough to instill in Leuthardt the certitude of a true believer; one who might sound like a crackpot, were he not a brain surgeon who deals in the life-and-death realm of the operating room, where there is no room for hubris or delusion. Leuthardt knows better than most that brain surgery is dangerous, scary and difficult for the patient. But his understanding of the brain has also given him a clear-eyed view of its inherent limitations and the potential of technology to help overcome them. Once the rest of the world understands the promise, he insists – and once the technologies progress – the human race will do what it has always done. It will evolve. This time with the help of chips implanted in our heads.

“A true fluid neural integration is going to happen,” Leuthardt says. “It’s just a matter of when. If it’s 10 or 100 years in the grand scheme of things, it’s a material development in the course of human history.”

Leuthardt is by no means the only one with exotic ambitions for what are known as brain-computer interfaces. Last March Elon Musk, a founder of Tesla and Spacex, launched Neuralink, a venture aiming to create devices that facilitate mind-machine melds. Facebook’s Mark Zuckerberg has expressed similar dreams, and last spring his company revealed that it has 60 engineers working on building interfaces that would let you type using just your mind. Bryan Johnson, the founder of the online payment system Braintree, is using his fortune to fund Kernel, a company that aims to develop neuroprosthetics he hopes will eventually boost intelligence, memory, and more.

These plans, however, are all in their early phases and have been shrouded in secrecy, making it hard to assess how much progress has been made or whether the goals are even remotely realistic. The challenges of brain-computer interfaces are myriad. The kinds of devices that people like Musk and Zuckerberg are talking about won’t just require better hardware to facilitate seamless mechanical connection and communication between silicon computers and the messy grey matter of the human brain. They’ll also have to have sufficient computational power to make sense out of the mass of data produced at any given moment as many of the brain’s nearly 100 billion neurons fire. One other thing: we still don’t know the code the brain uses. We will have to, in other words, learn how to read people’s minds.

But Leuthardt, for one, expects he will live to see it. “At the pace at which technology changes, it’s not inconceivable to think that in a 20-year time frame everything in a cellphone could be put into a grain of rice,” he says. “That could be put into your head in a minimally invasive way, and would be able to perform the computations necessary to be a really effective brain-computer interface.”

DECODING THE BRAIN

Scientists have long known that the firing of our neurons is what allows us to move, feel, and think. But breaking the code by which neurons talk to each other and the rest of the body – developing the capacity to actually listen in and make sense of precisely how it is that brain cells allow us to function – has long stood as one of neuroscience’s most daunting tasks.

In the early 1980s, an engineer named Apostolos Georgopoulos, at Johns Hopkins University in Maryland in the US, paved the way for the current revolution in braincomputer interfaces. Georgopoulos identified neurons in the higher-level processing areas of the motor cortex that fired prior to specific kinds of movement, such as a flick of the wrist to the right, or a downward thrust with the arm. What made Georgopoulos’s discovery so important was that you could record these signals and use them to predict the direction and intensity of the movements. Some of these neuronal firing patterns guided the behaviour of scores of lowerlevel neurons working together to move the individual muscles and, ultimately, a limb.

Using arrays of dozens of electrodes to track these high-level signals, Georgopoulos demonstrated that he could predict not just which way a monkey would move a joystick in three-dimensional space, but even the velocity of the movement and how it would change over time.

It was, it seemed clear, precisely the kind of data one might use to give a paralysed patient mind control over a prosthetic device. Which is the task that one of Georgopoulos’s protégés, Andrew Schwartz, took on in the 1990s. By the late 1990s, Schwartz, who is currently a neurobiologist at the University of Pittsburgh, had implanted electrodes in the brains of monkeys and begun to demonstrate that it was indeed possible to train them to control robotic limbs just by thinking.

Leuthardt, in St Louis to do a neurosurgery residency at Washington University in 1999, was inspired by such work: when he needed to decide how to spend a mandated year-long research break, he knew exactly what he wanted to focus on. Schwartz’s initial success had convinced Leuthardt that science fiction was on the verge of becoming reality. Scientists were finally taking the first tentative steps towards the melding of man and machine. Leuthardt wanted to be part of the coming revolution.

He thought he might devote his year to studying the problem of scarring in mice: over time, the single electrodes that Schwartz and others implanted as part of this work caused inflammatory reactions, or ended up sheathed in brain cells and immobilised. But when Leuthardt and his adviser sat down to map out a plan, the two came up with a better idea. Why not see if they might be able to use a different brain recording technique altogether?

“We were like, ‘Hey, we’ve got humans with electrodes in them all the time!’“Leuthardt says. “Why don’t we just do some experiments with them?”

Georgopoulos and Schwartz had collected their data using a technique that relies on microelectrodes next to the cell membranes of individual neurons to detect voltage changes. The electrodes Leuthardt used, which are implanted before surgery in epilepsy patients, were far larger and were placed on the surface of the cortex, under the scalp, on strips of plastic, where they recorded the signals emanating from hundred of thousands of neurons at the same time. To install them, Leuthardt performed an initial operation in which he removed the top of the skull, cut through the dura (the brain’s outermost membrane), and placed the electrodes directly on top of the brain. Then he connected them to wires that snaked out of the patient’s head in a bundle and plugged into machinery that could analyse the brain signals.

Such electrodes had been used successfully for decades to identify the exact origin in the brain of an epilepsy patient’s intractable seizures. After the initial surgery, the patient stops taking anti-seizure medication, which will eventually prompt an epileptic episode; and the data about its physical source helps doctors like Leuthardt decide which section of the brain to resect in order to forestall future episodes.

But many were sceptical that the electrodes would yield enough information to control a prosthetic. To help find out, Leuthardt recruited Gerwin Schalk, a computer scientist at the Wadsworth Centre, a public-health laboratory of the

New York State Department of Health. Progress was swift. Within a few years of testing, Leuthardt’s patients had shown the capacity to play Space Invaders – moving a virtual spaceship left and right – simply by thinking. Then they moved a cursor in three-dimensional space on a screen.

In 2006, after a speech on this work at a conference, Schalk was approached by Elmar Schmeisser, a programme manager at the US Army Research Office. Schmeisser had in mind something far more complex. He wanted to find out if it were possible to decode ‘imagined speech’; words not vocalised, but simply spoken silently in one’s mind. Schmeisser, also a science fiction fan, had long dreamt of creating a ‘thought helmet’ that could detect a soldier’s imagined speech and transmit it wirelessly to a fellow soldier’s earpiece.

Leuthardt recruited 12 bedridden epilepsy patients, confined to their rooms and bored as they waited to have seizures, and presented each one with 36 words that had a relatively simple consonant-vowelconsonant structure, such as ‘bet’, ‘bat’, ‘beat’ and ‘boot’. He asked the patients to say the words out loud and then to simply imagine saying them – conveying the instructions visually (written on a computer screen), with no audio and again vocally, with no video, to make sure that he could identify incoming sensory signals in the brain. Then he shipped the data to Schalk for analysis.

Schalk’s software relies on pattern recognition algorithms. His programs can be trained to recognise the activation patterns of groups of neurons associated with a given task or thought. With a minimum of 50 to 200 electrodes, each one producing 1 000 readings per second, the programs must churn through a dizzying number of variables. The more electrodes and the smaller the population of neurons per electrode, the better the chance of detecting meaningful patterns if sufficient computing power can be brought to bear to sort out irrelevant noise.

“The more resolution the better, but at the minimum it’s about 50 000 numbers a second,” Schalk says. “You have to extract the one thing you are really interested in. That’s not so straightforward.”

Schalk’s results, however, were surprisingly robust. As one might expect, when Leuthardt’s subjects vocalised a word, the data indicated activity in the areas of the motor cortex associated with the muscles that produce speech. The auditory cortex, and an area in its vicinity long believed to be associated with speech processing, were also active at the exact same moments. Remarkably, there were similar yet slightly different activation patterns even when the subjects only imagined the words silently.

Schalk, Leuthardt and others involved in the project believe they have found the little voice that we hear in our mind when we imagine speaking. The system has never been perfect: after years of effort and refinements to his algorithms, Schalk’s program guesses correctly 45 per cent of the time. But rather than attempt to push that number higher (they expect performance to improve with better sensors), Schalk and Leuthardt have focused on decoding increasingly complex components of speech.

In recent years, Schalk has continued to extend the findings on real and imagined speech (he can tell whether a subject is imagining speaking Martin Luther King Jr’s ‘I Have a Dream’ speech or Lincoln’s Gettysburg Address). Leuthardt, meanwhile, has attempted to push on into the

next realm: identifying the way the brain encodes intellectual concepts across different regions.

The data on that effort is not published yet, “but the honest truth is we’re still trying to make sense of it,” Leuthardt says. His lab, he acknowledges, may be approaching the limits of what’s possible using current technologies.

“The moment we got early evidence that we could decode intentions,” Leuthardt says, “I knew it was on.”

Soon after obtaining those results, Leuthardt took seven days off to write, visualise the future and think about both short- and long-term goals. At the top of the list of things to do, he decided, was preparing humanity for what’s coming, a job that is still very much in progress.

With sufficient funding, Leuthardt insists, reclining in a chair in his office after performing surgery, he could already create a prosthetic implant for a general market that would allow someone to use a computer and control a cursor in threedimensional space. Users could also do things like turn lights on and off, or turn heat up and down, using their thoughts alone. They might even be able to experience artificially induced tactile sensations and access some rudimentary means of turning imagined speech into text. “With current technology, I could make an implant, but how many people are going to want that now?” he says. “I think it’s very important to take practical, short interval steps to get people moved along the pathway towards this road of the longterm vision.”

To that end, Leuthardt founded Neurolutions, a company aimed at demonstrating that there is a market, even today, for rudimentary devices that link mind and machine and at beginning to use the technology to help people. NeuroLutions has raised several million so far; also, a non-invasive brain interface for stroke victims who have lost function on one side is currently in human trials. The device consists of brain-monitoring electrodes that sit on the scalp and are attached to an arm orthosis; it can detect a neural signature for intended movement before the signal reaches the motor area of the brain. The neural signals are on the opposite side of the brain from the area usually destroyed by the stroke and thus are usually spared any damage. By detecting them, amplifying them, and using them to control a device that moves the paralysed limb, Leuthardt has found, he can actually help a patient regain independent control over the limb far faster and more effectively than is possible with any approach currently on the market. Importantly, the device can be used without brain surgery.

Though the technology is decidedly modest compared with Leuthardt’s grand designs for the future, he believes this is an area where he can meaningfully transform people’s lives right now. There are about 700 000 new stroke patients in the US each year, and the most common motor impairment is a paralysed hand. Finding a way to help more of them regain function – and demonstrating that he can do it faster and more effectively – would not only demonstrate the power of braincomputer interfaces, but also meet a huge medical need.

Using non-invasive electrodes that sit on the outside of the scalp makes the invention much less off-putting for

patients, but it also imposes severe limitations. The voltage signals coming from brain cells may be muffled as they travel through the scalp to reach the sensors, and they may be diffused as they pass through bone. Either makes them harder to detect and their origins harder to interpret.

Leuthardt can achieve far more transformative feats using his implanted electrodes that sit directly on the cortex of the brain. But he has learnt through painful experience that elective brain surgery is a tough sell, not just with patients, but with investors as well. When he and Schalk founded Neurolutions, in 2008, they hoped to restore movement to the paralysed by bringing just such an interface to market. But the investment community wasn’t interested. For one thing, neuroscientist-led start-ups have been testing brain-computer interfaces for more than a decade, but have had little success in turning the technology into a viable treatment for paralysed patients. The population of potential patients is limited – at least compared with some of the other conditions being targeted by medical-device start-ups competing for venture capital. (Roughly 40 000 people in the US have complete quadriplegia.) And most of the tasks that could be accomplished using such an interface can already be handled with non-invasive devices. Even most locked-in patients can still blink an eye or perhaps wiggle a finger. Methods that rely on this residual movement can be used to input data or move a wheelchair without the danger, recovery time, or psychological wherewithal involved in implanting electrodes directly on one’s cortex.

So after their initial fund-raising efforts failed, Leuthardt and Schalk set their sights on a more modest goal. Unexpectedly, they found that many patients continued to recover additional function even after the orthosis was removed – extending to, for instance, fine motor control of their fingers. Often, it turned out, all the patients needed was a little push. Then, once new neural pathways were established, the brain continued to remodel and expand them so that they could convey more complex motor commands to the hand.

The initial success Leuthardt expects in these patients, he hopes, will encourage some to move on to a more robust invasive system. “A couple years down the road you might say, ‘You know what? For that non- invasive version, you can get this much benefit, but I think that now, given the science that we know and everything, we can give you this much more benefit,’“he says. “We can enhance your function even more.”

Leuthardt is so eager for the world to share his passion for the technology’s potentially transformative effects that he has also sought to engage the public through art. In addition to writing his novels and play, he is working on a podcast and Youtube series with a fellow neurosurgeon, in which the two discuss technology and philosophy over coffee and doughnuts.

In Leuthardt’s first book, Reddevil 4, one character uses his “cortical prosthetic” to experience hiking the Himalayas while sitting on his couch. Another, a police detective, confers telepathically with a colleague about how to question a murder suspect standing right in front of them. Every character has instant access to all the knowledge in the world’s libraries and can access it as quickly as a person can think any spontaneous thought. No one ever has to be alone, and our bodies no longer limit us. On the flip side, everyone’s brain is vulnerable to computer viruses that can turn people into psychopaths.

Leuthardt acknowledges that at present, we still lack the power to record and stimulate the number of neurons it would take to replicate these visions. But he claims his conversations with some Silicon Valley investors have only fuelled his optimism that we’re on the brink of an innovation explosion.

Schalk is a little less sanguine. He’s sceptical that Facebook, Musk and others are adding much of their own to the quest for a better interface.

“They are not going to do anything different from the scientific community by itself,” Schalk says. “Maybe something is going to come of it, but it’s not like they have this new thing that nobody else has.”

Schalk says it’s “very, very obvious” that in the next five to 10 years some form of brain-computer interface will be used to rehabilitate victims of strokes, spinal cord injuries, chronic pain, and other disorders. But he compares the current recording techniques to the IBM computers of the 1960s, saying that they are now “archaic”. For the technology to reach its true long-term potential, he believes, a new sort of brain-scanning technology will be needed – something that can read far more neurons at once.

“What you really want is to be able to listen to the brain and talk to the brain in a way that the brain cannot distinguish from the way it communicates internally, and we can’t do that right now,” Schalk says. “We really don’t know how to do it at this point. But it’s also obvious to me that it is going to happen. And if and when that happens, our lives are going to change, and our lives are going to change in a way that is completely unprecedented.”

Where and when the breakthroughs will come from is unclear. After decades of research and progress, many of the same technological challenges remain daunting. Still, the progress in neuroscience and computer hardware and software makes the outcome – at least to true believers – inevitable.

At the very least, says Leuthardt, the buzz emanating from Silicon Valley has generated “real excitement and real thinking about brain-computer interfaces being a practical reality”. That, he says, is “something we haven’t seen before”. And though he acknowledges that if this turns out to be hype it could “set the field back a decade or two”, nothing, he believes, will stop us from reaching the ultimate goal: a technology that will allow us to transcend the cognitive and physical limitations previous generations of humankind have taken for granted.

“It’s going to happen,” he insists. “This has the potential to alter the evolutionary direction of the human race.”

Eric Leuthardt believes that in the near future implants will be like plastic surgery or tattoos, undertaken with hardly a second thought.

Leuthardt has learnt through painful experience that elective brain surgery is a tough sell, not just with patients, but with investors as well.

Leuthardt specialises in operating on patients with intractable epilepsy, all of whom must spend several days before their main surgery with electrodes implanted on their cortex as computers aggregate information about neural firing patterns preceding their seizures. Opposite: Using non-invasive electrodes on the outside of the scalp is less off-putting , but imposes severe limitations.