THE BRAIN GAIN
AN ARTIFICIAL-INTELLIGENCE REVOLUTION WAS BORN IN A VANCOUVER HOTEL ROOM. BUT WAS IT A MISSED OPPORTUNITY FOR CANADA?
Mel Silverman walked over to a whiteboard and picked up a marker, listing all the academic disciplines that the band of renegade scientists asking him for money represented.
Assembled there 12 years ago at Vancouver’s Metropolitan Hotel was a group of about 15 people, ranging from computer scientists to biologists to experimental engineers. What united them was their interest in a concept that was, at the time, generally perceived as the domain of the lunatic fringe.
They believed it was possible to teach a machine to learn the same way a child does, through artificial neural networks that mimic the function of the human brain. In the process of teaching a machine to learn like a human, they figured there was likely a lot to discover about how humans learn as well.
The consensus among most computer scientists at the time was that this was nuts. The way to get a computer to do something was to program it to do it, not ask it to learn the task itself. If he had been a computer scientist, Silverman probably would have thanked them for their time and moved on.
But Silverman, who was in charge of recommending what programs to green- light at the not- for- profit, mostly publicly funded Canadian Institute for Advanced Research (CIFAR), was a physician by training — a profession he says has a tendency to question authority. He had noticed how the group went quiet and listened reverently when Geoffrey Hinton, a University of Toronto researcher lured to Canada by CIFAR decades earlier, spoke. He liked the ambitious scope of the problem they were trying to tackle and the persistence of the group willing to risk professional ostracism to tackle it.
Silverman asked the group why CIFAR should agree to fund them. Sebastian Seung, an American computational neuroscientist who is now a professor at Princeton University, responded.
“He said, ‘ Well, we’re kind of weird,’ ” Silverman recalled. “‘If CIFAR is looking for a highrisk, adventurous kind of group that’s willing to step out of its usual areas of comfort, this is the group.’ I said, ‘ OK. That sounds cool.’ ”
Silverman persuaded CIFAR to give that band of self- identified weirdos about $ 10 million over 10 years, making it pretty much the only organization at the time to back the research of artificial neural networks.
Today, it’s clear they were anything but nuts.
The world’s biggest tech companies are currently spending billions of dollars exploring the technology, which researchers have used to train computers to recognize handwritten characters, understand speech and even identify cats in YouTube videos.
Neural networks are being used to help doctors interpret medical images and give better treatment advice. They’re making machines employers can teach to do factory work. Just about any industry that wants to make the best possible use of vast amounts of data could potentially benefit from artificial intelligence (AI).
As a research project, CIFAR’s mostly taxpayer- funded 2004 program has clearly been a winner. But its long-term legacy for the Canadian economy is not as certain. Some of the key researchers and their students have founded some interesting Canadian startups, but as tends to be the case in the tech world, the centre of the action has shifted to California.
Hinton now splits his time between the University of Toronto and Google Inc., while his former postdoctoral student Yann LeCun is now head of AI at Facebook Inc.
Did Canada miss the chance to become ground zero for a new high-tech industry?
“I would look at it another way. I would say, here’s an example of a Canadian success,” Silverman said. “Life is what happens while you’re still planning. Individuals need to follow their own dreams and look at where they can have the greatest impact.”
The impact of Hinton’s re- search is only beginning to be felt in the wider world.
CIFAR brought the Britishborn researcher to Canada in the 1980s. He was one of the youngest members of the program of about 30 artificial intelligence researchers and the only one working on neural networks.
“Occasionally it was a bit annoying, because I was convinced my approach was right and they all thought it was awful,” Hinton said. “It was seen as kind of disreputable and obviously hopeless.”
Undaunted, Hinton went about assembling like- minded people. He invited LeCun, a French student with a background in electrical engineering, to do post- doctoral studies with him at the University of To- ronto. He also collaborated with Yoshua Bengio, a computer scientist who is now a professor at the University of Montreal.
But Hinton said his faith never wavered for one simple reason: “Because the brain must be doing it somehow.”
Children learn the difference between cats and dogs by looking at lots of cats, looking at lots of dogs, processing the incoming visual information and teaching themselves to recognize distinguishing features — not because their parents plugged USB keys into their heads filled with code describing different breeds.
Similarly, artificial neural networks are inspired by the way neurons in the brain pass information to each other. Mimicking the way a child learns, scientists can train an artificial neural network to recognize a cat by inputting millions of images of cats and also millions of images of other things, such as dogs. The computer starts to learn to learn the features that distinguish cats from dogs until it can look at an image it hasn’t seen before and determine which category to put it in — cat or dog — with the help of some feedback. ( Terrifyingly, the people who build neural networks aren’t always sure exactly how this process works.)
Progress in the field was slow until the mid-’ 00s because researchers underestimated the amount of computing power necessary to handle the vast quantities of data involved.
By around that time, LeCun said, his colleagues were getting a little tired of labouring in obscurity. “You don’t want to be seen as completely crazy and be marginalized. Also, you have students who need to find jobs. You want them to publish things that will see the light of day,” he said. “I wanted to work in this area again, reboot the whole effort, be involved.”
Funding from CIFAR turned out to be just what they needed. It provided the means to help the diverse group of researchers Hinton had helped assemble from around the world meet and collaborate.
Breakthroughs f ol l owed quickly. A 2006 paper in the journal Science described a method for training multi- layered neural networks, similar to how the brain processes a signal from the retina and passes it on to another neuron for analysis. This allowed machines to learn more abstract concepts and finally started generating interest from academia at large.
Industry started to take note in 2012. Hinton’s group showed neural networks could do a better job recognizing objects than other artificial intelligence techniques.
Google snapped up Hinton in short order and acquired DNNResearch, a company he had founded with two graduate students. In June, Twitter Inc. purchased Whetlab, another deep learning startup founded by a group of Hinton’s students.
In 2010, Hinton said, he approached Research in Motion — now BlackBerry Ltd. — to ask if the company was interested in having an intern to work on speech recognition. The Waterloo- based company passed — perhaps missing the chance to get a head start on a technology that its major competitors are now investing in heavily.
“It wasn’t unreasonable of them,” Hinton said. “You need a big team to do all the engineering and they didn’t have that team in place. It wasn’t something they were betting on.”
Alan Bernstein, chief executive of CIFAR, said that’s the thing about the type of fundamental research the organization funds. It’s impossible to predict what the researchers are going to learn and whether it will be useful to industry at all. If a program’s discoveries happen to be useful to Silicon Valley while Canadian companies and venture capitalists pass — well, so it goes.
CIFAR’s mandate is to bring the world’s pre- eminent minds together to tackle big, complicated questions. Judged on that criteria, the 2004 neural computation program was a clear success.
“Where they started was not really, how do we build an industry, but rather, how can we understand how brains learn?” Bernstein said. “If you knew the path you were going to be on to solve a big problem in research, in a real sense, the problem’s already solved.”
Bengio, the University of Montreal professor who looked up to Hinton in graduate school, is now a co- director of CIFAR’s neural computation program. He said if the public wants to encourage Canadian economic and research excellence, the worst thing anyone could do would be to attach strings to funding — for example, requiring that researchers stay in Canada or only work with Canadian companies.
“The problem isn’t that we are giving some money to researchers who are outside Canada. The problem is that ... we don’t have the environment that will keep them here,” he said. “It’s not by closing in that we can keep the best people. It’s by opening up.”
HE SAID, ‘WELL, WE’RE KIND OF WEIRD. IF CIFAR IS LOOKING FOR A HIGH-RISK, ADVENTUROUS KIND OF GROUP THAT’S WILLING TO STEP OUT OF ITS USUAL AREAS OF COMFORT, THIS IS THE GROUP.’ I SAID, ‘OK. THAT SOUNDS COOL.’
— MEL SILVERMAN ON A KEY 2004 MEETING LIFE IS WHAT HAPPENS WHILE YOU’RE STILL PLANNING.