Focusing on the big picture
Images from Google Street View rich in data
What vehicle is most strongly associated with Republican voting districts? Extended-cab pickup trucks. For Democratic districts? Sedans.
Those conclusions may not be particularly surprising. After all, market researchers and political analysts have studied such things for decades.
But what is surprising is how researchers working on an ambitious project based at Stanford University reached those conclusions: by analyzing 50 million images and location data from Google Street View, the streetscene feature of the online giant’s mapping service.
For the first time, helped by recent advances in artificial intelligence, researchers are able to analyze large quantities of images, pulling out data that can be sorted and mined to predict things like income, political leanings and buying habits. In the Stanford study, computers collected details about cars in the millions of images it processed, including makes and models.
“All of a sudden we can do the same kind of analysis on images that we have been able to do on text,” said Erez Lieberman Aiden, a computer scientist who heads a genomic research centre at the Baylor School of Medicine. He provided advice on one aspect of the Stanford project.
For computers, as for humans, reading and observation are two distinct ways to understand the world, Lieberman Aiden said. In that sense, he said, “computers don’t have one hand tied behind their backs anymore.”
Text has been easier for AI to handle, because words have discrete characters — 26 letters, in the case of English. That makes it much closer to the natural language of computers than the freehand chaos of imagery. But image recognition technology, much of it developed by major technology companies, has improved greatly in recent years.
The Stanford project gives a glimpse at the potential. By pulling the vehicles’ makes, models and years from the images, and then linking that information with other data sources, the project was able to predict factors like pollution and voting patterns at the neighbourhood level.
“This kind of social analysis using image data is a new tool to draw insights,” said Timnit Gebru, who led the Stanford research effort.
In the end, the car-image project involved 50 million images of street scenes gathered from Google Street View. In them, 22 million cars were identified, and then classified into more than 2,600 categories like their make and model, located in more than 3,000 ZIP codes and 39,000 voting districts.
But first, a database curated by humans had to train the AI software to understand the images.
The researchers recruited hundreds of people to pick out and classify cars in a sample of millions of pictures. Some online contractors did simple tasks like identifying the cars in images. Others were car experts who knew nuances like the subtle difference in the tail lights on the 2007 and 2008 Honda Accords.
“Collecting and labelling a large data set is the most painful thing you can do in our field,” said Gebru, who received her PhD from Stanford in September and now works for Microsoft Research.
But without experiencing that datawrangling work, she added, “you don’t understand what is impeding progress in AI in the real world.”
Once the car-image engine was built, its speed and predictive accuracy was impressive. It successfully classified the cars in the 50 million images in two weeks. That task would take a human expert, spending 10 seconds per image, more than 15 years.
Identifying so many car images in such detail was a technical feat. But it was linking that new data set to public collections of socioeconomic and environmental information, and then tweaking the software to spot patterns and correlations, that makes the Stanford project part of what computer scientists see as the broader application of image data.