It’s all about the camera

As we look to the future, the cameras in our smartphones are only going to get better, writes Brian X. Chen

Smartphones will incorporate smarter features like face scanning technologies in the near future. model with a starting price of US$999 (RM4,208), the company will introduce infrared facial recognition as a new method for unlocking the device.

How would the new iPhone do that exactly? Apple declined to comment. But Qualcomm’s Spectra, a so-called depthsensing camera system, is one example of how face scanning works.

The Spectra system includes a module that sprays an object with infrared dots to gather information about the depth of an object based on the size and the contortion of the dots. If the dots are smaller, then the object is farther away; if they are bigger, the object is closer. The imaging system can then stitch the patterns into a detailed 3D image of your face to determine if you are indeed the owner of your smartphone before unlocking it.

“You’re seeing the contours of the head — it’s not just the front of the face as you’re typically thinking about,” said Sy Choudhury, a senior director of product security for Qualcomm.

Because of the uniqueness of a person’s head shape, the likelihood of bypassing facial recognition with the incorrect face is one in a million, he added. That compares with a false acceptance rate of one in 100 for previous facial recognition systems, which had very poor security.

Older facial recognition systems worked by simply using the camera to take a photo of yourself and comparing that with an image that was stored on the device. All a thief would need to do to fool the system was hold a photo of your face in front of the camera — which some people already did with Samsung’s facial-recognition feature.

There are, however, limitations to infrared-scanning technologies. For example, objects that you wear, like a hat or a scarf, might throw off the camera, according to Qualcomm. In addition, experts said infrared light can get drowned out by bright sunlight outdoors, so face scanning might work less reliably on the beach.

It remains to be seen how exactly face scanning will work in the next iPhone. But Apple is well acquainted with depth-sensing camera technologies. In 2013, the iPhone maker acquired PrimeSense, a company that developed sensors for Microsoft’s Kinect, a depth-sensing camera system that let Xbox players control games using body movements. Analysts expect some rendition of PrimeSense’s technology to appear in future iPhones.

AUGMENTED REALITY

Depth-sensing cameras may be crucial to enhancing augmented reality, a jargony industry term that probably makes your eyes glaze over. But bear with me for one moment: Augmented reality will have major implications for future mobile apps.

It’s no secret that Apple is bullish about augmented reality. In a recent financial earnings call, Timothy Cook, Apple’s chief executive, called augmented reality “big and profound,” with major implications for gaming, entertainment and business products. This fall, Apple will release iOS 11, its next mobile operating system that includes support for applications made with ARKit, a tool kit for app developers to easily create augmented-reality applications.

ARKit uses a combination of the iPhone’s camera and motion sensors, including the accelerometer and gyroscope, to let people lay digital objects on top of the real world and interact with them with precise movements.

I got a demo of ARKit from Ikea, the furniture maker, with its coming app Ikea Place. I placed an Ikea bed on the floor and was able to move around and look underneath it. This type of application would be useful for getting a sense of how an item looks and fits alongside other furniture in a space before placing an order.

“This is like a real application that real people can use to make real-life decisions,” said Michael Valdsgaard, the head of digital transformation at Ikea. But the limitations of the Ikea Place app underscore what’s missing from ARKit. For placing virtual objects, the app can detect horizontal surfaces, like a table surface or the ground, but it cannot yet detect walls.

Vertical planes like walls are trickier to detect because they are not as smooth as floors — with doors, windows and picture frames getting in your way. Depthsensing cameras make wall detection much easier for future iPhones, said Blair MacIntyre, a research scientist who is working on augmented reality for Mozilla, the organisation that makes the Firefox web browser.

All the tech giants are betting big on augmented reality. For years, Microsoft has been developing HoloLens, an augmentedreality headset. In April, Facebook announced Camera Effects Platform, an environment for software developers to build augmented-reality apps for Facebook. This week, Google unveiled ARCore, an augmented-reality tool kit for Android devices, in response to Apple’s ARKit.

MacIntyre said augmented reality has huge potential when it matures. He envisioned people being able to take a tour of a natural-history museum, pointing their smartphone cameras at a fossil exhibit to bring a dinosaur back to life. But he said that augmented reality on smartphones was a stopgap to the inevitable: Wearing data in front of your face at all times through some kind of headset.

“If you look at science fiction, a lot of it has this characteristic of being always on and serendipitous,” he said. “You get a lot closer to that when you get a headmounted display.”

Until that happens, smartphones are about to become much smarter.