Popular Mechanics (South Africa)
The iPad’s handwriting recognition tech shows how Apple does machine learning
THE MORE INTUITIVE A TASK IS FOR humans, generally, the harder it is for artificial intelligence. Think of when Alexa can’t hear your commands, or when your spam filter traps an important email. A computer’s ability to read handwriting, then translate it into letters and numbers it can understand, has been a challenge going back decades. Think of the hit-or-miss capabilities of the Windows Transcriber in the early 2000s, or the PalmPilot in the late ’90s. Handwriting is so nuanced that just analysing a static letter’s shape doesn’t work.
Apple, it seems, found a solution. In the newest update to iPadOS, when you write with the Pencil stylus, the iPad can understand your scrawl. It works like most machine learning – examples inform rules that help predict and interpret a totally new request – but taps into a smarter data set and greater computing power to do what had stumped generations of previous machines. While Alexa and Siri rely on a connection to faraway data centres to handle their processing, the iPad needs to be able to do all that work on the device itself to keep up with handwriting (and drawing – machine learning also helps the Notes app straighten out an imperfect doodle of a polygon, for example). That takes way more effort than you’d think.
‘When it comes to understanding [handwriting] strokes, we do data-gathering. We find people all over the world, and have them write things,’ explains
Craig Federighi, senior VP of software engineering at Apple. ‘We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation.’ That methodology is distinct from the comparatively simple approach of scanning and analysing existing handwriting. Federighi says that for Apple’s tech, which the company calls Scribble, static examples weren’t enough. They needed to see the strokes that formed each letter. ‘If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written.’
That dynamic understanding of how people write means Apple’s software can reliably know what you’re writing as you’re writing it, but combined with data on a language’s syntax, Scribble can also predict what stroke, character or word you’ll write next. The massive amount of statistical calculations needed to do this are happening on the iPad itself, rather than at a data centre. ‘It’s got to be happening in real time, right now, on the device that you’re holding,’ Federighi says. ‘Which means that the computational power of the device has to be such that it can do that level of processing locally.’ The use cases for Scribble: You’re handwriting notes on your iPad with the Pencil during a meeting, and you want to see a map of Zanzibar. You can now swipe to the Maps app and write ‘Zanzibar’ into the search field, rather than pecking at the screen’s keyboard. Or, you want to email a few lines of those handwritten notes. You use your finger to select that section, copy, then paste into an email, where it shows up as if you typed them. Or you write down a phone number, and you can tap to call it.
If you spend R2 300 on an Apple Pencil, Scribble’s precision offers an additional method of input, along with speech and the keyboard, to communicate with the iPad. The use-case is narrow, but it’s a digital bridge for handwriting diehards, and easier than carrying a keyboard with your tablet. It works so well it makes translating your writing into functional text feel like a natural behaviour. This is the kind of novelty Apple introduces often, one that feels so organic, you’ll look for excuses to use it.