Apple Scribble Review: AI faces cursive
29.6.2020
Translation: Patrik Stainbrook
Apple's upcoming iPadOS 14 software will be able to recognise notes written with their branded Pencil. This tool is almost ready. The software handles cursive nicely. German? Not so much.
Apple's iPadOS 14 will include a feature called Scribble. Scribble is quite simple when you get down to it: another attempt of a tech titan trying to recognise and interpret human calligraphy. After upgrading my two-year old iPad Pro, it's time to analyse the system and then attempt to write in cursive. Up until now, this type of writing has been the Achilles heel of any computer.
And just cause it's fun: let's see how much Scribble can handle. Generally, my handwriting has been described as atrocious – sometimes I can't even decipher it myself, but luckily I usually remember what I wrote. Now its time to check out how horribly I have to scrawl across the screen before Scribble taps out.
If you're not interested in the problems and complexity of handwriting recognition, skip forward to the section titled «Scribble: hidden but active».
Handwriting: more complex than you think
Before we jump headfirst into testing, a few words on handwriting and its features, problems and noteworthy tidbits. When programming handwriting recognition software, you'll inevitably face a sheer infinite amount of problems. After all, when a brain reads handwriting, it's doing a whole lot of work.
I asked three colleagues to write «Digitec» on a piece of paper.
If you show these to a machine, it'll be faced with images, not writing. When typing, no matter what font you're using, a «D» will remain a «D». Even when using Wingdings, your machine has a programmed setting for easily interpreting the letter «D».
But if you write by hand, then it has to suddenly analyse and image. Just the first letter, «D», reveals some problems. In cursive, «D» contains a loop that can also be interpreted as «d». You'd just have to change the «D» loop a tiny bit. If you scribble an «e», it might look like a «c» depending on how sloppy your handwriting was. A «g» can be open or closed at the top, similar to a «y». Then add in the fact that every letter looks slightly different, even when written by the same person. The above pictures contain six different «i»s, written by three different people.
Still, handwriting recognition on a tablet is a neat feature. If you can simply scribble and the machine gets it, then that's already enough for a lot of people: a digital notebook that more than deserves its title. Artificial Intelligence (AI) recognises your scribbles and assigns each of your personal letters a setting, as if you had been using a keyboard. Completely legible notes, written by hand.
Understandably, the idea for handwriting recognition isn't new at all. In the 90s, a company called Palm used a form of handwriting recognition on their Personal Digital Assistant Palm Pilot – most recognisable as a «digital notebook». As the software wasn't fast or smart enough to comprehend the intricacies of handwriting, developers created a type of stenography called «Graffiti». If you wrote a symbol onto the cross, Palm would mostly recognise it correctly.
But it didn't last, and the hype around handwriting recognition slowly quieted down. Apple and Android both included keyboards on their tablets, not once mentioning handwriting recognition. Hardware keyboards were built, both wired and wireless.
Still, the idea remained in the minds of developers. Now Apple claims to have cracked the case after 25 years of testing.
Scribble: hidden but active
During Apple's Keynote at the World Wide Developer Conference (WWDC), the feature was finally showed off. Not how it was implemented though. Good News: Scribble is enabled by default. You can find its settings on iPadOS under Settings → Apple Pencil.
As is expected from Apple, you don't get a large selection of preferences aside from On/Off. But what is there to change? You write, the iPad recognises it, the AI does its job. Forget any settings such as «Level of recognition: High» or whatever.
Scribble is deeply integrated into the system. You can edit any text field with your Apple Pencil by just writing into it. This includes apps created by third-party developers. You can even google by hand. I can enter an address into the Snowhaze Browser. Or browse through iPads on digitec.ch. A bit useless, after all, I already own one. Even if you go slightly outside the allotted box it won't really matter. The iPad understands which box you want to use and where you're writing.
It takes some time to get used to. I always catch myself thinking: «will it recognise this?» Most of the time it does. But not always.
AI enables everything
Turns out that the iPad has a massive problem in understanding the word «Digitec». Or «digitec», if we're following our CDCI. The «g» is often interpreted as a «j» or «y». Sometimes the «c» at the end is an «r». But if I search for the song «Everything is awesome», then the system works its magic. Or shows how it works, at least. Apple is keeping quiet about what exactly Scribble is doing. What's without a doubt: AI plays a big part.
I start writing in the Google app. After «Everything», my scribbles get transformed into digital writing. Suddenly, the Google search bar contains the word «Everything». The same happens after «is». I don't even need to write «Awesome», Google's recommended search results are good. This means that both Google and the iPad's AIs know what I want. Or at least what most users are looking for when entering «everything is». That means «awesome» can be as ugly as I want: the «a» is just barely recognisable, but iPadOS already knows what's up, hitting me up with «The Lego Movie».
Maybe that's why it's struggling with «Digitec» or «digitec». Even though our company is a titan in Switzerland, digitec is nearly irrelevant in the global context.
So, is Apple pinching some of Google's search data? Possibly. It's much more likely that Google's search data gets integrated into the entire interpretation. There could be a whole host of data hidden behind the veil. Theoretically, the software could scan your mail and find out which terms you use most often and correlate this with your scribbles to better find out what you're writing. If you give it permission of course.
Scribble is fast. The AI is so quick that even taking a screenshot of my scribbles in the search bar is sometimes impossible. Impressive.
Copy/Paste: a hassle, but functional.
Scribble even works in Notes, Apple's own notebook app. Write something, and you can highlight and copy it. Careful: if you only select «Copy», then iPadOS will only capture an image of your handwriting. You'll need to choose «Copy as Text» to actually copy the digital format. Kind of annoying, but understandable. What if you want to actually copy your scribble?
Another thing: iPadOS doesn't show you what you're copying when you select «Copy as Text». Am I copying «Digitec» or «Dijiter»? Roll the dice and find out. You're basically gambling on what you're pasting when you copy anything. Would it really have been so hard to pop up a little window containing the text selected?
The stress test: when does Scribble fail?
In the USA and Canada cursive isn't really that common. It's sometimes taught at schools, but never forced upon anyone. In Switzerland, basic handwriting is always taught first. It can be used at your discretion to connect letters should this make the flow of writing easier for you. Classical cursive like I know it has had its time, just like Graffiti. Or stenography in general.
The problem: I don't want to learn a completely new way to write just because some software can't understand my concept of cursive. If a developer claims that their software will understand my handwriting, I want it to understand my style. And I write in cursive from years of scribbling with pen on paper.
Either way, there must be a point at which the software gives up. For testing purposes, I'll use the word «Apple». I write it down in Notes, copy it and search whatever the software recognises in Google.
«Apple» in standard writing works perfectly, as expected.
Time to write Apple in cursive. It's pretty nice for my standards. This is how I write when I want other people to recognise what I'm writing. That's not always the case, but Scribble seems to be fine with it.
This is how I write when I'm making notes. I'm at the point of «I can't read what I wrote, but I remember». iPadOS offers «Copy as Text», so it's deciphering something. I recognises text as text. Even better. iPadOS recognises the scribble as «Apple».
So let's add in some complexity. Only in cursive this time. I immediately notice how writing on the iPad is extremely arduous and slippery. If you're planning on writing on the iPad by hand a lot, then I strongly recommend you get a screen protector such as Paperlike.
Normally I'd write my search term directly into Google, but then I couldn't show you what Scribble needs to deal with. That's why I'm using Notes first before pasting into Google.
This kind of works. Scribble doesn't recognise «Everything is awesome», but «Everything is ansane». Is «ansane» even a word? I'm hoping that Google's AI notices this incongruity, recognises the order of symbols and makes «Awesome» out of it. Nope, I only get weird band names and the like, but no movie theme song.
My handwriting is so scuffed that I've found certain words that no one can decipher at first glance through the years. My versions of «n», «m», «i», «s», «v», «w», «e» and «u» all consist of the same visual element: ∧.
Let's test this with a special word:
Yes, this is a grammatical German word. It's even in the Duden dictionary. Exactly how I've written it here. And that's exactly why I think that Scribble won't be able to handle this: it's a random assortment of slashes.
I write, copy and paste it into Google. «Nomuinn». One thing off the bat, Scribble was right about the «om». The rest? Completely false. Searching «Nomuinn» in Google doesn't result in anything useful either. Climate data from the US state of Kansas, among other things.
Whoever finds out what I wrote will win a grand prize of... nothing. It's like a normal award, but you won't get anything except for a thank you. With a heart. Written on my iPad.
Scribble can handle a lot before it folds. I don't need to change my natural script in a big way, but the AI does work better in English than German. Even if I translate my hellish word into English, Scribble can't handle it. The shapes are probably to unclear. Otherwise, when I enter «Das ist ein Test» it spits out «Pas int ein Test». Not very correct, even for German. «This is a test» works flawlessly on the other hand. «Winter is coming» as well, and «Best ever» – written in the style Tim Cook, of course – works without a problem.
The catch
You're probably thinking that you'll be able to write whole articles like this one using your Apple Pencil. But that only works to an extent. You can scribble anything you want into Notes before copying it into Pages or a Google Doc. But Google's neat search engine recognition isn't present in Notes, Pages or Google Docs. You can start off writing by hand, but the software won't transform it into digital text. That would be next level.
Outlook does this. But it's kinda broken. I decide to write Editor Luca Fontana an email, as I haven't had the chance to annoy him yet today. Recognition only works really well when I write in huge letters, much larger than the text box allows me. Fine, I can deal with that, but then there's also the fact that I can't create new lines.
Scribble is surprisingly good. It's extremely fast, can recognise cursive and makes working with the iPad significantly easier. But it's still not ready for long form articles though. Yes, cancelling out wrong words is relatively simple, but in the end, using a keyboard is quicker than relying on Scribble's AI. On top of having to rely on it recognising my handwriting correctly But if I just want to make a quick note or google something, the Scribble system is useful and reliable.
Scribble is about 90% of the way to an ideal place. It isn't the expected quantum leap in tablet use. It's still missing some accuracy, an understanding of German – never mind Swiss – and support for extra tools such as line breaks. But I'm sure that's yet to come. Because being able to write on top of drawing by hand on a tablet is too good of an idea to just leave in the dust.
Bam. Done. Time to wait and see what Luca has to say about my fantastic email.
Journalist. Author. Hacker. A storyteller searching for boundaries, secrets and taboos – putting the world to paper. Not because I can but because I can’t not.
Smartphone
Follow topics and stay updated on your areas of interest
22 comments
later