Sender 11

Nokia's new touch UI

Mobile, open development platform with touchscreen, haptic feedback, presence, GPS location, VoIP and an AJAX capable web browser. Only Google Gears is missing.

Who is this good news for?

Symbian developers rejoice, the big question is: what about Java and Browser developers?

As Dean Bubley observes; "normal people don't download applications to handsets". S60 Symbian development is a all about enterprise applications. In order to access these great new features in consumer applications, Nokia has to provide OS bindings from the browser and from Java ME. All Nokia’s senior VP for software platforms, Lee Williams said about this in his presentation yesterday was that support for cool stuff would come in an "iterative way". And if Java ME support goes thru the Java Community Process, it will take a couple of years to agree on a JSR.

Well, its certainly cool stuff here:

Nokia showed a touch UI as you have probably already seen elsewhere. It is not multi-touch, darn! The touch screen in the demo was a resistive type single point touch screen. The UI changes are evolutionary. Nokia still sees the phone as a single-hand device and they are keeping menus, they are just making them finger-sized. Nokia said they wanted to be backwards compatible, keep old application working. There is no iPhone style finger keyboard, you need to use a stylus for writing.

The new phones will support haptics. Probably by using the vibration motor. Nokia licensed this technology from Immersion earlier this summer. This is fairly primitive so don't expect any religious experiences. But its a start.

Cool gestures!

They introduced a "Sensor framework" and mentioned both motion, orientation, proximity and light sensors. They showed both motion and acceleration. A video showed a woman diverting a call by flipping over the phone while it was lying on a table. All these sensors makes gestures an additional UI component. This is very exciting and certainly opens some new opportunities for application and game developers!

You may have seen the crazy guys who made it possible to use a Wii remote to control a Nokia phone, next year you may be able to control the Wii with your Nokia phone!

Gestures are going to be user configurable.

ScreenPlay and FreeWay

ScreenPlay and FreeWay are Symbian 9.5 technologies for video acceleration and network access. Snazzy names! Something must have happened since the iPhone. Symbian used to call their stuff something like "Host Controller Transport Layer" or "Logical Link Control and Adaptation Protocol (L2CAP)".

ScreenPlay goes hand in hand with Symbian's recently announced support for Symmetric Multi Processing (SMP). SMP is basically a Duo Core processor on your phone. Current Symbian phones are painfully slow and desperately needs faster drawing speed. Thanks, iPhone!

Symbian Freeway architecture. Nokia has promised UMA devices next year. The promise is that you can move seamless between networks. It you are on the phone and move into a known WiFi zone, the call seamlessly switches to VoIP. Almost sounds too good to be true.

Flash Lite 3 was announced, that means Flash Video in the browser and will cater to all your YouTube entertainment needs. Potentially a better solution than today's awkward video playback.

Links:

Official S60 touch UI info

Video

The Apple multi-touch gesture dictionary patent

Apple has filed a patent application for a "multi-touch gesture dictionary". This has spurred some debate in the IxD community. Can you patent a language? Is it good? Is it evil? Etc. etc.

1) Gestures used for controlling machines via touch screens do not constitute a language, linguistically. Gestures are simply a collection of commands. No language has ever been patented, not braille, not sign language, not esperanto etc.
There is 20+ years of prior art in this field (research on touch screens and gestures) and anyone trying to patent a set of gestures will have a really hard time finding gestures that are not already "prior art". All the good ones are already "taken".

2) Many of us have tried our hand at designing icons and know how hard that can be; very few commands can easily be visualized iconically. The best touch screen gestures are symbolic/iconic. A symbol/icon is based on imitation; the "flick" scroll gesture is one example. You might argue that Option-C or Ctrl-C are symbols as well, they "imitate" the word "Copy". Trouble is that you soon run out of "symbolic"/iconic gestures and you have to use signs instead. Signs have an arbritrary relationship to their 'meaning' - the command they signal, and they have to be learned and memorized. What is the relationship between Option-V or Ctrl-V and "Paste"? (The only "relation" is spatial on a qwerty-keyboard: the V key is next to the C on the keyboard, and "Paste" often follows "Copy".)

And this might just be the point of the Apple patent! Apple seeks to patent every imaginable way to assist learning sign gestures. Since there are so few symbolic gestures, Apple knows you are going to have to start learning the sign ones. And they want to ensure that you can't get help without some money tickling into the Apple bank account.

I believe they simply want to patent how to learn gestures.

Touch is the new Click *

If you are a hardware person, touch is the new thin, if you are a software person then touch is the new click. If you are a little of both then I guess touch is the new thin click...

The need for feedback

Like many people, we have a car. Nothing fancy, 4 year old, reliable and not particularly high-tech. (Yes, the missus decided...) The car has an electronic flasher for the indicator lights/turn signals.

In the old days, cars used a thermal flasher. A thermal flasher is a mechanical device consisting of some pieces of spring metal and wire, cleverly arranged, inside a little tin can, to turn the blinker lights on and off every second or so. And it has a nice tick-tock-tick-tock sound.

The sound is important. The sound is the primary feedback mechanism for the indicator lights. You don't see the indicator lights from inside most cars. You could look down at the instrument panel, but you probably prefer to keep your eyes on the road, especially when driving through an intersection.

As time went by, electronic devices became less expensive than mechanical devices. But electronic devices do not generate sounds voluntarily. To solve this problem manufacturers had to fit a "sound circuit" and a "loudspeaker" into the electronic flasher to make it sound like an old-fashioned mechanical device.

Haptics

The current mobile phone touch screens are the equivalent of flashers without sound. Current touchscreen phones, like the iPhone, do not give adequate feedback. Look at the problems people have typing on the device. But all is not lost. Haptics is the "science of touch" and there is a lot of interesting work going on in that field.

Vibration as feedback

The vibration actuator common in most mobile phones is used for "announcements"; when you receive a call in silent mode, the phone vibrates. This rudimentary behavior can be expanded, and the idea would be to apply very detailed control over the vibration motor in order to give feedback from touch.

There are physical limitations to what you can do with vibration. The screen is fairly small and has to be made from a transparent and scratch resistant material. If the screen has a hard surface, there is currently no way to make only a part of the screen vibrate. If you apply mechanical vibration, the entire screen will vibrate. In many cases, this is OK, like when you have a single point of touch and you want to give a sense of "hit" or "miss".

I'm not at all sure that multi-touch gestures really need different feedback for each finger.

Problem with bandwidth

Experiments have shown that most people can distinguish between 6, 75 and 250 Hz vibration. They can also distinguish between sine and sawtooth vibration. This is not a lot of resolution. The problem with vibration is that the communication bandwidth is very narrow. It would be nice if is was possible to feel the keys on the control surface (the screen). To me, vibration seems inadequate for this type of "haptic rendering". It probably won't be possible to simulate texture and shape well enough. I think it would be possible to render any surface shape and size using vibration, but its hard for me to see how to create the sensation of an "edge". It would likely be a fuzzy object.

Audio + Vibration

A way to increase bandwidth and redundancy is obviously to combine vibration and sound. For example a little "tap" combined with one sound could mean a click. A little "buzz" combined with a different sound could mean an error.

Combining haptic and auditory feedback has a lot of promise and there is even a conference (there is always a conference!): 2nd International Workshop on Haptic and Audio Interaction Design. If you are interested in audio + vibration you should also take a look at this MIT Technology Review article and google Prof Stephen Brewster.

Movement matters

Imagine you touch something with your index finger. The contact area is about the size of a pea. If you hold your finger still, you get a somewhat imprecise impression of what you are touching. As soon as you move your finger you get a much more precise indication. If you hold still, you may not be able to tell if you touch, say, paper, plastic or wood. As soon as you move your finger over the surface, you can easily tell the difference. Humans are good at distinguishing texture and shape. Braille readers can distinguish 43 different shapes each with an 2,5 * 5mm (0.1 by 0.2 inches) area.

Instead of simulating via vibration, I want the real thing: adaptive surface texture and shape. Not only for the fingertip, but for the whole hand. There is relevant feedback to give also after the finger is lifted off the screen.

Liquid metal skin? Nanotech to the rescue!

We need a material that is transparent and durable in its resting state but can change its surface enough to make it possible for us to feel shapes. The material should also be touch sensitive. The entire phone should be covered in the stuff, no reason to limit the interaction to the screen only. The phone could be able to tell us when a download is completed or give directions to the post office while in the pocket or in the hand.
Btw, does anybody know what phone the T-1000 from Terminator 2 uses?

*) The title is snagged/derived from CEnriqueOrtiz. If you develop mobile phone software, I recommend you add his blog to your reader.