Behind the Scenes: Mouse Gestures in a Usable Way

Culture Section
Our new .com was a great excuse to play around with some big ideas we’ve had kicking around.

And, if you’ve visited the new AgencyNet.com, you probably noticed that we have a bit of a thing for user interface.  Our new site explores a range of UX paradigms and inputs — exploring a building, navigating a slideshow, even using the touch-sensitive iPhone as an input mechanism.

We also employed a unique navigation system in our “Culture” section, exploring the ability to browse content without clicking buttons or scrolling in a page.  The system was designed to facilitate a more human friendly interaction between the Flash application and the user by using “mouse gestures”.

Most of us are already familiar with this concept.  It is used to particularly good effect on the iPhone and other touch screen devices, where users can give commands using specific fingers movements for specific actions (i.e. zoom in by separating two fingers or by doing the opposite to zoom out).

We’ve already revealed some of the secrets behind our eye-catching homepage and, in the spirit of open source, we thought we’d do the same here.

Creating Gesture Navigation

Developer note: Gesture navigation is not suitable for every application.  Also, when using this technique, the developer should choose which gestures to use wisely.  Try to make the gestures self explanatory and intuitive.  Gestures need to be consistent and fairly simple in design, yet have distinct differences between them; we don’t want the users to get frustrated because the app is confusing the gestures.   Finally, a personal tip is to avoid letters, because a word in English may begin with a different letter in another language.

In this mini tutorial we will create a simple image gallery that will be controlled through the use of mouse gestures.   Here is a list of the gestures that we will catch:

  • A horizontal line to zoom out.

zoomOut

  • A vertical line to zoom in.

zoomIn

  • An arrow pointing to the right to go to the next image.

right

  • An arrow pointing to the left to go to the previous image.

left

And here is the plan we will follow:

  1. Create a “gesture listener” that will call the commands and execute the behaviors on the gallery depending on the gesture that the user provided. This will act as a bridge between the gesture library and our app.
  2. Create a gallery image class that has the next public behavior or functions: show next, show previous, zoom in and zoom out.
  3. We will connect these by using the command pattern to add some sugar to the code, in this way we will be able to use buttons and/or gestures or whatever we want to use as an input for our gallery.

If you haven’t already, take a look at our Culture section and familiarize yourself with it so you have a better idea of what we are going to be talking about.  And from here, you can download the source code.

The gestures handler

The core of this is to have a way to recognize the mouse gesture (or the mouse movement trace).  The good news is that we don’t really need to get too much into it, there is already a library in AS3 that’s really simple to use it and it works great. The Mouse Gesture Recognition lib for AS3, created by Didier Brun, implements all of the functionality that we need to detect mouse gestures; you only need to translate your mouse movement into numbers, where each number represents a direction.

mouseMovement2

After this, the numbers entered with the mouse are compared with the saved gestures, using the Levenshtein distance (the minimum number of edits to transform one sequence into another), and then it selects the best candidate, provided it’s enough close to a defined gesture.

Please refer to the official page to see a clear image of how it works and how it uses the Levenshtein algorithm to choose the best candidate of the registered gestures.

For this task, I have created a class called GesturesHandler that listens to the mouse gesture library events.  If a mouse gesture is recognized, it will execute the command specified by our class.

First we need to register the movement for each gesture that we want to match:

_mouseGesture.addGesture(RIGHT, "1112333");
_mouseGesture.addGesture(LEFT, "3332111");
_mouseGesture.addGesture(ZOOMIN, "222222");
_mouseGesture.addGesture(ZOOMOUT, "0000000");
//inverted
_mouseGesture.addGesture(RIGHT + "2", "7776555");
_mouseGesture.addGesture(LEFT + "2", "5556777");
_mouseGesture.addGesture(ZOOMIN + "2", "666666");
_mouseGesture.addGesture(ZOOMOUT + "2", "444444");

We need to register the same gesture twice since the user might draw the same gesture backwards, like the zoom in line, it can be draw from the top to the bottom or from bottom to the top.

pathrightinverse

Next, we need to add some handlers for the events that the Mouse Gesture library dispatches; here is a brief explanation of the event handlers inside our class:

  • handleStartCapture(event:GestureEvent): When the gesture library starts capturing a gesture, we need to start drawing that gesture in the UI to give the user feedback that a gesture is being handled.
  • handleStopCapture(event:GestureEvent): We will use this event to clear the graphics of the DisplayObject that we use to draw the gesture.
  • handleGestureMatch(event:GestureEvent): If a gesture of the ones we registered is matched, a command specific to each gesture is executed.
private function handleGestureMatch(event:GestureEvent):void
{
   if (event.datas) {
      var gestureString:String = event.datas as String;
      if (gestureString.indexOf(RIGHT) > -1) {
         _commands[RIGHT].execute();
      } else if (gestureString.indexOf(LEFT) > -1) {
         _commands[LEFT].execute();</pre>
      } else if (gestureString.indexOf(ZOOMIN) > -1) {
         _commands[ZOOMIN].execute();
      } else if (gestureString.indexOf(ZOOMOUT) > -1) {
         _commands[ZOOMOUT].execute();
      }
   }
}

So how do we register the commands?

public function addCommand(key:String, command:ICommand):void
{
   _commands[key] = command;
}

As you can see, there is an addCommand() function that adds a command to a dictionary.  The handleGestureMatch() function uses those commands from the dictionary to call each function.  We can add commands to our gesture handler dictionary from our Main class.  The good thing about using the command pattern is that our gesture handler is not tied to any other class; we can use our class to control anything that our concrete commands implements.

The Image Gallery

This class creates an Array of images defined in an xml, as soon as the first image is loaded that image is added, this class implements four public functions that will be called from our commands, here is a brief explanation of what they do although their names are pretty self explanatory:

  • zoomIn: Increases the size of the current image, maintaining the image centered.
  • zoomOut: Decreases the size of the current image.
  • showNext: It moves the current image to the left leaving the stage, and it shows the next image in the array coming from the right of the stage.
  • showPrev: It moves the current image to the right leaving the stage, and it shows the previous image in the array coming from the left of the stage.

So far, we have created the two main components of our small app, but we need to connect them using commands.  I think you are starting to feel my love for this pattern; it’s just a nice way to maintain objects loosely coupled and keep our code as abstract as possible.

Commands

Finally for the commands we will create an interface called ICommand that will define the signature for the “execute” function; with this we can create our concrete commands that will call the behaviors from our image gallery class using the “execute” function, here is a brief description of each of the commands class that I have created:

  • NextImageCommand: It Calls the showNext function of the image gallery.
  • PreviousImageCommand: It Calls the showPrev function of the image gallery.
  • ZoomInCommand: It Calls the zoomIn function of the image gallery.
  • ZoomOutCommand: It Calls the zoomOut function of the image gallery.

Here is one of the commands classes the only difference between them is the line of code inside the execute command function.

public class NextImageCommand implements ICommand
{
   private var _receiver:ImageGallery;
      public function NextImageCommand(receiver:ImageGallery)
      {
         _receiver = receiver;
      }
      /* INTERFACE com.agencynet.gesturegallery.commands.ICommand */

      public function execute():void
      {
         _receiver.showNext();
      }
}

Mixing Things

So in our Main class we link our objects using the next lines of code:

//create the gallery using the xml
_gallery = new ImageGallery(xml.images.image);
addChild(_gallery);

//create the commands to control the gallery
var nextCommand:ICommand = new NextImageCommand(_gallery);
var prevCommand:ICommand = new PreviousImageCommand(_gallery);
var zoomInCommand:ICommand = new ZoomInCommand(_gallery);
var zoomOutCommand:ICommand = new ZoomOutCommand(_gallery);

//create and add the gestureHandler that will receive the mouse events
_gestureHandler = new GesturesHandler();
_gestureHandler.resize(stage.stageWidth, stage.stageHeight);
addChild(_gestureHandler);

//assign a command to a gesture type
_gestureHandler.addCommand(GesturesHandler.RIGHT, nextCommand);
_gestureHandler.addCommand(GesturesHandler.LEFT, prevCommand);
_gestureHandler.addCommand(GesturesHandler.ZOOMIN, zoomInCommand);
_gestureHandler.addCommand(GesturesHandler.ZOOMOUT, zoomOutCommand);

As you can see we create our ImageGallery, then we create one instance of each of our commands. After that, we create the GestureHandler instance and, finally, we add the commands to it using the addCommand() function.

A Bonus

So for all of you who kept reading and watched the source you may have noticed that there is another class available called Menu, this class is also able to add commands just like our GestureHandler, this class contains four buttons on the edges of our stage, clicking on each of the buttons will trigger one of our commands.  By using the same commands we are able to control our ImageGallery from a different source and we don’t need to tell our ImageGallery class anything.

I hope you have found this useful and that you can use it in your future works, and please don’t hesitate to post your comments either if you have questions or suggestions about how can I improve the way I write my examples, and keep flashing the world.

For more Behind The Scenes content check out:
Behind the Scenes: AgencyNet 2.0