The LeapMotion, a small usb device used to gather motion sensing data, has a variety of applications in science, medicine, and the arts. Previously, I worked on a Leap DrumKit using Ruby and MIDI and it was a great experience. For the next Leap app, I opted to get a little more sophisticated. This time, I wanted to create an environment where swimming manatees are controlled with your fingertips.
###How the LeapMotion uses Websockets
Since I wanted to get my manatees to swim on the live web, I did a little research on how the LeapMotion handles the websocket handshake. As it turns out, every time you connect the device, it broadcasts tracking information in JSON format via a web socket server accessible at ws://127.0.0.1:6437. This makes it pretty simple to talk the server’s stream:
The server itself is provided by the
leapd process, which runs as a daemon on OS X and Linux.
Once we are connected to the socket, we can hook into existing listeners, such as
onclose. This example illustrates the
onmessage capability for getting frame data:
Each of these events, called deviceEvents, has a state and tracking data associated with it. For instance, if you were looking for data on specific finger (named Pointables by the Leap team), the server would respond to the client in the following format:
We can also see that there is a base connection that sets up the socket differently based on whether it has been detected as being in a browser vs. node:
There’s our friendly
onmessage handler from earlier!
After reading more into the source, it became clear that I would not even need to enable websockets when deploying to Heroku. As we can see, the LeapJS exposes data from the controller to the user’s browser, which persists to the local, lightweight server that is already running. Since there is no remote connection needed (unless you are saving the frame data), deploying an app is a pretty simple task.
###Using LeapJS to Make Manatees
Okay, back to manatees! LeapJS It provides a variety of really helpful namespaces for using the LeapMotion API. After loading the CDN in the head script tags, I immediately had access to the Leap object. The most important function for gathering frames is
Leap.loop, which we can pass options (such as frameEventName to use requestAnimationFrame) and a callback function:
I referenced the LeapJS examples when figuring out the best way to simulate Manatees swimming. LeapJS just released a few great plugins that extend the functionality of the LeapMotion in the browser. As you can see in the example, I used screenPosition, allowing me to access the on-screen position of “any point in Leap-space.”
Next, I created the Manatee and gave it some properties:
The real interesting part is setTransform, which uses the image height and width to determine positioning. On every frame, the app passes each finger’s screenPosition to that manatee’s setTransform function, which will update the image’s location on the screen.
This was my first draft of the app, where all I used were two files (manatee.html & manatee.js) to get up to 10 manatees swimming together! You can see the code in a spike branch on github
###Deploying the Manatees
Next, I used Node.js, Express.js and the Jade templating engine to setup a bit of structure around the manatees:
I had planned on using the npm packages for LeapJS & its plugins, but struggled when trying to get frame data to the view. Eventually I landed on just loading up the the CDN directly in the view:
Once I could get the manatees swimming locally again, I created a Procfile with
web node web.js and a package.json file. From there, I ran
heroku create and
git push heroku master. Before I knew it, I could swim with manatees over the live web!
For this experiment, it would be fun to add collision detection and a bit of side scrolling in ocean. Right now, the poor thing just floats in air.
As part of this improvement, I want to figure out a better structure to deploying LeapJs applications. I would prefer to send frame data with express.js somehow and setup routes that are bound to certain Leap events. If that is too heavy, I could at least set up listeners that interact with the html elements directly, and most likely use canvas if I end up doing more intense drawing on the DOM.