Maestro Development Blog

Commercial Title: Maestro

Academic Title: Audio manipulation system

Name: Michael Doyle

Abstract: Maestro is a software system that introduces the art of composition to the digital world. It allows the user to interact and manipulate audio in real-time through the use of hand gestures. Gesturing will be interpreted by a standard web cam which in turn will conduct and generate the audio output of the system. The system will define a physical movement as a specific command that will have its own unique influence on audio.

Disciplines: Object Oriented Programming, Audio (Digital/Analog), Animation, GUI Development, HCI, I/O, Real Time Interaction
Hardware/Software Technologies: Windows, Apache/Tomcat, Java, Processing, MyEclipse, MIDI, Web Cam

Just came across this and skimmed through it, getting late!

Link

It mentions the Hidden Markov Model and the Kalman Filter. Some pages on which gestures work with the software.

GART is a library for processing, here is an example video for it:

Link

This is a publication from Machine Vision Group about recognition about hand gestures. It give some algorithms and basis it's recognition software on the Kalman Filter which basically reads an input and gets a more accurate representation of the source of the input by filtering out noise.

Link to article

Karl discussed the havok software at the first meeting.

Wiki says:

Havok Physics, better known as simply Havok, is a physics engine developed by Irish company Havok. It is designed for computer and video games by allowing interaction between objects or other characters in real-time and by giving objects physics-based qualities in three dimensions. By using dynamical simulation, Havok allows for more lifelike worlds and animation, such as ragdoll physics or intelligence in massive falling things.

I found this article about hand gesture recognition using the Hidden Markov Model (HMM).

Wikipedia defines the HMM:

A hidden Markov model (HMM) is a statistical model in which the system being modeled is assumed to be a Markov process with unknown parameters, and the challenge is to determine the hidden parameters from the observable parameters. The extracted model parameters can then be used to perform further analysis, for example for pattern recognition applications.

and a Markov process is:

A Markov process, named after the Russian mathematician Andrey Markov, is a mathematical model for the random evolution of a memoryless system, that is, one for which the likelihood of a given future state, at any given moment, depends only on its present state, and not on any past states.

Link to paper

Abstract

The proposed work is part of a project that aims at the control of a videogame based on hand gesture recognition. This goal implies the restriction of real-time response and the use of unconstrained environments. In this paper we present a new algorithm to track and recognise hand gestures for interacting with a videogame. This algorithm is based on three main steps: hand segmentation, hand tracking and gesture recognition from hand features. For the hand segmentation step we use the colour cue due to the characteristic colour values of human skin, its invariant properties and its computational simplicity. To prevent errors from hand segmentation we add the hand tracking as a second step. Tracking is performed assuming a constant velocity model and using a pixel labeling approach. From the tracking process we extract several hand features that are fed into a finite state classifier which identifies the hand configuration. The hand can be classified into one of the four gesture classes or one of the four different movement directions. Finally, the system’s performance is evaluated by showing the usability of the algorithm in a videogame environment.

HandVu is a hand gesture recognition piece of software that tracks a users hand in real time. The software uses OpenCV which contains over 500 algorithms for "real time computer vision".

There are a few people working on getting the OpenCV Library to work with processing.

Obtained from this link
The conductor uses his hands, arms, body, head and face to keep the performers together and to encourage them to give the very best performance possible.

1. Preparation. To show the musicians what the music is going to sound like before they perform it. With gesture, the conductor indicates the tempo, volume, articulation and exact starting time of the next beat. He also shows you who performs the beat (cues). When he does this to start the piece, it usually includes a model of the breath you should take.
   1. Tempo. Through the speed of the gesture.
   2. Volume. Through the size of the gesture.
   3. Articulation. Through the angle and "weight" of the gesture.
   4. Starting Moment. Through a breath and the predictable speed of the gesture.
   5. Cue. By looking at the performers involved. Note that the conductor does not have to be looking at you when you begin singing a note, as long as he prepared you to come in.
2. Contrast. The conductor shows the musicians how the next note, measure, or phrase is different from the last.
3. Style. The conductor is a physical representation of what the music sounds like and feels like to the audience.
4. Pulse. Using established beat patterns the conductor shows the pulse and what beat you are on in the measure.
5. Clarity & Economy. The conductor should communicate only what he intends to communicate and only what you need. No more, no less.
6. Listening. The conductor is your ears. The conductor is the only one who can hear what the group sounds like. The conductor is charged with making adjustments as needed to perfect the performance.

Machine Vision Group are a Dublin based research company that are currently working on Real-Time gesture recognition of human hand for the Irish Sign Language Association.

For the project they use a bright yellow glove on a hand and place it in front of a camera. The camera picks up the color and shape, but also picks up some background objects of the same color. They then filter out these objects as they are smaller in size compared to the hand. Once the hand is picked the hand shape is detected and a match for the gesture has to be found.

A "nearest neighbor" approach would take too long (compare the gesture to 1000's of image of gestures and find a similar gesture).