Sep 19 2008
Project Proposal
I intend to create a piece of software that will allow a user to orchestrate audio through the use of hand gestures. Hand gestures will be performed by the user and captured by a webcam; each hand gesture will be assigned to an audio sample by the user. To record a hand gesture, a user will be presented with a countdown timer on screen (3, 2, 1) they will then perform their gesture. While the user gestures, they will see their gesture highlighted by a trail on the web cam feed. They then pick their sound from an audio library to attach to their gesture.
A technique known as frame differencing will be applied to quantify the amount of movement in the gesture. This data will then be manipulated so that if the user performs the gesture again, the gesture does not need to be an exact replica, but a similar movement (i.e., a clockwise circular movement and a clockwise elliptical movement will be perceived as the same gesture).
Once gestures are allocated an audio sample, the user can then orchestrate the samples in free time (no set pattern). When the user correctly performs a gesture, the assigned audio sample will play and they will also be given a graphical representation of this. The graphical representation of a successful gesture will be displayed alongside the live web cam feed as an animated character playing an instrument.
The user will also be able to load an mp3 from their computer into the software, the software then will analyse the audio file and detect its beat. After detecting its beat, the user will play along to the song using their hand gestures to play in time to the song. If the user successfully performs a gesture on time with the beat, the animated character will perform an animation symbolising an unsuccessful attempt, if performed on time, the character will animate a successful attempt. When the song finishes, the user will be given shown their score of how many unsuccessful attempts they have made.
For the purpose of creating this project, a web cam will be needed. The project will be developed with processing in the Windows environment with Java installed. Processing along with the JMyron (video capture), ezGestures (gesturing), Minim (audio and beat detection) and controlP5 (virtual controller interface) libraries. Some of these libraries will be manipulated for motion detection (i.e., ezGestures is designed specifically for a mouse). In addition, research will have to be done into object oriented java to code with processing.