Multimodal Interaction / Eye Movement / Gesture / Body Motion / Haptic Feedbak

One of the biggest challenges of building gesture interfaces is on high-level interpretation. The interface system isn't necessarily easy to learn but should effortlessly replicate after users learned it. The approach to building a comprehensive gesture interface system is to find a proper gesture mental model and apply it to proper tasks. Therefore, instead of directly using these conducting gestures into my application, I first did some researches on the mental model behind the conducting gestures system. My question is why the conductor uses this gesture and how I can abstract these gestures and apply them to a general HCI scenario.

The core function of these conducting gestures is to offer clear signals to the musicians about important playing-related points in real-time music playing: the starting point, the end pointing and the beats. These gestures are required to be obvious enough among the sequential movements of the conductor’s hands so that these musicians can differentiate and then interpret them correctly even from a long distance away from the conductor. The tricks conductors used to differentiate these gestures from others are using different hand shape, like palm face front indicating starts while palm face up means stop. Thus, in the game interaction design, I used this trick for selection and on/off tasks; the user uses two different hand shapes of the left hand — palm face down and up — to control the beating music clip play and stop. These two hand shapes are so differentiated with each other so that users won’t easily get confused, so does the computer. Plus, there are three beating clips that are located in three different locations, which are visually indicated by three spheres with different colors. The user needs to use the direction of the left hand to tell the system which one they want to play or stop. These gestures as a system can still be naturally learned and performed by users in real-time.

Besides these functional gestures, the conductor performs some expressive gestures while interacting with the ensembles. For example, beyond the basic hand movements for music beat, they will use dynamic hitting stress for each beat to communicate their understanding of the music with the musicians: whether the music is edgy(hit hard between beating points) or smooth. Moreover, their hand movement range will indicate the overall feeling of music playing like vigorous or elegant. Similarly, in this music and lighting generation game, the right hand takes in charge of speed control. The movement speed of the right hand will cause the speed of music playing and the intensity of lights changed correspondingly. A bigger range of the hand movement will lead to more intensive lighting change, which creates a vibe that matches the player’s potential emotional expression.

​