American Sign Language Recognition in Game Development for Deaf Children (Brashear, et al – 2006)

Current Mood: studious

Blogs I Commented On:

• Josh Peschel
• Pankaj Rajan
• Brandon Paulson

Summary:
This is an application paper in the form of a computer game that utilizes computer gesture recognition technology to develop American Sign Language (ASL) skills to children in a system called CopyCat. The paper’s goal was to augment ASL learning in a fun manner for a children’s curriculum. The game focuses on correctly practicing repetitions of ASL phrases through the use of a computer. Due to lack of prior recognition engine for this area, the authors use a Wizard of Oz study by emulating missing functionality of the system for later implementation. For their data set, the authors selected phrases with three and four signs, and their recognition engine is currently limited to a subset of ASL single- and double-handed signs. To segment the samples, users were asked to use a mouse to indicate the start and end of their gestures, allowing their system to perform recognition on the pertinent phrases. Data consists of video of the user signing along with wireless accelerometers mounted on pink-colors for easy processing through a computer vision algorithm. Image pixel data is converted to HSC space for image segmentation purposes. For accelerometer processing, each data packet consists of four values: some sequence number and each of the three spatial dimensions. These packets are first synched to their video feed, then smoothed to account for variable number of packets associated in each frame. Finally, the feature vectors themselves are a combination of both the vision data and accelerometer data, where the accelerometer data consists of the spatial dimensions for both hands, and the vision data consists of various hand characteristics.

Discussion:
This is another applications paper for hand gesture recognition, but what I liked about this paper was that it was both something different from the typical applications papers concerning the domain our class is focusing on and potentially useful. I say potentially because in its current form, it still has some work to do. The results given in the paper are not very insightful on how well it really performs, and I’m still critical of the toolkit used due to its possible limitations (mentioned in a prior blog post), but I don’t see anything that would stop it from being a useful final product for their target audience of children.