Fashionable Wearable American Sign Language Recognition System

With WearASL, we explored how computer vision and wearable computing could support two-way communication between people who use American Sign Language (ASL) and people who communicate through speech. Rather than treating the technology as separate equipment, we incorporated the cameras, computer, controls, displays, speakers, and power supply into clothing and accessories.

Communication concept

At the fashion show, we presented Van and WearASL as coordinated wearable systems. A camera embedded in the hat observed the wearer's hand movements and sent the images to a computer carried in the backpack. We designed the system to recognize signed gestures and convert them into synthesized speech played through speakers in the backpack straps.

For communication in the other direction, a microphone captured spoken input and the wearable system presented the translated output through a Braille display. An LCD necklace provided visual feedback so that the wearer could check the system's interpretation.

Our public presentations documented the sign-to-speech path most concretely: the hat camera observed the hands and the backpack speakers voiced the recognized signs. Speech-to-Braille was part of the broader two-way design concept, rather than a separately evaluated translation system.

Technical approach

Our project poster describes a computer-vision pipeline for segmenting and tracking the hands, followed by gesture recognition using coupled or single hidden Markov models. We distributed the hardware across the outfit: the camera was integrated into the hat, the computer was carried in the backpack, and the batteries were stored in the shoes.

WearASL was a fashion-integrated research prototype.

Collaboration and presentation

We worked with Barbara Rosario, Joshua Weaver, and Alex Pentland at the MIT Media Lab and Maria Ella Carrera, Ricardo Prado, and Josefina Batres at Domus Academy in Milan. We presented the project in October 1997 as part of the MIT Media Lab's Beauty and the Bits Smart Clothes Fashion Show.

The Beauty and the Bits exhibit page describes Van and WearASL.

The original WearASL poster, reproduced from a higher-resolution scan.