DyPERS: Dynamic Personal Enhanced Reality System

Nuria Oliver, Tony Jebara, Bernt Schiele and Alex Pentland — MIT Media Lab

Problem and approach

DyPERS (Dynamic Personal Enhanced Reality System) uses augmented reality and computer vision to overlay video and audio clips onto real objects that the user is paying attention to. The system is wearable and adaptively learns an audio-visual memory, associating everyday objects with relevant media to evoke or play back in the future.

This was a laboratory prototype, not a self-contained pair of smart glasses. The user wore the camera, display, headphones, microphone, and controls, while an SGI workstation performed the recognition and media processing.

System Architecture

The object-recognition component learned visual cues selected by the user, formed statistical representations from sample images, and grouped multiple objects into an audio-visual association. It was designed to tolerate changes in scale, translation, rotation, lighting, and modest deformation.

The three main components of DyPERS were:

1. Audio-visual memory — accumulates personal memories and associates them with objects.
2. Generic trainable object recognition — identifies the objects linked to those memories.
3. Wearable interface — provides audio-visual input/output via a Heads-Up Display, wireless mouse, and wireless microphone.

Real-time audio-visual recording and playback.

Object recognition and association.

Hardware

• ELMO CCD QN401E color camera
• Wireless 3-button mouse
• Wireless microphone
• Glasstron heads-up display and headphones
• SGI O2 workstation
• Wavecom Jr. transmitter/receiver units

Interface

The interface paradigm is Record and Associate: two buttons on a wireless mouse let the user record video and audio in real time. A third button provides negative feedback to delete an incorrect association.

Demonstrated and proposed uses

The archived project page gave the following demonstrations and application ideas:

1. Clock — displays the user's daily schedule.
2. Demo poster — plays a short video associated with the poster.
3. Multilingual teacher — speaks the name of recognized objects in multiple languages.
4. Stuffed animal — triggers an associated story when a child looks at the toy.
5. Augmented storybook — associates pages of a children's book with narration.
6. Keypad door lock — reminds the user of the correct combination.
7. Business card — plays a recorded conversation to recall who the card belongs to.
8. Origami — plays step-by-step instructions for creating origami objects.
9. Machinery — plays maintenance or operating instructions associated with an appliance.
10. CD / movie poster — plays a preview clip associated with the media.
11. Accessibility — associates objects with audio descriptions for visually impaired users.
12. Medicine — plays dosage instructions when the user looks at the medication container.
13. Store logo — displays the nearest location, hours, and relevant items.
14. Art objects — plays explanations about paintings, sculptures, or other works.

Videos

Publications

@inproceedings{jebara1998dypers,
  author = {Jebara, Tony and Schiele, Bernt and Oliver, Nuria and Pentland, Alex},
  title = {DyPERS: Dynamic personal enhanced reality system},
  booktitle = {DARPA Image Understanding Workshop},
  year = 1998,
  url = {http://nuriaoliver.com/dypers/},
  cites = 70
}

@inproceedings{schiele1999dypers,
  author = {Bernt Schiele and Nuria Oliver and Tony Jebara and Alex Pentland},
  title = {An Interactive Computer Vision System {DyPERS}: Dynamic Personal Enhanced Reality System},
  booktitle = {International Conference on Computer Vision Systems (ICVS 1999)},
  publisher = {Springer},
  pages = {51--65},
  year = 1999,
  doi = {10.1007/3-540-49256-9_4},
  cites = 36,
  citesdate = {2026-03-23}
}

@article{schiele2001sensory,
  author = {Schiele, Bernt and Jebara, Tony and Oliver, Nuria},
  title = {Sensory-augmented computing: wearing the museum's guide},
  journal = {IEEE Micro},
  year = 2001,
  volume = 21,
  number = 3,
  pages = {44--52},
  doi = {10.1109/40.928764},
  cites = 47,
  citesdate = {2026-03-23}
}

@inproceedings{starner1999augmented,
  author = {Starner, Thad and Schiele, Bernt and Rhodes, Bradley J. and Jebara, Tony and Oliver, Nuria and Weaver, Joshua and Pentland, Alex},
  title = {Augmented realities integrating user and physical models},
  booktitle = {Augmented Reality},
  year = 1999,
  pages = {93--100},
  doi = {10.1201/9781439863992-14},
  cites = 7,
  citesdate = {2026-03-23}
}