DyPERS: Dynamic Personal Enhanced Reality System

Nuria Oliver, Tony Jebara, Bernt Schiele and Alex Pentland, MIT Media Lab

Problem and approach

With DyPERS (Dynamic Personal Enhanced Reality System), we used augmented reality and computer vision to associate video and audio clips with real objects that the wearer was looking at. The wearable system learned an audio-visual memory from everyday objects and could play the associated media when they appeared again.

We built a laboratory prototype, not a self-contained pair of smart glasses. The wearer carried the camera, display, headphones, microphone, and controls, while an SGI workstation performed the recognition and media processing.

DyPERS demonstration with its wearable display and project poster.

System Architecture

Our object-recognition component learned visual cues selected by the user, formed statistical representations from sample images, and grouped multiple objects into an audio-visual association. It was designed to tolerate changes in scale, translation, rotation, lighting, and modest deformation.

We organized DyPERS around three main components:

1. Audio-visual memory: accumulates personal memories and associates them with objects.
2. Generic trainable object recognition: identifies the objects linked to those memories.
3. Wearable interface: provides audio-visual input/output via a Heads-Up Display, wireless mouse, and wireless microphone.

Real-time audio-visual recording and playback.

Object recognition and association.

Hardware

• ELMO CCD QN401E color camera
• Wireless 3-button mouse
• Wireless microphone
• Glasstron heads-up display and headphones
• SGI O2 workstation
• Wavecom Jr. transmitter/receiver units

Interface

The interface paradigm is Record and Associate: two buttons on a wireless mouse let the user record video and audio in real time. A third button provides negative feedback to delete an incorrect association.

The broader goal was to move from a personal computer to a personalized one: a system adapted to its wearer and able to use the surrounding context. In a museum demonstration, we associated reproductions of paintings with recorded explanations from a guide. When the wearer looked at a work, DyPERS replayed the corresponding video. This provided a simple alternative to an audio tour that required every visitor to follow the same sequence.

Demonstrated and proposed uses

We demonstrated the system with a clock and project poster. The museum scenario provided another concrete demonstration, while the following list also includes applications that we proposed rather than implemented:

1. Clock: displays the user's daily schedule.
2. Demo poster: plays a short video associated with the poster.
3. Multilingual teacher: speaks the name of recognized objects in multiple languages.
4. Stuffed animal: triggers an associated story when a child looks at the toy.
5. Augmented storybook: associates pages of a children's book with narration.
6. Keypad door lock: reminds the user of the correct combination.
7. Business card: plays a recorded conversation to recall who the card belongs to.
8. Origami: plays step-by-step instructions for creating origami objects.
9. Machinery: plays maintenance or operating instructions associated with an appliance.
10. CD / movie poster: plays a preview clip associated with the media.
11. Accessibility: associates objects with audio descriptions for visually impaired users.
12. Medicine: plays dosage instructions when the user looks at the medication container.
13. Store logo: displays the nearest location, hours, and relevant items.
14. Art objects: plays explanations about paintings, sculptures, or other works.

Videos

Publications

@inproceedings{jebara1998dypers,
  author = {Jebara, Tony and Schiele, Bernt and Oliver, Nuria and Pentland, Alex},
  title = {DyPERS: Dynamic personal enhanced reality system},
  booktitle = {DARPA Image Understanding Workshop},
  year = 1998,
  url = {http://nuriaoliver.com/dypers/},
  cites = 70
}

@inproceedings{schiele1999dypers,
  author = {Bernt Schiele and Nuria Oliver and Tony Jebara and Alex Pentland},
  title = {An Interactive Computer Vision System {DyPERS}: Dynamic Personal Enhanced Reality System},
  booktitle = {International Conference on Computer Vision Systems (ICVS 1999)},
  publisher = {Springer},
  pages = {51--65},
  year = 1999,
  doi = {10.1007/3-540-49256-9_4},
  cites = 36,
  citesdate = {2026-03-23}
}

@article{schiele2001sensory,
  author = {Schiele, Bernt and Jebara, Tony and Oliver, Nuria},
  title = {Sensory-augmented computing: wearing the museum's guide},
  journal = {IEEE Micro},
  year = 2001,
  volume = 21,
  number = 3,
  pages = {44--52},
  doi = {10.1109/40.928764},
  cites = 47,
  citesdate = {2026-03-23}
}

@inproceedings{starner1999augmented,
  author = {Starner, Thad and Schiele, Bernt and Rhodes, Bradley J. and Jebara, Tony and Oliver, Nuria and Weaver, Joshua and Pentland, Alex},
  title = {Augmented realities integrating user and physical models},
  booktitle = {Augmented Reality},
  year = 1999,
  pages = {93--100},
  doi = {10.1201/9781439863992-14},
  cites = 7,
  citesdate = {2026-03-23}
}