LAFTER: Lips and Face Real Time Tracking and Facial Expression Recognition

Nuria Oliver, Francois Berard and Alex Pentland, MIT Media Lab

We built LAFTER when reliable real-time face tracking still required careful engineering. It found a face and mouth in live video, kept tracking them as the person moved, and classified mouth shapes associated with expressions such as happiness and sadness.

Our system joined detection and tracking rather than treating them as separate steps. Detection could recover the face after tracking failed; tracking provided the stable measurements needed for expression recognition. It was tested with hundreds of users.

We also used the tracker as an input to several other systems. An automatic camera operator kept a remote camera pointed toward the user. A virtual window changed its view as the user moved their head, and a graphics application animated a character from the tracked facial movement.

LAFTER running in the laboratory. The photograph comes from a low-resolution video source and is displayed at a correspondingly modest size.

The virtual-window application used LAFTER to direct a remote camera from the user's head movement.

The expression labels were based on visible mouth shape, so they should not be treated as measurements of a person's internal emotional state.

The first paper received the conference's Best Presentation Award and was a Best Paper Award nominee. The later journal article received a Special Mention from the Pattern Recognition Society. A related project modeled the lips in three dimensions rather than classifying their visible appearance; see 3D Modeling of Human Lip Motion.

Publications

@inproceedings{oliver1997lafter,
  author = {Oliver, Nuria and Pentland, Alex P. and Berard, Francois},
  title = {Lafter: Lips and face real time tracker},
  booktitle = {Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition},
  year = 1997,
  pages = {123--129},
  publisher = {IEEE},
  doi = {10.1109/cvpr.1997.609309},
  award = {Best Paper Award Nominee. Winner to the Best Presentation Award},
  cites = 376,
  citesdate = {2026-03-23}
}

@article{oliver2000lafter,
  author = {Nuria Oliver and Alex P. Pentland and Fran{\c{c}}ois B{\'{e}}rard},
  title = {{LAFTER}: A Real-Time Face and Lips Tracker with Facial Expression Recognition},
  journal = {Pattern Recognition},
  volume = 33,
  number = 8,
  pages = {1369--1382},
  year = 2000,
  doi = {10.1016/s0031-3203(99)00113-2},
  award = {Special Mention Award from the Pattern Recognition Society as an Outstanding Publication},
  cites = 219,
  citesdate = {2026-03-23}
}