LAFTER: Lips and Face Real Time Tracking and Facial Expression Recognition

Nuria Oliver, Francois Berard and Alex Pentland — MIT Media Lab

LAFTER was built when reliable real-time face tracking still required careful engineering. It found a face and mouth in live video, kept tracking them as the person moved, and classified mouth shapes associated with expressions such as happiness and sadness.

The system joined detection and tracking rather than treating them as separate steps. Detection could recover the face after tracking failed; tracking provided the stable measurements needed for expression recognition. It was tested with hundreds of users.

The archived account describes the system as reliable and accurate but gives no participant protocol or accuracy figures. Its expression labels were based on visible mouth shape, so they should not be treated as measurements of a person's internal emotional state.

The first paper received the conference's Best Presentation Award and was a Best Paper Award nominee. The later journal article received a Special Mention from the Pattern Recognition Society. A related project modeled the lips in three dimensions rather than classifying their visible appearance; see 3D Modeling of Human Lip Motion.

Publications

Conference paper
DOI
Cite
Formatted citation

Nuria Oliver, Alex P. Pentland, Francois Berard (1997). Lafter: Lips and face real time tracker. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 123-129. https://doi.org/10.1109/cvpr.1997.609309

BibTeX
@inproceedings{oliver1997lafter,
  author = {Oliver, Nuria and Pentland, Alex P. and Berard, Francois},
  title = {Lafter: Lips and face real time tracker},
  booktitle = {Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition},
  year = 1997,
  pages = {123--129},
  publisher = {IEEE},
  doi = {10.1109/cvpr.1997.609309},
  award = {Best Paper Award Nominee. Winner to the Best Presentation Award},
  cites = 376,
  citesdate = {2026-03-23}
}
★ Best Paper Award Nominee. Winner to the Best Presentation Award

Lafter: Lips and face real time tracker

Nuria Oliver, Alex P. Pentland, Francois Berard
Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition · 1997
Journal paper
PDF DOI
Cite
Formatted citation

Nuria Oliver, Alex P. Pentland, Fran\ccois B\'erard (2000). LAFTER: A Real-Time Face and Lips Tracker with Facial Expression Recognition. Pattern Recognition 33(8), 1369-1382. https://doi.org/10.1016/s0031-3203(99)00113-2

BibTeX
@article{oliver2000lafter,
  author = {Nuria Oliver and Alex P. Pentland and Fran{\c{c}}ois B{\'{e}}rard},
  title = {{LAFTER}: A Real-Time Face and Lips Tracker with Facial Expression Recognition},
  journal = {Pattern Recognition},
  volume = 33,
  number = 8,
  pages = {1369--1382},
  year = 2000,
  doi = {10.1016/s0031-3203(99)00113-2},
  award = {Special Mention Award from the Pattern Recognition Society as an Outstanding Publication},
  cites = 219,
  citesdate = {2026-03-23}
}
★ Special Mention Award from the Pattern Recognition Society as an Outstanding Publication

LAFTER: A Real-Time Face and Lips Tracker with Facial Expression Recognition

Nuria Oliver, Alex P. Pentland, Fran\ccois B\'erard
Pattern Recognition · 2000