3D Modeling and Tracking of Human Lip Motion

Sumit Basu, Nuria Oliver and Alex Pentland — MIT Media Lab

This project asked whether the three-dimensional shape of a person's lips could be recovered from ordinary two-dimensional video. The model began with generic finite-element physics, then learned the characteristic deformations of real lips from observed motion.

Those learned modes constrained the shape between sparse observations. The system could estimate lip shape and pose without lipstick or special markers, and could use the same representation for both analysis and synthesis.

The result was a method for reconstructing and synthesizing lip shape, not a complete speech-reading or facial-expression system. The archived project description does not report the scale or demographic range of the video data, so its robustness across speakers cannot be assessed from that account.

This was related to LAFTER, but the two systems solved different problems. LAFTER concentrated on robust real-time face and mouth tracking; this work built a deformable 3D model of the lips themselves.

The model combined physical constraints, learned deformation modes, and measurements from video.

Publications

Journal paper
External link
Cite
Formatted citation

Sumit Basu, Nuria Oliver, Alex Pentland (1998). 3D lip shapes from video: A combined physical–statistical model. Speech Communication 26(1-2), 131-148. https://doi.org/10.1016/s0167-6393(98)00055-7

BibTeX
@article{basu19983d,
 author = {Sumit Basu and Nuria Oliver and Alex Pentland},
 doi = {10.1016/s0167-6393(98)00055-7},
 journal = {Speech Communication},
 number = {1-2},
 pages = {131--148},
 publisher = {Elsevier BV},
 title = {3D lip shapes from video: A combined physical–statistical model},
 url = {https://doi.org/10.1016/s0167-6393%2898%2900055-7},
 volume = {26},
 year = {1998}
}

3D lip shapes from video: A combined physical–statistical model

Sumit Basu, Nuria Oliver, Alex Pentland
Speech Communication · 1998
Conference paper
External link
Cite
Formatted citation

Sumit Basu, Nuria Oliver, Alex Pentland (1998). Coding human lip motions with a learned 3D model. Workshop on Very Low Bitrate Video (VLBV'98), 8-9. http://alumni.media.mit.edu/~sbasu/papers/vlbv98.pdf

BibTeX
@inproceedings{basu1998coding,
  author = {Basu, Sumit and Oliver, Nuria and Pentland, Alex},
  title = {Coding human lip motions with a learned 3D model},
  booktitle = {Workshop on Very Low Bitrate Video (VLBV'98)},
  year = 1998,
  address = {Urbana},
  pages = {8--9},
  url = {http://alumni.media.mit.edu/~sbasu/papers/vlbv98.pdf}
}

Coding human lip motions with a learned 3D model

Sumit Basu, Nuria Oliver, Alex Pentland
Workshop on Very Low Bitrate Video (VLBV'98) · 1998
Conference paper
DOI
Cite
Formatted citation

Sumit Basu, Nuria Oliver, Alex Pentland (1998). 3D modeling and tracking of human lip motions. Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271), 337-343. https://doi.org/10.1109/ICCV.1998.710740

BibTeX
@inproceedings{basu1998modeling,
  author = {Basu, Sumit and Oliver, Nuria and Pentland, Alex},
  title = {3D modeling and tracking of human lip motions},
  booktitle = {Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271)},
  year = 1998,
  pages = {337--343},
  publisher = {Narosa Publishing House},
  doi = {10.1109/ICCV.1998.710740},
  cites = 144,
  citesdate = {2026-03-23}
}

3D modeling and tracking of human lip motions

Sumit Basu, Nuria Oliver, Alex Pentland
Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271) · 1998