3D Modeling and Tracking of Human Lip Motion

Sumit Basu, Nuria Oliver and Alex Pentland, MIT Media Lab

We asked whether the three-dimensional shape of a person's lips could be recovered from ordinary two-dimensional video. We began with generic finite-element physics, then learned the characteristic deformations of real lips from observed motion.

The learned modes constrained the shape between sparse observations. We could estimate lip shape and pose without lipstick or special markers and use the same representation for both analysis and synthesis.

That common representation worked in both directions. For analysis, the system estimated the model's deformation and pose from an image sequence. For synthesis, changing those same parameters generated a new lip shape. We did not need separate descriptions for tracking and animation.

The result was a method for reconstructing and synthesizing lip shape, not a complete speech-reading or facial-expression system.

This was related to LAFTER, but we were solving a different problem. LAFTER concentrated on robust real-time face and mouth tracking; here, we built a deformable 3D model of the lips themselves.

Diagram connecting video measurements, physical constraints, learned deformation modes, and a reconstructed lip model.

The model combined physical constraints, learned deformation modes, and measurements from video so that the same parameters could support analysis and synthesis.

Publications

Journal paper
External link
Cite
Formatted citation

Sumit Basu, Nuria Oliver, Alex Pentland (1998). 3D lip shapes from video: A combined physical–statistical model. Speech Communication 26(1-2), 131-148. https://doi.org/10.1016/s0167-6393(98)00055-7

BibTeX
@article{basu19983d,
 author = {Sumit Basu and Nuria Oliver and Alex Pentland},
 doi = {10.1016/s0167-6393(98)00055-7},
 journal = {Speech Communication},
 number = {1-2},
 pages = {131--148},
 publisher = {Elsevier BV},
 title = {3D lip shapes from video: A combined physical–statistical model},
 url = {https://doi.org/10.1016/s0167-6393%2898%2900055-7},
 volume = {26},
 year = {1998}
}

3D lip shapes from video: A combined physical–statistical model

Sumit Basu, Nuria Oliver, Alex Pentland
Speech Communication · 1998
Conference paper
External link
Cite
Formatted citation

Sumit Basu, Nuria Oliver, Alex Pentland (1998). Coding human lip motions with a learned 3D model. Workshop on Very Low Bitrate Video (VLBV'98), 8-9. http://alumni.media.mit.edu/~sbasu/papers/vlbv98.pdf

BibTeX
@inproceedings{basu1998coding,
  author = {Basu, Sumit and Oliver, Nuria and Pentland, Alex},
  title = {Coding human lip motions with a learned 3D model},
  booktitle = {Workshop on Very Low Bitrate Video (VLBV'98)},
  year = 1998,
  address = {Urbana},
  pages = {8--9},
  url = {http://alumni.media.mit.edu/~sbasu/papers/vlbv98.pdf}
}

Coding human lip motions with a learned 3D model

Sumit Basu, Nuria Oliver, Alex Pentland
Workshop on Very Low Bitrate Video (VLBV'98) · 1998
Conference paper
DOI
Cite
Formatted citation

Sumit Basu, Nuria Oliver, Alex Pentland (1998). 3D modeling and tracking of human lip motions. Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271), 337-343. https://doi.org/10.1109/ICCV.1998.710740

BibTeX
@inproceedings{basu1998modeling,
  author = {Basu, Sumit and Oliver, Nuria and Pentland, Alex},
  title = {3D modeling and tracking of human lip motions},
  booktitle = {Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271)},
  year = 1998,
  pages = {337--343},
  publisher = {Narosa Publishing House},
  doi = {10.1109/ICCV.1998.710740},
  cites = 144,
  citesdate = {2026-03-23}
}

3D modeling and tracking of human lip motions

Sumit Basu, Nuria Oliver, Alex Pentland
Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271) · 1998