Coupled Hidden Markov Models for Complex Action Recognition

Nuria Oliver, Matthew Brand and Alex Pentland — MIT Media Lab

Many actions contain several processes unfolding at once. The two hands in a T'ai Chi movement, for example, follow their own trajectories but are not independent of each other.

This project introduced coupled hidden Markov models (CHMMs), linking separate state sequences so that each could influence the others. The paper developed algorithms for training the coupled model and compared it with a conventional HMM on a vision task classifying two-handed movements.

The CHMMs trained faster, produced better model likelihoods, and were less sensitive to their initial conditions in the reported experiments. The same idea later supported models of interactions between people and driver maneuvers in the SmartCar.

The comparison was made on a bounded vision task with two synchronized processes: the trajectories of the two hands in a set of T'ai Chi gestures. It established the value of coupling for that task, not performance on arbitrary actions or larger groups of interacting processes.

Publication

@inproceedings{brand1997coupled,
  author = {Brand, Matthew and Oliver, Nuria and Pentland, Alex},
  title = {Coupled hidden Markov models for complex action recognition},
  booktitle = {Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition},
  year = 1997,
  pages = {994--999},
  publisher = {IEEE},
  doi = {10.1109/cvpr.1997.609450},
  cites = 1622,
  citesdate = {2026-03-23}
}

Gesture examples

Examples of gestures robustly classified by CHMMs: