Perceptual Intelligence: Modeling Human Interactions
Nuria Oliver, Barbara Rosario and Alex Pentland — MIT Media Lab
Detection was only the start. The harder problem was recognizing what was
happening between two people: whether they were meeting, following one
another, changing direction to talk, or continuing separately.
The system combined bottom-up visual tracking with top-down behavioral models
in a feedback loop. Hidden Markov models represented individual activity;
coupled hidden Markov models represented the dependencies between two people.
The experiments found that the coupled models recognized interactions more
efficiently and accurately than treating each person's behavior independently.
Real labeled footage was scarce. A synthetic-agent simulator supplied examples
instead, and models trained on those generated trajectories could then
classify real sequences without additional tuning. It was an unusual shortcut,
but a practical response to the cost of collecting interaction video in the
late 1990s.
The evaluation covered a defined vocabulary of encounters, such as following,
meeting, talking, and separating, rather than unrestricted social behavior.
Its success on those sequences therefore should not be read as general
understanding of people's intentions.
Nuria Oliver, Barbara Rosario, Alex Pentland (1998). Statistical modeling of human interactions. CVPR Workshop on Interpretation of Visual Motion, 39-46. https://nuriaoliver.com/papers/cvpr98.pdf
BibTeX
@inproceedings{oliver1998statistical,
author = {Oliver, Nuria and Rosario, Barbara and Pentland, Alex},
title = {Statistical modeling of human interactions},
booktitle = {CVPR Workshop on Interpretation of Visual Motion},
year = 1998,
pages = {39--46},
url = {https://nuriaoliver.com/papers/cvpr98.pdf},
cites = 54
}
Nuria Oliver, Barbara Rosario, Alex Pentland (1999). A Bayesian computer vision system for modeling human interactions. International Conference on Computer Vision Systems (ICVS). Springer, 255-272. https://doi.org/10.1007/3-540-49256-9_16
BibTeX
@inproceedings{oliver1999bayesian,
author = {Oliver, Nuria and Rosario, Barbara and Pentland, Alex},
title = {A Bayesian computer vision system for modeling human interactions},
booktitle = {International Conference on Computer Vision Systems (ICVS). Springer},
year = 1999,
pages = {255--272},
doi = {10.1007/3-540-49256-9_16},
cites = 2255,
citesdate = {2026-03-23}
}
Nuria Oliver, Barbara Rosario, Alex Pentland (1999). Graphical Models for Recognizing Human Interactions. Advances in Neural Information Processing Systems (NIPS 1999), 924-930. https://proceedings.neurips.cc/paper_files/paper/1998/hash/3a20f62a0af1aa152670bab3c602feed-Abstract.html
BibTeX
@inproceedings{oliver1999graphical,
author = {Nuria Oliver and Barbara Rosario and Alex Pentland},
title = {Graphical Models for Recognizing Human Interactions},
booktitle = {Advances in Neural Information Processing Systems (NIPS 1999)},
pages = {924--930},
year = 1999,
url = {https://proceedings.neurips.cc/paper_files/paper/1998/hash/3a20f62a0af1aa152670bab3c602feed-Abstract.html},
cites = 63
}
Barbara Rosario, Nuria Oliver, Alex Pentland (1999). A Synthetic Agent System for Bayesian Modeling of Human Interactions. Proceedings of the 3rd Annual Conference on Autonomous Agents (ICAA 1999), 342-343. https://doi.org/10.1145/301136.301225
BibTeX
@inproceedings{rosario1999synthetic,
author = {Barbara Rosario and Nuria Oliver and Alex Pentland},
title = {A Synthetic Agent System for {Bayesian} Modeling of Human Interactions},
booktitle = {Proceedings of the 3rd Annual Conference on Autonomous Agents (ICAA 1999)},
pages = {342--343},
year = 1999,
doi = {10.1145/301136.301225},
cites = 9,
citesdate = {2026-03-23}
}
Nuria Oliver, Barbara Rosario, Alex P. Pentland (2000). A Bayesian Computer Vision System for Modeling Human Interactions. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8), 831-843. https://doi.org/10.1109/34.868684
BibTeX
@article{oliver2000bayesian,
author = {Nuria Oliver and Barbara Rosario and Alex P. Pentland},
title = {A {Bayesian} Computer Vision System for Modeling Human Interactions},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
volume = 22,
number = 8,
pages = {831--843},
year = 2000,
doi = {10.1109/34.868684},
cites = 1569,
citesdate = {2026-03-23}
}