A Bayesian Computer Vision System for Modeling Human Interactions

Nuria Oliver, Barbara Rosario and Alex Pentland — MIT Media Lab / Microsoft Research

Abstract

We describe a real-time computer vision and machine learning system for modeling and recognizing human behaviors in a visual surveillance task. The system is particularly concerned with detecting when interactions between people occur and classifying the type of interaction. Examples include following another person, altering one's path to meet another, and so forth.

Our system combines top-down with bottom-up information in a closed feedback loop, with both components employing a statistical Bayesian approach. We propose and compare two state-based learning architectures — HMMs and CHMMs — for modeling behaviors and interactions. The CHMM model is shown to work much more efficiently and accurately.

A synthetic agent training system is used to develop a priori models for recognizing human behaviors and interactions. We demonstrate the ability to use these a priori models to accurately classify real human behaviors and interactions with no additional tuning or training.

Publications

A Bayesian Computer Vision System for Recognizing Human Interactions Nuria Oliver, Barbara Rosario and Alex Pentland. CVPR 1998 Workshop on Interpretation of Visual Motion, Santa Barbara, June 1998.

Graphical Models for Recognizing Human Interactions Nuria Oliver, Barbara Rosario and Alex Pentland. NIPS 1998, Denver, December 1998.

A Bayesian Computer Vision System for Modeling Human Interactions Nuria Oliver, Barbara Rosario and Alex Pentland. ICVS 1999, Gran Canaria, January 1999.

A Synthetic Agent System for Bayesian Modeling of Human Interactions Nuria Oliver, Barbara Rosario and Alex Pentland. Autonomous Agents 1999, Seattle, May 1999.

Videos

Examples of interactions recognized by the system:

• Follow
• Meet, talk and continue together
• Meet, talk and go on separately
• Change direction, meet, talk and go on separately