Graphical Models for Driver Behavior Recognition and Prediction in a SmartCar

Nuria Oliver and Alex Pentland — MIT Media Lab / Microsoft Research

Problem and testbed

We developed the SmartCar to study whether a vehicle could recognize a driver's tactical maneuver from both the driver's actions and the surrounding context. The testbed combined real-time data acquisition and playback with dynamical graphical models. Four video signals captured surrounding traffic, the driver's head position, and the driver's viewpoint; the car also recorded brake, gear, steering-wheel angle, speed, and throttle signals.

The archived account states that, over two months, more than 70 drivers operated the SmartCar for 1.25 hours in the greater Boston area. Hidden Markov models (HMMs), with coupled HMMs considered as an extension, were trained on these data to represent seven maneuvers: passing, changing lanes left and right, turning left and right, starting, and stopping.

The resulting models recognized and predicted these labeled maneuvers and exposed their internal state as a drive unfolded, as the passing video below illustrates. This made the work relevant to car simulators and adaptive driver interfaces. The experiments were limited to a fixed set of tactical maneuvers and driving around greater Boston; they did not establish that the models could prevent dangerous situations in general.

Publications

Conference paper
PDF DOI
Cite
Formatted citation

Nuria Oliver, Alex P. Pentland (2000). Graphical Models for Driver Behavior Recognition in a SmartCar. Proceedings of the IEEE Intelligent Vehicles Symposium 2000, 7-12. https://doi.org/10.1117/12.389351

BibTeX
@inproceedings{oliver2000graphical,
  author = {Nuria Oliver and Alex P. Pentland},
  title = {Graphical Models for Driver Behavior Recognition in a {SmartCar}},
  booktitle = {Proceedings of the IEEE Intelligent Vehicles Symposium 2000},
  pages = {7--12},
  year = 2000,
  doi = {10.1117/12.389351},
  cites = 446,
  citesdate = {2026-03-23}
}

Graphical Models for Driver Behavior Recognition in a SmartCar

Nuria Oliver, Alex P. Pentland
Proceedings of the IEEE Intelligent Vehicles Symposium 2000 · 2000
Journal paper
PDF DOI
Cite
Formatted citation

Nuria Oliver, Alex P. Pentland (2000). Driver Behavior Recognition and Prediction in a SmartCar. Proceedings of SPIE 4023, 280-290. https://doi.org/10.1117/12.389351

BibTeX
@article{oliver2000driver,
  author = {Nuria Oliver and Alex P. Pentland},
  title = {Driver Behavior Recognition and Prediction in a {SmartCar}},
  journal = {Proceedings of SPIE},
  volume = 4023,
  pages = {280--290},
  year = 2000,
  doi = {10.1117/12.389351},
  cites = 140,
  citesdate = {2026-03-23}
}

Driver Behavior Recognition and Prediction in a SmartCar

Nuria Oliver, Alex P. Pentland
Proceedings of SPIE · 2000

Videos