Graphical Models for Driver Behavior Recognition and Prediction in a SmartCar

Nuria Oliver and Alex Pentland, MIT Media Lab / Microsoft Research

Problem and testbed

We developed the SmartCar to study whether a vehicle could recognize a driver's tactical maneuver from both the driver's actions and the surrounding context. The testbed combined real-time data acquisition and playback with dynamical graphical models. Four video signals captured surrounding traffic, the driver's head position, and the driver's viewpoint; the car also recorded brake, gear, steering-wheel angle, speed, and throttle signals.

Collecting those signals was a substantial part of the project. In the first acquisition system, speed, steering, brake, and gear sensors shared a serial connection and were polled individually by the host computer. Each reading was stored with its sensor identifier and a timestamp so that the recorded drive could later be replayed at approximately the original pace. On a Pentium 180, the system collected about 22 sets of readings per second.

The vehicle had forward-facing and driver-facing cameras, but their recordings still had to be synchronized with the sensor playback manually. We also implemented the first stage of GPS support by parsing position messages, although displaying the car on a digital road map remained future work. These constraints shaped the testbed as much as the later modeling did.

More than 70 drivers operated the SmartCar for 1.25 hours in the greater Boston area over two months. We trained hidden Markov models (HMMs), with coupled HMMs as an extension, to represent seven maneuvers: passing, changing lanes left and right, turning left and right, starting, and stopping.

Our models recognized and predicted these labeled maneuvers and exposed their internal state as a drive unfolded, as the passing video below illustrates. This made them relevant to car simulators and adaptive driver interfaces. We limited the experiments to a fixed set of tactical maneuvers and driving around greater Boston; they did not establish that the models could prevent dangerous situations in general.

The SmartCar testbed recorded the driver, vehicle controls, and surrounding traffic.

Sensor wiring connected the vehicle signals to the data-acquisition system.

Publications

@inproceedings{oliver2000graphical,
  author = {Nuria Oliver and Alex P. Pentland},
  title = {Graphical Models for Driver Behavior Recognition in a {SmartCar}},
  booktitle = {Proceedings of the IEEE Intelligent Vehicles Symposium 2000},
  pages = {7--12},
  year = 2000,
  doi = {10.1117/12.389351},
  cites = 446,
  citesdate = {2026-03-23}
}

@article{oliver2000driver,
  author = {Nuria Oliver and Alex P. Pentland},
  title = {Driver Behavior Recognition and Prediction in a {SmartCar}},
  journal = {Proceedings of SPIE},
  volume = 4023,
  pages = {280--290},
  year = 2000,
  doi = {10.1117/12.389351},
  cites = 140,
  citesdate = {2026-03-23}
}

Videos

• : labels in the bottom-right corner indicate the current HMM state at each moment.