Towards Perceptual Intelligence: Nuria Oliver's MIT Thesis
Nuria Oliver — MIT Media Lab doctoral thesis, 2000
The thesis asked whether a computer could move beyond detecting that a person
was present and begin to recognize what the person was doing. Could it follow
an action as it unfolded, account for interactions with other people or
machines, and make a useful prediction about what might happen next?
Nuria called this goal perceptual intelligence. The central idea was to
connect perception and interpretation rather than treat them as separate
steps. Cameras and other sensors supplied observations from the physical
world. Statistical models used those observations, together with expectations
learned from earlier examples, to identify a behavior and feed predictions
back to the perceptual system.
One framework, four testbeds
The thesis tested this approach on behaviors of increasing complexity:
Facial expressions.LAFTER detected and tracked a
face and mouth in real time, then recognized a small set of expressions.
Two-handed gestures.Coupled hidden Markov
models represented the separate but coordinated movements
of both hands in T'ai Chi.
Interactions between people. A visual system distinguished encounters
such as following, meeting, talking, and separating. This became the
human-interaction modeling project.
Behavior involving a person and a machine. The
SmartCar combined signals from the driver, the
vehicle, and surrounding traffic to recognize and predict driving
maneuvers.
Using the same general approach across these systems was important. A smile,
a two-handed gesture, a conversation between pedestrians, and a lane change
look very different, but all unfold over time and depend on context.
The modeling idea in plain language
A hidden Markov model, or HMM, represents an activity as a sequence of states
that cannot be observed directly but leave visible evidence. A driver's
intention to change lanes is hidden, for example, while head movement,
steering, speed, and nearby traffic can be measured.
The thesis extended this idea with coupled hidden Markov models. Instead of
modeling two streams independently, the coupled model allowed each to affect
the other. That made it possible to represent two hands moving together or two
people responding to one another.
Training such models normally requires many labeled examples. Those examples
were difficult and expensive to collect, particularly for interactions
between people. The thesis therefore also explored using simulated behavior
to create an initial model, which could then be refined with a smaller amount
of real data.
What the thesis demonstrated
Across the four testbeds, the models recognized defined behaviors from noisy
sensor data and could often identify an action before it was complete. The
work showed how bottom-up measurements and top-down predictions could operate
in one probabilistic framework.
The scope was deliberately bounded. Each system recognized a specified set of
expressions, gestures, encounters, or driving maneuvers. It did not give a
computer a general understanding of human intentions, and results from these
late-1990s testbeds should not be read as evidence of performance across
unrestricted settings or populations.
Nuria Oliver (2000). Towards Perceptual Intelligence: Statistical Modeling of Human Individual and Interactive Behaviors. Massachusetts Institute of Technology. https://nuriaoliver.com/thesis/thesisNuriaOliver.pdf
BibTeX
@phdthesis{oliver2000thesis,
author = {Nuria Oliver},
title = {Towards Perceptual Intelligence: Statistical Modeling of Human Individual and Interactive Behaviors},
school = {Massachusetts Institute of Technology},
year = 2000,
url = {https://nuriaoliver.com/thesis/thesisNuriaOliver.pdf},
cites = 51
}