Seer: Layered Representations for Learning and Inferring Office Activity from Multiple Streams of Information
System
We built Seer as a real-time office-awareness system that combined video,
audio, and keyboard and mouse activity. We used Layered Hidden Markov Models
(LHMMs): lower layers interpreted individual sensor streams, and a higher
layer combined those outputs over a longer time scale. The system modeled six
situations: phone and face-to-face conversations, computer work, a distant
conversation, a presentation, and an empty office.
We used layers to reduce the number of parameters and isolate changes in
individual sensors. In an experiment with 60 minutes of office activity from
three users, split equally between training and testing, our LHMM used 670
parameters and classified the six activities with 99.7% accuracy. A single
Cartesian-product HMM used 1,360 parameters and reached 72.7% on the same
data.
This was a small, controlled evaluation rather than evidence of general office
deployment. When we moved Seer between offices, the higher-level models
transferred well, but some low-level models still needed retraining for
differences in ambient noise, background, and illumination.
S-Seer later addressed another practical constraint: deciding
which expensive sensor analyses to run at any moment.
SEER demonstration during Bill Gates' keynote at IJCAI 2001.
Nuria Oliver, Eric Horvitz (2005). A Comparison of HMMs and Dynamic Bayesian Networks for Recognizing Office Activities. International Conference on User Modeling (UM 2005), 199-209. https://doi.org/10.1007/11527886_26
BibTeX
@inproceedings{oliver2005hmm,
author = {Nuria Oliver and Eric Horvitz},
title = {A Comparison of {HMMs} and Dynamic {Bayesian} Networks for Recognizing Office Activities},
booktitle = {International Conference on User Modeling (UM 2005)},
publisher = {Springer},
pages = {199--209},
year = 2005,
doi = {10.1007/11527886_26},
cites = 135,
citesdate = {2026-03-23}
}
Nuria Oliver, Eric Horvitz (2005). Selective Perception Policies for Guiding Sensing and Computation in Multimodal Systems: A Comparative Analysis. Computer Vision and Image Understanding 100(1-2), 198-224. https://doi.org/10.1016/j.cviu.2004.12.004
BibTeX
@article{oliver2005selective,
author = {Nuria Oliver and Eric Horvitz},
title = {Selective Perception Policies for Guiding Sensing and Computation in Multimodal Systems: A Comparative Analysis},
journal = {Computer Vision and Image Understanding},
volume = 100,
number = {1-2},
pages = {198--224},
year = 2005,
doi = {10.1016/j.cviu.2004.12.004},
cites = 41,
citesdate = {2026-03-23}
}
Nuria Oliver, Ashutosh Garg, Eric Horvitz (2004). Layered Representations for Learning and Inferring Office Activity from Multiple Sensory Channels. Computer Vision and Image Understanding 96(2), 163-180. https://doi.org/10.1016/j.cviu.2004.02.004
BibTeX
@article{oliver2004layered,
author = {Nuria Oliver and Ashutosh Garg and Eric Horvitz},
title = {Layered Representations for Learning and Inferring Office Activity from Multiple Sensory Channels},
journal = {Computer Vision and Image Understanding},
volume = 96,
number = 2,
pages = {163--180},
year = 2004,
doi = {10.1016/j.cviu.2004.02.004},
cites = 482,
citesdate = {2026-03-23}
}
Nuria Oliver, Eric Horvitz, Ashutosh Garg (2002). Layered Representations for Human Activity Recognition. Proceedings. Fourth IEEE International Conference on Multimodal Interfaces, 3-8. https://doi.org/10.1109/ICMI.2002.1166960
BibTeX
@inproceedings{oliver2002layered,
author = {Nuria Oliver and Eric Horvitz and Ashutosh Garg},
title = {Layered Representations for Human Activity Recognition},
booktitle = {Proceedings. Fourth IEEE International Conference on Multimodal Interfaces},
pages = {3--8},
year = 2002,
doi = {10.1109/ICMI.2002.1166960},
award = {ACM ICMI Ten Year Technical Impact Award},
awardyear = {2002/2014},
cites = 472,
citesdate = {2026-03-23}
}
★ ACM ICMI Ten Year Technical Impact Award (2002/2014)
Nuria Oliver, Eric Horvitz, Ashutosh Garg (2002). Hierarchical Representations for Learning and Inferring Office Activity from Multiple Sensory Channels. Microsoft Research. https://www.microsoft.com/en-us/research/publication/hierarchical-representations-for-learning-and-inferring-office-activity-from-multiple-sensory-channels/
BibTeX
@techreport{oliver2002hierarchical,
author = {Oliver, Nuria and Horvitz, Eric and Garg, Ashutosh},
title = {Hierarchical Representations for Learning and Inferring Office Activity from Multiple Sensory Channels},
institution = {Microsoft Research},
year = 2002,
url = {https://www.microsoft.com/en-us/research/publication/hierarchical-representations-for-learning-and-inferring-office-activity-from-multiple-sensory-channels/}
}
Nuria Oliver, Eric Horvitz, Ashutosh Garg (2007). Layered models for context awareness.
BibTeX
@patent{oliver2007context,
author = {Oliver, Nuria and Horvitz, Eric and Garg, Ashutosh},
title = {Layered models for context awareness},
number = 7203635,
type = {patentus},
year = 2007,
note = {Priority 2002-06-27, Filed 2002-06-27, Granted 2007-04-10},
cites = 185
}
