Seer: Layered Representations for Learning and Inferring Office Activity from Multiple Streams of Information

System

Seer was a real-time office-awareness system that combined video, audio, and keyboard and mouse activity. It used Layered Hidden Markov Models (LHMMs): lower layers interpreted individual sensor streams, and a higher layer combined those outputs over a longer time scale. The system modeled six situations—phone and face-to-face conversations, computer work, a distant conversation, a presentation, and an empty office.

Layering was intended to reduce the number of parameters and isolate changes in individual sensors. In an experiment with 60 minutes of office activity from three users, split equally between training and testing, the LHMM used 670 parameters and classified the six activities with 99.7% accuracy. A single Cartesian-product HMM used 1,360 parameters and reached 72.7% on the same data.

This was a small, controlled evaluation rather than evidence of general office deployment. When Seer was moved between offices, the higher-level models transferred well, but some low-level models still needed retraining for differences in ambient noise, background, and illumination. S-Seer later addressed another practical constraint: deciding which expensive sensor analyses to run at any moment.

SEER demonstration during Bill Gates' keynote at IJCAI 2001.

Publications

Conference paper
DOI
Cite
Formatted citation

Nuria Oliver, Eric Horvitz (2005). A Comparison of HMMs and Dynamic Bayesian Networks for Recognizing Office Activities. International Conference on User Modeling (UM 2005), 199-209. https://doi.org/10.1007/11527886_26

BibTeX
@inproceedings{oliver2005hmm,
  author = {Nuria Oliver and Eric Horvitz},
  title = {A Comparison of {HMMs} and Dynamic {Bayesian} Networks for Recognizing Office Activities},
  booktitle = {International Conference on User Modeling (UM 2005)},
  publisher = {Springer},
  pages = {199--209},
  year = 2005,
  doi = {10.1007/11527886_26},
  cites = 135,
  citesdate = {2026-03-23}
}

A Comparison of HMMs and Dynamic Bayesian Networks for Recognizing Office Activities

Nuria Oliver, Eric Horvitz
International Conference on User Modeling (UM 2005) · 2005
Journal paper
PDF DOI
Cite
Formatted citation

Nuria Oliver, Eric Horvitz (2005). Selective Perception Policies for Guiding Sensing and Computation in Multimodal Systems: A Comparative Analysis. Computer Vision and Image Understanding 100(1-2), 198-224. https://doi.org/10.1016/j.cviu.2004.12.004

BibTeX
@article{oliver2005selective,
  author = {Nuria Oliver and Eric Horvitz},
  title = {Selective Perception Policies for Guiding Sensing and Computation in Multimodal Systems: A Comparative Analysis},
  journal = {Computer Vision and Image Understanding},
  volume = 100,
  number = {1-2},
  pages = {198--224},
  year = 2005,
  doi = {10.1016/j.cviu.2004.12.004},
  cites = 41,
  citesdate = {2026-03-23}
}

Selective Perception Policies for Guiding Sensing and Computation in Multimodal Systems: A Comparative Analysis

Nuria Oliver, Eric Horvitz
Computer Vision and Image Understanding · 2005
Journal paper
PDF DOI
Cite
Formatted citation

Nuria Oliver, Ashutosh Garg, Eric Horvitz (2004). Layered Representations for Learning and Inferring Office Activity from Multiple Sensory Channels. Computer Vision and Image Understanding 96(2), 163-180. https://doi.org/10.1016/j.cviu.2004.02.004

BibTeX
@article{oliver2004layered,
  author = {Nuria Oliver and Ashutosh Garg and Eric Horvitz},
  title = {Layered Representations for Learning and Inferring Office Activity from Multiple Sensory Channels},
  journal = {Computer Vision and Image Understanding},
  volume = 96,
  number = 2,
  pages = {163--180},
  year = 2004,
  doi = {10.1016/j.cviu.2004.02.004},
  cites = 482,
  citesdate = {2026-03-23}
}

Layered Representations for Learning and Inferring Office Activity from Multiple Sensory Channels

Nuria Oliver, Ashutosh Garg, Eric Horvitz
Computer Vision and Image Understanding · 2004
Conference paper
PDF DOI
Cite
Formatted citation

Nuria Oliver, Eric Horvitz, Ashutosh Garg (2002). Layered Representations for Human Activity Recognition. Proceedings. Fourth IEEE International Conference on Multimodal Interfaces, 3-8. https://doi.org/10.1109/ICMI.2002.1166960

BibTeX
@inproceedings{oliver2002layered,
  author = {Nuria Oliver and Eric Horvitz and Ashutosh Garg},
  title = {Layered Representations for Human Activity Recognition},
  booktitle = {Proceedings. Fourth IEEE International Conference on Multimodal Interfaces},
  pages = {3--8},
  year = 2002,
  doi = {10.1109/ICMI.2002.1166960},
  award = {ACM ICMI Ten Year Technical Impact Award},
  awardyear = {2002/2014},
  cites = 472,
  citesdate = {2026-03-23}
}
★ ACM ICMI Ten Year Technical Impact Award (2002/2014)

Layered Representations for Human Activity Recognition

Nuria Oliver, Eric Horvitz, Ashutosh Garg
Proceedings. Fourth IEEE International Conference on Multimodal Interfaces · 2002
Technical report
PDF External link
Cite
Formatted citation

Nuria Oliver, Eric Horvitz, Ashutosh Garg (2002). Hierarchical Representations for Learning and Inferring Office Activity from Multiple Sensory Channels. Microsoft Research. https://www.microsoft.com/en-us/research/publication/hierarchical-representations-for-learning-and-inferring-office-activity-from-multiple-sensory-channels/

BibTeX
@techreport{oliver2002hierarchical,
  author = {Oliver, Nuria and Horvitz, Eric and Garg, Ashutosh},
  title = {Hierarchical Representations for Learning and Inferring Office Activity from Multiple Sensory Channels},
  institution = {Microsoft Research},
  year = 2002,
  url = {https://www.microsoft.com/en-us/research/publication/hierarchical-representations-for-learning-and-inferring-office-activity-from-multiple-sensory-channels/}
}

Hierarchical Representations for Learning and Inferring Office Activity from Multiple Sensory Channels

Nuria Oliver, Eric Horvitz, Ashutosh Garg
Microsoft Research · 2002
Patent
Cite
Formatted citation

Nuria Oliver, Eric Horvitz, Ashutosh Garg (2007). Layered models for context awareness.

BibTeX
@patent{oliver2007context,
  author = {Oliver, Nuria and Horvitz, Eric and Garg, Ashutosh},
  title = {Layered models for context awareness},
  number = 7203635,
  type = {patentus},
  year = 2007,
  note = {Priority 2002-06-27, Filed 2002-06-27, Granted 2007-04-10},
  cites = 185
}

Layered models for context awareness

Nuria Oliver, Eric Horvitz, Ashutosh Garg
2007

Presentation

Videos