S-Seer: Guiding Perceptual Sensing and Analysis with Value of Information

Selective sensing

The original Seer processed video, audio, sound localization, and computer-interaction signals to recognize office activity. S-Seer asked a practical follow-up question: when perceptual analysis is expensive, which observation is worth computing next?

S-Seer used expected value of information (EVI) to compare the expected diagnostic benefit of a sensor feature with its computational cost. It could pause a feature when the current evidence was already decisive and reactivate it as confidence fell. The experiments compared this policy with always-on processing, fixed-rate sampling, and random feature selection.

On 600 office-activity sequences—100 for each of six activities—the EVI policy produced per-activity accuracies from 97.8% to 100%, close to processing every feature continuously. Depending on the activity, CPU use fell from 44.3–67.1% with all features to 19.6–56.5% with EVI. The saving came with a limitation: because features remained off while the model was confident, S-Seer could take longer to recognize a transition. Its choices also depended on the utility and cost values supplied to the policy.

S-Seer publications

Journal paper
PDF DOI
Cite
Formatted citation

Nuria Oliver, Eric Horvitz (2005). Selective Perception Policies for Guiding Sensing and Computation in Multimodal Systems: A Comparative Analysis. Computer Vision and Image Understanding 100(1-2), 198-224. https://doi.org/10.1016/j.cviu.2004.12.004

BibTeX
@article{oliver2005selective,
  author = {Nuria Oliver and Eric Horvitz},
  title = {Selective Perception Policies for Guiding Sensing and Computation in Multimodal Systems: A Comparative Analysis},
  journal = {Computer Vision and Image Understanding},
  volume = 100,
  number = {1-2},
  pages = {198--224},
  year = 2005,
  doi = {10.1016/j.cviu.2004.12.004},
  cites = 41,
  citesdate = {2026-03-23}
}

Selective Perception Policies for Guiding Sensing and Computation in Multimodal Systems: A Comparative Analysis

Nuria Oliver, Eric Horvitz
Computer Vision and Image Understanding · 2005
Conference paper
DOI
Cite
Formatted citation

Nuria Oliver, Eric Horvitz (2005). S-seer: Selective Perception in a Multimodal Office Activity Recognition System. International Workshop on Machine Learning for Multimodal Interaction (MLMI 2004), 122-135. https://doi.org/10.1007/978-3-540-30568-2_11

BibTeX
@inproceedings{oliver2005sseer,
  author = {Nuria Oliver and Eric Horvitz},
  title = {{S-seer}: Selective Perception in a Multimodal Office Activity Recognition System},
  booktitle = {International Workshop on Machine Learning for Multimodal Interaction (MLMI 2004)},
  publisher = {Springer},
  pages = {122--135},
  year = 2005,
  doi = {10.1007/978-3-540-30568-2_11},
  cites = 24,
  citesdate = {2026-03-23}
}

S-seer: Selective Perception in a Multimodal Office Activity Recognition System

Nuria Oliver, Eric Horvitz
International Workshop on Machine Learning for Multimodal Interaction (MLMI 2004) · 2005
Conference paper
External link
Cite
Formatted citation

Andrew Wilson, Nuria Oliver (2005). Multimodal Sensing for Explicit and Implicit Interaction. 11th International Conference on Human-Computer Interaction (HCI International 2005). https://www.microsoft.com/en-us/research/publication/multimodal-sensing-explicit-implicit-interaction/

BibTeX
@inproceedings{wilson2005multimodal,
  author = {Andrew Wilson and Nuria Oliver},
  title = {Multimodal Sensing for Explicit and Implicit Interaction},
  booktitle = {11th International Conference on Human-Computer Interaction (HCI International 2005)},
  address = {Las Vegas, Nevada, USA},
  year = 2005,
  url = {https://www.microsoft.com/en-us/research/publication/multimodal-sensing-explicit-implicit-interaction/},
  cites = 31
}

Multimodal Sensing for Explicit and Implicit Interaction

Andrew Wilson, Nuria Oliver
11th International Conference on Human-Computer Interaction (HCI International 2005) · 2005

Seer foundations

Journal paper
PDF DOI
Cite
Formatted citation

Nuria Oliver, Ashutosh Garg, Eric Horvitz (2004). Layered Representations for Learning and Inferring Office Activity from Multiple Sensory Channels. Computer Vision and Image Understanding 96(2), 163-180. https://doi.org/10.1016/j.cviu.2004.02.004

BibTeX
@article{oliver2004layered,
  author = {Nuria Oliver and Ashutosh Garg and Eric Horvitz},
  title = {Layered Representations for Learning and Inferring Office Activity from Multiple Sensory Channels},
  journal = {Computer Vision and Image Understanding},
  volume = 96,
  number = 2,
  pages = {163--180},
  year = 2004,
  doi = {10.1016/j.cviu.2004.02.004},
  cites = 482,
  citesdate = {2026-03-23}
}

Layered Representations for Learning and Inferring Office Activity from Multiple Sensory Channels

Nuria Oliver, Ashutosh Garg, Eric Horvitz
Computer Vision and Image Understanding · 2004
Conference paper
PDF DOI
Cite
Formatted citation

Nuria Oliver, Eric Horvitz, Ashutosh Garg (2002). Layered Representations for Human Activity Recognition. Proceedings. Fourth IEEE International Conference on Multimodal Interfaces, 3-8. https://doi.org/10.1109/ICMI.2002.1166960

BibTeX
@inproceedings{oliver2002layered,
  author = {Nuria Oliver and Eric Horvitz and Ashutosh Garg},
  title = {Layered Representations for Human Activity Recognition},
  booktitle = {Proceedings. Fourth IEEE International Conference on Multimodal Interfaces},
  pages = {3--8},
  year = 2002,
  doi = {10.1109/ICMI.2002.1166960},
  award = {ACM ICMI Ten Year Technical Impact Award},
  awardyear = {2002/2014},
  cites = 472,
  citesdate = {2026-03-23}
}
★ ACM ICMI Ten Year Technical Impact Award (2002/2014)

Layered Representations for Human Activity Recognition

Nuria Oliver, Eric Horvitz, Ashutosh Garg
Proceedings. Fourth IEEE International Conference on Multimodal Interfaces · 2002
Technical report
PDF External link
Cite
Formatted citation

Nuria Oliver, Eric Horvitz, Ashutosh Garg (2002). Hierarchical Representations for Learning and Inferring Office Activity from Multiple Sensory Channels. Microsoft Research. https://www.microsoft.com/en-us/research/publication/hierarchical-representations-for-learning-and-inferring-office-activity-from-multiple-sensory-channels/

BibTeX
@techreport{oliver2002hierarchical,
  author = {Oliver, Nuria and Horvitz, Eric and Garg, Ashutosh},
  title = {Hierarchical Representations for Learning and Inferring Office Activity from Multiple Sensory Channels},
  institution = {Microsoft Research},
  year = 2002,
  url = {https://www.microsoft.com/en-us/research/publication/hierarchical-representations-for-learning-and-inferring-office-activity-from-multiple-sensory-channels/}
}

Hierarchical Representations for Learning and Inferring Office Activity from Multiple Sensory Channels

Nuria Oliver, Eric Horvitz, Ashutosh Garg
Microsoft Research · 2002

Videos