Recommender systems typically learn a person's preferences from past
behavior. This research began with inconsistent movie ratings and sparse data,
then moved into context-aware models, ranking objectives, and mobile
recommendations used outside the lab.
"I Like It, I Like It Not": Noise in User Ratings
We asked participants to rate movies more than once and measured the
consistency of their answers. The experiment also considered the order in
which films were presented and the time between ratings.
Repeated ratings contained a measurable level of noise. A rating recorded on
one occasion is therefore not necessarily a definitive statement of a
person's preferences.
Xavier Amatriain, Josep M. Pujol, Nuria Oliver (2009). I Like It... I Like It Not: Evaluating User Ratings Noise in Recommender Systems. International Conference on User Modeling, Adaptation, and Personalization (UMAP 2009), 247-258. https://doi.org/10.1007/978-3-642-02247-0_24
BibTeX
@inproceedings{amatriain2009noise,
author = {Xavier Amatriain and Josep M. Pujol and Nuria Oliver},
title = {I Like It... I Like It Not: Evaluating User Ratings Noise in Recommender Systems},
booktitle = {International Conference on User Modeling, Adaptation, and Personalization (UMAP 2009)},
publisher = {Springer},
pages = {247--258},
year = 2009,
doi = {10.1007/978-3-642-02247-0_24},
cites = 323,
citesdate = {2026-03-23}
}
International Conference on User Modeling, Adaptation, and Personalization (UMAP 2009) · 2009
The follow-up study tested selective re-rating as a way to reduce that noise.
Repeatedly rating every item produced the largest improvement, but asking
people to reconsider a much smaller set of carefully selected ratings also
improved accuracy while requiring less additional effort. The experiments
concerned movie ratings; a production system would still have to decide when
the expected gain justifies asking a person to repeat work.
Xavier Amatriain, Josep M. Pujol, Nava Tintarev, Nuria Oliver (2009). Rate It Again: Increasing Recommendation Accuracy by User Re-Rating. Proceedings of the 3rd ACM Conference on Recommender Systems (RecSys 2009), 173-180. https://doi.org/10.1145/1639714.1639744
BibTeX
@inproceedings{amatriain2009rateit,
author = {Xavier Amatriain and Josep M. Pujol and Nava Tintarev and Nuria Oliver},
title = {Rate It Again: Increasing Recommendation Accuracy by User Re-Rating},
booktitle = {Proceedings of the 3rd ACM Conference on Recommender Systems (RecSys 2009)},
pages = {173--180},
year = 2009,
doi = {10.1145/1639714.1639744},
cites = 234,
citesdate = {2026-03-23}
}
Xavier Amatriain, Josep M. Pujol, Nava Tintarev, Nuria Oliver
Proceedings of the 3rd ACM Conference on Recommender Systems (RecSys 2009) · 2009
The same denoising approach was also described in a related patent.
Patent
Cite
Formatted citation
Xabier Amatriain, Josep M. Pujol, Nuria Oliver (2011). Denoising explicit feedback for recommender systems.
BibTeX
@patent{amatriain2011denoising,
author = {Amatriain, Xabier and Pujol, Josep M. and Oliver, Nuria},
title = {Denoising explicit feedback for recommender systems},
number = {EP2312516},
type = {patentep},
year = 2011,
note = {Application EP20090382209. Filed 2009-10-20, Published 2011-04-20}
}
Denoising explicit feedback for recommender systems
Xabier Amatriain, Josep M. Pujol, Nuria Oliver
2011
"The Wisdom of the Few": Collaborative Filtering via Expert Opinions
The second study used a relatively small set of expert opinions. Rather than
searching the entire ratings dataset for similar users, the method identifies
experts whose preferences are close to those of each user and weights their
opinions accordingly.
We evaluated the approach with the Netflix dataset and in a user study. It
produced accurate recommendations using substantially fewer ratings, and
participants preferred its recommendations. It also reduced the amount of
data and computation needed to generate a recommendation. The method depends,
however, on an independent source of expert ratings that covers the relevant
items; not every recommendation domain has one.
Xavier Amatriain, Neal Lathia, Josep M. Pujol, Haewoon Kwak, Nuria Oliver (2009). The Wisdom of the Few: A Collaborative Filtering Approach Based on Expert Opinions from the Web. Proceedings of the 32nd International ACM SIGIR Conference, 532-539. https://doi.org/10.1145/1571941.1572033
BibTeX
@inproceedings{amatriain2009wisdom,
author = {Xavier Amatriain and Neal Lathia and Josep M. Pujol and Haewoon Kwak and Nuria Oliver},
title = {The Wisdom of the Few: A Collaborative Filtering Approach Based on Expert Opinions from the Web},
booktitle = {Proceedings of the 32nd International ACM SIGIR Conference},
pages = {532--539},
year = 2009,
doi = {10.1145/1571941.1572033},
cites = 233,
citesdate = {2026-03-23}
}
Xavier Amatriain, Neal Lathia, Josep M. Pujol, Haewoon Kwak, Nuria Oliver
Proceedings of the 32nd International ACM SIGIR Conference · 2009
Adding context to recommendation
Preferences can depend on circumstances that a user-item matrix does not
represent. Multiverse Recommendation modeled users, items, and contextual
dimensions together through tensor factorization. Across the datasets used in
the paper, the context-aware model improved on the tested non-contextual and
context-aware baselines, although the benefit depended on having useful
contextual variables. Adding dimensions also increases the data and
computation needed to estimate the model.
Alexandros Karatzoglou, Xavier Amatriain, Linas Baltrunas, Nuria Oliver (2010). Multiverse Recommendation: N-Dimensional Tensor Factorization for Context-Aware Collaborative Filtering. Proceedings of the 4th ACM Conference on Recommender Systems (RecSys 2010), 79-86. https://doi.org/10.1145/1864708.1864727
BibTeX
@inproceedings{karatzoglou2010multiverse,
author = {Alexandros Karatzoglou and Xavier Amatriain and Linas Baltrunas and Nuria Oliver},
title = {Multiverse Recommendation: N-Dimensional Tensor Factorization for Context-Aware Collaborative Filtering},
booktitle = {Proceedings of the 4th ACM Conference on Recommender Systems (RecSys 2010)},
pages = {79--86},
year = 2010,
doi = {10.1145/1864708.1864727},
award = {Best Paper Award Nominee},
cites = 1105,
citesdate = {2026-03-23}
}
Later work addressed recommendation as a ranking problem rather than only
trying to predict rating values. CLiMF optimized reciprocal rank directly,
while TFMAP extended ranking-oriented learning to top-N, context-aware
recommendation by optimizing mean average precision. These evaluations used
implicit feedback, where an observed interaction is evidence of relevance but
an unobserved item is ambiguous; their results do not answer how the same
objectives behave with explicit ratings.
Yue Shi, Alexandros Karatzoglou, Linas Baltrunas, Martha Larson, Nuria Oliver, Alan Hanjalic (2012). CLiMF: Learning to Maximize Reciprocal Rank with Collaborative Less-Is-More Filtering. Proceedings of the 6th ACM Conference on Recommender Systems (RecSys 2012), 139-146. https://doi.org/10.1145/2365952.2365981
BibTeX
@inproceedings{shi2012climf,
author = {Yue Shi and Alexandros Karatzoglou and Linas Baltrunas and Martha Larson and Nuria Oliver and Alan Hanjalic},
title = {{CLiMF}: Learning to Maximize Reciprocal Rank with Collaborative Less-Is-More Filtering},
booktitle = {Proceedings of the 6th ACM Conference on Recommender Systems (RecSys 2012)},
pages = {139--146},
year = 2012,
doi = {10.1145/2365952.2365981},
award = {Best Paper Award},
cites = 446,
citesdate = {2026-03-23}
}
Yue Shi, Alexandros Karatzoglou, Linas Baltrunas, Martha Larson, Alan Hanjalic, Nuria Oliver (2012). TFMAP: Optimizing MAP for Top-N Context-Aware Recommendation. Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, 155-164. https://doi.org/10.1145/2348283.2348308
BibTeX
@inproceedings{shi2012tfmap,
author = {Yue Shi and Alexandros Karatzoglou and Linas Baltrunas and Martha Larson and Alan Hanjalic and Nuria Oliver},
title = {{TFMAP}: Optimizing {MAP} for Top-N Context-Aware Recommendation},
booktitle = {Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval},
pages = {155--164},
year = 2012,
doi = {10.1145/2348283.2348308},
cites = 268,
citesdate = {2026-03-23}
}
Yue Shi, Alexandros Karatzoglou, Linas Baltrunas, Martha Larson, Alan Hanjalic, Nuria Oliver
Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval · 2012
Recommendations in everyday mobile use
Frappe moved the question into the field, examining how people used and
perceived mobile application recommendations in everyday contexts. Its
Android-market deployment reached about 1,000 users, accompanied by a
33-person local study. Usage-based measures could look favorable even when
participants described poor recommendations, a useful warning that offline or
behavioral accuracy alone does not establish a good recommendation
experience.
Linas Baltrunas, Karen Church, Alexandros Karatzoglou, Nuria Oliver (2015). Frappe: Understanding the usage and perception of mobile app recommendations in-the-wild. https://doi.org/10.48550/arXiv.1505.03014
BibTeX
@misc{baltrunas2015frappe,
author = {Baltrunas, Linas and Church, Karen and Karatzoglou, Alexandros and Oliver, Nuria},
title = {Frappe: Understanding the usage and perception of mobile app recommendations in-the-wild},
year = 2015,
eprint = {1505.03014},
archiveprefix = {arXiv},
doi = {10.48550/arXiv.1505.03014},
cites = 145,
citesdate = {2026-03-23}
}
