A photograph can be technically clear and still fail to appeal to the person
looking at it. We examined whether computational models could represent some
of that subjective judgment and use it when organizing or searching consumer
photos and videos.
In our first search study, we combined textual relevance with an automatically
estimated aesthetic score. The 37 participants made one thing clear: visual
appeal influenced their choices, but how much it mattered shifted with the
query and with the person doing the searching. There was no universal ranking.
We then represented photographic composition through features inspired
by simplicity, visual balance, golden-mean placement, and golden triangles. It
We also tested category-specific models rather than assuming that portraits,
landscapes, and other kinds of photographs should be judged in the same way.
Video posed a different problem. From 160 consumer clips, we selected seven
visual features and reported 73% accuracy on a binary high-versus-low
aesthetic classification task.
We also used comments as data. In a 58-person evaluation of image-search
results, comment-based aesthetic predictions were stronger than visual
content alone for that particular setting.
Our models represented reported preferences and signals from particular
datasets, not an objective definition of beauty. Their hand-designed features,
categories, and comments may carry the assumptions of a platform or community
and may not transfer to other cultures, genres, or periods.
The work overlaps with MAMI in its interest in personal
photographs, but asks a different question: MAMI retrieves images through
spoken annotations, while these studies estimate visual appeal and use it in
ranking.
Pere Obrador, Xavier Anguera, Rodrigo de Oliveira, Nuria Oliver (2009). The Role of Tags and Image Aesthetics in Social Image Search. Proceedings of the 1st ACM SIGMM Workshop on Social Media, 65-72. https://doi.org/10.1145/1631144.1631158
BibTeX
@inproceedings{obrador2009tags,
author = {Pere Obrador and Xavier Anguera and Rodrigo de Oliveira and Nuria Oliver},
title = {The Role of Tags and Image Aesthetics in Social Image Search},
booktitle = {Proceedings of the 1st ACM SIGMM Workshop on Social Media},
pages = {65--72},
year = 2009,
doi = {10.1145/1631144.1631158},
cites = 38,
citesdate = {2026-03-23}
}
Anush K. Moorthy, Pere Obrador, Nuria Oliver (2010). Towards Computational Models of the Visual Aesthetic Appeal of Consumer Videos. European Conference on Computer Vision (ECCV 2010), 1-14. https://doi.org/10.1007/978-3-642-15555-0_1
BibTeX
@inproceedings{moorthy2010aesthetic,
author = {Anush K. Moorthy and Pere Obrador and Nuria Oliver},
title = {Towards Computational Models of the Visual Aesthetic Appeal of Consumer Videos},
booktitle = {European Conference on Computer Vision (ECCV 2010)},
publisher = {Springer},
pages = {1--14},
year = 2010,
doi = {10.1007/978-3-642-15555-0_1},
cites = 104,
citesdate = {2026-03-23}
}
Pere Obrador, Ludwig Schmidt-Hackenberg, Nuria Oliver (2010). The Role of Image Composition in Image Aesthetics. 2010 IEEE International Conference on Image Processing (ICIP), 3185-3188. https://doi.org/10.1109/icip.2010.5654231
BibTeX
@inproceedings{obrador2010composition,
author = {Pere Obrador and Ludwig Schmidt-Hackenberg and Nuria Oliver},
title = {The Role of Image Composition in Image Aesthetics},
booktitle = {2010 IEEE International Conference on Image Processing (ICIP)},
pages = {3185--3188},
year = 2010,
doi = {10.1109/icip.2010.5654231},
cites = 119,
citesdate = {2026-03-23}
}
Pere Obrador, Michele A. Saad, Poonam Suryanarayan, Nuria Oliver (2012). Towards Category-Based Aesthetic Models of Photographs. International Conference on Multimedia Modeling (MMM 2012), 63-76. https://doi.org/10.1007/978-3-642-27355-1_9
BibTeX
@inproceedings{obrador2012aesthetic,
author = {Pere Obrador and Michele A. Saad and Poonam Suryanarayan and Nuria Oliver},
title = {Towards Category-Based Aesthetic Models of Photographs},
booktitle = {International Conference on Multimedia Modeling (MMM 2012)},
publisher = {Springer},
pages = {63--76},
year = 2012,
doi = {10.1007/978-3-642-27355-1_9},
cites = 39,
citesdate = {2026-03-23}
}
Jose San Pedro, Tom Yeh, Nuria Oliver (2012). Leveraging User Comments for Aesthetic Aware Image Search Reranking. Proceedings of the 21st International Conference on World Wide Web (WWW 2012), 439-448. https://doi.org/10.1145/2187836.2187896
BibTeX
@inproceedings{sanpedro2012aesthetic,
author = {Jose San Pedro and Tom Yeh and Nuria Oliver},
title = {Leveraging User Comments for Aesthetic Aware Image Search Reranking},
booktitle = {Proceedings of the 21st International Conference on World Wide Web (WWW 2012)},
pages = {439--448},
year = 2012,
doi = {10.1145/2187836.2187896},
cites = 71,
citesdate = {2026-03-23}
}
Proceedings of the 21st International Conference on World Wide Web (WWW 2012) · 2012
Related patents
Our photo and video models also led to patents on assessing photographic
aesthetics and classifying videos.
Patent
Cite
Formatted citation
Pere Obrador, Mhd Saad, Poonam Suryanarayan, Nuria Oliver (2014). Method to assess the aesthetic quality of photographs.
BibTeX
@patent{obrador2014aesthetic,
author = {Obrador, Pere and Saad, Mhd and Suryanarayan, Poonam and Oliver, Nuria},
title = {Method to assess the aesthetic quality of photographs},
number = 8660342,
type = {patentus},
year = 2014,
note = {Priority 2012-01-24, Filed 2012-01-24, Granted 2014-02-25},
cites = 82
}
Method to assess the aesthetic quality of photographs
Pere Obrador, Mhd Saad, Poonam Suryanarayan, Nuria Oliver
2014
Patent
Cite
Formatted citation
Arun Moorty, Pere Obrador, Nuria Oliver (2015). Method for the classification of videos.
BibTeX
@patent{moorty2015videoclassification,
author = {Moorty, Arun and Obrador, Pere and Oliver, Nuria},
title = {Method for the classification of videos},
number = {WO/2012/001048},
type = {patentwipo},
year = 2015,
note = {Priority 2010-07-01, Filed 2011-06-29, Granted 2015-01-27},
cites = 10
}