Understanding user perceptions of diversity in recommender systems reveals a paradox: while users favor intuitive, metadata-driven metrics like genres, their ability to distinguish finer variations in diversity is limited, highlighting the need for user-aligned algorithms that balance diversity with relevance.

In this study, in collaboration with Patrik Dokoupil and Ladislav Peska, and published in the proceedings of ACM UMAP ’24, we investigate user perceptions of diversity in recommender systems (RS), highlighting the complicate balance between algorithmic design and user-centric objectives.

Diversity in RS enhances user satisfaction and engagement by offering a range of recommendations that differ in genres, attributes, or user ratings. Beyond accuracy, diversity helps avoid “echo chambers” and promotes discovery. However, the success of these systems is measured not only by metrics assessing recommendations’ effectiveness but by how users perceive and value the diversity presented.

Research Questions

Our study tackled three main questions:

1. Which diversity metric aligns best with user perceptions? Users were presented with recommendations generated using three types of diversity metrics: metadata-based, content-based, and collaborative. The study revealed a strong user preference for metadata and content-based metrics, highlighting their alignment with user expectations of diversity.
2. How do users perceive differences in diversity magnitude? The research found that users can recognize significant variations in diversity levels. However, this perception diminishes when the differences between recommendations are subtle, suggesting a nonlinear relationship.
3. Which metrics best predict user evaluations of diversity? Collaborative filtering metrics, while theoretically robust, often diverge from user perceptions. Conversely, metadata-based approaches, such as those leveraging genres, resonated more closely with user evaluations.

Methodology

We conducted experiments using datasets from two domains: movies (MovieLens) and books (GoodBooks). Recommendation lists were generated using a bounded greedy optimization algorithm, incorporating various diversity metrics. Users evaluated the lists through a detailed study flow that included preference elicitation, metric selection, and assessments of diversity magnitude.

To ensure robustness, the study analyzed user feedback through drag-and-drop interfaces (an example is shown in the figure) and statistical models, capturing both absolute and relative perceptions of diversity.

Main Findings

1. Preference for metadata-based metrics. Users consistently preferred recommendations that incorporated genre-based diversity, suggesting that intuitive and relatable metrics resonate more with end-users.
2. Nonlinear perception of diversity. While users could distinguish large differences in diversity, their sensitivity to smaller variations diminished, emphasizing the need for careful calibration of diversity thresholds.
3. Interplay between diversity and relevance. Users sometimes perceived less relevant recommendations as more diverse, highlighting a potential tradeoff between these two objectives.
4. Domain-specific variations. The effectiveness of diversity metrics varied between movies and books, underscoring the importance of context in designing RS.

Implications for Recommender Systems

1. User-centric design. Developers should prioritize diversity metrics that align with user perceptions, such as metadata and content-based approaches.
2. Balancing objectives. Careful tuning is required to balance diversity with relevance, avoiding over-diversification that may compromise user satisfaction.
3. Adaptation to context. The choice of metrics should consider domain-specific nuances, as user expectations differ across content types.
4. Future directions. The study highlights the need for more research on user-aligned diversity metrics and the role of presentation formats in shaping user perceptions.

Conclusion

Diversity is a multifaceted and user-centric concept in recommender systems. By bridging the gap between algorithmic advancements and user perceptions, we can design systems that not only perform well but also meet the different expectations of their users.