A recommender system is only as effective as its understanding of user propensities. The SM-RS dataset links contextual impressions with self-reported preferences, enabling the development of personalized, multi-objective recommendations.

Recommender systems (RS) have long focused on delivering accurate results, aiming to align recommendations with user profiles. However, as user expectations evolve, beyond-accuracy metrics such as novelty, diversity, and fairness have gained prominence.

The SM-RS dataset, which we introduced at SIGIR ’24 with Patrik Dokoupil and Ladislav Peska, addresses this need by integrating single- and multi-objective recommendation strategies with contextual user impressions and self-reported preferences.

Traditional RS datasets lack nuanced information about user preferences for beyond-accuracy objectives like diversity or novelty. SM-RS fills this gap by:

• Introducing ground truth propensities. The dataset captures self-declared user preferences toward relevance, novelty, and diversity.
• Providing contextual impressions. SM-RD logs detailed user interactions with displayed recommendations, offering insights into decision-making processes.
• Enabling multi-objective recommendation. This dataset enables fine-grained optimization for both single- and multi-objective recommenders.

SM-RS Features

1. User behavior and contextual impressions.
   • Contains 55,000 displayed items and 11,000 user selections.
   • Tracks the impact of impressions on user decision-making.
   • Includes normalized marginal gains (NMG) for relevance, novelty, and diversity.
2. Self-reported propensities.
   • Gathers user feedback on the importance of beyond-accuracy metrics after each iteration.
   • Enables within-session analysis of propensity changes.
3. Additional components.
   • MovieLens-derived metadata enriched with visuals and descriptions.
   • JSON-based structure for user profiles, interactions, and propensity data.
   • Pre-computed relevance, novelty, and diversity metrics.

Possible use cases

The dataset supports diverse applications, including:

1. Unbiased evaluation of Multi-Objective Recommenders (MORS).
   • Reduces bias inherent in single-objective training datasets.
   • Evaluates the interplay between relevance, novelty, and diversity.
2. User behavior analysis.
   • Maps user actions to self-reported propensities, deepening understanding of decision patterns.
   • Explores the influence of contextual impressions.
3. Pareto optimization and decision-making.
   • Evaluates algorithms selecting recommendations from the Pareto frontier.
   • Assesses user satisfaction with proportionality in multi-objective trade-offs.
4. Beyond-accuracy criteria exploration.
   • Serves as a foundation to innovate on newer beyond-accuracy metrics.

Possible tasks

The dataset supports diverse tasks, including:

1. Propensity prediction.
   • Predict user preferences toward relevance, novelty, and diversity.
   • Baseline methods include locally and globally normalized NMG computations.
2. Proportional recommendation construction.
   • Create recommendations matching user-defined propensities.
   • Metrics include mean absolute error (MAE) and KL-divergence.
3. Impression-aware re-ranking.
   • Optimize single-objective recommendations by incorporating contextual impressions.
   • Evaluated using nDCG@10 and precision@5.

Future directions

SM-RS sets a new benchmark for recommender systems by combining contextual impressions with multi-objective strategies. Nevertheless, it has some limitations:

• Small user base. 227 participants may restrict scalability for certain algorithms.
• Short-term context. Limited longitudinal insights due to brief study duration.

Future research can expand SM-RS by increasing participant diversity, extending study periods, and integrating additional domains beyond movies.