Recommender systems by borattoludovico

KGGLM: A Generative Language Model for Generalizable Knowledge Graph Representation Learning in Recommendation

Current recommender systems struggle to unify knowledge representation across tasks, leading to inefficiencies and reduced interpretability. KGGLM addresses this by leveraging generative language models for generalizable and task-adaptive knowledge graph learning, achieving state-of-the-art performance in both knowledge completion and recommendation.

Recommender systems are central in personalizing user experiences across domains, from e-commerce to entertainment. A growing trend is leveraging knowledge graphs (KGs) to enhance the utility, explainability, and accuracy of recommendations. However, existing approaches for knowledge graph-based recommendation face critical challenges, including inefficiencies in representation learning and the need for task-specific model tuning.

In a paper, in collaboration with Giacomo Balloccu, Gianni Fenu, Mirko Marras, and Alessandro Soccol, and published in the proceedings of ACM RecSys ’24, we introduce KGGLM (Knowledge Graph Generative Language Model), a Transformer-based model designed to unify and optimize knowledge representation learning across multiple steps of the recommendation pipeline. KGGLM employs a generative framework to address limitations in existing systems, particularly in knowledge completion and path reasoning.

Knowledge Graphs and the need for generalizable representation

A knowledge graph organizes data into entities and their relationships, represented as triples. In recommender systems, KGs enable richer contextual understanding by integrating user-product interactions with auxiliary information.

Current KG-based recommendation methods often:

  • Use different representation learning methods for knowledge completion (predicting missing links) and path reasoning (discovering multi-hop relations).
  • Suffer from inefficiencies due to task-specific training and storage requirements.
  • Lack unified representations, leading to reduced interpretability.

KGGLM tackles these challenges by proposing a generative language model approach that learns generalizable representations applicable across tasks.

KGGLM: A Generative Language Model for KGs

Methodology Overview

KGGLM leverages the decoder-only Transformer architecture to build a generalizable framework for KG representation learning. The methodology consists of:

  1. Path sampling
    • A random walk algorithm extracts paths (sequences of entities and relations) from the KG.
    • Paths are tokenized and prepared as sequences for the model.
  2. Tokenization and embedding
    • Paths are tokenized into sequences, mapped to integers, and embedded into continuous vector spaces.
    • Embedding incorporates semantic (token), category, and positional information.
  3. Model training
    • KGGLM is pre-trained on generic paths to learn foundational representations.
    • Fine-tuning specializes these representations for tasks like knowledge completion (single-hop paths) and path reasoning (multi-hop paths).
  4. Inference
    • The model predicts missing entities or relations by generating sequences that extend a given prefix.
    • Diversity is ensured by excluding previously seen or decoded paths.

Properties

  • KGGLM uses causal masking to ensure auto-regressive predictions.
  • Fine-tuning involves domain-specific paths, making the model adaptable to varied recommendation scenarios.
  • The loss function combines log-likelihood maximization across training sequences.

Experimental Validation

Datasets

The experiments were conducted on two public datasets:

  • ML1M: Dense with higher average product degrees.
  • LFM1M: Larger and sparser, with more interactions and entities.

Metrics

Performance was evaluated using:

  • Mean Reciprocal Rank (MRR): Average rank of the first correct prediction.
  • Hits@1: Proportion of cases where the target entity appears first.
  • NDCG@10: Quality of the recommended list based on ranking positions.

Results

Knowledge Completion

  • KGGLM significantly outperformed 22 baselines on the ML1M dataset, achieving the highest MRR and Hits@1.
  • On LFM1M, domain-specific baselines like TorusE performed better, highlighting the need for dataset-specific fine-tuning.

Recommendation

  • KGGLM consistently surpassed all baselines, achieving a 31% increase in MRR and a 41% increase in NDCG@10 on ML1M compared to the best baseline.
  • On LFM1M, KGGLM demonstrated an even greater margin of improvement, emphasizing its robustness.

Conclusion and future directions

KGGLM bridges the gap between knowledge completion and path reasoning in recommender systems by introducing a generalizable and unified framework.

As future work, we propose exploring larger model architectures (e.g., GPT-3). Moreover, we would like to consider diverse and larger path samples to improve robustness. We would also like to validate the generalizability of our framework across domains and datasets.

For further details, you can access KGGLM’s source code.