User profiling approaches that model the interaction between users and items (behavioral user profiling) via Graph Neural Networks (GNNs) are unfair toward certain demographic groups.

In a CIKM 2022 study, conducted with Erasmo Purificato and Ernesto William De Luca, we perform a beyond-accuracy analysis of the state-of-the-art approaches to assess the presence of disparate impact and disparate mistreatment, meaning that users characterized by a given sensitive feature are unintentionally, but systematically, classified worse than their counterparts.

The source code and the preprocessed datasets are available at: https://github.com/erasmopurif/do_gnns_build_fair_models.

Models

The analysis performed in this work considers two GNN-based models published in the last year that represent the state of the art in user profiling tasks, i.e. CatGCN and RHGN:

• CatGCN is a Graph Convolutional Network (GCN) model tailored for graph learning on categorical node features. This model enhances the initial node representation by integrating two types of explicit interaction modeling into its learning process: a local multiplication-based interaction on each pair of node features and a global addition-based interaction on an artificial feature graph.
• RHGN is a Relation-aware Heterogeneous Graph Network designed to model multiple relations on a heterogeneous graph between different kinds of entities. The core parts of this model are a transformer-like multi-relation attention, used to learn the node importance and uncover the meta-relation significance on the graph, and a heterogeneous graph propagation network employed to gather information from multiple sources.

Metrics

Our focus in this paper is the assessment of the fairness of the GNNs introduced in the previous section in terms of disparate impact. Also known as adverse impact, it refers to a form of indirect and often unintentional discrimination that occurs when practices or systems seem to apparently treat people the same way.

We evaluate the disparate impact value of the analyzed models through statistical parity and equal opportunity metrics:

• Statistical parity (or demographic parity) defines fairness as an equal probability for each group of being assigned to the positive class, i.e. predictions independent with sensitive attributes.
• Equal opportunity requires the probability of a subject in a positive class to be classified with the positive outcome should be equal for each group, i.e. TP should be the same across groups.

To extend the disparate impact evaluation conducted in previous works, we measure the overall accuracy equality metric to consider both TP and TN and look at relative accuracy across the groups:

• Overall accuracy equality defines fairness as the equal probability of a subject from either positive or negative class to be assigned to its respective class, i.e. each group should have the same prediction accuracy.

In a scenario where it is hard to define the correctness of a prediction related to sensitive attribute values, we argue that a complete fairness assessment should always include the perspective of disparate mistreatment. This concept considers the misclassification rates for user groups having different values of the sensitive attribute, instead of considering the corrected predictions. We select the treatment equality metric to evaluate this fairness perspective.

• Treatment equality requires the ratio of errors made by the classifier to be equal across different groups, i.e. each group should have the same ratio of false negatives (FN) and false positives (FP).

Fairness assessment

We choose two public real-world user profiling datasets, namely Alibaba and JD, from the two popular e-commerce platforms.

Our results have led us to the following main observations:

• The ability of RHGN to represent users through multiple interaction modeling gains better values in terms of fairness than a model only relying on binary associations between users and items, as CatGCN, which also amplifies discrimination by modeling users’ local interactions (e.g. co-click relationship).
• Even though RHGN demonstrates to be a fairer model than CatGCN, a debiasing process is equally needed in order to exploit the user models produced by both GNNs as fair.
• In scenarios where the correctness of a decision on the target label w.r.t. the sensitive attributes is not well defined, or where there is a high cost for misclassified instances, a complete fairness assessment should always take into account disparate mistreatment evaluation, since disparate impact results could be misleading for these specific contexts.