Daniele Moltisanti
Data Scientist Manager @ Sky

29 November 2024

14m

Why Conditional Data Permutations Are Essential for Accurate XAI Analysis

Introduction

When explaining the decision-making process of AI models, eXplainable AI (XAI) methods like variable importance, partial dependence plots (PDP), and individual conditional expectation (ICE) plots are widely used. However, applying these tools to non-linear, black-box models trained on correlated features can lead to misleading insights.

This article explores why traditional XAI methods fall short in such cases, highlights the problem of correlation breakdown, and presents conditional data permutations as an effective solution.

The Background: XAI and Correlated Features

The Role of XAI Methods

XAI methods aim to:

• Quantify feature importance for predictive models.
• Visualize relationships between input variables and outputs.

Common tools like variable importance and PDPs rely on univariate permutations, where one feature is varied across the entire dataset to evaluate its influence on the model.

The Problem of Correlation

In datasets with correlated features (e.g., socio-economic variables or genetic traits), univariate permutations disrupt the correlation structure. This leads to:

1. Overstated Importance: Correlated features appear more influential than they truly are.
2. Neglect of Uncorrelated Factors: Important but uncorrelated features are undervalued.

Example: Imagine two correlated features, X1 (income) and X2 (education level). If X1 is permuted without considering its relationship to X2, the resulting analysis misrepresents their combined effect.

The Solution: Conditional Data Permutations

Conditional data permutations address this issue by maintaining the correlation structure of features during analysis. Here’s how it works:

1. Conditional Permutations: Instead of shuffling a feature across the entire dataset, it is permuted conditionally based on related features.

2. Maintaining Correlation: This ensures that the relationships between variables remain intact, leading to unbiased and accurate measures of feature importance.

3. Implementation with Trees: For tree-based models like Random Forests:

   • Permutations occur within the final nodes of the tree.
   • This approach respects the feature splits learned during training.

Practical Applications of Conditional Permutations

Variable Importance

Traditional variable importance scores overstate the role of correlated features. Conditional permutations provide unbiased measures, ensuring fair evaluation of all features.

Partial Dependence and ICE Plots

Conditional methods for PDPs and ICE plots prevent extrapolation errors by limiting the evaluation to the observed feature space. This improves interpretability and reliability.

Example Scenario: Using Conditional Permutations in Random Forests

Scenario

You’re analyzing a medical dataset to determine which factors influence the risk of a disease. Two features, X1 (blood pressure) and X2 (heart rate), are highly correlated.

Traditional Approach

Using univariate permutations, the model overemphasizes X1 and underrepresents X2. This skews the interpretation, leading to potentially harmful medical recommendations.

Conditional Permutations Approach

1. Within Node Permutations: Both X1 and X2 are permuted only within their respective leaf nodes.
2. Preserved Correlation: The dependency between blood pressure and heart rate is maintained.
3. Accurate Results: The variable importance scores reflect the true influence of both features.

Outcome

This approach ensures that the model insights are reliable and actionable, preventing misinterpretation that could harm patient care.

Supporting Evidence: Key Research Papers

Conditional data permutations are supported by groundbreaking research:

1. Hooker et al. (2021):

   • Highlights the risks of traditional permutations in overemphasizing correlated features.
   • Advocates for additional models or methods like conditional permutations to ensure unbiased results.
   • Read the full paper

2. Strobl et al. (2008):

   • Proposes conditional variable importance for Random Forests, demonstrating its effectiveness in preserving correlation structures.
   • Read the full paper

Conclusion

XAI methods like variable importance, PDPs, and ICE plots are invaluable for interpreting machine learning models. However, when applied to datasets with correlated features, traditional approaches can mislead. Conditional data permutations offer a robust solution, preserving feature relationships and providing unbiased insights.

By adopting conditional techniques, practitioners can achieve more accurate and reliable interpretations, unlocking the full potential of XAI.