Model Explainability
As machine learning models are deployed in high-stakes domains like healthcare, finance, and lending, the ability to explain why a model made a specific prediction becomes critical. Explainability builds trust, satisfies regulatory requirements, and helps debug model behavior.
Explainability Methods​
| Method | What It Does | Scope | Best For |
|---|---|---|---|
| SHAP Values | Assigns each feature a contribution to each individual prediction | Local (per prediction) | Explaining why a specific decision was made |
| Feature Importance (Global) | Overall ranking of features by contribution across all predictions | Global (entire model) | Understanding which features drive the model overall |
| Partial Dependence Plots | Shows the effect of a single feature on predictions, averaging over other features | Global | Visualizing the relationship between one feature and the target |
SHAP Values​
SHAP (SHapley Additive exPlanations) is grounded in game theory. It treats each feature as a "player" and calculates how much each player contributes to the final prediction.
import shap
# Train your model
model.fit(X_train, y_train)
# Create SHAP explainer
explainer = shap.Explainer(model, X_train)
shap_values = explainer(X_test)
# Visualize a single prediction
shap.plots.waterfall(shap_values[0])
# Global feature importance from SHAP
shap.plots.beeswarm(shap_values)
Key properties of SHAP:
- Shows which features pushed a prediction higher or lower
- Based on Shapley values from cooperative game theory
- Works with any model (model-agnostic)
- Can provide both local (per-prediction) and global (aggregate) explanations
"Explain why the model denied this specific loan application" = SHAP values. SHAP gives you a breakdown showing that, for example, high debt-to-income ratio pushed the prediction toward denial while long employment history pushed it toward approval.
Feature Importance (Global)​
Feature importance provides a high-level view of which features matter most across the entire dataset.
| Source | Method | Notes |
|---|---|---|
| XGBoost | Gini importance (gain-based) | Built-in; fast but can be biased toward high-cardinality features |
| Random Forest | Permutation importance | Measures accuracy drop when a feature is shuffled |
| Any model | SHAP-based importance | Aggregate SHAP values across all predictions — most reliable |
# XGBoost built-in feature importance
import xgboost as xgb
model = xgb.XGBClassifier()
model.fit(X_train, y_train)
xgb.plot_importance(model, importance_type='gain')
# Permutation importance (model-agnostic)
from sklearn.inspection import permutation_importance
result = permutation_importance(model, X_test, y_test, n_repeats=10)
Partial Dependence Plots​
Partial Dependence Plots (PDPs) show how the model's prediction changes as a single feature varies, while averaging over all other features.
from sklearn.inspection import PartialDependenceDisplay
# Show effect of two features
PartialDependenceDisplay.from_estimator(
model, X_train, features=['income', 'age']
)
PDPs are useful for understanding directional relationships: does increasing income increase or decrease the predicted default probability?
When to Use Each Method​
| Question | Method |
|---|---|
| "Why did the model make this specific prediction?" | SHAP values (local explanation) |
| "Which features are most important overall?" | Feature importance (global) |
| "How does changing one feature affect predictions?" | Partial Dependence Plots |
| "Is the model biased against certain groups?" | SHAP + bias metrics (e.g., via fairness tools) |
Model explainability is not just a nice-to-have. In regulated industries (finance, healthcare), you may be legally required to explain automated decisions. SHAP and feature importance are the standard tools for meeting these requirements.
LIME (Local Interpretable Model-agnostic Explanations)​
LIME is another local explanation method that works by fitting a simple interpretable model (like linear regression) around the prediction of interest.
import lime
from lime.lime_tabular import LimeTabularExplainer
explainer = LimeTabularExplainer(
X_train, feature_names=feature_names, class_names=['No', 'Yes']
)
explanation = explainer.explain_instance(X_test[0], model.predict_proba)
explanation.show_in_notebook()
SHAP vs LIME: SHAP has stronger theoretical guarantees (based on Shapley values) and is more consistent. LIME is faster for individual explanations but can be less stable across similar inputs.
Flashcards​
What are SHAP values and when should you use them?
Click to revealSHAP (SHapley Additive exPlanations) assigns each feature a contribution to an individual prediction, based on game theory. Use them when you need to explain why a specific prediction was made — e.g., why a loan was denied.
Built-in feature importance (Gini/gain-based) can be biased toward high-cardinality features or features with many possible split points. Use permutation importance or SHAP-based importance for a more reliable ranking.