Interpretability
In short
Interpretability is the meaning of an AI system's output in the context of the purpose the system was designed for. NIST separates it from explainability, a representation of the mechanisms underlying the system's operation.
- Also called
- AI interpretability, Model interpretability, Interpretable AI
- Topic
- Safety, ethics and risk
- Level
- Some background helps
- First described
- In 2016, Zachary Lipton argued that machine-learning papers used interpretability for several different ideas, including transparency and post-hoc explanation, and that the term had no formal technical meaning2.
NIST's AI Risk Management Framework uses interpretability for the meaning of a system's output in the context of its designed purpose, and explainability for a representation of the mechanisms underlying its operation. Interpretability, it says, can answer why a decision was made and what that decision means to the user. It says the risk from weak interpretability can be addressed by describing why the system made a particular prediction or recommendation1.
Outside that framework the word is less settled. Lipton's 2016 review found that machine-learning papers used interpretability for several different ideas, and that the term had no formal technical meaning. He contrasts transparency, a person following how a model works, with post-hoc explanations that add a text, a picture or an example after a prediction without showing the mechanism2.
Lipton argues that simple models are not automatically interpretable. A decision tree or a linear model can be too large for a person to step through, and a model whose inputs are anonymous engineered features is hard to take apart. He writes that a compact neural network can be more transparent than a deep decision tree. The post-hoc methods he discusses include written explanations, visualisations of what a network responds to, and explanations that point to similar examples2.
He uses a pneumonia model to show why the meaning of an output can matter. The model assigned a lower risk of death to patients who also had asthma, because those patients had received more aggressive treatment. If it were used for triage, those patients would then receive less aggressive treatment, and the prediction would stop holding2. NIST lists validity, safety, security, privacy, fairness and interpretability as separate characteristics, and says handling them one at a time does not make a system trustworthy1.
Example
Lipton describes a model that predicted a lower risk of death from pneumonia when the patient also had asthma, because those patients had been treated more aggressively. If the score were used for triage, those patients would receive less aggressive treatment, and the prediction would no longer hold2.
Why it matters
Lipton argues that people ask what an output means when a score does not say whether the pattern will survive once the model starts to change what happens2. NIST treats that question as separate from showing that the system is accurate or fair1.
Questions people ask
What is the difference between interpretability and explainability?
NIST uses interpretability for the meaning of an output in the context of the system's purpose, and explainability for a representation of how the system operates1. Lipton found that research papers already used interpretability for several ideas, including transparency and explanations added after a prediction, and that they often left the word undefined2.
Are simple models always interpretable?
No. Lipton writes that linear models, rule lists and decision trees are not interpretable just because of their type. A large tree can be harder to follow than a compact neural network, and a model is hard to interpret when its input features have no plain meaning2.
Can interpretability make an AI model safe?
No. NIST lists safety, validity, security, privacy and fairness separately from interpretability. It says that dealing with trustworthiness characteristics one at a time does not make a system trustworthy1.