Explainability
In short
Explainability is a representation of the mechanisms underlying an AI system's operation, so people can see how an output was produced. NIST's principles say the reasons should be understandable to the intended user and accurate to the system's process.
- Also called
- AI explainability, Explainable AI, XAI
- Topic
- Safety, ethics and risk
- Level
- Some background helps
- First described
- In September 2021, Phillips and colleagues at NIST published four principles for explainable AI systems: explanation, meaningfulness, explanation accuracy and knowledge limits2.
NIST's AI Risk Management Framework defines explainability as a representation of the mechanisms underlying an AI system's operation. It defines interpretability separately, as the meaning of the system's output in the context of the purpose the system was designed for. Explainability, it says, can answer how a decision was made, and the description should be matched to a person's role, knowledge and skill1.
A 2021 NIST report states four principles. The system supplies evidence or reasons for its outputs or processes. Those reasons are understandable to the intended users. They correctly reflect why the output was produced, or they reflect the system's process. The system operates only under conditions it was designed for, and only when it has enough confidence in its output. Supplying a reason, the report says, does not by itself make that reason correct or intelligible2.
The report groups techniques into models a person can read directly, such as decision trees and regression models, and post-hoc explanations made by another tool. Post-hoc examples it describes include a score for each input feature, a counterfactual that names a small input change which would flip the decision, and saliency maps on images. A post-hoc tool can query a model for outputs without seeing its internals, and one explanation may not show how the whole system works2.
The audience changes what is useful. NIST says a system builder may want an explanation for debugging a model, while policymakers and end users may need something else. Its loan example is a system whose output is approve or deny, and which may give a rationale for that decision. The report also says explanations can expose proprietary detail, and that some methods can be adjusted so unfair behaviour in a model is harder to see2.
The risk framework says explainable systems are easier to debug and monitor, and lend themselves to documentation, audit and governance. It lists explainability apart from validity, safety, security, privacy and fairness, and says covering any one of these characteristics does not make a system trustworthy1.
Example
NIST uses a loan application as an illustration. The system's output is a decision to approve or deny, and one style of explanation is to state a rationale for that decision2.
Why it matters
NIST says an explanation can help people debug, monitor and audit a system1. That help depends on the intended user understanding the reason, and on the reason matching the process that produced the output2.
Questions people ask
What is an example of explainable AI?
NIST uses a loan decision. The system approves or denies an application and can supply a rationale. Under the report's principles, that rationale should be understandable to the person it is for, and it should match the process that produced the decision2.
Is explainability the same as interpretability?
NIST uses explainability for a representation of how a system operates, and interpretability for what an output means given the system's designed purpose. It treats the two as distinct characteristics of a trustworthy system1.