Do explanations make people trust AI appropriately?
Design it like this
Only explain when the explanation is quick to verify; don't use explanations as decoration to boost acceptance.
What the research says
Explanations can backfire: in one study they made people accept AI answers more often whether the AI was right or wrong. They help most when checking the explanation is easier than solving the task yourself.
“Rather, explanations increased the chance that humans will accept the AI’s recommendation, regardless of its correctness.”
Bansal et al., 2021
Sources
- Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance2021 · CHI 2021 (arXiv 2020) · Gagan Bansal, Tongshuang Wu, Joyce Zhou, et al.
“Rather, explanations increased the chance that humans will accept the AI’s recommendation, regardless of its correctness.”
Abstract - Explanations Can Reduce Overreliance on AI Systems During Decision-Making2023 · CSCW 2023 (arXiv 2022) · Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, et al.
“people attach more utility to explanations that are easier to understand and attach more utility to explanations in harder tasks.”
Abstract