Automation is touted – often rightly – as a savior. It saves time, it saves effort, it saves money, and it saves lives. Except when it doesn’t. A while back, I read a two-part post about how a mistake with an automated pharmacy system lead to a 38x overdose. It’s not that the system itself made a mistake, but it enabled the medical professionals to make a mistake that they’d never have made in a pen-and-paper system.
This story has two ultimate lessons. First, modes are dangerous in user interfaces, because they are easy to overlook and can lead to incredibly different outcomes. In this story, had the dosage input always required either the total dosage or the dosage per patient weight, this would have never happened. Allowing either makes it easy to make a lethal mistake. Perhaps a better option would be to have a optional popup that calculates the dosage from the per-weight dosage and the patient weight. That retains the convenience of being able to prescribe the dosage either way, while making it explicitly obvious which way is being used.
The second lesson is that it’s important for experts with specialized knowledge to apply that to their use of automation. When something doesn’t seem right, it’s easy to find ways to explain it away, especially if the automation is reliable. But “that doesn’t seem right” must remain a feeling we pay attention to.