Apothek Grand Rounds: Unintended Consequences, Patient Safety, and Medical Ethics for Bioengineers
Guest Lecture on Bioengineering at George Mason University
A few weeks ago, I delivered a guest lecture to bioengineering students at George Mason University. I’ve delivered lectures to undergraduates in the past, and I always approach the challenge by thinking about what I wish someone had told me earlier in my career.
Considering that some of the students in the audience might one day build next generation health technologies, I shared my reflections on medical ethics, dealing with risk, and avoiding unintended consequences related to diagnosis, treatment, and care.
Here’s the short version of our conversation.
The Promise vs The Reality
We’re living through a remarkable moment in healthcare with the deployment of digital technologies in research, record keeping, diagnosis, and treatment. Moore’s Law tells us computing power doubles roughly every two years, promising a future of exponential gains in digital capability. At the same time, Eroom’s Law (Moore’s intentionally spelled backward) tells the opposite story in the pharmaceutical industry: drug discovery has gotten slower and more expensive every decade since the 1950s. The promise of digital health — mobile, wearables, AI — is that it can help fill that gap, improving population health in the absence of new medicine discoveries.
But not all innovation leads to progress. And even when it does, we often deal with unintended consequences associated with its deployment.
I shared a story from my own clinical career about the rollout of electronic health records. I went from exclusive use of paper charts during medical school, to exclusive use of electronic health records as an emergency medicine resident. The pitch for EHRs in that era was compelling: better data, improved patient safety, seamless information sharing. The reality? Hours of documentation after every shift. The barrier of a computer screen between physicians and patients. Systems optimized primarily for billing, not healing or bedside efficiency. Alarm fatigue rendering pop-up warnings meaningless. A generation of health providers burned out by thousands of clicks through unwieldy interfaces. To read more about this phenomenon, these articles explore more concrete experiences: Death by 1,000 Clicks (Schulte & Fry, KFF), and Why Doctors Hate their Computers (Gawande, The New Yorker).
The failure here lies in the gap between design intent of EHRs and the complex health systems in which the technology is deployed. I’m not calling for a return to paper-based records, just a recognition that technology can simultaneously enhance and encumber the practice of medicine.
On Unintended Consequences
One of the core ideas I wanted to share with the students was that unintended consequences are not random. They fall into recognizable categories:
First-order consequences are direct impacts that weren’t caught in development (e.g. drug off-target binding). These are the ones we’re best at finding, because they’re closest to the intervention. We design a drug to treat a disease, and then we seek to identify negative side effects through clinical trials.
Second-order consequences are impacts that ripple through social or biological systems. Consider an AI triage tool that reduces the cognitive load on nurses in a busy emergency department waiting room. What happens when the use of and reliance on this tool erodes the clinical judgement they were utilizing every day?
Emergent consequences are the hardest to predict, arising from the interaction of individual changes to the system. Consider the feedback loop as an AI platform’s outputs, riddled with hallucinations, corrupt the next generation of training data (see: AI Is Inventing Academic Papers That Don’t Exist — And They’re Being Cited in Real Journals, Rolling Stone). Along with Luke Shors, I described the challenge of AI hallucinations and sycophancy for health research in a past Substack post linked below:
For all of these categories, the classic iceberg analogy holds: we see a visible event and immediate impacts, but the larger mass of contributing factors and future consequences sit hidden below the surface.
Three Sources of Error We Underestimate
Following a discussion on the taxonomy of consequences, I talked about the different sources of error that lead to patient safety risks.
Human behavior. Engineering design in healthcare assumes rational actors. Behavioral science has proven over decades that we are anything but rational actors. Instead, we rely on mental shortcuts, we respond to social norms, we are prone to loss aversion and decision paralysis. If a device or algorithm only works when users behave rationally, it will fail in the real world.
Complex systems. Healthcare systems are highly complex. Interventions that appear obvious and simple in a design document must interact with chaotic clinical environments, fluctuating staffing levels, and perverse individual and institutional incentives when they are deployed. The work on systems thinking by Donella Meadows notes that systems are more than the sum of their parts, and their interactions often defy our intuitive understanding of them.
Bias. In health research, bias is often about what questions are asked, what gets studied, and who is left out. In AI, biased training data means biased outputs. The problem for health AI is often that the populations these tools are trained on are rarely the populations they’re deployed on. Consider the experience of IBM Watson for Oncology, which was trained using data and feedback from a leading oncology center in New York City to create a seemingly powerful tool. When deployed around the world however, it generated inappropriate and sometimes dangerous clinical guidance.
What To Do About It
I believe that the future opportunity for health technology is real. It can speed up workflows, automate rote tasks, and analyze large amounts of multimodal data. I also believe that the potential negative consequences of deploying these technologies in healthcare settings demand something different from the tech industry’s battle cry of “move quickly and break things.”
A few practical frameworks I shared with the students:
Stress-test your idea before you build it. Consider what happens if your health technology succeeds. What second-order changes might occur related to clinician or patient behaviors, and could these changes undermine the benefit you designed for? Consider what might happen if your technology is deployed in a less than ideal setting (understaffed, under-resourced, overcrowded, etc.).
Audit your algorithms, before and after deployment. Test your technology across demographic subgroups. Evaluate its performance on “edge cases.” Once deployed, monitor for unintended consequences. We need to build systems that generate predictable and reliable outputs within a known degree of confidence.
Take ethics seriously. The National Society for Professional Engineers code of ethics begins with the clause: “Hold paramount the safety, health, and welfare of the public.” The Association for Computing Machinery code of ethics highlights the dangers of data aggregation and emergent properties, and stresses the need to assess potential harms before software deployment.
Closing Thoughts
I closed with a difficult question on accountability for health technology deployment:
A physician who makes an error affects one patient. You may one day write code or design a device that affects a million. Does that change what you owe the people you’ll never meet?
In the past, a “tombstone mentality” of deploy-harm-regulate has driven progress in medicine, but it has also left wreckage along the way. Thalidomide, OxyContin, and Theranos are a few extreme examples. But subtler harms from poorly implemented health technologies accumulate every day with fewer headlines and less accountability.
Risk mitigation tools and ethical frameworks don’t work automatically. They only work if researchers and technology developers decide they matter.
Thank you for joining Apothek Dispatch. I'm excited to share this journey with you! This newsletter is free to everyone, thanks to the generous support of fellow Apothek Dispatch members. To support the effort, subscribe or upgrade below: