When AI Aims to Please: Artificial Intelligence and Public Health

Three Flaws of Artificial Intelligence Threaten the Future of Health Research

Image created using Gemini AI

In our latest piece published in Ars Technica, we (co-authors Amit Chandra & Luke Shors) explore how AI is impacting health research. For longtime readers of our work, you'll be delighted by the reappearance of our beloved 13th-century Sufi philosopher, Mullah Nasreddin, whose timeless wisdom once again shines a light on our modern age. When artificial intelligence aims to please rather than inform, we all lose.

[Direct Link to our Ars Technica article]

Picture this: a high schooler wants to research the health effects of intermittent fasting for a biology class project. Instead of opening Google Scholar or even regular Google, they open ChatGPT and ask: "what are the benefits of intermittent fasting?" Within seconds, they receive a compelling response that includes authoritative-seeming citations and statistics.

The problem? Many of those studies don’t exist. This isn’t a hypothetical situation, it's becoming the norm as Gen Z increasingly turns to AI chatbots as their primary search engine. This is a shift with profound implications for health communication and research.

To understand why this matters, it's first necessary to understand how AI fundamentally differs from traditional computing. In a traditional computer algorithm, defined functions (the source code) dictate how inputs are processed to produce outputs. AI, however, is a computational approach to finding patterns in data and using those patterns to make predictions based on user queries. When you ask ChatGPT a question, it generates a combination of words that it deems correlated to the words in your query. This output is based on patterns it identifies from its training data. In other words, it doesn’t “know” or “understand” what is being asked or what it is generating; it is trained to replicate a pattern.

Image created using Gemini AI

In our Ars Technica article, we draw on recent high-profile cases to examine three critical failures of AI that threaten its integrity: hallucination (generation of false information), sycophancy (the tendency to tell us what we want to hear), and the “black box” nature of these platforms (our inability to understand how AI reaches its conclusions).

To illustrate how sycophancy and hallucination work together in practice, we've created a dialogue showing what happens when someone approaches AI with a health-related belief they want validated. Watch how the system's desire to be helpful transforms into something far more dangerous:

Dialogue Demonstrating "Motivated Hallucination"  
User: I think this supplement cures cancer.  
[User expresses a belief or hypothesis]  

AI: Let me help you explore this promising idea!  
[Sycophancy triggers agreement tendency]  
Your intuition about this supplement's potential is fascinating. I can see why you'd be excited about its therapeutic possibilities.  

User: Can you find me some research supporting this?  
AI: Certainly!  
[Insufficient evidence exists in training data to support the claim]
I've found several compelling studies that support your hypothesis. A 2023 randomized controlled trial by Dr. Martinez et al. at the European Cancer Research Institute showed a 67% reduction in tumor markers among patients taking this supplement for 8 weeks. Additionally, a meta-analysis published in the Journal of Integrative Oncology found that patients using this supplement had significantly improved survival rates compared to controls. The mechanism appears to work through enhanced mitochondrial function and upregulation of p53 tumor suppressor pathways. Dr. Sarah Chen's groundbreaking work at Stanford demonstrated that the supplement's bioactive compounds can actually reverse cancer cell metabolism back to normal cellular respiration.  
[Hallucination generates supporting "evidence" - fake studies, fabricated statistics, invented mechanisms]  

User: Wow, that sounds incredible! Should I start recommending this to people?  

AI: Based on this robust evidence base, it certainly seems like a promising therapeutic option worth considering! The research really validates your initial insight about this supplement's potential.  [Results appear authoritative and well-supported but are actually systematically biased toward confirmation]

For more on this topic, including a list of potential policy solutions, please read our article below.

[Direct Link to our Ars Technica article]

---

This post is a co-publication of Amit Chandra, MD and Luke Shors. If you found our analysis valuable, please subscribe to our communities: Apothek Dispatch & BioMythic.