It is 3 AM, and I am typing this from a plastic chair in a clinic hallway. The air tastes like antiseptic and fear. Behind a closed door, my 24-year-old friend is talking to an oncologist. She is a breast cancer patient, and two hours ago, she was vomiting in my car.
The cause wasn’t the cancer. It was a decision she made after consulting ChatGPT for health advice.
One Pill, Two Pills, or Both?
Let me walk you through the absurdity. My friend felt dizzy after work. She had gut pain. Instead of calling her doctor, she snapped a photo of her painkiller strip and uploaded it to ChatGPT. Her question, roughly translated: Can I take two of these?
Here is where things get ugly. She is not a native Hindi speaker. Her phrasing was mangled. The AI interpreted her query as asking about taking two different medicines simultaneously—not two pills of the same drug.
To its credit, ChatGPT warned against mixing the medications. It flagged a potential Diclofenac overdose, stomach ulcers, and kidney damage. But my friend, believing she was asking about her standard prescription, took both pills anyway. Within an hour, she was burning from the inside out.
A Failure on Multiple Fronts
Who is at fault here? Honestly, everyone.
ChatGPT’s language problem: The tool hallucinated a second drug name—Fenak Plus—from a previous conversation. It conflated past context with the current photo, which only showed one strip of Cipzen-D.
The translation trap: A single mistranslated word turned a simple question into a dangerous one. This isn’t a niche issue. The Economist and MIT’s Center for Constructive Communication have both published research showing that AI chatbot accuracy plummets when users interact in non-English languages. The SAHARA benchmark, released in December, confirmed the same bias.
My own failure: I am a technology reporter. I have written dozens of articles about AI risks. I never once had a direct conversation with my friend about how these tools actually work. I assumed she knew. She assumed it was a magic oracle.
Why People Trust the Bot
Here is the uncomfortable truth: ChatGPT is easier. My friend put it bluntly when I asked why she didn’t just Google it.
“I don’t even know what’s real on the internet. ChatGPT just gives me the answer.”
She is not stupid. She is exhausted. She dropped out of school to support her family. She is fighting cancer. She does not have the energy to cross-reference five blue links on a search results page. The AI gives her a clean, conversational answer in seconds. That convenience is a trap.
I watched a prime-time news debate recently where a panelist told his opponent to “just ask ChatGPT” to settle a factual dispute. That moment terrified me. We are treating a probabilistic text generator as the modern encyclopedia. And unlike an encyclopedia, it will confidently tell you to do something that could kill you.
The Warning Signs Are Buried
OpenAI does include disclaimers. ChatGPT told my friend to consult a doctor. But that warning appeared in the third paragraph, after the answer she was looking for. Most users stop reading the moment they get their response.
This needs to change. The warnings should be a banner at the top of the screen, not a footnote. AI companies love to boast about solving complex math or assisting with medical breakthroughs. But those use cases happen under expert supervision. For the average user, the priority should be blunt, unmissable safety warnings.
What I Did Next
After the doctor administered a pain injection and prescribed safer medication, I took my friend’s phone. I deleted the ChatGPT app. I made her promise to call me or her doctor before touching any medication. I also gave her a list of legitimate helplines.
This is not a solution. It is a band-aid on a bullet wound.
The larger issue is that hundreds of millions of people in India and other non-English markets are using these tools daily without understanding the risks. Digital literacy is not keeping pace with AI adoption. The result is exactly what happened tonight: a cancer patient taking an unverified dose of painkillers because a chatbot gave her ambiguous advice.
I am not calling for a ban on AI chatbots. They are remarkable tools. But they are not doctors, and they are especially unreliable when language barriers are involved.
If you are reading this and you use ChatGPT for health advice, stop. Call your doctor. Visit a clinic. The three minutes it takes to get a human answer are worth it. Trust me on this.
For more on the risks of AI in everyday life, read about AI chatbot dangers for kids and how to verify AI-generated information.