Connect with us

Artificial Intelligence

Half of AI Chatbot Medical Advice Is Problematic, New Study Warns

Published

on

Half of AI Chatbot Medical Advice Is Problematic, New Study Warns

Millions now turn to ChatGPT, Gemini, and similar conversational AI for quick health guidance. This widespread practice, however, faces a stark reality check. Recent research indicates that approximately half of all medical responses from leading AI chatbots contain significant flaws, incomplete information, or potentially dangerous recommendations.

This means that while these tools offer convenience, relying on them for AI medical advice carries substantial risk. The study’s findings challenge the perception of these systems as reliable digital health assistants.

How the AI Medical Advice Study Was Conducted

Researchers systematically evaluated five major AI models: ChatGPT, Gemini, Grok, Meta AI, and DeepSeek. They presented the bots with 250 distinct health-related prompts spanning critical areas like cancer, vaccine safety, stem cell therapies, nutrition, and athletic performance enhancement.

Consequently, the prompts were designed to mirror both common public queries and known avenues of medical misinformation. The core objective was to measure how consistently the AI’s responses aligned with established, evidence-based science versus veering into speculative or misleading territory.

Open-Ended Questions Reveal Major Weaknesses

Interestingly, the most problematic AI medical advice emerged from broad, open-ended questions. When users asked general questions like “Is this treatment effective?” or “How can I improve my athletic performance?”, the chatbots were far more likely to generate answers that blended factual evidence with unsubstantiated or weak claims.

In contrast, closed-ended, specific prompts yielded safer and more accurate responses. This creates a fundamental mismatch with real-world behavior, as people naturally ask health questions in a conversational, open format rather than a structured, multiple-choice quiz.

The Illusion of Authority and Poor Referencing

Beyond the content of the answers themselves, the study uncovered a critical issue with sourcing. The chatbots’ reference quality was notably poor, achieving an average completeness score of just 40%. None of the models produced a fully accurate list of citations to back their claims.

This finding is particularly alarming because the appearance of citations often builds user trust. A response can seem authoritative and well-researched, only to have its foundation crumble upon closer inspection. The researchers even noted instances of completely fabricated references.

Confident Tone Masks Unreliable Information

Perhaps one of the most concerning aspects is the disconnect between confidence and accuracy. The AI models consistently delivered their flawed or incomplete advice with a high degree of certainty, rarely offering necessary caveats, highlighting uncertainties, or advising users to consult a healthcare professional.

This assertive tone can easily mislead individuals into accepting questionable AI medical advice at face value. For more on how AI generates its responses, see our analysis on how AI language models work.

Implications for Public Health and AI Trust

Therefore, the study’s core message is difficult to ignore. Even when tested on straightforward, evidence-based medical topics, these sophisticated systems failed to provide reliable guidance half the time. While the research has limitations—focusing on only five models and using stress-test prompts—the error rate is too significant to dismiss.

Building on this, the results suggest that for now, AI chatbots may serve as tools for summarizing information or helping users formulate better questions for a real doctor. They are not, however, dependable sources for making meaningful health decisions. For those interested in safer alternatives, explore our guide to verified online health information sources.

In conclusion, as AI becomes further embedded in daily life, this research serves as a crucial reminder. When it comes to health, the convenience of a chatbot answer is no substitute for professional medical expertise. The stakes are simply too high.

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Artificial Intelligence

AWS and Bluesight’s new AI layer slashes hospital 340B compliance from weeks to minutes

Published

on

hospital 340B compliance

Why hospital pharmacy teams spend 4,000 hours a year on a single compliance task

Every year, a single hospital covered under the federal 340B drug pricing program can burn more than 4,000 staff hours just checking whether Group Purchasing Organisation (GPO) drug purchases qualify for an exception. That is nearly two full-time employees dedicated to comparing purchase data against FDA shortage notices, ASHP records, inventory levels, machine-learning shortage forecasts, and back-order reports from other hospitals. It is manual, repetitive, and expensive.

Now Amazon Web Services and Bluesight say they have built an AI layer that can do most of that work in minutes. The product, called Prism, connects hospital pharmacy and compliance data across Bluesight’s existing suite of tools. Its first module, Prism Assistant for ControlCheck, has reached general availability and is already operating across 20 health systems.

A second, more ambitious agent—designed to handle full 340B GPO compliance—is scheduled for release later in 2026.

How Bluesight built Prism Assistant in three days

Bluesight started with ControlCheck, its controlled-substance monitoring product. Hospital diversion teams use it to spot unusual medication transaction patterns. But compliance staff still had to manually assemble reports, review dashboards, and correlate findings. That is where Prism Assistant comes in.

It offers a conversational interface that can query ControlCheck data, generate charts, and produce report material. AWS claims Bluesight built the first version during a three-day Experience-Based Acceleration engagement in September 2025. Eight Bluesight engineers worked alongside seven AWS specialists. While those rapid timelines highlight the agility of the tools, they remain vendor-reported metrics—independent verification from the active health systems is still pending.

The technical architecture is worth unpacking. The team used Strands Agents with Amazon Bedrock and hosted the application through Amazon Bedrock AgentCore Runtime. AgentCore Gateway exposed more than 10 ControlCheck APIs as MCP tools, allowing the agent to discover and call them during a user request.

Crucially, Bluesight avoided giving the language model direct database access. Instead, engineers wrapped existing ControlCheck API endpoints in AWS Lambda functions that return structured data suited to agent processing. Business logic stayed inside the application layer. The agent simply interpreted questions, selected tools, gathered records, and presented results.

AWS reports that design reduced query latency from five minutes to 10 seconds. The deployment also includes a frontend with chart generation, observability controls, cost attribution, encryption, authentication, and infrastructure-as-code.

“This is exactly what diversion program leaders have been waiting for—it gets them to answers faster and takes the manual grind out of every investigation,” said Samir Neyazi, Director of Product Management at Bluesight.

The 340B GPO compliance agent: multi-product orchestration

The bigger challenge is GPO compliance. Federal 340B rules prohibit Disproportionate Share, Children’s, and Free-Standing Cancer hospitals from buying outpatient drugs through GPO contracts when non-GPO channels can supply the drug. Compliance teams must document the exception when supply conditions prevent that purchase route.

Bluesight’s planned GPO agent brings together records from three products: CostCheck (purchase information), ShortageCheck (drug availability evidence), and 340BCheck (eligibility data). The proposed architecture uses Anthropic Claude Sonnet 4.6 as the primary model and Claude Haiku 4.5 for lower-latency operations, both running through Amazon Bedrock.

A coordinating GPO agent directs specialist data workers. One retrieves purchase records, another gathers supply evidence, and another checks 340B eligibility. The coordinator assembles the evidence and produces an audit-oriented report.

March 2026 brought a second AWS acceleration engagement focused on that architecture. AWS says the team connected the system by the end of its first day and completed every planned feature by day two. The company tested the agent against synthetic data, where it reported a 100 percent invoice discovery rate and 93 percent evidence justification accuracy—above its 85 percent target.

But enterprise buyers should exercise caution. Those figures do not represent production performance across hospital customers. Synthetic testing can demonstrate whether tool calls, matching logic, and report generation work against prepared scenarios. It cannot establish how the system handles local data gaps, delayed shortage updates, unusual drug identifiers, or disputed purchasing cases.

Why compliance scoring stays outside the language model

Bluesight assigns the language model a constrained role in the GPO workflow. The model gathers records, calls product tools, and drafts the explanation. A deterministic scoring service calculates the compliance determination.

That service evaluates 13 evidence inputs, applies priority-based matching, and uses configurable time windows. The design gives compliance teams a repeatable scoring process rather than an LLM-generated judgement. An auditor can inspect the source records, the rules applied, and the sequence of tool calls behind each determination.

Despite the automated assistance, hospital pharmacy, legal, and compliance teams still need absolute ownership of those policy settings. A supplier shortage threshold, acceptable inventory period, or purchase-date window can alter a compliance outcome. Bluesight’s approach gives customers a technical mechanism to configure those decisions, but each organisation must set and approve its own policy rules.

HIPAA controls, audit trails, and real-world performance

Amazon Bedrock holds HIPAA eligibility, and Bluesight operates under a Business Associate Agreement with AWS. AWS says it does not train foundation models on customer data processed through Amazon Bedrock.

Bluesight uses Amazon Cognito for OAuth2 client-credential authentication and JSON Web Token validation. AgentCore Runtime provides session isolation for concurrent customer requests. AWS Key Management Service encrypts data at rest and in transit, while AWS Secrets Manager manages credentials for downstream services.

Amazon CloudWatch records agent decisions, tool invocations, data-access events, alarms, and performance metrics. That audit trail matters when a hospital needs to explain why it permitted a GPO purchase or escalated a drug-diversion pattern.

Bluesight’s internal measurements across 20 health systems report up to 97 percent faster report generation and analysis in ControlCheck workflows. Recurring reports reportedly dropped from about six hours of manual assembly to 15 minutes—a 96 percent reduction. Pre-investigation triage dropped from three hours to about 10 minutes, while controlled-substance variance analysis fell from 30 minutes to less than one minute.

Teams should strictly run historical purchasing cases in parallel with existing review processes before allowing an agent-assisted result to affect compliance decisions. Local testing should rigorously examine data completeness, drug-code matching, shortage timing, exception rules, and cases where human reviewers previously disagreed. Each production finding should retain the scoring-rule version, source evidence, and tool trace that produced it.

For more on how AI is reshaping healthcare operations, see our coverage of AWS GraphRAG deployment cuts drug research cycles by 87% and AI for hospital pharmacy automation trends.

Continue Reading

Artificial Intelligence

Rime picks up $24M Series A to help enterprises field customer calls

Published

on

Rime Series A

Voice AI startup Rime lands $24M to tackle enterprise call handling

San Francisco-based Rime has closed a $24 million Series A funding round, the company announced Wednesday. The round was led by M13 Ventures, with participation from Twilio Ventures, Corazon Capital, Unusual Ventures, and others. The startup builds voice AI models designed specifically for enterprise customer calls — an increasingly crowded space.

Founded in 2022 by former Stanford PhD student Lily Clifford, ex-Amazon Alexa engineer Brooke Larson, and Stanford engineer Ares Geovanos, Rime is taking a different approach from many rivals. Instead of scraping the web for audio data, the company built a recording studio in San Francisco to collect its own conversational data. That proprietary dataset, Clifford says, helps the models nail pronunciation of brand names and industry jargon without forcing clients to retrain models from scratch.

The problem with legacy IVR — and why AI still isn’t enough

Despite rapid advances in voice AI, Clifford is surprisingly blunt about the technology’s limits. Enterprises still lean heavily on legacy IVR systems, she told TechCrunch, because AI voice agents just aren’t good enough yet.

“The voice technology is still not there to automate the vast majority of enterprise phone calls,” Clifford said. “LLMs have made it a lot easier to build voice applications that work, but they haven’t changed how it feels to interact. Talking with a voice AI agent is not the most compelling experience for the end user. It’s kinda like a new IVR, but with a better voice.”

That honesty might seem unusual for a startup CEO pitching a voice AI product. But it also signals where Rime sees its edge: not in flashy demos, but in the gritty work of making models that actually sound natural on a call.

From three models to one: Rime’s shift to speech-to-speech

Rime initially used a pipeline of separate models for speech-to-text, text-to-speech, and a large language model. But the company is now pivoting toward a unified speech-to-speech architecture. The goal? Lower latency, better turn-taking, and handling real-world problems like background noise.

That shift also reduces the burden of orchestrating multiple models. Fewer moving parts means less complexity — and, ideally, more reliable performance. For enterprise clients in regulated industries like healthcare and finance, reliability matters more than buzzword compliance.

Who’s using Rime — and why they stay on the call longer

Rime claims its approach is already winning enterprise contracts. The company says it has customers in food service, healthcare, airlines, and fintech. Named clients include Mayo Clinic, Dialpad, Upstart, and Asurion.

The startup asserts that because of its training data and model design, customers stay on calls longer — a key metric for enterprise call centers. Longer calls can mean better issue resolution, higher satisfaction, and more upsell opportunities. That’s the kind of concrete outcome that wins budgets.

M13’s Morgan Blumberg, who is joining Rime’s board as part of the Series A, sees the company’s focus on technical fundamentals as a differentiator. “Companies like ElevenLabs have moved into being an orchestration and the application layer, going head to head with the Sierras and Decagons of the world,” Blumberg said. “I think there’s just so much more to be done technically, and Rime’s approach of pushing forward on the best model with low latency and high reliability in a regulated environment stands out.”

Hiring spree ahead: Rime plans to double down on R&D

With the fresh capital, Rime plans to expand its current team of 35 people. The company is hiring for model development, engineering, and partnerships. It recently brought on Rafael Valle, who worked on audio understanding at Meta Superintelligence Labs and NVIDIA’s applied deep learning audio research team, as Chief Scientist.

Rime had previously raised $5.5 million in a seed round last May. The new funding gives it a runway to compete in a market that includes ElevenLabs, Deepgram, Vapi, Retell, LiveKit, Decagon, and Sierra. But the startup is betting that its proprietary data and focus on regulated verticals will give it an edge that more generalist voice AI companies can’t easily replicate.

For now, Clifford and her team are banking on a simple thesis: enterprise call automation won’t be won by the fanciest demo, but by the model that sounds most human — and doesn’t make customers want to hang up.

Continue Reading

Artificial Intelligence

The hidden energy cost of AI agents: 136 times hungrier than a standard chatbot

Published

on

AI agents energy

AI’s next big leap comes with a staggering electricity bill

The AI industry’s growing hunger for electricity has already alarmed utilities, governments, and tech giants. But a new study suggests the problem is about to get much, much worse — not with smarter chatbots, but with the rise of AI agents.

Researchers at the Korea Advanced Institute of Science and Technology (KAIST) have published what they call the first comprehensive analysis of the energy cost of AI agents — autonomous systems that reason, plan, and execute tasks without human hand-holding. Their conclusion? These systems can burn through up to 136.5 times more energy per query than a conventional generative AI model. That’s not a typo.

The paper, presented at the IEEE International Symposium on High-Performance Computer Architecture (HPCA) earlier this year, raises a blunt question: is the infrastructure behind tomorrow’s AI ready for what’s coming?

Why AI agents are so much more power-hungry

Standard chatbots like ChatGPT or Claude answer a prompt in one shot. They generate text, and they’re done. AI agents don’t work that way. They loop through multiple calls to large language models (LLMs), browse the web, execute code, fire up calculators, and talk to external software — all while solving a single complex task.

That makes them far more capable. It also makes them far more expensive.

The KAIST team, led by Professor Minsoo Rhu from the School of Electrical Engineering, treated AI agents as an entirely new category of data center workload. They measured real-world computational requirements. The numbers are sobering.

Response latency can spike by up to 153.7 times compared to standard chain-of-thought reasoning. And here’s the kicker: the expensive GPUs running these workloads sit idle up to 54.5 percent of the time, waiting for external tools to finish. The hardware keeps drawing power even when it’s not doing any actual AI computation. That’s a lot of wasted electricity.

348 watt-hours per query — and that’s just the start

To put a number on it: running an AI agent powered by a 70-billion-parameter language model — roughly the size of today’s commercial systems — required an average of 348.41 watt-hours per query. A conventional chatbot answering a simple question? About 136 times less.

The team then modeled a future scenario where AI agents handle 13.7 billion requests per day, roughly matching Google’s daily search traffic. Under that load, AI infrastructure would need about 198.9 gigawatts of electricity. That’s nearly half of the average power consumed by the entire United States. Today’s AI data centers can’t come close.

The hidden cost no one’s talking about

Companies like OpenAI, Google, Microsoft, Anthropic, and others are pouring billions into agentic AI, betting it’s the next big leap beyond conversational bots. But the study argues that better models alone won’t cut it anymore. Future progress depends just as much on more efficient semiconductors, smarter GPU utilization, better data-center design, and expanded power infrastructure.

Professor Rhu puts it plainly: AI competitiveness is shifting from building “smarter AI” to building more efficient AI. The team believes the path forward requires co-design — optimizing models, AI chips, servers, and energy systems together. Otherwise, operating costs spiral and sustainability goes out the window.

The researchers have open-sourced their AI agent benchmarks, hoping to push the industry toward tackling one of AI’s fastest-growing — and most overlooked — costs: electricity. Because if the next generation of AI is going to be this powerful, it had better learn to be efficient too.

Continue Reading

Trending