AI Models Grow More Confident Delivering Wrong Answers
Key insights
- AI models now hallucinate less frequently but present wrong answers with greater confidence, making errors significantly harder for users to detect.
- A Yale study found AI medical scribes produced authoritative-looking clinical notes that omitted critical details needed for accurate patient care.
- UC Berkeley's Dan Klein classifies current AI systems as 'plausibility engines' optimized for convincing output rather than factual accuracy.
Why this matters
The shift from frequent, obvious hallucinations to rare, confident errors changes the failure mode from detectable to invisible, which breaks the verification workflows that enterprises and regulators are currently building around AI output. Medical AI systems generating authoritative but incomplete clinical notes represent a concrete patient safety risk at scale, and the Yale findings signal that high-stakes deployment is already outpacing error-detection infrastructure. For technical leaders, accuracy benchmarks measuring hallucination frequency are now a lagging indicator; the relevant metric is calibration: whether model confidence actually tracks correctness.
Summary
AI systems are getting fewer things wrong, but more confidently wrong, making errors harder to catch.
UC Berkeley professor Dan Klein calls these models 'plausibility engines': optimized to sound right, not be right. A Yale study on AI medical scribes found notes that appeared authoritative but omitted critical clinical details, raising stakes across research, legal, and medical settings.
Essentially: (UC Berkeley, Yale) research shows the problem has shifted from obvious errors to invisible ones that pass a credibility test.
- Yale AI medical notes appeared authoritative while omitting details that affect patient care decisions.
- Confident wrong answers are harder to catch than hallucinations, raising liability exposure in high-stakes deployments.
The risk isn't AI that seems broken; it's AI that seems fine.
Potential risks and opportunities
Risks
- Hospitals deploying AI medical scribes (Nuance DAX, Suki, Abridge) face malpractice exposure if AI-omitted clinical details contribute to adverse patient outcomes in 2026
- Legal AI tools (Harvey, Casetext) face bar association scrutiny and potential sanctions if practitioners file briefs containing confidently wrong AI-generated citations
- Enterprise AI deployments in finance and pharma could face regulatory audits if AI outputs used in compliance filings contain confidence-calibration gaps that surface post-submission
Opportunities
- AI output grounding and verification tools (Vectara, Cohere Grounded Generation, GroundTruth AI) gain budget priority as enterprises add accuracy layers above base models
- Clinical AI audit vendors and healthcare IT compliance firms gain leverage to sell post-deployment monitoring contracts to hospital systems already running AI scribes
- Model calibration startups focused on uncertainty quantification have a clear commercial pitch as the gap between expressed confidence and actual accuracy becomes a recognized liability
What we don't know yet
- Which specific AI medical scribe products were evaluated in the Yale study, and whether findings have been shared directly with those vendors
- Whether major AI labs (OpenAI, Google DeepMind, Anthropic) publish internal confidence-calibration metrics and what those scores show as of early 2026
- Whether the FDA or CMS have issued updated guidance on AI-generated clinical documentation standards in response to findings like the Yale study
Originally reported by axios.com
Read the original article →Original headline: AI Is Still Getting Things Wrong, More Confidently Than Ever