arXiv paper: GPT-4o podcasts drift, 'catch-n-repair' fixes turns

There is a small but useful result on the arXiv this week that anyone shipping AI-generated audio should probably read. The paper, On Improving Faithfulness of Podcasts from Documents, is billed as the first systematic study of whether long-form, multi-speaker LLM podcasts actually stay grounded in the source document they were built from. The short answer, per the authors, is not reliably, even when the generator is GPT-4o.

The setup is straightforward. Soumya Dutta, Tejas Indulal Dhamecha and Pannaga Shivaswamy construct a dataset of over 1500 documents spanning five domains, generate podcast transcripts with multiple LLMs, then score each conversational turn with an LLM-as-a-judge framework that they validate against human raters. Their finding is blunt: 'state-of-the-art models, including GPT-4o, frequently generate ungrounded content'. In other words, turns that scan smoothly in a two-host conversation but are not actually supported by the underlying document.

The proposed fix, catch-n-repair, is a model-agnostic pass that detects unfaithful turns and rewrites them while preserving conversational flow. The authors report 'consistent improvements in faithfulness across both in-domain and out-of-domain settings'. The honest caveat is that the abstract does not give the delta numbers, does not name which five domains are in the test bench, and does not enumerate every generator model beyond GPT-4o, so take the specifics as reported rather than settled until the full paper and code land.

Why this is worth attention if you are not writing papers: automated document-to-podcast generation is already a shipped consumer feature in more than one tool, and the paper's core point is that engaging narration is not the same as accurate narration. A repair-pass framework that lives outside the generator is exactly the kind of thing enterprise buyers of AI audio briefings will start asking for, an accountability layer they can bolt on regardless of whose model is under the hood.