AI humanizers alter facts in 35-65% of documents, audit finds

Eleven commercial tools designed to make AI-generated text pass detection scans altered or dropped a checkable fact (a number, a date, a person or organization name, or a quoted span) in 35 to 65 percent of the documents that contained one, according to an October 2026 arXiv preprint auditing AI-text humanizers.

The paper writes that public leaderboards 'score evasion and embedding similarity, but not whether checkable details survive.' Its method is narrow and literal: run text through each humanizer, then run a rule-based checker over the output to see 'whether numbers, dates, person and organization names, and quoted spans survive each rewrite.'

Most of the damage was to quotes. Even stripping those out and counting only numbers and names, 10 of the 11 tools still altered at least one in 20 percent or more of the documents. The September cycle looked similar at 19 to 50 percent. An open 12B humanizer, run against fact-dense U.S. federal agency text, kept 'at least 97% of numbers and names' but reworded 67 to 78 percent of the quoted spans, 'against its own prompt.'

A tool that erodes fewer facts is not necessarily worse at fooling a detector. The authors report a Spearman rho of -0.46 (95% CI -0.65 to 0.08) between unit retention and detector evasion, and write that the two 'were not clearly related across tools.'

The detection side is messier. Perplexity-based detectors, Binoculars and Fast-DetectGPT, 'scored below chance (AUROC 0.37 and 0.39)' on 2026-model AI-styled text, with line breaks removed. A general-purpose model prompted with a single zero-shot question, which the paper calls Jev, reached AUROC 0.864 while 'calling 24% of the human originals AI at a 0.5 threshold.'

The authors say hypotheses, data rules and analyses were frozen before the main runs, and that they are releasing the checker and per-item results.