Vauhini Vara tests whether readers can tell AI from her fiction

"Of my seven readers, none correctly identified more than half the passages," novelist Vauhini Vara writes in The New Yorker. Vara had fed her own books to a large language model, had it generate new passages in her voice, mixed those with excerpts of her actual writing, and asked friends to sort human from machine. Her friend the novelist Karan Mahajan misidentified all four excerpts he was shown.

Vara's essay is built around a preregistered study by Tuhin Chakrabarty of Stony Brook, Jane C. Ginsburg of Columbia Law School, and Paramveer Dhillon of the University of Michigan, testing whether AI models trained on copyrighted books can pass for the writers whose books trained them. The team compared ChatGPT, Claude, and Gemini against MFA-trained professional writers across 50 award-winning authors' styles. When the models were merely prompted in context, MFA-trained readers strongly preferred the human writing. When the same models were fine-tuned on the author's full oeuvre, MFA readers reversed and favored the AI, and the fine-tuned outputs evaded automated detection about 97 percent of the time, against 3 percent for the in-context runs. Median production cost per author: $81.

Authors whose work was fed into the pipeline included the Nobel laureate Han Kang, Junot Díaz, Sigrid Nunez, and Tony Tulathimutte. "When I wrote to the authors whose work was used in the study, most declined to be interviewed or didn't respond," Vara writes.

Vara ran her own oeuvre through the same procedure. "Reading some of its lines next to my own, I couldn't remember which was which," she writes. "Unlike Díaz or Nunez, I even preferred some of the doppelgänger's versions."

"It's tempting to believe A.I. doesn't really pose a threat to literature," she writes. Chakrabarty, she reports, supports a ban on what he himself did. Three of the AI experts on our Who's Who list shared the essay.