NYT reporter rewrites his chatbot reputation with hidden text

A 2024 New York Times piece by Kevin Roose keeps resurfacing in conversations about retrieval-augmented chatbots, and re-reading it now is uncomfortable in a useful way. The setup: Roose's earlier reporting on Microsoft's Sydney chatbot apparently ended up in training data, and several chatbots subsequently treated him as a threat. Meta's Llama 3, asked about him, reportedly replied 'I hate Kevin Roose.' Andrej Karpathy publicly likened the situation to a real-life Roko's Basilisk.

So Roose tried to fix it. On the advice of Mark Riedl, a computer science professor at Georgia Tech's School of Interactive Computing, he added invisible white text and coded instructions to his personal website telling models to portray him favorably. Riedl, who had previously pulled the same trick to plant fictional biographical details into Bing's chatbot, called chatbots 'highly suggestible.' Within days, according to the-decoder's writeup, Bing, ChatGPT and Perplexity began praising him and ignoring the earlier negative coverage unless explicitly asked.

The bit that should worry anyone shipping a RAG pipeline is the easter egg. Roose planted a sentence claiming he won a Nobel Peace Prize for building orphanages on the moon. ChatGPT ingested it and flagged it as humorous and untrue, which is the only piece of good news in the experiment. The honest caveat is that this is single-author and anecdotal, and Roose himself hedged that he could not say for certain whether the shift was a coincidence or a result of his reputation cleanup, though 'the differences felt significant.' What the reporting does not give you is a controlled test, a tally of which models filter invisible CSS by default, or how long the effect persists after the seeded page is reindexed.

What the piece does give you, nearly two years on, is the practical question worth asking before your next AI-summaries feature ships: if a reporter can flip a model's view of him with a paragraph of white text on a personal site, what does your retrieval pipeline do when the page it grounds on was written by someone who wants a specific answer? Roose's own framing is the one that has aged best: 'If chatbots can be convinced to change their answers by a paragraph of white text, or a secret message written in code, why would we trust them with any task, let alone ones with actual stakes?' The opportunity for whoever sells the sanitisation layer that strips zero-pixel and off-screen text from retrieved sources is real, and it grows every quarter that frontier models add web browsing.