DeepMind's SynthID-Text watermarks live Gemini responses

DeepMind ran the watermark on live Gemini traffic and it did not move the needle on quality. According to the Nature paper, the team assessed feedback from nearly 20 million Gemini responses. On the watermarked model, thumbs-up rates rose by 0.01% and thumbs-down rates fell by 0.02%. Both differences were called "statistically insignificant."

The scheme, called SynthID-Text, does not touch training. "SynthID-Text does not affect LLM training and modifies only the sampling procedure," the authors write, and "watermark detection is computationally efficient, without using the underlying LLM." To make it deployable at scale, they folded it into speculative sampling, the efficiency technique that production LLM stacks already use.

The cost, on the paper's own benchmark: a Gemma 7B-IT model served on 4 v5e tensor processing units generates text at 15.527 ms per token, rising to 15.615 ms with 30-layer Tournament sampling. The paper calls it "a latency increase of only 0.57%."

"We hope that the availability of SynthID-Text will facilitate further development of watermarking and responsible use of LLM systems," the authors write. The paper reports SynthID-Text is now used to watermark Gemini and Gemini Advanced.

Evaluations described in the paper cover Gemma 2B-IT, Gemma 7B-IT and Mistral 7B-IT variants. The abstract publishes no per-attack robustness numbers.