Asked Five Times for the Same Text, an Open Model Carrying the Kind of Watermark Claude Now Uses Gave Two Answers With at Least 30 Identical Opening Words in 24 of 30 Requests. Without It, in None

A consultant asked an open AI model, five times, for about 250 words in the first person about a memorable trip to a coastal town, with a text watermark switched on. All five answers began “My heart still…”. Two of them ran identically for their first 43 words, down to “it was the perfect escape from the urban chaos”. Without the watermark, the same model began three answers with “Last summer” and two with “My heart still skips a beat”, and no two shared more than 11 opening words. Across 30 different requests, at temperature 1, at least two of the five watermarked answers began with 30 or more identical words in 24 requests. Without the watermark this happened in none: the longest shared opening was 17 words.

The method is SynthID-Text, published by Google DeepMind in Nature in October 2024 and used since then in the Gemini app. On 14 August Anthropic announced that Claude’s text would carry a version of it, to comply with the EU AI Act’s Code of Practice on transparency of AI-generated content; its help page, updated on 28 October, lists text watermarks for its current models, including Fable 5.1, Opus 5.5, Sonnet 5.5 and Haiku 5.5. The idea is that when a model faces a close choice of word, “overcast” or “grey” in Anthropic’s example, the choice is settled not by an ordinary random number but by a secret key combined with the few words that precede it. Anyone with the key can then check whether a text follows that pattern. “Watermarking does not impact the quality of Claude’s output,” Anthropic writes; in its internal testing it saw “no impact” on content, creativity or readability. It also cites Google’s experiment on about 20 million Gemini responses, where thumbs-up and thumbs-down rates did not differ significantly, and a side-by-side rating by people who saw no difference.

Those tests judge one answer at a time. The 56-page supplement to the Nature paper is more specific. The watermark is designed so that each single response is, on average, as good as an unwatermarked one; but, the authors write, this “does not necessarily preserve diversity across multiple responses”: when several responses are sampled for the same prompt, they are more similar to each other with the watermark than without. They add that this “could lower the overall quality of a system which generates many responses then selects the best one”, and that an adversary could use the repeated biases to steal the watermark. The remedy they describe, masking the watermark more often, makes it harder to detect. In a post on 24 August, the blogger Keyvan pointed out the gap between this passage and Anthropic’s sentence.

Natzir Turrado, a Spanish consultant in search marketing and data science, measured it. In an article published on 18 August and updated on 24 October, with code and data on GitHub, he ran Google’s open implementation of SynthID-Text with a key of his own and the paper’s settings, on Qwen2.5-7B-Instruct compressed to 4 bits, on a laptop. It is a replica of the method, not Claude’s watermark: nobody outside Anthropic has the key, and Anthropic’s detection service is open only to some organisations. One answer at a time, he found no measurable difference: on 100 English texts, average perplexity measured with a model from another family, Llama-3.1-8B, was 5.27 with the watermark and 5.42 without, a difference that is not statistically significant, and a language model asked to compare the two versions, in both orders, consistently preferred the watermarked one 5 times, the unwatermarked one 7 times, and neither in the other 88.

Then he asked for five answers to each of 30 requests, from why sleep matters to a story about a lighthouse keeper, with and without the watermark, at temperature 1 and at 0.7. He measured how much each answer reuses the word sequences of the other four, with Self-BLEU, the same yardstick the Nature paper uses: 0 means no shared sequences, 100 identical texts. At temperature 1 the score was 38.6 with the watermark and 19.4 without, higher with the watermark in all 30 requests; at 0.7 it was 45.8 against 28.6, higher in 27 of 30. Liminal Finds downloaded the 600 published answers and recomputed these figures with the method in his code, getting the same numbers. The count of identical openings in the first paragraph is ours, not his. At temperature 0.7 the median longest shared opening was 62.5 words with the watermark and 8 without, and in 5 requests two watermarked answers shared more than 100 words; no two answers were identical from start to finish.

What this touches is not the single answer a reader sees, but every use that relies on asking more than once: pressing the button to regenerate an answer, generating several drafts and keeping the best, or repeating a question to see whether the answers agree. If the watermark steers the same request towards the same words, agreement between answers says less than it seems. This is an inference from the replica and from the paper’s own text; nobody has published such a measurement on Claude.

The limits are substantial. The replica used a 7-billion-parameter model, one key, 30 English requests for short articles, and the configuration from the paper; Anthropic has not published how its watermark is configured. Self-BLEU measures shared word sequences, not quality as people judge it. When Turrado posted his result on X, Sumanth Dathathri, one of the SynthID-Text authors at Google DeepMind, replied on 20 August: “There's a lot of easy fixes to the diversity issue. I think if you tried the same prompt with Gemini/Claude many times, you'd get very diverse responses :)”. Turrado answered that his tests used the open implementation with its default configuration, agreed that production models might differ, and asked which fixes do not weaken detection. In his post, Keyvan wrote that no answer had come so far.