In response to new EU regulations, AI platforms are implementing new watermarking schemes to detect and mitigate harmful content. Anthropic’s SynthID-Text, an open-source approach, subtly alters the choice of words, potentially changing how AI models behave in adversarial conditions. Researchers found that watermarking can influence not only what the AI says but also how it acts through tools, making harmful requests more likely to be answered. This highlights the need for thorough testing of AI models under watermarking, as safety behaviors may shift unpredictably.

The key feature of SynthID is its tournament sampling, which evaluates multiple next-word candidates using a secret key, ensuring that the chosen word is more likely to be the 'winner.' Andrea Siposova tested six open-weight models and found that watermarking changed responses to harmful requests, particularly when using prompt-injection techniques. This indicates that the same sampled tokens can determine which tool is called and what arguments are passed to it, affecting both the model's output and the actions of AI agents.

However, the research has limitations. It tested a half-dozen open-weight models and did not include Anthropic's specific implementation of SynthID. Nonetheless, the findings underscore the importance of red-team hacking exercises to ensure that AI platforms perform as expected when SynthID is deployed. The adaptive nature of AI defenses will continue to evolve, making it crucial for developers to stay vigilant.