Calibrate text-watermark detection on the artifacts you actually review

Anthropic says Claude's text watermark has less room to operate where the next token must be exact. Code and factual passages can therefore carry less signal than freer prose. Short samples create another limitation.

Those details are explicit in Anthropic's explanation of its watermark. They are a reason to change the evaluation dataset before treating a detector as a routine check on engineering documents.

My acceptance rule is artifact-specific: a result demonstrated on long narrative answers does not establish a useful detection threshold for configuration files, brief incident notes or code changes.

Count opportunities for signal, not just words

The watermark works through choices in generated text. Where several continuations are suitable, the method can leave a detectable pattern without changing the intended meaning. Where the output must be exact, there is less freedom to do that. Anthropic says code comments can still provide opportunities even when other parts of the code do not.

That means document length alone is an incomplete description of the test material. A long file containing repeated structure and exact identifiers may offer a different signal from a similarly sized explanatory essay. This is a mechanism-based expectation, not a measured false-negative rate for either category.

I would resist turning “less signal” into “undetectable.” The vendor's explanation does not provide a universal boundary for every technical artifact. The practical response is to measure the cases the organisation intends to submit and allow the process to return an inconclusive result.

As an illustration, I would assemble an authorised, labelled test set for a hypothetical engineering-document workflow. It would separate explanatory prose, short factual notes, generated code with comments and lightly edited human text. The known process history would provide labels; the detector's own output would not.

I would then test the intended detector version and configuration, reporting results separately for each category. If access or the required validation data is unavailable, I would leave the proposed check unevaluated rather than inventing a reliability estimate from the mechanism description.

Abstention is a useful product behaviour

A workflow needs somewhere to put a sample for which the detector has too little evidence. Forcing every result into “AI” or “human” conceals the uncertainty precisely where the material may be least informative.

In the hypothetical test, I would record how often the process abstains as well as its errors among the results it does return. A system that avoids difficult samples can look accurate while answering very few of the questions the team actually cares about. Coverage and accuracy need to travel together.

The next check is whether the result changes an appropriate decision. An authorship or policy review still needs process evidence; a watermark is not a correctness test for the code. Even a confidently detected mark cannot tell the reviewer whether the proposed change is safe to merge.

The narrow contribution of artifact-level calibration is to prevent a broad detector claim from being inherited by every file type in an organisation. It helps determine where a signal is informative enough to include and where the workflow should rely on other evidence.

Before adding a watermark check to a technical review, take a sample of the actual artifacts the team handles. If the evaluation materials look nothing like them, the published detection result has not yet answered your operating question.

A watermark threshold belongs to a tested artifact class, not to every document that happens to contain text.