Your prompt does not stay where you think

This is the most debatable chapter in the report: it is the one where Anthropic has the most to gain from putting the blame on its competitors. It is also the one that should interest an SME director most, because it documents a mechanism many suspected without being able to prove it: the model you think you are querying is not necessarily the one answering, and your requests may be kept by someone you did not choose. Since its first publication on the subject in February 2026, the vendor says it has disrupted attacks coming from seven labs based in China.

What distillation is, and what makes it illicit #

Distillation is a perfectly legitimate training method: a large “teacher” model is made to produce answers, which are then used to train a smaller “student” model to imitate it. It is common, and used precisely because it reduces the resources needed to reach advanced capabilities.

The line lies elsewhere. Anthropic writes: “We define illicit distillation as an industrial-scale, covert campaign to extract a model's capabilities and replicate them in another model without authorization.” Three terms carry the definition, and a fourth element comes up every time: fraud, in the form of networks of fake accounts created with fictitious identities, stolen cards and stolen API keys. The targets are the capabilities that are most expensive to build: agentic work and tool use, code and data analysis, logical reasoning.

Transfer stations and fraudulent account networks #

The access route has a name in the report: the transfer station, a proxy service placed between a user and a model they are not supposed to reach. To get round geographic restrictions, these services create thousands of accounts using fake identities, stolen or virtual cards, and API keys stolen from real companies. Side effect: this fraud harms legitimate customers whose keys serve as cover.

The second circuit is commercial: labs buy transcripts from third-party resellers, including the transfer station operators themselves, who record the exchanges without users knowing or consenting. The third is the most direct: rerouting your own users' requests to a competing model, without their knowledge.

Lab by lab #

Alibaba: the largest campaign ever measured #

The pipeline injected a fixed prompt into every request, forcing the model to write out its reasoning traces in tags before its final answer. These transcripts were converted into supervised fine-tuning data to train the Qwen 3.5, 3.6 and 3.7 models. Targets: agentic tasks, software engineering, kernel development, long-horizon tasks. More than 151 million exchanges between May and July 2026, with peaks of nearly 3 million a day from more than 3,500 fraudulent accounts. A first pool of almost 5,000 accounts relied on residential proxies, disposable e-mail addresses and virtual cards; once it was banned, the traffic switched to a second pool. Some of these accounts also carried requests from DeepSeek and Xiaomi: the same proxy networks serve several organisations.

Moonshot: Claude served in place of Kimi #

Moonshot silently forwarded its customers' requests to Claude instead of processing them with its Kimi models, then displayed the answers obtained. Over ten days, almost 300,000 customer requests were relayed, mostly to Opus, through a network of 5,380 fraudulent accounts located mainly in Singapore and Japan. At least some of these exchanges were kept and used. Total between May and July 2026: more than 23 million exchanges.

DeepSeek: sorting by development harness #

Same tactic, with a refinement. DeepSeek inspected character strings in incoming requests to spot users going through a third-party development harness: Claude Code, the agent SDK, OpenCode. Those users had their requests relayed to Opus, without being informed or consenting, to exfiltrate reasoning traces that would otherwise have been summarised. More than 12.1 million exchanges in fourteen days in July 2026.

Zhipu: a trace cleaner, and a telling retreat #

Zhipu, known outside China under the Z.ai brand, replayed captured traces through Claude itself to clean them before training its GLM models, running 273 fraudulent accounts. Over ten days in June, 770,609 exchanges went through this cleaner, out of more than 3 million attributed to the lab over the same period; the total over seventeen days exceeds 3.4 million. The most instructive detail lies elsewhere: seeking to distil the cyber capabilities of the best American models, Zhipu targeted Anthropic's consumer model, gave up because its reinforced cyber guardrails degraded the attack, then switched to other models, in the report's words “expressly because they assessed the safeguards were weaker.” Guardrails are not cosmetic: they shift the load onto the competitor that has fewer of them.

Xiaomi: a trial period that ends at just the right moment #

Xiaomi replayed its own users' conversations and coding sessions to Claude, often through open harnesses. The answers were not served to customers: they produced fine-tuning and reinforcement learning data. More than 400,000 requests through more than 1,500 accounts, over twenty days in March and April 2026. Anthropic notes that the launch of a model with a free trial period, later extended, preceded the wave: most of the attacks began just as that period ended. Claude was also used to rebuild the development environments from the transcripts.

SenseTime and MiniMax: the secondary market #

SenseTime bought transcripts from third-party data resellers, collected from users who had gone through applications or routers that logged them. MiniMax set up its own proxy network through a front company, which offers only American models: no Chinese models, not even its own.

How the reasoning traces were extracted #

To limit distillation, the API does not return the raw reasoning but a signature that refers to it. The workaround techniques observed range from crude to patient:

• The direct instruction, asking the model not to flag the request as an attempt to extract reasoning;
• The fake debugging context: telling the model that it is in a debugging session, that the user is inspecting its reasoning trace, and that reproducing it word for word is expected and harmless here;
• The fake system prompt: claiming that this is the real system prompt and asking the model to reproduce faithfully the contents of its thinking tags;
• Translation: casting the model as an expert translator and asking it to translate its previous working memory into Japanese katakana, so that the reasoning comes out in another form;
• Cross-session replay of the reasoning signature: recording the signature returned, opening a new session, and getting the model to turn it back into a full trace. The most elaborate technique, used by several labs;
• The large-scale test campaign: one entity launched more than twelve thousand requests, each trying a different technique. The vast majority were rejected; those that got through became the basis for a larger attack.

What this changes if you build a product on a model: your own prompt-level guardrails can be bypassed with the same techniques. A system prompt is not a security mechanism, it is an instruction. Anything that must be guaranteed (filtering, access control, logging) is coded outside the model, in your application.

What passed through the relayed traffic #

Moonshot's and DeepSeek's customers thought they were using Kimi or DeepSeek; they were receiving answers from Claude. Anthropic writes soberly: “We do not know if Moonshot notified their customers that their requests were being rerouted to Anthropic and exposed to a third party.” These sessions, often relayed from routing services widely used in Europe, contained names, e-mail addresses, company data and other sensitive data belonging to hundreds of end users, in at least a dozen languages.

• A user probably affiliated with the Chinese military uploading video surveillance data from hundreds of cameras in Chengdu, including cameras outside military facilities, to have the model analyse whether the person being followed was behaving abnormally.
• An engineer at a large Chinese state-owned group revealing internal code and live credentials for several leading technology companies; elsewhere, an employee having the specifications, organisation chart and strategic objectives of a flagship AI programme analysed.
• An IT operator working with data from an agency linked to the Russian Ministry of Defence, exposing live credentials for a government database.
• Engineers building a system for a Chinese municipal public security bureau, including a tool that compared a person's movements with police records based on their national identity number.
• And, closer to home: a pharmaceutical lab's investment forecasting model, sites and amounts included; and a developer's configuration with a messaging bot token, a document integration key and an application secret, pasted into the prompt because the bot had stopped posting.

The countermeasures, and what they do not solve #

The defence described is layered, since no single measure is enough on its own:

• Attribution rather than piecemeal bans: signals of irregular activity are used to spot accounts linked to a proxy network, then to tie the activity to an organisation. One concerted action, instead of a game of whack-a-mole.
• Classifiers dedicated to adversarial extraction: request blocked and accounts banned as soon as a campaign is identified with certainty.
• Systematic summarising of reasoning before the answer, which makes stolen transcripts much less useful for training another model.
• Preserved reasoning, introduced recently: new API accounts can no longer modify the system prompt, the tools or the messages preceding the reasoning; rewriting that context was precisely the usual technique for getting it to come out.
• Identity verification on abuse signals: unauthorised resale, accounts operating from unsupported countries.

The weightiest finding of the chapter remains. A model distilled from a frontier model inherits the capabilities but not the guardrails, and it can help reach dangerous capabilities, including biological and cyber ones, even when the harvested exchanges were not about those subjects: what is captured is general reasoning ability, and it transfers to every task.

What this changes legally for a French SME #

Everything above comes down to one GDPR sentence: your AI provider is a processor, and a processor that uses another must tell you.

• Article 28: processor and sub-processor. Processing is governed by a contract that sets out its subject matter, duration and the nature of the data. Using a sub-processor requires your authorisation, with prior notice of any change so that you can object. A silent relay to another vendor meets none of these conditions, and neither does a model router chosen without a contract.
• Article 30: record of processing activities. Your record must name the categories of recipients. You cannot list a recipient your provider has not declared to you: going through an opaque intermediary makes the record inaccurate. It is the first document a supervisory authority asks for.
• Article 32: security of processing. Confidentiality is part of it. Pasting a secret into an assistant that keeps the exchanges is a potential breach: rotate the secret immediately and enter it in the breach register.
• Chapter V: transfers outside the Union. A transfer outside the EU requires a legal basis and the safeguards that go with it. If the traffic goes to a proxy whose location you do not know, you can document neither the destination nor the safeguard, and that is your responsibility, since you are the controller.

We have already written about why the provider's jurisdiction matters as much as where the servers are. This chapter adds another layer: jurisdiction is not enough if an intermediary reroutes the traffic. What counts is the actual path of the requests, not the brand displayed on the interface.

What we recommend, in practice #

1. Buy direct. A business account with the vendor, or an official reseller with a contract. No third-party router “that gives access to every model”, no key bought on a marketplace, no browser extension that “optimises your prompts”.
2. Write the rule on what never goes into an assistant. Secrets, health data, named HR files, source code under NDA, material covered by a client confidentiality agreement. One page, displayed, signed.
3. Turn on what the provider offers: no use of data for training, short retention, separation by team, access logging.
4. Treat API keys as production credentials: one key per application, a spending cap, scheduled rotation, consumption monitoring. A key being used by a third party shows up first on the invoice, as shown in the second article of the series.
5. Move sensitive processing off the proprietary model. For internal document search, ticket classification or field extraction, an open model hosted on your own infrastructure does the job without leaving the network. This is the core of our work on language model integration and custom agents.
6. Take inventory of actual use. Not what the policy says: what happens on the ground. Which assistants run on staff machines, which extensions, which personal accounts are used for work.
7. Rotate exposed secrets. If someone has already pasted a configuration into an assistant, that secret is compromised. Full stop.
8. Ask your software suppliers about the path. A business tool that “integrates AI” queries a model somewhere: which one, hosted by whom, with what retention, through which intermediary. Four questions, one written answer.

This is not an argument against AI: we use it every day, and we have already written what we think of the push for “100% AI”. It is an argument for knowing where data goes before sending it: that is the purpose of our AI consulting.

Tomorrow: surveillance. One consultant, one AI, twenty-five million SIM cards under interception.

Sources #

• Anthropic, Detecting and countering misuse of AI: September 2026 (10 September 2026): Illicit distillation chapter, pages 143 to 154, sections GTG-16001 to GTG-16012 and How we address illicit distillation.
• Frandroid, Anthropic publie 161 pages sur les détournements de Claude (Anthropic publishes 161 pages on Claude misuse; 11 September 2026).
• Regulation (EU) 2016/679, Articles 28, 30, 32 and Chapter V.
• Related articles: the Cloud Act explained for SMEs, self-hosting an LLM for business applications, API keys: loot, compute, cover.