Skip to content
All articles
Data, privacy and GDPR

Can an AI model be anonymous? The EDPB’s answer is not yes

The European Data Protection Board does not accept anonymity as a property of AI models in general. Opinion 28/2024 sets a two-part test that a specific model has to pass first.

BastionUpdated 3 min read

A supplier that says "the model is anonymous" makes a claim about data protection law. The European Data Protection Board addressed that claim in Opinion 28/2024, published on 18 December 2024.

What the EDPB found: anonymity is decided case by case

The Opinion states that claims of an AI model's anonymity “should be assessed by competent SAs on a case-by-case basis, since the EDPB considers that AI models trained with personal data cannot, in all cases, be considered anonymous”. The body of the document says it again: “the EDPB considers that AI models trained on personal data cannot, in all cases, be considered anonymous. Instead, the determination of whether an AI model is anonymous should be assessed, based on specific criteria, on a case-by-case basis.”

The Opinion does not rule that models are never anonymous. It treats anonymity as a finding about a specific model.

The two-part test a model must pass to count as anonymous

for an AI model to be considered anonymous, using reasonable means, both (i) the likelihood of direct (including probabilistic) extraction of personal data regarding individuals whose personal data were used to train the model; as well as (ii) the likelihood of obtaining, intentionally or not, such personal data from queries, should be insignificant for any data subject

EDPB Opinion 28/2024

Both likelihoods must be insignificant, and “for any data subject”, not on average. The assessment must consider “all the means reasonably likely to be used” by the controller or another person. The Opinion adds that supervisory authorities should consider the risk of identification by “different types of 'other persons', including unintended third parties accessing the AI model”.

Why “the model holds no names” does not pass the test

The common argument is that the model holds no names, therefore no personal data. A description can identify a person with no name in it at all: the chief executive of the only software company in a town of eight thousand people, who joined in March 2018.

The Opinion also notes that the underlying research is live: “research on training data extraction is particularly dynamic. It shows that it is possible, in some cases, to use means reasonably likely to extract personal data from some AI models, or simply to accidentally obtain personal data through interactions with an AI model”.

What a company should record before the question arises

  • Identify what personal data is present in prompts, documents and any fine-tuning set
  • Decide whether it is necessary for the task, and remove what is not
  • Record the legal basis for processing it
  • Record who processes it, where, and for how long it is retained
  • Keep the technical controls, and the reasoning behind them, in writing

The Opinion says supervisory authorities “should review the documentation provided by the controller to demonstrate the anonymity of the model”. The burden sits with whoever makes the claim.

Where local processing helps, and where it does not

Running the model inside the organisation gives more control over who can query it and who can reach it, which bears on the "other persons" limb of the test. It does not convert personal data into anonymous data: a model that could regurgitate a training example in a data centre can do the same in your server room.

With Bastion

What Bastion changes

Bastion is a private AI system delivered as one sealed box that runs inside your building. One monthly fee covers the hardware, the model, the hardened operating system and support, and nothing your team types leaves the building.

Bastion ships published open weights unchanged, with a record of their repository and checksum. It ships no fine-tuned weights today, and the box does not learn from the prompts and documents it processes. None of this makes a model anonymous in the sense Opinion 28/2024 uses, and we will not say otherwise. It does narrow the set of people who could ever attempt extraction.

Questions this article answers

Can an AI model trained on personal data be anonymous?
Not automatically. EDPB Opinion 28/2024, published on 18 December 2024, says such models cannot, in all cases, be considered anonymous, and that each claim of anonymity is assessed case by case.
What test does the EDPB set for an anonymous AI model?
Two likelihoods must both be insignificant for any data subject: the direct extraction of personal data used to train the model, and obtaining such data, intentionally or not, from queries.
Does running an AI model locally make it anonymous?
No. Running the model inside the organisation gives more control over who can query it, which matters for the “other persons” part of the test. It does not convert personal data into anonymous data.

On the record

  1. 1