Data Privacy — GDPR, CCPA, and Privacy-Preserving AI

In October 2006 Netflix published 100 million film ratings from 480,189 subscribers as a public competition dataset. Names were removed. Subscriber IDs were replaced with random numbers. By the standards of the time this was a careful release.

Two years later Narayanan and Shmatikov showed that an attacker who knew roughly eight of a person's film ratings, with dates accurate to within a fortnight, could uniquely identify 99% of records in the set — and two of the eight ratings could even be wrong. They demonstrated it by cross-referencing public IMDb reviews. Film-viewing history reveals political leanings, religion and sexuality. A closeted subscriber sued in 2009; Netflix cancelled the planned sequel competition in 2010.

The lesson generalises well beyond film ratings. Latanya Sweeney had already shown that 87% of the US population is uniquely identified by just three fields — five-digit postal code, gender and date of birth — and used that fact to pull the Governor of Massachusetts's hospital records out of a supposedly anonymised state dataset. In 2006 AOL released 20 million "anonymised" search queries; journalists identified user 4417749 as a 62-year-old woman in Georgia within days, from her searches alone.

Privacy law exists because "we removed the names" has never worked. Understanding why it does not work is what turns compliance from a legal formality into an engineering constraint.

GDPR: six principles and one idea underneath them

Article 5 sets out six principles. The underlying idea is that personal data is borrowed, not owned: you hold it for a stated purpose, for a stated period, and the person it describes keeps rights over it throughout.

Article 5(2) adds accountability: you must be able to demonstrate compliance. In practice that means the documentation is the compliance. An untested claim that your training set was minimised is worth nothing without a record of what you excluded and why.

Article 9 sits on top and prohibits processing special-category data — racial or ethnic origin, political opinions, religious belief, trade union membership, genetic data, biometric data used to identify someone, health, sex life and sexual orientation — unless a specific exception applies. Note the collision this creates: the attributes you most need in order to test a model for discrimination are exactly the attributes that are hardest to lawfully hold.

Penalties run to €20 million or 4% of worldwide annual turnover, whichever is higher, for the serious tier.

Article 22: the automated-decision rule that most teams misread

Article 22 gives a person the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects or similarly significantly affects them. Credit refusal, job rejection, insurance pricing and benefit denial all qualify.

Three exceptions permit it anyway: necessity for a contract, authorisation by Union or Member State law, or explicit consent. But even then, safeguards are mandatory — the right to obtain human intervention, to express a point of view, and to contest the decision.

The word that gets misread is "solely". Teams assume that inserting a human reviewer removes them from Article 22. Regulators do not accept nominal review. If the human lacks the authority, information or time to reach a different conclusion, the decision is still automated in substance. A caseworker approving 400 model outputs an hour is a formality, not an intervention.

The CJEU sharpened this in the SCHUFA judgment of December 2023 (case C-634/21). A German credit agency argued it merely produced a score; the lender made the decision. The Court held that where a third party draws strongly on that score to grant or refuse credit, producing the score is itself an automated decision under Article 22. Building the model does not put you outside the rule because someone else presses the button.

A follow-up case in February 2025 (C-203/22, Dun & Bradstreet Austria) addressed what Article 15(1)(h)'s "meaningful information about the logic involved" requires. The answer: not the source code or the full algorithm, but enough concise, intelligible detail about the procedure and the principal factors that the person can understand the result and contest it. Trade secrets do not licence a blanket refusal — the protected material goes to the supervisory authority or court, which decides what must be disclosed.

"A human reviews it" only removes you from Article 22 if the human can, in practice, reach a different answer — with the information, authority and time to do so.

Data subject rights, and the one that breaks machine learning

Erasure versus a trained model

Deleting a row from a database is easy. Removing that row's influence from a model trained on it is a research problem.

The influence is real and measurable, not theoretical. Membership inference attacks (Shokri et al., 2017) determine whether a specific record was in the training set, by exploiting the fact that models are slightly more confident on data they memorised. Worse, models can regurgitate training data verbatim: Carlini and colleagues inserted a synthetic nine-digit number into an email corpus and extracted it back out of the trained language model, then in 2021 recovered names, phone numbers and email addresses verbatim from a public model. If a model can emit someone's phone number, that model contains personal data. EU regulators take the same line: in Opinion 28/2024 (December 2024) the European Data Protection Board said a model trained on personal data counts as anonymous only if identifying people from it, or extracting their data through queries, is very unlikely — and that this has to be shown case by case.

SISA (Bourtoule et al., 2021) is the practical answer: split training data into shards, train an isolated sub-model on each, and aggregate predictions. An erasure request then requires retraining only the shard containing that record. If you expect erasure requests at volume — any consumer product — this is an architecture decision to make before training, not after the first request arrives.

CCPA and CPRA: the same problem, a different philosophy

California's regime applies to for-profit businesses doing business in California that meet one of three thresholds: annual gross revenue above 26,625,000 US dollars (the original 25 million, adjusted for inflation every two years; this figure applies from 2025); buying, selling or sharing the personal information of 100,000 or more consumers or households a year; or deriving half or more of annual revenue from selling or sharing personal information.

Core rights: to know what is collected and how it is used; to delete; to opt out of the sale or sharing of personal information; to correct inaccurate information; to limit the use of sensitive personal information; and non-discrimination for exercising any of these — you cannot degrade the service of someone who opts out.

The practical consequence for an ML team is that GDPR forces a question at collection time — on what basis am I allowed to have this? — while CCPA forces a capability at request time: can you actually locate and remove one person's data across every store you own? Most organisations fail the second test not for legal reasons but because nobody knows where all the copies are.

India's DPDP Act: consent first, with a long runway

India's Digital Personal Data Protection Act, 2023 covers digital personal data processed in India, and processing abroad when it is connected with offering goods or services to people in India. The DPDP Rules, 2025, notified in November 2025, bring it into force in steps: the Data Protection Board's provisions at once, registration of consent managers after one year, and most duties on businesses — notices, security safeguards, breach notification, children's data, rights requests — eighteen months after notification, in May 2027.

Three differences from GDPR matter to an ML team:

• A narrower lawful basis. Processing needs consent, or one of a closed list of "certain legitimate uses" in Section 7. There is no open-ended legitimate-interests basis to lean on.
• No special categories and no automated-decision right. The Act has no Article 9-style tier of sensitive data and no Article 22-style right about automated decisions. That makes fairness testing less constrained by privacy law, and gives people fewer rights to contest a model's decision.
• Public data and children. The Act does not apply to personal data the person themselves made public (Section 3(c)(ii)) — which covers what someone chose to post, not everything a scraper can reach. A child is anyone under 18: their data needs verifiable parental consent, and tracking, behavioural monitoring and targeted advertising aimed at them are barred, subject to exemptions the Rules set out.

Penalties go up to ₹250 crore for failing to keep reasonable security safeguards, and up to ₹200 crore each for breach-notification failures and breaches of the children's-data duties.

Privacy by design: what actually works

Minimisation is the cheapest control there is

Data you never collected cannot leak, cannot be subpoenaed, cannot be re-identified and does not need erasing. Before adding a feature, ask what its measured lift is. A field that adds 0.3 points of AUC and carries health information is a bad trade, and the ML team is the only group in the organisation positioned to notice.

Pseudonymisation is not anonymisation

Under GDPR, pseudonymised data — identifiers replaced by tokens, with a key held separately — is still personal data and still fully in scope. Only genuinely anonymous data, where re-identification is not reasonably possible by anyone, falls outside the regulation. The Netflix and AOL cases are what "reasonably possible" looks like in practice.

k-anonymity requires every record to be indistinguishable from at least k−1 others on the quasi-identifiers, achieved by generalising (age 34 becomes 30–39) and suppressing. It has two well-known holes. If all k records in a group share the same sensitive value, you learn it anyway — hence l-diversity, requiring diversity of sensitive values within each group, and t-closeness, requiring the group's distribution to resemble the whole. And all of them collapse in high dimensions: with hundreds of columns, almost every row is unique, and generalising enough to fix that destroys the data.

Differential privacy: a guarantee rather than a hope

Every technique above tries to hide individuals in a crowd, and attackers keep finding them. Differential privacy inverts the approach: instead of hiding the person, it guarantees that the output barely changes whether or not any one person is in the data.

Formally, a randomised mechanism $M$ is $\varepsilon$-differentially private if, for any two datasets $D$ and $D'$ differing in a single record, and any set of outputs $S$:

Read $e^{\varepsilon}$ as the most an adversary's odds can shift by learning whether you were included. At $\varepsilon = 0.1$, that factor is 1.105 — about a 10% shift, strong protection. At $\varepsilon = 1$, it is 2.72. At $\varepsilon = 10$ it is 22,026, which is a guarantee in name only.

The Laplace mechanism achieves this for a numeric query by adding noise drawn from a Laplace distribution with scale $b = \Delta f / \varepsilon$, where $\Delta f$ is sensitivity — the most the answer can change if one person is added or removed.

1import numpy as np23def dp_count(true_count, epsilon, sensitivity=1.0, rng=np.random.default_rng(0)):4    return true_count + rng.laplace(loc=0.0, scale=sensitivity / epsilon)56# A counting query has sensitivity 1. With epsilon = 0.5, scale b = 2.7# For Laplace, P(|noise| > t) = exp(-t / b), so 95% of draws land within8# t = b * ln(20) = 2 * 3.0 = 6.0 of the truth.

The number that matters

Work through the consequence, because it is the single most important fact about deploying DP. With $\varepsilon = 0.5$ and sensitivity 1, the noise scale is $b = 1/0.5 = 2$, and 95% of the time the added noise falls within ±6.

The noise is absolute, so its damage is inversely proportional to group size. Large groups barely notice; small groups are obliterated. This is the mechanism behind the documented finding that differentially private training degrades accuracy far more for under-represented groups than for well-represented ones. DP-SGD compounds it: per-example gradients are clipped to a fixed norm before noise is added, and atypical examples — which is what members of a small group look like to the model — have the largest gradients and therefore lose the most information to clipping.

Differential privacy protects everyone equally in the guarantee and unequally in the outcome: the same absolute noise is a rounding error for a large group and total destruction for a small one.

For model training, DP-SGD (Abadi et al., 2016) clips per-example gradients to norm $C$, adds Gaussian noise proportional to $C$, and uses a privacy accountant to track cumulative $\varepsilon$ across steps. That accounting matters: the privacy budget composes, so every query and every epoch spends from a finite total. A team that answers "just one more analysis" fifty times has no guarantee left.

Federated learning does not, by itself, give privacy

Keeping raw data on the device and sending only gradients sounds sufficient. It is not — gradients leak. Work on deep leakage from gradients showed that training images can be reconstructed at pixel level from the gradients alone. Federated learning is a useful component, but a private system needs it combined with secure aggregation (so the server sees only sums) and differential privacy (so the sums themselves are protected).

The DPIA, written so it is useful

Article 35 requires a Data Protection Impact Assessment before processing likely to result in high risk — explicitly including systematic and extensive automated evaluation with legal or similarly significant effects, large-scale processing of special-category data, and large-scale systematic monitoring of public areas. Most consequential AI systems hit at least one trigger.

Four required elements: a systematic description of the processing and its purposes; an assessment of necessity and proportionality; an assessment of the risks to individuals' rights and freedoms; and the measures that address those risks. If high risk remains after mitigation, Article 36 requires prior consultation with the supervisory authority before you go live.

The section that determines whether a DPIA is real is necessity and proportionality. "Could we achieve this purpose with less data, less identifiable data, or no model at all?" A DPIA that never once concludes "we will not collect this" is a document nobody read.

Where privacy and fairness pull against each other

These two goals are usually presented as allies. They collide in three specific, concrete places, and knowing where saves a great deal of confused argument.

The middle row is the one that surprises people. Adding privacy protection can make a system less fair, and unless you are measuring per-group outcomes you will not see it happen.

What to build differently because of this

Design for deletion at the start. Decide before training how an erasure request will be honoured — sharded training, a documented retraining cadence, or a defensible written position on why the model itself contains no recoverable personal data. Retrofitting is a rebuild, and "we cannot delete you from the model" is not an answer you want to give a regulator for the first time under a deadline.

Instrument the lineage. When a request arrives you need to answer, in hours, which raw tables, feature stores, caches, logs, backups and model versions contain that person. If that answer requires a person to think, you do not have a compliant system; you have a compliant intention.

Treat every new field as a liability with a measured benefit. Log the AUC lift each feature buys. A field that buys 0.2 points and carries health or ethnicity information is a straightforward decision once the number is on the table, and an interminable argument while it is not.

Never deploy differential privacy without per-group utility checks. Compute what the noise does to your smallest group before choosing ε, using the arithmetic above. If a subgroup of 12 comes back with 50% error, you have not protected those people — you have erased them from the analysis, which is its own kind of harm.

And keep the fairness data separate and locked. Collect the protected attribute for the sole, documented purpose of bias testing; hold it in a store that the training pipeline cannot read; grant access to the audit function only. That configuration satisfies both the regulator asking why you hold ethnicity and the auditor asking why you cannot produce per-group error rates.