Differential Privacy for LLMs

LLMs memorize. Carlini et al. (2021) showed that production language models will reproduce verbatim training text when prompted for it. Differential privacy is the formal defense: train so the output is provably insensitive to any single training example. The 2024-2025 evidence says DP-SGD is necessary, but the ε values people deploy may not match the threat model they care about.

(ε, δ)-differential privacy

A randomized algorithm M is (ε, δ)-DP if for any two datasets differing in one example and any event S: P(M(D) in S) <= e^ε * P(M(D') in S) + δ.

Interpretation: the output distribution is close enough (parametrized by ε) that the contribution of any single individual cannot be reliably inferred, except with probability δ.

DP-SGD

Abadi et al. 2016. The standard recipe:

1. Sample a mini-batch.
2. Compute per-example gradients.
3. Clip each per-example gradient to a threshold C.
4. Sum the clipped gradients and add Gaussian noise with std σ * C.
5. Use the noisy sum to update parameters.

Privacy cost is tracked by an accountant (Moments Accountant, Rényi DP accountant). Reported ε values in the LLM literature vary widely by threat model, data sensitivity, and utility target; there is no universally "safe" default ε. Published examples span roughly ε ≈ 1–10 in some LLM training settings, but treat those as illustrations, not recommended defaults. Lower ε generally needs more noise, and more noise tends to cost utility.

LoRA + DP-SGD

Full DP-SGD on a frontier model is prohibitive. LoRA (Hu et al. 2022) confines the updates to a small adapter, which shrinks per-example gradient storage. LoRA + DP-SGD is the common 2025 configuration. The DP guarantee covers the adapter, and the base model stays fixed.

The 2024-2025 tension

Two lines of evidence:

• Canary MIA (Duan et al. 2024). Insert unique canaries into training data, measure whether a membership-inference attacker can identify them. It reports limited success on language models, which suggests MIA is hard.
• Training-data extraction (Carlini 2021, Nasr et al. 2025). Prompt the model with a prefix; measure whether it recovers verbatim text from training. It reports substantial memorization, which suggests that in the sense that matters, MIA is easy.

The March 2025 paper (arXiv:2503.06808) resolves it by pointing out that the two measure different things. MIA asks "is example e in D?" on inserted canaries. Extraction asks "what can I recover of D?" For privacy, the "most extractable" example is the one that counts, and canaries under-report it because nobody optimized them to be extractable.

The paper also introduces new canary designs. They support loss-based MIA without shadow models, and they yield the first nontrivial DP audit of an LLM trained on real data with realistic DP guarantees.

Alternatives to DP training

• PMixED (arXiv:2403.15638). Private prediction at inference time. Mixture of experts on next-token distributions; each expert sees a shard of training data; aggregation adds noise for DP. Avoids DP training entirely.
• DP synthetic data generation (Google Research 2024). LoRA-fine-tune with DP-SGD, sample synthetic data, train a downstream classifier on the synthetic data.

Both sidestep the utility cost of full DP training, and the price is a different threat model.

Differential Privacy Reversal via LLM Feedback

This is an emerging 2025 attack. It uses a DP-trained model's confidence scores as an oracle to re-identify individuals. Outputs can look clean while the confidence distributions still leak.

The defense is to withhold confidences, or to truncate or quantize them before exposure. That is a separate requirement on top of (ε, δ)-DP training.

Where this fits in Phase 18

Lessons 20-21 cover bias and fairness, this lesson covers privacy, Lesson 23 covers provenance via watermarking, and Lesson 27 covers the regulatory data-provenance layer.

main.py runs a DP-SGD toy on a 2-feature binary logistic regression with 500 training and 200 test points. It sweeps the noise multiplier σ over 0.0, 0.5, 1.0, 2.0, and 4.0 with the clipping norm C fixed at 1.0, and prints an approximate ε and the test accuracy for each. The ε is an analytical Gaussian-mechanism proxy, not output from a real accountant, so read it as a rough trend. There is no canary attack in the file. Exercise 2 asks you to add one.

This lesson produces outputs/skill-dp-audit.md. Hand it a DP claim about a language model deployment and it audits four things: the (ε, δ) values, the accountant used, the MIA evaluation protocol, and whether anyone has assessed confidence-score exposure.

1. Run code/main.py. Sweep σ in {0.5, 1.0, 2.0} and report the (ε, δ)-accuracy trade-off. Identify the point at which utility collapses.

2. Implement a canary insertion and a log-loss test. Measure detection rate before and after DP-SGD at σ = 1.0.

3. Read Nasr et al. 2025 on training-data extraction. Why does extraction success not collapse under moderate ε? What does this imply about MIA-as-evaluation?

4. Design a deployment using PMixED (arXiv:2403.15638) that operates entirely at inference time. What is the threat model that PMixED addresses that DP-SGD does not?

5. Sketch the DP Reversal via LLM Feedback attack. Design a countermeasure that limits confidence-score leakage and estimate its deployment cost.

• Abadi et al.: DP-SGD (arXiv:1607.00133): the standard DP training algorithm
• Carlini et al.: Extracting Training Data (arXiv:2012.07805): the canonical extraction paper
• Duan et al.: Canary MIA on LLMs (arXiv:2402.07841, 2024): limited-success MIA
• Kowalczyk et al.: Auditing DP for LLMs (arXiv:2503.06808, March 2025): resolution of the tension
• PMixED (arXiv:2403.15638): inference-time private prediction