How Differential Privacy Works and Why It Matters for Real-World Data Collection

Differential privacy is one of the more mathematically rigorous approaches to a problem that organisations routinely handle poorly: how do you collect and analyse data about people’s behaviour while limiting the privacy risk to individuals? The problem is harder than it appears, because many intuitive privacy-preserving measures—anonymisation, aggregation, pseudonymisation—provide weaker protection than people assume, and breaches of those weak protections have repeatedly demonstrated that “anonymised” data is often re-identifiable.

Differential privacy provides a formal mathematical guarantee rather than an informal assurance, which makes it unusual among privacy techniques and is the source of both its power and its practical constraints.

The Core Problem: Why Aggregation Isn’t Enough

The intuition behind data anonymisation is that if you remove or obscure personally identifying information from a dataset, the remaining data can’t be traced back to individuals. This intuition has been repeatedly invalidated by research and real-world re-identification attacks.

In 2006, Netflix published a “anonymised” dataset of movie ratings for a machine learning competition. Researchers subsequently de-anonymised a substantial fraction of the dataset by correlating it with IMDb reviews—the combination of ratings and their timestamps was sufficiently distinctive to uniquely identify individuals even without names or account IDs. In 2019, a study re-identified 99.98% of individuals in an “anonymised” location mobility dataset using only four spatiotemporal points from each trajectory.

The mathematical reason anonymisation is brittle: most people’s data records are highly distinctive even without obvious identifiers. The combination of age, gender, zip code, and one or two behavioural characteristics uniquely identifies a large fraction of the population. Datasets that appear to have removed identifying information still contain enough unique combinations to allow re-identification against auxiliary datasets.

Aggregate statistics have a related vulnerability: given sufficiently many queries about aggregate data from a dataset, an attacker can reconstruct information about individuals through differencing attacks. If you can ask “how many people in this dataset have property X?” and also “how many people in this dataset minus person P have property X?”, the difference reveals person P’s status on property X. Even without direct access to individual records, query access to aggregate statistics can leak individual information.

The Differential Privacy Definition

Differential privacy provides a formal definition of what it means for an analysis mechanism to be “private.” Informally: an algorithm is differentially private if its output looks essentially the same whether or not any particular individual’s data is included in the input dataset. An observer who sees only the output cannot meaningfully infer whether any specific person’s data was included.

More precisely, an algorithm M satisfies ε-differential privacy if for any two datasets D₁ and D₂ that differ in exactly one individual’s record, and for any possible output S:

Pr[M(D₁) ∈ S] ≤ e^ε × Pr[M(D₂) ∈ S]

The parameter ε (epsilon) is the “privacy budget”—it controls the privacy-utility trade-off. A smaller ε provides stronger privacy (the output distributions for D₁ and D₂ are more similar) but typically requires more noise to achieve, which reduces the accuracy of the analysis. A larger ε allows more accurate analysis at the cost of weaker privacy guarantees.

The key mechanism for achieving differential privacy is adding carefully calibrated random noise to the output of an analysis. The noise is drawn from a distribution (typically Laplace or Gaussian) scaled to the “sensitivity” of the query—how much any individual’s data could affect the query answer—and to the desired privacy budget ε.

The Two Models: Local and Central Differential Privacy

Differential privacy can be applied at two different points in the data pipeline, with different trust assumptions and different utility characteristics.

Central differential privacy assumes a trusted curator who receives raw data and applies the privacy-preserving analysis. The noise is added to the final output, not to individual data records. The curator must be trusted with the raw data; the privacy guarantee applies to what is released externally. This model can achieve high utility because the noise is added only once to the aggregate, not to each individual record.

Local differential privacy applies the noise at the individual level, before data leaves the user’s device. Each individual adds randomisation to their own data before submitting it; the data collector never sees the true individual-level values. This eliminates the need to trust the data collector—privacy is preserved even if the collector is malicious—but requires substantially more noise to achieve the same privacy guarantee, which reduces accuracy for the same dataset size.

Apple’s use of differential privacy in iOS for collecting keyboard usage statistics and emoji usage patterns uses the local model: the device randomises the data before sending it to Apple’s servers. Apple gets useful aggregate statistics about what users are typing; individual users’ private keystrokes are not revealed. Google’s use of differential privacy in Chrome’s RAPPOR system for collecting browser configuration statistics uses a similar local approach.

Real-World Deployments

Differential privacy has moved from academic theory to production deployment at several major technology companies and government agencies over the past decade:

Apple began using local differential privacy in iOS 10 (2016) for collecting usage statistics about keyboard and emoji usage, Health app features, and Safari browsing. The implementation details—including the ε values used—were published and subject to academic scrutiny, though the choice of ε has been debated in the research community.

Google has used differential privacy in multiple products, including Chrome usage statistics (RAPPOR), Google Maps traffic patterns, and the Google Analytics feature set. Their open-source differential privacy library has been widely adopted.

The US Census Bureau applied differential privacy to the 2020 Census data, introducing randomisation to protect individual records before publication of detailed geographic data. This was the largest-scale government deployment of differential privacy and generated significant discussion about the privacy-accuracy trade-off, particularly for small geographic areas and small populations where the noise had proportionally larger effects on count accuracy.

The Privacy Budget and Its Limitations

The privacy budget ε is composable: if you run k analyses each with budget ε, the total privacy cost is kε. This means that answering many queries about a dataset consumes the privacy budget quickly, and once the budget is exhausted, further analysis can no longer maintain the privacy guarantee. This creates a challenge for interactive data analysis where the number of queries isn’t known in advance.

The choice of ε in practice is difficult and somewhat arbitrary. The mathematical definition provides a bound on privacy loss, but translating that bound into “this is safe enough for this application” requires judgment about the adversary’s capabilities and the sensitivity of the data. Research on what ε values are appropriate for different applications is ongoing; current practice uses values ranging from 0.1 to 10 depending on the application, with little consensus on what constitutes appropriate calibration for real-world deployment.

Differential privacy also makes certain types of analysis difficult or impossible at acceptable accuracy. Fine-grained statistics about small subpopulations require large amounts of noise to achieve meaningful privacy guarantees, which can make the statistics so noisy as to be useless. The 2020 Census differential privacy deployment created significant accuracy problems for small-population groups and geographies that required rebalancing the privacy parameters.

Why It Still Matters

Despite its limitations, differential privacy represents something rare in privacy technology: a formal guarantee rather than a heuristic. Most privacy-preserving techniques—k-anonymity, data masking, aggregation—provide informal assurances that have been repeatedly broken. Differential privacy’s mathematical foundation means that when the guarantee holds, it holds provably, and the privacy loss is bounded rather than unknown.

For organisations collecting sensitive data at scale, differential privacy provides a principled framework for answering the question “how private is this analysis?”—a question that most data practices can’t answer with precision. As regulatory pressure on data collection increases and the failure modes of weaker anonymisation techniques continue to be exposed, differential privacy’s role in responsible data collection practices is likely to grow.