LLMs Can Infer What Users Never Disclosed, Outpacing Privacy Law

The usual framing of AI and privacy centers on data collection: what companies gather, how long they keep it, who can access it. Writing for Tech Policy Press, Ikenna Ogbogu, an undergraduate at Harvard studying Computer Science and Economics, argues that this framing misses the deeper problem. Foundation models do not just store what you share. They infer what you never shared at all.

The mechanism is not exotic. These systems learn statistical representations of behavior at scale, enabling reliable predictions about mental health status, political affiliation, and income level derived from aggregated behavioral data that looks, on its surface, like nothing sensitive. The article illustrates this with a 2024 study in which researchers prompted GPT-4 to identify where a Reddit user lived. The model correctly placed the user in Melbourne, Australia, based on a post mentioning "hook turn," without any explicit location disclosure.

The legal architecture built to address privacy was not designed for this. Laws like CCPA rest on the assumption that individuals can anticipate how their data will be used, and offer remedies including access, correction, and deletion. Those remedies presuppose information lives in discrete records. When sensitive attributes are encoded across a model's internal representations rather than in a database row, deletion requests become ambiguous. Model editing research, the piece notes, shows that supposedly removed information leaves "recoverable traces" in the weights. Compounding this: these inferential capabilities may be unknown even to the developers of the models themselves.

Ogbogu proposes three policy directions: expanding the definition of covered data to include inferred and probabilistic attributes; requiring audits of inference capabilities before deployment, which he calls capability-rooted governance; and mandating public impact assessments disclosing what sensitive attributes a system can infer. The honest caveat is that the piece offers a sharp diagnosis more than a treatment plan, and does not address what accuracy thresholds would trigger oversight or how audits would be scoped in practice.

For developers and compliance teams building on foundation models, the forward-looking implication is plain: audit what your system can infer, not only what it was trained on. The market for inference-auditing tools does not really exist yet. If capability-rooted governance gains regulatory traction, that gap closes quickly.