Is it safe to share models that were trained on sensitive data? This is the question I investigate in my dissertation, in the context of federated learning. To answer it, I went through the literature to find out which attacks can reveal sensitive information about a model’s training data. Three of the most prominent are:

Membership inference asks a simple question: was this particular person’s data used to train the model? [1] If a hospital trains a diagnostic model on patients with a specific condition, learning that someone’s record was in the training set can reveal that they have that condition.

Reconstruction attacks go further. Instead of answering a yes/no question, the attacker tries to recover the training data itself, whether that’s attributes of a record, an image, or a piece of text. [2, 3] When it works, it is the most damaging of the three.

Property inference isn’t about any single person. It targets the dataset as a whole, for example the share of a demographic group in the training data, or a characteristic an organization never meant to disclose. [4, 5] The risk here is to confidentiality rather than individual privacy.

Why study them?

First, regulation and trust depend on it. Laws like the GDPR require organizations to protect personal data, and “the data never left the building” is a weak argument if the model itself can give it away.

Second, these risks are easy to over- or underestimate. Many attacks in the literature assume an attacker with access and knowledge that real deployments rarely grant, a mismatch that a survey of 271 industry practitioners also found [6]. Others are more practical than people expect. Without studying them carefully, organizations either spend effort defending against threats that are unlikely or miss the ones that matter.

Third, defenses cost something. Protecting a model usually means giving up accuracy, computing power, or development time, so decision-makers need to know which risks justify which costs.

Why I consider them in my PhD

My PhD focuses on privacy risks in federated learning, where data stays local and only models are shared. That setup is often described as privacy-friendly, but the models exchanged during training are exactly what these three attacks exploit. Membership, property, and reconstruction attacks cover the range of what could leak, from a single person’s presence to a whole dataset’s statistics to the raw data itself.

I’m also asking the people who work with sensitive data which of these threats they consider realistic, and how much leakage they would accept. That way the attacks I study reflect real-world concerns and not only what is technically possible in a lab.

References

[1] Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017). Membership Inference Attacks Against Machine Learning Models. IEEE Symposium on Security and Privacy, 3–18.

[2] Fredrikson, M., Jha, S., & Ristenpart, T. (2015). Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures. ACM CCS, 1322–1333.

[3] Zhu, L., Liu, Z., & Han, S. (2019). Deep Leakage from Gradients. NeurIPS 32, 14747–14756.

[4] Ganju, K., Wang, Q., Yang, W., Gunter, C. A., & Borisov, N. (2018). Property Inference Attacks on Fully Connected Neural Networks Using Permutation Invariant Representations. ACM CCS, 619–633.

[5] Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019). Exploiting Unintended Feature Leakage in Collaborative Learning. IEEE Symposium on Security and Privacy, 691–706.

[6] Grosse, K., et al. (2024). Towards More Practical Threat Models in Artificial Intelligence Security. USENIX Security Symposium.