What’s patient privacy for? The Hippocratic Oath, considered one in all the earliest and most generally known medical ethics texts on this planet, reads: “Whatever I see or hear within the lives of my patients, whether in reference to my skilled practice or not, which ought to not be spoken of out of doors, I’ll keep secret, as considering all such things to be private.”
As privacy becomes increasingly scarce within the age of data-hungry algorithms and cyberattacks, medicine is one in all the few remaining domains where confidentiality stays central to practice, enabling patients to trust their physicians with sensitive information.
But a paper co-authored by MIT researchers investigates how artificial intelligence models trained on de-identified electronic health records (EHRs) can memorize patient-specific information. The work, which was recently presented on the 2025 Conference on Neural Information Processing Systems (NeurIPS), recommends a rigorous testing setup to make sure targeted prompts cannot reveal information, emphasizing that leakage have to be evaluated in a health care context to find out whether it meaningfully compromises patient privacy.
Foundation models trained on EHRs should normally generalize knowledge to make higher predictions, drawing upon many patient records. But in “memorization,” the model draws upon a singular patient record to deliver its output, potentially violating patient privacy. Notably, foundation models are already known to be susceptible to data leakage.
“Knowledge in these high-capacity models could be a resource for a lot of communities, but adversarial attackers can prompt a model to extract information on training data,” says Sana Tonekaboni, a postdoc on the Eric and Wendy Schmidt Center on the Broad Institute of MIT and Harvard and first creator of the paper. Given the danger that foundation models could also memorize private data, she notes, “this work is a step towards ensuring there are practical evaluation steps our community can take before releasing models.”
To conduct research on the potential risk EHR foundation models could pose in medicine, Tonekaboni approached MIT Associate Professor Marzyeh Ghassemi, who’s a principal investigator on the Abdul Latif Jameel Clinic for Machine Learning in Health (Jameel Clinic), a member of the Computer Science and Artificial Intelligence Lab. Ghassemi, a college member within the MIT Department of Electrical Engineering and Computer Science and Institute for Medical Engineering and Science, runs the Healthy ML group, which focuses on robust machine learning in health.
Just how much information does a nasty actor need to reveal sensitive data, and what are the risks related to the leaked information? To evaluate this, the research team developed a series of tests that they hope will lay the groundwork for future privacy evaluations. These tests are designed to measure various forms of uncertainty, and assess their practical risk to patients by measuring various tiers of attack possibility.
“We actually tried to emphasise practicality here; if an attacker has to know the date and value of a dozen laboratory tests out of your record in an effort to extract information, there could be very little risk of harm. If I have already got access to that level of protected source data, why would I want to attack a big foundation model for more?” says Ghassemi.
With the inevitable digitization of medical records, data breaches have turn into more commonplace. Previously 24 months, the U.S. Department of Health and Human Services has recorded 747 data breaches of health information affecting greater than 500 individuals, with the bulk categorized as hacking/IT incidents.
Patients with unique conditions are especially vulnerable, given how easy it’s to select them out. “Even with de-identified data, it will depend on what sort of data you leak concerning the individual,” Tonekaboni says. “When you discover them, you already know rather a lot more.”
Of their structured tests, the researchers found that the more information the attacker has about a specific patient, the more likely the model is to leak information. They demonstrated find out how to distinguish model generalization cases from patient-level memorization, to properly assess privacy risk.
The paper also emphasized that some leaks are more harmful than others. As an illustration, a model revealing a patient’s age or demographics could possibly be characterised as a more benign leakage than the model revealing more sensitive information, like an HIV diagnosis or alcohol abuse.
The researchers note that patients with unique conditions are especially vulnerable given how easy it’s to select them out, which can require higher levels of protection. “Even with de-identified data, it really will depend on what sort of data you leak concerning the individual,” Tonekaboni says. The researchers plan to expand the work to turn into more interdisciplinary, adding clinicians and privacy experts in addition to legal experts.
“There’s a reason our health data is private,” Tonekaboni says. “There’s no reason for others to find out about it.”
This work supported by the Eric and Wendy Schmidt Center on the Broad Institute of MIT and Harvard, Wallenberg AI, the Knut and Alice Wallenberg Foundation, the U.S. National Science Foundation (NSF), a Gordon and Betty Moore Foundation award, a Google Research Scholar award, and the AI2050 Program at Schmidt Sciences. Resources utilized in preparing this research were provided, partly, by the Province of Ontario, the Government of Canada through CIFAR, and firms sponsoring the Vector Institute.
