Imagine that a team of scientists has developed a machine-learning model that may predict whether a patient has cancer from lung scan images. They wish to share this model with hospitals around the globe so clinicians can start using it in diagnosis.

But there’s an issue. To show their model the right way to predict cancer, they showed it hundreds of thousands of real lung scan images, a process called training. Those sensitive data, which at the moment are encoded into the inner workings of the model, could potentially be extracted by a malicious agent. The scientists can prevent this by adding noise, or more generic randomness, to the model that makes it harder for an adversary to guess the unique data. Nevertheless, perturbation reduces a model’s accuracy, so the less noise one can add, the higher.

MIT researchers have developed a method that permits the user to potentially add the least amount of noise possible, while still ensuring the sensitive data are protected.

The researchers created a latest privacy metric, which they call Probably Roughly Correct (PAC) Privacy, and built a framework based on this metric that may robotically determine the minimal amount of noise that should be added. Furthermore, this framework doesn’t need knowledge of the inner workings of a model or its training process, which makes it easier to make use of for various kinds of models and applications.

In several cases, the researchers show that the quantity of noise required to guard sensitive data from adversaries is way less with PAC Privacy than with other approaches. This might help engineers create machine-learning models that provably hide training data, while maintaining accuracy in real-world settings.

“PAC Privacy exploits the uncertainty or entropy of the sensitive data in a meaningful way,  and this enables us so as to add, in lots of cases, an order of magnitude less noise. This framework allows us to know the characteristics of arbitrary data processing and privatize it robotically without artificial modifications. While we’re within the early days and we’re doing easy examples, we’re excited concerning the promise of this method,” says Srini Devadas, the Edwin Sibley Webster Professor of Electrical Engineering and co-author of a latest paper on PAC Privacy.

Devadas wrote the paper with lead creator Hanshen Xiao, an electrical engineering and computer science graduate student. The research can be presented on the International Cryptography Conference (Crypto 2023).

Defining privacy

A fundamental query in data privacy is: How much sensitive data could an adversary get better from a machine-learning model with noise added to it?

Differential Privacy, one popular privacy definition, says privacy is achieved if an adversary who observes the released model cannot infer whether an arbitrary individual’s data is used for the training processing. But provably stopping an adversary from distinguishing data usage often requires large amounts of noise to obscure it. This noise reduces the model’s accuracy.

PAC Privacy looks at the issue a bit in a different way. It characterizes how hard it will be for an adversary to reconstruct any a part of randomly sampled or generated sensitive data after noise has been added, moderately than only specializing in the distinguishability problem.

For example, if the sensitive data are images of human faces, differential privacy would concentrate on whether the adversary can tell if someone’s face was within the dataset. PAC Privacy, however, could have a look at whether an adversary could extract a silhouette — an approximation — that somebody could recognize as a specific individual’s face.

Once they established the definition of PAC Privacy, the researchers created an algorithm that robotically tells the user how much noise so as to add to a model to forestall an adversary from confidently reconstructing a detailed approximation of the sensitive data. This algorithm guarantees privacy even when the adversary has infinite computing power, Xiao says.

To seek out the optimal amount of noise, the PAC Privacy algorithm relies on the uncertainty, or entropy, in the unique data from the perspective of the adversary.

This automatic technique takes samples randomly from an information distribution or a big data pool and runs the user’s machine-learning training algorithm on that subsampled data to supply an output learned model. It does this repeatedly on different subsamplings and compares the variance across all outputs. This variance determines how much noise one must add — a smaller variance means less noise is required.

Algorithm benefits

Different from other privacy approaches, the PAC Privacy algorithm doesn’t need knowledge of the inner workings of a model, or the training process.

When implementing PAC Privacy, a user can specify their desired level of confidence on the outset. For example, perhaps the user wants a guarantee that an adversary won’t be greater than 1 percent confident that they’ve successfully reconstructed the sensitive data to inside 5 percent of its actual value. The PAC Privacy algorithm robotically tells the user the optimal amount of noise that should be added to the output model before it’s shared publicly, with the intention to achieve those goals.

“The noise is perfect, within the sense that should you add lower than we let you know, all bets may very well be off. However the effect of adding noise to neural network parameters is complicated, and we’re making no guarantees on the utility drop the model may experience with the added noise,” Xiao says.

This points to 1 limitation of PAC Privacy — the technique doesn’t tell the user how much accuracy the model will lose once the noise is added. PAC Privacy also involves repeatedly training a machine-learning model on many subsamplings of knowledge, so it may well be computationally expensive.  

To enhance PAC Privacy, one approach is to change a user’s machine-learning training process so it’s more stable, meaning that the output model it produces doesn’t change very much when the input data is subsampled from an information pool.  This stability would create smaller variances between subsample outputs, so not only would the PAC Privacy algorithm should be run fewer times to discover the optimal amount of noise, however it would also must add less noise.

An additional benefit of stabler models is that they often have less generalization error, which implies they’ll make more accurate predictions on previously unseen data, a win-win situation between machine learning and privacy, Devadas adds.

“In the following few years, we’d like to look somewhat deeper into this relationship between stability and privacy, and the connection between privacy and generalization error. We’re knocking on a door here, however it shouldn’t be clear yet where the door leads,” he says.

This research is funded, partially, by DSTA Singapore, Cisco Systems, Capital One, and a MathWorks Fellowship.