Preserving Privacy in AI Training Data

Models trained on sensitive medical, financial, and pharmaceutical data have training data itself as an asset. Back in 2023, Google DeepMind research already showed that GPT-3.5-turbo could regurgitate PII verbatim, and Amazon researchers reproduced three attacks targeting a single model, federated learning(FL) gradients, and a shared global model. They also verified where differential privacy(DP) and secure multiparty computation(MPC) each provide defense.

• membership inference uses differences in a model's confidence scores to determine whether a specific record was included in training. The attacker trains a proxy model on data mimicking the original distribution to find a cutoff, and in ResNet-50/ImageNet-1k experiments, 97% of the samples the attack identified as training data were indeed training data.
• DP-SGD trains by adding Gaussian noise to batch gradients. On EMNIST, accuracy was 78% at ε=1.5 and 82% at ε=3.0, both lower than the 90% of the model without DP, but showing a meaningful privacy-utility tradeoff.
• In FL, sample reconstruction was possible from local gradients alone. Through optimization that searches for the batch closest to the target gradient, a single-sample batch from EMNIST was reconstructed exactly, and 3 samples were recovered from a batch of size 7. However, when gradients were secret-shared and summed via MPC, as in PFL, the same attack failed.
• The shared global model attack used nB neurons in a ReLU preprocessing layer to decompose the average gradient into per-sample gradients. On EMNIST, most samples from other participants' local batches were reconstructed, but when DP-SGD(ε=1.5) was applied again to the global gradient, meaningful reconstruction disappeared.

Ultimately, MPC blocks information exchanged during training, while DP blocks information exposed by the final model, and using both technologies together is needed to realistically protect sensitive AI training data.