Differentially Private Clustered Federated Learning with Privacy-Preserving Initialization and Normality-Based Aggregation

PINA separates initialization from aggregation to reduce the convergence delay caused by non-IID data in clustered Federated Learning with Differential Privacy (DP). It directly targets the problem where naive DP noise excessively perturbs client updates, breaking down cluster structure.

• In the initialization stage, LoRA (r=1) is attached to a pretrained Transformer, fine-tuning only the value projection matrix of the last attention layer. Each client keeps only the largest positive/negative entries of its update, zeroes out the rest, applies LDP, and sends it to the server, which then builds initial cluster models from these sparse sketches.
• In the training stage, clients identify their cluster via loss values, then train the selected cluster model on local data. Updates are clipped, noised, and aggregated via SecAgg-based Distributed DP (DDP); the server initially aggregates using the smallest ℓ2 norm, then later readjusts aggregation size based on normality estimated via the Shapiro-Wilk test statistic.

Experiments assumed 10,000 clients under ϵ = 2, 8 conditions on rotated CIFAR-10 (C = 2), rotated FMNIST (C = 4), and FEMNIST (C = 2). A 22M-parameter ViT-Small pretrained on ImageNet-21k was compared against FedAvg, FedProx, FedNova, IFCA, with PINA outperforming SOTA on all three datasets.

Notably, at ϵ = 2, PINA showed accuracy improvements of up to 2.9%, 2.9%, and 3.1% on rotated CIFAR-10, rotated FMNIST, and FEMNIST respectively. Ablation studies showed that stronger privacy conditions actually helped preserve clustering structure better, while IFCA—which applies LDP throughout the entire training process—damaged the structure. The proposed initialization converged to optimal clustering far faster than random initialization. Overall, PINA achieved an average test accuracy improvement of 2.9%.