Locking Pre-trained Weights via Deep Low-Rank Residual Distillation

DLR-Lock is a defense technique that makes it difficult for users to perform unauthorized fine-tuning or modifications for redistribution purposes, even after inspecting the weights of a pre-trained language model. Model providers replace each pre-trained MLP with a Deep Low-Rank Residual Network (DLR-Net) with a similar number of parameters.

DLR-Net is designed to incur higher costs during backpropagation than during inference. During training, activation memory increases proportionally with depth, and the backpropagation overhead becomes disproportionately larger than forward propagation, making standard fine-tuning difficult.

Key features include:

• Efficient training of DLR-Net via module-wise knowledge distillation
• Activation memory burden proportional to depth during backpropagation
• Increased optimization difficulty for standard fine-tuning due to structural mismatch
• Resilience against adaptive attackers who know the full defense strategy
• Preservation of the original model's inference performance and functionality

Researchers evaluated through LLM experiments that DLR-Lock operates effectively not just as a simple structural defense, but also against attackers who fully analyze model weights and architecture.