Optimizing LoRA Target Module Selection for Efficient Fine-Tuning

LoRA freezes the original model weights and inserts small adapters into specific sublayers to reduce the cost of parameter updates. This means where the adapter is inserted simultaneously governs accuracy, latency, and inference cost.

Amazon conducted an ablation study based on Nova 2.0 Lite to find a standard target-module configuration that works across most customer use cases. Comparing qkv, o_proj, and fc1/fc2 alone or in combination, o_proj showed the best efficiency-accuracy balance among single modules.

In a Transformer block, attention consists of query/key/value (qkv) and o_proj, while the feed-forward layer is handled by fc1/fc2. Generally, applying LoRA to more modules improves performance, but training and inference costs increase accordingly.

The experiment was conducted across 7 datasets spanning text and image, reasoning and non-reasoning tasks. It examined generality across financial reasoning, long-document summarization, medical QA, JSON extraction, multimodal Q&A, and OCR, comparing SFT LoRA under identical hyperparameters.

The key trade-offs are as follows.

• qkv only: Lowest latency and a reasonable baseline, but not peak performance.
• o_proj only: Very efficient, also important for reasoning, and suitable for ultralow-latency environments.
• qkv + o_proj: Best balance of accuracy and latency for most NLP tasks.
• qkv + fc1/fc2: Delivers high accuracy close to near-full fine tuning, but at a larger latency cost.
• o_proj + fc1/fc2: Suitable for tasks like classification and sentiment analysis where qkv isn't necessary.
• All modules: Delivers peak performance, but is usually inefficient in production due to latency.

Ultimately, for Nova 2.0 Lite, o_proj alone was identified as the best efficiency point, and qkv + o_proj was presented as the most practical choice for general-purpose production settings.