Low-Rank Adaptation (LoRA)

Understanding Its Purpose First

LoRA freezes the original weights and changes the model's behavior on the target task by training a small number of low-rank adaptation parameters.

Understanding with an Example

To make a model more consistently use a certain lesson plan format, you can prepare high-quality examples and fine-tune using LoRA. Compared to attaching a template in a conversation, this trains some parameters, but still requires data and compute.

Going a Step Deeper

Updates to large weight matrices can be represented as the product of two smaller matrices—this is the intuitive meaning of "low-rank." LoRA chooses to train these smaller matrices instead of performing full updates on all original weights. It can be combined with training objectives like supervised fine-tuning.

What It Does Not Explain

LoRA is a parameter adaptation method, not a new model architecture, nor a protocol for connecting models to external data sources. More efficient parameters do not mean you can pick any data and learn well; whether domain facts are accurate still requires retrieval or evaluation verification.

What to Learn Next

• Supervised Fine-Tuning (SFT)
• Parameters
• Retrieval-Augmented Generation (RAG)

References

• LoRA: Low-Rank Adaptation of Large Language Models: Freezes pretrained weights and trains low-rank matrices; a parameter-efficient adaptation method.

Reference verified on 2026-09-09; the original paper is used to explain the mechanism, and the examples in the text are for instructional purposes.