Fine-Tuning vs LoRA vs Full Fine-Tuning: What's the Difference

You have a base model that is impressively general but frustratingly generic. It knows a bit about everything and not enough about your thing. So you decide to fine-tune it, and within about ten minutes of reading you run into a wall of terms: full fine-tuning, LoRA, QLoRA, PEFT, adapters, rank, quantization. Half the blog posts use "fine-tuning" and "LoRA" as if they were competing products, and the other half use them as if they were the same thing. Neither framing is correct, and the confusion costs people real money in wasted GPU hours and real time in failed experiments.

Here is the short version before we go deep. Fine-tuning is the umbrella. It is the general act of taking a pretrained model and training it further on your own data. Full fine-tuning and LoRA are two different methods of doing that fine-tuning. Full fine-tuning updates every weight in the model. LoRA updates a tiny set of extra weights and leaves the original model frozen. They are not rivals in the way people think. They are points on a spectrum that trades cost against control. This article walks through what each one actually does under the hood, when to pick which, how much hardware you really need, and the mistakes that quietly sabotage most first attempts.

What Fine-Tuning Actually Means

Fine-tuning is not a single algorithm. It is a category. When someone says "we fine-tuned the model," all they are telling you is that they took an existing pretrained model and continued training it on a narrower dataset to shift its behavior. That is it. The word says nothing about how many parameters they touched, what hardware they used, or whether they froze anything.

This matters because the base model you start with already knows an enormous amount. A modern language model has already read a large slice of the public internet, absorbed grammar, facts, reasoning patterns, and code. Pretraining is the expensive part that costs millions of dollars and months of compute. Fine-tuning is the cheap, targeted follow-up where you nudge that giant pile of knowledge toward your specific task: your writing style, your domain vocabulary, your support tickets, your classification labels.

Think of it like hiring a brilliant graduate. Pretraining is their entire education up to the day they walk in the door. Fine-tuning is the two weeks of onboarding where you teach them how your company does things. You are not re-educating them from scratch. You are specializing what they already have.

The key mental model to carry through the rest of this article is the spectrum. On one end you have full fine-tuning, which is maximally powerful and maximally expensive. On the other end you have lightweight methods like LoRA that are cheap and fast but touch far less of the model. Every technique we discuss lives somewhere on that line. Once you internalize the spectrum, the "versus" framing dissolves. You are not choosing a team. You are choosing a position on a tradeoff curve.

Full Fine-Tuning: The Heavyweight Approach

Full fine-tuning is the original, no-shortcuts method. You take the pretrained model and you make every single weight in it eligible for updating. During training, gradients flow through the entire network and adjust all of its parameters. Nothing is frozen. The whole model is clay again.

Conceptually this is the most powerful option because you are giving the optimizer the maximum degrees of freedom. If your task requires the model to genuinely shift its internal representations, full fine-tuning can do that in a way that lighter methods struggle to match. For deep domain adaptation, where the target data looks very different from anything in pretraining, this flexibility is a real advantage.

The problem is cost, and the cost is brutal. When you fine-tune all the weights, you have to store several things in GPU memory at once. You store the model weights themselves. You store the gradients, which are the same size as the weights. And if you use a common optimizer like Adam, you store two additional optimizer states per parameter, which is twice the size of the weights again. A rough rule of thumb is that full fine-tuning with Adam needs somewhere around four times the memory of the raw model just for the training state, before you even account for activations.

Here is what that looks like in practice for a mid-size model:

Model: 7 billion parameters, fp16
- Weights:            ~14 GB
- Gradients:          ~14 GB
- Optimizer (Adam):   ~28 GB (two states, fp32-ish)
- Activations:        several GB more, depends on batch and sequence
--------------------------------------------------------------
Total working set:    roughly 60+ GB before headroom

That is why full fine-tuning a 7B model comfortably wants an 80 GB data-center GPU, and larger models want several of them wired together with distributed training. You also produce a full-size copy of the model for every task you fine-tune, so ten specialized models means ten times the storage. For teams with the hardware and a genuinely demanding task, full fine-tuning remains the gold standard for quality. For everyone else, it is often overkill that never makes it past the budget conversation.

Enter LoRA: The Clever Shortcut

LoRA, short for Low-Rank Adaptation, came out of a simple and slightly rebellious question: do we actually need to update all the weights to get most of the benefit? The researchers behind it observed that the change a model undergoes during fine-tuning tends to have low "intrinsic rank." In plain language, the update the model needs is far simpler than the model itself, so you do not need a full-size update to express it.

Here is the core trick. Instead of modifying the big original weight matrix directly, LoRA freezes it completely and injects two small matrices alongside it. When a weight matrix would normally be a large square, LoRA approximates the change to it as the product of two skinny matrices: one that projects down to a small dimension called the rank, and one that projects back up. Because the rank is tiny, usually something like 8, 16, or 32, these two matrices together have a minuscule number of parameters compared to the original.

During training, only those two little matrices learn. The giant frozen weight is never touched. At inference time you can add the low-rank product back into the original weight and the model behaves as if it had been fine-tuned, with no extra latency.

The math is easier to feel than to describe:

Original weight W:  d x d       (say 4096 x 4096 = ~16.7M numbers, frozen)

LoRA replaces the UPDATE to W with:
    delta_W = B x A
    A: r x d        (r=16, so 16 x 4096  = 65,536 numbers)
    B: d x r        (4096 x 16           = 65,536 numbers)

Trainable per matrix: ~131K instead of ~16.7M
That is well under 1% of the original, and you never
store gradients or optimizer state for the frozen weight.

The payoff is dramatic. Because you only train a tiny fraction of the parameters, you only need gradients and optimizer states for that tiny fraction. Memory drops by an order of magnitude. Training gets faster. And the artifact you save at the end is not a full-size model, it is just the small adapter, often only a few megabytes. You can keep a single frozen base model on disk and swap in different LoRA adapters for different tasks, which is a massive operational win compared to storing many full copies.

LoRA is the best-known member of a family called PEFT, or Parameter-Efficient Fine-Tuning. The whole family shares the same philosophy: freeze most of the model, train a small add-on. LoRA just happens to be the variant that hit the sweet spot of simple, effective, and widely supported.

QLoRA and the Rest of the PEFT Family

Once LoRA proved you could freeze the base model, the natural next question was whether you could also shrink it. That is what QLoRA does. The Q stands for quantized. Instead of keeping the frozen base model in 16-bit precision, QLoRA loads it in 4-bit precision, which roughly quarters the memory the base model occupies. The LoRA adapters still train in higher precision on top, so you keep most of the quality while slashing the footprint further.

The combination is what put fine-tuning within reach of ordinary hardware. With QLoRA, models that used to demand a data-center GPU can be adapted on a single consumer card. A 7B model that needed 60-plus GB for full fine-tuning can be adapted with QLoRA in a fraction of that, because the base is compressed and only the adapters carry training state.

Rough memory comparison, 7B model, illustrative:

Full fine-tuning:   needs a big data-center GPU (80 GB class)
LoRA (fp16 base):   fits on a mid-range card, base in 16-bit
QLoRA (4-bit base): fits on a single consumer GPU, base compressed

The exact numbers shift with batch size, sequence length,
and library, but the ordering is always the same.

LoRA and QLoRA are not the only PEFT methods, though they dominate in practice. Other approaches include prefix tuning and prompt tuning, which learn a small set of virtual tokens that steer the model without touching its weights, and various adapter designs that insert small trainable modules between layers. There is also DoRA, a more recent refinement of LoRA that decomposes the weight update in a way that closes some of the quality gap with full fine-tuning. For most people starting out, though, the decision tree is refreshingly short: try LoRA, and reach for QLoRA when memory is tight.

Head to Head: How to Actually Choose

Now that the pieces are on the table, the real question is which one you should reach for. The honest default answer for most projects in 2026 is LoRA or QLoRA. Not because it is always the best, but because it captures most of the benefit at a tiny fraction of the cost and risk, and you can run it without begging for a hardware budget.

Reach for full fine-tuning when several of these are true at once:

• Your target domain is genuinely far from the base model's training data, for example a specialized scientific or legal corpus with unusual structure.
• You have the hardware, meaning access to large GPUs or a multi-GPU cluster, and the budget to run them.
• Quality is the overriding priority and small gains justify large costs.
• You are producing one flagship model rather than many task-specific variants.

Reach for LoRA when most of these hold:

• You are adapting the model to a style, a format, a tone, or a moderately specialized domain.
• You want to train on modest hardware and finish in hours, not days.
• You expect to maintain several specialized versions and want cheap, swappable adapters.
• You value being able to roll back or A/B test by simply loading a different small file.

Reach for QLoRA specifically when you like everything about LoRA but you are memory-constrained, such as fine-tuning a larger model on a single consumer GPU where even the 16-bit base would not fit.

There is one more option people forget in the rush to train something: do not fine-tune at all. A great deal of what people try to fix with fine-tuning is actually a knowledge problem, not a behavior problem, and knowledge problems are usually better solved with retrieval. If you want the model to answer questions about your constantly changing internal documents, retrieval-augmented generation will beat fine-tuning almost every time, because you can update the documents without retraining anything. Fine-tuning teaches behavior and form. Retrieval supplies facts. Confusing the two is the single most common strategic mistake in this whole space.

A Concrete Walkthrough With LoRA

To make this tangible, here is the shape of a LoRA fine-tuning run using the widely used Hugging Face stack. This is not a full script you should copy blindly, but it shows the pieces and how little configuration LoRA actually needs.

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = "some-open-base-model"
model = AutoModelForCausalLM.from_pretrained(base_model)
tokenizer = AutoTokenizer.from_pretrained(base_model)

# The heart of LoRA: a small config object.
lora_config = LoraConfig(
    r=16,                 # the rank, the single most important knob
    lora_alpha=32,        # scaling for the update, often ~2x the rank
    lora_dropout=0.05,    # light regularization
    target_modules=[      # which layers get adapters
        "q_proj", "v_proj"
    ],
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)

# This line is the reality check. Print how much you are training.
model.print_trainable_parameters()
# Typical output: trainable params ~4M / all params ~7B (well under 1%)

That last line is worth staring at. It reports how many parameters are actually learning versus how many exist in total, and the first time you see the trainable share come out under one percent, the whole idea of LoRA clicks. You are teaching the model new behavior by adjusting a rounding error's worth of its weights.

A few knobs matter more than the rest. The rank, r, controls how much capacity the adapter has. Small ranks like 8 are cheap and often enough for style and format tasks. Larger ranks like 64 give the adapter more room to learn harder shifts, at more cost. The lora_alpha value scales how strongly the adapter's update is applied, and a common convention is to set it to roughly twice the rank. The target_modules choice decides which layers receive adapters. Attention projection layers are the usual starting point, and applying LoRA to more layers generally helps quality at the price of more trainable parameters. Start conservative, measure, then widen only if the results demand it.

The Mistakes That Wreck Fine-Tuning Runs

Most failed fine-tuning attempts do not fail because someone picked LoRA over full fine-tuning or vice versa. They fail for boringly consistent reasons that have nothing to do with the method.

The first and biggest is bad data. Fine-tuning is extraordinarily sensitive to the quality and format of your training examples. A few hundred clean, consistent, well-formatted examples will outperform tens of thousands of noisy ones. If your examples contradict each other, or the formatting drifts, or the label distribution is skewed, the model faithfully learns the mess. Garbage in, garbage out is not a cliche here, it is the primary failure mode. Spend the majority of your effort on the dataset, not the hyperparameters.

The second is trying to inject knowledge that belongs in retrieval. As said above, if the thing you want is up-to-date facts, fine-tuning is the wrong tool. You will burn compute, and worse, you may teach the model to confidently state facts that were true on training day and are stale now.

The third is catastrophic forgetting. When you push a model hard on a narrow task, especially with full fine-tuning or a high learning rate, it can lose general capabilities it used to have. It gets great at your niche and mysteriously worse at everything else. LoRA is somewhat more resistant to this because the base weights stay frozen, but it is not immune. Keeping learning rates modest and not over-training helps a lot.

The fourth is overfitting through too many epochs. People assume more training is more better and run the data through many times. Past a point the model stops learning the task and starts memorizing the examples, and your validation performance quietly degrades even as training loss keeps dropping. Watch a held-out validation set, not just the training loss, and stop when validation stops improving.

The fifth is skipping evaluation entirely. A shocking number of teams fine-tune, eyeball a handful of outputs, declare victory, and ship. Without a proper evaluation set and a clear metric, you have no idea whether you improved anything or just changed it. Build the evaluation before you build the model.

Bringing It All Together

Step back and the landscape is simpler than the jargon suggests. Fine-tuning is the umbrella term for continuing to train a pretrained model on your own data. Under that umbrella, full fine-tuning updates every weight and delivers maximum control at maximum cost, wanting serious hardware and producing full-size models. LoRA freezes the original model and trains a tiny pair of low-rank matrices, capturing most of the benefit for a fraction of the memory, the time, and the storage. QLoRA goes further by compressing the frozen base to 4-bit so even large models fit on a single consumer GPU. And sometimes the right answer is to reach for retrieval instead of training at all.

The practical playbook for most people is short. Start by asking whether your problem is behavior or knowledge. If it is knowledge, use retrieval. If it is behavior, default to LoRA, drop to QLoRA when memory is tight, and only graduate to full fine-tuning when you have proven that the cheaper method leaves real quality on the table and you have the hardware to chase it. Above all, obsess over your data and your evaluation, because those decide the outcome far more than the choice of method ever will.

If you want to go from understanding these ideas to actually building and shipping systems that use them, that is exactly the gap the AI Engineering Roadmap course on teachyou.ai is built to close. It walks you through fine-tuning, LoRA and QLoRA, retrieval pipelines, and evaluation in a hands-on, project-driven way, so the spectrum in this article turns into skills you can put to work. The models are ready to learn from you. This is where you learn to teach them.