Fine-Tuning Small Language Models

Fine-tuning small models means taking a pretrained language model in the roughly 0.5B to 8B parameter range and continuing to train it on your own labeled examples so it learns your task, format, or domain. For most engineering teams this is cheaper, faster, and more private than calling a frontier API, and a small model that has seen a few thousand of your examples often beats a much larger general model on the one narrow job you care about. This guide walks through the whole loop: picking a base model, building a dataset, running a LoRA or QLoRA training job on a single GPU, evaluating honestly, and serving the result.

Why fine-tuning small models beats reaching for a bigger one

The instinct when a model gets something wrong is to reach for a larger one. That works, but it has a cost you pay on every single request: latency, dollars per token, and sending your data to someone else's servers. Fine-tuning small models flips the economics. You pay the training cost once, then run inference on hardware you control.

Small models win in a few concrete situations:

• The task is narrow and repetitive: classify a support ticket, extract fields from an invoice, rewrite text into a fixed house style, route a query to a tool.
• You have or can generate a few hundred to a few thousand clean examples.
• You need low latency or high throughput, so a 3B model returning in 200ms beats a 200B model returning in 3s.
• Data cannot leave your infrastructure for compliance reasons.

Small models lose when the task needs broad world knowledge, long multi-step reasoning, or open-ended generation across many domains. Do not fine-tune a 1B model and expect it to replace a frontier model as a general assistant. Match the tool to the job.

A useful rule: if you can write a clear rubric for what a correct output looks like, and you can produce examples that follow that rubric, the task is a good fit for fine-tuning small models.

Full fine-tuning vs LoRA vs QLoRA

There are three training approaches you will actually consider. Full fine-tuning updates every weight in the model. It gives the most capacity to change behavior but needs enough GPU memory to hold the model, its gradients, and optimizer states, which is roughly four times the model size in bytes just for the optimizer with Adam. For a 7B model in bf16 that pushes you past a single consumer GPU fast.

LoRA (Low-Rank Adaptation) freezes the original weights and trains small rank-decomposition matrices that get added to specific layers. You end up training well under 1 percent of the parameters. The base model stays untouched on disk, and your trained output is a small adapter file, often tens of megabytes. You can keep many adapters for one base model and swap them per request.

QLoRA is LoRA on top of a base model quantized to 4-bit. The frozen base is loaded in 4-bit precision to slash memory, while the LoRA adapters train in higher precision. This is what lets you fine-tune a 7B or 8B model on a single 16GB to 24GB GPU. For most teams doing fine-tuning of small models on one GPU, QLoRA is the default starting point.

Quick decision guide:

• Under 3B params and you have a 24GB+ GPU: plain LoRA in bf16 is simple and fast.
• 7B to 8B on a single 16GB to 24GB GPU: use QLoRA.
• You have multiple A100/H100 GPUs and need maximum quality with no adapter merge step: full fine-tuning is on the table.

Set up the environment

You need a recent Python, PyTorch with CUDA, and the Hugging Face stack. Create a clean virtual environment first so you do not fight version conflicts later.

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install torch transformers datasets accelerate peft bitsandbytes trl

Verify the GPU is visible before you waste time on a training run that will fall back to CPU:

python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

If that prints False, stop and fix your CUDA install. Training small models on CPU is technically possible and practically useless for anything past a toy example.

Build the dataset

Data quality decides everything. A thousand clean, consistent examples beat ten thousand noisy ones. Before writing any training code, decide on one exact input and output format and enforce it across every row.

The most common format is a chat-style JSONL file where each line is a conversation. Store it so every example teaches the model the same shape of behavior.

{"messages": [{"role": "system", "content": "You classify support tickets into one of: billing, technical, account, other. Reply with only the label."}, {"role": "user", "content": "I was charged twice for my subscription this month."}, {"role": "assistant", "content": "billing"}]}
{"messages": [{"role": "system", "content": "You classify support tickets into one of: billing, technical, account, other. Reply with only the label."}, {"role": "user", "content": "The app crashes every time I open the reports page."}, {"role": "assistant", "content": "technical"}]}

A few rules that save you from garbage results:

• Keep the system prompt identical across examples if you plan to use the same one at inference. The model learns the mapping in context.
• Make outputs deterministic and minimal. For classification, output the bare label, not a sentence. Every extra token is something the model can get wrong.
• Cover the edge cases you actually see in production, including the "other" or "none" case. Models that never saw a negative example will hallucinate a positive one.
• Split off a held-out test set before you train, and never let those examples leak into training. A 90/10 split is a fine default.

Load and split with the datasets library:

from datasets import load_dataset

data = load_dataset("json", data_files="tickets.jsonl", split="train")
data = data.train_test_split(test_size=0.1, seed=42)
train_ds, eval_ds = data["train"], data["test"]
print(len(train_ds), len(eval_ds))

If you are generating synthetic training data with a larger model, still review a sample by hand. Synthetic data drifts toward a house style that may not match your real inputs, and it happily reproduces any bias in your generation prompt.

Fine-tune with QLoRA: a runnable script

Here is a complete QLoRA training script using the TRL library's SFTTrainer, which handles chat templating and loss masking for you. It targets a 7B-class base model but works for smaller ones by changing the model id.

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

base_model = "Qwen/Qwen2.5-3B-Instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(base_model)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)

data = load_dataset("json", data_files="tickets.jsonl", split="train")
data = data.train_test_split(test_size=0.1, seed=42)

sft_config = SFTConfig(
    output_dir="./ticket-classifier",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    logging_steps=10,
    eval_strategy="epoch",
    save_strategy="epoch",
    bf16=True,
    max_length=1024,
    gradient_checkpointing=True,
)

trainer = SFTTrainer(
    model=model,
    args=sft_config,
    train_dataset=data["train"],
    eval_dataset=data["test"],
    peft_config=lora_config,
    processing_class=tokenizer,
)

trainer.train()
trainer.save_model("./ticket-classifier/final")

Run it and watch the loss. A healthy run shows training loss dropping steadily and evaluation loss dropping then flattening. If evaluation loss starts climbing while training loss keeps falling, you are overfitting, so reduce epochs or add more data.

What the hyperparameters actually do

You do not need to tune all of these, but you should understand the ones that matter when a run goes wrong.

• r (LoRA rank): the capacity of the adapter. Start at 16. Raise to 32 or 64 if the model is underfitting a complex task, but higher rank costs memory and can overfit small datasets.
• lora_alpha: a scaling factor for the adapter's contribution. A common convention is alpha equals two times r. It interacts with learning rate, so if you double alpha you may want to nudge the learning rate down.
• target_modules: which layers get adapters. Targeting all attention and MLP projections, as above, is the strong default. Targeting only q_proj and v_proj is lighter but weaker.
• learning_rate: 2e-4 is a reliable starting point for LoRA. Full fine-tuning needs a much lower rate, often 1e-5 to 5e-5, because you are moving every weight.
• num_train_epochs: for a few thousand examples, 2 to 4 epochs is typical. More than that on a small dataset invites memorization.
• per_device_train_batch_size and gradient_accumulation_steps: their product is your effective batch size. If you hit an out-of-memory error, lower the batch size and raise accumulation to keep the effective batch the same.
• gradient_checkpointing: trades compute for memory by recomputing activations in the backward pass. Turn it on when memory is tight, off when you have room and want speed.

When you get an out-of-memory error, work through this order: lower batch size, enable gradient checkpointing, shorten max_length, then switch from LoRA to QLoRA, then pick a smaller base model. Do not jump straight to a smaller model, since sequence length and batch size are usually the real culprits.

Merge and run inference

After training you have an adapter, not a standalone model. For inference you can either load the base model plus adapter, or merge them into one set of weights. Loading the adapter separately keeps flexibility; merging simplifies deployment.

Load base plus adapter:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = "Qwen/Qwen2.5-3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "./ticket-classifier/final")
model.eval()

messages = [
    {"role": "system", "content": "You classify support tickets into one of: billing, technical, account, other. Reply with only the label."},
    {"role": "user", "content": "My password reset email never arrives."},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=8, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Note do_sample=False for a classification task. You want the deterministic argmax output, not a sampled one. Save temperature and sampling for creative generation.

To merge for a simpler deployment:

from peft import PeftModel
from transformers import AutoModelForCausalLM
import torch

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", torch_dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(base, "./ticket-classifier/final").merge_and_unload()
merged.save_pretrained("./ticket-classifier/merged")

One caveat: do not merge a QLoRA adapter back into a 4-bit base. Load the base in bf16 for the merge, apply the adapter, then save. Merging into the quantized weights degrades quality.

Evaluate honestly

Loss curves tell you the model is learning something, not that it is doing your task well. Evaluate on the held-out set with the metric that matches the job. For classification that is accuracy and a per-class breakdown, since an 88 percent overall accuracy can hide a class the model never gets right.

from sklearn.metrics import classification_report

def predict(messages):
    inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
    out = model.generate(inputs, max_new_tokens=8, do_sample=False)
    return tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True).strip()

y_true, y_pred = [], []
for ex in eval_ds:
    msgs = ex["messages"][:-1]
    gold = ex["messages"][-1]["content"]
    y_true.append(gold)
    y_pred.append(predict(msgs))

print(classification_report(y_true, y_pred))

Compare three numbers before you declare victory: the base model with no fine-tuning, the fine-tuned model, and a strong prompted frontier model if that is your alternative. The base model number tells you how much the fine-tune actually added. If fine-tuning only moves accuracy a couple of points, your data may be too small or too noisy, or the base model may already be good enough to just prompt.

For generation tasks where there is no single right answer, use a held-out set scored by a rubric, ideally with a second model acting as a judge against explicit criteria, plus a manual read of a sample. Never ship on training-loss alone.

Deploy the fine-tuned model

For real traffic, load the merged model in an inference server built for throughput rather than calling generate in a loop. Servers like vLLM or Text Generation Inference batch requests and manage GPU memory far better than a hand-rolled loop.

A minimal vLLM launch against your merged model:

pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model ./ticket-classifier/merged \
  --max-model-len 2048 \
  --dtype bfloat16

That exposes an OpenAI-compatible endpoint, so your existing client code that talks to an OpenAI-style API mostly just needs a new base URL and model name. Keep the same system prompt you trained with, or the model will underperform.

Operational notes worth building in from day one:

• Version your datasets and adapters together. When quality drifts, you need to know exactly which data produced which model.
• Log a sample of live inputs and outputs so you can build the next training set from real traffic, which is the highest-value data you will ever get.
• Watch for input drift. A classifier trained on last quarter's tickets will slowly go stale as products and phrasing change. Plan to retrain, not to train once.

A realistic end-to-end workflow

Putting it together, a first fine-tuning project on small models looks like this. Collect 500 to 2000 real examples and clean them into one consistent JSONL format. Hold out 10 percent. Pick a 3B instruct model as the base. Run QLoRA for 3 epochs with the script above. Evaluate against the base model and your prompted-API baseline. If the fine-tune wins clearly, merge, serve with vLLM, and start logging live traffic for the next round. If it does not win, the fix is almost always more or cleaner data, not a bigger rank or more epochs.

Resist two temptations. The first is jumping to a larger base model when the real problem is thin data. The second is training for many epochs to force the loss down, which produces a model that memorized your training set and fails on anything new. Fine-tuning small models rewards discipline in the dataset far more than cleverness in the config.

FAQ

How much data do I need to fine-tune a small model? For a narrow, well-defined task like classification or extraction, a few hundred clean examples can move the needle and a couple of thousand often gets you to production quality. Broad or nuanced tasks need more. Data quality and consistency matter more than raw count, so 500 carefully labeled examples beat 5000 noisy ones nearly every time.

Do I need a big GPU to fine-tune small models? No. QLoRA lets you fine-tune a 7B or 8B model on a single 16GB to 24GB GPU, and smaller 1B to 3B models train comfortably on that same hardware with plain LoRA. If you hit out-of-memory errors, lower the batch size, enable gradient checkpointing, and shorten the sequence length before considering a smaller model.

What is the difference between LoRA and QLoRA? LoRA freezes the base model and trains small low-rank adapter matrices, so you update a tiny fraction of parameters. QLoRA does the same but loads the frozen base model in 4-bit precision to cut memory use dramatically. Use QLoRA when memory is the constraint and plain LoRA when you have GPU headroom and want simpler, slightly faster training.

Will fine-tuning make the model forget its general abilities? Full fine-tuning can cause catastrophic forgetting, where the model loses general skills while learning your task. LoRA and QLoRA reduce this risk because the original weights stay frozen and only the adapter changes. Even so, keep your dataset focused on the target task, keep epochs modest, and evaluate on a held-out set to catch regressions early.

Should I fine-tune or just write a better prompt? Try prompting first. If a well-crafted prompt with a few examples on a capable model already hits your quality bar, you may not need to fine-tune at all. Reach for fine-tuning when prompting plateaus, when you need lower latency or cost at scale, when outputs must follow a strict format the model keeps breaking, or when data cannot leave your infrastructure.

How do I know if my fine-tune actually worked? Compare three numbers on a held-out test set the model never trained on: the base model without fine-tuning, your fine-tuned model, and your realistic alternative such as a prompted frontier API. The base model score shows how much the fine-tune added. If the gain is small, the answer is almost always more or cleaner training data rather than more epochs or a higher LoRA rank.

Can I run several fine-tuned tasks on one base model? Yes, and this is a real advantage of LoRA. Because each trained adapter is a small file that sits on top of the shared base model, you can keep many adapters for one base and load the right one per request. Some inference servers support hot-swapping adapters, which lets you serve multiple fine-tuned behaviors from a single GPU-resident base model.