Fine-Tuning, RAG or Prompting: How an Architect Decides

Fine-tuning is the most over-prescribed remedy in enterprise AI. When teams weigh fine-tuning vs RAG vs prompting, most of the gaps they try to train away are instruction, knowledge or rules problems, and those are cheaper to fix, audit and roll back outside the weights. This article covers how to tell the difference, and what a governed adaptation lifecycle looks like when tuning is genuinely justified.

12.0 Architect-level mental model

The first question is not:

“How do we fine-tune the model?”

It is:

“Do we need to modify the model at all?”

There are three fundamentally different levers:

             APPLICATION NEED
                    |
                    v
        What is actually missing?
                    |
       +------------+-------------+
       |            |             |
       v            v             v
 Instructions   Knowledge      Behaviour
       |            |             |
       v            v             v
   PROMPTING       RAG       FINE-TUNING

Think of them this way:

Technique	Changes	Best for
Prompting	Instructions given at runtime	Task definition, constraints, output format
RAG	Information available at runtime	Private/current/domain knowledge
Fine-tuning	Model's learned behaviour	Repeated patterns, style, task behaviour, specialised capability

And they are not mutually exclusive.

A sophisticated enterprise system might do:

Fine-tuned model
      +
System prompt
      +
RAG context
      +
Tools

The mistake is treating fine-tuning as a substitute for all of them.

---

12.1 Prompting vs RAG vs fine-tuning: which should you use?

This distinction should be automatic.

---

Prompting

Use prompting when the base model already has the capability but needs better instructions.

Example:

Base capability:
Model understands invoices.

Requirement:
Extract:
- invoice number
- date
- GST number
- total
- currency

Solution:
Prompt + structured schema

You probably do not need fine-tuning.

Prompting is generally:

• fastest
• cheapest to change
• easiest to test
• easiest to roll back
• lowest operational complexity

Rule

Start with prompting unless there is evidence prompting cannot reliably meet the requirement.

---

12.2 When is RAG the right lever?

Use RAG when the problem is missing or changing knowledge.

Example:

Question:
"What is our current employee travel reimbursement policy?"

The model doesn't know today's company policy.

Solution:
Retrieve current HR policy
        ↓
Supply it to model
        ↓
Generate grounded answer

Fine-tuning is usually the wrong solution because:

Policy changes tomorrow
        ↓
Retrain model?

That is operationally absurd.

With RAG:

Update document
     ↓
Re-index
     ↓
new knowledge immediately available

The same holds for domain vocabulary, misspellings and specialist terms, which teams often assume need a tuned model. A worked example of fixing them in the retrieval layer instead.

---

12.3 When is fine-tuning the right lever?

Fine-tuning is most useful when you repeatedly need the model to behave differently.

Examples:

• specialised classification
• domain-specific language patterns
• particular writing conventions
• structured outputs
• specialised extraction
• tool-selection behaviour
• consistent response style
• reducing verbose instructions
• teaching task-specific transformations
• adapting a smaller model to a narrow task

Example:

Input:
procurement request

Desired output:
enterprise-specific category + risk + approval class

If you have thousands of high-quality examples:

input → correct output

fine-tuning may substantially improve consistency.

---

12.4 The most important decision rule

Ask:

Is the problem instruction?

→ Prompt

Is the problem knowledge?

→ RAG

Is the problem behaviour?

→ Fine-tune

Is the problem deterministic execution?

→ Code / workflow / tool

This fourth option matters enormously.

Do not fine-tune a model to calculate tax rules that can be implemented deterministically.

---

12.5 Example decision

Suppose an insurance company wants an AI claims assistant.

Requirement 1

Speak professionally and return JSON.

Use:

Prompt / structured output

Requirement 2

Know the latest insurance policy wording.

Use:

RAG

Requirement 3

Classify claim narratives into the insurer's proprietary 150-category taxonomy.

Potentially:

Fine-tuning

Requirement 4

Calculate policy deductible.

Use:

Deterministic code/tool

Requirement 5

Process large numbers cheaply.

Potentially:

Fine-tune/distill smaller model

That is the architectural mindset.

---

12.6 Why do enterprises overuse fine-tuning?

Because it sounds like:

“We'll teach the AI our company.”

But fine-tuning is not a database update mechanism.

It creates several new responsibilities:

training data governance
dataset versioning
training infrastructure
evaluation
model registry
deployment
rollback
drift monitoring
retraining
security
licensing
cost

You have now created another ML lifecycle.

Therefore:

Fine-tuning should solve a measurable deficiency, not be an ideological requirement.

---

12.7 Supervised Fine-Tuning: SFT

Supervised fine-tuning is the most important adaptation technique to understand.

You start with a pretrained model:

Foundation model
       |
       v
General language capability

Then train it further using labelled examples:

Input X
   ↓
Expected output Y

Example:

Input:
"The supplier is requesting an advance payment
of 80% against an unverified bank account."

Target:
{
  "risk": "HIGH",
  "reasons": [
    "high advance percentage",
    "bank account unverified"
  ]
}

Repeat over thousands of examples.

The model adjusts its weights to increase the probability of producing outputs resembling the desired targets.

---

12.8 Training objective intuition

Suppose the desired output tokens are:

HIGH RISK

The model predicts probabilities:

LOW       0.20
MEDIUM    0.30
HIGH      0.50

Training penalises it when the correct token probability is insufficient.

Over examples, gradient descent changes parameters so that:

P(correct output | input)

increases.

You don't need the complete optimisation mathematics to make architecture decisions.

Know:

training data
    ↓
forward pass
    ↓
loss
    ↓
backpropagation
    ↓
gradient update
    ↓
new model parameters

---

12.9 Full fine-tuning

Traditional fine-tuning updates all or most model parameters.

Suppose:

Model = 70 billion parameters

Full fine-tuning can potentially update billions of values.

Advantages:

• maximum adaptation flexibility
• potentially strongest task adaptation

Disadvantages:

• huge GPU memory demand
• expensive
• slower
• large checkpoints
• operational complexity
• higher risk of damaging general model capability

This led to Parameter-Efficient Fine-Tuning, PEFT.

---

12.10 What is PEFT?

PEFT (parameter-efficient fine-tuning) adapts a model by training a small number of added or selected parameters while the base weights stay frozen.

PEFT means:

Adapt the model while training only a small fraction of its parameters.

Instead of:

70B model
↓
train 70B parameters

you might:

freeze base model
      +
train 0.1–2% additional/adaptation parameters

Conceptually:

              BASE MODEL
         mostly / fully frozen
                 |
           +-----+-----+
           |           |
           v           v
      Adapter     Adapter
       train       train

Benefits:

• lower compute requirements
• lower GPU memory
• smaller training artifacts
• faster iteration
• easier workload-specific variants

---

12.11 What is LoRA (Low-Rank Adaptation)?

This is the PEFT technique you absolutely need to understand. It was introduced in the original LoRA paper, which freezes the pretrained weights and trains small low-rank matrices injected into the transformer layers. (arXiv)

Suppose the model has a weight matrix:

W

Traditional fine-tuning updates it:

W' = W + ΔW

LoRA says:

Instead of learning the huge matrix ΔW, approximate it using two much smaller matrices:

ΔW = B × A

Therefore:

W' = W + B × A

where:

W = original model matrix

A = small trainable matrix
B = small trainable matrix

rank r << original matrix dimensions

---

12.12 LoRA intuition

Imagine:

Original weight matrix

10000 × 10000
= 100 million values

Instead of training another:

10000 × 10000

matrix, LoRA could approximate the update with:

A: 8 × 10000
B: 10000 × 8

Total:

160,000 trainable values

instead of:

100,000,000

The exact numbers vary, but the principle is:

The model adaptation may lie in a much lower-dimensional space than the entire model.

---

12.13 What gets trained under LoRA?

Typically:

Base weights
     ↓
frozen

LoRA matrices
     ↓
trained

At inference:

Base model
   +
LoRA adapter
   ↓
adapted behaviour

Potentially:

Base model
 ├── Finance LoRA
 ├── Legal LoRA
 ├── Healthcare LoRA
 └── Procurement LoRA

This can be useful operationally because you do not necessarily need a completely independent copy of the entire base model per domain.

---

12.14 LoRA rank

LoRA rank sets the size of the low-rank matrices, and with it how much the adapter can learn.

The parameter:

r

controls the low-rank dimension.

Simplistically:

lower r
→ fewer trainable parameters
→ cheaper
→ less expressive

higher r
→ more trainable parameters
→ potentially more adaptation
→ higher memory/compute

But higher rank does not automatically mean better quality.

Treat it as a hyperparameter and evaluate.

---

12.15 LoRA alpha

LoRA alpha is a scaling factor that controls how strongly the learned update is applied to the base weights.

You'll sometimes encounter:

LoRA alpha

It controls scaling of the LoRA update.

Conceptually:

W' = W + scaling × (B × A)

You don't need deep mathematical knowledge, but recognise:

rank
alpha
dropout
target modules
learning rate

as common LoRA tuning parameters.

---

12.16 What are target modules?

You don't necessarily attach LoRA everywhere.

For transformers, adaptation is often applied to parts of attention or other linear layers, such as matrices associated with:

Query
Key
Value
Output

or feed-forward components.

Architecture decision:

Which layers/modules require adaptation?

More target modules:

greater adaptation capacity
+
greater training cost

---

12.17 What is QLoRA?

QLoRA is LoRA applied on top of a quantized, frozen base model, which cuts the memory needed to fine-tune large models.

QLoRA combines:

Quantized base model + LoRA adaptation

The technique comes from the QLoRA paper, which fine-tunes LoRA adapters on top of a frozen 4-bit quantized base model to cut the memory needed for fine-tuning large models. (arXiv)

Conceptually:

Full precision model
       ↓
Quantize base model
       ↓
Keep base frozen
       ↓
Train LoRA adapters

For example:

Base weights
≈ 4-bit representation

LoRA parameters
trained at higher precision

Why?

Because the largest memory consumer is the base model itself.

Quantising it significantly lowers GPU memory requirements.

---

12.18 LoRA vs QLoRA

	LoRA	QLoRA
Base model	Usually higher precision	Quantized
Base weights trained?	No	No
Adapter trained?	Yes	Yes
GPU memory	Lower than full FT	Even lower
Training accessibility	Good	Better
Potential complexity	Moderate	More quantisation considerations

Mental model:

Full fine-tune
$$$$$$$$

LoRA
$$$

QLoRA
$$

Not exact economics, just the relative intuition.

---

12.19 Important QLoRA distinction

Do not confuse:

QLoRA

with merely serving a quantized model.

QLoRA is primarily a fine-tuning technique:

quantized frozen base
      +
trainable LoRA weights

Quantized inference is a separate serving topic.

---

12.20 Adapters

“Adapter” can be used broadly, but traditionally it means inserting small trainable modules into the network.

Conceptually:

Transformer layer
       |
       v
+----------------+
| frozen layer   |
+----------------+
       |
       v
+----------------+
| small adapter  | ← trained
+----------------+
       |
       v
next layer

Instead of changing all model parameters, you train these small modules.

LoRA is itself generally considered part of the broader PEFT family, although its mechanism modifies linear transformations through low-rank updates rather than simply inserting the classic adapter bottleneck.

---

12.21 PEFT architecture comparison

FULL FINE-TUNING
----------------
Base model
████████████████
all weights updated


ADAPTER TUNING
--------------
Base model
████████████████
      ↑
small modules inserted
and trained


LoRA
----
Base model frozen
████████████████
 + low-rank updates


QLoRA
-----
Quantized base frozen
████████
 + low-rank updates

For most architect discussions, this conceptual distinction is sufficient.

---

12.22 Dataset curation

Fine-tuning success often depends more on dataset quality than clever training technique.

A model trained on 100,000 bad examples can be worse than one trained on 5,000 excellent examples.

Your dataset lifecycle should look like:

Raw enterprise data
        |
        v
Source validation
        |
        v
Deduplication
        |
        v
Cleaning
        |
        v
PII / secret handling
        |
        v
Quality filtering
        |
        v
Labelling
        |
        v
Train / validation / test

---

12.23 What makes a good training example?

It should be:

• correct
• representative
• clear
• unambiguous
• relevant
• consistently labelled
• sufficiently diverse
• legally permitted for training
• free of unnecessary secrets/PII

Bad example:

Input:
"Supplier sent strange banking details"

Output:
"Risky"

What exactly should the model learn?

Better:

Input:
"Supplier requested bank account change.
New account is not present in supplier master."

Output:
{
  "risk": "HIGH",
  "risk_codes": ["UNVERIFIED_BANK_CHANGE"],
  "required_action": "SECONDARY_VERIFICATION"
}

---

12.24 Dataset representativeness

Suppose production contains:

60% normal procurement requests
20% service purchases
10% urgent procurements
 7% imports
 3% suspected fraud

Your training dataset contains:

95% normal
5% everything else

The model may look excellent on average but perform terribly on your critical 3%.

This introduces:

• class imbalance
• long-tail coverage
• rare-event evaluation

For risk/fraud/security workloads, rare cases may be the cases that matter most.

---

12.25 Dataset provenance

Provenance records where every training example came from and whether it may be used for training.

Enterprise fine-tuning needs provenance:

Training example
     |
     +-- source?
     +-- owner?
     +-- licence?
     +-- tenant?
     +-- consent?
     +-- version?
     +-- labeler?
     +-- timestamp?

You want to be able to answer:

“Why was this information allowed to influence this model?”

Especially in:

• healthcare
• BFSI
• legal
• HR
• customer data

---

12.26 Multi-tenant warning

Training one shared model on several tenants' data can transfer one customer's information into another customer's outputs.

Suppose your SaaS platform serves:

Customer A
Customer B
Customer C

Do not casually train:

A data
+
B data
+
C data
        ↓
shared tuned model

without a clear contractual and privacy basis.

Potential risks:

• cross-tenant information transfer
• confidentiality breach
• training rights issues
• inability to delete data
• model inversion/memorisation risk

Tenant isolation applies to training pipelines, not merely inference.

---

12.27 Training / validation / test split

This should be familiar from standard ML.

You divide examples into:

TRAIN
   ↓
used to update weights

VALIDATION
   ↓
used during development/tuning

TEST
   ↓
held back for final evaluation

Typical illustrative split:

80% train
10% validation
10% test

or:

70 / 15 / 15

There is no universal correct ratio.

The key requirement:

The test set must represent genuinely unseen data.

---

12.28 Training set

Used for gradient updates.

examples
   ↓
loss
   ↓
backprop
   ↓
weight update

The model directly learns from this data.

---

12.29 Validation set

Used to make training decisions:

• hyperparameters
• learning rate
• number of epochs
• LoRA rank
• checkpoint selection
• early stopping

The model doesn't normally train directly on validation examples, but you use their performance to optimise the training process.

Therefore validation is not completely unbiased final evidence.

---

12.30 Test set

The test set should ideally remain untouched until the model design is mostly final.

train
↓
iterate
↓
validation
↓
iterate
↓
FINAL candidate
↓
test once / sparingly

If you repeatedly optimise based on test performance:

test set
becomes
validation set

and its estimate becomes optimistic.

---

12.31 Time-based splits

Enterprise data often changes over time.

Random splitting may hide this.

Suppose:

2024 transactions
2025 transactions
2026 transactions

Better evaluation may be:

Train:
2024–2025

Validation:
early 2026

Test:
recent 2026

because that resembles deployment:

train on past → predict future.

This is particularly important for domains experiencing behavioural drift.

---

12.32 Entity-based splits

Suppose examples come from customers.

If:

Customer ABC examples

appear in both training and test data, the model may partially memorise customer-specific patterns.

Better:

Train:
customers A–M

Test:
customers N–Z

when the real requirement is:

generalise to unseen customers.

Split according to the generalisation question.

---

12.33 Data quality

Think in multiple dimensions.

Correctness

Are labels right?

Consistency

Do labelers follow the same rules?

Completeness

Are important scenarios missing?

Representativeness

Does it match production?

Diversity

Are linguistic/domain variations present?

Balance

Are important classes represented?

Freshness

Are examples still valid?

Noise

Are irrelevant fields confusing the model?

---

12.34 Label consistency

Label consistency means similar inputs always receive the same label; inconsistent labels teach the model contradictions.

Imagine:

Reviewer A:
Bank account change → HIGH

Reviewer B:
Bank account change → MEDIUM

Reviewer C:
Bank account change → HIGH only if > ₹10L

Your model is being trained on organizational disagreement.

No algorithm fixes undefined policy.

First define:

label taxonomy
+
annotation guideline
+
edge-case handling

Then measure inter-annotator agreement where appropriate.

This is a classic enterprise problem:

Fine-tuning often reveals that humans themselves have never agreed on the process.

---

12.35 Garbage in, model-shaped garbage out

The line worth keeping:

Fine-tuning compresses the patterns present in the dataset, including the bad ones.

If the dataset contains:

• bias
• obsolete processes
• contradictory decisions
• accidental policy violations

the tuned model may learn them.

Therefore dataset governance is part of AI governance.

---

12.36 What is data leakage in fine-tuning?

One of the most important evaluation failures.

Data leakage means information improperly crosses boundaries and gives unrealistic performance.

---

12.37 Leakage type 1: exact duplicate

Exact-duplicate leakage happens when the same example appears in both the training set and the test set.

Training:
Example X

Test:
Example X

Obviously invalid.

---

12.38 Leakage type 2: near duplicate

Near-duplicate leakage happens when test examples differ from training examples only trivially, so the test measures memory rather than generalisation.

Training:

Invoice 123 from ABC Ltd

Test:

Invoice 124 from ABC Ltd

with almost identical structure and content.

Your test may look stronger than genuine deployment.

---

12.39 Leakage type 3: same document chunks

A particularly relevant LLM/RAG issue.

Suppose you split documents into chunks first:

Document A:
chunk 1 → training
chunk 2 → test

The model may effectively have seen most of the test document.

Better:

split at document level
        ↓
then chunk

when appropriate.

---

12.40 Leakage type 4: future information

Imagine predicting credit risk at loan approval time but training data includes:

"loan defaulted after 90 days"

inside features.

The model appears brilliant because it sees the future.

Classic target leakage.

---

12.41 Leakage type 5: benchmark contamination

If a model has already been trained on the evaluation dataset or variants of it, benchmark results may be misleading.

For enterprise work, build private evaluation sets from your domain.

---

12.42 Leakage from synthetic data

Synthetic data can distort evaluation when the same model generates the training data and grades the result.

Suppose:

Frontier Model A
creates synthetic training data
       ↓
fine-tune Model B

Then:

Model A
also grades Model B

You risk creating an evaluation ecosystem where everything reflects Model A's preferences.

Independent human/task-level validation remains important.

---

12.43 Synthetic data

Synthetic data means using generated rather than directly observed examples.

Example:

Real examples: 2,000

Generate:
10,000 controlled variations

Potential uses:

• augment rare classes
• create edge cases
• produce paraphrases
• generate instruction-response pairs
• produce adversarial cases
• bootstrap a new domain

---

12.44 Synthetic data pipeline

A stronger architecture:

Seed examples
      |
      v
Generation model
      |
      v
Synthetic candidates
      |
      v
Validation / filtering
      |
  +---+----+
  |        |
reject   accept
           |
           v
      training set

Never:

LLM generates data
      ↓
train immediately

---

12.45 Synthetic data benefits

Scale

Generate many examples cheaply.

Coverage

Generate rare cases.

Privacy

Potentially reduce dependence on raw real-user data.

Controlled variation

Change:

• language
• format
• difficulty
• edge conditions

---

12.46 Synthetic data risks

Hallucinated labels

Generated target itself may be wrong.

Distribution collapse

Model-generated text becomes overly uniform.

Bias amplification

Teacher model biases propagate.

Unrealistic examples

Synthetic data may not resemble production.

Error reinforcement

Teacher creates error → student learns error.

Therefore:

Synthetic data is augmentation, not automatically truth.

---

12.47 Good synthetic-data strategy

A good synthetic-data strategy anchors generated examples to real data and filters them before they enter training.

Combine:

Real high-quality examples
        +
Synthetic augmentation
        +
Human/automated filtering
        +
Production evaluation

Not:

100% unchecked generated dataset

---

12.48 Fine-tuning evaluation

Before fine-tuning, establish the baseline.

Base model + prompt
        ↓
Evaluation
        ↓
Baseline score

Then:

Fine-tuned model
        ↓
same evaluation
        ↓
comparison

Without the baseline you cannot answer:

“Did fine-tuning actually help?”

---

12.49 Evaluation dimensions

Task quality

Examples:

classification accuracy
precision
recall
F1
exact match
field accuracy

Generative quality

correctness
groundedness
relevance
completeness

Behavioral compliance

JSON validity
format adherence
policy adherence
tone

Agentic behaviour

correct tool choice
argument correctness
task success
unnecessary tool calls

Operational metrics

latency
throughput
cost
token count

Safety

harmful behavior
data leakage
prompt injection robustness
policy violations

---

12.50 Evaluate the full system

Another important architect point:

Fine-tuned model quality alone is insufficient.

Production system:

Prompt
  +
RAG
  +
Fine-tuned model
  +
Tools
  +
Guardrails
  +
Workflow

Therefore test:

end-to-end task success

not only isolated model accuracy.

---

12.51 A/B evaluation

Suppose:

Model A = base
Model B = tuned

Run controlled traffic:

             requests
            /        \
           v          v
        Model A    Model B
           |          |
           v          v
       business metrics

Measure:

• resolution
• acceptance
• corrections
• escalation
• task completion
• cost

Offline improvement does not guarantee production improvement.

---

12.52 Fine-tuning can reduce prompt size

Consider:

Base model prompt:
4,000 tokens of instructions/examples

because you continually explain how to perform your specialised task.

Fine-tuned model may require:

500-token prompt

because some behaviour has moved into model parameters.

Potential benefits:

• lower token costs
• lower latency
• greater consistency

This can make fine-tuning economically attractive at sufficient volume.

---

12.53 What is catastrophic forgetting?

Fine-tuning a model heavily on a narrow dataset can make it worse at capabilities it previously possessed.

Conceptually:

Before:

General reasoning      ██████████
Domain task            ██████


After aggressive tuning:

General reasoning      █████
Domain task            ██████████

The model has adapted too aggressively to the narrow distribution.

This is catastrophic forgetting.

---

12.54 Example of catastrophic forgetting

Suppose you tune a model heavily on:

legal contract extraction

with repetitive short JSON outputs.

Afterward it may become worse at:

• general conversation
• broader reasoning
• unrelated tool calls
• longer explanations

because training pushed it strongly toward the narrow target distribution.

---

12.55 Mitigating catastrophic forgetting

Approaches include:

• lower learning rate
• fewer epochs
• PEFT instead of full fine-tuning
• mix general examples into training
• maintain diverse training examples
• stop training earlier
• compare broad capability benchmarks
• keep the base model available separately

Architecture principle:

Evaluate both what you want to gain and what you might lose.

---

12.56 Overfitting

Overfitting means a model performs well on its training data but poorly on data it has not seen.

Overfitting means:

model performs extremely well on training examples but poorly on unseen examples.

Example:

Training accuracy      99%
Validation accuracy    83%
Test accuracy          78%

The model learned training-specific details instead of general patterns.

---

12.57 Why LLM fine-tunes overfit

Common causes:

• too little data
• too many epochs
• high learning rate
• repetitive examples
• narrow dataset
• excessive model capacity
• duplicated data

---

12.58 Training curves

Conceptually:

Loss
 ^
 |\
 | \
 |  \ training loss
 |   \____________
 |
 |      validation loss
 |     \____
 |          \__
 |             \___
 |                 /
 |                /
 +----------------------> epochs

If training continues improving while validation starts worsening:

overfitting

Use techniques such as:

• early stopping
• regularisation
• lower learning rate
• more/diverse data
• fewer epochs

---

12.59 Overfitting vs catastrophic forgetting

Overfitting is failure to generalise to new data; catastrophic forgetting is loss of capabilities the model already had.

Know the distinction.

Overfitting

memorised/adapted too strongly
to training examples
→ poor generalisation

Catastrophic forgetting

new adaptation damages
previously learned capabilities

They can happen together, but they describe different problems.

---

12.60 Model merging concepts

Model merging combines the weights or adapters of separately fine-tuned models into one model.

Suppose you have:

Base model
   |
   +-- Finance-tuned model
   |
   +-- Legal-tuned model

Can we combine adaptations?

That is the broad idea behind model merging.

Conceptually:

W_merged = α · W_A + (1 − α) · W_B

although real techniques can be considerably more sophisticated.

---

12.61 LoRA merging

A common operational case:

Base model
   +
LoRA adapter

You can sometimes:

merge adapter weights
into base weights

producing:

standalone adapted model

Advantages:

simpler inference
potentially lower runtime adapter overhead

But:

less easy to switch adapters dynamically
new model artifact to manage

---

12.62 Multi-adapter serving

Multi-adapter serving keeps one base model in memory and applies a different LoRA adapter per request.

Alternative:

                 Base model
                     |
          +----------+----------+
          |          |          |
          v          v          v
       Legal      Finance   Procurement
        LoRA        LoRA       LoRA

Request:

tenant/workload
      ↓
adapter router
      ↓
base + selected adapter

Architecturally attractive when many related specialised variants share one base.

But complexity appears in:

• adapter compatibility
• model version coupling
• cache management
• scheduling
• isolation
• testing

---

12.63 Model merging caveat

Do not assume:

Great legal model
+
Great finance model
=
great legal-finance model

Weight-space interactions may damage capabilities.

Merged models require complete evaluation like any new model.

---

12.64 What is model distillation?

Distillation is a different idea.

You have:

Teacher model
    ↓
high capability
expensive

and train:

Student model
    ↓
smaller / cheaper / faster

to imitate useful behaviour.

Architecture:

             TEACHER
         large strong model
                |
      generates outputs /
        supervision
                |
                v
             dataset
                |
                v
         STUDENT MODEL
       smaller / cheaper

---

12.65 Why distill?

Suppose the frontier model gives:

95% task success
$0.10/request

but your workload is narrow.

A smaller student may achieve:

93% task success
$0.01/request

Illustrative numbers only, but the principle is:

Trade a small amount of general capability for dramatically better economics on a constrained workload.

---

12.66 Distillation vs fine-tuning

Fine-tuning adapts a model's behaviour from examples; distillation trains a smaller model to reproduce a larger model's outputs.

Fine-tuning asks:

“How do I adapt this model to my dataset/task?”

Distillation asks:

“How do I transfer useful behaviour from a stronger model into a smaller model?”

Often combined:

Strong teacher
      ↓
generate/label task examples
      ↓
fine-tune smaller student

---

12.67 Distillation and synthetic data

These concepts often meet.

Teacher LLM
    ↓
generates high-quality responses
    ↓
synthetic training dataset
    ↓
student fine-tuning

But:

The student also inherits the teacher's mistakes.

Use quality filtering and task-specific evaluation.

---

12.68 Distillation use case for agent systems

Imagine your agent needs:

Intent classification
Entity extraction
Tool selection

You initially use a frontier model for all three.

After collecting high-quality execution traces:

Frontier model
      ↓
label/teacher
      ↓
small tuned model

Then:

simple agent steps
     ↓
small student

complex planning
     ↓
frontier model

This fits perfectly with the complexity-based routing from the previous section.

---

12.69 Fine-tuning deployment lifecycle

This is the most important architect piece beyond the algorithms.

Do not think:

dataset
  ↓
fine-tune
  ↓
production

Think:

              MODEL ADAPTATION LIFECYCLE

Business requirement
        |
        v
Baseline evaluation
        |
        v
Is fine-tuning justified?
        |
        v
Dataset sourcing
        |
        v
Governance / privacy / licensing
        |
        v
Cleaning + labelling
        |
        v
Version dataset
        |
        v
Train / validation / test split
        |
        v
Fine-tune
        |
        v
Offline evaluation
        |
        v
Safety / security evaluation
        |
        v
Model registry
        |
        v
Shadow deployment
        |
        v
Canary / A-B
        |
        v
Production
        |
        v
Monitoring
        |
        v
Feedback / retraining

---

12.70 Step 1: define measurable need

Example:

Bad:

“We want a customised model.”

Good:

Current tool-selection accuracy: 84%
Required: ≥97%

Current prompt: 6,000 tokens/request
Target: <1,500 tokens

Current classifier cost: $X
Target: reduce by 70%

Now fine-tuning has a business case.

---

12.71 Step 2: baseline

Before tuning:

Base model
+
best reasonable prompt
+
RAG where appropriate

Evaluate it.

Otherwise you might spend weeks fine-tuning only to discover that an improved prompt achieved the same result.

---

12.72 Step 3: dataset versioning

Training data should be immutable/versioned.

Example:

procurement-risk-v1.0
procurement-risk-v1.1
procurement-risk-v2.0

Record:

• source
• transformations
• filters
• label schema
• split
• dataset hash/version
• approvals

Model reproducibility requires knowing exactly which dataset created it.

---

12.73 Step 4: training run tracking

Track:

base model
base model version
dataset version
training method
LoRA rank
learning rate
epochs
batch size
random seed
training code version
training metrics

Otherwise:

“Why is version 17 better?”

becomes impossible to answer.

---

12.74 Step 5: model registry

After training:

Model Registry
     |
     +-- Base model
     +-- Adapter
     +-- Dataset version
     +-- Evaluation
     +-- Approval status
     +-- Owner
     +-- Deployment status

Possible lifecycle:

EXPERIMENT
   ↓
CANDIDATE
   ↓
APPROVED
   ↓
PRODUCTION
   ↓
DEPRECATED
   ↓
RETIRED

---

12.75 Step 6: offline gates

Before production:

task quality
     ✓

safety
     ✓

privacy
     ✓

regression
     ✓

latency
     ✓

cost
     ✓

Remember regression testing.

If:

domain task:
+12%

tool calling:
-20%

the fine-tune may not be acceptable.

---

12.76 Step 7: shadow deployment

Real request
    |
    +------→ production model → user
    |
    +------→ tuned candidate   → evaluation only

Allows comparison on real traffic without user impact.

---

12.77 Step 8: canary

95% → current model
 5% → tuned model

Observe:

• quality
• latency
• failures
• safety
• business metrics

Then gradually:

5%
↓
20%
↓
50%
↓
100%

with rollback criteria.

---

12.78 Step 9: production monitoring

Monitor both infrastructure and model behaviour.

Operational

latency
error rate
throughput
GPU utilisation
cost

Model

task success
class distribution
tool accuracy
human correction rate
refusal rate

Data

input distribution
new terminology
new classes
drift

---

12.79 Data drift

Data drift is the gap that opens when production inputs change after a model was trained.

Production inputs change.

Training:

2025 procurement terminology

Production:

2027 new supplier categories
new policy
new regulations

Even without model degradation, the world changed.

Monitor input and performance drift.

---

12.80 Feedback loops

A useful enterprise architecture:

Production inference
        |
        v
Human corrections
        |
        v
Feedback store
        |
        v
Quality filtering
        |
        v
Candidate training data
        |
        v
Next model version

But never automatically train every user correction.

A malicious or simply mistaken user could poison the model.

---

12.81 Model poisoning risk

Fine-tuning introduces another attack surface.

If attackers can influence training examples:

malicious examples
       ↓
training pipeline
       ↓
model learns
malicious behaviour

Controls:

• trusted sources
• provenance
• anomaly detection
• approval workflow
• dataset versioning
• restricted write access
• human validation
• security testing

This connects directly to your earlier memory poisoning / RAG poisoning topics.

---

12.82 Training data deletion

Deleting a record from training data does not remove what a trained model already learned from it.

This becomes difficult.

Suppose customer A says:

“Delete all our data.”

Removing raw records from the data lake is straightforward.

But:

Customer data
    ↓
training dataset
    ↓
model weights

Once information influenced parameters, deletion is not as simple as deleting one row.

Possible response might require:

remove source data
↓
remove training examples
↓
rebuild/retrain affected model

depending on policy/regulatory requirements.

This is a major reason enterprises need strict rules about what is permitted into training.

---

12.83 Fine-tuning architecture at enterprise scale

The overall shape:

                       DATA PLANE

Enterprise sources
       |
       v
+------------------+
| Training Data    |
| Pipeline         |
+------------------+
| ingest           |
| clean            |
| redact           |
| deduplicate      |
| label            |
| quality gate     |
+--------+---------+
         |
         v
+------------------+
| Versioned Dataset|
+--------+---------+
         |
         v
+------------------+
| Training Service |
+------------------+
| SFT              |
| LoRA / QLoRA     |
| adapters         |
+--------+---------+
         |
         v
+------------------+
| Evaluation       |
+--------+---------+
         |
         v
+------------------+
| Model Registry   |
+--------+---------+
         |
         v
+------------------+
| Deployment       |
| Gateway / Router |
+--------+---------+
         |
         v
Production inference
         |
         v
Telemetry / feedback

Across the entire system:

             CONTROL / GOVERNANCE PLANE

IAM
Dataset provenance
Privacy
Tenant controls
Training approvals
Model approvals
Audit
Versioning
Evaluation policy
Deployment policy

---

12.84 Fine-tuning and the model gateway

Connect this with the previous topic.

Your application should still not know:

procurement-risk-lora-v17-q4

Instead:

Application
    ↓
AI Gateway
    ↓
Model Router
    ↓
Logical capability:
procurement-risk
    ↓
Registry resolution
    ↓
fine-tuned model v17

Then you can replace:

v17 → v18

without application changes.

---

12.85 Fine-tuning and RAG together

Very important.

Suppose you build a legal assistant.

Fine-tuning:

teaches:
how legal analysis should be structured
which categories matter
how citations should be expressed
how tools should be called

RAG:

provides:
current contract
current regulations
customer policies
previous case documents

Architecture:

User question
      |
      v
Retriever
      |
      v
Current evidence
      |
      +--------+
               |
               v
       Fine-tuned model
               |
               v
      Domain-consistent answer

Fine-tuning gives behaviour; RAG gives evidence.

---

12.86 Fine-tuning and agents

For agent systems, fine-tuning can be especially valuable for narrow repeated decisions.

Examples:

Supervisor:
complex planning
→ frontier/base model

Task classifier:
millions of repetitive classifications
→ small fine-tuned model

Tool router:
stable tool catalogue
→ fine-tuned model potentially

Document extractor:
domain-specific
→ fine-tuned model

Final verifier:
high consequence
→ stronger model

This produces:

different models
for different cognitive roles

instead of fine-tuning one giant universal agent.

---

12.87 What I would NOT fine-tune into a model

Generally avoid using model weights as the primary store for:

Frequently changing facts

product prices
policy wording
inventory
customer records
regulations

Use RAG/database/tools.

Secrets

API keys
passwords
credentials

Never.

Strict business calculations

tax
interest calculation
approval thresholds

Use deterministic code.

User-specific memory

customer preference
session state
workflow state

Use memory/state stores.

This distinction is extremely important.

---

12.88 Fine-tuning strategy maturity

Level 0

Base model

Level 1

Prompt engineering

Level 2

Prompt + RAG

Level 3

Fine-tuning for demonstrated gap

Level 4

PEFT / specialist models

Level 5

Routing + specialist tuned models

Level 6

Distillation + adaptive model portfolio

Don't jump to level 6 because it sounds sophisticated.

---

12.89 Common failure modes

Failure 1: fine-tuning for knowledge

“We trained the employee handbook into the model.”

Problem

The handbook changes.

Better

RAG.

---

Failure 2: no baseline

Fine-tuned model scores 92%.

Great?

Base model already scored:

94%

Fine-tuning made things worse.

---

Failure 3: dataset leakage

train ↔ test overlap

Results look excellent but are false.

---

Failure 4: bad labels

Model perfectly learns inconsistent human decisions.

Fix

Govern annotation before training.

---

Failure 5: too much synthetic data

Model learns synthetic distribution rather than production reality.

Fix

Maintain real-data grounding.

---

Failure 6: overfitting

Great training performance, poor production performance.

---

Failure 7: catastrophic forgetting

Specialised capability improves but general behaviour collapses.

---

Failure 8: no regression suite

Only the target task is tested.

Fix

Evaluate retained capabilities too.

---

Failure 9: training on prohibited customer data

Creates privacy/legal problems and possibly irreversible model artifacts.

---

Failure 10: treating tuned model as permanent

Base model changes, regulations change, business taxonomy changes.

Fine-tuned models have a lifecycle.

---

12.90 Prompt vs RAG vs Fine-Tune decision table

The table to keep at hand:

Requirement	Preferred mechanism
Better instruction following	Prompt
Output schema	Prompt first
Current enterprise knowledge	RAG
Private documents	RAG
Latest policy	RAG
Repeated specialised behaviour	Fine-tune
Proprietary classification taxonomy	Fine-tune candidate
Style/tone consistency at scale	Fine-tune candidate
Reduce huge few-shot prompts	Fine-tune candidate
Strict calculation	Code
Live inventory	Tool/API
User/session state	State/memory
Complex domain behaviour + current data	Fine-tune + RAG

---

12.91 Full FT vs LoRA vs QLoRA

Another table worth knowing:

Technique	Base weights	Extra trainable params	Compute	Typical reason
Full FT	Updated	Huge	Highest	Maximum adaptation
Adapter tuning	Frozen	Small modules	Low/moderate	Modular adaptation
LoRA	Frozen	Low-rank matrices	Low/moderate	Efficient fine-tuning
QLoRA	Quantized + frozen	LoRA matrices	Lower memory	Fine-tune larger models economically

---

12.92 LoRA vs adapter vs distillation

Do not confuse these.

LoRA
→ adapt same model using low-rank parameter updates

Adapter
→ attach/train small model modules

Distillation
→ transfer capability into a smaller student

Model merging
→ combine trained parameter adaptations/models

Different purposes.

---

12.93 The enterprise-wide view

If asked:

“How would you define a fine-tuning strategy for an enterprise?”

A strong answer is:

“I would first establish whether the gap is actually instructions, knowledge or model behaviour. Prompting is my first lever, RAG handles dynamic enterprise knowledge, and I introduce fine-tuning only when we have a measurable repeated behavioural gap or a strong economics case. If fine-tuning is justified, I would prefer parameter-efficient approaches such as LoRA or QLoRA where they meet quality requirements rather than defaulting to full-model training. The larger architecture is really a governed ML lifecycle: provenance-controlled datasets, clean train/validation/test separation, leakage checks, offline and regression evaluation, model registry, shadow and canary deployment, monitoring and rollback. For enterprise data I'd also explicitly govern tenant boundaries, PII, training rights and deletion obligations.”

That's the right level.

---

12.94 FAQ: Why not fine-tune everything?

“Because fine-tuning creates a new model lifecycle and embeds patterns into weights, which makes changing knowledge and governance more difficult. If the problem can be solved through prompting or retrieval, those are typically easier to update, audit and roll back. Fine-tuning makes sense when I want persistent behavioural adaptation or better economics from a specialised model, not merely because I have enterprise data available.”

---

12.95 FAQ: When should you use LoRA?

“When the base model already has most of the underlying capability and I need task or domain adaptation without paying the compute and storage cost of updating all parameters. I freeze the base model and learn low-rank updates to selected weight matrices, which gives me small adapter artifacts and enables multiple specialised variants to share the same base model.”

---

12.96 FAQ: What is QLoRA?

“QLoRA takes the PEFT idea further by keeping the frozen base model quantized, often at very low precision, while training higher-precision LoRA adapters. That significantly reduces memory required to fine-tune large models while retaining much of the benefit of LoRA adaptation.”

---

12.97 FAQ: How do you know fine-tuning worked?

Do not answer merely:

“The training loss decreased.”

Say:

“I establish a production-representative baseline before tuning and evaluate the tuned model against an untouched test set using workload-specific quality metrics, safety and regression tests, latency and cost. Then I'd shadow it against real production traffic and use a controlled canary or A/B rollout. The relevant question isn't whether training loss fell; it's whether end-to-end task performance improved without unacceptable regressions.”

---

12.98 FAQ: How do you prevent data leakage?

“I split data at the correct semantic boundary before preprocessing where necessary, for example at customer or document level rather than individual chunks, deduplicate near-identical examples, isolate time periods when future leakage is possible, preserve a genuinely untouched test set, and track dataset provenance so training and evaluation examples can be traced.”

---

12.99 FAQ: Should you train on production conversations?

“Not automatically. Production conversations are useful candidates for improving the dataset, but I'd require explicit data-use policy, privacy and tenant controls, filtering for PII and secrets, quality validation and provenance. User feedback is noisy and potentially adversarial, so it should enter a controlled data-curation pipeline rather than flowing directly into model training.”

---

12.100 FAQ: How do you prevent catastrophic forgetting?

“Fine-tuning can improve the target distribution while degrading capabilities learned during pretraining. I mitigate that through conservative adaptation, often PEFT, appropriate learning rates and training duration, diverse or replay data where needed, and regression evaluation against capabilities we intend to preserve.”

---

12.101 FAQ: How do you detect and prevent overfitting?

“I look for divergence between training and validation performance and use held-out data representative of production. Mitigations include better data diversity, fewer epochs, lower learning rate, early stopping and reducing adaptation capacity. For LLMs I also pay particular attention to duplicates and near-duplicates because they can make evaluation appear far stronger than genuine generalisation.”

---

12.102 The full adaptation lifecycle in one diagram

Remember this diagram:

                    BUSINESS REQUIREMENT
                           |
                           v
                  BASELINE EVALUATION
                           |
                   +-------+-------+
                   |               |
             Gap is knowledge   Gap is behaviour
                   |               |
                   v               v
                  RAG         FINE-TUNING
                                   |
                                   v
                         DATA GOVERNANCE
                                   |
                  +----------------+----------------+
                  |                |                |
               privacy         provenance       quality
                  |                |                |
                  +----------------+----------------+
                                   |
                                   v
                           VERSIONED DATASET
                                   |
                                   v
                             SFT / PEFT
                            LoRA / QLoRA
                                   |
                                   v
                              EVALUATION
                      quality + regression
                                   |
                                   v
                           MODEL REGISTRY
                                   |
                                   v
                        SHADOW → CANARY → PROD
                                   |
                                   v
                        GATEWAY / MODEL ROUTER
                                   |
                                   v
                             MONITORING
                                   |
                                   v
                              FEEDBACK
                                   |
                              controlled
                                retraining

If you can explain every box, you understand this topic at AI Architect rather than model-user level.

---

12.103 What you should know cold

In short:

• Prompting fixes instructions, RAG fixes knowledge, code fixes rules; fine-tuning fixes learned behaviour.
• Establish a prompt and RAG baseline before any training run.
• Fine-tuning quality is a data problem: provenance, splits and leakage decide the result.
• Prefer LoRA or QLoRA unless full fine-tuning is demonstrably needed.
• A tuned model is a governed artefact: registry, regression gates, shadow and canary rollout, monitoring.

The full list:

1. Prompting changes runtime instructions.
2. RAG changes runtime knowledge/context.
3. Fine-tuning changes learned behaviour.
4. Deterministic business logic should generally remain code.
5. SFT learns from input → desired-output examples.
6. Full fine-tuning updates most/all parameters.
7. PEFT adapts a model using only a small number of trainable parameters.
8. LoRA learns low-rank updates (BA) while freezing the base model.
9. QLoRA uses a quantized frozen base plus trainable LoRA parameters.
10. Adapters are small task-specific trainable components.
11. Fine-tuning quality is primarily a data-quality problem.
12. Train updates parameters; validation guides model development; test estimates final generalisation.
13. Split at the right semantic boundary, not blindly by rows.
14. Near-duplicate leakage is as dangerous as exact duplication.
15. Synthetic data must be validated; generation does not make it ground truth.
16. Overfitting means poor unseen-data generalisation.
17. Catastrophic forgetting means new adaptation damages old capabilities.
18. Model merging combines adaptations/weight information but requires fresh evaluation.
19. Distillation transfers capability from a strong teacher to a smaller student.
20. Fine-tuning requires a complete dataset → training → evaluation → registry → rollout → monitoring lifecycle.
21. Never fine-tune sensitive multi-tenant data without explicit governance and training rights.
22. Always establish a base-model + prompt/RAG baseline before fine-tuning.
23. Evaluate regression as well as target-task improvement.
24. A tuned model should remain behind the same model gateway/registry abstraction as any other model.

The one sentence to remember

“I use prompting for instructions, RAG for changing knowledge, deterministic tools for rules, and fine-tuning only for persistent behavioural adaptation or a demonstrated economics advantage; when tuning is justified, I treat the dataset and model as governed versioned artifacts and take the resulting model through leakage-controlled evaluation, regression testing, registry, shadow/canary rollout and continuous monitoring rather than treating fine-tuning as a one-time training exercise.”

That is the Principal/AI Architect framing for Fine-Tuning & Model Adaptation.

---

Related reading

• Prompt and Context Engineering as an Architectural Concern, the first lever, and usually the cheapest.
• RAG Architecture: The Full Pipeline and Where Each Stage Fails, the second lever, for knowledge that changes.
• How Enterprises Evaluate LLM Features Before Shipping, the evaluation discipline a tuned model has to pass.