LLM Fundamentals for Architects: Tokens, Context, Latency, Throughput and Cost

An AI architect does not need to derive transformer mathematics from first principles. You do need to understand the mechanics deeply enough to answer:

• Why does the system behave this way?
• What creates latency/cost?
• Why does a model hallucinate?
• Why can a larger context window still perform badly?
• What happens on the GPU during inference?
• Why does KV caching matter?
• When should you change the model versus change the surrounding architecture?
• Which controls are probabilistic versus deterministic?

A useful mental model is:

Raw input
   ↓
Tokenizer
   ↓
Token IDs
   ↓
Token embeddings + positional information
   ↓
Transformer layers
   ├─ Self-attention
   ├─ Feed-forward network
   ├─ Residual connections
   └─ Normalization
   ↓
Probability distribution over next token
   ↓
Decoder / sampling strategy
   ↓
Next token
   ↓
Repeat autoregressively

Everything else, meaning RAG, agents, tool calling, structured outputs and memory, is largely architecture wrapped around this probabilistic token-generation engine.

---

2.1 Tokens and Tokenization

What is a token?

LLMs do not operate directly on words.

They operate on tokens, integer IDs representing pieces of text.

Example:

"unbelievable"
might become:
["un", "believ", "able"]

Or:

"OpenAI"
→ ["Open", "AI"]

depending on the tokenizer.

Internally:

"Hello world"
      ↓ tokenizer
[9906, 1917]
      ↓ embedding lookup
vectors

The vocabulary might contain tens or hundreds of thousands of possible tokens.

---

Why not use whole words?

A word-level vocabulary becomes enormous and handles unseen words badly.

Character-level tokenization creates extremely long sequences.

Subword tokenization provides a compromise:

Characters      → tiny vocabulary, huge sequences
Words           → huge vocabulary, unknown words
Subword tokens  → manageable vocabulary + manageable sequences

Common approaches include variants of:

• BPE: Byte Pair Encoding
• WordPiece
• SentencePiece
• byte-level tokenization

Exact implementation varies by model family.

---

Architect implication

Token count, not character count, is usually what drives:

• context consumption
• inference cost
• latency
• maximum request size
• output limits
• KV-cache memory

So:

Cost ≈ input_tokens × input_price
     + output_tokens × output_price

for token-priced APIs.

---

Tokenization is language-dependent

Different languages may require different numbers of tokens for equivalent semantic content.

That means two 1,000-character documents can have very different:

token counts
cost
latency
context consumption

This matters when designing multilingual enterprise applications.

---

Important interview distinction

#### Tokenizer vocabulary ≠ model knowledge

The fact that something has a token does not mean the model understands it.

And something not represented by a single token can still be understood through multiple tokens.

---

Interview answer

Q: Why should an AI architect care about tokenization?

Because tokens are the actual computational unit consumed by the model. Tokenization affects context utilisation, inference cost, latency and sometimes multilingual performance. I therefore budget systems in tokens rather than characters or words and inspect token distributions for production workloads.

---

2.2 Embeddings

An embedding is a dense numerical vector representing information in a high-dimensional space.

Example conceptually:

"database"
     ↓
[0.14, -0.71, 0.03, ...]

Similar meanings tend to appear near each other in embedding space.

king
queen
monarch
royalty
→ relatively close
king
airplane
→ farther apart

---

Two meanings of "embedding" you must distinguish

This is an interview trap.

A. Token embeddings inside the LLM

Each token ID maps to a learned vector.

token ID 4217
    ↓
embedding table
    ↓
[... vector ...]

These vectors become the initial representation processed by the transformer.

---

B. Semantic embeddings used for retrieval

An embedding model converts:

sentence
paragraph
document
image
etc.

into a single vector suitable for:

• similarity search
• clustering
• recommendations
• semantic retrieval
• RAG

Example:

"What is our maternity leave policy?"
                ↓
        embedding model
                ↓
            vector Q
Policy document chunks
                ↓
        embedding model
                ↓
       vectors D1...Dn
nearest vectors → candidate documents

Do not confuse these two concepts.

---

Similarity

Common metrics:

#### Cosine similarity

[ cos(A,B)=\frac{A\cdot B}{||A||||B||} ]

Measures direction.

#### Dot product

[ A \cdot B ]

Often used when embeddings are normalized or trained around dot-product scoring.

#### Euclidean distance

[

	A-B

]

Measures geometric distance.

We'll cover this much more deeply under RAG.

---

2.3 Transformer Architecture

The transformer is the core architecture behind modern LLMs.

A simplified decoder-only transformer:

Input text
   ↓
Tokenization
   ↓
Token embeddings
   +
Position information
   ↓
┌──────────────────────────┐
│ Transformer Block 1      │
│                          │
│ Self-Attention           │
│      ↓                   │
│ Residual + Norm          │
│      ↓                   │
│ Feed-Forward Network     │
│      ↓                   │
│ Residual + Norm          │
└──────────────────────────┘
   ↓
┌──────────────────────────┐
│ Transformer Block 2      │
└──────────────────────────┘
   ↓
...
   ↓
Final hidden representation
   ↓
Linear projection
   ↓
Softmax
   ↓
Probability of next token

Large models may contain dozens or hundreds of such layers.

---

2.3.1 Self-Attention

Attention allows each token to determine which other tokens are relevant when constructing its representation.

Consider:

"The animal didn't cross the road because it was tired."

When processing:

"it"

the model needs to associate it strongly with:

"animal"

Attention provides this mechanism.

Conceptually:

Current token
     ↓
Look at other relevant tokens
     ↓
Assign attention weights
     ↓
Combine their information

---

2.3.2 Query, Key and Value

Every token representation is projected into three vectors:

Q = Query
K = Key
V = Value

The intuitive interpretation:

#### Query

What information am I looking for?

#### Key

What information do I represent?

#### Value

What information should I contribute if selected?

For each token:

[ Q=XW_Q ]

[ K=XW_K ]

[ V=XW_V ]

Attention scores are approximately:

[ Attention(Q,K,V) ================

softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V ]

---

Intuition

Imagine:

Query:
"What tokens are relevant to me?"
            similarity
Q(token) ---------------- K(other token)
                         ↓
                    attention score
                         ↓
             weighted contribution of V

Example:

"The bank approved the loan because it..."
                     attention
"it" ──────────────────────────► "bank"

The exact patterns learned are much more complex than grammatical reference resolution, but this intuition is useful.

---

2.3.3 Causal / Masked Self-Attention

Decoder LLMs must not see future tokens during training.

Suppose:

The capital of France is Paris

When predicting:

Paris

the model may see:

The capital of France is

but not the future answer.

Therefore attention uses a causal mask:

Token 1 can attend to: 1
Token 2 can attend to: 1 2
Token 3 can attend to: 1 2 3
Token 4 can attend to: 1 2 3 4

Matrix:

       T1 T2 T3 T4
T1     ✓
T2     ✓  ✓
T3     ✓  ✓  ✓
T4     ✓  ✓  ✓  ✓

---

2.3.4 Multi-Head Attention

Instead of performing attention once, transformers perform several attention operations in parallel.

Input
 ├── Attention Head 1
 ├── Attention Head 2
 ├── Attention Head 3
 └── Attention Head N
          ↓
       concatenate
          ↓
     linear projection

Different heads can learn different relationships.

Conceptually:

Head 1 → local syntax
Head 2 → long-range dependency
Head 3 → entity relationship
Head 4 → positional relationship
...

Do not state that specific heads always correspond cleanly to particular linguistic concepts. That is only intuition.

---

2.3.5 Positional Encoding

Attention itself does not inherently know token order.

Without positional information:

dog bites man

and:

man bites dog

contain the same tokens.

Therefore transformers inject information about position.

Classical transformers used sinusoidal positional encodings.

Modern LLMs often use approaches such as:

RoPE: Rotary Positional Embeddings

Other architectures may use different schemes.

Conceptually:

Token representation
      +
Position representation
      ↓
"dog at position 1"
different from:
"dog at position 3"

---

Why architects should care

Position encoding strongly influences:

• maximum context length
• context extension strategies
• long-context behaviour
• extrapolation beyond trained lengths

Simply increasing an API's context number does not guarantee equally good reasoning across the whole context.

---

2.3.6 Feed-Forward Layers

After attention, each token passes through a neural network, usually called the:

FFN
MLP
feed-forward network

Simplified:

[ FFN(x)=W_2\sigma(W_1x+b_1)+b_2 ]

Modern architectures frequently use gated variants.

Attention roughly answers:

What information elsewhere in the sequence matters?

The FFN transforms the resulting representation.

A useful simplified model:

Attention = communicate information between tokens
FFN = transform information inside each token representation

---

2.3.7 Residual Connections

Deep neural networks are difficult to train.

Residual connections add the original input back:

[ output=x+f(x) ]

Instead of:

x → transformation → output

we get:

x ───────────────────────────┐
  ↓                          │
transformation               │
  ↓                          │
  + ◄────────────────────────┘
  ↓
output

Benefits include:

• better gradient flow
• easier optimization
• preservation of information
• ability to train very deep networks

---

2.3.8 Layer Normalization

Layer normalization stabilizes activations during training.

Conceptually:

wildly varying activation scales
             ↓
      layer normalization
             ↓
more controlled distribution

Modern transformers often use pre-normalization architectures:

x
│
├────────────────────┐
↓                    │
Norm                  │
↓                     │
Attention             │
↓                     │
+ ◄───────────────────┘

Exact architecture differs by model.

---

2.3.9 Transformer Block: Know This Diagram

You should be able to draw:

                ┌───────────────────────┐
Input ─────────►│ Layer Norm            │
                │      ↓                │
                │ Self Attention        │
Input ─────────►│      + residual       │
                │      ↓                │
                │ Layer Norm            │
                │      ↓                │
                │ Feed Forward / MLP    │
                │      + residual       │
                └──────────┬────────────┘
                           ↓
                        Output

Repeated N times.

---

2.4 Autoregressive Generation

Decoder LLMs generate text one token at a time.

Given:

"The capital of France is"

the model computes:

P(next_token | previous_tokens)

Example:

Paris      0.89
Lyon       0.03
France     0.02
London     0.01
...

A decoding strategy chooses the next token.

Suppose:

Paris

is selected.

Then the next inference becomes:

"The capital of France is Paris"

and the process repeats.

Formally:

[ P(x_1,\ldots,x_n) =================

\prod_{t=1}^{n} P(x_t|x_1,\ldots,x_{t-1}) ]

---

Critical architectural implication

Generation is sequential:

Token 1
 ↓
Token 2
 ↓
Token 3
 ↓
Token 4

You cannot normally generate token 4 before token 3 exists.

This is one major reason why:

generation latency

behaves differently from normal parallel GPU workloads.

---

2.5 Pre-training

Pre-training creates the foundation model.

The model sees enormous datasets and learns to predict tokens.

Example:

Input:
"The Eiffel Tower is located in"
Target:
"Paris"

Repeated across massive datasets.

The objective sounds simple:

predict next token

but doing this at enormous scale causes the network to learn representations involving:

• language
• grammar
• concepts
• factual associations
• style
• code
• patterns
• some forms of reasoning

---

Important distinction

Pre-training produces:

a language model

not necessarily:

a useful assistant

It may simply continue text.

Example:

User: Explain TLS to me.
Raw pretrained model:
User: Explain TLS to me.
Assistant: ...

Instruction tuning transforms the behaviour.

---

2.6 Instruction Tuning

Instruction tuning trains the model on examples like:

Instruction:
Summarize the following...
Response:
...

or:

User:
How do I configure X?
Assistant:
...

The model learns:

When given instructions, produce responses aligned with them.

Usually this is supervised fine-tuning:

SFT: Supervised Fine-Tuning

Pipeline:

Pretrained model
       ↓
Instruction dataset
       ↓
Supervised fine-tuning
       ↓
Instruction-following model

---

2.7 RLHF

RLHF = Reinforcement Learning from Human Feedback.

Simplified pipeline:

Pretrained model
      ↓
Instruction tuning
      ↓
Generate several answers
      ↓
Humans rank answers
      ↓
Train reward model
      ↓
Optimize model against reward

Example:

Prompt
   ↓
Response A
Response B
Response C
   ↓
Human preference:
B > A > C

This preference data teaches desired behaviours.

---

Why RLHF?

Next-token prediction alone doesn't directly optimize for:

• helpfulness
• instruction following
• safety
• preferred tone
• refusal behaviour
• answer usefulness

RLHF attempts to shape these behaviours.

---

2.8 RLAIF

RLAIF = Reinforcement Learning from AI Feedback.

Instead of humans evaluating every response:

AI model / constitutional rules
              ↓
      preference feedback
              ↓
      model optimization

Potential advantages:

• cheaper
• scalable
• faster preference generation

Risks:

• evaluator bias
• model reinforcing model errors
• systematic blind spots

---

2.9 DPO

DPO = Direct Preference Optimization.

It simplifies preference alignment.

Instead of:

preference data
      ↓
reward model
      ↓
reinforcement learning
      ↓
aligned model

DPO approximately does:

preferred vs rejected responses
             ↓
      direct optimization
             ↓
         aligned model

Example training record:

Prompt: Explain OAuth.
Chosen response:
A technically correct and useful explanation.
Rejected response:
Incorrect/confusing explanation.

The model is trained to increase the relative probability of the chosen response.

---

Interview-level comparison

Method	Core idea
SFT	Learn directly from desired responses
RLHF	Humans generate preference signals; optimize model using them
RLAIF	AI generates preference/evaluation signals
DPO	Directly train from chosen vs rejected responses

You do not need to claim that every commercial model uses exactly one of these. Modern training pipelines can combine multiple techniques.

---

2.10 Context Window

The context window is approximately how much tokenized information the model can operate on during one inference interaction.

It may contain:

System instructions
+
Developer instructions
+
Conversation history
+
User prompt
+
Retrieved documents
+
Tool definitions
+
Tool results
+
Generated tokens

Conceptually:

┌─────────────────────────────┐
│ Context Window              │
│                             │
│ System prompt               │
│ Conversation                │
│ Retrieved documents         │
│ Tool schemas                │
│ User query                  │
│ Generated answer            │
└─────────────────────────────┘

All compete for context budget.

---

Context window ≠ memory

Very important.

A context window is:

Information available to the model during this inference.

Long-term memory is:

Information stored outside the model and retrieved into future contexts.

Example:

Postgres / vector DB
        ↓ retrieval
Relevant memories
        ↓
Current model context

The model does not inherently "remember" something just because it saw it in an earlier API call.

---

2.11 Attention Limits

Standard self-attention compares token relationships.

For sequence length (n):

[ QK^T ]

produces approximately an:

[ n \times n ]

attention matrix.

Traditional attention therefore has roughly:

[ O(n^2) ]

interaction complexity with sequence length.

If sequence length doubles:

n → 2n

the number of pairwise interactions becomes approximately:

n² → 4n²

This historically made very long contexts expensive.

Modern implementations use substantial optimizations, including memory-efficient attention implementations, architectural modifications and sparsity strategies, but longer context is still not free.

---

2.12 Lost-in-the-Middle

A model with a 100K+ context window does not necessarily use every token equally well.

Models can perform better when relevant information appears:

near beginning
or
near end

and worse when critical information is buried deep in the middle.

This phenomenon is often called:

lost in the middle.

Example:

Token 1
Important instructions
...
Token 40,000
Critical contract clause   ← model may underweight this
...
Token 90,000
Question

---

Architect implication

This is why:

"Our model has a huge context window, therefore we don't need RAG"

is usually poor architecture reasoning.

A huge context window does not eliminate:

• retrieval quality problems
• relevance selection
• cost
• latency
• distraction
• authorization filtering
• citation requirements
• information freshness

Better architecture:

Large data corpus
      ↓
Retrieve relevant subset
      ↓
Rerank
      ↓
Assemble concise context
      ↓
LLM

Long context complements retrieval; it does not automatically replace it.

---

2.13 KV Cache

KV cache is one of the most important inference concepts.

Remember attention:

Q
K
V

During autoregressive generation:

Token 1
Token 2
Token 3
...

previous tokens do not change.

Without caching, the model would repeatedly calculate their K and V vectors.

Example:

Generate token 101
→ calculate tokens 1–100
Generate token 102
→ calculate tokens 1–101 again
Generate token 103
→ calculate tokens 1–102 again

Extremely wasteful.

Instead:

Tokens 1–100
       ↓
Store K/V representations
       ↓
KV cache

For token 101:

calculate Q/K/V only for new token
+
reuse stored K/V for previous tokens

---

Conceptually

Prompt
 ↓
Prefill
 ↓
KV cache created
 ↓
Generate next token
 ↓
append new K/V
 ↓
Generate next token
 ↓
append new K/V
 ...

---

Why KV cache matters

It drastically improves generation efficiency.

But it consumes GPU memory.

KV-cache memory increases with things such as:

• sequence length
• concurrent requests
• layer count
• number/dimensions of KV heads
• numerical precision

Therefore:

Longer contexts
+
higher concurrency
→ much larger KV memory demand

---

MHA vs MQA vs GQA

Classic multi-head attention may maintain separate K/V representations for every head.

Optimizations include:

#### MQA: Multi-Query Attention

Multiple query heads share one set of K/V heads.

#### GQA: Grouped-Query Attention

Groups of query heads share K/V heads.

Conceptually:

MHA:
Q1 K1 V1
Q2 K2 V2
Q3 K3 V3
Q4 K4 V4
GQA:
Q1 ┐
Q2 ├─ K1 V1
Q3 ┐
Q4 ├─ K2 V2
MQA:
Q1 ┐
Q2 │
Q3 ├─ K V
Q4 │

These can reduce KV-cache requirements and improve inference efficiency.

---

2.14 Prefill vs Decode

This distinction is extremely useful when discussing inference.

Prefill phase

The initial prompt:

10,000 input tokens

can largely be processed in parallel.

This is relatively compute intensive.

During prefill:

Prompt tokens
     ↓
Transformer
     ↓
KV cache populated

---

Decode phase

Then tokens are generated sequentially:

token 1
   ↓
token 2
   ↓
token 3

Decode is often strongly constrained by memory bandwidth/KV movement and sequential dependency.

Thus inference has two distinct performance phases:

                    INFERENCE
Prompt ───────► PREFILL ───────► DECODE ─────► response
                parallel-ish     sequential

This distinction helps explain:

• TTFT
• tokens/second
• batching behaviour
• prompt caching
• hardware utilisation

---

2.15 Decoding

The model doesn't directly output text.

It produces a probability distribution.

Example:

Paris       0.64
Lyon        0.10
London      0.06
Berlin      0.04
...

The decoder determines how to select the next token.

---

2.15.1 Greedy Decoding

Always choose the highest probability token.

Paris 0.64 ← choose
Lyon  0.10
...

Advantages:

• deterministic-ish
• simple
• low randomness

Problems:

• repetitive
• locally optimal choices may lead to weak overall sequences
• poor diversity

---

2.15.2 Temperature

Temperature changes the probability distribution before sampling.

Conceptually:

Low temperature
→ sharper distribution
→ more predictable
High temperature
→ flatter distribution
→ more diverse

Example:

#### Low

Paris       .90
Lyon        .04
London      .02

#### Higher

Paris       .45
Lyon        .18
London      .12

Temperature mathematically scales logits before softmax.

[ softmax(z/T) ]

---

Architect heuristic

For:

classification
extraction
structured workflows
tool arguments
policy decisions

you generally want lower variability.

For:

brainstorming
creative writing
ideation

more sampling diversity may be acceptable.

But temperature is not a correctness control.

Setting:

temperature = 0

does not guarantee truth.

---

2.15.3 Top-K

Restrict sampling to the K highest-probability tokens.

Example:

Vocabulary: 100,000 tokens
top-k = 5
Only 5 most probable tokens remain candidates.

---

2.15.4 Top-P / Nucleus Sampling

Instead of selecting exactly K tokens, choose the smallest group whose cumulative probability reaches P.

Suppose:

A .50
B .25
C .10
D .07
E .03
...

For:

top-p = .90

sampling considers:

A + B + C + D
≈ .92

The candidate set dynamically changes depending on distribution shape.

---

Top-K vs Top-P

Top-K:
fixed number of candidates
Top-P:
dynamic number covering probability mass

---

2.15.5 Stop Sequences

A stop sequence instructs the inference engine to stop when certain generated content appears.

Example:

stop = ["</answer>"]

or:

stop = ["\nUser:"]

Useful for controlling response boundaries.

But do not use stop sequences as a security control.

---

2.16 Hallucination

A hallucination is generated content that is unsupported, false or invented while appearing plausible.

The fundamental reason is important:

An LLM is optimizing probable token continuation, not querying an internal database for verified truth.

Given:

Who won obscure award X in 1974?

the model may create a statistically plausible answer.

---

Why hallucinations happen

Several categories:

#### 1. Missing knowledge

The model simply does not know.

#### 2. Ambiguous prompt

Insufficient constraints.

#### 3. Retrieval failure

RAG retrieves irrelevant or incomplete documents.

#### 4. Context conflict

Two documents disagree.

#### 5. Forced answering

Prompt structure encourages answering even when evidence is absent.

#### 6. Long-context degradation

Evidence exists but isn't properly attended to.

#### 7. Reasoning error

Relevant facts exist but model combines them incorrectly.

#### 8. Tool failure

Tool returned incomplete or stale information.

#### 9. Model tendency toward plausible continuation

The core generative mechanism rewards plausible language.

---

You cannot "eliminate hallucination" with prompt engineering

Architectural mitigation requires layers:

User request
   ↓
Input validation
   ↓
Knowledge retrieval / tools
   ↓
Relevant authoritative evidence
   ↓
LLM
   ↓
Structured generation
   ↓
Validation
   ↓
Grounding / citation checks
   ↓
Business rules
   ↓
Human review where required

---

Controls

Use:

• RAG
• tool access
• grounding
• authoritative sources
• citations
• structured output
• deterministic validation
• confidence thresholds
• refusal / abstention
• secondary verification
• human approval
• monitoring
• evaluation datasets

---

Critical architect principle

Separate:

What the LLM proposes

from:

What the system permits

Example:

LLM:
"Approve ₹8 crore transaction"
        ↓
Policy engine:
User limit = ₹10 lakh
        ↓
DENY

The policy decision should not depend solely on an LLM having "reasoned correctly."

---

2.17 Reasoning Models

"Reasoning model" generally refers to models optimized to perform stronger multi-step problem solving, frequently using additional inference-time computation.

Instead of:

Prompt
 ↓
immediate answer

conceptually:

Prompt
 ↓
additional internal computation
 ↓
evaluate intermediate possibilities
 ↓
answer

The exact mechanisms differ between models and providers.

---

Test-time / inference-time compute

An important direction is:

More compute during inference
        ↓
potentially stronger reasoning

rather than depending only on:

larger pretrained model

This can improve tasks such as:

• mathematics
• coding
• planning
• difficult analysis
• multi-step logic

But costs may increase through:

• latency
• compute
• reasoning-token usage
• output delay

---

Reasoning model ≠ guaranteed correct

A model can produce an elegant chain of reasoning based on a false assumption.

Therefore:

reasoning
≠
verification

For high-consequence systems:

Reasoning model
        ↓
candidate decision
        ↓
independent verifier / tool / rule

---

2.18 Multimodal Models

A multimodal model can process more than one modality.

Examples:

text
images
audio
video
documents

Potential request:

Invoice image
   +
"Extract supplier and amount"

or:

architecture diagram
   +
"Find security problems"

---

Simplified architecture

Image example:

Image
 ↓
vision encoder / representation
 ↓
multimodal token representation
 ↓
language model
 ↓
text output

The architecture differs among model families.

---

Enterprise use cases

• document understanding
• OCR enhancement
• chart interpretation
• claims processing
• manufacturing inspection
• medical document workflows
• screenshot analysis
• UI testing
• video analysis
• call transcription + analysis

---

Important risk

Multimodal does not mean deterministic perception.

For example:

image → LLM says invoice total = ₹14,500

still requires validation for financial workflows.

---

2.19 Structured Outputs

Normal LLM output:

"The customer's risk rating appears to be high..."

Applications often need:

{
  "risk": "HIGH",
  "score": 83
}

Structured output capabilities constrain or strongly guide the model toward a schema.

Example conceptual schema:

{
  "type": "object",
  "properties": {
    "risk": {
      "enum": ["LOW", "MEDIUM", "HIGH"]
    },
    "score": {
      "type": "integer"
    }
  }
}

---

Why structured outputs matter

Without them:

LLM
 ↓
free text
 ↓
regex/parser
 ↓
fragile application

With structured outputs:

LLM
 ↓
schema-constrained generation
 ↓
typed object
 ↓
application

Much safer operationally.

---

But schema correctness ≠ semantic correctness

The model may return perfectly valid JSON:

{
  "invoice_amount": 9000000
}

when the actual invoice says:

90000

So distinguish:

syntactic validity

from:

semantic validity

You still need domain validation.

---

2.20 Function / Tool Calling

Tool calling allows an LLM to request that an external capability be executed.

Example tools:

get_customer(customer_id)
lookup_weather(city)
create_ticket(...)
query_database(...)
send_email(...)

The model receives tool definitions.

Example:

Tool:
get_customer_balance
Parameters:
customer_id: string

User:

"What is customer C123's balance?"

Model produces conceptually:

{
  "tool": "get_customer_balance",
  "arguments": {
    "customer_id": "C123"
  }
}

---

The model normally does NOT execute the tool itself

This distinction is critical.

User
 ↓
LLM
 ↓
Tool call request
 ↓
ORCHESTRATOR
 ↓
validate permissions
 ↓
execute actual function/API
 ↓
tool result
 ↓
LLM
 ↓
response

The runtime executes the tool.

---

Enterprise control plane

Never:

LLM
 ↓
direct unrestricted production API

Prefer:

LLM
 ↓
requested action
 ↓
Schema validation
 ↓
Identity / authorization
 ↓
Policy
 ↓
Approval if required
 ↓
Tool execution
 ↓
Audit log

An LLM chooses intent.

Your deterministic system decides whether that intent is allowed.

---

Example

LLM emits:

{
  "tool": "refund_customer",
  "amount": 100000
}

Orchestrator checks:

Agent identity
Tenant
Customer
Refund limit
User authorization
Approval requirement
Idempotency
Risk policy

Then:

execute
or
reject

---

Tool-call failure modes

Know these:

• wrong tool selected
• fabricated arguments
• missing arguments
• incorrect argument types
• unauthorized operation
• tool unavailable
• timeout
• non-idempotent retry
• prompt injection through tool result
• malicious tool output
• excessive tool loops
• stale tool results

Tool calling is therefore an orchestration problem as much as an LLM problem.

---

2.21 Model Context Limits

There are several limits people casually collapse into "context window."

You should think separately about:

Maximum total context
Maximum input
Maximum output
Provider-specific reasoning budget
Tool/schema overhead

Conceptually:

[ Context = System + Conversation + Prompt + RAG + ToolSchemas + ToolResults + GeneratedTokens ]

If:

maximum = 128K

you cannot assume you have:

128K for source documents

because everything else consumes capacity.

---

Context budgeting

Suppose:

Maximum context                  128K
System instructions                3K
Conversation history              15K
Tool definitions                   5K
Expected response                  5K
Safety margin                      5K
-------------------------------------
Available for retrieved context   95K

But blindly filling all 95K is still often undesirable.

Relevant context is usually better than maximal context.

---

2.22 Quantization

Model weights are numbers.

They may originally be represented at precisions such as:

FP32
FP16
BF16

Quantization stores/executes them using lower precision, for example:

INT8
INT4

or related low-bit formats.

---

Why?

Suppose a model has:

70 billion parameters

Very approximately:

#### FP16

70B × 2 bytes
≈ 140 GB just for weights

#### INT8

≈ 70 GB

#### INT4

≈ 35 GB

There is additional runtime memory overhead, so these numbers are not total server-memory requirements.

---

Benefits

Quantization can reduce:

• memory consumption
• hardware requirements
• memory bandwidth
• deployment cost

and sometimes increase inference speed.

---

Trade-off

Potential loss of:

• accuracy
• numerical fidelity
• reasoning quality

depending on:

• quantization method
• bit depth
• model
• workload

Modern quantization techniques can preserve surprisingly strong quality.

---

Architect decision

If deploying a model yourself:

Accuracy
Latency
Throughput
GPU memory
Cost
Model size
Context requirement
Concurrency

must be evaluated together.

Do not say:

"INT4 is always better because it is cheaper."

Benchmark against your actual task.

---

2.23 Inference Architecture

A production self-hosted LLM system may look like:

                       ┌──────────────┐
Users ─► API Gateway ─►│ AI Gateway   │
                       └──────┬───────┘
                              │
                    authentication
                    quotas
                    routing
                    safety
                    observability
                              │
                              ▼
                     Request Scheduler
                              │
                   ┌──────────┴──────────┐
                   │                     │
                   ▼                     ▼
             GPU Worker 1           GPU Worker N
             Model shard            Model shard
                   │                     │
                   └──────────┬──────────┘
                              ↓
                          Streaming
                              ↓
                            Client

Additional components can include:

• model registry
• tokenizer
• prompt cache
• KV-cache manager
• batching engine
• autoscaler
• GPU scheduler
• rate limiter
• model router
• guardrails
• telemetry
• evaluation pipeline

---

Model parallelism

A model may not fit onto one GPU.

Then weights can be distributed.

#### Tensor parallelism

Split tensor operations across GPUs.

Layer
 ├─ GPU 1
 ├─ GPU 2
 ├─ GPU 3
 └─ GPU 4

#### Pipeline parallelism

Different stages/layers reside on different GPUs.

GPU 1 → layers 1–20
GPU 2 → layers 21–40
GPU 3 → layers 41–60

#### Data parallelism

Replicate model across workers to serve/train different workloads.

Inference systems frequently combine techniques.

---

2.24 GPU Fundamentals

You do not need to become a CUDA engineer, but an AI Architect should understand why GPUs matter.

CPUs contain relatively few powerful general-purpose cores.

GPUs contain massive parallel computation capability.

Neural networks heavily use:

matrix multiplication

For example:

[ Y = XW ]

where (X) and (W) can be huge matrices.

GPUs are exceptionally effective at these parallel numerical operations.

---

GPU components you should know conceptually

#### Compute units / CUDA-style cores

General parallel numerical operations.

#### Tensor cores

Specialized matrix operations useful for deep learning.

#### VRAM / HBM

Very high bandwidth memory holding:

• model weights
• activations
• KV cache
• intermediate tensors

#### Memory bandwidth

How rapidly data moves between GPU memory and computation units.

This is critical for LLM inference.

---

Training vs inference

Training requires:

forward pass
+
loss calculation
+
backpropagation
+
optimizer states

Inference requires primarily:

forward pass

Training therefore requires much more memory/compute.

---

Compute-bound vs memory-bound

During parts of inference:

GPU computation capacity

may be the bottleneck.

Elsewhere:

moving model/KV data through memory

may dominate.

LLM decoding is frequently heavily constrained by memory bandwidth.

This explains why simply quoting:

GPU FLOPS

doesn't tell you actual inference performance.

---

2.25 Throughput vs Latency

These are different system objectives.

Latency

How long one request takes.

Example:

User request
      ↓
      2.4 seconds
      ↓
First/complete response

---

Throughput

How much total work the system handles.

For LLMs:

requests/sec
tokens/sec

Example:

1 user:
100 tokens/sec
100 concurrent users:
8,000 aggregate tokens/sec

---

Trade-off

Batching may improve:

throughput

while hurting:

individual request latency

So you cannot optimize one number in isolation.

---

2.26 Time-to-First-Token: TTFT

TTFT is:

Time from request submission until the first output token reaches the user.

Request
   ↓
Queue
   ↓
Tokenize
   ↓
Prefill
   ↓
First decode
   ↓
FIRST TOKEN   ← TTFT

TTFT depends heavily on:

• queue time
• prompt length
• prefill processing
• model size
• batching
• hardware
• caching
• network overhead

---

Why TTFT matters

For interactive applications, perceived responsiveness is strongly related to:

how quickly generation starts

A response that starts streaming in 500 ms and finishes in six seconds can feel better than one that waits five seconds and appears all at once.

---

2.27 Tokens Per Second

After generation begins:

Token 1
Token 2
Token 3
...

generation speed is often measured as:

output tokens / second

Suppose:

TTFT = 600ms
Generation = 50 tokens/sec
Output = 500 tokens

Generation alone is approximately:

500 / 50 = 10 seconds

So total user-visible latency is roughly:

TTFT + decode time

ignoring other overhead.

---

TTFT vs Tokens/sec

Know this cold:

TTFT
= how fast output begins
Tokens/sec
= how fast output continues

Two models can have:

same TTFT
different generation speed

or vice versa.

---

2.28 Batching

GPUs work efficiently when processing larger workloads in parallel.

Instead of:

Request A → GPU
Request B → GPU
Request C → GPU

you can batch:

A
B ─────► GPU batch
C
D

This improves GPU utilization and throughput.

---

Static batching problem

Requests have different:

prompt lengths
generation lengths
arrival times

Waiting to fill a fixed batch can increase latency.

Modern inference systems therefore often use:

#### Continuous/dynamic batching

As requests arrive and finish:

Time 1:
[A B C D]
Time 2:
[A B   D]   C finished
Time 3:
[A B E D]   E enters

This allows the scheduler to continuously use available compute capacity.

---

Batching trade-off

Larger batches
   ↓
higher GPU utilisation
   ↓
higher throughput
BUT
larger queue/batch pressure
   ↓
potentially higher per-request latency

The architect chooses based on SLA.

---

2.29 Prompt Caching

Many applications repeatedly send identical prefixes.

Example:

System prompt          4K tokens
Policy document       20K
Tool definitions       5K
--------------------------
Repeated prefix       29K

for every request.

Instead of repeatedly processing it:

Repeated prefix
       ↓
cached representation
       ↓
reuse

Prompt caching can reduce:

• input computation
• latency
• sometimes API cost

---

Example

Without cache:

Request 1:
process 30K prompt
Request 2:
process 30K prompt
Request 3:
process 30K prompt

With caching:

Request 1:
process 30K → cache
Request 2:
reuse matching prefix
Request 3:
reuse matching prefix

---

Prompt cache vs KV cache

This distinction frequently causes confusion.

KV cache

Used within an inference/generation lifecycle to avoid recomputing previous token K/V representations.

Prompt caching

Allows repeated prompt/prefix computation to be reused across requests or request executions, depending on the inference system/provider.

Conceptually:

KV CACHE
Current generation efficiency
PROMPT CACHE
Repeated request-prefix efficiency

---

Cache invalidation consideration

If your cached prompt contains:

policy
instructions
tools
tenant configuration

and they change, the cache must not incorrectly continue using stale state.

Cache identity may need to include:

model
model version
prompt version
tenant
policy version
tool version

depending on implementation.

---

2.30 Putting Model Inference Together

You should be able to whiteboard this entire flow:

                     USER REQUEST
                          │
                          ▼
                    API / Gateway
                          │
                          ▼
                     Tokenizer
                          │
                          ▼
              ┌─────────────────────┐
              │ Prompt cache check  │
              └──────────┬──────────┘
                         │
                         ▼
                       PREFILL
                         │
                    GPU execution
                         │
                         ▼
                    Build KV cache
                         │
                         ▼
                      DECODING
                         │
             ┌───────────┴──────────┐
             │                      │
       Transformer             KV cache
             │                      │
             └──────────┬───────────┘
                        ↓
                Next-token logits
                        ↓
                 Decoding policy
            temperature/top-p/etc.
                        ↓
                     token
                        ↓
                 append to context
                        ↓
              append to KV cache
                        ↓
                  repeat until:
                ┌───────┼─────────┐
                │       │         │
              EOS    stop seq   max tokens
                        │
                        ▼
                     RESPONSE

---

2.31 Complete Production Mental Model

Now put the LLM inside an enterprise system:

                           CLIENT
                              │
                              ▼
                     API / AI Gateway
                              │
                 ┌────────────┴────────────┐
                 │ AuthN / AuthZ           │
                 │ Tenant resolution       │
                 │ Rate limits             │
                 │ Cost controls           │
                 └────────────┬────────────┘
                              │
                              ▼
                       ORCHESTRATOR
                              │
          ┌───────────────────┼────────────────────┐
          │                   │                    │
          ▼                   ▼                    ▼
         RAG                Memory                Tools
          │                   │                    │
          └───────────────────┼────────────────────┘
                              ↓
                       Context Builder
                              │
                              ↓
                         LLM Gateway
                              │
                  ┌───────────┼───────────┐
                  │           │           │
                Model A     Model B     Model C
                  │
                  ↓
           Structured response
                  │
                  ↓
              Validation
                  │
           ┌──────┴────────┐
           │               │
        accepted        rejected
           │               │
           ▼               ▼
     business flow      retry/escalate

This is the level at which an architect should think.

The model itself is only one component.

---

2.32 Key Architect Trade-Offs

Model size

larger model
→ potentially stronger capability
→ more memory
→ more compute
→ higher latency/cost

Choose based on task, not prestige.

---

Context size

larger context
→ potentially more evidence
→ higher cost
→ higher TTFT
→ larger KV cache
→ greater distraction risk

---

Output length

longer output
→ more sequential decoding
→ higher latency
→ higher cost

Often output tokens are more expensive operationally than input tokens because they require sequential generation.

---

Quantization

lower precision
→ lower memory/cost
→ potentially higher speed
→ possible quality reduction

---

Batching

more batching
→ throughput ↑
→ utilization ↑
→ latency may ↑

---

Reasoning depth

more inference-time reasoning
→ potential accuracy ↑
→ latency ↑
→ cost ↑

---

Model vs system controls

Never try to solve everything by selecting a "smarter model."

Example:

Problem: customer-specific data leakage
Wrong:
Use larger LLM.
Correct:
Tenant-aware authorization and retrieval.

Likewise:

Problem: invoice amount must be <= PO balance
Wrong:
Prompt model to be careful.
Correct:
Deterministic validation.

---

2.33 What Is Probabilistic vs Deterministic?

This distinction is central to enterprise architecture.

#### Probabilistic

LLM generation
classification
summarization
reasoning
entity interpretation
tool selection
semantic retrieval scores

#### Deterministic controls

authentication
authorization
schema validation
financial limits
tenant isolation
workflow state transitions
idempotency
policy enforcement
database constraints
audit logging
approval gates

A strong architecture looks like:

          PROBABILISTIC
               │
         propose / interpret
               │
               ▼
        DETERMINISTIC
       validate / control
               │
               ▼
             ACT

Not:

LLM said yes
     ↓
do it

---

2.34 Common Interview Traps

#### Trap 1

"Temperature zero eliminates hallucination."

Wrong.

It reduces sampling variability, not factual error.

---

#### Trap 2

"A 1M-token context eliminates RAG."

Wrong.

Retrieval still addresses:

• relevance
• authorization
• freshness
• cost
• citations
• noise

---

#### Trap 3

"Function calling means the model calls APIs."

Incomplete.

Usually:

model proposes function call
runtime validates and executes it

---

#### Trap 4

"Valid JSON means correct answer."

Wrong.

Schema correctness ≠ semantic correctness.

---

#### Trap 5

"Embedding is what the transformer uses internally."

Only partly.

Distinguish:

token embeddings

from:

semantic/document embeddings

---

#### Trap 6

"More parameters always means better architecture."

Wrong.

Production decisions consider:

task accuracy
latency
cost
throughput
privacy
deployment
context
operability

---

#### Trap 7

"Long context means perfect recall."

Wrong.

Lost-in-the-middle and attention degradation exist.

---

#### Trap 8

"Quantization only reduces model size."

Incomplete.

It can affect:

• memory bandwidth
• throughput
• deployment footprint
• accuracy

---

2.35 Interview Questions You Should Be Able to Answer

Q1. Explain how an LLM generates text.

30-second answer:

The prompt is tokenized and converted to learned representations. Those representations pass through stacked transformer blocks containing self-attention and feed-forward layers. The model produces logits representing probabilities for the next token. A decoding strategy selects a token, it is appended to the sequence, and the process repeats autoregressively until a stop condition is reached.

---

Q2. Explain self-attention.

Each token produces query, key and value representations. Query-key similarity determines how strongly that token should attend to previous tokens, and the corresponding values are combined using those attention weights. Multi-head attention performs this through multiple learned attention projections in parallel, allowing the model to capture different relationships.

---

Q3. Why do transformers need positional encoding?

Attention on its own does not encode sequence order. Positional information allows the model to distinguish sequences containing the same tokens in different orders. Modern models commonly use mechanisms such as rotary positional embeddings.

---

Q4. Why is generation slow?

Prompt processing can be highly parallelized during prefill, but autoregressive decoding has a sequential dependency because token N+1 depends on token N. Generation therefore often becomes latency-sensitive and memory-bandwidth-heavy, particularly because the model repeatedly accesses weights and KV-cache data.

---

Q5. What is the KV cache?

During autoregressive decoding, keys and values for previously processed tokens don't change. The inference engine caches them rather than recomputing them for every generated token. This dramatically improves decoding efficiency but consumes memory proportional to context and concurrency.

---

Q6. What is TTFT?

Time-to-first-token measures the latency between submitting a request and receiving the first generated token. It includes queueing, prompt processing or prefill and initial decoding. It is particularly important for interactive user experience.

---

Q7. Throughput versus latency?

Latency measures how long an individual request takes, whereas throughput measures total work served, such as tokens per second or requests per second. Techniques such as batching can improve throughput while potentially increasing individual latency, so optimization depends on the workload SLA.

---

Q8. Why do LLMs hallucinate?

Because their training objective fundamentally optimizes probable token generation rather than verified factual retrieval. Hallucinations can additionally result from missing knowledge, retrieval failures, conflicting context or reasoning errors. I therefore manage hallucination architecturally through grounding, retrieval, tools, validation, abstention and deterministic controls rather than relying purely on prompting.

That is an excellent architect-level answer.

---

Q9. How do you reduce hallucinations in enterprise applications?

I first identify whether the problem is knowledge, reasoning or control. For knowledge, I ground the model through authoritative retrieval or tools. I require citations where appropriate and establish abstention behaviour when evidence is insufficient. Outputs are schema validated and important domain facts are independently checked. High-consequence actions pass through deterministic policy and approval controls rather than allowing the LLM to act directly.

---

Q10. Explain tool calling.

Tool calling allows the model to produce a structured request identifying an external function and its arguments. The application runtime, not the LLM, should authenticate the request, validate parameters, authorize the action, enforce policy, execute the tool and return the result. This separation is critical because the model's decision remains probabilistic.

---

Q11. What is quantization?

Quantization represents model weights or computation at lower numerical precision, such as reducing FP16 representations toward 8-bit or 4-bit formats. It can substantially reduce GPU memory and memory-bandwidth requirements and therefore deployment cost, but quality and hardware performance must be benchmarked for the target workload.

---

Q12. Temperature vs top-p?

Temperature reshapes the entire probability distribution, making sampling sharper or flatter. Top-p restricts sampling to the smallest candidate set whose cumulative probability reaches a threshold. Both influence generation diversity but neither should be treated as a factual-correctness control.

---

Q13. Why use RAG if the model supports huge contexts?

Context capacity and information relevance are different problems. Putting an entire corpus into a large context creates cost, latency, authorization and distraction issues and doesn't guarantee the model will use evidence equally well across the sequence. RAG lets me retrieve a tenant-authorized, relevant and fresh evidence set before generation.

---

2.36 Whiteboard Question: "Design an Enterprise LLM Platform"

Start here:

                         Enterprise Applications
                                  │
                                  ▼
                             AI Gateway
                                  │
                ┌─────────────────┼─────────────────┐
                │                 │                 │
             Identity          Quotas            Audit
                │
                ▼
                         Model Orchestrator
                ┌─────────────────┼──────────────────┐
                │                 │                  │
                ▼                 ▼                  ▼
              RAG              Tools              Memory
                │                 │                  │
                └─────────────────┼──────────────────┘
                                  ↓
                           Context Builder
                                  │
                                  ↓
                            Model Router
                      ┌───────────┼───────────┐
                      ▼           ▼           ▼
                    Small       Large      Reasoning
                    model       model        model
                      │
                      ▼
                         Structured Output
                                  │
                                  ▼
                              Validator
                                  │
                     ┌────────────┼────────────┐
                     ▼            ▼            ▼
                  Policy       Human Gate    Execute
                     │
                     ▼
                Observability
                     │
        ┌────────────┼───────────────┐
        ▼            ▼               ▼
      Cost        Quality         Security

Then explain:

• model routing based on task
• tenant-aware RAG
• deterministic tool authorization
• structured outputs
• evaluation
• prompt/model versioning
• cost/latency monitoring
• traceability
• fallback strategy
• human approval for high-risk actions

That turns an LLM-fundamentals answer into an architect answer.

---

2.37 Metrics You Should Know Cold

For inference:

Metric	Meaning
Input tokens	Prompt/context size
Output tokens	Generated sequence size
TTFT	Time before first token appears
TPOT	Time per output token
Tokens/sec	Generation speed
End-to-end latency	Total request duration
Throughput	Aggregate served work
Requests/sec	Concurrent request capacity
GPU utilisation	Compute utilization
GPU memory utilisation	Weight/cache/runtime memory
Cache hit rate	Reused prompt/cache opportunities
Queue latency	Time waiting for inference
Error rate	Failed requests
Cost/request	Workload economics
Cost/successful task	Usually more meaningful than raw token cost

That last metric is especially important.

A cheap model costing:

₹0.20/request

but succeeding only 60% of the time may be worse than a model costing:

₹0.60/request

with 98% success.

Architectural optimization should be:

[ \text{Cost per successful business outcome} ]

not merely:

[ \text{Cost per token} ]

---

2.38 Model Selection Framework

If asked:

How would you choose an LLM?

Don't answer only with benchmark scores.

Use:

                    MODEL SELECTION
Functional capability
├── task accuracy
├── reasoning
├── coding
├── multilingual
├── multimodal
├── tool calling
└── structured output
Operational
├── latency
├── throughput
├── context length
├── rate limits
└── availability
Economic
├── input cost
├── output cost
├── reasoning cost
└── infrastructure cost
Security
├── data residency
├── retention
├── provider controls
└── private deployment
Enterprise
├── SLA
├── compliance
├── support
├── version stability
└── observability
Architecture
├── API
├── self-host
├── fine-tuning
├── quantization
└── routing capability

And then say:

I would benchmark candidate models against a representative evaluation dataset from the actual business workload rather than selecting purely from public benchmarks.

That is the right architecture answer.

---

2.39 The Model Is Not the Application

This is probably the single most important conclusion of this topic.

Weak design:

User
 ↓
LLM
 ↓
Business action

Enterprise design:

                  Identity
                     │
                     ▼
User ─► Orchestrator ─► Context / RAG
                     │
                     ├─► Memory
                     │
                     ├─► LLM
                     │
                     └─► Tools
                            │
                         Policy
                            │
                        Validation
                            │
                         Approval
                            │
                         Execution
                            │
                           Audit

The LLM provides capabilities such as:

interpretation
reasoning
generation
planning
semantic understanding

The surrounding architecture provides:

truth sources
security
state
identity
permissions
reliability
determinism
governance
observability
business rules

---

2.40 What You Should Know Cold Before Moving On

You should be able to explain these without notes:

#### Transformer mechanics

Tokens
↓
Embeddings
↓
Positional information
↓
Q/K/V
↓
Self-attention
↓
Multi-head attention
↓
FFN
↓
Residual + normalization
↓
Next-token probabilities

#### Generation

Prompt
↓
Prefill
↓
KV cache
↓
Decode token
↓
append K/V
↓
decode next token

#### Training lifecycle

Pretraining
↓
Instruction/SFT
↓
Preference alignment
   ├─ RLHF
   ├─ RLAIF
   └─ DPO

#### Sampling

Greedy
Temperature
Top-k
Top-p
Stop sequences

#### Performance

TTFT
tokens/sec
throughput
latency
batching
KV cache
prompt caching
GPU memory

#### Architecture

LLM ≠ database
LLM ≠ policy engine
LLM ≠ authorization system
LLM ≠ memory system
LLM ≠ verifier

And the most important sentence:

The LLM is a probabilistic inference engine embedded inside a larger deterministic enterprise control system.

If you can defend that sentence and everything underneath it, you have the foundation needed for the later RAG, agent, state/memory, tool-use, governance and production architecture sections.

---

Related reading

• LLMs Aren't Magic: What CXOs Must Know Before Going In-House, the same mechanics translated for a non-technical audience.
• Enterprise LLM Deployment Cost in India, what these throughput and latency properties cost in practice.
• AI FinOps: A Practical Framework to Control Enterprise AI Cost, turning token economics into a controllable budget.