The AI Architect Operating Model: Turning a Business Objective into an Architecture

This is where the handbook starts, because it is the operating system for everything else. Once this becomes instinctive, RAG, agents, LLMOps, cloud and security stop being disconnected technologies. They become architectural choices made against business requirements.

The target is that somebody can put an unfamiliar AI problem in front of you and you can still structure a strong answer.

The fundamental shift is this:

Developer question:
"How do I build this?"
Architect question:
"What system should exist, why should it exist in this form,
what can go wrong, and how will we know it is working?"

An architect continuously moves through this chain:

BUSINESS OUTCOME
      ↓
USE CASE / ACTORS
      ↓
FUNCTIONAL REQUIREMENTS
      ↓
NON-FUNCTIONAL REQUIREMENTS
      ↓
RISK + CONSTRAINTS
      ↓
ARCHITECTURE OPTIONS
      ↓
TRADE-OFF
      ↓
DECISION
      ↓
IMPLEMENTATION
      ↓
OPERATING CONTROLS
      ↓
MEASUREMENT
      ↓
FEEDBACK / EVOLUTION

Never begin an architecture interview with technology.

If the interviewer says:

"Design an AI system that detects invoice leakage."

Don't begin with:

"I'll use LangGraph with Bedrock and Pinecone."

Begin with:

"Before choosing the implementation, I'd establish what constitutes leakage, whether the system only detects it or is expected to take corrective action, the financial impact of false positives and false negatives, where the source-of-truth data lives, the required response time, and the level of autonomy we're prepared to allow."

That is architect behaviour.

---

1.1 Business objective → architecture

Architecture exists to achieve an outcome under constraints.

Consider two superficially similar requirements.

#### System A

Answer employees' HR-policy questions.

#### System B

Detect contractual overbilling and automatically recover money.

Both might contain:

LLM
RAG
Agents
Tools

But architecturally they are radically different.

For the HR assistant:

wrong answer
   ↓
moderate operational impact
Typical architecture:
RAG
citations
access control
feedback
good observability

For financial execution:

wrong decision
      ↓
wrong action
      ↓
financial / contractual consequence

Therefore:

Evidence
   ↓
Reasoning
   ↓
Structured proposed action
   ↓
Policy
   ↓
Authorization
   ↓
Approval threshold
   ↓
Execution
   ↓
Verification
   ↓
Immutable evidence

Business risk changed the architecture.

In a platform organisation, the business objective might instead be something such as:

Build a reusable enterprise AI capability for many customers, business units or application teams.

Now architecture is influenced by entirely different concerns:

reuse
standardisation
multi-tenancy
provider abstraction
governance
regional deployment
self-service
security
cost attribution
platform operations

Same AI technologies.

Different business objective.

Different architecture.

---

1.2 Functional vs non-functional requirements

This distinction must become automatic.

Functional requirements

They describe what the system does.

Example:

Requirement
Upload a contract
Extract clauses
Retrieve relevant terms
Compare contract against invoice
Detect discrepancy
Generate explanation
Create dispute
Ask finance for approval
Update ERP

Non-functional requirements

They describe how well, under what conditions, and within what boundaries the system must operate.

Think across these dimensions:

Category	Architect question
Availability	How often must it be reachable?
Reliability	How often must it behave correctly?
Latency	How quickly must results appear?
Throughput	How much work must it process?
Scalability	What happens at 10× or 100× load?
Durability	What information can never be lost?
Consistency	When must everybody see the same state?
Security	Who can access or perform what?
Privacy	What data can be processed/stored?
Isolation	Can one customer ever see another's data?
Compliance	Which legal/regulatory controls apply?
Auditability	Can a decision be reconstructed?
Maintainability	How safely can the system change?
Portability	How dependent are we on one provider?
Observability	Can failures and degradation be detected?
Cost	What unit economics must be maintained?
DR	What happens after catastrophic failure?

AI introduces additional NFRs:

AI NFR	Example
Groundedness	Answer supported by retrieved evidence
Task-success rate	Agent successfully completes requested task
Tool correctness	Correct tool with correct parameters
Safety	No prohibited or unauthorized action
Hallucination tolerance	Depends heavily on use case
Evaluation threshold	Release blocked below defined quality
Model portability	Ability to change provider/model
Explainability	Evidence behind recommendation available
Autonomy boundary	Maximum action an agent may execute
Cost/task	Maximum acceptable AI execution cost

A common weak architecture answer spends 90% of its time on functional flow and almost none on NFRs.

At senior level, NFRs often determine the architecture more than functional requirements do.

---

1.3 Build vs buy vs managed service

You will face this constantly.

There are really three choices:

BUILD
You own implementation + operation.
BUY
Another product provides the capability.
MANAGED SERVICE
You configure/use a capability,
while the provider operates much of the infrastructure.

Consider model inference.

You could:

Self-host model
       vs
Managed model endpoint
       vs
Managed GenAI service

The architect evaluates more than feature capability.

Dimension	Build tends to win when…	Managed/buy tends to win when…
Differentiation	Capability is strategic IP	Capability is commodity
Control	Deep customization required	Standard capability sufficient
Time	Development time acceptable	Time-to-market critical
Operations	Team has expertise	Provider operation is valuable
Compliance	Full infrastructure control needed	Managed service meets requirements
Scale	Economics justify ownership	Variable demand favours managed
Portability	Provider independence critical	Lock-in acceptable
Innovation	Unique capability matters	Ecosystem velocity matters

One extremely useful architect principle is:

Build differentiation. Buy or manage commodity capability unless there is a concrete reason not to.

For enterprise AI, your differentiating assets are often things such as:

business workflow
domain ontology
proprietary data
agent policy
evaluation assets
integration logic
customer experience
institutional knowledge

Not:

running GPUs
building a vector database
implementing OAuth
writing another queue

There are exceptions, but they require a reason.

#### Avoid the portability trap

Another common architecture mistake is attempting to abstract everything.

Suppose AWS offers feature X and Azure offers feature Y.

You can create:

UniversalEverythingProviderInterface

until every cloud feature is reduced to the lowest common denominator.

Don't.

A better strategy is:

Portable where strategic
Native where advantageous

For example, a model gateway is frequently worth abstracting because switching or routing models is useful.

Abstracting every queue/storage/networking capability may add enormous complexity without business value.

The architect asks:

"What is the cost of lock-in compared with the cost of avoiding lock-in?"

---

1.4 Probabilistic AI inside deterministic control boundaries

This is one of the most important ideas in the entire course.

LLM output is probabilistic.

Enterprise authorization should not be.

Consider:

User:
"Refund this customer's ₹4 lakh payment."
LLM:
"I think this request is valid."

The LLM must not therefore receive authority to transfer ₹4 lakh.

Separate four things:

REASONING
"What appears to be happening?"
DECISION
"What action is appropriate?"
AUTHORIZATION
"Is this actor/system permitted to take that action?"
EXECUTION
"Perform the actual state change."

The model may participate heavily in the first two.

It should not become the security authority for the third.

A strong enterprise architecture looks like:

            UNTRUSTED / PROBABILISTIC
                     |
                  LLM
                     |
          structured action proposal
                     |
                     v
          -----------------------
          DETERMINISTIC BOUNDARY
          -----------------------
                     |
              schema validation
                     |
                policy engine
                     |
              authorization
                     |
             approval if needed
                     |
             tool permission
                     |
                execution
                     |
               verification
                     |
                 audit

The deterministic layer can check:

action_type = refund
amount = 400000
customer_id = X
requested_by = Y
tenant = Z

Against rules such as:

user Y has permission refund:create
AND
amount <= user's approval limit
AND
customer belongs to tenant Z
AND
invoice status permits refund
AND
bank account equals verified account
AND
required approval exists

This is immensely important for autonomous enterprise agents.

The architectural principle is:

The model may propose. The control plane authorizes. The execution layer acts.

That doesn't mean everything must require a human.

It means autonomy exists within deterministic boundaries.

---

1.5 Control plane vs data/runtime plane

This distinction will make many of your answers much cleaner.

Runtime/data plane

This is where actual workload execution occurs.

User request
     ↓
Agent
     ↓
Retrieval
     ↓
Model call
     ↓
Tool call
     ↓
Result

Runtime concerns include:

latency
throughput
state
model execution
retrieval
tool execution
retries
request isolation

Control plane

This defines how the runtime is allowed/configured to operate.

MODEL REGISTRY
PROMPT REGISTRY
AGENT DEFINITIONS
TOOL REGISTRY
POLICIES
TENANT CONFIGURATION
QUOTAS
BUDGETS
MODEL ROUTING RULES
EVALUATION CONFIGURATION
DEPLOYMENTS
FEATURE FLAGS
APPROVAL POLICIES

Conceptually:

                CONTROL PLANE
                     |
       --------------------------------
       |       |        |       |
     Policy   Models   Tools   Tenant cfg
       |       |        |       |
       --------------------------------
                     |
                 configures
                     ↓
               RUNTIME PLANE
 User → Agent → RAG → Model → Tool → Result

Why separate them?

Because configuration has a different lifecycle from requests.

You might change:

model routing
allowed tools
approval threshold
prompt version
tenant quota

without redeploying the runtime application.

It also creates cleaner governance.

---

1.6 Synchronous vs asynchronous execution

This is another decision you should explicitly mention.

#### Synchronous

request
   ↓
processing
   ↓
response

Caller waits.

Best when:

execution is short
result required immediately
failure can be returned directly

Example:

Ask question
→ RAG
→ LLM
→ answer

#### Asynchronous

request
   ↓
job accepted
   ↓
queue
   ↓
worker
   ↓
long-running execution
   ↓
result stored/event emitted

Use this when tasks are:

long-running
bursty
retryable
batch-oriented
dependent on external systems
not required immediately

Examples include:

document ingestion
bulk embedding
large-scale evaluation
long-running agents
invoice reconciliation
report generation
data synchronization

Production asynchronous execution introduces additional architectural requirements:

job ID
queue
durable state
retry policy
idempotency
timeout
dead-letter handling
status model
result storage
cancellation

Most sophisticated AI systems are hybrid.

For example:

User
 ↓
POST /analyse
 ↓
202 Accepted + job_id
 ↓
Queue
 ↓
Agent workflow
 ↓
callback / notification

Meanwhile a conversational request might stay synchronous.

---

1.7 Stateless vs stateful services

A stateless service contains no durable request-specific state inside the running process.

Request
   ↓
API instance
   ↓
DB/cache/state store
   ↓
response

Any API instance can process the next request.

That makes horizontal scaling straightforward.

             Load balancer
           /      |       \
        API1     API2     API3
           \      |       /
             external state

Agent systems, however, are intrinsically stateful at the workflow level.

Suppose an agent performs:

1 Retrieve invoice
2 Retrieve contract
3 Compare
4 Request approval
5 Wait 6 hours
6 Receive approval
7 Update ERP

You cannot keep this inside Python memory for six hours.

You want:

compute = disposable
state   = durable

Persist:

workflow ID
current step
inputs
outputs
decisions
tool results
approval state
retry counters
timestamps

Then if a worker dies:

new worker
   ↓
load checkpoint
   ↓
resume

This principle becomes central later when we study durable agents.

---

1.8 Event-driven architecture

An event describes something that has happened.

InvoiceReceived
ContractUpdated
ApprovalGranted
PaymentFailed
LeakageDetected

A command asks something to happen:

AnalyseInvoice
ApproveRefund
CreateDispute

The distinction matters.

Consider:

ERP
 |
 | InvoiceCreated
 v
Event bus
 |
 +----> Leakage detection
 |
 +----> Analytics
 |
 +----> Audit
 |
 +----> Notification

The ERP does not need knowledge of all downstream consumers.

That's decoupling.

But event systems introduce serious architecture questions:

Can events arrive twice?
Can they arrive out of order?
Can they disappear?
What happens if processing fails?
Can events be replayed?
How is schema evolution handled?

A central principle:

Assume duplicate delivery unless the infrastructure explicitly guarantees otherwise, and make consumers idempotent.

Suppose:

PaymentApproved event

is delivered twice.

Bad consumer:

pay supplier()
pay supplier()

Good consumer:

if event_id already_processed:
    return
execute_payment(idempotency_key=transaction_id)

#### Choreography vs orchestration

Choreography:

A emits event
B reacts
B emits event
C reacts

No central workflow controller.

Orchestration:

Workflow Engine
    |
    +--> A
    |
    +--> B
    |
    +--> C

For complex, financially consequential, long-running workflows, explicit orchestration frequently gives you better visibility and recovery semantics.

For loosely coupled notifications and integration, event choreography can be excellent.

Again: trade-off rather than doctrine.

---

1.9 Multi-tenant architecture

Multi-tenancy does not mean:

"Every table has tenant_id."

That's one control.

Tenant isolation must exist across the system.

AWS's SaaS guidance explicitly treats tenant isolation as foundational and distinguishes pooled/shared environments from stronger silo-style isolation approaches. (AWS Documentation)

Think:

Identity
Data
Retrieval
Memory
Compute
Cache
Queues
Files
Encryption
Tools
Logs
Metrics
Secrets
Rate limits
Costs
Configuration

Every one of those can leak across tenants.

#### Three broad isolation patterns

POOL
Tenant A ─┐
Tenant B ─┼→ Shared service → Shared DB
Tenant C ─┘
Lowest cost
Highest isolation engineering burden

SILO
Tenant A → dedicated stack
Tenant B → dedicated stack
Strong isolation
Higher cost / operational overhead

BRIDGE
Some shared
Some dedicated

Often enterprise SaaS ends up bridge-like.

Perhaps:

shared API
shared workers
shared model gateway
but
tenant-specific encryption key
tenant-specific index
or dedicated DB for regulated tenant

#### Important rule

Never trust this:

POST /search
{
  "tenant_id": 327
}

as your tenant identity.

Derive tenant context from authenticated identity:

JWT / identity
       ↓
tenant membership
       ↓
authorized tenant context
       ↓
query

And enforce it downstream.

For RAG:

user
 ↓
auth
 ↓
tenant + ACL claims
 ↓
retrieval
 ↓
metadata/security filter
 ↓
ONLY authorized chunks

Not:

retrieve everything
 ↓
ask LLM not to reveal forbidden data

That would be a catastrophic security design.

---

1.10 Security-by-design

Security cannot be something added after architecture.

Start with trust boundaries.

Internet
   |
   v
API Gateway
   |
---------------- TRUST BOUNDARY
   |
Application
   |
---------------- TRUST BOUNDARY
   |
AI services / tools
   |
---------------- TRUST BOUNDARY
   |
Enterprise systems

For every boundary ask:

Who is calling?
How are they authenticated?
What are they authorized to do?
What data crosses?
Is it encrypted?
Can inputs be trusted?
Can outputs be trusted?
What happens if this component is compromised?

And distinguish:

Authentication = who are you?
Authorization = what may you do?

For AI, assume all of these can be malicious:

user input
retrieved document
website content
email
tool output
external API response
memory

Therefore an LLM's context is not automatically trusted simply because your application retrieved it.

Security becomes defense-in-depth:

Identity
 ↓
Authorization
 ↓
Input controls
 ↓
Retrieval ACL
 ↓
Model isolation
 ↓
Structured output
 ↓
Policy validation
 ↓
Tool authorization
 ↓
Egress controls
 ↓
Audit

Later, when we reach AI security, we'll go much deeper into prompt injection, data exfiltration, tool abuse and poisoning.

---

1.11 Observability-by-design

Traditional observability has three major telemetry families:

METRICS
LOGS
TRACES

But for AI you need another conceptual dimension:

QUALITY

An AI system can be completely healthy operationally:

HTTP 200
CPU 27%
latency 900 ms
no exceptions

while producing terrible answers.

Therefore you observe both:

SYSTEM HEALTH
+
AI / BUSINESS QUALITY

A production agent trace might be:

agent_run #85472
├── authenticate
├── retrieve
│   ├── embedding
│   ├── vector_search
│   └── reranking
│
├── model_call_1
├── tool_call
│   └── SAP.read_invoice
│
├── model_call_2
├── policy_check
├── approval_request
├── tool_call
│   └── SAP.create_dispute
│
└── verification

You should be able to reconstruct:

who initiated it
which tenant
what model
what prompt version
what sources
what tools
what arguments
what result
what policy
what approval
what final outcome
what latency
what token usage
what cost

#### Audit vs observability

These are related but not identical.

Observability answers:

Why is the system behaving like this?

Audit answers:

Who did what, when, under what authority, and what changed?

An audit trail may require:

strong retention
tamper resistance
restricted access
complete state-changing actions

whereas traces may be sampled or have shorter retention.

Don't confuse the two.

---

1.12 Evaluation-by-design

This is another major shift from conventional architecture.

Do not:

Build application
      ↓
"Let's see whether the AI is good."

Instead:

Define acceptable behaviour
         ↓
Create evaluation cases
         ↓
Build system
         ↓
Continuously evaluate

Suppose the use case is contract analysis.

Before production you should already know what success means:

retrieval recall
clause identification accuracy
unsupported-claim rate
citation correctness
financial calculation accuracy
action recommendation accuracy
unsafe-action rate

And ultimately:

business outcome

For financial execution:

How much valid leakage was identified?
How many false claims were generated?
How much was successfully recovered?
How many interventions were required?

AI evaluation should exist at multiple layers:

retrieval
model
agent
tool execution
workflow
safety
business outcome

The reason is diagnostic.

If your end-to-end score falls from 94% to 82%, you need to know whether:

retrieval broke
OR
model changed
OR
tool API changed
OR
prompt changed
OR
business data changed.

Evaluation becomes an architectural subsystem rather than a QA afterthought.

NIST's AI Risk Management Framework similarly structures AI risk work around Govern, Map, Measure and Manage, explicitly treating measurement and ongoing risk management as part of the lifecycle rather than an end-stage activity. (NIST AI Resource Center)

---

1.13 Cost-by-design

AI architects have to think differently about cost because a feature can execute an unpredictable number of model/tool operations.

Consider:

User request
 ↓
query rewrite         = LLM call
 ↓
classification        = LLM call
 ↓
retrieval
 ↓
reranking             = model call
 ↓
planner               = LLM call
 ↓
worker 1              = LLM call
 ↓
worker 2              = LLM call
 ↓
critic                = LLM call
 ↓
final response        = LLM call

One user request became eight model operations.

Now make that multi-agent.

Costs can explode.

So architect around unit economics:

cost / request
cost / successful task
cost / tenant
cost / workflow
cost / customer
cost / recovered rupee

AWS's Well-Architected cost guidance specifically recommends thinking about increasing efficiency by decreasing cost per business outcome, rather than simply reducing total infrastructure spend. (AWS Documentation)

An agent might cost ₹20 to run and recover ₹20,000.

Excellent economics.

A customer-service query might cost ₹20 and replace a ₹6 operation.

Terrible economics.

Same absolute cost.

Different architecture decision.

Cost controls should therefore exist from the beginning:

model routing
token budgets
context budgets
tool-call limits
step limits
caching
batching
quotas
tenant budgets
usage telemetry

---

1.14 Failure-mode-first design

A junior designer asks:

"How will this work?"

An architect also asks:

"How will this fail?"

Take this:

Agent
 ↓
analyse invoice
 ↓
create ERP dispute
 ↓
send supplier notice
 ↓
update CRM

Now attack it.

What if the LLM times out?
What if ERP succeeds but returns timeout?
What if ERP succeeds and email fails?
What if the worker crashes after ERP success?
What if the same event is delivered twice?
What if the model provider goes down?
What if the retrieval index is stale?
What if the invoice changed during execution?
What if approval expires?
What if the action succeeds but verification fails?

Now architecture begins appearing.

timeout
retry policy
idempotency key
checkpoint
workflow state
version check
fallback
circuit breaker
DLQ
compensation
manual intervention
verification

Notice something important:

Failure analysis generates architecture.

You don't bolt reliability on afterward.

A useful mental model is:

Dependency
    ↓
Failure
    ↓
Detection
    ↓
Containment
    ↓
Recovery
    ↓
Data/state correctness
    ↓
User/business consequence

---

1.15 Human oversight and bounded autonomy

There isn't only:

human
vs
fully autonomous

There is an autonomy spectrum.

LEVEL 0
AI provides information.
LEVEL 1
AI recommends action.
LEVEL 2
AI prepares action; human executes.
LEVEL 3
AI executes after approval.
LEVEL 4
AI autonomously executes within defined boundaries.
LEVEL 5
Broad autonomous operation with human oversight.

Most enterprise AI will occupy different levels depending on action risk.

For example:

Read supplier balance
→ automatic
Draft dispute email
→ automatic
Send low-value dispute
→ perhaps automatic
Create ₹5 lakh adjustment
→ approval
Modify supplier bank account
→ strong human controls

The important thing is that autonomy is not configured as:

agent.autonomous = true

It depends on:

action
risk
amount
confidence/evidence
user
tenant
regulation
novelty
reversibility

NIST's AI RMF guidance explicitly calls for identifying AI capabilities that require human oversight in light of context and risk. (NIST AI Resource Center)

#### Human-in-the-loop

Human decision blocks execution.

AI → Human → Execute

#### Human-on-the-loop

AI operates, but humans monitor and can intervene.

AI → Execute
      ↑
   Human oversight

#### Critical nuance

Do not rely solely on an LLM saying:

confidence = 0.97

as the reason to execute a dangerous action.

That number may not be calibrated.

Use broader evidence:

retrieval strength
rule consistency
independent verification
historical performance
transaction risk
model agreement
business constraints

---

1.16 Architecture trade-offs

There is almost never a universally best architecture.

Architectural maturity sounds like:

"Given requirement X, I would choose A over B because of Y. The cost is Z, which I would mitigate using Q. If requirement R changes, I would revisit the decision."

Not:

"Kafka is scalable, so I use Kafka."

Consider several common AI trade-offs.

Choice	Benefit	Cost
Larger model	Better capability	Cost + latency
More retrieved context	More knowledge	Noise + tokens
Reranker	Better retrieval	Added latency
Multi-agent	Specialization	Coordination + cost
Managed AI service	Speed	Provider dependence
Self-hosting	Control	Operational complexity
Pooled tenancy	Efficiency	Isolation complexity
Silo tenancy	Isolation	Cost
Multi-cloud	Resilience/portability	Engineering complexity
Synchronous	Simple UX	Coupling/timeouts
Asynchronous	Resilience	More state/UX complexity
Strong guardrails	Safety	False blocks
Higher availability	Less downtime	More cost/complexity

A reviewer isn't merely looking for the choice.

They want to know whether you understand what you gave up.

---

1.17 Architecture Decision Records: ADRs

An ADR records an important architectural decision and why it was made.

A useful structure is:

ADR-017: Use durable workflow engine for agent execution
Status
Accepted
Context
Agent workflows may run for hours and perform multiple
side-effecting operations.
Decision
Persist workflow state externally and use a durable
orchestration mechanism rather than in-process loops.
Alternatives
In-memory Python loop
Queue-based custom orchestration
Durable workflow engine
Consequences
+ Resumability
+ Observable workflow state
+ Retry semantics
+ Long-running execution
Additional infrastructure
New operational dependency
Assumptions
Agent workflows regularly exceed request lifecycle.
Review trigger
Revisit if workflows become short-lived and stateless.

An ADR is not a 40-page architecture document.

Its purpose is to preserve:

context
decision
reason
trade-off

Six months later somebody can understand:

Why the hell did we build it this way?

Without ADRs, organizations repeatedly relitigate old architecture decisions.

---

1.18 Well-Architected principles

AWS currently organizes its Well-Architected Framework around six pillars: Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, and Sustainability. (AWS Documentation)

You should know them, but more importantly know how to think through them.

Pillar	Architect asks
Operational Excellence	Can we deploy, operate, observe and improve it safely?
Security	Are data, identities and systems protected?
Reliability	Does it recover and remain correct when components fail?
Performance Efficiency	Are resources and architecture appropriate for workload needs?
Cost Optimization	Are we achieving the outcome economically?
Sustainability	Are resources being used efficiently and unnecessarily wasteful work avoided?

For AI, mentally overlay:

EVALUATION
GOVERNANCE
DATA QUALITY
MODEL RISK
AGENT AUTONOMY

across every pillar.

So if they say:

"Perform a Well-Architected review of this GenAI platform."

Don't just recite six words.

Start asking questions.

Example under Reliability:

What happens if Bedrock is unavailable?
Can we route to another model?
What happens to in-flight agents?
Can tool actions be replayed safely?
How is workflow state recovered?

Under Cost:

Cost per task?
Token budget?
Model routing?
Tenant attribution?
Unexpected agent loops?

Under Security:

Prompt injection?
Tenant boundaries?
Tool authorization?
Secrets?
Data residency?

That's what knowledge of the framework actually means.

---

1.19 SLI, SLO and SLA

These terms need to become precise.

Google SRE defines an SLO as a target value or range for a service level measured using an SLI. (Google SRE)

Think:

SLI
What did we measure?
SLO
What level are we trying to maintain?
SLA
What level have we contractually committed to,
often with consequences if we miss it?

Example:

#### SLI

successful requests / total requests

Measured:

99.96%

#### SLO

99.9% successful requests per rolling 30 days

#### SLA

99.5% monthly availability,
otherwise service credit applies

Usually:

internal SLO > contractual SLA

You want operational margin.

#### Error budget

If:

SLO = 99.9%

then:

error budget = 0.1%

The organization can explicitly reason about how much unreliability it can tolerate.

For AI architecture you may have multiple SLI families:

Operational
availability
latency
error rate
throughput

and:

Quality
grounded-answer rate
task completion rate
tool success
retrieval recall
unsafe-output rate

Example:

SLI:
Percentage of evaluated contract answers
supported by retrieved evidence.
SLO:
≥ 98%.
SLI:
P95 interactive response latency.
SLO:
< 4 seconds.

Now you're operating AI as a production service rather than a science project.

---

1.20 HA, DR, RTO and RPO

These terms often get mixed together.

HA: High Availability

Goal:

Keep the service operating despite ordinary infrastructure failure.

Examples:

multiple instances
multiple availability zones
load balancing
automatic failover
redundant workers

DR: Disaster Recovery

Goal:

Restore service after a major failure.

Examples:

region loss
database destruction
catastrophic corruption
large-scale provider outage

These are different.

HA:
avoid interruption.
DR:
recover after major interruption.

#### RTO

Recovery Time Objective

How long can the system remain unavailable?

Example:

RTO = 30 minutes

Meaning architecture must restore service within the defined target.

#### RPO

Recovery Point Objective

How much data loss, expressed as time, can be tolerated?

RPO = 5 minutes

Potentially up to roughly five minutes of recent data may be lost in a disaster scenario.

Now imagine an AI assistant.

Perhaps:

vector index RPO = 24 hours

because the index can be rebuilt from source documents.

But:

financial execution audit log RPO ≈ 0

because losing records of autonomous financial actions would be unacceptable.

Again:

business importance determines architecture.

---

1.21 Capacity and scale planning

Architects do not say:

"Kubernetes will scale it."

You first need a workload model.

Suppose:

10 requests / second
average end-to-end duration = 5 sec

You have roughly:

50 concurrent requests

using the basic relationship:

[ Concurrency \approx ArrivalRate \times AverageLatency ]

Now ask:

How many model calls/request?
Tokens/request?
Peak-to-average ratio?
Embedding QPS?
Vector search QPS?
DB connections?
Concurrent tool calls?
External API quotas?
Agent execution duration?

AI introduces unusual scaling dimensions.

Traditional API:

requests/sec
CPU
memory

AI:

requests/sec
tokens/sec
tokens/request
context size
model concurrency
GPU memory
provider quotas
retrieval workload
number of agent steps
tool latency

For asynchronous workflows:

queue depth
      ↓
consumer throughput
      ↓
processing latency

becomes crucial.

#### Backpressure

Suppose you can process:

100 jobs/minute

but receive:

1,000 jobs/minute

Scaling isn't the only response.

You need:

queueing
rate limiting
admission control
priority
tenant quotas
load shedding

Otherwise a noisy customer can consume the whole platform.

#### Never size only for average

Think:

normal
peak
failure mode
recovery surge

After an outage, queued work may produce more load than normal production traffic.

---

1.22 Production-readiness gates

This separates prototype architecture from enterprise architecture.

A prototype asks:

Does it work?

Production asks:

Can we trust it,
operate it,
recover it,
secure it,
measure it,
afford it,
and safely change it?

Before production, I would expect explicit readiness across:

Gate	Question
Business	Is success measurable?
Functional	Does the workflow work?
Data	Is data trustworthy and governed?
AI quality	Does it pass representative evaluations?
Security	Threat model and controls complete?
Privacy	Data usage/retention acceptable?
Tenant isolation	Cross-tenant access prevented?
Reliability	Failure/retry/recovery tested?
DR	Recovery tested, not merely documented?
Performance	Expected and peak load validated?
Cost	Unit economics measured?
Observability	Can production behaviour be reconstructed?
Audit	Are consequential actions attributable?
Operations	Who supports it at 2 AM?
Change	Versioning/deployment/rollback defined?
Governance	Model/tool/prompt changes controlled?
Human oversight	Escalation path exists?
Rollout	Blast radius constrained?

This is why:

"the prompt works"

is perhaps 10% of a production AI system.

---

Putting it together

Suppose the interviewer gives you this:

"Design an agent that reviews vendor invoices against contracts and automatically raises disputes."

Do not jump immediately into RAG.

Your brain should execute this sequence:

• Outcome: What constitutes business success: detected leakage, recovered leakage, cycle-time reduction?
• Risk: What is the cost of false positive/negative? Does it merely recommend or execute?
• Requirements: Volume, latency, tenants, data sources, residency, audit obligations.
• Architecture: Ingestion, knowledge layer, retrieval, reasoning, orchestration, tool execution.
• State: Long-running workflow? Checkpoints? Approval state?
• Security: Identity, tenant isolation, source ACLs, tool authorization.
• Control: Probabilistic reasoning → deterministic policy → bounded execution.
• Failure: Duplicate event? ERP timeout? Partial completion? Model outage?
• Evaluation: Retrieval accuracy, discrepancy accuracy, false claims, action correctness.
• Observability: Trace evidence → reasoning → policy → action.
• Scale: Workload model, quotas, concurrency, queues.
• Cost: Cost per invoice and cost per rupee recovered.
• Availability/DR: What state cannot be lost? Required RTO/RPO?
• Rollout: Shadow → recommendations → approval → bounded autonomy.
• Trade-offs: Explicitly state what you chose, what you sacrificed, and why.

That is your master system-design algorithm.

Eventually I want it happening almost unconsciously.

---

The 90-second answer I want in your head

If someone asks:

"How do you approach architecture?"

A very strong answer would sound approximately like this:

I normally start with the business outcome rather than the technology. I want to understand the actors, required decision or action, cost of being wrong, scale, latency, data sensitivity, regulatory constraints and acceptable level of autonomy. From there I separate functional requirements from the non-functional requirements that will actually shape the architecture.

>

For AI systems I make a particular distinction between probabilistic reasoning and deterministic control. An LLM can interpret evidence and propose an action, but authorization, policy, transaction integrity and high-risk execution sit outside the model. I also design evaluation, observability, security, cost and failure recovery from the beginning rather than treating them as production hardening.

>

Once I have those constraints I compare architecture options explicitly, document important choices and their trade-offs, define the SLIs/SLOs and failure semantics, and design the rollout so autonomy and blast radius increase only as evidence supports it. The architecture isn't finished when the happy path works; it's finished when we understand how it behaves when dependencies fail, load increases, models change and the AI is wrong.

If you can say that naturally, rather than from memory, you already sound considerably more senior.

---

The deeper idea

Everything we study next can be hung on this skeleton:

LLM fundamentals
       ↓
What does the probabilistic component do?
Prompt engineering
       ↓
How is behaviour/context controlled?
RAG
       ↓
How does trustworthy knowledge enter?
Data architecture
       ↓
Where does enterprise truth come from?
Agents
       ↓
How does reasoning become multi-step work?
Memory
       ↓
How does state survive?
Tools/MCP
       ↓
How does AI interact with the real world?
Evaluation
       ↓
How do we know it works?
Security
       ↓
What is it permitted to do?
LLMOps
       ↓
How do we operate/change it?
Cloud/Kubernetes
       ↓
Where/how does everything execute?
Operating model
       ↓
WHY IS EACH OF THOSE DESIGNED THAT WAY?

That last question is the architect's job.

Exit test

You should be comfortable answering these without notes:

Question	What I'm testing
Why shouldn't architecture start with technology selection?	outcome orientation
Give me 10 NFRs for an enterprise AI agent.	requirement maturity
How do you decide build vs buy?	commercial/technical judgment
How do you safely let a probabilistic system take deterministic actions?	AI architecture
Explain control plane vs runtime plane.	platform architecture
When would you make an AI workflow asynchronous?	distributed-system judgment
Why should agent compute be stateless while workflow state is durable?	agent production design
Event vs command?	event architecture
How do you enforce tenant isolation beyond tenant_id?	SaaS architecture
Audit vs observability?	operations/governance
Why is evaluation an architectural component?	production AI
What should AI FinOps optimize?	economic architecture
Design for an ERP call succeeding but returning a timeout.	failure thinking
Human-in-loop vs human-on-loop?	autonomous systems
Explain one architecture decision and its trade-off.	architect judgment
What belongs in an ADR?	architecture governance
Explain AWS's six Well-Architected pillars.	platform standards
SLI vs SLO vs SLA?	operational maturity
HA vs DR? RTO vs RPO?	resilience
How do you capacity-plan an agent platform?	scale
What separates an AI prototype from production?	entire topic

AWS describes its Well-Architected Framework as a method for understanding the pros and cons of architecture decisions across its six pillars, which is exactly the mindset to apply rather than memorizing service names. (AWS Documentation)

---

Related reading

• AI Adoption Is an Operating-Model Change, Not a Software Installation, the same argument made to executives rather than architects.
• Enterprise AI Operating Model: Who Owns AI After the Pilot?, where accountability sits once the system is live.
• From AI Pilot to Production: The Twelve Gates, the readiness checks this operating model has to satisfy.