Model Strategy: Selection, Gateways, Routing and Fallbacks

Model strategy is where most enterprise AI programmes quietly couple themselves to one provider. The durable architecture is not the model. It is the execution layer around the model: selection criteria, a governed gateway, a registry, routing, fallbacks and capacity planning, arranged so that any model can be replaced without rewriting the business logic that calls it.

11.0 The architect-level mental model

The wrong way to think about this topic is:

“Which LLM should we use?”

The architect-level question is:

“How do I build a model execution layer that can select, govern, route, observe, replace, and scale models without coupling the enterprise application to any one model or provider?”

The model should be treated as a replaceable probabilistic compute dependency.

                APPLICATION / AGENT LAYER
                          |
                          v
                 +----------------+
                 |   AI GATEWAY   |
                 +----------------+
                   |            |
           Policy / Auth     Observability
                   |
                   v
              MODEL ROUTER
        +----------+-----------+
        |          |           |
        v          v           v
     Model A    Model B     Model C
    cheap/fast  strong      specialist
        |          |           |
      AWS        Azure       GCP /
    Bedrock     Foundry     self-hosted

There are really two planes:

CONTROL PLANE
-------------
Model registry
Model evaluation
Allowed providers
Routing policy
Cost policies
Tenant policies
Residency policy
Quotas
Model lifecycle
Deployment configuration


DATA PLANE
----------
Prompt
    ↓
Gateway
    ↓
Policy enforcement
    ↓
Route decision
    ↓
Inference endpoint
    ↓
Response

That separation is extremely important.

A production AI application should ideally ask for a capability, not hard-code a vendor model.

Instead of:

call("provider-x-model-abc")

think:

generate(
    capability="complex_reasoning",
    latency_class="interactive",
    data_classification="confidential",
    max_cost=0.05
)

The infrastructure decides which approved model satisfies those constraints.

That is the foundation for almost this entire section.

---

11.1 How to choose an LLM: model selection criteria

Model selection is multi-dimensional optimisation, not a leaderboard exercise.

A model that scores highest on a benchmark may still be completely wrong for your workload because of latency, price, residency, tool-use reliability, capacity, or operational constraints.

A practical selection function looks conceptually like:

Score(m) = w_q·Q + w_l·L + w_c·C + w_t·T + w_p·P + w_a·A

where:

• Q = workload quality
• L = latency suitability
• C = cost suitability
• T = tool/function-use suitability
• P = privacy/compliance suitability
• A = availability/reliability

But before scoring, you usually apply hard eligibility constraints.

Candidate models
      |
      v
+-----------------------+
| Eligibility filtering |
+-----------------------+
 residency?
 modality?
 approved vendor?
 context length?
 privacy?
 tools?
 region?
      |
      v
Eligible models
      |
      v
+-----------------------+
| Ranking / evaluation  |
+-----------------------+
 quality
 cost
 latency
 reliability
      |
      v
Selected model

This prevents a high-quality model from winning even though it violates a non-negotiable requirement.

---

11.1.1 Capability

Ask whether the model can perform the actual workload.

Capabilities may include:

• summarisation
• extraction
• classification
• reasoning
• coding
• mathematical reasoning
• multilingual generation
• structured JSON generation
• function/tool calling
• vision
• audio
• long-document processing
• agentic planning

Do not evaluate “general intelligence” when the production task is something narrow like invoice extraction.

For example:

Expense classification
     ↓
Small model may be sufficient

Contract risk analysis
     ↓
Stronger reasoning model

Image + text insurance claim
     ↓
Multimodal model

Complex agent planning
     ↓
Strong reasoning + reliable tool use

Architect principle

Evaluate against the workload, not against public benchmark averages.

---

11.2 Quality

Quality needs an explicit definition.

Depending on the task, quality could mean:

Classification
→ F1 / precision / recall

Extraction
→ field accuracy

RAG
→ groundedness + answer correctness

Agent
→ task completion rate

Tool calling
→ correct tool + correct arguments

Coding
→ tests passed

Customer service
→ resolution + policy adherence

For enterprise systems, build an evaluation dataset from realistic production examples.

Golden Dataset
       |
       +---- Model A
       +---- Model B
       +---- Model C
               |
               v
       Automated evaluation
               +
        Human evaluation
               |
               v
      Quality / cost matrix

Never make the production decision from one prompt typed into a playground.

---

11.3 Latency

There are several latency dimensions.

Time to first token: TTFT

How long before the model starts answering?

Important for conversational UX.

Inter-token latency

How quickly subsequent tokens stream.

End-to-end latency

request
  ↓
gateway
  ↓
routing
  ↓
model
  ↓
tool
  ↓
model
  ↓
response

The user experiences the total, not merely model latency.

Tail latency

Architects should care about:

P50
P95
P99

A model averaging two seconds is not necessarily production-grade if its P99 is 25 seconds.

For agents, latency compounds.

Suppose an agent invokes an LLM five times:

2 sec × 5 calls
≈ 10 sec minimum

before counting tools, retrieval, networking, retries, etc.

This is one reason smaller models and routing can matter even when the strongest model is affordable.

---

11.4 Cost

Do not reduce model economics to “price per million tokens.”

Real cost is closer to:

Cost = InputTokens + OutputTokens + Retries + ToolLoops
     + Retrieval + Caching + ServingInfrastructure

For managed models:

Input tokens
Output tokens
Cached tokens
Reasoning consumption
Batch pricing
Provisioned capacity

For self-hosted models:

GPU hours
Idle GPU capacity
Kubernetes / orchestration
Storage
Networking
Engineering
Observability
On-call
Model upgrades

This gives an important point:

Open-weight does not automatically mean cheaper.

If a managed API costs ₹X per request but your self-hosted GPUs sit at 20% utilisation, managed inference may still be dramatically cheaper.

Self-hosting becomes economically interesting when enough of these are true:

• high utilisation
• stable traffic
• suitable smaller/open models
• strict residency requirements
• customisation needs
• sufficient platform engineering capability

For what those economics look like with real inference, integration, operations and governance costs, see Enterprise LLM Deployment Cost in India.

---

11.5 Context length

The advertised context window tells you what a model can accept, not what you should routinely send.

Large contexts increase:

• input-token cost
• preprocessing time
• latency
• irrelevant information
• potential attention dilution

Therefore:

1 million token support
        ≠
send 1 million tokens every time

Architecture should still use:

• retrieval
• summarisation
• hierarchical context
• memory selection
• document decomposition

Another subtle issue appears with model routing: the routing layer must respect the context capacity of every possible destination. Microsoft explicitly documents that its Foundry Model Router's effective context window can be constrained by the smallest selected underlying model unless you configure an appropriate model subset. (Microsoft Learn)

---

11.6 Tool use

For agent systems, tool-use reliability may be more important than prose quality.

Evaluate:

Did it select the correct tool?

Did it produce the right arguments?

Did it obey the schema?

Did it avoid hallucinating tools?

Did it understand tool failures?

Can it use parallel tools?

Can it continue correctly after observation?

Imagine:

{
  "tool": "approve_payment",
  "arguments": {
    "invoice_id": "123",
    "amount": 450000
  }
}

A model being eloquent is irrelevant if its tool arguments are unreliable.

For an enterprise agent platform, I would maintain separate model scores for generation and tool execution.

---

11.7 Modality

The architecture must determine whether workloads require:

Text
Image
Audio
Video
Documents
Embedding
Multiple modalities simultaneously

Do not force everything through one multimodal model merely because it supports every modality.

You might use:

Audio
   ↓
specialised transcription model
   ↓
text reasoning model

instead of an expensive general multimodal model throughout the pipeline.

Again: route by capability.

---

11.8 Residency

Residency can become a hard constraint rather than a scoring criterion.

Suppose:

Tenant A
Data allowed in India only
       ↓
Only India-compliant deployments eligible

Tenant B
EU-regulated workload
       ↓
Only approved EU processing geography

Then routing becomes:

request
   ↓
tenant policy
   ↓
data classification
   ↓
approved geography
   ↓
eligible models
   ↓
quality/cost routing

This is particularly important in multi-region and multi-cloud architecture.

For example, Bedrock supports geographic cross-region inference that keeps processing within defined geographic boundaries as well as global cross-region inference for workloads without the same residency constraint. (AWS Documentation)

---

11.9 Privacy

Ask:

• Is provider training performed on submitted data?
• What are retention policies?
• Can logging contain prompts?
• Is private networking available?
• What encryption controls exist?
• Where does processing occur?
• Can secrets or PII reach the model?
• Which models are approved for which classification?

A strong enterprise design can classify traffic:

PUBLIC
   ↓
Broad model eligibility

INTERNAL
   ↓
Approved managed providers only

CONFIDENTIAL
   ↓
Private endpoints / restricted models

HIGHLY RESTRICTED
   ↓
Self-hosted / specific controlled deployment

Do not leave this decision to application developers.

---

11.10 Availability

Model availability includes much more than whether an endpoint “exists.”

Consider:

provider outage
region outage
capacity exhaustion
429 throttling
model retirement
deployment upgrade
quota exhaustion
network failure
provider policy change

Therefore model selection must include an operational dimension.

This leads directly to:

• model gateways
• fallbacks
• multi-deployment routing
• multi-region strategies

---

11.11 A practical selection matrix

One way to summarise it:

Workload	Primary concern	Likely model strategy
Classification	Cost + latency	Small model
Basic summarisation	Cost	Small/medium model
Complex legal reasoning	Quality	Strong reasoning model
Interactive chatbot	TTFT + quality	Fast general model
Coding	Coding eval	Coding-capable model
Enterprise agent	Tool reliability	Strong tool-use model
Image claims	Multimodality	Vision model
Restricted data	Residency/privacy	Approved regional/self-hosted
High-volume extraction	Throughput + cost	Small model / batch
Complex planner	Reasoning	Strong model, selectively invoked

The important thing is that there is normally no single enterprise-wide winner.

---

11.12 Closed vs open-weight models: what is the difference?

Be careful with terminology.

Closed model

You consume the model through an API but don't receive the weights.

Application
    ↓
Provider API
    ↓
Provider-controlled inference

Advantages:

• minimal infrastructure
• latest frontier capability
• managed scaling
• provider optimisations
• low operational burden

Disadvantages:

• provider dependence
• pricing dependence
• less deployment control
• availability/residency constraints
• limited deep customisation

---

Open-weight model

Weights are available and you can deploy them yourself, subject to the model's licence.

Model weights
      ↓
Your runtime
      ↓
Your GPU infrastructure
      ↓
Your endpoint

Advantages:

• deployment control
• infrastructure control
• customisation
• potentially better residency
• potentially lower unit economics at scale
• reduced API-provider dependence

Disadvantages:

• GPU management
• inference optimisation
• capacity planning
• security patching
• model lifecycle
• operational expertise
• potentially weaker frontier capability depending on workload

Interview trap

Do not casually use:

open source = open weight

They are not necessarily identical. A model can expose weights while having licence restrictions that do not make the complete system conventionally open-source.

---

11.13 Managed vs self-hosted models

This is a different axis.

You can conceptually have:

                    HOSTING

              Managed       Self-hosted
            +-------------+-------------+
Closed      | Common      | Generally   |
model       | pattern     | impossible  |
            +-------------+-------------+
Open-weight | Managed API | Full control|
            | or endpoint |             |
            +-------------+-------------+

A cloud provider might offer an open-weight model as a managed endpoint.

Therefore:

open vs closed describes model access; managed vs self-hosted describes serving responsibility.

That is a useful distinction.

---

11.14 When would you self-host?

Use self-hosting when there is a real architectural reason.

Examples:

1. Data sovereignty

Model must operate inside a controlled environment.

2. Economics at scale

Predictable high utilisation can justify reserved GPU capacity.

3. Model customisation

You require fine-tuning or inference modifications not available through the provider.

4. Edge/offline deployment

External inference APIs are unsuitable.

5. Operational independence

Provider outage/availability constraints are unacceptable.

6. Specialised smaller models

A domain-specific 8B model may outperform an expensive frontier model on a narrow workload while being economical to host.

But:

“We want to avoid vendor lock-in” alone is generally not enough justification for building a GPU platform.

A gateway/provider abstraction usually gives you significant portability at much lower operational cost.

---

11.15 What is an AI model gateway?

This is one of the most important concepts in this domain.

A model gateway is the controlled enterprise ingress layer to AI inference.

            Apps / Agents / Services
                    |
                    v
          +---------------------+
          |      AI Gateway     |
          +---------------------+
          | Authentication      |
          | Authorization       |
          | Model policy        |
          | Routing             |
          | Rate limits         |
          | Quotas              |
          | Cost controls       |
          | Logging             |
          | Safety controls     |
          | Retry / circuit     |
          +---------------------+
           /        |         \
          v         v          v
      Bedrock    Foundry    Vertex /
                             vLLM

Gateway responsibilities

It may enforce:

• authentication
• tenant identification
• allowed models
• token limits
• spend budgets
• routing
• retry
• fallback
• circuit breakers
• semantic caching
• prompt/response telemetry
• content policies
• audit logging

Microsoft now exposes AI-specific API Management capabilities including per-consumer token quotas and semantic caching, and its architecture guidance explicitly positions API Management as an AI gateway in front of Foundry/model deployments. (Microsoft Learn)

---

11.16 What the gateway should not become

A common architecture failure is putting all application intelligence into the gateway.

Do not turn it into:

authentication
+ prompts
+ workflows
+ business rules
+ agent planning
+ RAG
+ model logic
+ approval logic
+ everything

Then you have simply created another monolith.

Keep concerns separated:

AI Gateway
→ cross-cutting inference policy

Agent runtime
→ planning/orchestration

RAG service
→ retrieval

Business services
→ business rules

Tool layer
→ enterprise operations

---

11.17 Provider abstraction

Provider abstraction keeps application code independent of any one provider's API, so models can be swapped without rewriting business logic.

Provider APIs differ.

For example, providers may represent:

• messages
• tool schemas
• streaming
• reasoning
• images
• token usage
• safety
• errors

differently.

Create an internal abstraction:

ModelRequest(
    messages=[],
    tools=[],
    temperature=0.2,
    response_schema=...,
    capability="reasoning"
)

Adapter:

Internal request
       |
       +---- AWS adapter
       |
       +---- Microsoft adapter
       |
       +---- Google adapter
       |
       +---- vLLM adapter

Response:

ModelResponse(
    content=...,
    tool_calls=[],
    usage=...,
    latency=...,
    model=...
)

---

But avoid the lowest-common-denominator trap

This is subtle and worth dwelling on.

If your abstraction exposes only features supported identically by every model, you lose provider innovation.

Better:

COMMON CAPABILITIES
-------------------
messages
streaming
tools
structured output
token metrics


OPTIONAL CAPABILITIES
---------------------
reasoning controls
prompt caching
special modalities
provider-specific safety
special inference parameters

Then perform capability negotiation.

if model.supports("structured_output"):
    ...

rather than pretending all models behave identically.

---

11.18 What is a model registry?

Do not confuse a model registry with a provider model catalogue.

An enterprise registry answers:

Which model/deployment/version is approved for which workload under which conditions?

Example:

logical_name: reasoning-standard

provider: microsoft
model: ...
deployment: prod-reasoning-01

status: approved

capabilities:
  reasoning: high
  tool_calling: true
  structured_output: true
  vision: false

regions:
  - india

data_classes:
  - public
  - internal
  - confidential

evaluation:
  task_success: 0.94
  tool_accuracy: 0.98

routing:
  tier: premium

owner: ai-platform-team

Store:

• model/provider
• model version
• deployment
• capability metadata
• context window
• modalities
• approved regions
• data classifications
• evaluation results
• cost profile
• latency profile
• status
• owner
• fallback
• lifecycle dates

Google's managed AI platform itself uses Model Registry for deployed/tuned models, which illustrates the same broader lifecycle principle: model identity and model serving endpoint should be managed separately. (Google Cloud Documentation)

---

11.19 How does LLM model routing work?

Model routing is the decision, made before inference, about which model should handle a given request.

Routing means deciding:

Which model/deployment should execute this request?

There are multiple strategies.

---

11.19.1 Static routing

Support bot
   ↓
Model A

Developer copilot
   ↓
Model B

Simple, predictable.

Good starting point.

---

11.19.2 Rule-based routing

IF vision_required:
    multimodal_model

ELIF confidential:
    private_model

ELIF task == extraction:
    cheap_model

ELSE:
    general_model

This is easy to audit.

---

11.19.3 Capability-based routing

Request declares requirements:

needs:
  tool_use: true
  vision: false
  reasoning: high
  max_latency_ms: 5000

Router searches registry for eligible models.

This is much more scalable than application-specific model IDs.

---

11.20 What is complexity-based routing?

Complexity-based routing sends simple requests to small, cheap models and reserves strong reasoning models for the requests that need them.

This is particularly important.

The insight:

Most tasks do not require the strongest model.

Example:

User request
     |
     v
Complexity classifier
     |
 +---+----------------+
 |                    |
Simple             Complex
 |                    |
 v                    v
Small model       Strong model

Examples:

Simple

• extraction
• formatting
• classification
• basic summary
• straightforward tool invocation

Medium

• synthesis
• moderate reasoning
• multi-document comparison

Complex

• planning
• ambiguous decisions
• long chains of reasoning
• complex coding
• difficult agent recovery

---

Complexity routing flow

A more production-oriented version:

Request
   ↓
Hard policy filters
   ↓
Data/residency check
   ↓
Task classifier
   ↓
Complexity estimation
   ↓
Candidate models
   ↓
Cost/latency optimisation
   ↓
Chosen model

Notice:

Complexity routing happens only after policy eligibility.

You don't choose a cheaper model and then discover it violates residency requirements.

---

How complexity can be estimated

You can use:

Heuristics

token count
number of documents
number of tools
task type
workflow depth

Classifier

Small classifier predicts:

simple / medium / complex

Small LLM router

Ask a lightweight model to classify difficulty.

Learned router

Train a routing model from historical evaluation data.

Microsoft Foundry now implements essentially this concept natively: its Model Router analyzes characteristics including task complexity, reasoning and task type, then routes to an eligible underlying model. It supports routing modes that trade off quality and cost. (Microsoft Learn)

That makes complexity-based routing a mainstream pattern in 2026, not a niche optimisation.

---

11.21 Complexity routing economics

Suppose:

80% simple
15% medium
5% complex

Instead of:

100% → expensive frontier model

do:

80% → small model
15% → medium model
 5% → frontier model

If quality remains acceptable, cost and latency can fall dramatically.

But the architect must measure:

routing accuracy
quality by route
escalation rate
cost/request
latency/request
fallback rate

A cheap router that misclassifies hard requests can destroy product quality.

---

11.22 How should model fallbacks work?

A fallback model takes over when the chosen model cannot serve a request because of throttling, timeouts, errors or outages.

Fallback answers:

What do we do when the chosen model cannot serve the request?

Example:

Primary model
    |
    +--- success → response
    |
    +--- 429 / timeout / unavailable
             ↓
        Fallback model

Fallback triggers might include:

• 429 throttling
• timeout
• 5xx
• regional outage
• model unavailable
• capacity exhaustion

Critical rule

Fallback compatibility must be tested in advance.

If:

Primary supports tool X
Fallback doesn't

then it isn't really a fallback.

Registry metadata should therefore contain:

model A
 fallback:
   model B

compatibility:
   tools: yes
   modality: yes
   schema: yes

---

11.23 Cascades

A cascade tries a cheaper model first and escalates to a stronger one only when the first answer is not good enough.

Cascades are different.

A fallback reacts to failure.

A cascade reacts to insufficient answer quality or confidence.

Cheap model
    |
    v
Is result sufficient?
  /       \
yes        no
 |          |
return      v
       Strong model

Example:

Invoice extraction
      ↓
small model
      ↓
confidence > 0.95?
   /       \
 yes       no
  |         |
done    stronger model

The goal is:

start cheap and escalate only when necessary.

---

11.24 Ensembles

An ensemble runs multiple models and combines their results.

             Request
         /      |      \
        v       v       v
     Model A Model B Model C
        \       |       /
             Judge
               |
               v
             Result

Methods include:

• majority voting
• scoring
• judge model
• consensus
• specialist combination

Useful for:

• high-value decisions
• evaluation
• difficult reasoning
• robustness

But expensive:

Cost_ensemble ≈ Cost_A + Cost_B + Cost_C + Cost_judge

Therefore you usually do not ensemble every request.

---

11.25 Fallback vs cascade vs ensemble vs load balancing

Pattern	Why second model runs
Fallback	First model unavailable/fails
Cascade	First model result insufficient
Router	Choose best model before inference
Ensemble	Multiple models intentionally contribute
Load balancing	Spread requests across equivalent capacity

These are routinely confused.

---

11.26 Rate limits

Rate limiting is something you enforce to control consumption.

Typical dimensions:

Requests/minute
Tokens/minute
Tokens/day
Concurrent requests
Cost/day
Requests/user
Requests/tenant

For an enterprise platform:

Tenant A
100k tokens/minute

Tenant B
20k tokens/minute

Internal batch workload
2M tokens/hour

Without tenant-aware limits:

one workload can consume all shared model capacity.

Microsoft API Management's AI gateway supports token limits by consumer/subscription-style keys specifically to solve this shared-backend problem. (Microsoft Learn)

---

11.27 Quotas

A quota is usually externally imposed or contractually allocated capacity.

Providers may constrain:

RPM
TPM
daily tokens
concurrent inference
regional capacity
GPU capacity

For example, Bedrock model inference uses model-specific quota controls, including token-based limits and, for some models/endpoints, request-rate controls. (AWS Documentation)

Your gateway therefore needs to know:

Application demand
        ↓
Internal rate limits
        ↓
Provider quota capacity

You want:

InternalAllocatedCapacity ≤ ProviderAvailableCapacity

with some safety margin.

---

11.28 Load balancing

Traditional HTTP load balancing often assumes equivalent servers.

Model serving is more complicated.

Consider:

Deployment A
quota remaining = 80%

Deployment B
quota remaining = 10%

Deployment C
latency degraded

Simple round-robin gives:

A → B → C

which is poor.

Better AI-aware routing considers:

health
latency
remaining quota
concurrency
token demand
cost
region
model compatibility

Conceptually:

RouteScore = Capacity + Health + Latency + PolicyFit

---

11.29 Backpressure

When capacity is exhausted, don't allow infinite retry storms.

Use:

admission control
    ↓
queue
    ↓
rate limit
    ↓
bounded retry
    ↓
circuit breaker

Respect provider Retry-After behavior.

Microsoft's API Management architecture explicitly recommends honoring throttling/Retry-After and provides circuit-breaker functionality for 429 responses. (Microsoft Learn)

This is exactly the kind of production detail that differentiates an architect answer from:

“We'll just retry.”

---

11.30 Multi-region serving

Multi-region serving spreads inference across regions for availability and capacity, within the residency limits each workload allows.

Why multi-region?

• latency
• capacity
• resilience
• residency
• disaster recovery

Architecture:

                Global ingress
                     |
              Regional policy
               /           \
              v             v
          Region A       Region B
             |              |
         Model pool      Model pool

But don't blindly fail from:

EU → US

if data policy says processing must remain in Europe.

Therefore region routing is:

availability
    +
residency
    +
latency
    +
capacity

not just latency.

---

11.31 Multi-cloud serving

Example:

                    AI Gateway
            /           |          \
           v            v           v
       AWS Bedrock   MS Foundry   Google
           \            |           /
              common telemetry

Reasons can include:

• different model availability
• customer cloud preference
• regulatory requirements
• acquisition/business environment
• resilience
• specialised capabilities

But multi-cloud has significant costs:

• IAM complexity
• networking
• multiple observability systems
• inconsistent APIs
• billing
• data movement
• feature differences
• operating skills

Therefore a strong architect answer is:

I would design the inference layer so multi-cloud is possible, but I would not deploy multi-cloud solely for architectural elegance. There must be a business, regulatory, resilience or capability justification.

That is much better than “multi-cloud avoids lock-in.”

---

11.32 Amazon Bedrock

Think of Bedrock as AWS's managed foundation-model execution platform.

For this topic, know five things especially well:

Bedrock
├── Multiple models
├── Managed inference
├── Inference profiles
├── Prompt routing
└── Provisioned throughput

---

Bedrock architecture

Agent / Application
        |
        v
API / AI Gateway
        |
        v
IAM + policy
        |
        v
Bedrock
        |
  +-----+------+
  |            |
Model A      Model B
  |
Inference profile
  |
Region routing

---

Bedrock cross-region inference

Bedrock inference profiles can route inference across regions.

AWS currently supports both geographic and global cross-region inference profiles. Geographic routing can preserve a defined geography such as APAC/EU/US, while global profiles can use supported commercial regions more broadly. AWS states that cross-region traffic remains on the AWS network and is encrypted in transit. (AWS Documentation)

Architecturally:

Application
    ↓
APAC inference profile
    |
  +---+---+
  |   |   |
Region Region Region

Useful when regional capacity varies.

---

Bedrock intelligent prompt routing

Bedrock also supports intelligent prompt routing for supported model sets, meaning the request can be directed between models based on the prompt instead of hard-coding one model. It supports both single-region models and eligible cross-region inference profiles. (AWS Documentation)

The broader signal:

AWS itself is moving from “call this model” toward “route the request to the appropriate model/capacity.”

That reinforces the architecture we're discussing.

---

Bedrock Provisioned Throughput

For predictable capacity, Bedrock provides Provisioned Throughput using model units. AWS documents Model Units in terms of the model-specific input/output throughput they can process, with commitment options for provisioned capacity. (AWS Documentation)

Use:

On-demand
→ variable / early-stage traffic

Provisioned throughput
→ sustained predictable production demand

Do not provision expensive fixed capacity merely because the application is “production.”

Measure utilisation first.

---

11.33 Azure OpenAI / Microsoft Foundry

Current terminology: Microsoft's platform is now Microsoft Foundry; Microsoft documents Azure AI Studio/Azure AI Foundry as previous branding. Existing Azure OpenAI resources can be upgraded to Foundry resources while retaining their endpoint, keys and state. (Microsoft Learn)

Be comfortable with both terms, because job descriptions and existing enterprise estates will continue to contain “Azure OpenAI” and “Azure AI Foundry.”

---

Foundry architecture

Application / Agent
        |
        v
Azure API Management
      AI Gateway
        |
        v
Microsoft Foundry
        |
   +----+-----+
   |          |
Model       Model
deployment deployment

Microsoft Foundry currently brings agents, models and tools under a common management environment with RBAC, networking, policies, monitoring and evaluations. (Microsoft Learn)

---

11.34 Microsoft Foundry Model Router

This is extremely relevant to your routing section.

Current Foundry Model Router operates as a deployable model endpoint:

Application
     |
     v
Model Router
     |
     +---- simple → smaller model
     |
     +---- reasoning → reasoning model
     |
     +---- complex → stronger model

Microsoft says the router evaluates characteristics such as:

• complexity
• reasoning requirement
• task type

and can optimise around quality/cost through routing modes. (Microsoft Learn)

The important architectural idea:

Application code sees one endpoint while routing policy decides the underlying model.

That's exactly what a good enterprise model strategy should aim for.

---

11.35 Azure API Management as AI Gateway

This is one of the most widely used enterprise patterns on Azure.

Microsoft's recommended architectures explicitly use Azure API Management as the AI gateway in front of Foundry/model deployments. APIM can apply AI-aware policies such as token limits, metrics, safety controls, caching, routing and request policies. (Microsoft Learn)

Architecture:

             Enterprise Applications
                       |
                       v
             Azure API Management
              +----------------+
              | Entra identity |
              | rate limits    |
              | token quotas   |
              | routing        |
              | cache          |
              | telemetry      |
              | policies       |
              +----------------+
                  /        \
                 v          v
             Foundry A   Foundry B

Microsoft also documents gateway designs across multiple deployments/instances and circuit-breaking on throttled endpoints. (Microsoft Learn)

This is a strong reference architecture for Azure-centred estates.

---

11.36 Microsoft deployment strategy

Microsoft Foundry exposes multiple deployment types, and the choice affects capacity, latency behavior, region/data-zone constraints and SLA characteristics. Microsoft currently documents provisioned deployment types as providing guaranteed throughput/lower latency variance versus standard best-effort types. (Microsoft Learn)

The architect decides based on:

traffic predictability
       +
SLA
       +
residency
       +
cost
       +
capacity

rather than simply choosing “provisioned because it's production.”

---

11.37 Vertex AI / Google AI Platform

The terminology is currently evolving. Google Cloud's current documentation surfaces many capabilities under Gemini Enterprise Agent Platform, while existing APIs and resources still carry aiplatform / Vertex terminology. It is still worth understanding Vertex AI, because that term remains deeply embedded in enterprise architectures and APIs. (Google Cloud Documentation)

Core mental model:

Model Garden
     |
     +---- managed models
     |
     +---- open models
              |
              v
        Model Registry
              |
              v
           Endpoint
              |
              v
          Inference

Google's model platform supports models available directly through managed APIs as well as models that must be deployed to endpoints using backing compute. (Google Cloud Documentation)

---

11.38 Google Model Garden

Think:

discovery + model catalogue + deployment pathways.

Model Garden can include:

• Google models
• partner models
• open models

and provides ways to discover/test/deploy them. (Google Cloud Documentation)

For enterprise architecture:

Model Garden
    ↓
evaluation
    ↓
approved model
    ↓
Model Registry
    ↓
endpoint

Do not give developers arbitrary production access to every catalogue model.

---

11.39 Google Provisioned Throughput

Google also supports Provisioned Throughput for workloads requiring reserved model capacity.

Conceptually:

Pay-as-you-go
→ variable traffic

Provisioned Throughput
→ predictable reserved capacity

Google measures provisioned generative capacity using GSUs and allows handling of excess traffic according to configured consumption behavior; for example, workloads can spill into pay-as-you-go or be constrained to dedicated provisioned capacity. (Google Cloud Documentation)

Again, the cross-cloud concept is the same:

AWS                 Google
Model Unit          GSU
Provisioned         Provisioned
Throughput          Throughput

Specific mechanics differ, but the architectural decision is:

shared/on-demand capacity versus reserved/predictable capacity.

---

11.40 Self-hosted inference

Now leave the hyperscaler-managed world.

Model weights
      |
      v
Inference runtime
      |
      v
GPU node(s)
      |
      v
Serving endpoint
      |
      v
AI Gateway

Components might be:

Kubernetes
GPU nodes
Model storage
Inference engine
Autoscaling
Load balancing
Metrics
Model cache
Gateway

The key architecture responsibility becomes:

efficiently convert GPU capacity into tokens.

---

11.41 vLLM concepts

You do not need to be a vLLM implementation expert to make serving decisions, but you should know why it exists.

vLLM is an inference/serving runtime designed for efficient LLM execution.

Current vLLM capabilities include:

• PagedAttention
• continuous batching
• chunked prefill
• prefix caching
• several quantisation schemes
• speculative decoding
• distributed execution
• multiple GPU parallelism techniques. (vLLM)

---

11.42 KV cache

The KV cache stores attention keys and values for tokens already processed, so each new token does not recompute the whole sequence.

During autoregressive generation:

Token 1
Token 2
Token 3
...

the transformer would waste enormous compute if it recomputed all prior attention states for every new token.

So it caches:

Keys
Values

from previous tokens.

This is the KV cache.

Rough mental model:

Request 1 → KV cache █████████
Request 2 → KV cache ███
Request 3 → KV cache █████████████

KV-cache memory can become a major serving constraint, especially with:

• many concurrent users
• long contexts
• long generated outputs

---

11.43 PagedAttention

Traditional allocation can waste GPU memory because request lengths vary.

PagedAttention uses a virtual-memory-like idea:

Logical KV blocks
      ↓
mapped onto
      ↓
Physical GPU memory blocks

Instead of needing large contiguous memory allocations:

██████.....████........

memory can be managed in smaller blocks.

vLLM specifically identifies PagedAttention as one of the techniques it uses for efficient KV memory management. (vLLM)

At architect level:

PagedAttention improves utilisation of scarce GPU memory by managing KV-cache storage in page-like blocks rather than relying on large contiguous allocations.

Enough.

---

11.44 Continuous batching

Continuous batching adds and removes requests from a running batch at each generation step instead of waiting for a whole batch to finish.

Traditional static batching:

wait
collect requests
execute batch
wait
collect next batch

Bad for variable online workloads.

Continuous batching:

Request A ───────────────
Request B    ─────────
Request C       ─────────────
Request D             ───────

As requests finish, new ones enter the batch.

Benefits:

• higher GPU utilisation
• more throughput
• less waiting than fixed batches

vLLM supports continuous batching specifically for this purpose. (vLLM)

---

11.45 Prefill vs decode

A useful serving distinction.

Prefill

Process the input prompt.

10,000-token document
       ↓
prefill

This is relatively compute-heavy and parallelisable.

Decode

Generate tokens one at a time.

token
 ↓
token
 ↓
token

Decode tends to have different compute/memory characteristics.

Modern inference engines increasingly optimise them separately; current vLLM even supports disaggregated prefill/decode configurations. (vLLM)

At architect level, know that:

long-context ingestion and token generation are different workload phases and can stress serving infrastructure differently.

---

11.46 GPU scheduling concepts

Suppose you have:

Node A: 8 GPUs
Node B: 8 GPUs
Node C: 8 GPUs

Your scheduler needs to decide:

Where does model X run?
How many replicas?
How many GPUs per replica?
How do requests reach replicas?
What happens on failure?
How do we prevent one tenant monopolising capacity?

Key concepts:

Replica

Complete serving copy.

Model instance 1 → GPU group A
Model instance 2 → GPU group B

Allows request-level parallelism.

---

11.47 Data parallelism

Conceptually:

Model copy A ← requests
Model copy B ← requests
Model copy C ← requests

Same model replicated across devices.

Useful when model fits in one GPU/group and you need throughput.

---

11.48 Tensor parallelism

Split calculations for a model layer across GPUs.

Transformer layer
   /   |   |   \
GPU1 GPU2 GPU3 GPU4

Needed when one model does not fit or when parallel computation is beneficial.

Requires substantial GPU-to-GPU communication.

vLLM supports tensor-parallel inference across GPUs. (vLLM)

---

11.49 Pipeline parallelism

Split model layers into stages.

Layers 1-10
   GPU 1
     ↓
Layers 11-20
   GPU 2
     ↓
Layers 21-30
   GPU 3

vLLM supports pipeline parallelism and supports combining it with tensor parallelism for models that span multiple nodes. (vLLM)

The easy distinction:

Tensor parallel
→ split computation within layers

Pipeline parallel
→ split groups of layers across stages

---

11.50 GPU utilisation matters enormously

Imagine a GPU capable of serving 1,000 useful token-units but workload uses 300.

Capacity   ██████████
Used       ███
Idle       .......

You still pay for the GPU.

Therefore optimise:

• batching
• model size
• request scheduling
• KV cache
• concurrency
• quantisation
• replica count
• autoscaling

The self-hosting economics depend heavily on utilisation.

---

11.51 Quantized serving

Models normally store weights using some numeric precision.

Conceptually:

FP32
↓
FP16/BF16
↓
FP8
↓
INT8
↓
INT4

Reducing precision can reduce:

• memory
• bandwidth
• serving cost

and potentially increase throughput.

But it may reduce model quality depending on model, workload and quantisation technique.

vLLM currently supports numerous quantisation formats including FP8, INT8, INT4, GPTQ/AWQ and others. (vLLM)

---

Example

Suppose roughly:

70B parameters × 2 bytes
≈ 140 GB

just for FP16/BF16 weight storage before other runtime memory.

At approximately 4-bit:

70B × 0.5 bytes
≈ 35 GB

for raw parameter storage.

Real runtime memory is higher because you also need things such as:

• KV cache
• activations/buffers
• runtime overhead

But that rough calculation shows why quantisation dramatically changes deployment economics.

---

11.52 Quantisation trade-off

Higher precision
      ↑
quality potential
memory
cost


Lower precision
      ↓
memory
cost
potential quality

Never decide quantisation from synthetic benchmarks alone.

Run your real evaluation set:

FP16 baseline
   ↓
INT8
   ↓
INT4
   ↓
compare:
quality
latency
throughput
GPU memory

---

11.53 AI gateway / routing layers

For a multi-provider enterprise, the reference shape looks like this:

                         ENTERPRISE
                   Applications / Agents
                           |
                           v
                +----------------------+
                | Enterprise AI Gateway|
                +----------------------+
                | Entra/IAM Identity   |
                | Tenant isolation     |
                | Data classification  |
                | Rate limits          |
                | Token quotas         |
                | Cost budgets         |
                | Safety policy        |
                | Routing              |
                | Circuit breaker      |
                | Telemetry            |
                +----------+-----------+
                           |
                    MODEL ROUTER
              +------------+-----------+
              |            |           |
              v            v           v
          Bedrock      MS Foundry   Google AI
              |                        |
              |                        |
              +---------+--------------+
                        |
                        v
                 Self-hosted vLLM
                    when justified

Microsoft's current architecture guidance explicitly supports AI-gateway patterns through API Management and even describes multi-backend/multi-region model-gateway architectures, which makes this a very defensible reference architecture. (Microsoft Learn)

---

11.54 Add the model registry

Full version:

                      CONTROL PLANE
      +---------------------------------------------+
      | Model Registry                              |
      |                                             |
      | approved models                             |
      | capability metadata                         |
      | eval scores                                 |
      | cost                                        |
      | latency                                     |
      | residency                                   |
      | version                                     |
      | lifecycle                                   |
      +----------------------+----------------------+
                             |
                             v
                       MODEL ROUTER
                             |
             +---------------+---------------+
             |                               |
             v                               v
        Managed models                 Self-hosted
   Bedrock/Foundry/Google                 vLLM

And:

                       OBSERVABILITY
                            |
        +-------------------+-------------------+
        |                   |                   |
       cost               quality             SRE
     tokens              eval score        latency
     spend               failures           429
     tenant              hallucination      5xx
                                             |
                                             v
                                   Feed back into routing

That feedback loop is important.

---

11.55 Routing should evolve from static to adaptive

Do not start with sophisticated ML-based routing unless needed.

A good maturity model:

Stage 1: single model

app → model

Stage 2: gateway

app → gateway → model

Stage 3: policy routing

app → gateway → rule router → models

Stage 4: evaluation-driven routing

task + policy + eval results
              ↓
             model

Stage 5: adaptive routing

historical:
quality
latency
cost
availability
        ↓
routing optimisation

This gives you a strong principle:

Do not build an elaborate model router before you have evidence that you need multiple models. But establish the abstraction boundary early so you can add routing later without rewriting applications.

---

11.56 Model lifecycle

Another architect concern that is often forgotten.

Models change.

Model v1
    ↓
New model v2 released
    ↓
evaluate
    ↓
shadow test
    ↓
canary
    ↓
production
    ↓
deprecate v1

Do not:

provider announces model
→ change model ID
→ production

Use:

candidate
    ↓
offline evaluation
    ↓
security/compliance
    ↓
shadow traffic
    ↓
canary
    ↓
A/B where suitable
    ↓
approved

The model registry is part of that lifecycle.

---

11.57 Shadow testing

Suppose production uses Model A.

Production request
       |
       +------→ Model A → user
       |
       +------→ Model B → evaluation only

Model B does not affect the user.

Compare:

quality
tool success
latency
cost
safety

This is one of the safest ways to evaluate model replacement.

---

11.58 Failure modes you should know

Failure 1: hard-coded provider

every service → vendor SDK

Changing model requires modifying dozens of services.

Fix

Gateway + provider adapters.

---

Failure 2: strongest model everywhere

Result:

excellent capability
+
terrible economics
+
unnecessary latency

Fix

Workload evaluation + routing.

---

Failure 3: cheapest model everywhere

Result:

poor task quality.

Fix

Quality thresholds and escalation.

---

Failure 4: router ignores compliance

Router optimises cost and sends confidential data to forbidden geography.

Fix

Policy filtering before optimisation.

---

Failure 5: fallback incompatible

Primary has tool/vision capability, fallback doesn't.

Fix

Capability-aware fallback sets.

---

Failure 6: retry storm

429
 ↓
retry
 ↓
429
 ↓
retry
 ↓
429

multiplied by thousands of agents.

Fix

backoff + jitter + queue + circuit breaker + alternate capacity.

---

Failure 7: self-hosting with poor utilisation

GPU estate costs more than managed API.

Fix

measure utilisation and TCO.

---

Failure 8: abstraction hides useful features

One universal API supports only the lowest common denominator.

Fix

common core + optional capabilities.

---

Failure 9: no model version governance

Provider upgrades/deprecations unexpectedly change behaviour.

Fix

registry + evaluation + controlled rollout.

---

Failure 10: model metrics only

Platform tracks:

tokens
latency

but not:

task success

A cheap fast model that produces incorrect actions is not an optimisation.

---

11.59 Metrics I would expect from the model platform

Performance

TTFT
total latency
tokens/sec
P50/P95/P99

Capacity

RPM
TPM
concurrency
queue depth
429 rate
GPU utilisation
KV-cache utilisation

Economics

cost/request
cost/tenant
cost/workflow
tokens/request
cache hit rate

Quality

task success
groundedness
tool-call accuracy
schema-valid response %
fallback/cascade rate

Routing

requests/model
routing classification
escalation rate
route quality
cost saved by routing

Reliability

5xx
timeouts
circuit breaker state
regional failures
provider availability

---

11.60 The design decision hierarchy

When a request arrives, think in this order:

1. Is the model ALLOWED?
       ↓
2. Is it CAPABLE?
       ↓
3. Is capacity AVAILABLE?
       ↓
4. Does it meet QUALITY?
       ↓
5. Can we optimise LATENCY/COST?

Not:

Which model is cheapest?

This hierarchy is worth remembering.

---

11.61 Applying this to a multi-agent platform

In a multi-agent platform, model strategy connects directly to the agent hierarchy described in Multi-Agent Systems.

That hierarchy already has:

Supervisor
   ↓
Task agents
   ↓
Deterministic workflows

Now add model routing:

                   SUPERVISOR
                       |
                Model capability:
                strong reasoning
                       |
          +------------+------------+
          |                         |
          v                         v
      Task Agent A              Task Agent B
 classification                negotiation
          |                         |
      small model                strong model
          |
          v
 deterministic workflow

The key insight:

Model choice can differ by agent role and even by individual reasoning step.

You should not necessarily run the Supervisor, extractor, classifier, planner, evaluator and summariser on the same model.

---

Complexity-routing architecture for agents

Task arrives
    |
    v
Supervisor
    |
    v
Task classification
    |
    +-------- simple --------→ Small/cheap model
    |
    +-------- medium --------→ Standard model
    |
    +-------- complex -------→ Strong reasoning model
                                      |
                                      v
                           deterministic execution

Even better:

request
 ↓
policy eligibility
 ↓
complexity
 ↓
model route
 ↓
agent reasoning
 ↓
deterministic workflow
 ↓
verification

This preserves the principle that runs through the whole handbook:

Use probabilistic intelligence where judgement is required and deterministic execution where correctness must be bounded.

---

11.62 The enterprise-wide view

At enterprise scale, zoom out.

The question is often:

“How would you define an enterprise model strategy?”

A strong answer:

“I wouldn't standardise the enterprise on one foundation model. I'd establish an approved model portfolio behind a governed AI gateway. Models would be evaluated against workload-specific quality, latency, cost, context, tool use, modality, privacy, residency and availability criteria. The gateway would provide provider abstraction, identity, quotas, observability and policy enforcement, while a model registry stores approved model capabilities and lifecycle status. Routing can start static and evolve toward capability- and complexity-based routing. For critical workloads I'd design compatible fallbacks and regional capacity strategies. Managed platforms such as Bedrock, Microsoft Foundry or Google's AI platform would generally be preferred unless economics, customisation or sovereignty justified self-hosted inference such as vLLM.”

That answer touches almost every bullet in this topic.

---

11.63 FAQ: Bedrock vs Azure OpenAI/Foundry vs Vertex AI, which should you choose?

Do not turn it into a brand comparison.

Say:

“I would choose based on the enterprise's cloud estate, model requirements, security controls, data residency, available capacity, operational model and economics. Bedrock is a natural fit for AWS-centric organisations, Foundry integrates strongly with the Azure/Entra/APIM governance ecosystem, and Google's platform provides its managed Gemini ecosystem plus Model Garden and broader ML serving. I would still put an enterprise model abstraction and policy layer above them where portability or multi-model operation is required.”

Then discuss workload-specific requirements.

That's an architect answer.

---

11.64 FAQ: Why not just use the best model?

“Because ‘best’ is workload-specific. The strongest reasoning model may be unnecessarily expensive and slower for extraction or classification. I'd establish quality thresholds for each task and select the cheapest/fastest model that reliably clears that threshold, while retaining stronger models for complex cases. That's where complexity routing or cascades become valuable.”

---

11.65 FAQ: How do you avoid LLM vendor lock-in?

Avoid:

“We'll make everything multi-cloud.”

Say:

“I separate application semantics from provider semantics through an AI gateway and provider adapters, keep prompts and evaluation datasets independent where practical, maintain model capability metadata in a registry, and avoid exposing provider-specific model IDs throughout business code. I still allow provider-specific capabilities through extensions rather than forcing everything to a lowest-common-denominator interface. That gives us portability without paying the full operational cost of active multi-cloud unless there's a business reason.”

---

11.66 FAQ: When should you self-host with vLLM?

“When there is a demonstrated requirement, such as sovereignty, specialised open-weight models, custom inference behaviour or sufficiently stable high utilisation to make GPU economics attractive. I wouldn't self-host merely to avoid API fees because then I inherit GPU scheduling, model lifecycle, scaling, availability, inference optimisation and on-call responsibility. With vLLM I'd think about PagedAttention and KV-cache efficiency, continuous batching, quantisation and the appropriate tensor/pipeline/data-parallel strategy based on model size and workload.”

vLLM's current serving architecture supports the major optimisation/parallelism concepts referenced in that answer. (vLLM)

---

11.67 Reference architecture

If there is time for only one diagram, draw this:

                     USERS / AGENTS
                           |
                           v
                  +-----------------+
                  |   AI GATEWAY    |
                  +-----------------+
                  | Auth / Tenant   |
                  | Data policy     |
                  | Rate / quota    |
                  | Cost controls   |
                  | Observability   |
                  +--------+--------+
                           |
                           v
                 +------------------+
                 |   MODEL ROUTER   |
                 +------------------+
                 | capability       |
                 | complexity       |
                 | latency          |
                 | cost             |
                 | availability     |
                 +--------+---------+
                          |
        +-----------------+----------------+
        |                 |                |
        v                 v                v
     Bedrock           Foundry          Google
        |                                  |
        +-----------------+----------------+
                          |
                    self-hosted
                       vLLM
                          |
                          v
                  GPU infrastructure


                  CONTROL PLANE
                        |
        +---------------+---------------+
        |                               |
  Model Registry                   Evaluation
 capability                       golden sets
 version                          quality
 policy                           safety
 cost                             tool success
 regions
 lifecycle

If you can explain every box on that diagram, you understand this topic at architect level.

---

11.68 What you should know cold

In short:

• Select models per workload against explicit quality, latency, cost and policy thresholds.
• Put every model behind a governed gateway and a registry, never behind hard-coded IDs.
• Route on eligibility first, then capability, then cost.
• Fallback, cascade, ensemble and load balancing solve different problems.
• Self-host only when sovereignty, customisation or sustained utilisation justify it.

These distinctions should require zero thought:

1. Model selection = deciding which models meet workload requirements.
2. Model gateway = enterprise policy/control boundary around inference.
3. Provider abstraction = prevent application coupling to provider-specific APIs.
4. Model registry = approved models, capabilities, versions, policies and lifecycle.
5. Routing = choose a model before inference.
6. Complexity routing = cheap/small models for simple tasks; stronger models when justified.
7. Fallback = first model failed.
8. Cascade = first model answer wasn't sufficient.
9. Ensemble = intentionally use multiple models.
10. Load balancing = distribute equivalent inference capacity.
11. Rate limit = consumption control you impose.
12. Quota = capacity limit/allocation you must operate within.
13. Open-weight vs closed = access to weights.
14. Managed vs self-hosted = who operates inference.
15. vLLM = efficient open-model inference runtime.
16. PagedAttention = efficient KV-cache memory management.
17. Continuous batching = dynamically batch active requests.
18. Tensor parallelism = split computation across GPUs.
19. Pipeline parallelism = split model layers/stages across GPUs.
20. Quantisation = lower numerical precision to reduce serving footprint/cost, subject to quality validation.
21. Multi-region ≠ blindly fail over across residency boundaries.
22. Multi-cloud should be enabled architecturally but adopted only with justification.
23. Policy eligibility comes before cost optimisation.
24. The model is replaceable; the AI execution platform is the durable architecture.

---

11.69 The one-sentence version

If this entire domain has to be compressed into one answer:

“I treat models as replaceable probabilistic compute behind a governed execution layer: an enterprise AI gateway enforces identity, tenancy, privacy, quotas and observability; a registry tracks approved capabilities and versions; and routing selects the cheapest and fastest eligible model that meets the workload's quality threshold, with compatible fallbacks, regional capacity strategies and self-hosted inference only where sovereignty, customisation or economics justify the operational complexity.”

That is the core of Model Strategy, Serving & Routing at Principal/AI Architect level.

---

Related reading

• LLM Fundamentals for Architects, the latency, throughput and cost mechanics that routing decisions trade on.
• Enterprise LLM Deployment Cost in India, managed versus self-hosted economics with real numbers.
• AI FinOps: A Practical Framework to Control Enterprise AI Cost, the cost governance that quotas and budgets feed into.