Cloud Architecture for AI Workloads: Isolation, Identity, Networking and Serving

By Aakash Ahuja··17 min read

Cloud architecture for AI workloads is the design of the isolation, identity, networking, serving and residency layers that an enterprise AI platform runs on. AI workloads break several assumptions that ordinary web architecture takes for granted: they call external model providers over the public internet, they are often GPU-bound and expensive to scale, they move sensitive data into prompts, and they are subject to data-residency rules that care exactly where a token is processed. This article covers how an architect designs the cloud substrate — tenant and network isolation, workload identity, egress control to model providers, private connectivity, data residency, and the serving tier for both API-based and self-hosted inference.

The whole domain reduces to one question:

Where does my data physically go, which identity is allowed to do what, and can this layer stay available and affordable when the workload is GPU-bound and partly operated by a third party?

Ordinary cloud architecture answers "is it available, secure and scalable?" AI cloud architecture must additionally answer "does a prompt leave my trust boundary, where is it processed, under whose identity, and what happens when the GPU or the provider I depend on is unavailable?"


18.0 Architect-level mental model

The mental model is that an AI platform has an unusual data-gravity and trust problem: the most sensitive data in the company (contracts, customer records, source code) flows into prompts, and those prompts often leave your infrastructure to reach a model. So cloud architecture for AI is first about controlling that flow, and only second about the usual availability and scale concerns.

Hold four questions in mind for every component:

QuestionWhat it decides
Where does data go?Residency, egress control, provider boundary
Who is acting?Workload identity, IAM, least privilege
How is it reached?Networking, private endpoints, isolation
How does it serve?API vs self-hosted, GPU, autoscaling, availability
The deployment model you choose — SaaS API, cloud-provider model service, or self-hosted weights — changes the answer to all four. That choice is the first real decision in this domain.


18.1 What are the deployment models for enterprise AI?

There are three, and they trade control against operational burden. The architect's job is to match the model to the data sensitivity and the latency, cost and compliance requirements — not to pick one dogmatically.

ModelYou controlYou operateData leaves you?
External API (SaaS)Prompts, configNothing (provider runs model)Yes — to provider
Cloud-provider model servicePrompts, config, VPC placementLittle (managed in your cloud)Stays in cloud region/tenant
Self-hosted weightsEverythingGPUs, serving, scaling, updatesNo — stays in your infrastructure
The decision drivers:

data sensitivity / residency   → pushes toward in-cloud or self-hosted
latency / throughput control   → pushes toward self-hosted or regional serving
capability (best model needed) → often pushes toward external API
cost at scale                  → depends sharply on utilisation
operational maturity           → self-hosting demands GPU + MLOps skill

Most enterprises end up with a mix: external APIs for frontier capability on lower-sensitivity data, in-cloud managed models for regulated data, and selective self-hosting where residency or economics demand it. The model-strategy domain covers how to route across them; this domain covers where they run.


18.2 How do you control what data leaves your trust boundary?

You control it at the egress layer, with an explicit, allowlisted, logged path to each model provider — never as an incidental outbound internet call from application code.

The default failure is quiet: an SDK makes an HTTPS call to a provider from anywhere in the application, and now sensitive prompt content is leaving the building with no central control, no logging, and no residency guarantee.

The architecture that fixes it:

Application
     │  (no direct internet egress)
     ▼
AI Gateway  ──────────────► egress allowlist (provider endpoints only)
     │                       data-loss / PII redaction hooks
     │                       request/response logging (metadata)
     ▼
Egress proxy / NAT
     │  deny-by-default outbound
     ▼
Model provider (named, approved endpoints)

Key controls:

deny-by-default egress; allowlist only approved provider endpoints
route all model calls through the gateway (the same code boundary as the model gateway)
apply redaction / classification before content leaves the boundary
log that a call left, to which provider, in which region (metadata, not raw content by default)
use provider "no-training" / enterprise data terms where residency matters

This is the cloud expression of the AI security control map: the egress boundary is where "the model should not see this data" becomes an enforceable network control.


18.3 Data residency: where is a token actually processed?

Data residency for AI means knowing — and being able to prove — the physical region where prompt and output data is processed and stored, including inside the model provider.

This is harder than ordinary residency because the processing can happen inside a third party. An architect must be able to answer:

In which region does the application run?
In which region is retrieval data stored and indexed?
In which region does the model provider process the prompt?
Where are logs, traces and captured completions stored?
Where is any caching or fine-tuning data held?

Design levers:

pin the provider's regional endpoint (not a global endpoint that may route anywhere)
use in-cloud model services deployed to a specific region for regulated data
keep retrieval indexes and vector stores in-region
redact or tokenise sensitive fields before they reach a cross-border provider
store traces/completions in-region with redaction

The honest architectural statement: with an external provider you are trusting a contract and a regional endpoint, not a network boundary you own. For data that cannot leave a jurisdiction, that often forces in-cloud managed models or self-hosting, regardless of capability trade-offs. The economics of this choice for one jurisdiction are worked through in Enterprise LLM Deployment Cost in India.


18.4 Workload identity: who is calling the model and the tools?

Every component that calls a model or a tool should act as a distinct, least-privilege workload identity — not as a shared API key and not as the end user's unrestricted credentials.

The anti-pattern is a single god-key, shared across services, with broad permissions, sitting in an environment variable. When it leaks, everything is exposed; when something misbehaves, you cannot tell which component did it.

The baseline:

each service / agent  → its own workload identity (managed identity / IAM role / service account)
                         short-lived, auto-rotated credentials
                         least-privilege scoped to exactly what it needs
                         no long-lived static keys in config
tool access           → authorized per (identity, action, resource), via policy
provider keys         → held by the gateway, injected at runtime from a vault

For agents specifically, a child agent should not automatically inherit the supervisor's privileges, and tool execution should pass through a policy decision that considers user, tenant, agent identity and the specific action. This is the cloud-identity underpinning of the agent trust-boundary model.


18.5 Network isolation and private connectivity

AI components should sit in a private network and reach dependencies — databases, vector stores, in-cloud model endpoints — over private connectivity, not the public internet.

                 ┌─────────────────────────────────────┐
                 │            Private network           │
                 │                                      │
  users ──► edge │  API ─► AI Gateway ─► Agent/RAG       │
          (WAF)  │                 │        │            │
                 │                 │        ▼            │
                 │                 │   Vector store      │ (private endpoint)
                 │                 │   Databases         │ (private endpoint)
                 │                 ▼                     │
                 │        in-cloud model endpoint        │ (private endpoint)
                 │                 │                     │
                 └─────────────────┼─────────────────────┘
                                   ▼
                        egress proxy (allowlist)
                                   ▼
                        external model provider

Principles:

private subnets for compute; no public IPs on AI workloads
private endpoints / PrivateLink-style connectivity to managed model services and data stores
a single controlled egress path for any external provider calls
network segmentation between tenants / environments / risk tiers
WAF and rate limiting at the public edge only

The point is that the only thing reaching the public internet is the deliberate, allowlisted call to an approved external provider — everything else stays private.


18.6 Multi-tenancy and isolation

If the platform serves multiple tenants, business units or risk tiers, isolation must be designed deliberately across compute, data, identity and telemetry — not assumed.

Isolation spans several layers:

network      → segmentation / separate subnets or accounts per tier
data         → per-tenant stores or enforced row/namespace isolation
retrieval    → tenant-scoped indexes or filters (never cross-tenant retrieval)
identity     → per-tenant workload identities and keys
telemetry    → tenant A cannot query tenant B traces
cost         → usage attributable per tenant

The spectrum runs from a shared platform with logical isolation (cheaper, denser, more blast-radius risk) to fully separate accounts/projects per tenant (stronger isolation, higher cost and operational overhead). The right point on that spectrum is set by the sensitivity and regulatory status of the data, and it is one of the decisions worth capturing in an ADR. The tenant-context propagation that makes logical isolation actually hold is covered in Tenant Context in Multi-Tenant Microservices.


18.7 What makes serving AI workloads different?

Serving is where AI cloud architecture departs most sharply from web architecture, because inference — especially self-hosted — is GPU-bound, memory-heavy, slow to start and expensive to leave idle.

The properties that break ordinary autoscaling intuition:

GPUs are scarce, expensive, and not instantly available
model weights are large → long cold starts (load into GPU memory)
inference holds GPU memory for the KV cache during generation
throughput depends on batching requests together
latency and throughput trade off against each other
scaling to zero means paying a large cold-start penalty on the next request

So the serving tier has to be designed around these realities rather than treated like a stateless web pod that scales in seconds.


18.8 Serving patterns for self-hosted models

When you self-host, choose the serving pattern by latency sensitivity and traffic shape. The three common patterns:

PatternFitsTrade-off
Real-time endpointInteractive, low-latency, chat/agentsMust keep GPUs warm; idle cost
Batch inferenceBulk scoring, embeddings, offline jobsHigh throughput, not latency-sensitive
Async / queueSpiky or long-running generationDecouples load; adds queueing latency
Engineering levers that matter for self-hosted serving:

continuous / dynamic batching       → throughput without killing latency
KV-cache-aware scheduling           → memory is the real constraint, not just FLOPs
GPU autoscaling with warm pools     → avoid cold-start penalty on scale-out
quantisation / smaller models       → fit more on a GPU, cheaper per token
request admission control           → protect GPUs from overload and retry storms

The architect does not need to implement the serving engine, but must know that memory and batching, not CPU, govern inference capacity, and that naive "scale to zero" autoscaling produces terrible tail latency.


18.9 Availability and resilience for AI serving

AI serving has failure modes ordinary services do not, and the resilience design has to account for a dependency you may not control.

Design for:

provider outage / rate limiting  → fallback model, routing, graceful degradation
GPU capacity unavailable         → queueing, degraded model, backpressure
regional failure                 → multi-region or documented degraded mode
cold-start latency               → warm pools / minimum replicas
retry amplification              → bounded retries, jittered backoff, circuit breakers

The key design decision is the fallback posture: when the primary model (self-hosted or provider) is unavailable, does the system route to a secondary model, degrade to a cheaper/smaller model, queue the work, or refuse safely? That decision belongs in the gateway and is set by the business criticality of each use case. The broader catalogue of failure modes is the subject of the reliability and resilience domain.


18.10 Cost shape of AI cloud workloads

AI workloads have a cost shape that punishes idle capacity and rewards utilisation, which is the opposite of the "scale to zero" instinct that works for ordinary web services.

External API      → cost ∝ tokens; near-zero idle cost; cost scales with usage
Self-hosted GPU   → cost ∝ reserved GPU time; high idle cost; cheap only at high utilisation

The practical implication:

low / spiky volume     → external API usually cheaper (no idle GPU bill)
high / steady volume   → self-hosting can win if GPUs stay well-utilised
mixed                  → route by use case; reserve GPUs for the steady base load

The metric that matters is cost per successful task, not cost per GPU-hour or cost per token in isolation — a well-utilised GPU and a cheap-per-token API can both be the wrong choice if the task success rate is low. This feeds directly into the cost and performance architecture and AI FinOps work.


18.11 A reference cloud architecture

Putting the layers together, a defensible enterprise AI cloud architecture looks like this:

                         Public edge (WAF, rate limit, auth)
                                      │
                         ┌────────────┴─────────────┐
                         │      Private network       │
                         │                            │
                         │  API tier (workload id)     │
                         │        │                    │
                         │  AI Gateway ── policy / redaction / telemetry
                         │        │                    │
                         │  ┌─────┴──────┐             │
                         │  │ Agent/RAG   │  (per-tenant isolation)
                         │  └─────┬──────┘             │
                         │        │                    │
                         │  private endpoints:         │
                         │   ├── Vector store (in-region)
                         │   ├── Databases (in-region) │
                         │   └── In-cloud model service (in-region)
                         │        │                    │
                         └────────┼────────────────────┘
                                  │
                        egress proxy (deny-by-default, allowlist)
                                  │
                        External model provider (regional endpoint, enterprise terms)

        Cross-cutting: workload identities + vault, central telemetry (in-region, redacted),
                       cost attribution per tenant, IaC-managed, multi-region posture

Every arrow is a place to ask the four mental-model questions: where does data go, who is acting, how is it reached, how does it serve.


18.12 The cloud-readiness checklist

[ ] Deployment model chosen per use case by data sensitivity + latency + cost
[ ] Deny-by-default egress; all external model calls go through an allowlisted gateway
[ ] Data residency known and provable end to end (app, retrieval, provider, logs)
[ ] Every workload has its own least-privilege, short-lived identity
[ ] Provider keys in a vault, injected at runtime, never in code/config/logs
[ ] AI compute in private subnets; private endpoints to data + model services
[ ] Tenant isolation designed across network, data, retrieval, identity, telemetry, cost
[ ] Serving pattern matched to latency/traffic; memory + batching understood
[ ] Warm capacity / fallback posture defined for provider and GPU unavailability
[ ] Retries bounded with backoff; circuit breakers on provider calls
[ ] Cost measured as cost per successful task; GPU utilisation tracked
[ ] Whole substrate defined as infrastructure as code and reproducible

18.13 Common mistakes

MistakeConsequence
Direct provider calls from application codeUncontrolled egress, no residency guarantee, no log
Global provider endpoint for regulated dataData processed in an unknown region
One shared god-key for all servicesLeak exposes everything; no attribution
Public IPs / internet path to data storesUnnecessary exposure of the most sensitive data
Assuming logical multi-tenancy "just works"Cross-tenant retrieval and telemetry leakage
Autoscaling GPUs to zeroSevere cold-start tail latency
Self-hosting at low utilisationPaying for idle GPUs; worse economics than an API
Treating cost as cost per token / GPU-hourOptimising the wrong number; ignoring task success
Residency assumed from app region onlyProvider and logs quietly break the guarantee
---

18.14 FAQ: Should we use a model API or self-host?

It depends on data sensitivity, volume and operational maturity, and most enterprises do both. External APIs give the strongest capability with near-zero idle cost and no GPU operations, which suits lower-sensitivity data and spiky volume. Self-hosting keeps data inside your infrastructure and gives full control of latency and versioning, but only makes economic sense at high, steady GPU utilisation and demands real MLOps and GPU skill. In-cloud managed model services sit between the two: the model runs in your cloud region and tenant, which often satisfies residency without the burden of operating GPUs yourself. Decide per use case and route accordingly, rather than standardising on one model for everything.


18.15 FAQ: How do you keep prompt data in a required region?

You control residency at every hop, not just the application. Run the application and retrieval stores in the required region, use an in-cloud model service deployed to that region or a provider's regional endpoint with contractual data terms, and store logs, traces and any captured completions in-region with redaction. For data that legally cannot leave a jurisdiction, an external provider with a global endpoint is usually not acceptable, and the architecture is forced toward in-cloud managed models or self-hosting in that region. The non-negotiable is that you can prove where a prompt is processed, which means pinning regional endpoints and avoiding "global" routes that may process data anywhere.


18.16 FAQ: Why can't AI serving just autoscale like a web app?

Because inference is GPU-bound and memory-heavy, not CPU-bound and stateless. Model weights are large and take time to load into GPU memory, so a scaled-from-zero replica has a long cold start; generation holds GPU memory for the KV cache, so capacity is governed by memory and batching rather than CPU; and GPUs are scarce and expensive, so you often cannot scale out instantly even if you want to. The result is that naive scale-to-zero produces severe tail latency, and serving has to be designed with warm pools or minimum replicas, dynamic batching for throughput, and admission control to protect GPUs from overload and retry storms.


18.17 FAQ: What is the single biggest AI cloud risk?

Uncontrolled egress of sensitive prompt data to external providers. It is the most common and most damaging failure because it happens silently: an SDK call leaves the application, sensitive content crosses your trust boundary, and there is no central log, no redaction and no residency guarantee. The fix is architectural, not procedural — route every external model call through a gateway behind a deny-by-default egress proxy with an allowlist, apply classification and redaction before content leaves, and log that the call happened. Once egress is controlled, identity, isolation and serving are comparatively tractable.


18.18 What you should know cold

  • An AI platform's defining cloud problem is data gravity and trust: the company's most sensitive data flows into prompts that often leave your infrastructure, so controlling that flow comes before availability and scale.
  • Choose the deployment model (external API, in-cloud managed, self-hosted) per use case by data sensitivity, latency, cost and operational maturity; most enterprises run a routed mix.
  • Control egress with a deny-by-default allowlisted gateway, prove data residency at every hop, and give every workload its own least-privilege, short-lived identity with provider keys in a vault.
  • Put AI compute in private networks with private endpoints to data and model services, and design tenant isolation deliberately across network, data, retrieval, identity, telemetry and cost.
  • Serve AI as a GPU- and memory-bound workload: warm capacity over scale-to-zero, batching for throughput, a defined fallback posture for provider/GPU unavailability, and cost measured as cost per successful task.

The one sentence to remember

"For an AI workload I design the cloud around one question — where does my data go, under whose identity, over which private path, and how does it stay available and affordable when the serving tier is GPU-bound and partly operated by a third party — and I make egress, residency and identity enforceable at the infrastructure layer rather than trusting application code."

That is the substrate every other domain in this handbook ultimately runs on.


Part of the series

The Enterprise AI Architect's Handbook
  1. 1.The Enterprise AI Architect Roadmap: The 29 Domains the Role Actually Owns
  2. 2.The AI Architect Operating Model: Turning a Business Objective into an Architecture
  3. 3.LLM Fundamentals for Architects: Tokens, Context, Latency, Throughput and Cost
  4. 4.Prompt and Context Engineering as an Architectural Concern
  5. 5.RAG Architecture: The Full Pipeline and Where Each Stage Fails
  6. 6.Knowledge Architecture: Ontologies, Entity Resolution and Graph Retrieval
  7. 7.Agent Architecture: Loops, Planning, Verification and Termination
  8. 8.Agent State and Memory Architecture: Scoping, Retention and Provenance
  9. 9.Multi-Agent Systems: When They Help, and How They Fail
  10. 10.Agent Orchestration: Frameworks, Durable Execution and Framework-Independent Design
  11. 11.MCP Architecture and the Enterprise Tool Gateway
  12. 12.Model Strategy: Selection, Gateways, Routing and Fallbacks
  13. 13.Fine-Tuning, RAG or Prompting: How an Architect Decides
  14. 14.Evaluating LLM, RAG and Agent Systems: Metrics, Judges and Quality Gates
  15. 15.LLMOps and Observability: Tracing, Metrics, Drift and Feedback Loops
  16. 16.AI Security: The Full Threat and Control Map for Architects
  17. 17.Responsible AI, Privacy and Governance as Architecture, Not Paperwork
  18. 18.Software Engineering for AI Platforms: The Non-Negotiable Baseline
  19. 19.Cloud Architecture for AI Workloads: Isolation, Identity, Networking and Serving← you are here
  20. 20.Containers, Infrastructure as Code and Delivery for AI Systemscoming soon
  21. 21.Cost and Performance Architecture: Designing for Cost per Successful Taskcoming soon
  22. 22.Reliability and Resilience: The Twenty Failure Modes of AI Systemscoming soon
  23. 23.Enterprise AI Platform Architecture: Control Plane and Runtime Planecoming soon
  24. 24.Production and Launch Readiness for AI Systemscoming soon
  25. 25.Domain Architecture: Applying the Model to a Real Business Functioncoming soon
  26. 26.AI System Design Practice: Fifteen Problems and How to Approach Themcoming soon
  27. 27.Architecture Artefacts: The Diagrams an AI Architect Must Be Able to Drawcoming soon
  28. 28.Structured Answers: System Design, Trade-offs, Incidents and Reviewscoming soon
  29. 29.Experience Narratives: The Stories an Architect Must Be Able to Tellcoming soon
  30. 30.Architecture Leadership and Technical Strategycoming soon
View full series →
AISeriesNovember 6, 2026
Share
Aakash Ahuja

Aakash Ahuja

Enterprise AI, Cybersecurity & Platform Engineering

Aakash writes about secure AI agents, microservices architecture, enterprise platforms, and production engineering. He has 20+ years of experience building and operating software systems across banking, cloud, cybersecurity, AI, and enterprise workflow automation. He is Director of Technology at itmtb Technologies and teaches AI, Big Data, and Reinforcement Learning at top institutes in India.