Quick takeaways
- Start with managed APIs for speed, then move to self-hosted or dedicated inference only when cost, latency, or data residency demand it.
- Right-size compute by measuring tokens per second, concurrent users, and context length rather than assuming bigger GPUs are always better.
- Treat caching, load balancing, and retries as first-class infrastructure concerns, not afterthoughts.
AI infrastructure and orchestration
Google Cloud Tech covers agent orchestration and the infrastructure patterns behind reliable AI systems.
Inference hosting models
Choosing where to run inference shapes cost, latency, privacy, and operational burden. Most teams should progress from managed APIs toward more control only when the business case is clear.
Start simple. Many teams over-engineer self-hosting before they understand their traffic pattern, latency requirements, and cost floor.
GPUs and compute sizing
Model size, batch size, context length, and throughput all drive compute requirements. The wrong GPU strategy wastes budget or creates unpredictable latency.
Sizing checklist:
- Estimate peak concurrent requests and acceptable p99 latency.
- Measure tokens generated per second under realistic context lengths.
- Account for KV-cache memory growth on long contexts.
- Prefer batching only when latency targets allow it.
Use quantization and efficient serving frameworks when you need more throughput from the same hardware, but validate output quality after each compression step.
Scaling patterns
Inference traffic is bursty and unpredictable. A single instance works for prototypes; production needs horizontal scaling, queueing, and graceful degradation.
Replica scaling
Add identical inference replicas behind a load balancer to handle throughput spikes.
Model routing
Route simple queries to smaller, cheaper models and complex queries to larger ones.
Fallbacks
Keep a backup provider or model ready for outages, rate limits, or latency spikes.
Request shaping
Trim prompts, chunk context, and limit output tokens before they hit the model.
Define scaling triggers before you need them. Autoscale on queue depth or latency, not just CPU, because inference is memory-bound and asynchronous.
Caching and load balancing
Caching and load balancing reduce redundant inference and distribute work evenly. They are often the fastest ways to cut cost and improve response times.
Caching layers:
- Exact-match cache: Store identical prompts and reuse responses when deterministic output is acceptable.
- Semantic cache: Cache responses for similar queries using embeddings, useful for support and documentation use cases.
- Short-term session cache: Avoid regenerating context-heavy summaries within the same conversation.
Load balancing should consider GPU memory, queue length, and current latency, not just round-robin routing. A busy replica will degrade everyone on it.
Monitoring and observability
LLM infrastructure needs different observability than traditional web services. Track cost, quality, and latency together to spot regressions early.
Key metrics:
- Tokens per second, time to first token, and end-to-end latency.
- Cost per request, per user, and per workflow.
- Error rates, retry counts, and fallback activations.
- Output quality signals like human thumbs-down or automated eval drift.
Pair infrastructure dashboards with the LLM evaluation guide to connect system health with model behavior.
Cost optimization
AI infrastructure costs scale with usage. Without guardrails, a popular feature can blow the budget before anyone notices.
Cost levers:
- Compress prompts and reduce max output tokens.
- Use smaller models for simpler tasks and reserve large models for high-value tasks.
- Implement caching and deduplicate repeated requests.
- Set per-user, per-team, and per-feature budgets with alerts.
- Review model API pricing regularly and renegotiate at volume.
Start with budgets and alerts before you optimize architecture. Visibility creates the pressure to optimize smarter.
Security and compliance
Infrastructure choices determine who can access prompts, outputs, and model weights. Align your architecture with data classification and regulatory requirements early.
Controls to consider:
- Encrypt data at rest and in transit; isolate sensitive workloads.
- Log access to models, prompts, and outputs with retention policies.
- Use private endpoints or VPCs for self-hosted inference.
- Review provider data usage terms for training opt-outs and retention.
Cross-reference this with the AI security guide and data privacy guide before deploying production workloads.
Recommended infrastructure stack
Managed API or vLLM/TGI
Use managed APIs for speed; use vLLM, TGI, or similar for self-hosted throughput and model control.
Kubernetes or serverless containers
Run inference workloads on container platforms with autoscaling, secrets management, and node pools.
LLM proxy or router
Centralize rate limiting, retries, fallbacks, caching, and usage tracking across providers.
OpenTelemetry + cost dashboards
Trace requests end-to-end and attribute cost and latency to users, features, and models.
90-day rollout
- Month 1: Audit current AI usage, set budgets, and choose a hosting model based on latency, privacy, and volume needs.
- Month 2: Add caching, retries, fallbacks, and basic observability. Right-size GPU or API commitments.
- Month 3: Implement per-feature cost allocation, model routing, and a review cycle for model selection and provider pricing.
Without AI vs. with AI
| Task | Without AI | With AI |
|---|---|---|
| Hosting choice | Teams default to managed APIs without comparing cost, latency, and data residency. | The guide maps workloads to managed APIs, self-hosted clouds, dedicated capacity, or edge deployment. |
| GPU sizing | Engineers buy the largest available GPUs assuming bigger is always better. | Compute is sized by tokens per second, concurrency, and context length for the actual workload. |
| Scaling | Teams add capacity only after latency spikes and outages. | Caching, load balancing, autoscaling, and retries are designed into the stack from the start. |
| Cost tracking | API bills arrive as a surprise with no per-workload breakdown. | Teams measure cost per request, per token, and per use case to guide architecture decisions. |
| Observability | Failures are debugged by guessing which model or path was slow. | Monitoring tracks latency percentiles, error rates, throughput, and cost by model and endpoint. |
FAQ
When should we self-host instead of using a managed API?
Self-host when data residency, custom models, committed capacity, or per-token cost justify the operational burden. Most teams should validate product-market fit on managed APIs first.
How do I size GPUs for LLM inference?
Size for peak concurrent users, target latency, and context length. Benchmark tokens per second and memory usage on real prompts before buying hardware.
What is the fastest way to reduce inference cost?
Start with prompt compression, output token limits, caching, and model routing. Then evaluate quantization and smaller models for specific tasks.
Should I autoscale LLM replicas by CPU?
No. Use queue depth, request latency, or GPU memory utilization. Inference is memory-bound and asynchronous, so CPU is a poor scaling signal.
How do I monitor AI infrastructure effectively?
Track latency, throughput, errors, retries, cost per request, and quality drift together. Correlate infrastructure metrics with user-facing outcomes.
Is edge deployment worth the complexity?
Edge makes sense for low-latency, offline, or privacy-sensitive workloads. For most server-side applications, cloud or self-hosted inference is simpler and more capable.
What AI infrastructure should I start with?
Start with managed APIs for speed; move to self-hosting only when cost, latency, or data residency require it.
How do I size GPUs for AI inference?
Measure tokens per second, concurrent users, and context length rather than assuming larger GPUs are always better.
What is the cheapest way to run AI at scale?
Dedicated or reserved capacity for stable workloads, smaller models for simple tasks, and aggressive caching.
How important is caching?
Very. Reusing similar prompts can cut cost and latency dramatically for repeated queries.