Closing the AI Infrastructure Readiness Gap

Owning accelerators, a Kubernetes cluster and model access does not establish production readiness. Infrastructure is ready when the organisation can sustain defined quality, latency, throughput and cost through peak demand, failure, a model change and deteriorating data. Investment decisions should therefore follow the service profile and tests on the organisation's workload—not a comparison of GPU counts.

Adoption does not close this gap by itself. In the CNCF Annual Survey 2025, published in January 2026, 66% of the 213 respondents hosting AI models used Kubernetes for some or all inference, while 7% reported daily model deployment and 47% deployed occasionally. These are responses from a specific survey subgroup rather than a market-wide measure, but they illustrate the distinction between having an execution layer and being able to deliver controlled changes frequently.

Separate workload classes before sizing capacity

“AI” is not one infrastructure profile. Training, fine-tuning, batch processing, vector search, online inference and an agent making a chain of tool calls have different constraints. Begin with a service catalogue that records ownership, criticality, data, model, traffic pattern and degraded mode.

For online inference, tail latency, concurrent requests, input and output length, throughput and queue time matter. Batch work may tolerate waiting but has a completion deadline and backlog constraint. Training depends on securing an entire resource set, network throughput and checkpointing. An agent can multiply demand because one user request triggers several model and tool operations.

Do not confuse an experimental environment with a critical service. Experiments may prioritise iteration speed; production needs isolation, consumption limits, approved model provenance, observability and recovery. A shared pool can be sensible only with explicit priorities and protection that prevents a research job from displacing production.

Model demand in units of work

Capacity planning starts with expected traffic and request shape. For a language service, measure input and output tokens, concurrency, time to first token, generation rate and context-length distribution. An average hides peaks and the cost of long requests, so include distributions and scenarios such as a campaign, month-end or dependency failure.

On the resource side, observe accelerator utilisation and memory, queue time, out-of-memory errors, data transfer, CPU, cache and network. Low GPU utilisation need not mean surplus capacity; data preparation, scheduling or an external API may be the bottleneck. High utilisation is not success when response time breaches its objective and no failure headroom remains.

The model should support decisions: scale, queue, alter batch size, route to a smaller model, limit context or shed lower-priority traffic. Headroom must follow the resilience objective and the time required to add capacity. There is no universal safe percentage.

Release the model, data and configuration together

An AI service change is rarely just a model file. Output depends on the system prompt, orchestration code, generation parameters, tokeniser, retrieval index, filters, tools and evaluation data. Each element needs an identifiable version and a promotion trail between environments.

The model registry should retain provenance, usage conditions, evaluation artefacts, approvals and links to deployments. The pipeline must reproduce the exact combination that passed testing. Replacing a model or prompt manually in production destroys comparability and makes rollback unreliable.

Roll out progressively, routing a limited share of traffic to the new version and comparing it across the same request categories. Define promotion and automatic-stop criteria before the release so that a team is not interpreting declining results for the first time under incident pressure.

Join technical SLOs with quality evaluation

A service can respond quickly while returning poor answers. Operate two connected signal sets. Technical signals include availability, latency, queueing, errors, throughput and resource consumption. Model signals depend on the task: correctness, completeness, source support, safety, format compliance and escalation to a human.

NIST AI 600-1 describes risks specific to generative AI and actions that complement the AI Risk Management Framework. It does not provide one quality threshold for every product. The organisation must define acceptable error in context, who judges output and when the process reverts to human handling.

The evaluation set should represent actual traffic, including difficult and rare cases, and its version must be controlled. Offline evaluation does not replace production monitoring because user data, tools and context change. Quality signals from samples, complaints and human corrections need to be linked to the complete release version.

Design degradation before failure

Not every AI feature needs the same continuity plan. An assistive feature may require a clear message and manual process. A critical service may need a smaller fallback model, another region or provider, reduced functionality, priority queueing, low-priority load shedding or a safe non-generative response.

Test the plan under load. Failover can increase latency, change model behaviour or breach data constraints. Rollback needs to restore the model, prompt, index and configuration as well as the application image. Operations must know who can change mode, which signals trigger it and how recovery is confirmed.

An AI resilience exercise need not begin by taking down a data centre. Simulate an unavailable accelerator, a growing queue, an API quota, a failed agent tool or a damaged index. Observe whether the system contains impact or amplifies demand through retries.

Benchmark the actual service and full cost

MLPerf Inference results provide a valuable reference for comparable workloads and configurations, but they do not predict an application with a different model, context, batching policy, network and quality requirement. A procurement test should run a representative request set on candidate configurations under steady state, peak and failure.

Keep the quality threshold constant during comparison. A cheaper request is not equivalent if the output needs more correction. Total cost includes accelerator, CPU, memory, storage, network, licences, idle time, reserve capacity, platform operation, evaluation and human review. Energy is both an operating constraint and a cost input. The IEA's “Energy and AI” explains why data-centre planning must account for electricity and grid availability, not hardware alone.

Issue readiness based on evidence

Before production launch, the service owner should present a defined SLO, quality evaluation, peak and degraded-mode tests, unit-cost estimate, release-level observability, rollback procedure, data constraints and an owned on-call path. A missing element may justify a constrained pilot, but not unconditional production approval.

Repeat the review after a material change in model, architecture, traffic profile or provider. The decisive measure is not available GPU count. It is the ability to deliver a useful result predictably and to behave in a controlled way when assumptions stop being true.