Almost every article about AI cost control is written for somebody who is not you. It opens with a frontier lab, a training cluster with thousands of accelerators, and a capex number with nine digits in it. Then it recommends a reserved capacity strategy and a dedicated FinOps team. If you are five engineers who shipped an AI feature last quarter, none of that reaches your bill.
The bill you actually get has a different shape. It is inference, and unlike a training run it does not end. AWS states the split plainly in its own Well-Architected guidance for machine learning: up to 90 percent of the infrastructure spend for developing and running ML applications goes to inference rather than training (Machine Learning Lens, MLCOST05-BP02). Training is a project with an end date. Inference is a subscription you sold to yourself, and the renewal is daily.
This post is the working playbook for a five to twenty person team with an AI feature in production and a cloud bill growing faster than revenue. It covers what to measure before you change anything, the levers that produce most of the savings in practice, how to buy GPU capacity in the right shape, and the controls that cut cost and close a real security gap at the same time. It assumes you have no FinOps hire, because you do not.
Quick context: what changed by 2026
Two datasets frame the year. The FinOps Foundation's State of FinOps 2026 survey, covering 1,192 respondents who collectively represent more than 83 billion dollars of annual cloud spend, found that 98 percent of practitioners now manage AI spend, up from 63 percent in 2025 and 31 percent in 2024. AI cost management ranked as the single skillset teams most need to build, and the two hardest open problems reported were visibility into AI costs and allocating those costs to a business unit.
The second dataset says where the money leaks. Flexera's 2026 State of the Cloud Report, its fifteenth annual edition and based on 753 cloud decision makers, put estimated wasted cloud spend at 29 percent, the first increase after a five year downward trend, and attributed the reversal to AI cost complexity, new pricing models, and underused commitment discounts.
Takeaway: the industry did not get worse at FinOps in 2026. It added a workload class whose cost behaviour breaks the old playbook, and most teams are still running the old playbook.
1. First decide whether you should own a GPU at all
The largest cost decision in your AI stack is made before any optimisation: hosted model API, or your own weights on your own accelerator. Teams skip this because the choice usually gets made by whoever prototyped first, and a prototype on a hosted API silently becomes the architecture.
Do the arithmetic properly, once. On the API side, your unit cost is published per million tokens on the provider pricing page (Anthropic and every major vendor publish theirs), so your monthly cost is tokens multiplied by rate. On the self-hosted side your cost is the instance hourly rate multiplied by 730 hours, whether or not a single request arrives, plus storage, plus the engineer hours to run it. The break even is not a token volume, it is a utilisation level: self hosting wins only when you keep the accelerator genuinely busy.
Practitioner opinion: below roughly a steady stream of continuous traffic, a pre-seed team is nearly always better off on a hosted API, because the real cost of self hosting is not the instance, it is the on-call rotation for a component nobody on the team has run before.
Takeaway: compute your own break even from published prices and your own peak-to-average traffic ratio before you buy a single GPU hour.
2. Measure the right number, because utilisation is not allocation
The most common measurement mistake is reading a GPU as busy when it is merely occupied. A pod holds the device, so dashboards show the GPU as allocated, and nobody notices that the silicon is idle between requests.
Use hardware telemetry, not scheduler state. NVIDIA's DCGM Exporter runs as a DaemonSet on your GPU nodes and publishes per-device metrics to Prometheus (NVIDIA DCGM Exporter). The metric that answers "is this thing actually working" is DCGM_FI_PROF_GR_ENGINE_ACTIVE, the fraction of time the graphics and compute engine is active. Pair it with memory utilisation, because inference servers commonly reserve a large block of device memory at startup for the key-value cache, which makes memory look saturated while compute sits near zero.
On top of hardware metrics, keep two application numbers: tokens served per hour, and queue depth. Cost per thousand tokens served is the only figure that lets you compare a self-hosted deployment against a hosted API honestly.
Takeaway: instrument engine-active percentage and cost per thousand tokens before you touch instance types. Without those two numbers every optimisation is a guess.
3. Allocate every rupee to something a human owns
State of FinOps 2026 identified allocation as the unsolved problem, and it bites hardest on shared inference infrastructure: one endpoint serves six features, so the bill arrives as a single undifferentiated number and no engineer feels responsible for it.
Fix it at the request layer, not the billing layer. Every call into your inference path should carry a small, mandatory set of labels: feature, customer tier or tenant, environment, and whether the call is interactive or background. Log token counts against those labels. Your provider's usage API can then be reconciled against your own counts, which is also how you catch a runaway loop. The FinOps Foundation's FOCUS open billing specification is the destination format if you want cloud, SaaS and AI vendor spend in one schema, and its token economics working group is where the emerging practice for per-token unit economics is being written down.
Takeaway: a label schema enforced in code on day one costs an afternoon. Retrofitting attribution onto six months of untagged inference traffic costs a sprint and still produces estimates.
4. Idle time is line item number one
For a startup, the dominant waste is not inefficient inference, it is paid-for silicon serving nothing. A GPU node reserved for a demo environment, an internal tool used twice a day, or a staging replica nobody scaled down bills continuously at the same rate as production.
The remedy is event driven autoscaling with a real floor of zero for anything that is not customer facing. KEDA scales a Kubernetes deployment from an external signal such as queue depth or a Prometheus query, including scaling to and from zero, and the managed Kubernetes services document the pattern directly (see Microsoft's guide to autoscaling GPU workloads with KEDA on AKS). Pair it with a node autoscaler so that removing the last pod also removes the node, otherwise you have scaled the workload to zero and kept paying for the hardware.
The trade is cold start. Loading a large model into device memory is not instant, so scale to zero belongs on internal and batch paths, while user facing endpoints hold a warm minimum of one replica and scale the rest on queue depth.
Takeaway: set a hard rule that no non-production GPU workload may have a minimum replica count above zero, and enforce it in the manifest review.
5. Batching is the cheapest throughput you will ever buy
On an accelerator you have already paid for, serving strategy decides your unit cost. Naive serving processes one request at a time and leaves the device mostly idle waiting on memory. Continuous batching schedules at the iteration level instead, so new requests join the running batch and finished sequences leave it without waiting for the slowest request in the group.
This is what vLLM implements, using PagedAttention to store attention key-value tensors in non-contiguous blocks in the manner of operating system virtual memory, which removes the fragmentation that otherwise wastes device memory. The original paper reports roughly two to four times the throughput of prior serving systems at comparable latency, and the project documents the tuning knobs, chiefly memory utilisation fraction and maximum sequence count, in the vLLM docs.
Two to four times the throughput on the same hardware is a two to four times reduction in cost per token. No purchasing decision on this list moves the number that far.
Takeaway: if you self host and you are not using a continuous-batching server, that is the first change to make, before instance shopping or commitment purchases.
6. If you buy tokens, buy them in the right tier
Teams on hosted APIs assume there is one price. There are usually three, and the gap between them is large.
Asynchronous batch tiers exist precisely for work that does not need an answer in seconds. Anthropic's Message Batches API is documented at a 50 percent discount on both input and output tokens, and Amazon Bedrock batch inference is likewise priced below on-demand. Every nightly enrichment job, every backfill, every offline classification pass belongs there.
Prompt caching is the second tier and is routinely left on the table. Anthropic's prompt caching documentation prices a cache write at 1.25 times the base input rate for the five minute lifetime, or 2 times for the one hour lifetime, and a cache read at 0.1 times base input. If your system prompt, tool definitions and retrieved context are stable across calls, that is a ninety percent reduction on the repeated portion. Note the minimum cacheable prompt length, which varies by model from 512 to 4,096 tokens; below it, caching silently does not happen and no error is returned. The discounts stack with the batch tier.
Takeaway: audit your call sites for two things this week, which jobs can move to batch, and whether your cache read counts are non-zero in the usage fields.
7. Route by task, not by habit
Most production AI features send every request to whichever model the prototype used. That is a pricing decision made by accident and never revisited, and it is expensive because model prices differ by an order of magnitude across a single vendor's lineup.
Split your traffic by what the task actually requires. Classification, extraction, routing, short summarisation and formatting rarely need your largest model. Reserve the top tier for the requests where quality genuinely differentiates the product, and let a smaller model handle the volume. The engineering work is a routing layer plus an evaluation set, and the evaluation set is the part teams skip, which is why they cannot tell whether the cheaper model would have been fine.
Practitioner opinion: build the evaluation set first, even a hand-labelled hundred examples. Without it, model downgrades are argued on vibes and get reverted the first time somebody sees one bad output.
Takeaway: a routing layer plus a small evaluation harness pays for itself faster than any infrastructure change, because it needs no capacity planning and no migration.
8. Buy capacity in the shape your workload actually has
If you do run your own accelerators, on-demand pricing is the worst rate available and the default everyone lands on. The alternatives map to workload shape.
Interruptible capacity fits anything that can checkpoint and retry, which includes most batch and offline inference. Short-horizon reservations fit bounded projects: AWS EC2 Capacity Blocks for ML reserve GPU capacity for a defined future window at a rate below on-demand, so you pay only for the period you booked. On Google Cloud, flex-start provisioning, backed by Dynamic Workload Scheduler, obtains GPU or TPU capacity for runs of up to seven days at discounted rates without a long-term commitment. Only steady, predictable, always-on baseline load justifies a one or three year commitment, and Flexera's finding on underused commitment discounts is a warning about buying that shape too early.
Takeaway: classify each workload as interruptible, bounded, or always-on before shopping, then buy the matching instrument. Buying a commitment for bursty traffic converts a variable cost into a fixed one at exactly the wrong moment.
9. One GPU, several workloads
Small teams frequently run several small models, none of which fills a modern accelerator. Sharing one device is supported, and the three mechanisms differ in isolation rather than convenience.
Time-slicing lets you declare a number of replicas for a GPU and hand each to a different pod, multiplexing them in time. NVIDIA's GPU Operator documentation is explicit that this provides no memory or fault isolation between replicas, so one workload can exhaust memory and take down its neighbours. Multi-Instance GPU partitions supported hardware into hardware-isolated instances with their own memory and fault domain, which is the right choice for anything multi-tenant. MPS sits between the two.
Practitioner opinion: time-slicing is fine for internal and development workloads on a single trust boundary. If any two workloads sharing a device belong to different customers, use MIG or separate devices. A cost optimisation that removes a fault boundary is a security decision wearing a finance costume.
Takeaway: pick the sharing mode from the isolation requirement, not from the utilisation number you want to hit.
10. The cost controls that are also security controls
Uncapped inference is a security finding, not only a budget one. OWASP tracks it as LLM10:2025 Unbounded Consumption, covering excessive and uncontrolled inference that leads to service degradation, model extraction, and direct economic loss. The industry nickname for the last one is denial of wallet: an attacker cannot take you down, so they run up your bill instead.
The mitigations OWASP lists are the same controls a FinOps review would ask for. Rate limit and set per-identity quotas. Validate and cap input length, because token count is your cost unit and an unbounded input is an unbounded charge. Monitor consumption per identity and alert on anomalies rather than reading the invoice at month end. Add two more from the security side: scope every provider API key to one service with its own spend limit so a leaked key cannot drain the account, and set a hard provider-side budget alert, since a runaway retry loop in your own code produces the identical bill to an attack.
Takeaway: per-identity quotas, input length caps, scoped keys and budget alerts are one piece of work that satisfies a security requirement and a cost requirement at once.
Summary table
| Lever | Applies to | Effort | Typical impact |
|---|
| Scale non-production GPUs to zero | Self hosted | Low | Removes idle hours entirely |
| Continuous batching server | Self hosted | Medium | Multiples of throughput on the same device |
| Move offline jobs to a batch tier | Hosted API | Low | About half price on eligible traffic |
| Prompt caching on stable context | Hosted API | Low | Cache reads at a tenth of input rate |
| Model routing by task | Both | Medium | Largest gap, model prices differ by an order of magnitude |
| Capacity shape matched to workload | Self hosted | Medium | Discount versus on-demand |
| GPU sharing (MIG or time-slicing) | Self hosted | High | Consolidates several small models |
| Quotas, input caps, scoped keys | Both | Low | Caps the worst case, closes OWASP LLM10 |
What to do at your stage
Pre-seed. Stay on hosted APIs. Do four things and stop: tag every inference call with feature and environment, move every offline job to the batch tier, turn on prompt caching for your system prompt and tool definitions, and set per-identity quotas plus a provider budget alert. That is a day of work and it covers most of your realistic exposure.
Seed. Add measurement and routing. Track cost per thousand tokens per feature, build the small evaluation set, and route non-critical traffic to a cheaper model. If you have started self hosting, run a continuous-batching server and put a hard zero floor under every non-production GPU workload.
Series A. Now capacity purchasing earns its keep. Classify workloads as interruptible, bounded or always-on, buy the matching instrument, deploy DCGM telemetry so utilisation is a measured number rather than an assertion, and decide sharing mode from your isolation requirements. This is also the point where a monthly review with a named owner stops being overhead.
If your AI bill is growing faster than your usage and you cannot yet say which feature owns which share of it, that is a visibility problem before it is a cost problem, and the fix is measurement plus four controls, not a migration. MatrixGard runs this as part of fractional DevSecOps for pre-seed and seed teams: the cloud, infrastructure and security work that needs an experienced owner but does not yet need a full-time hire. If you want a second pair of eyes on where your inference spend is actually going, start with the free cloud security and cost checklist.
Avinash S is the founder of MatrixGard, a fractional DevSecOps practice working with pre-seed and seed startups across India, Singapore, the UAE, the US and the UK. He has spent around a decade in cloud infrastructure and security.
Methodology note: survey figures are drawn from the FinOps Foundation State of FinOps 2026 (1,192 respondents) and the Flexera 2026 State of the Cloud Report (753 respondents), both linked above. The inference share of ML infrastructure spend is AWS's own published figure in the Well-Architected Machine Learning Lens. Pricing multipliers, discount tiers, isolation properties and autoscaling behaviour are cited to current vendor documentation at the time of writing; verify against the linked pages before making a purchasing decision, since pricing and limits change. Passages labelled "Practitioner opinion" are the author's judgement from field work and are not sourced claims. No client names, engagement outcomes or private figures appear in this post.
