A technical deep-dive on how a single oversized GPU turned into ~90% of a bill, and the four changes that brought it back down.
Framing note. This is a sanitized write-up of a real cost-optimization project. Service names, cluster names, and currency figures have been swapped for representative equivalents. Everything technical (the bottleneck analysis, the async design, the autoscaling metric, the pool right-sizing, and the cold-start fix) is described as it actually happened.
The names below are aliases, standing in for the real service and hardware: -
image-gen-svc— the GPU-backed image-generation service this post is about. - BigGPU — a flagship, high-end data-center GPU with large VRAM (think A100-class). - SmallGPU — a mid-tier inference GPU with less VRAM, roughly one-third the price (think L4-class).
Introduction
Hi, I'm Sandeep, an AI engineer at Gaudiy. For a while, one GPU-backed image-generation feature was quietly one of the most expensive things our team ran. This post is how we cut its production GPU cost by about 75% without making the product any slower for users. I'll walk through the bottleneck analysis, the async autoscaling design (which is the interesting part), the node right-sizing, and the cold-start problem that nearly undid all of it.
TL;DR — one interactive image-generation feature had grown into ~90% of our cloud bill. It ran on a single always-on flagship GPU, and because one request nearly fills the GPU's memory, that GPU could only handle one request at a time. So it sat there billing 24/7 for a peak that rarely showed up. Four changes, stacked, cut production GPU cost by ~75%, all verified in production with no downtime:
- a test environment you can switch off, so it costs nothing when idle;
- a move to a GPU about a third of the price, made possible by a chunked-batch fix;
- autoscaling on a custom queue-depth metric we had to build ourselves, because the stock metric doesn't see async backlog;
- right-sizing the shared GPU node on top of that.
The one thing still open, cold-start latency under a sudden spike, is at the end.
- Introduction
- 1. Problem & background
- 2. Two problems, two fixes
- 3. Lever 1 — A test environment you can switch off
- 4. The backbone: moving from BigGPU to SmallGPU
- 5. Lever 2 — Async processing + a custom queue-depth autoscaler
- 6. Lever 3 — Right-sizing the shared GPU node pool
- 7. Lever 4 — Cold start: baking the weights into the image
- 8. Results
- 9. What we looked at and didn't do
- 10. What's still open
- We're hiring
1. Problem & background
The situation
One user-facing AI image-generation feature had become one of the most expensive line items we ran. It sat on an always-on flagship data-center GPU, and the monthly bill for that single feature was high enough (and its usage flat enough) that leadership started asking whether it was worth keeping at that cost.
That question set the bar. This wasn't about trimming a few percent. We needed a cut big enough to change the keep-or-kill conversation, and we couldn't make the product slower to get it. A 10% saving wouldn't have been worth the effort.
The problem, precisely
Three things defined it:
- GPU compute was ~90%+ of the bill. Storage, egress, monitoring, and managed APIs barely registered next to it. Start to finish, this was a GPU cost problem.
- The production GPU ran 24/7 and was oversized. One flagship GPU, sized for a peak that rarely arrived, never scaling down when idle. We paid peak rates around the clock.
- A full copy of the stack ran in test, also 24/7, at about 5–10% utilization. Nobody runs tests at 3 a.m., but the meter didn't know that.
The constraints
The constraints are what ruled out the easy answers:
- A hard latency cap. From the user's side the feature is synchronous: click generate, watch a spinner, get results. There's a 60-second ceiling on that wait, and going past it counts as a failure.
- Data residency. The data had to stay in one country. That took the biggest theoretical lever, moving to a cheaper GPU in another region, off the table before we started.
- Production reliability. No spot/preemptible GPUs on the interactive path, since a mid-request preemption shows up as a user-facing error, and no community-tier providers without real SLAs.
So everything below is what was left after those constraints removed the obvious options.
The workload
image-gen-svc produces four image variations per request, in roughly ten seconds on the BigGPU. Each request runs a short pipeline: a couple of detection and segmentation passes, then the expensive part, a diffusion UNet (the denoising loop, about 70% of the total time), a VAE decode, and an upscale.
The number that actually drove the cost, though, wasn't time. It was VRAM. A single request is memory-hungry enough to nearly fill the SmallGPU and take up most of the larger BigGPU too. At that size, two requests can't share a GPU; they simply don't both fit in memory. So each GPU handled exactly one request at a time, and the only way to add capacity was to add whole GPUs.
That one fact is the root of the cost. The service ran synchronously: one always-on BigGPU serving live traffic, plus a full copy idling in test. A flagship GPU locked to one-request-at-a-time throughput, billing 24 hours a day. That's where the money went.
2. Two problems, two fixes
It helped to split the GPU spend from §1 into two separate problems, because they have completely different fixes and it's easy to waste effort attacking the wrong one:
Problem A — the test environment Two GPU services running 24/7 at ~5–10% utilization. Pure waste: nobody is testing at 3 a.m., but the meter runs. → Fix: make it switch off. (Lever 1) Problem B — the production environment One always-on flagship BigGPU, sized for a peak that rarely arrives, never scaling down, and far more powerful than the workload needs. → Fix: cheaper GPU + autoscaling + right-sized nodes. (Levers 2–4)
The rest of the post is the four levers, in the order we shipped them. Lever 1 was a config change that took an afternoon. Levers 2–4 were the real engineering.
3. Lever 1 — A test environment you can switch off
Test ran the same GPU stack as production, around the clock, for a workload that was idle almost all the time. The obvious fix, scale it to zero, hits a Kubernetes quirk: the Horizontal Pod Autoscaler (HPA) won't let you set minReplicas: 0. The floor is 1, and the HPA overrides any attempt to set the Deployment to replicas: 0, because keeping replicas up is its whole job.
So turning it "off" means removing the HPA and zeroing the Deployment together. We did that with a single GitOps toggle, one line in the overlay pointing at a Kustomize patch:
# test/kustomization.yaml patches: - path: deployment.yaml - path: horizontal_pod_autoscaler.yaml - path: service.yaml - path: disable-gpu.yaml # add this line = OFF ; remove it = ON
The disable-gpu.yaml patch does two things:
- deletes the HPA, so it can't override the replica count, and
- sets the Deployment to
replicas: 0.
ADD disable-gpu.yaml ──▶ GitOps sync (~3 min) ──▶ pods terminate
no GPU demand
cluster autoscaler
removes the GPU node
(~10 min) ──▶ cost = 0
REMOVE disable-gpu.yaml ──▶ GitOps sync ──▶ HPA + replicas restored
autoscaler adds a GPU node
pod boots (~7–15 min)
cost resumes
A few things make this cheap and safe:
- The node pool is never created or deleted. The GKE cluster autoscaler adds and removes nodes based on whether any pod still needs a GPU. Turn the workload off, the node goes idle, and the autoscaler reclaims it. No infra permissions needed.
- It's a pull request. Whoever needs the environment adds or removes one line and merges. No
kubectl, no console, no on-call SRE, and it's auditable and revertible like any other commit. - Billing is per-second, so a one-hour test costs an hour, not a month. There's no minimum charge for spinning up briefly.
Roughly what that buys, depending on how the environment is used:
| Usage pattern | Savings vs always-on |
|---|---|
| Always on (24/7) | — (baseline) |
| Business hours only (~12 h × weekdays) | ~64% |
| ~2 h/day, weekdays only | ~94% |
| Off | 100% |
This shipped first, and it was a low-risk way to prove out the "scale GPUs to zero when idle" idea before we touched production.
4. The backbone: moving from BigGPU to SmallGPU
The big production lever was swapping the always-on flagship BigGPU for the much cheaper SmallGPU, at roughly a third of the price. You can't just change the machine type, though. The workload had to actually fit and perform on the smaller card, and the reason it didn't at first is the most reusable lesson here.
4.1 Compute-bound vs memory-bound
We ran the same pipeline on both GPUs and watched the hardware counters. The two cards are limited by different things:
BigGPU (large VRAM, more compute) SmallGPU (less VRAM, fewer compute)
───────────────────────────────── ────────────────────────────────────
Compute: nearly all units busy Compute: ALL units busy
during the denoising loop during every active stage
Memory : most used, real headroom Memory : nearly full, ~no headroom
left free
Verdict: COMPUTE-bound. Verdict: BOTH compute-bound
Spare VRAM is useless — AND memory-bound.
the compute units are the wall. No room to breathe.
This is what saved us from a couple of dead-end "optimizations":
- Cross-user batching would gain basically nothing here. The instinct is that the BigGPU has spare memory, so you should batch more requests to use it. But throughput isn't limited by memory. It's limited by the compute units (the streaming multiprocessors), which are already saturated during the denoising pass. Adding more images just serializes more work onto units that are already busy. Spare VRAM doesn't turn into spare throughput.
- Pipeline parallelism across the stages would also gain almost nothing. The UNet is ~70% of the wall-clock and saturates compute during that window, so overlapping the small pre/post stages around it doesn't buy much.
The changes that do help are the ones that cut compute per denoising step: attention optimizations, fewer steps, quantization. The lesson we took away was to measure which resource is actually saturated before optimizing, because the intuitive move is often aimed at the wrong one.
4.2 The VRAM wall and the chunked-batch fix
There was a hard blocker on the SmallGPU. Generating the full batch in one pass needs more memory than the SmallGPU has, so it OOMs immediately. The BigGPU had room to spare; the SmallGPU just couldn't run the production payload.
The fix was chunked batching: instead of generating the whole batch in one forward pass, split it into smaller chunks and clear the allocator's cache between them.
BEFORE (OOM on SmallGPU): generate(full batch) ──▶ peak memory > SmallGPU capacity ──▶ OOM AFTER (fits on SmallGPU): chunk 1: generate(part of the batch) → empty_cache() chunk 2: generate(the rest) ──────────────────────────────────────────────────────── peak memory per chunk now fits
Clearing the cached-but-unused GPU memory between chunks resets peak usage, so the second chunk starts clean. You pay a little wall-clock time (more than one pass) to run on a GPU that costs a third as much, and since the async design below hides that extra latency from users, it was an easy trade.
Once the memory wall was gone, the migration went all the way through: both flagship-GPU pools deleted, the workload live on SmallGPU, no downtime. The headline ~75% cut comes from this swap. But the swap was only safe because of the next three levers.
5. Lever 2 — Async processing + a custom queue-depth autoscaler
This is the part I'd reuse on any async GPU service.
Autoscaling is how you stop paying for a peak you rarely hit: scale up under load, drop to a floor when it's quiet. But it only works if the autoscaler can actually see the load, and for an async service the obvious metric doesn't. We had to build one that does.
5.1 Sync vs async request paths
The service supports both paths, picked per request:
SYNC path (interactive, holds the connection)
─────────────────────────────────────────────
client ──POST──▶ [inference server] ──▶ run full pipeline (~29 s)
◀── 200 OK with 4 image URLs
client waits the full ~29 s.
ASYNC path (what production uses)
─────────────────────────────────
client ──POST──▶ [inference server]
├─▶ 200 OK { id, url:"" } ◄── returned in ~50 ms
└─▶ submit real work to a background worker
│ (single worker thread, one job at a time)
▼
run full pipeline (~29 s)
│
▼
upload images + publish a
"job complete" message to a pub/sub topic
│
▼
a subscriber updates the DB row; the frontend,
polling or subscribed, shows the images
Async is good for the user (no 29-second held connection) and it's what makes autoscaling worthwhile, because requests can queue instead of each holding a live GPU. It also quietly breaks the autoscaler, which is what the next few sections are about.
5.2 Where the backlog actually hides
An async request can sit in three places:
watched by autoscaler? bounded? ───────────────────────────────── ────────────────────── ──────── 1 inference-server scheduler queue yes soft cap 2 background-worker internal queue NO ← the problem unbounded 3 pub/sub results topic (outbound) no (results, not work) n/a
Queue #2 is the one that bites. The instant the server returns the {id, url:""} placeholder, it treats the request as done, even though the GPU work hasn't started yet. Those jobs pile up in the worker's queue, and nothing on the outside is watching it.
5.3 Why the stock autoscaler was blind
The metric an inference server exposes out of the box is roughly "requests received but not yet dispatched." For a sync service that's a fine proxy for backlog. For our async service it's useless, because the server dispatches (and "completes") every request in about 50 ms. Here's what the autoscaler sees when 10 async requests hit one pod in two seconds:
t (s) server "pending" real worker backlog autoscaler thinks reality ───── ──────────────── ─────────────────── ───────────────── ───────────── 0.0 0 0 "idle" idle 1.0 1 5→9 "still low" 9 jobs behind 2.0 0 10 "IDLE" ◀── ~310 s of work owed 30 0 9 "idle" worker on job #1 300 0 1 "idle" job #10 times out
The autoscaler sees a brief blip, then a flatline at zero, and never scales up. In reality one pod is now ten jobs deep, about five minutes of work behind, and the last request in the burst eventually hits its timeout. Nothing on the dashboards shows it.
5.4 The custom queue-depth metric
The fix is to stop asking the inference server about backlog and instead publish the background worker's real queue length as a metric, then scale on that.
# expose a gauge that reports the ACTUAL background backlog queue_depth = Gauge("async_queue_depth", "async tasks waiting in the worker") def _update_queue_metric(self): queue_depth.set(self.worker._work_queue.qsize()) # called on every submit and every completion
Scrape that gauge, expose it as an external metric, and point the autoscaler at it:
metrics: - type: External external: metric: { name: async_queue_depth } target: type: AverageValue averageValue: "6" # scale up when the summed backlog exceeds 6 × replicas
One thing worth being precise about: AverageValue sums the metric across pods rather than checking each pod against the target.
desiredReplicas = ceil( SUM(queue_depth over ALL pods) / 6 )
Pod A Pod B TOTAL ceil/6 action
5 5 10 2 stay at 2
9 9 18 3 +1 → 3
12 12 24 4 +2 → 4
Now the autoscaler reacts to real work owed instead of the server's version of events. In load tests it decided to scale about 17 seconds after a burst started, which is fast. (The slow part after that is cold start, covered in Lever 4.) In production it mostly sits at its floor of 2 replicas because real backlog is near zero, and it scaled correctly the one time load actually called for it.
One thing that cost me real debugging time: this HPA is GitOps-managed with self-heal on. If you bump minReplicas by hand in the console to "test scaling," ArgoCD reverts it in about three seconds and nothing happens. The only real ways to test scaling here are a load test or a PR. Load-driven scaling is fine, since the HPA changing replica counts under load isn't treated as config drift.
6. Lever 3 — Right-sizing the shared GPU node pool
With the workload on the SmallGPU and autoscaling working on real backlog, one more thing stood out: the node itself was oversized.
The GPU node pool used a machine shape with far more vCPU and RAM than the workload needed. It only really used ~1 GPU and a small slice of the CPU and memory, so we were paying for a big machine to use a sliver of it.
BEFORE AFTER
┌──────────────────────┐ ┌──────────────────────┐
│ oversized machine │ recreate │ right-sized machine │
│ 1× SmallGPU per node │ ───────▶ │ 1× SmallGPU per node │
│ far more vCPU / RAM │ pool │ fits the workload │
│ than needed │ │ exactly │
└──────────────────────┘ └──────────────────────┘
│
▼
~37% lower cost PER NODE,
right-sized to the workload,
savings scaling with replica count
The one wrinkle is that machine type is immutable in a GKE node pool, so "right-sizing" really means creating a new pool, draining onto it, and deleting the old one. With a parallel-pool pattern that's zero-downtime.
That ~37% per-node saving stacks on top of the BigGPU-to-SmallGPU swap. It's a discount on already-cheaper hardware.
7. Lever 4 — Cold start: baking the weights into the image
Autoscaling down to a low floor only pays off if scaling back up is fast. Ours wasn't, and the reason is a good reminder that saving money can introduce new failure modes.
Right after the production cutover, one new pod took about 41 minutes to become ready. Another pod, same image and same machine type, came up in about 2 minutes. The difference:
Several GB of model weights were downloaded from a public model hub on every cold start. Nothing was baked into the image and there was no cache volume, so cold-start time was really just download time: slow and unpredictable. And a startup probe would kill a download that ran too long, forcing the whole thing to start over.
Attempt 1: download weights ... too slow ... startup probe kills it at ~25 min
(no cache persists — the partial download is thrown away)
Attempt 2: download weights again ... this time fast ... Ready
────────────────────────────────────────────────────────
TOTAL ≈ 41 min = 26 (wasted) + 15
That 2-vs-41-minute spread was pure network variance on the download. You can't size autoscaler windows around a cold start that ranges from 2 to 41 minutes, and worse, if the model hub had an outage during a scale-up or a node replacement, new pods could never reach ready at all. This wasn't just slow, it was an availability risk.
The fix was to bake the weights into the image at build time and stop the pod from fetching anything at boot:
Docker build: prefetch all model weights into an image layer Runtime env : HUB_OFFLINE=1 (never touch the network at boot)
That turns startup into a local load with no network dependency:
WARM node (image cached, pool keeps a warm floor) schedule → model load from local disk (~2.5 min) ≈ 2.5 min to Ready COLD node (brand-new node needed) decision (~20 s) + node provision (~3 min) + first image pull (~3 min) + model load (~2.5 min) ≈ 6 min to Ready
The two most recent cold pods loaded in 144 and 148 seconds, within four seconds of each other, which is what a local load looks like. The 41-minute tail is gone. The only cost the bake adds is a bigger image, so the very first pull onto a brand-new node is a bit slower, and it's cached after that.
8. Results
Stacking the four levers on the BigGPU-to-SmallGPU move:
Lever Effect
────────────────────────────────────── ─────────────────────────────────────
BigGPU → SmallGPU (chunked-batch fix) ~75% lower production GPU cost ◀ headline
Async + queue-depth autoscaling scale to a floor when idle;
pay for load, not for peak
Shared-pool right-sizing ~37% less PER NODE, stacked on top
Test on/off toggle up to 100% off the test bill
Cold-start bake 41-min tail → deterministic 2.5–6 min
- Production: moved onto SmallGPU with both flagship pools deleted, no downtime, ~75% lower GPU cost. Realized and verified in production, not a projection.
- Autoscaling: healthy in production (floor 2, ceiling 10, scaling on real backlog), decides in ~15–30 s, cold start bounded at ~2.5–6 min.
- Test: on-demand, effectively free when idle.
Every number here was checked against live production logs and metrics after the fact. The rule on this project was realized savings, not estimates.
9. What we looked at and didn't do
A few options we considered and turned down, with the reasons:
- Spot / preemptible GPUs. A mid-request preemption on the interactive path is a user-facing error, so this was out for that path.
- Cheaper GPUs in another region. Blocked by the data-residency requirement. This was the biggest theoretical saving and it simply wasn't available to us.
- Cross-user dynamic batching. Ruled out by the compute-bound analysis. It looks like free throughput given the spare VRAM, but compute is the limit, so it would have been a lot of work for ~0% gain.
- A dedicated node pool per service. Same per-replica cost as sharing, but more surface area and more ownership overhead, so we kept it shared.
10. What's still open
One problem is still open.
Cold start can't absorb a sharp spike. Even after the fix, it's about 2.5 minutes on a warm node and 6 on a new one. Autoscaling handles sustained load well, but it can't spin up a pod fast enough for a sudden 0-to-burst spike that lands inside that window, and during those minutes the extra requests just wait. A few ways to close the gap:
- Raise the warm floor (more idle pods) → costs money, defeats some savings
- Over-provision a "pause pod" per node → keeps a node pre-warmed for instant scheduling
- Pre-pull the (now larger) image onto nodes → shaves the ~3 min first-pull on new nodes
- Move the queue OUT of the pod (event-driven autoscaling on a durable topic)
→ the autoscaler could then see backlog before any pod exists,
and even scale to true zero
The durable-queue approach (event-driven autoscaling reading a pub/sub topic's depth) is where this is headed. It makes scaling predictive and would let the service scale to true zero. It's a bigger change on the caller side, though, so we didn't let it block the cutover.
We're hiring
Gaudiy is hiring engineers, product managers, and more. If any of this is the kind of work you like (cost-aware ML infrastructure, GPU autoscaling, building at that intersection), I'd be happy to talk. Casual chats are welcome, no commitment.
Casual talk: https://gaudiy.com/recruit/casualtalk
