The pods that exist to be evicted.

"Doesn't the autoscaler already do this?"

The objection goes like this. Set the autoscaler to add capacity at 70% utilisation and it fires on its own, well before you run out of room. So why pay to run pods whose entire job is to sit there doing nothing until something more important displaces them from the node?

It's a reasonable read, and it contains one factual mix-up worth clearing up first, because clearing it up makes the case for balloon pods stronger, not weaker.

There are two autoscalers, and only one of them has a percentage

The 70% number belongs to the Horizontal Pod Autoscaler. HPA watches a utilisation target and adds pods. That's the loop with a threshold you tune.

Cluster Autoscaler — the thing that adds nodes — doesn't work that way at all. It has no scale-up percentage. It watches for pods that are stuck in Pending because no existing node can fit them, and only then does it go ask the cloud provider for another machine.

The utilisation percentage you may be thinking of is --scale-down-utilization-threshold, default 0.5 — that governs when nodes get removed. It's the other direction entirely.

Ninety seconds is not a setting

Suppose that setting existed. You tell the autoscaler to order a node at 70% full instead of 90. Then 60. Then 50. Each time, you have changed when the order goes in. You have not changed how long the delivery takes.

The node still has to be allocated by the cloud provider, boot, run cloud-init, start the kubelet, register with the API server, and pass its readiness checks. On EC2 with a warm-ish AMI that's somewhere between sixty and a hundred and twenty seconds, and no YAML you write changes it.

That's the actual choice. The wait is fixed. What's negotiable is whether it happens while a user is staring at a spinner, or quietly in the background while your pod is already serving.

Where the ninety seconds goes

Same traffic spike, same cluster, same instance type. The only variable is whether there was a balloon holding a seat.

waiting on infrastructure instant work that happens either way

Toggle it. Notice that the red bar doesn't disappear in the second view — it moves. The node still takes ninety seconds to arrive. It just arrives after your pod started serving, because your pod took the seat the balloon was keeping warm.

A pod with negative priority is a pod that can be displaced

The trick rests on preemption. Give the balloon a PriorityClass with a negative value and it sits below every normal workload in the cluster. When a real pod can't find a home, the scheduler doesn't wait politely — it evicts the balloon and puts the real pod on that node. Sub-second, no cloud API call, no boot.

The elegant part is what happens next. The evicted balloon goes to Pending, which is exactly the condition Cluster Autoscaler is watching for. So it requests a new node, the balloon lands there, and your headroom is restored — all off the critical path.

# The balloon sits below everything real.
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: balloon
value: -1
globalDefault: false
description: "Placeholder capacity. Evict me freely."
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: balloon
spec:
  replicas: 3
  selector:
    matchLabels: {app: balloon}
  template:
    metadata:
      labels: {app: balloon}
    spec:
      priorityClassName: balloon
      terminationGracePeriodSeconds: 0   # leave instantly
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.9
        resources:
          requests:
            # size this to your LARGEST real pod
            cpu: "2"
            memory: 4Gi

The resource requests are the whole point. A balloon reserves capacity by asking for it — the pause container itself uses about a megabyte and does nothing but sleep. You're not buying compute, you're buying a reservation.

The failure modes are all silent

None of these throw an error. The balloons sit there looking perfectly healthy while quietly doing the wrong thing, which is the expensive kind of broken.

The manifest, with the parts people leave out

Two objects. The comments mark the four lines that decide whether this works or quietly does nothing.

# ── 1 ── The priority class ─────────────────────────────
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: balloon
value: -1                       # above CA's -10 expendable cutoff
globalDefault: false
preemptionPolicy: Never          # balloons yield, never displace others
description: "Placeholder capacity. Evict me freely."
---
# ── 2 ── The balloons ───────────────────────────────────
apiVersion: apps/v1
kind: Deployment
metadata:
  name: balloon
  namespace: kube-system
spec:
  replicas: 3                    # replicas x requests = your headroom
  selector:
    matchLabels: {app: balloon}
  template:
    metadata:
      labels: {app: balloon}
    spec:
      priorityClassName: balloon
      terminationGracePeriodSeconds: 0

      # Without this, all three can land on one node —
      # which is one node of headroom, not three.
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway   # DoNotSchedule can deadlock
        labelSelector:
          matchLabels: {app: balloon}

      containers:
      - name: pause
        image: registry.k8s.io/pause:3.9
        resources:
          requests:                # the reservation IS the request
            cpu: "2"
            memory: 4Gi
          limits:
            cpu: "2"
            memory: 4Gi

Then confirm it does something, rather than assuming:

# Are the balloons actually spread, or stacked?
kubectl get pods -l app=balloon -o custom-columns=\
NAME:.metadata.name,NODE:.spec.nodeName

# Watch a preemption happen for real
kubectl get events -A --field-selector reason=Preempted -w

# Confirm CA will replace an evicted balloon
# (this must be ABOVE the balloon's -1)
kubectl -n kube-system get deploy cluster-autoscaler -o yaml \
  | grep expendable-pods-priority-cutoff

If that last command returns nothing, you're on the default of -10 and -1 is safe. If someone has lowered it past your balloon's value, the buffer works exactly once.

Balloons only pay off with KEDA

The scaler you choose decides whether balloons save you or just sit there costing money.

HPA sees CPU at 68% and concludes things are fine. KEDA sees 500 messages in the queue and knows more pods are needed. That gap in timing is the whole argument: balloons buy time while new replicas start, and KEDA fires while there's still headroom to land in.

If you're on HPA, the honest answer is to tune it properly and skip balloons entirely. A well-tuned threshold and a workload that tolerates the wait is a cheaper, simpler system than one paying month-round for headroom it consumes late. Balloons don't fix a scaler that reacts too slowly — they just make the bill larger.

How many balloons?

Every manifest in this post says replicas: 3. That number is a placeholder. The real one is a function of two things you have to measure yourself: how long a node takes to arrive, and how fast pods show up during a burst. Fleet size doesn't enter into it — a service running ten pods and one running two hundred need the same headroom if they burst at the same rate.

1 · Measure how long a node actually takes

Not the number in the vendor doc. Yours. Run this on a quiet cluster three to five times and take the median.

# note where you're starting from
kubectl get nodes | wc -l

# force pods that can't fit, so CA has to add a node
kubectl create deploy burst-probe \
  --image=registry.k8s.io/pause:3.9 --replicas=100
kubectl set resources deploy/burst-probe \
  --requests=cpu=1

# watch for the new node and note when it goes Ready
kubectl get nodes -w

kubectl delete deploy burst-probe

Sixty to a hundred and eighty seconds is the usual range. Image pull dominates it, so a fat image on a cold node sits at the top of that band. This number is stable until you change your node pool or your base image — measure it once and write it down.

2 · Find your burst arrival rate

How many pods land per second in your worst spike. If you're running KEDA this is derivable: queue depth over your value threshold, divided by the polling interval. Otherwise pull it from your own scale events — the honest source is your metrics, not a formula.

3 · Pick what you're willing to make wait

This is the step people skip, and it's the one that decides everything. Balloons cover the front of a burst, not the burst. With a KEDA trigger at value: 5, a five-hundred-message jump wants a hundred pods — no balloon fleet holds that, and you shouldn't pay for one that tries. So the question isn't "how big is my peak," it's "how many pods must start now, before nodes arrive." The tail waits, on purpose.

replicas = (burst_rate × seconds_you_refuse_to_wait) ÷ pods_per_balloon

Two pods a second, twenty seconds you're unwilling to lose, one app pod per balloon: forty replicas. If your balloon is sized to hold two app pods, twenty. That's the starting point — not the answer.

4 · Then watch it

The formula gets you to a first deploy. The cluster tells you the rest:

• Balloons evicted and re-inflating on every spike — sized about right.
• Balloons never touched — you're paying rent on headroom you don't use. Come down.
• App pods still Pending with balloons present — undersized, or the balloons are smaller than the pods they're supposed to make room for. Check which before you raise the count.

KEDA + balloons: complete example

Three objects. Deploy them in order, and you have a system that scales ahead of demand.

1. The balloons (same as before)

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: balloon
value: -1
globalDefault: false
preemptionPolicy: Never
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: balloon
  namespace: kube-system
spec:
  replicas: 5
  selector:
    matchLabels: {app: balloon}
  template:
    metadata:
      labels: {app: balloon}
    spec:
      priorityClassName: balloon
      terminationGracePeriodSeconds: 0
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels: {app: balloon}
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.9
        resources:
          requests:
            cpu: "2"
            memory: 4Gi

2. The KEDA ScaledObject (watches queue depth)

# Scales the worker on RabbitMQ queue depth,
# not CPU. Fires before utilisation climbs.
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: rabbitmq-auth
  namespace: production
spec:
  secretTargetRef:
  - parameter: host
    name: rabbitmq-secret        # holds the full amqp:// URI
    key: host
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: job-processor-scaler
  namespace: production
spec:
  scaleTargetRef:
    name: job-processor
  minReplicaCount: 2              # baseline, never scale to zero
  maxReplicaCount: 200
  pollingInterval: 15             # how often KEDA checks the queue
  cooldownPeriod: 120
  fallback:
    failureThreshold: 3
    replicas: 4                   # if the scaler breaks, hold this
  triggers:
  - type: rabbitmq
    metadata:
      protocol: amqp
      queueName: jobs
      mode: QueueLength
      value: "5"              # one replica per 5 queued messages
    authenticationRef:
      name: rabbitmq-auth

3. The worker deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: job-processor
  namespace: production
spec:
  replicas: 2                     # KEDA will override this
  selector:
    matchLabels: {app: job-processor}
  template:
    metadata:
      labels: {app: job-processor}
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels: {app: job-processor}
              topologyKey: kubernetes.io/hostname
      containers:
      - name: processor
        image: myregistry/job-processor:1.2.3
        resources:
          requests:
            cpu: "2"
            memory: 4Gi
          limits:
            cpu: "2"
            memory: 4Gi
        env:
        - name: RABBITMQ_HOST
          value: rabbitmq.rabbitmq
        - name: RABBITMQ_QUEUE
          value: jobs

To verify it's working:

# Watch KEDA respond to queue depth changes
kubectl get hpa keda-hpa-job-processor-scaler -w

# Is the trigger healthy? READY and ACTIVE should be True
kubectl get scaledobject job-processor-scaler -n production

# Confirm balloons are spread across nodes
kubectl get pods -l app=balloon -o wide

Most clusters shouldn't do this

Balloon pods convert money into latency, continuously, whether or not you ever spike. The pairing matters more than the balloons themselves.

Sixty seconds, and one way to lose all of them

So I built the cluster and tried it. Six pods, small nodes, a burst big enough that the cluster had to go and buy a node to finish the job.

No balloons: the last pod started serving at 65 seconds.

With balloons: five.

And it wasn't luck. The log had six Preempted events, each one naming the balloon that got out of the way and the pod that took its chair. Both runs used the same tiny image, so this is boot time only — a real image would make the gap bigger, not smaller.

Sixty seconds off the burst, for the price of one idle node.

Different knobs, different questions

A scale-up threshold answers when do we ask for more. Balloon pods answer where does the next pod land while we wait. They're not alternatives, and tuning the first one harder will never do the second one's job.

Whether that trade is worth it comes down to one question you can answer without any Kubernetes knowledge at all: during those ninety seconds, is anyone waiting?