Python Memory Management and the HPA Trap: Why Your Pods Never Scale Down

2026-08-18

If you’ve ever run a Python service in Kubernetes with a memory-based Horizontal Pod Autoscaler (HPA), you may have seen this pattern: memory usage climbs steadily, the HPA adds replicas, traffic drops, and… nothing scales back down. The pods sit there holding memory they no longer need, and the cluster keeps paying for it.

This isn’t a leak in the classic sense. It’s a deliberate design decision in Python’s memory allocator, and it interacts badly with how container orchestrators measure memory. Let’s dig into why.

How Python allocates memory: Pymalloc

CPython doesn’t hand every allocation to the operating system. For small objects (less than 512 bytes), it uses its own allocator called pymalloc, which sits on top of the C malloc and manages memory in a hierarchy:

When you create a small object — say, a 40-byte dict entry — pymalloc finds a pool for the 40-byte size class and hands you a block from it. When you delete the object, the block goes onto a free list for that pool. The pool stays alive. The arena stays alive.

Here’s the critical part: an arena is only returned to the OS when every pool in it is completely empty. If a single 40-byte object survives in a 256 KB arena, the entire arena stays mapped. Your process RSS stays high.

Larger objects (≥ 512 bytes) skip pymalloc entirely and go straight to the C malloc, which typically calls brk or mmap and can return memory to the kernel much more readily. That’s why this problem is most visible in applications that churn through lots of small objects — web frameworks, ORMs, JSON serializers, anything that builds and discards millions of small dicts, lists, and strings.

Watch it happen

Here’s a small script that demonstrates the behavior. It allocates 10 million small objects, deletes them, and reports RSS at each stage:

import gc
import os
import sys


def rss_mb():
    with open("/proc/self/status") as f:
        for line in f:
            if line.startswith("VmRSS:"):
                return int(line.split()[1]) / 1024
    return 0


class SmallObject:
    __slots__ = ("a", "b", "c")
    def __init__(self):
        self.a = 1
        self.b = "x" * 40
        self.c = [1, 2, 3]


print(f"Start:            {rss_mb():8.1f} MB")

objects = [SmallObject() for _ in range(10_000_000)]
print(f"After allocate:   {rss_mb():8.1f} MB")

del objects
print(f"After del:        {rss_mb():8.1f} MB")

gc.collect()
print(f"After gc.collect: {rss_mb():8.1f} MB")

Run it and you’ll see something like:

Start:               8.0 MB
After allocate:    640.0 MB
After del:         640.0 MB
After gc.collect:  640.0 MB

The objects are gone. The memory is not. Pymalloc is holding hundreds of megabytes of arenas that are mostly empty, waiting for future allocations. From the kernel’s perspective — and from Kubernetes’ perspective — the process is using 640 MB.

Why this breaks HPA

Kubernetes measures container memory from cgroup accounting, which is essentially the same kernel-level view as RSS. The HPA compares that number against your target utilization and computes desired replicas:

desiredReplicas = ceil(currentReplicas * (currentMetricValue / desiredMetricValue))

The trap works like this:

  1. Traffic spikes. Your Python app allocates millions of small objects. RSS climbs.
  2. HPA sees utilization exceed the target and scales up. More pods.
  3. Traffic drops. The app deletes the objects. But RSS barely moves, because pymalloc holds the arenas.
  4. HPA still sees high utilization. It never scales down.

Worse, the scale-down window keeps getting pushed out because the metric never drops below the threshold. You end up with a fleet of pods each holding phantom memory, and the autoscaler is convinced they’re all busy.

The same distortion affects scheduling: the kube-scheduler sees inflated memory requests and spreads pods across more nodes than necessary, and the kubelet may evict pods based on a memory pressure that isn’t real.

What you can do

1. Try jemalloc

jemalloc is a general-purpose allocator that manages memory in a way that’s friendlier to returning pages to the OS. You can swap it in without recompiling Python using LD_PRELOAD:

# Debian/Ubuntu
apt-get install libjemalloc2

# Run your app with jemalloc
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 python app.py

In a container, set it in the entrypoint or via an environment variable:

ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

I’ve seen jemalloc reduce steady-state RSS meaningfully for allocation-heavy workloads, but the results were inconsistent. Some workloads showed a 20–30% reduction; others showed no change at all. The variance seems to depend on allocation patterns — object sizes, lifetimes, and how fragmented the arenas get. It’s worth benchmarking against your own workload, but don’t treat it as a guaranteed fix.

2. Tune the garbage collector

CPython’s GC is generational and reference-counting based. The thresholds that trigger collection are tunable:

import gc

# Collect more aggressively: lower thresholds mean more frequent collections
gc.set_threshold(700, 10, 10)  # (gen0, gen1, gen2)

# Or force a collection at strategic points in your app
gc.collect()

More aggressive collection frees objects sooner, which gives pymalloc a better chance of emptying pools and arenas. But there’s a real trade-off: every collection pass costs CPU, and collecting too often can measurably hurt throughput. This is an area worth exploring for your specific workload — start with the defaults, lower the thresholds gradually, and watch both RSS and latency.

3. Recycle your workers

If your Python app runs under uWSGI, the most reliable mitigation is to recycle workers before their arenas become too fragmented. uWSGI has three knobs:

[uwsgi]
# Recycle a worker after it has processed 10,000 requests
max-requests = 10000

# Recycle a worker when its RSS exceeds 300 MB
reload-on-rss = 300

# Time-based recycling (least favored — a quiet worker can be killed
# mid-request, and a busy worker can outgrow the limit before the timer fires)
max-worker-lifetime = 3600

max-requests is the most predictable: each worker handles a bounded number of requests, then exits and is replaced by a fresh process with a clean heap. reload-on-rss is a good safety net for workloads with unpredictable per-request memory. Time-based recycling works, but it’s the least precise of the three — I’d use it only as a last resort.

This pattern should feel familiar to anyone who has administered IIS: the classic IIS worker pool behavior of slowly consuming more memory until the pool is recycled is the same phenomenon. Application pools that never recycle eventually eat all available memory. The fix is the same philosophy — bound the lifetime of the worker process so fragmentation can’t accumulate forever.

The fundamental tension

Garbage collection and memory management are always a balancing act. Every allocator has to choose between two competing goals:

Pymalloc optimizes hard for the first goal. It assumes memory is cheap and allocation speed is precious — a reasonable assumption on a dedicated server, and a problematic one in a containerized world where the orchestrator watches your RSS and makes decisions based on it.

Practical takeaways

Understanding where the memory actually lives — in pymalloc’s arenas, not in your objects — is the first step to building Python services that behave well in Kubernetes.