Sleeping workers: a zone that costs one small pod while nobody calls it
Since 1.0 a group's zone can go to sleep when it is idle: its workers are scaled to 0 and one small activator pod answers in their place. The first authenticated MCP call wakes the zone and is answered by the real workers. Sleeping is off by default, per group × zone.
Turning it on
A super admin or an admin of the group sets it; nobody else sees the control, and an API attempt by anyone else is
403.
- Group page → Zone actions → Sleep: Sleep after idle (minutes, 0 = never), then Save sleep. The same row
shows the zone's state (
awake,asleeporwaking) and a Sleep now or Wake now button. - API:
PUT /api/v1/groups/{g}/zones/{z}/workers {"sleep_after_secs": 900}.0means never (the default); anything else must be 60 or more, or the answer is422. - By hand:
POST /api/v1/groups/{g}/zones/{z}/sleepand…/wake, same roles.

b set to sleep after 15 idle minutes and asleep now, with Wake now; the zone row
above shows the state. (Seeded console: its stand-in workers keep answering, where a cluster zone would run none.)What "idle" means
The console checks every zone that has the setting on each pass of its scheduler loop (the interval is the auto-rebalance scheduler's, 60 s by default; the idle check runs whether or not auto-rebalance is enabled). A zone is put to sleep when all of this holds:
- every worker pod of the zone reports a last accepted call at least
sleep_after_secsago (or none at all), - every pod reports 0 calls in flight, 0 running tasks and 0 open
subscriptions/listenstreams, - the zone was last woken, deployed or put to sleep at least
sleep_after_secsago.
A pod that cannot be read, or a worker from before 1.0 (it reports no last-call time), keeps the zone awake.
What moves a worker's last-call time is an MCP call the node accepted: one that passed the credential, the throttle and the in-flight cap. So these do not keep a zone awake:
- calls that were refused (no or wrong credential, throttled, over the in-flight or task cap),
- health checks, gRPC reflection and the
Adminservice, - the console's own calls — opening the group page, the package list, golden cases. The console marks them with the zone's worker admin key, and the node does not count a call that carries it.
A running task and an open subscriptions/listen stream do keep a zone awake, for as long as they last. Opening a
stream does not itself move the last-call time; the open stream is what counts.
What a sleeping zone runs
Going to sleep, in this order: the console starts Deployment/worker-activator (one pod) in the zone's namespace
and waits until it is ready, then scales worker and worker-canary to 0. The stopping workers finish what is in
flight during their SIGTERM drain (RAMEN_STOP_DRAIN_SECS).
The activator is the zone's own worker image started with RAMEN_ACTIVATOR=1, with the zone's whole worker
configuration and no Python runtime. It sits behind the zone's existing worker Service, so the load
balancer's route and health check do not change, and it answers health as serving. Real workers are also reachable
inside the namespace through a second Service, worker-direct, which the activator forwards to and which no load
balancer points at.
Cost. One pod per sleeping zone, requesting 10m CPU and 32Mi memory (limits 250m / 128Mi), instead of the zone's worker count plus its canary. On GKE Autopilot the saving is smaller than those numbers suggest: Autopilot bills per pod and applies its own minimum to a pod's requests, so the activator is billed at that minimum.
On AWS the canary is taken out of the load balancer's weighted split before the workers stop, and put back on wake.
On the local compose stack the state changes and nothing else does: the one compose worker keeps running.
What a caller sees
The activator runs the same guards as a worker, with the zone's own settings: IP rules, Origin, credential,
protocol checks and throttle. A call the workers would refuse is refused the same way and never wakes the zone.
A call that is accepted:
- The activator asks the console to wake the zone. One request per wake; calls that arrive meanwhile share it.
- The call is held while the activator polls the workers' health every 500 ms.
- When a worker answers, the request is forwarded unchanged (HTTP: path, headers and body; gRPC: the
Mcp/Callbytes and metadata) and the answer is passed back as it arrives, an event stream included.
If no worker is serving within RAMEN_WAKE_TIMEOUT_SECS (default 25 s), the caller gets
HTTP/1.1 503 Service Unavailable
Retry-After: 10
{"error":"waking: the zone is starting, retry shortly"}
or gRPC UNAVAILABLE with the same message. The wake goes on, so the retry finds the zone awake or is held
again. The default sits under the load balancers' own backend timeouts (30 s on a GKE backend service, 60 s on an
AWS load balancer) so that a slow wake is Ramen's 503 with a Retry-After, not the load balancer's 504. How
long a wake takes is mostly how long a worker pod takes to become ready; when the cluster has to add a node for
it, expect the first call to get the 503.
Waking, in this order: the console scales worker back to the count the zone had (at least 1) and worker-canary
to what it had, waits until they are ready, marks the zone awake, and only then scales the activator to 0. While
both run the activator forwards, so no call is refused in between. A wake that fails leaves the zone asleep and
the next call asks again.
The telemetry consent check is not made by the activator; the worker makes it on the forwarded call.
The session secret
The activator tells the workers which client address its guard resolved, and that the throttle was already
counted, with a proof derived from the zone's RAMEN_SESSION_SECRET. A worker that verifies the proof uses that
address for its IP rules and its log, and does not throttle the call a second time. Zones the console deploys have
the secret (it is generated per group and written on every deploy). A worker started without one generates its
own per pod; its activator's proof then does not verify, the workers ignore what it says about the caller, and the
activator logs a warning at start. Do not use sleeping on workers that run without the deploy-provided secret.
With deploys, rebalance, autoscaling and IP rules
| Action on a zone that is not awake | What happens |
|---|---|
| Deploy | The zone is woken first (trigger:deploy), then deployed as usual. A deploy also counts as activity: the zone does not go back to sleep sooner than its setting afterwards. |
| Rebalance (the button or the API) | Wakes the zone first (trigger:manual). |
| Auto-rebalance scheduler | Skips the zone: it is neither rebalanced nor autoscaled. |
| Autoscaling | Holds, with reason zone is asleep or zone is waking. The 1..max bounds apply to an awake zone; asleep is the one case with 0 workers. |
| Count or size change | Wakes the zone first, then applies. |
| Max or sleep setting change | Stored; nothing is started for it. |
| IP rules | Applied to the activator as well: it is restarted with the new allow-list and enforces it. |
Audit and logs
| Audit action | Tags |
|---|---|
workers.sleep_after |
from:, to: (seconds) |
workers.sleep |
trigger:idle (actor scheduler) or trigger:manual |
workers.wake |
trigger:call (actor activator), trigger:manual or trigger:deploy; count: once the workers are up |
| either, failed | error: with the reason |
| any, refused | error:forbidden |
In the access log every line an activator writes carries activator: true. A forwarded call is one line there
with forwarded: true, woke (whether this call waited for the wake) and, on HTTP, upstream_status; the worker's
own line for the same call carries via: "activator". A call that timed out has reason waking; one whose worker
went away mid-call has reason upstream and gets the same 503.
Limits
- The first call to a sleeping zone is answered
503when the wake takes longer than the hold. Clients that do not retry on503see an error once. - A zone that sleeps needs the deploy-provided session secret (above).
- The activator is one pod. While it restarts (a node drain, an IP-rule change) a sleeping zone does not answer.
What is verified
By tests:
tests/conformance/test_activator.py: a real node in activator mode in front of a real worker — the guards before any wake, concurrent calls sharing one wake, the hold, forwarding on HTTP and gRPC with headers, metadata and the worker's own errors, an event stream and asubscriptions/listenstream through it, the503withRetry-After.tests/conformance/test_idle_metrics.py: what moves a worker's last-call time and what does not.tests/conformance/test_console_sleep_otel.py: the setting and its roles, sleep and wake through the console API, the internal wake endpoint, the audit entries.tests/kind/test_v1_sleep_build_otel.pyholds the cluster proofs (an idle zone sleeps, a call through the edge wakes it and is answered, an open stream or a running task keeps it awake, a deploy wakes it first).
On GKE, EKS and AKS (1.0.0, 2026-11-13 to 2026-11-14): an idle zone slept and a call through each cloud's real
load balancer woke it and was answered. EKS answered the waking call after 13 s (held, no retry). GKE and AKS
answered after 1 s, from workers that were still draining while the zone was woken behind them. On GKE Autopilot
a cold wake can take longer than the 25 s hold: the caller then gets the activator's 503 with Retry-After: 10
(seen once) and retries; it never got the load balancer's 504.
The kind numbers and the rest of the cloud run are on the Transport page. A worker skipping its throttle for a forwarded call is not exercised by a test.