Daemon Mode Design¶
Overview¶
Today Open Resource Broker (ORB) acts only when asked: a CLI call, a REST request, or an MCP tool call. Daemon mode adds an optional, long-running process that watches an external demand signal (for example the depth of an SQS queue, pending Kubernetes pods, or the waiting applications of a Spark Standalone master) and adjusts capacity by calling the same acquire and return orchestrators the API uses.
This document defines the process model, the extension interfaces, the single-actor guarantees, the scale-down behaviour, the configuration schema and the delivery plan. Daemon mode is opt-in and disabled by default. Kubernetes chart behaviour described here is the contract for the chart.
Goals¶
- Scale capacity up and down automatically from a pluggable demand signal.
- Reuse the existing acquire and return orchestrators. The daemon never talks to a provider directly.
- Be safe to run: dry run by default, hard capacity ceilings, no scaling on stale or missing data, no action on machines the daemon did not create.
- Let third parties add demand sources without changing ORB core.
- Run as a single process on bare metal or as its own Deployment in Kubernetes, with a clear path to multiple replicas.
Non-goals¶
- Replacing the REST API, CLI or MCP server. They remain request-driven and are unchanged.
- Co-hosting the daemon inside the API process.
- A general workflow or job scheduler. The daemon only converts a demand number into a machine count.
- Budget-based limiting, per-source release hooks and reload of the source list (see Open items).
Process model¶
The daemon is one asyncio process with one event loop, started with
orb daemon run in the foreground (suitable for containers and service
managers).
orb daemon run
|- DaemonSupervisor lifecycle, restart with backoff, shutdown ordering
| |- LeaderElector multi-replica releases only (backend "none" before that)
| |- DemandPoller[source] one per configured source; caches the latest reading
| |- Reconciler[policy] one per source + template pair; acts only as leader
| |- HealthServer /livez /readyz /metrics on daemon.health_port
Rules:
- Pollers run on every replica. They are read-only and keep
orb daemon statusaccurate on followers. Only the reconciler acts, and only while it holds leadership. - Pollers and reconcilers are isolated. An exception in one source's poll marks
that source
stale; it never stops the loop. - A stale reading (older than
max_staleness_seconds) makes the reconciler skip the cycle. The daemon never scales on missing data and never scales to zero because of an error. - The reconciler is level-triggered and idempotent. Each cycle recomputes the desired count from the latest reading and the actual count from the machine list; it does not remember earlier requests. Machines that are pending or provisioning count as actual, so a slow provider does not cause repeated over-requests.
Interfaces¶
DemandReading { source_id, value: float, unit: str, observed_at,
labels: dict, raw: dict | None }
DemandSourcePort (Protocol)
name: str
async start(config)
async stop()
async read() -> DemandReading # raises DemandSourceError(retryable: bool)
DemandSourceRegistry
register(name, factory)
create(name, config)
# third-party sources register through the entry-point group
# "orb.demand_sources"
ScalingPolicyPort (Protocol)
decide(reading, current: CapacityState, now) -> ScalingDecision
# pure: no I/O, testable with a simulated clock
ScalingDecision { target: int,
action: scale_up | scale_down | hold,
reason: str,
suppressed_by: cooldown | clamp | rate | max_instances | None }
CapacityState { actual: int, pending: int, returning: int,
spot_flagged_ids: list }
LeaderElectionPort (Protocol) # multi-replica only
acquire() / renew() / release()
is_leader: bool
fencing_token: str # opaque
Built-in sources ship as separate modules registered through the same entry-point mechanism that third parties use, so the plugin path is exercised by first-party code. The Spark source is distributed as an optional extra and is not part of core packaging.
Reconcile cycle¶
For each source and template pair, while leader:
- Take the latest cached reading. Skip the cycle if it is missing or stale.
- Build the capacity state from the machine list, filtered by template and by
the ownership label
orb:managed-by=daemon:<source>. The daemon only ever counts and acts on machines it created. - Call
policy.decide(reading, state, now). - Apply limits in order: policy
min/max,limits.max_instances,limits.max_actions_per_minute. - Record the intent (see Single-actor guarantees).
Unless
dry_runis set, callAcquireMachinesOrchestrator(input carriestemplate_id,countandclient_token) orReturnMachinesOrchestrator(input carries the machine ids andgrace_period_seconds). Actual state comes from theListMachinesquery. - Append a decision record (demand, target, actual, action, reason, dry-run flag) and run the orphan-adoption pass.
Single-actor guarantees¶
Storage-level optimistic locking prevents two writers from corrupting one record. It does not prevent two replicas from each deciding to scale up by five. The guarantees below address that.
First release: single replica¶
The first release runs exactly one daemon: leader_election.backend: none,
replicas: 1, deployment strategy Recreate.
Storage constraints
sqlanddynamodbstorage have no local data directory and need no local guard. This is the supported storage for the daemon in Kubernetes.- On bare metal, one
orb daemon runprocess may own its own data directory withjsonorsqlitestorage. - The daemon and
json/sqlitestorage are mutually exclusive in the Kubernetes chart. A second pod sharing the volume would either lose the lock race or become a second writer. Chart rendering fails for this combination.
Single-writer guard for json and sqlite
When the API process starts with json or sqlite storage, it takes an
exclusive, non-blocking flock on <data_dir>/.orb-writer.lock. If the lock is
held, the process exits non-zero with a clear error; there is no retry loop.
The lock lives on the data volume, so it holds across pods on the same node and
across any pods that mount the same volume. It is the enforcement mechanism.
A pidfile is not: it is host-local and sits on ephemeral storage in Kubernetes.
Network filesystems without flock semantics are unsupported.
The Helm chart adds a second layer: helm template fails when replicaCount > 1
with json/sqlite storage, when the daemon is enabled with json/sqlite
storage, or when daemon.replicas > 1 without leader election configured. The
API Deployment uses strategy: Recreate for json/sqlite.
Later: multiple replicas with leader election¶
A later release adds LeaderElectionPort with these backends:
| Backend | Use | Notes |
|---|---|---|
none |
single replica | always leader; default |
k8s_lease |
Kubernetes | first multi-replica backend; requires shared (sql/dynamodb) storage |
sql |
bare metal | lease row; follows k8s_lease |
Lease timing
| Setting | Value |
|---|---|
lease_seconds |
30 |
renew_seconds |
10 |
demote_after_seconds |
20 |
- The holder measures expiry on its local monotonic clock, starting when it sent the last successful renew. It never trusts a timestamp read from the lease. After 20 seconds without a successful renew it demotes: it starts no new cycles, cancels pending cycle work before the next orchestrator call, and stops writing.
- Followers never compare the lease's wall-clock timestamp with their own clock. They record the local monotonic time at which they last saw the holder's renewal change, and may attempt takeover only after 30 seconds have passed since then. The holder therefore always demotes at least 10 seconds before a follower can take over, regardless of clock skew between nodes.
- A SQL lease uses the database clock for expiry, with the same numbers.
- If the lease store is unavailable, every replica demotes after 20 seconds. No scaling happens; the API is unaffected.
Fencing
Checking is_leader before acting is not sufficient. A leader can pause (GC,
network partition, process stop) after the check and then call the provider with
an expired lease, and neither a Kubernetes Lease nor an AWS API can verify a
fencing token. These mechanisms close the gap:
- Conditional intent write. Before any provider call, the reconciler writes
its intent (source, template, window, desired count, client token) to
storage, conditioned on the current fencing token. A stale leader's write
fails and the call never happens. The SQL lease backend uses a condition on
the lease row. Kubernetes Leases have no fencing token, so with
k8s_leasethe stored token is the holder identity plus the Lease's transition count, written to a token row in sharedsqlordynamodbstorage in the same compare-and-swap that follows a successful acquire; older tokens are rejected, and the intent write is conditioned on that row (a transaction condition on DynamoDB). - Idempotency at the provider (next section).
- Orphan adoption (next section).
The residual risk is bounded: even if every mechanism fails, the number of
daemon-owned machines cannot exceed limits.max_instances.
Followers keep polling and report role: follower in orb daemon status.
AWS-side idempotency and orphan adoption¶
Client tokens. Each acquire carries a deterministic key:
client_token = sha256(source, template_id, window_start, desired_count)
window_start = floor(now / interval_seconds) * interval_seconds
A different desired_count in the same window yields a different token, so AWS
does not deduplicate those two requests. The conditional intent write and the
max_instances ceiling bound that case.
The token is passed as ClientToken to RunInstances, EC2 Fleet creation and
the other AWS create calls, so AWS deduplicates a repeated request from a second
replica or a retry. For providers without a client-token concept (Kubernetes
pods), deterministic object names and labels are derived from the same key so a
duplicate create returns AlreadyExists.
Orphan adoption. Every cycle, the reconciler lists machines labelled
orb:managed-by=daemon:<source> that have no matching ORB record. This happens
when a leader dies between the provider call and the record write, or when a
conditional write loses a race. Such machines are adopted into a record, or
returned if the policy no longer wants them.
Scale-down¶
Scale-down is two-phase: mark, then return.
- When the target is below the actual count, the reconciler selects victims and marks them as draining. Marked machines still count as actual for scale-up arithmetic.
- A marked machine is returned only when the grace period has elapsed and the latest reading still supports the lower target at that moment.
- If demand rebounds, marks are cleared and scale-up reuses draining machines before requesting new ones. Scale-up never waits on grace.
Grace period. scale_down.grace_period_seconds per source. The default is
the existing request.default_grace_period (300 seconds). The value is passed
to providers that support graceful termination.
Time source. draining_since is the storage-side timestamp of the mark
write (database time, or the write time recorded by DynamoDB). With the none
backend it is the process clock, which is the only clock. The grace comparison
uses the same source, so a timestamp written by one replica is never compared
with another replica's wall clock.
Fencing. Mark and unmark writes use the same conditional write as the intent record, so a stale leader can neither mark nor unmark.
Machine states. The daemon tracks scale-down with these transitions:
activetodraining: marked by the reconciler (fenced write).drainingtoactive: demand rebounded; mark cleared (fenced write).drainingtoreturning: grace elapsed and the reading still supports the lower target;ReturnMachinesOrchestratorcalled.returningtoreturned: the provider confirms the return. Areturningmachine is never selected again.
Victim selection order: longest-marked first, then spot-flagged, then youngest first.
Exclusions: machines with an active unfinished request, and machines already returning or with a return in flight, so a return is never issued twice.
Graceful shutdown¶
On SIGTERM, in this order:
- Stop starting reconcile cycles and set
/readyzto not ready. - Wait for the in-flight orchestrator call, up to
shutdown_timeout_seconds(default 25). A provider mutation is never cancelled mid-flight. - Drain the domain event queue, up to
event_drain_timeout_seconds(default 10). - Release the lease (multi-replica only; a no-op with backend
none). - Stop pollers, close source clients, exit 0.
The lease is released after the event drain so a follower cannot start acting while the old leader is still flushing events.
In Kubernetes, terminationGracePeriodSeconds must be at least
shutdown_timeout_seconds + event_drain_timeout_seconds + 10 (45 with the
defaults). The chart computes the value from the configured timeouts.
After SIGKILL, the lease expires and a follower takes over after 30 seconds.
Because the loop is level-triggered, the next leader recomputes from storage and
runs the orphan-adoption pass, so no state is lost.
The daemon must handle SIGTERM through the asyncio loop's signal handler, not
default Python behaviour, and orb server stop style escalation (SIGTERM
then SIGKILL on the process group) must not preempt the drain.
Health and readiness¶
The daemon serves a minimal listener on daemon.health_port (default 8081),
with no path prefix.
| Endpoint | Healthy when |
|---|---|
/livez |
The event loop is responsive and the supervisor heartbeat is within three times the interval. Liveness never depends on provider health. |
/readyz |
Config is valid, storage is reachable, and every enabled source has been read successfully within max_staleness_seconds. Independent of leadership, so followers stay in the Service. |
/metrics |
Prometheus metrics prefixed orb_daemon_. |
Pair /readyz with a generous startup probe (for example 30 failures at a 5
second period) so a slow provider credential exchange at boot does not stall a
rolling update.
Configuration¶
The daemon block is additive and defaults to disabled. Older ORB versions that
reject unknown configuration keys will reject a config containing daemon.
daemon:
enabled: false
dry_run: true # stays true until an operator changes it
interval_seconds: 15
max_staleness_seconds: 60
health_port: 8081
shutdown_timeout_seconds: 25
event_drain_timeout_seconds: 10
leader_election:
backend: none # none | k8s_lease | sql
lease_seconds: 30
renew_seconds: 10
demote_after_seconds: 20
limits:
max_instances: 100 # hard ceiling over recorded + adopted machines
max_actions_per_minute: 6
sources:
- name: sqs-jobs
type: sqs
config:
queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/jobs
region: us-east-1
template_id: batch-spot
policy:
type: target_tracking
target_per_machine: 10
min: 0
max: 50
cooldown_up_seconds: 60
cooldown_down_seconds: 300
scale_down:
grace_period_seconds: 300 # default: request.default_grace_period
A Spark Standalone source follows the same shape:
- name: spark-batch
type: spark_standalone
config:
master_url: http://spark-master:8080
template_id: spark-worker
policy:
type: target_tracking
target_per_machine: 8 # cores per worker
min: 1
max: 40
cooldown_up_seconds: 60
cooldown_down_seconds: 600
Scaling-policy schema¶
Source entry fields:
| Field | Type | Required | Default | Constraints |
|---|---|---|---|---|
name |
string | yes | unique across sources | |
type |
string | yes | a registered demand-source name | |
config |
map | yes | validated by the source | |
template_id |
string | yes | must reference an existing template | |
policy |
map | yes | see below | |
scale_down.grace_period_seconds |
int | no | request.default_grace_period |
>= 0 |
Common policy fields:
| Field | Type | Required | Default | Constraints |
|---|---|---|---|---|
type |
string | yes | target_tracking or step (or a registered policy) |
|
min |
int | no | 0 | >= 0 |
max |
int | yes | >= min |
|
cooldown_up_seconds |
int | no | 60 (proposed default) | >= 0 |
cooldown_down_seconds |
int | no | 300 (proposed default) | >= 0 |
target_tracking adds target_per_machine (float, required, > 0):
step adds steps, an ordered list evaluated against reading.value. Each
entry has exactly one of above or below (a threshold on reading.value)
and exactly one of add or remove (a positive integer machine count). List
above entries from the largest threshold down and below entries from the
smallest up. The first matching entry in each direction applies to the current
actual count, and the result is clamped to min/max:
policy:
type: step
min: 0
max: 50
steps:
- { above: 100, add: 5 }
- { above: 20, add: 1 }
- { below: 10, remove: 2 }
Cooldown semantics: cooldown_up_seconds starts at the last executed
scale-up action and suppresses further scale-up until it elapses;
cooldown_down_seconds does the same for scale-down marks and returns. A
suppressed decision is reported with suppressed_by: cooldown. Proposed: dry-run
actions do not start cooldowns.
Policy type registry: policy.type resolves to a ScalingPolicyPort
implementation through the proposed entry-point group orb.scaling_policies.
target_tracking and step are built in; third parties register additional
types the same way as demand sources.
Rules:
- Source credentials come from the environment or from files (
*_file,${ENV}), never inline in the config file. SQS uses ambient AWS credentials. - Schema validation rejects nonsensical values (zero or negative limits,
min > max) at load and reload, and the previous configuration stays active. SIGHUP(or the adminreload-configcall) re-reads configuration at a cycle boundary. Reloadable:dry_run, policy parameters,limits, intervals. Not reloadable: the source list, leader-election backend, storage and provider set; these need a restart.leader_election.backend: k8s_leaseis rejected withjsonorsqlitestorage, because the fencing token needs shared storage.
Failure modes¶
| Failure | Behaviour |
|---|---|
| Demand source down or slow | Source marked stale; reconciler skips; orb_daemon_source_stale set. No scaling on missing data. |
| Provider call fails or partially fulfils | Next cycle sees the real actual count and retries within rate limits. Repeated failures back off exponentially and show in status. |
| Flapping demand | Cooldown plus mark-then-return grace; marks clear when demand rebounds. |
| Replica race | Conditional intent write, deterministic client token, orphan adoption, and the max_instances ceiling. |
| Lease store outage | No scaling; API unaffected. |
| Machines the daemon did not create | Never touched; selection is restricted to the ownership label. |
| Restart during scale-down | Marks are stored under the fencing condition, so the new process resumes from storage. |
Rollback¶
Daemon mode is opt-in. To roll back, set daemon.enabled: false or stop the
process. Machines the daemon created remain ordinary ORB machines and can be
returned with the existing commands. Lease and token tables added for leader
election are additive and safe to leave in place.
Delivery plan¶
Planned capability milestones.
| Milestone | Capability |
|---|---|
| S1 | Core ports and registry: DemandSourcePort, DemandSourceRegistry with the orb.demand_sources entry-point group, ScalingPolicyPort, DTOs, and a fake source for tests. |
| S2 | Reconciler with dry run: reconcile cycle wired to the orchestrators, ownership label, deterministic client tokens, orphan adoption, in-memory decision records, simulated scale-up and scale-down tests. |
| S3 | Scaling policies: step scaling, target tracking, cooldown, clamping, and the mark/grace scale-down state machine with exclusions. |
| S4 | Single-replica daemon: daemon configuration schema, orb daemon run and orb daemon status, the data-directory writer lock, ordered shutdown, limited config reload, health listener. |
| S5 | Leader election: LeaderElectionPort, k8s_lease backend with the timing above, fenced intent and mark writes, multi-replica failover test. SQL lease follows. |
| S6 | Limits and observability: max_instances, rate limiting, orb_daemon_* metrics, structured decision log. |
| S7 | Demand sources: SQS and Kubernetes pending pods. The Spark Standalone source is delivered with the Spark integration work. |
| S8 | Documentation: integration guide and a guide to writing a demand source. |
S5 requires optimistic locking on SQL and DynamoDB storage. S4 requires the event-queue drain hook.
Test strategy¶
- Unit: policies with a simulated clock; reconciler with a fake source and fake orchestrators; registry and entry-point discovery; configuration validation; grace state-machine table tests.
- Concurrency: two reconcilers sharing one SQL database and a fake clock; at most one acts per window; with leader election disabled, the capacity caps still hold (negative control).
- Integration: SQS against a fake or a real queue; the Kubernetes pending-pod source against a kind cluster in CI.
- Shutdown: a subprocess test sends
SIGTERMand asserts exit code 0, event queue drained, in-flight call completed and no orphan tasks. With leader election, the lease is released after the drain and a follower acquires it within seconds. - Architecture: new ports are registered and the daemon package imports no provider code.
Open items¶
Deferred beyond the first release:
- DynamoDB lease backend and database advisory-lock alternatives.
- A per-source
can_releasehook so a source can veto returning a machine (for example, a Spark worker that is still running executors). - Budget-based limiting.
- Co-hosting the daemon inside the API process.
- Reload of the source list without a restart.
- Ownership tracking by tags beyond the single ownership label.
- A shared secret model for Spark worker bootstrap.