Skip to content
SHC Docs

The daemon's settings

Every setting the daemon reads is a key, and shc config is how a key is read and written. A key has a shipped default, and a row written with shc config set overrides it at one scope: global, a tenant, a stack, an environment, or one deployment. The most specific row wins. This page turns the keys a cluster operator actually turns and reads each one back where it takes effect. It runs on the cluster the restore page left, with every app from the app ladder on it.

These keys are the cluster’s. The restore page left prod selected, and a set with no --scope writes at the narrowest scope the context names, so clear the context first. From here a row lands at global scope unless it names another, and a command about one deployment names its tenant with -t.

home01
$ shc ctx --reset
...

A key you never set reads its default. --show-origin says where the value came from; ls --prefix lists the rows a section holds.

home01
$ shc config get timeouts.drain
300
$ shc config get timeouts.drain --show-origin
KEY VALUE FOUND SOURCE
timeouts.drain 300 true default
$ shc config ls --prefix backup
PATH VALUE SCOPE
backup.password ref+vault://backup/password global
backup.repo /var/backups/shc/repo global

The nightly prune keeps as many daily, weekly, monthly and yearly backups as these four keys say, and shc backup prune on the repositories page applies them now. Set the four. Each write prints when it takes effect: these are live, read by the next sweep.

home01
$ shc config set backup.retention.daily 7
takes effect on the running estate: the next reconcile sweep picks it up, no reinstall or restart needed
KEY VALUE SCOPE ACTIVATION
backup.retention.daily 7 global live
$ shc config set backup.retention.weekly 4
takes effect on the running estate: the next reconcile sweep picks it up, no reinstall or restart needed
KEY VALUE SCOPE ACTIVATION
backup.retention.weekly 4 global live
$ shc config set backup.retention.monthly 6
takes effect on the running estate: the next reconcile sweep picks it up, no reinstall or restart needed
KEY VALUE SCOPE ACTIVATION
backup.retention.monthly 6 global live
$ shc config set backup.retention.yearly 1
takes effect on the running estate: the next reconcile sweep picks it up, no reinstall or restart needed
KEY VALUE SCOPE ACTIVATION
backup.retention.yearly 1 global live
$ shc config get backup.retention.daily
7

A row at a narrower scope beats the global one for what it covers. Keep three daily backups of the prod tenant’s deployments: --scope tenant with the tenant named. Read with the tenant named and its row answers; read without it and the global row does. --scope on get reads exactly one tier, and on ls lists it.

home01
$ shc config set backup.retention.daily 3 --scope tenant -t prod
takes effect on the running estate: the next reconcile sweep picks it up, no reinstall or restart needed
KEY VALUE SCOPE ACTIVATION
backup.retention.daily 3 tenant live
$ shc config get backup.retention.daily -t prod
3
$ shc config get backup.retention.daily
...
7
...
$ shc config get backup.retention.daily --scope tenant -t prod
3
$ shc config get backup.retention.daily --scope global
7
$ shc config ls --scope tenant -t prod
PATH VALUE SCOPE
backup.retention.daily 3 tenant
this listing covers the tenant scope only: 22 further config rows are set at narrower scopes in tenant master and are NOT shown above
$ shc config ls --prefix backup.retention -t prod
...

The read without a tenant still answers 7, and it says on stderr that a row at tenant scope in prod was passed over, with the command that lists it. ls --all lists every row of a key at every scope, each with the tenant, stack and environment it is set at.

home01
$ shc config ls --all --prefix backup.retention.daily
PATH VALUE SCOPE TENANT
...

Not every key is read at every scope. Retention is read at the global, tenant and stack tiers and nowhere else, so a row at the environment tier would be stored and consulted by nothing; shc config set refuses it and names the tiers that work.

home01
$ shc config set backup.retention.daily 3 --scope environment -t prod -e staging
shc: [X400066CFGSCP] config key backup.retention.daily is never read at environment scope, nothing would consult this row, set it at stack, tenant, global instead

unset removes one row at one scope and nothing else; the global value shows through again.

home01
$ shc config unset backup.retention.daily --scope tenant -t prod
...
$ shc config get backup.retention.daily -t prod
7

A timeout is a deadline in seconds for one kind of operation, cluster wide. timeouts.drain is the budget for each relocation a drain makes, not for the whole drain: shc node drain --timeout bounds a run. The three backup budgets nest: timeouts.backup bounds the restic call, timeouts.backup_converter one app’s converter, timeouts.backup_stack the whole capture of one deployment. timeouts.quiesce bounds taking the freeze a backup asks an app for and timeouts.quiesce_hold keeping it; past the hold the backup stops the app rather than hold a freeze of unknown age. A row is a plain number and reads back as one.

home01
$ shc config set timeouts.drain 120
KEY VALUE SCOPE
timeouts.drain 120 global
$ shc config get timeouts.drain --show-origin
KEY VALUE FOUND SOURCE SCOPE
timeouts.drain 120 true row global
$ shc config set timeouts.backup 3600
KEY VALUE SCOPE
timeouts.backup 3600 global
$ shc config set timeouts.backup_converter 10800
KEY VALUE SCOPE
timeouts.backup_converter 10800 global
$ shc config set timeouts.backup_stack 43200
KEY VALUE SCOPE
timeouts.backup_stack 43200 global
$ shc config set timeouts.quiesce 60
KEY VALUE SCOPE
timeouts.quiesce 60 global
$ shc config set timeouts.quiesce_hold 600
KEY VALUE SCOPE
timeouts.quiesce_hold 600 global
$ shc config set timeouts.container_stop 20
KEY VALUE SCOPE
timeouts.container_stop 20 global
$ shc config set timeouts.compose_converge 1200
KEY VALUE SCOPE
timeouts.compose_converge 1200 global
$ shc config set timeouts.exec_idle 3600
KEY VALUE SCOPE
timeouts.exec_idle 3600 global
$ shc config set timeouts.log_idle 600
KEY VALUE SCOPE
timeouts.log_idle 600 global
$ shc config ls --prefix timeouts
...

The other timeouts bound the daemon’s own calls: its Keycloak and database connections, the store coming up and shutting down, a mesh ping, a node enrolling, a repository clone and fetch, and how long an exec waits after its stop signal before the kill. Their defaults fit a lab and they are set the same way.

explained, not walked: the daemon's own deadlines, whose defaults fit a lab
$ shc config set timeouts.keycloak 20
$ shc config set timeouts.db_connect 20
$ shc config set timeouts.rqlite_ready 120
$ shc config set timeouts.rqlite_shutdown 20
$ shc config set timeouts.mesh_ping 20
$ shc config set timeouts.enroll 120
$ shc config set timeouts.git_clone 1200
$ shc config set timeouts.git_fetch 600
$ shc config set timeouts.exec_kill_wait 10

A trailing ! on the key removes the row, the same as unset. Put the drain budget back and the default reads again.

home01
$ shc config set timeouts.drain!
...
$ shc config get timeouts.drain
300

shc constraint explain prints two of the scheduler’s keys for a deployment: its priority on the first line and, at the end, how long its replicas wait after their node’s lease expires before they move. Read web.

home01
$ shc constraint explain -s web -t prod
deployment prod/web/prod movability free priority normal
running on home03
status running
DERIVED CONSTRAINTS
limit condition ready only ready nodes take new work keeps far01 home01 home02 home03
limit architecture images the replica's images must have a manifest for the node's architecture keeps far01 home01 home02 home03
limit fit room the replica needs capacity for its CPU and memory reservations; home01: cpu: past capacity, 3100m committed of 2000m; home02: cpu: past capacity, 5350m committed of 2000m removes home01 home02; keeps far01 home03
eligible far01 home03
RELOCATION
far01: possible
shc update -s web -e prod --where 'node=far01'
failover 1m0s after its node's lease expires: a new assignment inside its eligible set

Both keys are read through the scope cascade, so a tenant, a stack or one deployment can be set apart from the rest. scheduler.priority orders the scheduler’s work, high first, and never preempts; scheduler.grace.stateless is the wait for a replica with no volume, scheduler.grace.member for one with a volume or in a group, which never waits less than a minute. Raise the priority and lengthen the grace for everything in prod, at the tenant tier, and explain reads both on web.

home01
$ shc config set scheduler.priority high --scope tenant -t prod
KEY VALUE SCOPE
scheduler.priority high tenant
$ shc config set scheduler.grace.stateless 90s --scope tenant -t prod
KEY VALUE SCOPE
scheduler.grace.stateless 90s tenant
$ shc constraint explain -s web -t prod
deployment prod/web/prod movability free priority high
running on home03
status running
DERIVED CONSTRAINTS
limit condition ready only ready nodes take new work keeps far01 home01 home02 home03
limit architecture images the replica's images must have a manifest for the node's architecture keeps far01 home01 home02 home03
limit fit room the replica needs capacity for its CPU and memory reservations; home01: cpu: past capacity, 3100m committed of 2000m; home02: cpu: past capacity, 5350m committed of 2000m removes home01 home02; keeps far01 home03
eligible far01 home03
RELOCATION
far01: possible
shc update -s web -e prod --where 'node=far01'
failover 1m30s after its node's lease expires: a new assignment inside its eligible set

The member grace and the move rate are global. scheduler.moves_per_minute caps the automatic moves the cluster starts inside any minute, a reschedule, a failover or a drain’s moves alike, and a manual move is never counted. It is read when the daemon starts, so a change waits for the next restart; the upgrade page restarts every node.

home01
$ shc config set scheduler.grace.member 10m
KEY VALUE SCOPE
scheduler.grace.member 10m global
$ shc config get scheduler.grace.member
10m
$ shc config set scheduler.moves_per_minute 3
KEY VALUE SCOPE
scheduler.moves_per_minute 3 global
$ shc config get scheduler.moves_per_minute
3

Take the two tenant rows away and web reads as it did.

home01
$ shc config unset scheduler.priority --scope tenant -t prod
...
$ shc config unset scheduler.grace.stateless --scope tenant -t prod
...
$ shc constraint explain -s web -t prod
deployment prod/web/prod movability free priority normal
running on home03
status running
DERIVED CONSTRAINTS
limit condition ready only ready nodes take new work keeps far01 home01 home02 home03
limit architecture images the replica's images must have a manifest for the node's architecture keeps far01 home01 home02 home03
limit fit room the replica needs capacity for its CPU and memory reservations; home01: cpu: past capacity, 3100m committed of 2000m; home02: cpu: past capacity, 5350m committed of 2000m removes home01 home02; keeps far01 home03
eligible far01 home03
RELOCATION
far01: possible
shc update -s web -e prod --where 'node=far01'
failover 1m0s after its node's lease expires: a new assignment inside its eligible set

The maintenance keys bound what the control plane keeps. Events are kept event_retention_days; a code its module declares chatter lives event_retention_short_days, one it declares a significant transition event_retention_long_days, and both are tiers over the first number. event_max_rows caps the table by count. A row still waiting to be broadcast to a peer is protected from the pruner for event_broadcast_grace_days; event_pending_max and event_pending_ttl_seconds bound how many such rows may wait and for how long. A node prunes images once its disk passes disk_auto_prune_threshold percent, exited job containers are reaped after stale_jobs_retention_hours, and the leader keeps control_plane_snapshot_retention nightly snapshots of the store. The four containment keys cap what one event row may cost and how many identical rows one code may write inside a window. All are global and live.

home01
$ shc config get maintenance.event_retention_days
14
$ shc config set maintenance.event_retention_days 7
...
$ shc config set maintenance.event_retention_short_days 1
...
$ shc config set maintenance.event_retention_long_days 30
...
$ shc config set maintenance.event_max_rows 200000
...
$ shc config set maintenance.event_broadcast_grace_days 2
...
$ shc config set maintenance.event_pending_max 5000
...
$ shc config set maintenance.event_pending_ttl_seconds 1800
...
$ shc config set maintenance.disk_auto_prune_threshold 70
...
$ shc config set maintenance.stale_jobs_retention_hours 12
...
$ shc config set maintenance.control_plane_snapshot_retention 14
...
$ shc config set maintenance.event_payload_max_bytes 16384
...
$ shc config set maintenance.event_message_max_bytes 4096
...
$ shc config set maintenance.event_coalesce_window_seconds 30
...
$ shc config set maintenance.event_coalesce_max_per_window 5
...
$ shc config get maintenance.event_retention_days
7
$ shc config ls --prefix maintenance
...

shc db purge on the upgrade page reports what the retention rules would take; with the days cut to seven it reports more than it did.

home01
$ shc db purge
...

Nine more maintenance keys stay as shipped on this ladder. Six of them bound the audit records: the audit table is kept 365 days and the audit events 90, neither is capped by count, and identical audit rows are never coalesced into one with a count. Shortening the days, capping the rows or coalescing is an estate owner’s ruling, because deleted audit records are gone and the audit chain shows the cut forever. The last three fold the vault’s internal reads into one audit row per window; zero restores a row per read.

explained, not walked: these delete or fold compliance records, an estate ruling and not a lab step
$ shc config set maintenance.audit_retention_days 180
$ shc config set maintenance.audit_max_rows 1000000
$ shc config set maintenance.event_audit_retention_days 90
$ shc config set maintenance.event_audit_max_rows 1000000
$ shc config set maintenance.audit_coalesce_window_seconds 60
$ shc config set maintenance.audit_coalesce_max_per_window 10
$ shc config set maintenance.vault_read_audit_window_seconds 0
$ shc config set maintenance.vault_read_audit_max_keys 2048
$ shc config set maintenance.vault_read_audit_steady_attest_seconds 3600

Put the event retention back to the shipped fourteen days.

home01
$ shc config unset maintenance.event_retention_days
...
$ shc config get maintenance.event_retention_days
14

log.level has no shipped row: the daemon starts at info unless its environment says otherwise. A row written here is picked up by every node’s running daemon, which logs the change. Raise it, read the daemon’s log, and take the row away again.

home01
$ shc config set log.level debug
KEY VALUE SCOPE
log.level debug global
$ shc config get log.level
debug
$ shc cluster logs home01 -S shc -n 5
...
$ shc config set log.level!
...

The self-heal reconcile runs on the leader. health.reconcile_enabled switches it, health.check_infrastructure and health.check_docker_networks switch two of its passes, and health.adopt_keycloak_realms is whether a realm found in the system Keycloak that is no tenant becomes one. Each is read on the next pass. deploy.placement_scoring is whether an install ranks every node or only the one that received it; it is read at each install. concurrency.queue_max_depth is the ceiling of the daemon’s work queues, read live. timing.tick_interval is how often each node measures its containers, read when the daemon starts, and timing.slow_poll the pace of its slow loops, read on the next cycle.

home01
$ shc config set health.reconcile_enabled true
KEY VALUE SCOPE
health.reconcile_enabled true global
$ shc config set health.check_infrastructure false
KEY VALUE SCOPE
health.check_infrastructure false global
$ shc config set health.check_docker_networks false
KEY VALUE SCOPE
health.check_docker_networks false global
$ shc config set health.adopt_keycloak_realms false
KEY VALUE SCOPE
health.adopt_keycloak_realms false global
$ shc config ls --prefix health
...
$ shc config set deploy.placement_scoring off
KEY VALUE SCOPE
deploy.placement_scoring off global
$ shc config get deploy.placement_scoring
off
$ shc config set concurrency.queue_max_depth 2000
KEY VALUE SCOPE
concurrency.queue_max_depth 2000 global
$ shc config get concurrency.queue_max_depth
2000
$ shc config set timing.tick_interval 'every 10 seconds'
KEY VALUE SCOPE
timing.tick_interval every 10 seconds global
$ shc config get timing.tick_interval
every 10 seconds
$ shc config set timing.slow_poll 2
KEY VALUE SCOPE
timing.slow_poll 2 global
$ shc config get timing.slow_poll
2

The deploy quota keys are the other activation class. A limit ratio, a reservation ratio or a quota is baked into a deployment’s render when it is installed; a running container keeps the figure it was installed with until shc update renders it again, and the write says so. An app’s deploy.resources.limits is read as the deployment’s quota when nothing higher sets one, and the container runs with quota x deploy.limit_ratio (1.5 by default), so gitlab’s cpus: "2.0" runs with a 3.0 CPU cap. shc inspect <stack> shows the rendered figure. shc config set deploy.limit_ratio 1, then shc update <stack>, makes the cap the authored number when that authored limit supplies the quota.

home01
$ shc config set deploy.limit_ratio 1.2
read only while a deployment is rendered: a running container keeps the value it was installed with, run `shc update <stack>` to apply it
resource figures are baked into the rendered compose: run `shc update <stack>` to re-render an existing deployment at the new value, and `shc inspect <stack>` to read the limits its rendered compose gives each container
KEY VALUE SCOPE ACTIVATION
deploy.limit_ratio 1.2 global install
$ shc config get deploy.limit_ratio --show-origin
KEY VALUE FOUND SOURCE SCOPE
deploy.limit_ratio 1.2 true row global

The rest of that family are set the same way: a CPU and a memory quota per deployment, the ratio of a declared limit that becomes the reservation the scheduler counts, and that ratio per axis. This ladder renders nothing again, so they are not typed.

explained, not walked: install-time figures, and this ladder renders nothing again
$ shc config set deploy.cpu_quota 2
$ shc config set deploy.memory_quota 4g
$ shc config set deploy.reservation_ratio 0.5
$ shc config set deploy.cpu_reservation_ratio 0.25
$ shc config set deploy.memory_reservation_ratio 1.0

Take the rows this section wrote away. The four health keys, the scoring, the queue depth and the two timing keys go back to their defaults; the limit ratio stays until the next render.

home01
$ shc config set health.reconcile_enabled!
...
$ shc config set health.check_infrastructure!
...
$ shc config set health.check_docker_networks!
...
$ shc config set health.adopt_keycloak_realms!
...
$ shc config set deploy.placement_scoring!
...
$ shc config set concurrency.queue_max_depth!
...
$ shc config set timing.tick_interval!
...
$ shc config set timing.slow_poll!
...
$ shc config set deploy.limit_ratio!
...
$ shc config ls --prefix health
...

Some keys are the daemon’s own, and shc config set is not how they are set. dev reads false and shc config set refuses it.

home01
$ shc config get dev
false

dev and the two collector cardinality caps are read once, from /etc/shc/config.yaml on the node, when the daemon starts; a row in the store is refused rather than stored and ignored.

explained, not walked: shc config set refuses them, they are read from /etc/shc/config.yaml when the daemon starts
$ shc config set dev true
$ shc config set collector.max_cardinality_per_instrument 2000
$ shc config set collector.diagnostic_cardinality 200

The log format, the store’s path or URL, the internal listener’s bind address and whether it is on, the token the nodes present to one another and the default environment are read once while the daemon starts, some before the store is even open, from its environment on the node: SHC_LOG_FORMAT, SHC_DATABASE_PATH, SHC_INTERNAL_TOKEN, SHC_ENVIRONMENT. A row written here is stored and read back, and the running daemon keeps what it started with.

explained, not walked: read from the daemon's environment on each node when it starts
$ shc config set log.format json
$ shc config set database.path /var/lib/shc/shc.db
$ shc config set database.url sqlite:///var/lib/shc/shc.db
$ shc config set server.tcp.enabled true
$ shc config set server.tcp.bind 0.0.0.0:4001
$ shc config set internal_api.token P@ssw0rd
$ shc config set environment staging

crons and modules are sections, not keys: each member is named in code and set by its own path. A cron’s schedule is overridden under its name, as shc cron ls on the upgrade page lists them; a platform module’s host, path or variables under the module’s name, as the observability page sets the OpenObserve password. This ladder keeps the shipped schedules and module set.

explained, not walked: sections whose members the daemon names in code, and this ladder keeps the shipped schedules and module set
$ shc config set crons.db_snapshot_cron '0 2 * * *'
$ shc config get crons
$ shc config set modules.keycloak.host auth.example.com
$ shc config get modules

The five license keys steer the licence the licence page installed from a file: whether a trial is accepted without the command, whether the daemon re-checks the licence with the broker, whether the DNS record carries it, under which domain, and license.offline for an estate that never dials out. The licence here came from a file and stays as it is.

explained, not walked: the licence here is the file the licence page installed, and a lab has no broker and no zone
$ shc config set license.accept_trial true
$ shc config set license.auto_check false
$ shc config set license.dns_enabled false
$ shc config set license.domain learn.example
$ shc config set license.offline true

The three cluster keys belong to an estate that shares a DNS zone with another. cluster.dns_authority set to standby stops this cluster publishing any managed record until it is made authoritative at the handover; cluster.dns_takeover is whether it may take over records another cluster wrote; cluster.generation is the generation the records carry, and the platform writes it. One cluster owns this ladder’s zone.

explained, not walked: one cluster owns this ladder's zone, and the platform writes the generation
$ shc config set cluster.dns_authority standby
$ shc config set cluster.dns_takeover allow
$ shc config set cluster.generation g2

The backup repository on this ladder is a directory. A repository in S3, Azure, Backblaze or Google Cloud Storage takes the account’s credentials from the backup keys, whose secrets shc config get prints redacted; a ref+vault:// reference, as backup.password is, keeps the secret in the vault.

explained, not walked: the repository here is a directory, and these are an account's credentials
$ shc config set backup.aws_access_key_id AKIAEXAMPLE
$ shc config set backup.aws_secret_access_key P@ssw0rd
$ shc config set backup.azure_account_name learn
$ shc config set backup.azure_account_key P@ssw0rd
$ shc config set backup.b2_account_id 000example
$ shc config set backup.b2_account_key P@ssw0rd
$ shc config set backup.google_application_credentials /etc/shc/gcs.json

The cluster is as the restore page left it, with the retention, the timeouts, the maintenance bounds and the two global scheduler rows this page set. shc config ls lists every row.

home01
$ shc config ls
...

That is the whole ladder.