Maintenance and diagnosis¶

This document covers the day-to-day operation of the platform itself. For functional use, see Getting started and the documents it references.

Maintenance¶

backup maintenance                 # applies retentions, reclaims space
backup maintenance --tenant acme   # one customer only
backup maintenance --full          # includes repository compaction
backup storage audit               # checks versioning and locking
backup support export --days 30    # a pseudonymised bundle for the vendor
backup support resolve <label>     # what a label the vendor quoted stood for

Maintenance runs from the platform, under an identity distinct from the agents'. Agents hold write permissions only: a compromised agent can erase nothing.

Diagnosis¶

Symptom	Where to look
The whole fleet appears modified overnight	A transport switched, or a normalisation rule changed. Check `transport_fell_back` in the report.
`CERTIFICATE_VERIFY_FAILED` on an API	Certificate not validated by the system authorities — the factory case on FortiOS, PAN-OS and BIG-IP. The message gives the fingerprint presented: pin it on the device page. See Pinned certificate.
A first SSH backup fails with “host key not approved”	Expected behaviour under the default posture. Approve the fingerprint from the device page, see Pending fingerprints.
A device is “stale” when it was not scheduled today	Should not happen — the state is computed on the cadence. Check that the cron expression is the one you expect, see Tolerance.
A job stays `pending`	No agent online for that device. It will turn `abandoned` after the delay.
An agent stops backing up with no error	Working copy full. The “Working copy” column on the Agents screen.
The rsync transfer does not decrease	The working copy is not persisting: check the volume mounted on `/var/lib/backupmssp/staging`.
Space does not free up after lowering the retention	Object lock. See Object lock.
A customer sees other customers	The application role is a superuser. Check `/ready`.
SAML unavailable	`libxmlsec1` absent. Use the platform image, or switch to OIDC.
An agent enrolment fails TLS verification	The name the agent uses is absent from `BKP_GRPC_SERVER_NAMES`.
A screen is slow without the volume explaining it	See The cost of isolation policies, below.

Backing up a Linux server: key or password¶

The transfer goes through rsync over SSH, and OpenSSH never offers a password prompt in non-interactive mode. The symptom misleads:

Permission denied (publickey,password)

You suspect a wrong password when in fact it was never offered at all.

The agent handles both modes:

Credential attached	Behaviour
SSH key	`-i` with `BatchMode=yes` and `IdentitiesOnly=yes` — no prompt possible
Password alone	`sshpass`, keys disabled, a single prompt allowed
Neither	explicit refusal before any transfer

The key is preferable. The password travels through an environment variable rather than the command line — ps is readable by all the agent's processes — but a dedicated read-only key stays cleaner, and can be revoked without changing an account's password.

The private key is never left on the agent's disk: OpenSSH requires a file, so it is created in memory (/dev/shm, permissions 0600) for the length of the transfer, then erased.

The account must be able to read the declared paths. For /etc with an unprivileged account, switch on the device's sudo option: the agent then uses --rsync-path "sudo rsync", which does not require a remote interactive shell.

Full configuration export¶

Settings → Configuration export → Whole configuration, or:

curl -s "$BASE/api/export/platform" -H "Authorization: Bearer $TOKEN" > platform.json

The document describes the installation: customers, agents, credentials, devices with their schedules and their options, retention rules at all three levels, global settings.

Nothing that grants access is part of it — no credential values, no repository keys, no agent certificates. That is what allows it to be kept outside the platform, which is exactly what a recovery plan demands: an export you cannot store elsewhere is of no use.

The document carries its own recovery notes (restore_notes):

credential values have to be re-entered;
repository keys are a separate, dual-control procedure;
agents have to be re-enrolled, their certificates not being transferable.

Backing up the platform itself¶

Three elements, and all three are indispensable:

Element	Without it
The PostgreSQL database	no inventory, no references to the backups
`BKP_MASTER_KEY`	the database is unusable: every customer key is encrypted under it
The object store	no backups at all

Moving an installation amounts to copying the database, storage and certificate authority volumes, plus the environment file. Without the authority, every agent has to be re-enrolled.

The cost of isolation policies¶

Isolation rests on PostgreSQL's row-level security, whose predicate reads the scope from a session variable. That is what makes it impossible to forget — and it is also what blinds the planner.

The planner cannot estimate a session variable. It has no statistics on current_setting(). Every cardinality on the isolated tables therefore falls back to rows=1. Measured on 400,000 versions: estimate 1 row, reality 92,000. No rewriting of the predicate corrects that; four formulations were compared, all estimate 1. It is a property of RLS, not a writing defect.

The consequence is brutal: believing it has one row on each side, the engine picks nested loops and re-walks the inner table as many times as there are outer rows. The dashboard thus reached 2.4 s on fifteen devices, discarding seven million rows to produce four.

What not to write¶

Never convert the scope column. tenant_id::text = current_setting(...) forbids using the index on tenant_id and deprives the engine of the column's statistics. Compare uuid to uuid.
Never mark the scope functions as anything other than STABLE PARALLEL SAFE. A function that is merely “parallel restricted” inside a policy makes every query on the table ineligible for parallelism: the scan of the versions table loses its worker processes and takes three times as long.
Do not grant LEAKPROOF lightly: the attribute lets the planner evaluate an expression before the policy, which can leak the content of forbidden rows through an error message. Neither scope function has any need of it — they take no argument and touch no column.
Do not count on the planner to rescue a long query. In an isolated context it has no means of doing so. Write short: every superfluous join is material for a bad plan to blow up on. The question “which devices changed since yesterday” went through version → run → device when the artifact already carries its device: 2,002 ms by the long path against 2 ms by the short one, on the same database and with the policies unchanged.
Index the columns of temporal clauses. The engine will not go and find them by itself. On 400,000 versions, a narrow window — the real case, 24 h over a year of history — goes from 7.7 ms to 0.6 ms with a dedicated index.

Migrating the policies¶

The policies are laid down at every startup by the bootstrap: a simple restart of the platform is enough to switch a database in service, with no manual intervention.

The replacement happens outside a transaction, so there is a brief moment when a table has RLS active and no policy. That moment closes access, it does not open it: PostgreSQL refuses every row of a table whose RLS is active and which has no policy. Verified — 2,000 rows visible before, 0 during, 2,000 after. An interrupted migration therefore leaves a platform that sees nothing, never one that sees everything.

Two deployment traps met along the way¶

The init and platform services share the same image. They are built from the same Dockerfile; without a common image:, a docker compose build platform leaves init on a stale image, and the schema is then created from a model older than the one the platform runs. The symptom is disconcerting: the platform issues valid queries that the database refuses.

Byte quantities are 64-bit. A 32-bit integer overflows at 2.1 GB, well before the footprint of a single server in the fleet. Any new column expressing a size or a number of files must use BigInteger.

Tests¶

pytest                                  # everything available
pytest -m "not db and not storage"      # with no external dependency
KOPIA_BIN=/usr/local/bin/kopia pytest -m storage   # real deduplication

Tests marked db recreate the schema: they are redirected automatically to a database carrying the _test suffix, created on the fly. Running the suite does not erase the development database.

Tests marked storage create a real repository in the object store and check deduplication between successive snapshots. They are skipped if the deduplication binary or the storage are unreachable.