Backups (production and staging)

Every Lumio host with customer data backs up its data layer once a day. The backup is encrypted, stored off the host in the EU, and deleted after 14 days. A backup counts only once a restore of it has been tested. The restore log records each test with its date.

The scripts and systemd units are in dev-stack/staging/backup/. Like the compose files next to them, they are reference files: the operator copies them to the host (see Setting up a host). The folder's README.md is the operator's quick reference on the host: commands for daily operation, the configuration table and troubleshooting. Agents have no access to the hosts. The operator applies every step and records the result.

What is backed up

Store	Content	Method
PostgreSQL (staging-postgres)	main database lumio + roles	pg_dump -Fc + pg_dumpall --globals-only
TimescaleDB (staging-tsdb)	lumio_tsdb (hypertable chunks, compressed chunks, continuous-aggregate materializations) + roles	pg_dump -Fc + pg_dumpall --globals-only
Extensions PostgreSQL (staging-extensions-postgres)	lumio_extensions (ext_* schemas) + the roles from initdb/extensions/10-roles.sh	pg_dump -Fc + pg_dumpall --globals-only
Redis (staging-redis)	the /data directory: AOF (appendonlydir/) and dump.rdb	file copy outside an AOF rewrite

Uploaded files are not on the host: the app stores them in its own R2 bucket ([storage] in the API config). They are not part of this backup.

Redis holds data that has no other copy until it is flushed: the chat buffer, parked events, extension storage and automation variables (see Data layer). The script copies /data only while no AOF rewrite runs. If the AOF manifest changes during the copy, it copies again.

Each dump is streamed straight into restic (--stdin-from-command), so no unencrypted dump file is written to disk. If pg_dump fails, restic creates no snapshot and the job fails.

Tool, schedule and target

Item	Value
Tool	restic ≥ 0.19 (0.19.0 fixed a hang in --stdin-from-command; tested with 0.19.1), one repository per environment
Schedule	lumio-backup.timer: daily 03:15 UTC (Persistent=true catches up a missed run). lumio-backup-check.timer: Sundays 05:00 UTC, restic check --read-data-subset=5%
Target	Any S3-compatible object storage in the EU, one bucket per environment (lumio-backup-prod, lumio-backup-staging), each with its own access key and its own restic password. Staging uses Cloudflare R2 with the EU jurisdiction today. See Choosing or switching the S3 provider
Monitoring	A failed run shows in systemctl --failed and journalctl -u lumio-backup. An optional dead-man's switch URL (BACKUP_PING_URL) gets a ping on success and /fail on failure

Encryption and key

restic encrypts all repository data with AES-256 in counter mode, authenticated with Poly1305-AES. The repository key is derived from the password with scrypt (restic design document, read 2026-10-27). The target only ever holds ciphertext.

• The password is in /etc/lumio-backup/restic.pass (owner root, mode 0600) on the database host. The host disk is fully encrypted.
• A second copy is in the operator's password manager. Without the password, no backup can be restored.
• The password is never stored at the backup target: not in the bucket and not in a file next to the repository.
• The S3 access key is in /etc/lumio-backup/env (owner root, mode 0600). It can reach only that one backup bucket. So a compromised staging host can neither read nor delete the production backups, and neither host can touch the app's upload bucket.

Retention: 14 days

lumio-backup.sh deletes every snapshot older than BACKUP_RETENTION_DAYS (default 14) and prunes the repository data after each backup. Age is measured against the clock, not against the newest snapshot. So if backups stop, old snapshots are still deleted on time. With one backup a night, there are about 14 restore points, and the oldest one goes back 14 days. Damage that is noticed later than that cannot be undone from a backup.

The value must stay below 30 days, because two deletion promises reach into backups:

• legal/privacy.mdx: deleted YouTube revenue data disappears from backups at the latest 30 days after the deletion. The same holds for every other deletion (a deleted profile, an erased chatter, a Shopify customers/redact).
• Shopify API Terms §6.2.3: merchant data is deleted within 30 days.

Retention depends on the bucket as well. Versioning and object lock (R2: bucket lock) must stay off, because they would keep deleted data beyond the 14 days or stop restic from deleting it. A lifecycle rule must not be added, because it would delete restic's data files underneath the repository.

Setting up a host

At the S3 provider, once per environment (details per provider in Choosing or switching the S3 provider):

1. Create the private bucket lumio-backup-<env> (prod or staging) in an EU location, with versioning, object lock and lifecycle rules off.
2. Create an access key that can reach only this bucket. Note the access key ID and the secret access key; most providers show the secret only once.

As root on the database host (production or staging):

1. Install restic ≥ 0.19 and jq. Distribution packages are often older (Ubuntu 24.04 ships 0.16.4, which lacks --stdin-from-command). In that case install the release binary from GitHub, compare its SHA-256 with the release's SHA256SUMS, put it in /usr/local/bin, and remove the distribution package. Check with restic version.
2. Copy the scripts and units:

   install -d -m 0700 /opt/lumio-backup /etc/lumio-backup
   install -m 0700 dev-stack/staging/backup/{lumio-backup.sh,lumio-backup-common.sh,restore-test.sh,restore.sh} /opt/lumio-backup/
   install -m 0644 dev-stack/staging/backup/README.md /opt/lumio-backup/
   install -m 0644 dev-stack/staging/backup/lumio-backup*.{service,timer} /etc/systemd/system/
   install -m 0600 dev-stack/staging/backup/lumio-backup.env.example /etc/lumio-backup/env

3. Fill in /etc/lumio-backup/env: the repository and region of the provider block you chose, the access key, LUMIO_ENV, the container names, REDIS_DATA_DIR and REDIS_PASSWORD.
4. Create the password and the repository:

   (umask 077; openssl rand -base64 32 > /etc/lumio-backup/restic.pass)
   set -a; . /etc/lumio-backup/env; set +a
   restic init

   Store the password in the password manager now.
5. Run a first backup by hand, then enable the timers:

   /opt/lumio-backup/lumio-backup.sh backup
   systemctl daemon-reload
   systemctl enable --now lumio-backup.timer lumio-backup-check.timer

6. Run the restore test and add the result to the restore log.

Choosing or switching the S3 provider

restic's S3 backend works with any S3-compatible object storage, and the scripts only pass RESTIC_REPOSITORY, AWS_DEFAULT_REGION and the access key through. The provider is therefore chosen in /etc/lumio-backup/env alone. lumio-backup.env.example has a ready block for each of these (provider docs read 2026-10-27):

Provider	RESTIC_REPOSITORY	AWS_DEFAULT_REGION	EU location	Key limited to one bucket
Cloudflare R2	s3:https://<ACCOUNT_ID>.eu.r2.cloudflarestorage.com/<bucket>	auto	bucket created with jurisdiction European Union (cannot be changed later)	API token with Object Read & Write on this bucket
Hetzner Object Storage	s3:https://fsn1.your-objectstorage.com/<bucket>	fsn1 (or nbg1, hel1)	all locations are in the EU	a Hetzner key opens every bucket of its project, so use one project per environment
DigitalOcean Spaces	s3:https://fra1.digitaloceanspaces.com/<bucket>	us-east-1 (signing only; the endpoint decides the location)	fra1 or ams3	Spaces key with Limited access to this bucket
AWS S3	s3:https://s3.eu-central-1.amazonaws.com/<bucket>	eu-central-1 (or another EU region)	EU regions	IAM user whose policy allows only this bucket
Self-hosted MinIO	s3:https://<minio-host>:9000/<bucket>	us-east-1 (MinIO's default)	a server in the EU that is not the database host	MinIO user whose policy allows only this bucket

For every provider: private bucket, versioning, object lock and lifecycle rules off. Prod and staging never share a bucket or a key.

A self-hosted MinIO only counts as off-host if it runs on a different machine than the databases (ideally at another provider or data centre), with an encrypted disk, reached over HTTPS. If its certificate comes from its own certificate authority, put the CA file into RESTIC_CACERT. The scripts load every variable of the env file into restic's environment. Tested against MinIO with a self-signed certificate (2026-10-27): backup, restore-test.sh (PASS) and restore.sh --dry-run. Without RESTIC_CACERT restic refuses the connection (x509: certificate signed by unknown authority).

Switching provider (for example from R2 to Hetzner):

1. Create the bucket and key at the new provider.
2. Keep the old configuration and its password: copy /etc/lumio-backup/env to env.<old-provider>, rename restic.pass to restic.pass.<old-provider>, and point RESTIC_PASSWORD_FILE in env.<old-provider> at the renamed file. Without that, the new password would overwrite the old one and the old backups could no longer be read.
3. Put the new provider's values into /etc/lumio-backup/env (RESTIC_REPOSITORY, AWS_DEFAULT_REGION, the key). Create a new restic.pass, store it in the password manager, and run restic init. The commands are in the folder's README.md.
4. Run lumio-backup.sh backup and restore-test.sh. The timers back up to the new provider from now on.
5. The old backups stay restorable for the transition: LUMIO_BACKUP_ENV=/etc/lumio-backup/env.<old-provider> /opt/lumio-backup/restore.sh --dry-run. Nothing prunes them any more, so delete the old bucket 14 days after the switch at the latest. That keeps the 30-day deletion promise, and by then the new provider holds a full set of restore points.

Copying the old snapshots over is not worth it: with 14 days of retention, the new provider holds a full set of restore points two weeks after the switch.

Several targets at once. With BACKUP_TARGETS="r2 hetzner" in the main env file and one file per target in /etc/lumio-backup/targets/<name>.env (its own bucket, key and restic password), every backup goes to all targets one after the other. A target that does not answer within BACKUP_TARGET_TIMEOUT seconds (default 120) or fails is reported, and the others still run; the run then ends with exit code 1 (backup failed for: <names>). Each target keeps its own 14 days. --target <name> limits lumio-backup.sh, restore-test.sh and restore.sh to one target; the restore scripts default to the first one. Run the restore test for every target. Setup steps are in the folder's README.md.

Restore test

Run it after setting up a host, after every change to the backup setup, and at least once a quarter:

/opt/lumio-backup/lumio-backup.sh backup
/opt/lumio-backup/restore-test.sh

restore-test.sh takes the newest snapshot of each store and restores it into throwaway containers. These containers run the same image as the live containers and have no network (--network none). TimescaleDB is restored between timescaledb_pre_restore() and timescaledb_post_restore(), into a database created from template0 with the same extension version as the dump. The installed extension can be older than the image's default (the -ha image ships older versions too), and a dump restores only into its own version. The backup records that version as the tag timescaledb-<version> on the TimescaleDB snapshot. Redis loads the restored AOF. The script then counts rows created before the snapshot time in the live and the restored databases:

• platform_event_logs and platform_chat_messages (TimescaleDB),
• users and accounts (PostgreSQL),
• the number of ext_* schemas (extensions PostgreSQL),
• the Redis key count after the AOF is loaded.

It prints a result block for the restore log. Afterwards it deletes the scratch containers and files, because they hold production data. The scratch directory is /var/tmp on the encrypted disk. Never copy a restored database or a dump to another machine.

Result	Meaning
PASS	all counts match, pg_restore reported no errors, Redis loaded its keys
CHECK	the restore worked, but a count differs or pg_restore reported errors. A smaller live count is expected when retention or a deletion removed rows after the snapshot. A smaller restored count, or any pg_restore error, is a failed test
FAIL	Redis did not load, or a count query failed on one side (ERR in the table, for example a missing table). Exit code 1

Restoring production

restore.sh brings the live data layer back to a snapshot. It overwrites the live databases and the Redis data, so it differs from restore-test.sh, which only ever restores into throwaway containers. A restore brings the data back to the snapshot time, which is at most about 24 hours old. Everything after that point is lost, including deletions.

1. Stop the API, the bots and the workers.
2. Look at the plan first. It lists the snapshot of each store and its time, and changes nothing:

   /opt/lumio-backup/restore.sh --dry-run

   --before 2026-10-27T03:00:00Z picks the newest snapshots older than that time. --only tsdb,redis restores only some stores (postgres, tsdb, extensions, redis).
3. Restore. The script asks for the host tag (lumio-<env>) before it changes anything:

   /opt/lumio-backup/restore.sh

   What it does:
   • It refuses to run while other sessions are connected to a target database. --terminate ends them instead.
   • It takes a safety backup of the current state first, without pruning, so a wrong restore can itself be undone. If a database is already gone, skip it with --no-safety-backup.
   • For each database: it creates only the missing roles (existing roles keep their current passwords), recreates the database from template0 with its previous owner, and runs pg_restore. TimescaleDB gets the extension version from the snapshot's timescaledb-… tag and is restored between timescaledb_pre_restore() and timescaledb_post_restore(). Every session runs with event_triggers=off, so the access-log login trigger does not interfere.
   • Redis: it stops the container, moves the current data directory aside (<dir>.before-restore-<time>, never deleted), puts the restored one in place, starts the container and waits until the AOF is loaded.
   • At the end it prints the snapshot IDs, the pg_restore errors (exit code 1 if there are any) and the row counts of the stores it restored.
4. Repeat the deletions made after the snapshot time. The audit log in the restored TimescaleDB does not contain them. Use what still exists outside the database: support requests, the Shopify Partner Dashboard webhook log (customers/redact, shop/redact) and Google revocations, which the daily check picks up again on its own.
5. Start the services. The TimescaleDB retention policies and the daily compliance and retention jobs run again on their normal schedule. Once everything works, delete the Redis directory that was moved aside.

Restore log

Date (UTC)	Host	Snapshots	Result	Tested by
—	production	—	pending: the production host is not set up yet	—
2026-10-27 22:59	staging	postgres 7d409dd8, tsdb e1b2f479 (TimescaleDB 2.31.0), extensions 621e9cff, redis 8b0670a6	PASS: users 18 = 18, accounts 11 = 11, platform_event_logs 16 031 = 16 031, platform_chat_messages 127 857 = 127 857, ext_* schemas 0 = 0; Redis 292 keys; 0 pg_restore errors	operator (founder)