Deployment

Lumio uses release-driven deployment for its container images: they are built and pushed only when a GitHub Release is published, or by manual dispatch -- there are no automatic image builds on branch pushes. The workflow lives in .github/workflows/release-images.yml. Other surfaces (docs, extensions, npm packages) have their own pipelines, covered under Other Deployment Pipelines.

Release Strategy

Every deploy starts with a GitHub Release. The release tag encodes which app to deploy and the version:

{app}@{version}

The workflow parses the tag, determines the target app, runs the verification gate, builds the image, pushes it to the container registry, and signs it.

Setting the Release Version

The product version is bumped by hand in the feature PR that changes app code, in the YYYY.M.PATCH format (e.g. 2026.8.14). The Rust workspace, all TS apps except apps/mobile, all shared/*, and the private root package.json share one version and move in lockstep with the API - set every manifest to the same value directly (npm has no version inheritance).

After editing the manifests, run the mandatory, in-order post-bump regeneration chain and commit the generated artifacts alongside the manifest edits:

1. cargo check --workspace - refreshes Cargo.lock.
2. just schemas - regenerates apps/api/openapi.json + apps/api/schema.graphql (both embed the version).
3. just bazel-vendor - regenerates vendor/ from the new Cargo.lock.
4. Commit Cargo.lock, vendor/, apps/api/openapi.json, and apps/api/schema.graphql.
5. just verify-release, then confirm git status is clean.

just verify-all does not detect schema or vendor drift - run just verify-release before opening the PR. Independent packages (packages/*, extensions/system/*) are not product-versioned; bump each one's own semver in the same PR that changes it. The mobile app (apps/mobile) is not product-versioned either: package.json version and app.json expo.version carry its own semver (starting at 1.0.0) and always agree, while the iOS buildNumber / Android versionCode store build counters are separate, monotonically increasing integers managed by EAS - never derived from the product version or the semver string.

How to Deploy

1. Go to GitHub Releases for the repository.
2. Click Draft a new release.
3. Create a tag following the {app}@{version} format.
4. Mark the release as a pre-release if the image should receive beta tags instead of latest.
5. Publish the release.

Tag examples:

[email protected]
[email protected]
[email protected]
[email protected]

Image Tags

The workflow generates multiple tags per image based on whether the release is marked as a pre-release.

Production (stable release)

Tag	Example	Description
{version}	2026.5.3	Exact version
{major.minor}	2026.5	Floating minor tag
{year}	2026	Floating year tag: the newest stable release of that year
latest	latest	Most recent stable release
{short_sha}	a1b2c3d	7-character commit SHA
{full_sha}	a1b2c3d...	Full 40-character commit SHA

Pre-release (beta)

Tag	Example	Description
{version}	2026.5.3-beta.1	Exact pre-release version
beta	beta	Floating beta tag
{short_sha}	a1b2c3d	7-character commit SHA
{full_sha}	a1b2c3d...	Full 40-character commit SHA

A version string containing beta, alpha, or rc is also treated as a pre-release regardless of the GitHub Release checkbox.

Verification Gate

Before anything is published, a Verify job must pass:

bazel test  //crates/... --build_tag_filters=-manual
bazel build //apps/...   --build_tag_filters=-manual

The build step is a pre-push gate: the release fails before publishing if any app binary does not compile. -manual skips the container _push targets, which are build-tagged manual.

Deployable Apps

Nine Rust binaries are built with Bazel (bazel run --config=release //apps/{app}:{app}_push) via rules_img:

App	Tag Prefix
api	api@
automation-worker	automation-worker@
bot-module-worker	bot-module-worker@
twitch-bot	twitch-bot@
youtube-bot	youtube-bot@
kick-bot	kick-bot@
discord-bot	discord-bot@
innertube-proxy	innertube-proxy@
crawler-twitch	crawler-twitch@

The Discord bot service authenticates to the API with DISCORD_BOT__API__SYSTEM_KEY. Create that key under Admin → System Keys in the same environment, grant bots:internal, and provide the one-time value to the bot service. The API does not provision System keys from config. System access required indicates the supplied credential is not an accepted System key; Missing permission: bots:internal indicates a valid key without the required grant.

Build mode

Release images are compiled optimised. --config=release is defined in the repo .bazelrc as build:release --compilation_mode=opt, which rules_rust turns into rustc opt-level=3 with debuginfo stripped. Bazel's default mode, fastbuild, compiles at opt-level=0. Unoptimised code is slower, and its async state machines are several times larger, which leaves less headroom on actix's worker-thread stacks. Never push an image without --config=release. Local cargo run uses [profile.dev] opt-level = 1. The build mode is a hardening measure. It does not replace fixing oversized futures in code (boxing them with Box::pin).

PR CI (Rust Tests, Rust Build, Clippy, Schema Check) and the release Verify gate stay on fastbuild. The two modes write to separate output directories (bazel-out/k8-opt vs bazel-out/k8-fastbuild) and have different action keys, so they never evict each other's BuildBuddy entries. Because no CI job builds in opt, the Build & Push job uploads its own results to the remote cache (--remote_upload_local_results). A release then recompiles only the crates that changed since the previous release, not the whole dependency tree. Schema generation runs under the same config, so it reuses the API's opt outputs.

To check which mode a target builds in, run bazel aquery --config=release 'mnemonic("Rustc", //apps/api:api)' and look for --codegen=opt-level=3.

API image environment defaults

//apps/api:api_image sets these environment variables. A host env file (e.g. .env.api in dev-stack/staging/) overrides any of them.

Variable	Image default	Why
TZ	Europe/Berlin	Log and cron timestamps
RUST_LOG	info	Default log level
LUMIO__SERVER__HOST / LUMIO__SERVER__PORT	0.0.0.0 / 3000	Listen on the container network
RUST_MIN_STACK	8388608 (8 MiB)	Stack size for every thread the Rust standard library spawns, including actix worker threads. The default is 2 MiB, and actix-server has no stack-size option of its own.

RUST_MIN_STACK is a safety margin for deep async request paths. It does not fix oversized futures. Those are fixed in code by boxing them (Box::pin), and the code must stay within the 2 MiB default because tests and local runs use it.

Staying within the 2 MiB default

Two things decide how much stack a GraphQL request needs in an unoptimised build:

• The depth of the root merge. Query and Mutation in apps/api/src/graphql/schema.rs are MergedObjects, which resolve a field by polling their members as a chain of nested futures. At opt-level=0 each link costs 25-35 KiB of stack. A flat list of 65 query types put a ChatQuery field about 1.7 MiB deep before its resolver ran. That is what overflowed staging on 2026.9.235. The root types are therefore a two-level tree of QueryPartN / MutationPartN groups with at most 8 members each. Add a new query or mutation type to the last part, and start a new part once that one holds 8. The SDL does not change, because the parts flatten into one Query / Mutation type.
• The size of each resolver future. Every byte a resolver future holds is carried by each future that wraps it. Box large inner futures (Box::pin), in particular each branch of a tokio::join! over network calls and each arm of a match that dispatches to per-platform async functions.

Tests guard both, and they also run in Bazel fastbuild: graphql::schema::tests::root_merge_tree_stays_shallow executes the deepest root fields on a 1 MiB stack, and the *_stay_within_budget tests in graphql/chat.rs and lo-chat's profile_service.rs cap the future sizes of the emote and user-profile paths. If one trips, box the new future. Do not raise the budget. The cost is small. The stack is reserved virtual memory, and the kernel commits physical pages only when a thread touches them, so resident memory grows only by what a request actually uses. Setting it in the image rather than in code is deliberate: std::env::set_var is unsafe in Rust 2024, racy once threads exist, and the standard library caches the value on the first thread spawn.

Three Next.js apps are built from Dockerfiles (docker/{app}.Dockerfile):

App	Tag Prefix
web	web@
admin	admin@
id	id@

The innertube-proxy image is deployed to a trusted-IP host rather than a shared cloud edge - see InnerTube Proxy for the placement requirement.

Bulk Deploys

Three special tag prefixes deploy multiple apps at once:

Tag Prefix	Deploys
all@{version}	All 12 apps (9 Rust + 3 Next.js)
all-rust@{version}	All 9 Rust apps
all-web@{version}	All 3 Next.js apps

Example: publishing a release tagged [email protected] builds and pushes every app image.

Manual Dispatch

For emergency deploys or rebuilds without creating a release, the workflow supports workflow_dispatch with two inputs:

Input	Description
app	Which app to deploy - a dropdown of all, all-rust, all-web, or any single app from the tables above
tag	The image tag to apply (required)

This is useful when the release flow is unavailable or when retagging an existing build.

Signing and Provenance

Every pushed image is signed and attested before the job finishes. Signing resolves the image's immutable digest first (via its unique full-SHA tag) and operates on that digest, never on a floating tag:

• Signature - keyless cosign sign using the workflow's ambient GitHub OIDC token (the job holds id-token: write). No long-lived signing key exists.
• Provenance - cosign attest --type slsaprovenance1 attaches a SLSA v1.0 provenance predicate recording the source repository, git ref and commit, the triggering event, the builder workflow ref, and the run's invocation URL.

Deploy-Side Signature Verification

A signature only protects anything if the deploy checks it. A core-product image (API, web, admin, id, stats, shopify, workers, bots, innertube-proxy, crawler-twitch) is deployed only if it verifies against this identity, and only by digest:

cosign verify \
  --certificate-identity-regexp '^https://github\.com/zaflun/lumio/\.github/workflows/release-images\.yml@refs/tags/(all|all-rust|all-web|api|automation-worker|bot-module-worker|twitch-bot|youtube-bot|kick-bot|discord-bot|innertube-proxy|crawler-twitch|web|admin|id|stats|shopify)@[0-9]{4}\.[0-9]{1,2}\.[0-9]+(-[0-9A-Za-z.]+)?$' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/zaflun/lumio/<app>:<tag>

The identity is the workflow file at a release tag ref, {app}@{version} (app is a core image or all / all-rust / all-web, version is YYYY.M.PATCH with an optional -<pre-release> suffix). A release published from a GitHub Release, or a manual dispatch started from such a tag, passes. A manual dispatch started from a branch is signed as …@refs/heads/<branch> and is refused, as is a run on any other tag (for example the @zaflun/* npm package tags), an unsigned image, or one signed by any other workflow, repository or OIDC issuer. Use Use workflow from → Tags and pick a release tag for an emergency rebuild that has to be deployable.

Where it is enforced

Deploy path	Gate
Staging server (own compose files on the host, :beta channel)	dev-stack/staging/verified-pull.sh pull <app> beta verifies, pulls ghcr.io/zaflun/lumio/<app>@sha256:…, and tags that digest locally as :beta. Each core-image service carries pull_policy: never (the reference files in dev-stack/staging/ and the host's copies), so docker compose up starts only the verified image, fails if none was pulled, and docker compose pull skips it. Roll back with verified-pull.sh pull <app> <older-version> beta.
Pinning a digest directly (e.g. the innertube-proxy and crawler-twitch trusted-IP hosts)	verified-pull.sh verify <app> <tag> prints the verified …@sha256:… ref for the compose image: line.
k3s cluster (planned)	An admission policy (sigstore policy-controller or Kyverno) enforcing the same identity is an acceptance criterion of the k3s cutover.

The compose files in dev-stack/staging/ are reference copies; the host runs its own. Third-party infrastructure images (PostgreSQL, Redis, brokers, Prometheus, Grafana) are not built by release-images.yml and are not covered by this gate.

The gate checks the documented path. It is not an admission controller: whoever has shell access to the host can still docker pull a tag or start any image by hand. Admission-time enforcement arrives with the k3s cutover. The full operator steps are in dev-stack/staging/README.md → Signature verification.

Schema Artifacts

Releases that include the API (api, all, or all-rust) additionally generate and attach the API contract to the GitHub Release:

bazel run --config=release //apps/api:generate-openapi -- openapi.json
bazel run --config=release //apps/api:generate-schema  -- schema.graphql

Both files are uploaded to the release with --clobber, so each API release carries the exact OpenAPI and GraphQL SDL it shipped.

Container Registry

All images are pushed to GitHub Container Registry (GHCR):

ghcr.io/{org}/lumio/{app}

For example:

ghcr.io/zaflun/lumio/api:2026.5.3
ghcr.io/zaflun/lumio/twitch-bot:latest
ghcr.io/zaflun/lumio/web:beta

Authentication uses the GITHUB_TOKEN provided by GitHub Actions with packages: write permission.

Environments

Two different promotion models are in use, and they must not be conflated.

Container apps (API, workers, bots, web, admin, id)

These are promoted by release channel, not by branch. release-images.yml never runs on a branch push - only on a published GitHub Release or a manual dispatch. The channel is decided by the pre-release flag:

Release kind	Floating tag	Meaning
Pre-release	beta	Pre-release channel
Full release	latest	Full release channel

Consumers pin an exact {version}, follow a stable {major.minor} or {year} line, or follow beta / latest. Which environment a given image lands in is decided by the deployment target that pulls it, and is convention-based rather than enforced by this workflow.

Docs family (docs, developer-docs, extension-apps)

Only these deploy on branch and tag pushes, and their branch-to-environment mapping is enforced in CI:

Branch / Tag	Environment
next	Staging
main	Production preview
Tag 20* (e.g. 2026.5.3)	Production

See the Installation guide for the full domain table per environment.

Other Deployment Pipelines

Container images are only one of several release paths. The rest are driven by their own workflows:

Surface	Workflow	Trigger
Docs site	deploy-docs.yml	Push to next / main, tag 20*
Developer docs	deploy-developer-docs.yml	Push to next / main, tag 20*
Extension supervisor + runtime	deploy-extension-apps.yml	Push to next / main or tag 20*, restricted to changes under apps/extension-supervisor/ or apps/extension-runtime/; also manual
System extensions	system-extensions.yml	Tag ext-{name}@{version} (e.g. [email protected]); also manual, per extension or all
Public npm packages	publish-npm.yml	Tag @zaflun/{package}@{version}; also manual, per package or all

The npm workflow publishes one of the five public @zaflun/* packages, or all of them in dependency order, through npm Trusted Publishing (no npm token), and supports a dry_run input that builds and tests without publishing. Those packages carry their own independent semver rather than the product version - see npm Publishing.

system-extensions.yml runs on the self-hosted EX44 runner (rollback via the repository variable RELEASE_RUNNER); publish-npm.yml stays on ubuntu-latest because npm Trusted Publishing only accepts GitHub-hosted runners - see CI Runner.

Security Audit

A nightly security audit runs via .github/workflows/security-audit.yml (cron: 0 3 * * * UTC). It scans both dependency ecosystems on both main (what is live) and next (what releases are cut from), as a ref: [main, next] matrix, because a scheduled trigger alone would only ever check out the default branch:

Check	Tool	Scope	Accepted-risk allowlist
Rust dependencies	cargo audit	All vulnerability advisories (.vulnerabilities.count)	.cargo/audit.toml
npm dependencies	pnpm audit	High and critical severity	pnpm.auditConfig.ignoreGhsas in the root package.json

Failure semantics: each matrix leg fails when it finds vulnerabilities or when the audit itself breaks (tool crash, unparseable JSON report, an earlier step failing). A red nightly run therefore always means "look at this"; a green one means every leg produced a clean, parseable report.

Alert flow:

• Each failing leg posts its own Discord alert via DISCORD_SECURITY_WEBHOOK_URL, labelled with the branch, listing the advisories, or flagged as an audit error with the tail of the tool's stderr.
• A daily summary always posts: either "All clear" (green), or "Action required" (yellow) with the Rust and npm job results.

The workflow also supports manual workflow_dispatch for on-demand scans.