Token Refresh

Overview

The Token Refresh Worker is the single, authoritative component responsible for refreshing OAuth access tokens for all platform connections. It runs as a background task inside the API server, continuously monitoring token expiry across channel_connections, login_connections, and bot_connections. Discord login tokens are refreshed; its global bot token is static, while server mappings store no OAuth token.

All workers and services that need a fresh token read it from the database using the utility functions in apps/api/src/oauth.rs. They never hold tokens in-memory across operations or call refresh logic inline.

Token Lifecycle

OAuth Provider (Twitch, YouTube, etc.)
    |
    | (user connects channel / logs in)
    v
DB: channel_connections / login_connections
    - access_token  (AES-256-GCM encrypted)
    - refresh_token (AES-256-GCM encrypted)
    - expires_at
    |
    | (background, always running)
    v
Token Refresh Worker (apps/api/src/workers/token_refresh.rs)
    - Wakes 10 min before next expiry
    - Calls platform token endpoint
    - Re-encrypts new tokens
    - Writes back to DB
    |
    | (on each API call)
    v
get_fresh_connection_token() / get_fresh_oauth_token()
    - Reads encrypted tokens from DB
    - Decrypts with config.auth.token_encryption_key
    - Returns FreshToken to caller

Which Tables Are Refreshed

Table	Purpose	Function to read
channel_connections	Platform API tokens (Twitch, YouTube, Kick, Spotify)	get_fresh_connection_token()
login_connections	OAuth login session tokens (used to authenticate users on sign-in)	get_fresh_oauth_token()
bot_connections	Bot identity tokens (refreshed for OAuth bot platforms; the Discord global bot token is static)	get_fresh_bot_token()

In all three tables, rows flagged reconnect_required = true are excluded - both from the refresh pass and from the "next expiry" calculation. A connection in that state needs the user to re-authorize; retrying its refresh token would only burn requests.

Cluster Safety

Every API replica starts its own Token Refresh Worker, but only one may refresh a given cycle. Before each pass the worker takes a non-blocking Postgres advisory lock (pg_try_advisory_lock, key 0x746F6B65 = "toke"); a replica that does not get the lock skips the cycle immediately without holding a connection.

This matters because several providers (Google, Kick, Discord, …) rotate the refresh token on use. Two replicas refreshing the same row would replay an already-consumed refresh token, and the provider may invalidate the whole token family - permanently breaking that connection's auth. The lock is released after the pass, and a crashed replica releases it automatically on disconnect.

Single-Flight Per Connection

The cluster lock above stops two replicas running the refresh pass at once, but it is not the only refresher of a given row. A login connection is also refreshed on demand by services::login_token::get_valid_account_login_token when the send path needs a token whose expiry is within 60 seconds - and that lazy path and the worker's 10-minute window overlap in the last minute before expiry. For a provider that rotates the refresh token on use (Kick is OAuth 2.1 and always returns a new refresh token), the loser of that race replays an already-consumed token, the provider rejects it, and the connection is flagged reconnect_required - the exact opposite of an automatic refresh.

There are therefore two distinct advisory locks, and they must not be confused:

Lock	Key	Blocking?	What it serialises
Global / session	pg_try_advisory_lock(0x746F6B65) - a constant	non-blocking	one replica per refresh cycle
Per connection	pg_advisory_xact_lock(hashtextextended(connection_id, 0)) - the row UUID	blocking	the two refreshers of a single row (the worker pass and the login-lazy path)

Every refresh path takes the per-connection lock before hitting the provider, via oauth::lock_connection_for_refresh(kind, connection_id), and then re-reads the expiry under the lock: if another flight already refreshed the row while it waited (a successful refresh pushes expires_at far into the future), it skips instead of calling the provider again. The worker's on-demand login twin (services::login_token) takes the same key, so the two serialise against each other and exactly one flight refreshes a given row. The refreshed tokens are persisted inside the lock transaction, whose commit releases the lock atomically with the write - so no token write happens outside the lock, and the COALESCE(refresh_token) update can never let a race loser clobber the winner's freshly rotated token. Connection-row UUIDs are unique across the three tables, so the single lock space needs no per-table namespacing.

Failure Backoff

A refresh that fails bumps the row's updated_at. The refresh query then skips any row matching expires_at < now() AND updated_at > now() - interval '1 hour' - that is, an already-expired token whose refresh was attempted within the last hour. This keeps a permanently broken connection (revoked grant, deleted app credential) from being retried on every 5-minute cycle. Tokens that are not yet expired are retried normally.

Supported Platforms

Platform	Token Endpoint	Auth Style
Twitch	https://id.twitch.tv/oauth2/token	Body params
YouTube / Google	https://oauth2.googleapis.com/token	Body params
Kick	https://id.kick.com/oauth/token	Body params
Spotify	https://accounts.spotify.com/api/token	Authorization: Basic header
Discord	https://discord.com/api/v10/oauth2/token	Body params

When Tokens Are Refreshed

The worker uses a smart sleep strategy to minimize wasted wakeups:

1. On startup: immediately runs a full pass to catch any already-expired tokens.
2. After each pass: queries the minimum expires_at across both tables and sleeps until 10 minutes before that expiry.
3. The sleep is capped at 5 minutes so the worker never goes dormant too long.
4. If a token is already within the 10-minute window (or already expired), the worker wakes in 5 seconds.

A token is considered in need of refresh when:

expires_at < now() + interval '10 minutes'

Only tokens with a refresh_token present are eligible. Tokens without a refresh token (e.g., non-expiring API keys) are skipped silently.

Encryption

All tokens in the database are encrypted at rest using AES-256-GCM. The encryption key comes from:

# apps/api/config/local.toml
[auth]
token_encryption_key = "<base64-encoded 32-byte key>"

Generate a key with:

openssl rand -base64 32

The encryption key can also be set via the LUMIO__AUTH__TOKEN_ENCRYPTION_KEY environment variable (the LUMIO__SECTION__KEY convention used by lo_config).

The worker decrypts tokens before sending them to the platform endpoint and re-encrypts the new tokens before writing them back to the database.

Rotating the key invalidates every stored secret. The ciphertext carries no key-version tag, so a value only decrypts under the exact key that wrote it. Changing token_encryption_key - or deploying a build that derives a different key from the same config (e.g. an env override that had been silently dropped) - makes all previously stored secrets unreadable at once: every access_token/refresh_token on channel_connections and bot_connections, and every client_id/client_secret in app_credentials. When the worker cannot decrypt a token - or the account's app_credentials client_id/secret needed to talk to the provider - it never sends the raw ciphertext upstream; it treats the failure as terminal, flags the connection reconnect_required, and fires the best-effort audit + email. Remediation: the owner re-enters app credentials in the platform tab’s App section and reconnects the affected channels; there is no automatic re-encryption migration.

OAuth App Credentials

For channel_connections and OAuth bot_connections, refresh follows the row's pinned credential_source.

credential_source	OAuth app pair	Token row
account	client_id + client_secret from app_credentials matched by (account_id, platform)	channel_connections or bot_connections
system	global config pair resolved by CredentialResolver::system_pair(platform)	channel_connections or bot_connections

The access token and refresh token are always stored encrypted on the connection row. account app credentials are also encrypted in the database; system app credentials come from config. The pin exists because a refresh token minted by one OAuth app cannot be refreshed by another.

If a row is pinned to system but the matching global key pair is absent at refresh time, the refresh fails clearly and the row is handled like any other refresh failure. If a row is pinned to account but the account's app_credentials row is absent, the account must save credentials and reconnect.

For login_connections, the client_id and client_secret are the config-level credentials set in [auth] (e.g., twitch_client_id, google_client_id, discord_client_id / discord_client_secret), resolved by services::login_token::get_provider_credentials. These are NOT stored in app_credentials. Discord rotates refresh tokens; the worker and the on-demand login path use the same single-flight lock. A provider with no configured credentials is skipped.

Utility Functions

Both functions live in apps/api/src/oauth.rs and are the correct way for any worker or feature to obtain a decrypted, ready-to-use token.

get_fresh_connection_token

Reads and decrypts a channel connection's access token and refresh token, then resolves the OAuth app pair from the pinned credential_source.

use crate::oauth::get_fresh_connection_token;

let token = get_fresh_connection_token(&db, connection_id, &resolver).await?;
// token.access_token  -- decrypted, ready for API use
// token.client_id     -- matching OAuth app client ID
// token.client_secret -- matching OAuth app client secret
// token.expires_at    -- optional expiry timestamp

Use this for: Twitch chat workers, YouTube polling, Spotify and Kick - workers that call a platform API on behalf of a channel. Discord has no channel connection.

get_fresh_oauth_token

Reads and decrypts a login connection's access token from login_connections. client_id and client_secret are returned as empty strings - the caller must supply the login config credentials separately.

use crate::oauth::get_fresh_oauth_token;

let token = get_fresh_oauth_token(&db, login_connection_id, &encryption_key).await?;
// token.access_token -- decrypted login token
// token.client_id    -- always "" (use config credentials)

Use this for: features that need the user's login token (e.g., reading scoped login-level platform data).

get_fresh_bot_token

Reads and decrypts a bot identity token from bot_connections, resolving the OAuth app pair from the row's pinned credential_source. Rows with reconnect_required = true are not returned. For Discord the returned client_id / client_secret are empty strings, because Discord bot tokens are static and are never refreshed.

use crate::oauth::get_fresh_bot_token;

let token = get_fresh_bot_token(&db, bot_connection_id, &resolver).await?;
// token.access_token   -- decrypted bot identity token
// token.refresh_token  -- decrypted, None when the platform issues none

Use this for: anything acting as a bot identity rather than as the channel.

How to Use in a New Worker

When writing a new worker or feature that calls a platform API:

Do this:

// In your worker's run loop or per-request handler:
let token = crate::oauth::get_fresh_connection_token(
    &db,
    connection_id,
    &resolver,
).await?;

// Use token.access_token directly
let response = platform_api_client
    .some_request(&token.access_token)
    .await?;

Never do this:

// DO NOT hold OAuthTokens in-memory across loop iterations
// DO NOT call tokens.try_refresh() inline in a worker
// DO NOT implement your own refresh logic
let mut tokens = OAuthTokens { ... };
tokens.try_refresh(&http).await?; // WRONG — Token Refresh Worker handles this

This is enforced in code where it used to be violated: lo-spotify-api's client-side refresh helper is marked #[deprecated] with the note "Use crate::oauth::get_fresh_connection_token in apps/api instead. Inline refresh races with the Token Refresh Worker." OAuthTokens::try_refresh and refresh_oauth_token have no callers outside workers/token_refresh.rs.

The Token Refresh Worker guarantees that channel_connections.access_token is always fresh by the time your worker reads it. You can safely read the token from the DB on each API call without worrying about expiry.

Architecture

apps/api/src/workers/token_refresh.rs  -- Background worker (runs_token_refresh_worker)
apps/api/src/oauth.rs                  -- Utility functions + OAuthTokens struct

Worker Internals

Function	Purpose
run_token_refresh_worker()	Entry point, starts the loop, handles cancellation
refresh_expiring_tokens_if_leader()	Takes the global/session advisory lock; runs the pass only on the winning replica
next_expiry_in_secs()	Queries MIN(expires_at) across the three tables to determine smart sleep duration
refresh_expiring_tokens()	Finds all expiring tokens and refreshes each one under the per-connection single-flight lock

oauth.rs Internals

Function	Purpose
get_fresh_connection_token()	Read + decrypt channel connection token, resolve credentials by pinned credential_source
get_fresh_oauth_token()	Read + decrypt login connection token
get_fresh_bot_token()	Read + decrypt bot connection token, resolve credentials by pinned credential_source
refresh_oauth_token()	Execute the HTTP token refresh request
lock_connection_for_refresh()	Take the per-connection single-flight lock and re-read the expiry under it
refresh_still_needed()	Whether a re-read expiry still needs refreshing within the window (the under-lock re-check)
update_connection_tokens()	Write refreshed tokens back to channel_connections (inside the lock tx)
update_oauth_tokens()	Write refreshed tokens back to login_connections (inside the lock tx)
update_bot_connection_tokens()	Write refreshed tokens back to bot_connections (inside the lock tx)
token_needs_refresh()	Returns true if token expires within 5-minute margin
OAuthTokens::try_refresh()	Used internally by the worker for channel connections only; persists via the caller's lock-tx executor

Key Files

File	Purpose
apps/api/src/workers/token_refresh.rs	Token Refresh Worker implementation
apps/api/src/oauth.rs	Utility functions: get_fresh_connection_token, get_fresh_oauth_token, refresh_oauth_token
apps/api/src/credentials.rs	CredentialResolver, source-aware global/account credential lookup
apps/api/src/crypto.rs	AES-256-GCM encrypt/decrypt used by all token operations
apps/api/config/default.toml	[auth] token_encryption_key config key