Chat and event durability

Platforms deliver most chat messages and events exactly once. EventSub over WebSocket, the Kick and StreamElements sockets and most webhooks do not redeliver, so whatever the API fails to store is gone. This page describes how the API keeps that data when TimescaleDB, Redis or the process itself has a problem, and what is still not covered.

Guarantees at a glance

Situation	Chat messages	Platform events (subs, gifts, bits, tips, raids …)
Normal operation	Redis buffer → batch insert every 5 s	Inserted directly, then broadcast
TimescaleDB down, slow or out of connections	Stay in the Redis buffer; after 5 failed flushes per account they move to the dead-letter list	Parked in the Redis dead-letter list; the live alert is still broadcast
TimescaleDB back	Flush resumes; dead letters replay within ~60 s	Parked events replay within ~30 s with their original id and timestamp
Redis refuses writes	Direct insert into TimescaleDB	Direct insert (normal path; Redis is only needed when that fails)
Both down	Lost, counted	Lost, counted
Deploy / docker stop (SIGTERM)	In-memory persistence queue drained into Redis, final flush	Events already parked stay in Redis for the next process

Chat messages

Every chat ingest path calls lo_chat::buffer_or_persist (crates/lo-chat/src/buffer.rs): the message is pushed onto lumio:chat:buffer:{account_id}; if that push fails it is inserted directly. The chat flush worker (apps/api/src/workers/chat_flush.rs) drains the buffers every 5 s. A batch that fails 5 times in a row moves to lumio:chat:deadletter:{account_id} (72 h TTL), which a leader-gated drain replays every ~60 s. The insert is idempotent (ON CONFLICT DO NOTHING), so replays never duplicate.

Platform events

Ingest paths (platform workers, webhooks, the 7TV and stream-status services) persist events with lo_events::persist_event (crates/lo-events/src/durable.rs):

1. The event gets its row id and created_at before the first insert attempt.
2. The insert is tried once, inline, for at most 3 s (EVENT_INSERT_TIMEOUT). Ingest loops are serial, so with a saturated pool waiting the full 30 s acquire timeout per event would stall alerts.
3. On a transient error (I/O, pool timeout, connection lost, statement timeout, the 3 s bound, serialization failure …) the event is appended to lumio:events:deadletter:{account_id} (72 h TTL, refreshed on every park) and the caller gets the row it will be stored as, so the live alert still goes out.
4. On a permanent error (SQLSTATE class 22, 23 or 42, or a decode error) nothing is parked; replaying it could never succeed. The error is returned and counted as rejected.

The event retry worker (apps/api/src/workers/event_retry.rs) replays the lists every 30 s. One replica per cycle replays, the holder of lumio:events:deadletter:drain:leader. For each account it drains the list and inserts each event with its original id and created_at, so the event lands in the stream session it happened in. It stops at the first transient error and puts the rest back, and that ends the whole pass: the remaining accounts wait for the next cycle, so replay never competes with live ingest for the TimescaleDB pool while it is still unhealthy. The same Redis scan feeds the replay and the events_dead_letter_depth gauge.

Replays are idempotent:

• events with an external_id keep the advisory-lock + EXISTS dedup on (account_id, external_id);
• events without one conflict on the primary key (id, created_at).

A replay that finds the event already stored counts as recovered.

Two callers keep the fail-fast lo_events::insert_event:

• REST POST /v1/events/emit: the client sees the error and can retry.
• The Spotify playback-control log (GraphQL and REST, the same on both). It records a dashboard user's own action, not a platform delivery.

Not covered

• An event whose insert fails while Redis is also unreachable is lost.
• A parked event older than 72 h (by its original timestamp) is dropped on replay.
• A replayed event is not broadcast again. Its alert already went out when it was parked.
• Follow-up writes that target the row before it lands find nothing. The gift-bundle recipient enrichment is one example: the header row is replayed without the recipient list.
• The dead-letter list is drained with an atomic read-and-delete. A process killed between that read and the insert or re-park loses the batch.

Shutdown

Docker and Kubernetes stop a container with SIGTERM, then SIGKILL after the grace period (Docker default: 10 s). The API handles SIGTERM and SIGINT itself (apps/api/src/shutdown.rs; actix's own signal handling is disabled):

1. The HTTP server stops gracefully. In-flight requests get 5 s, then open WebSocket sessions are closed.
2. At the same time, ingest and timer workers are cancelled. The per-account platform workers get 5 s to stop. The Twitch EventSub worker drains its in-memory persistence queue into the Redis chat buffer before it reports stopped.
3. The chat flush worker is cancelled last and runs one final flush, bounded to 3 s.

The drain takes at most 8 s, so keep the container stop grace period at 10 s or more. Whatever is still in Redis afterwards is picked up by the next process.

crawler-twitch also handles SIGTERM. Its writer flushes what is already on the result bus. A shard's current aggregation window, not yet published to the bus, is not covered.

Database timeouts

The API pools set statement_timeout on every connection ([database] / [timescale] statement_timeout_secs, default 30 in apps/api/config/default.toml; 0 = no limit). One runaway query is cancelled (SQLSTATE 57014) instead of holding one of the five TimescaleDB connections indefinitely and stalling chat flush, event ingest and audit writes at once. A cancelled event insert is transient, so the event is parked, not lost.

Work that legitimately runs longer lifts the limit for one transaction with lo_database::begin_unbounded (or the crate-local equivalent):

• retention deletes (chat, events, GIF content);
• GDPR erasure deletes and the data-export reads;
• the account dissolution DELETE.

Startup backfills run on a separate one-connection TimescaleDB pool without the limit, which is closed once they finish. Code on that path must never acquire from the pool while it holds a transaction from the same pool: the second acquire waits for the one connection the transaction already holds, fails after connect_timeout, and the boot stalls before the server binds. Reads inside a transaction go through the transaction (query helpers take impl sqlx::PgExecutor<'_>, not &PgPool).

Migrations (lo_database::run_migrations / run_tsdb_migrations) run on a dedicated connection with statement_timeout = 0 and lock_timeout = '5s'. A DDL statement queued behind a long query would otherwise block every later query on that table. With the timeout it gives up and is retried every 5 s, up to 12 times, then startup fails. The connection is closed afterwards, so neither setting reaches the pool. A second replica that waits on the migrator's advisory lock meanwhile gives up after the same budget, about 2 minutes.

crawler-twitch, bot-module-worker and automation-worker share lo-database but not the API config, so their pools have no statement_timeout unless their own config sets statement_timeout_secs.

Metrics

Metric	Meaning
events_persist_failures_total{outcome}	Failed event inserts by outcome: parked, recovered, lost, rejected. Any lost is event loss
events_dead_letter_depth	Parked events across all accounts, measured from Redis every 30 s. Not returning to 0 = replay is stuck
chat_*	See Metrics

No series carries an account label (cardinality rule).