Kick Chat-Moderation Listener

Kick sends no webhook when a message is deleted on kick.com or when a user is unbanned there. Kick declined to add one (KickEngineering/KickDevDocs#310, closed 2025-12-14). Both actions are broadcast only on Kick's unofficial public chat WebSocket, which runs on Pusher. The API keeps one outbound connection to it and handles exactly two events:

Pusher event	Handling
App\Events\MessageDeletedEvent	lo_chat::soft_delete_message, moderation log delete, WS chat:delete + chat:moderation_log
App\Events\UserUnbannedEvent	lo_chat::mark_user_unbanned, moderation log unban with the acting moderator, WS chat:moderation_log
everything else (ChatMessageEvent, UserBannedEvent, subs, hosts, …)	dropped by event name, nothing stored or forwarded

Chat messages come in through the official chat.message.sent webhook, and bans and timeouts through moderation.banned. The listener adds no new data category.

Unofficial means best effort. Kick documents no contract for this socket and is moving its own web client to Centrifugo. When the listener fails, deletes and unbans made on kick.com stop reaching Lumio. The webhook intake, chat, events and moderation issued from Lumio are not affected: the listener sits on neither the webhook path nor the request path.

Protocol

• Socket: wss://ws-{cluster}.pusher.com/app/{app_key}?protocol=7&client=js&version=8.4.0&flash=false. Connecting needs no authentication.
• Channel: chatrooms.{chatroom_id}.v2, subscribed with pusher:subscribe and an empty auth. Kick confirms each subscription with pusher_internal:subscription_succeeded.
• Payload: data is a JSON string inside the frame. A delete carries {id, message:{id}, aiModerated, violatedRules}. An unban carries {id, user:{id,username,slug}, unbanned_by:{id,username,slug}, permanent}.
• Keep-alive: the listener answers pusher:ping with pusher:pong. It sends its own pusher:ping after activity_timeout seconds of silence (from pusher:connection_established, 120 s by default) and reconnects when no answer arrives within 30 s.
• Message ids: MessageDeletedEvent.message.id is the id of the deleted ChatMessageEvent. A 2026-09-30 capture of 20 live chat rooms matched 25 of 25 deletes to a chat message seen earlier on the socket. The same UUID is the message_id Lumio stores from the chat.message.sent webhook as platform_message_id. An id Lumio does not know only increments a counter and can never delete another message.

Chat-room ids

The official API has no chat-room id. For each connected Kick channel (channel_connections with platform = 'kick'), the listener resolves one as follows:

1. It reads Redis lumio:kick:chatroom_id:{broadcaster_user_id} (7 d TTL).
2. On a miss, it gets the slug from the stored channel_login, or else from the official GET /public/v1/channels?broadcaster_user_id with the connection's token.
3. It reads chatroom.id from kick.com/api/v2/channels/{slug} (undocumented, see Connections → undocumented kick.com endpoints).

If the lookup fails, the channel is skipped with a warn! and retried after 10 minutes. There are at most 50 lookups per resync. No migration is involved.

Runtime shape

apps/api/src/workers/kick_chat_moderation.rs:

• One listener per deployment. A Redis lease (lumio:kick:realtime_listener:leader, 30 s TTL, renewed every 10 s by its owner) picks the replica that listens. The other replicas retry every 15 s. If the lease holder dies, another replica takes over within about 30 s. Processing an event twice during a hand-over is harmless: deletes are idempotent and unbans are deduplicated.
• Channels follow connections. The listener re-reads the connected Kick channels every resync_interval_secs. The same trigger that starts the Kick webhook worker (a connect, or a disconnect through the worker manager) also wakes the listener immediately.
• Sharding. Each connection carries at most max_channels_per_connection chat rooms (250 channels means 3 connections at the default 100). If Kick does not confirm a subscribe within subscribe_confirm_timeout_secs, or refuses it, the room is logged and moved to the next connection. After 3 moves it is parked for 10 minutes. A connection with no rooms closes its socket.
• Reconnect. A dropped or failed connection reconnects with backoff from 1 s up to 60 s, plus jitter. The backoff resets after a connection that lived at least 60 s.
• Provider-neutral. A connection talks only through the RealtimeAdapter trait (services/kick_chat_moderation.rs). PusherAdapter is the only implementation. A Centrifugo adapter would be one more implementation, not a rewrite.
• Fail-safe. A malformed event is dropped and counted, and a warn! is logged for the 1st, 2nd, 4th … occurrence. When the handler queue (1024 events) is full, the event is dropped and counted instead of blocking the socket. A delete that arrives before the chat buffer flush (Redis to TimescaleDB every 5 s) is retried once after 12 s.

Deduplication

• Delete. The deleted_at IS NULL guard in soft_delete_message makes the delete idempotent. A message already deleted from Lumio, or deleted twice on the socket, changes nothing and writes no second log line. The delete event names no moderator, so the log uses "moderator", the placeholder Twitch's channel.chat.message_delete path uses. When Kick marks the delete aiModerated, the log uses "Kick AutoMod" and stores violatedRules in details.
• Unban. An unban for the same user already logged within 60 s (KICK_BAN_WEBHOOK_DEDUP_SECS, e.g. an unban issued from Lumio) is skipped. The payload has the same shape as the Twitch channel.unban path, including duration_secs / ends_at (both null).

Pusher key store

The Pusher app_key is public, but kick.com hands it to its web client at runtime and does not hardcode it. The API keeps it the same way it keeps the InnerTube key (services/kick_realtime_credentials.rs):

override (emergency pin) -> Redis lumio:kick:realtime_pusher_credentials (24 h) -> fresh fetch -> cold-boot constant

• Fetch: POST https://web.kick.com/api/v1/realtime/connection with no auth and the body {"client":{"id":"<uuid>","type":"web"},"capabilities":{"accepted_providers":[{"provider":"pusher"}]}}. It answers {"data":{"connections":[{"provider":"pusher","credentials":{"app_key":"…","cluster":"us2"}}],"mode":"websocket"},"message":"success"}. Only a pusher entry with a valid app_key and cluster is accepted. Both values are charset-checked because they end up in the socket URL.
• Cold boot: cold_boot_app_key / cold_boot_cluster (32cbd69e4b950bf97679 / us2, measured 2026-09-30). These are used only when override, Redis and the fetch all came up empty.
• Refresher (workers/kick_realtime_credentials.rs): does a full resolution at startup, then runs every refresh_interval_secs (6 h). The leader is chosen through Redis SET NX on lumio:kick:realtime_refresh:leader, with a TTL of about 90 % of the interval. The leader fetches and writes Redis, and followers adopt the Redis value. A key change logs the old and new public values at info!, and the listener reconnects every connection with the new key.
• Failed fetch: the key already held stays in use, and a process still on the cold-boot key adopts the Redis value if one exists.

Emergency pin

Set both app_key_override and cluster_override (preferably through LUMIO__KICK__REALTIME__APP_KEY_OVERRIDE / LUMIO__KICK__REALTIME__CLUSTER_OVERRIDE). A pin wins over every other source and disables the refresher, and the API logs one warn! at startup. If only one of the two is set, or the pair is malformed, the pin is ignored. Unset both to resume auto-refresh.

Alarms

The leader drives two alarms. Each is edge-triggered and latched in Redis: one incident sends one Discord message, and one recovery message follows when the incident clears. Both are logged at ERROR even when no webhook is configured.

Alarm	Condition	Redis latch
Cold boot	No pin, Redis empty, fetch failed: the listener runs on the cold-boot key	lumio:kick:realtime_cold_boot_alarm
Pusher no longer offered	The fetch worked, but kick.com offered no pusher provider (Centrifugo only)	lumio:kick:realtime_pusher_not_offered_alarm

alert_after_failures (default 1) requires that many consecutive bad cycles before an alarm fires. A cycle that says nothing about a condition keeps its latch: a failed fetch neither raises nor clears "not offered". When "Pusher no longer offered" fires, the listener keeps running on the last known key until Kick shuts Pusher down. The follow-up is a Centrifugo adapter.

Configuration

[kick.realtime] in apps/api/config/default.toml:

Key	ENV	Default	Meaning
enabled	LUMIO__KICK__REALTIME__ENABLED	true	Disable switch for the listener and the key refresher.
max_channels_per_connection	LUMIO__KICK__REALTIME__MAX_CHANNELS_PER_CONNECTION	100	Chat rooms per socket.
subscribe_confirm_timeout_secs	LUMIO__KICK__REALTIME__SUBSCRIBE_CONFIRM_TIMEOUT_SECS	10	Wait for subscription_succeeded before a room moves on.
resync_interval_secs	LUMIO__KICK__REALTIME__RESYNC_INTERVAL_SECS	60	Channel re-read interval (minimum 10).
app_key_override / cluster_override	LUMIO__KICK__REALTIME__APP_KEY_OVERRIDE / …__CLUSTER_OVERRIDE	empty	Emergency pin (both or neither).
cold_boot_app_key / cold_boot_cluster	LUMIO__KICK__REALTIME__COLD_BOOT_APP_KEY / …__COLD_BOOT_CLUSTER	32cbd69e4b950bf97679 / us2	Stage-4 value, never a pin.
connection_url	LUMIO__KICK__REALTIME__CONNECTION_URL	https://web.kick.com/api/v1/realtime/connection	Key fetch endpoint.
refresh_interval_secs	LUMIO__KICK__REALTIME__REFRESH_INTERVAL_SECS	21600	Key refresh interval (minimum 60).
alert_webhook_url	LUMIO__KICK__REALTIME__ALERT_WEBHOOK_URL	empty	Discord webhook for both alarms (a SensitiveString, never logged).
alert_after_failures	LUMIO__KICK__REALTIME__ALERT_AFTER_FAILURES	1	Consecutive bad cycles before an alarm; 0 counts as 1.

Metrics

On the internal /metrics server. No series carries a channel, account or user id.

Metric	Type	Labels	Meaning
kick_chat_ws_connected	gauge	-	Open listener connections. 0 with Kick channels connected means the listener is down or not the leader on this replica.
kick_chat_ws_events_total	counter	event (message_deleted, user_unbanned, chat_message, user_banned, other, unknown), outcome (applied, noop, duplicate, dropped, malformed, unrouted, overflow, error)	Socket events and what happened to them. noop = message already deleted or unknown to Lumio.
kick_realtime_credentials_source	gauge	source (override, redis, fetch, cold_boot)	1 for the source of the key in use.
kick_realtime_pusher_offered	gauge	-	1 while kick.com still offers Pusher, 0 once it stops.
kick_realtime_credentials_refresh_total	counter	outcome (changed, unchanged, not_offered, fetch_failed)	Leader refresh cycles.

Audit

The listener emits no audit event. Deletes and unbans made on kick.com land in the chat moderation log, which is not a security record (see Audit Events, "do not emit").

Tests

• Parser and key-store unit tests with the measured payloads as fixtures (apps/api/tests/fixtures/kick_pusher/*.json, embedded via include_str!, compile_data in Bazel).
• DB integration tests apps/api/tests/kick_chat_moderation.rs: delete, AutoMod attribution, dedup against a delete issued from Lumio, unknown id, unban resets ban state, unban dedup.
• Live smoke test (network, not in CI): cargo test -p lumio-api --lib kick_chat_live_smoke -- --ignored --nocapture fetches the key, resolves a chat room through v2 and waits for the subscribe confirmation.

Code

File	Role
apps/api/src/services/kick_chat_moderation.rs	Frame model, RealtimeAdapter, PusherAdapter, ShardPlan, chat-room lookup, apply functions, listener metrics
apps/api/src/workers/kick_chat_moderation.rs	Lease, session, shards, event handler
apps/api/src/services/kick_realtime_credentials.rs	Key store, resolution chain, alarm decision, key metrics
apps/api/src/workers/kick_realtime_credentials.rs	Refresher, leader lock, Discord alarms
crates/lo-kick-api/src/client.rs	KickWebsiteChannel.chatroom_id