Error Tracking (Sentry)

Lumio's four Next.js apps - apps/web, apps/admin, apps/id, apps/shopify - each ship with a @sentry/nextjs integration. apps/stats has none. The Rust backend is on sentry-rust via lo-telemetry, covered in Rust backend at the end of this page.

When no DSN is configured, the integration is a no-op - apps boot normally, no events are sent. Local dev typically leaves the DSN blank; staging and production supply it via the container environment at run time (LUMIO_PUBLIC_SENTRY_DSN, see Environment variables).

Browser Sentry is consent-gated (web, ID) or off (admin, Shopify)

A DSN alone is not enough. Sentry is classified as a Functional cookie in the published policy (§ 25 TDDDG / GDPR), so the browser SDK stays fully inert - no Sentry.init, no capture, no storage writes, no network traffic - until the user grants functional consent. apps/web and apps/id each keep the lifecycle logic in src/lib/sentry-consent.ts (syncSentryConsent()).

The DSN reaches the browser at request time: the root layout (a server component) calls getBrowserSentryConfig() from src/lib/config.ts and passes dsn and environment as props to <SentryConsentSync />. The release images are built without a DSN, and a NEXT_PUBLIC_* value is frozen at next build, so a DSN read from process.env.NEXT_PUBLIC_SENTRY_DSN in browser code would always be empty in a deployed image.

<SentryConsentSync /> boots the SDK and re-runs the sync when the preference changes at runtime; a cross-tab storage listener tears the client down on withdrawal in another tab. It is mounted inside ConsentProvider in apps/web/src/app/(main)/layout.tsx and in the root layout of apps/id. Consent is read from the lumio_cookie_consent localStorage key via getStoredConsent() from @lumio/cookies-consent - that stored choice is the single source of truth, even on routes where ConsentProvider runs in exempt mode.

<SentryConsentSync /> is the entry point that actually matters: Next 16 builds with Turbopack, and @sentry/nextjs only injects sentry.client.config.ts into webpack builds. Under next dev / next build that file is never loaded, so anything placed only there does not run in the browser.

The extension Supervisor and Runtime frames follow the same rule - see Extension Supervisor and Runtime.

apps/admin has no consent banner, so it runs no browser Sentry at all: its sentry.client.config.ts is intentionally empty. Server-side capture (src/instrumentation.ts) is unaffected. Give the admin app a consent surface before adding a browser Sentry.init there.

apps/shopify follows the admin pattern for the same reason. Its embedded page renders inside the Shopify admin iframe and has no consent surface, so sentry.client.config.ts is empty and the CSP that src/proxy.ts sets has no Sentry origin in connect-src. Only the server runtime reports errors: the HMAC check and CSP in src/proxy.ts, the page's server render and the src/app/api/* forward routes.

Files per app

Each of apps/{web,admin,id,shopify}/ has the same five files (paths relative to the app):

File	Loaded by	Purpose
next.config.ts	build	Wraps withSentryConfig (source-map upload, build-time injection)
sentry.client.config.ts	browser (webpack builds only)	In apps/web and apps/id a one-line delegate to syncSentryConsent() (build-inlined DSN only); empty in apps/admin and apps/shopify. Turbopack never loads it - see the caution above
src/sentry.server.config.ts	Node runtime	SSR / route-handler / Server-Action capture
src/sentry.edge.config.ts	Edge runtime	Edge route handlers
src/instrumentation.ts	server	register() imports the right runtime config and binds the logger (apps/shopify has no @lumio/logger, so nothing to bind); also exports onRequestError = Sentry.captureRequestError

instrumentation.ts belongs in src/

All four apps use the src/ layout (src/app, src/proxy.ts). Next.js looks up instrumentation.ts - like proxy.ts and instrumentation-client.ts - only in the folder that holds app/, so here in src/. A copy in the app root is ignored without a warning: register() never runs, and server-side Sentry, the logger binding and onRequestError are all off whatever DSN is set. apps/web/__tests__/next-convention-files.test.ts fails if a convention file sits in the app root of web, admin, id, stats or shopify, and checks that web, admin, id and shopify register their runtime configs from src/instrumentation.ts. After next build the server output contains .next/server/instrumentation.js; if that file is missing, the hook was not found.

Configuration is duplicated across the four apps on purpose - the apps evolve independently. The browser side differs on purpose: apps/web and apps/id are consent-gated, apps/admin and apps/shopify have no browser Sentry, and none of them uses Session Replay. The server and edge configs are identical in web, admin and ID; apps/shopify has the same settings with a stricter scrub (see What gets stripped).

Environment variables

Variable	Where	Purpose
LUMIO_PUBLIC_SENTRY_DSN	Run time (browser + server)	Public DSN. Read per request by the root layout and handed to the browser SDK (web and ID, after Functional consent); also the server fallback. This is the name to set on deployed containers. apps/web also adds its ingest origin to the CSP connect-src.
NEXT_PUBLIC_SENTRY_DSN	Build time	Local-dev fallback for LUMIO_PUBLIC_SENTRY_DSN. Inlined at next build, so setting it on a container of a prebuilt image has no effect.
SENTRY_DSN	Server only	Optional - falls back to LUMIO_PUBLIC_SENTRY_DSN, then NEXT_PUBLIC_SENTRY_DSN. Useful when you want the SSR / Edge runtimes on a different project than the browser. In apps/admin and apps/shopify it is the only DSN that matters (no browser SDK).
SENTRY_ORG	Build (CI)	Sentry org slug - required for source-map upload. Leave blank locally.
SENTRY_PROJECT	Build (CI)	Sentry project slug - same.
SENTRY_AUTH_TOKEN	Build (CI)	Auth token for source-map upload. Generate one in Sentry → Settings → Auth Tokens.
LUMIO_ENV / NEXT_PUBLIC_LUMIO_ENV	Both	Tags every event with the environment (development / staging / production). Falls back to NODE_ENV. The browser SDK gets LUMIO_ENV from the root layout at request time, like the DSN.

Source-map upload only runs when SENTRY_AUTH_TOKEN + SENTRY_ORG + SENTRY_PROJECT are all set - typically only in CI for staging / production builds.

What gets captured

• Unhandled exceptions in browser code (React error boundaries, Promise rejections, sync throws) - web and ID only, and only after Functional consent. Never in admin or Shopify.
• SSR / route-handler errors through the onRequestError export in src/instrumentation.ts (Sentry's recommended hook).
• Anything emitted via logger.error(...) or logger.warn(...) - the logger calls Sentry.captureException / Sentry.captureMessage when bound. See Logging for the binding details.

What gets stripped

beforeSend in every runtime config deletes the authorization and cookie keys from event.request.headers before the payload leaves the process. In the Next.js apps, sendDefaultPii: false keeps Sentry's automatic IP / user-agent collection off, so cookie-issued JWTs and popout tokens never end up in Sentry; the extension frames get the same from dataCollection (see Extension Supervisor and Runtime).

apps/shopify strips more, in src/lib/sentry-scrub.ts, through both beforeSend and beforeSendTransaction. The embedded page is opened with Shopify's launch query, and the forward routes carry Lumio and order data:

• the authorization and cookie headers (any casing), request.cookies and request.data (no request body is ever attached);
• the query params id_token, hmac, session, code and state are replaced with [Filtered] in request.url and request.query_string, and in the string values of span data and breadcrumb data (where transactions repeat the URL as url.full / http.query).

apps/shopify/__tests__/sentry-scrub.test.ts covers it with a URL that carries all five params.

If you want to enrich events with explicit user info (e.g. account ID), set it manually after login:

import * as Sentry from "@sentry/nextjs";

Sentry.setUser({ id: account.id, ip_address: undefined });

ip_address: undefined opts out of Sentry's IP capture even if sendDefaultPii ever flips on later.

Session Replay

Not enabled in any app. Replay was removed from apps/web under ZAF-91 and from apps/id / apps/admin under ZAF-2681 - it is the highest-PII component, and the privacy policy states that Lumio does not use it. Do not re-add it, even consent-gated, without a disclosed processing entry in the policy first.

Performance traces

tracesSampleRate is 0.1 when NODE_ENV === "production" (10 % of transactions) and 1.0 everywhere else, in all four apps on both the browser and server sides.

Suppressed noise

denyUrls on the browser side is the same list in all three apps:

denyUrls: [/extensions\//i, /^chrome:\/\//i, /^moz-extension:\/\//i, /^safari-web-extension:\/\//i]

Note the second entry matches the chrome:// internal scheme, not chrome-extension:// - the generic /extensions\//i pattern is what catches Chrome extension frames. Add entries when you see recurring noise from a specific extension or third-party script.

Adding a new captured event manually

Most code should just throw / call logger.error. For business-event tracking (e.g. "user upgraded plan"), use Sentry's addBreadcrumb or captureMessage directly:

import * as Sentry from "@sentry/nextjs";

Sentry.captureMessage("user upgraded plan", {
  level: "info",
  extra: { account_id: accountId, plan_slug: plan.slug },
});

These don't surface as errors but let you search them in Sentry's "Issues" tab when triaging.

Debugging the integration locally

Set the DSN in .env.local:

NEXT_PUBLIC_SENTRY_DSN=https://<key>@o<org>.ingest.sentry.io/<project>
LUMIO_ENV=development

LUMIO_PUBLIC_SENTRY_DSN works the same way. In apps/web and apps/id you must also accept the Functional cookie category in the consent banner (or the browser SDK stays inert and nothing is sent). To reset, clear the lumio_cookie_consent localStorage key and reload.

Then trigger an error from any client component:

"use client";

import { useEffect } from "react";

export default function DebugPage() {
  useEffect(() => {
    throw new Error("sentry smoketest");
  }, []);
  return null;
}

Reload the page, open Sentry → Issues, the event should land within a few seconds with environment: development. Filter that out before shipping or the dashboard fills with smoketest noise.

Extension Supervisor and Runtime

apps/extension-supervisor and apps/extension-runtime run in the browser as iframes inside the dashboard and inside OBS overlays and widgets. Each has @sentry/browser with a build-time DSN (SENTRY_DSN_SUPERVISOR / SENTRY_DSN_RUNTIME), and neither can read the user's consent itself, so the host page decides:

• ExtensionWorkerManager sends errorTracking: true in the Supervisor init message only when the host sets its errorTracking prop and getStoredConsent() grants Functional consent. Only dashboard hosts set the prop (widget editor, extension designer). The OBS hosts /overlay and /widget have no consent surface and never set it, so Sentry stays off there.
• The Supervisor calls Sentry.init only on such an init, and adds errorTracking=1 to the Runtime frame URL (dropping any such parameter already in extensionUrl). The Runtime calls Sentry.init only with that parameter.
• Without it, both frames' Sentry.captureMessage calls are no-ops. The decision is taken when the frame starts; a frame started before the user withdraws consent keeps it until it is reloaded. No Session Replay in either frame.

Both frames are on @sentry/browser 11, which has no sendDefaultPii. Its replacement dataCollection collects user info, cookies and request/response bodies when left unset, so both Sentry.init calls pin it to the restrictive baseline: no user info, cookies, bodies, GenAI/MCP inputs and outputs, database query data, queue data or GraphQL documents, and IP-revealing headers and query parameters (forwarded, -ip, remote-, via, -user) denied. Do not drop or loosen that block. The Next.js apps are still on @sentry/nextjs 10 and keep sendDefaultPii: false.

Rust backend (lo-telemetry)

Every Rust service calls lo_telemetry::init_sentry(&config.sentry, "<service>") before init_tracing. Sentry stays off unless [sentry] enabled = true and dsn is set; config keys and ENV names are in Installation → Error tracking (Rust services). The service names are the same as the diagnostics file's: api, twitch-bot, kick-bot, youtube-bot, discord-bot, crawler-twitch, bot-module-worker, automation-worker and innertube-proxy. The proxy has no TOML config and builds its [sentry] / [logging] values from env vars, see InnerTube Proxy → Logging and error tracking.

Events and breadcrumbs

init_tracing installs three Sentry tracing layers from sentry::integrations::tracing, each behind the stdout filter (RUST_LOG / [logging] level) plus a filter of its own: BreadcrumbLayer (breadcrumbs), ErrorLayer (events) and SpanLayer (spans from info up become transactions, sampled by [sentry] traces_sample_rate). There is no LogLayer, so no Sentry structured logs are sent. For every line that passes the stdout filter, [sentry] event_level decides:

Level	event_level = "warn" (default)	event_level = "error"
error	event + breadcrumb	event + breadcrumb
warn	event + breadcrumb	breadcrumb
info	breadcrumb	breadcrumb
debug / trace	nothing	nothing

The breadcrumb layer runs before the error layer, so an event's own line is already its last breadcrumb. init_tracing reads event_level once, from the value init_sentry stored; that is why init_sentry must run first. With Sentry off the layers keep the library default (error → event, warn/info → breadcrumb).

Panics are captured by the default sentry panic integration.

Lines from the twitch_irc library (twitch_irc and its submodules) are breadcrumbs only, whatever their level (BREADCRUMB_ONLY_TARGETS in crates/lo-telemetry/src/sentry_events.rs). The library logs every dropped pool connection, missing PONG and server RECONNECT at error and rebuilds the connection itself. A lasting outage is reported by the silence watchdog in lo-twitch-irc (crates/lo-twitch-irc/src/silence.rs): when a client has joined channels and receives nothing for 5 minutes, it logs error "Twitch IRC receives nothing while channels are joined, connections keep failing" once per outage, under its own lo_twitch_irc target. twitch-irc pings every open connection every 30 s and forwards the PONG, so a healthy pool is never silent that long.

Reconnect loops

An upstream WebSocket that drops and reconnects is expected behaviour, not an error. A worker logs a single drop or failed reconnect at info, and error only when the reconnect keeps failing: at RECONNECT_ERROR_THRESHOLD (5) consecutive failures and every further 5, decided by lo_common::reconnect_failure_is_persistent. The escalation has its own fixed message, so it groups into one Sentry issue:

Worker	Drop / single failure (info)	Lasting failure (error)
StreamElements (api::workers::streamelements)	StreamElements connection lost, reconnecting	StreamElements reconnect keeps failing
Twitch EventSub (lo_twitch_api::eventsub, api::workers::twitch_eventsub)	EventSub WebSocket read error, reconnecting, EventSub keepalive timeout exceeded, reconnecting, Failed to connect to EventSub, retrying	EventSub reconnect keeps failing
Kick bot Pusher chat (kick_bot::bot)	Failed to connect to Kick WebSocket — retrying, Kick WebSocket error — reconnecting, Kick WebSocket closed by server — reconnecting	Kick WebSocket reconnect keeps failing

The StreamElements and Kick bot counters reset when a connection stayed up for 60 s (the Kick bot counts failed connects and connections that ended sooner), the EventSub counter when a new session took its subscriptions. Every line carries consecutive_failures. New reconnect loops follow the same pattern.

GraphQL request log levels

POST /v1/gql logs one line per request. A response with errors gets its level from the errors' extensions.code, not from their message (classify_error_log_level in apps/api/src/graphql/mod.rs):

Errors in the response	Level	Message
Every error is the uncoded Authentication required literal (unauthenticated popout pages)	debug	GraphQL auth-required (unauthenticated request)
Every error has an expected client code: NOT_FOUND, UNAUTHORIZED, UNAUTHENTICATED, VALIDATION_ERROR, BAD_REQUEST, PLAN_LIMIT_REACHED, FEATURE_DISABLED (the Authentication required literal also counts)	info	GraphQL request with expected client errors
Anything else: a server fault, FORBIDDEN, an error without a code, or a mix of expected and unexpected	warn	GraphQL request with errors

So a missing history report or a dead refresh token is a breadcrumb, not a Sentry issue. FORBIDDEN stays warn on purpose: a refused permission check is a security signal. A resolver error only gets an expected level if it carries a code, so new resolvers map their failures through ApiError + crate::graphql::errors::to_graphql_error. Resolvers that answer with a bare message such as Overlay not found or No active account use crate::graphql::errors::{not_found, unauthorized, bad_request} instead: they add the code of the REST twin's ApiError variant and leave the message unchanged. A bare async_graphql::Error::new(…) has no code and is logged at warn, so keep it for real faults (e.g. a row missing right after it was written). graphql_requests_total{outcome="error"} counts every response with errors, whatever its log level.

What every event carries

• release = "<service>@<version>", with the workspace product version, so a regression can be tied to a release. server_name is the service name. The release workflow fills that release with commits and debug files, see Releases, commits and debug files.
• Tags service and target (the log target, e.g. api::workers::twitch_eventsub), plus these fields when the log line has them: platform, sub_type, notice_type, event_type, error_kind. IDs (account_id, channel_id, user_id, …) stay in the Rust Tracing Fields context and are never tags, because tags must stay low-cardinality.
• Fingerprint [target, message template], where the template is the message with every digit run replaced by <n>. "Pool connection 3 has failed" and "Pool connection 7 has failed" are one issue. The stack trace stays on the event.

What gets stripped

before_send and before_breadcrumb always remove viewer names and message text from the event's fields and breadcrumb data. They use the same key lists as the diagnostics file's redact_user_fields (crates/lo-telemetry/src/redact.rs), including JSON documents logged as a field. This cannot be switched off. IDs, event types and numbers stay. The log message itself is not rewritten, so never interpolate a viewer name into the message string: log it as a field.

Volume guard

One bad token can log the same error dozens of times per start, e.g. one Failed to create EventSub subscription per subscription type. The guard sends at most max_events_per_fingerprint (default 10) events per fingerprint per fingerprint_window_secs (default 60). The rest is dropped and counted. The first event of the next window carries the count as suppressed_count in the Volume Guard context. The guard lives in the process (no Redis), remembers up to 1024 fingerprints (least recently seen is evicted) and fails open. max_events_per_fingerprint = 0 switches it off.

Releases, commits and debug files

release-images.yml (Build & Push job) has a step Sentry release, commits and debug files. It runs after every image of the run is pushed, signed and promoted, for each Rust app the run built:

1. Release. It creates the release <app>@<version> in the Sentry project <app>, or reuses it if the SDK already created it with the first event. <version> is [workspace.package] version from the root Cargo.toml: the CARGO_PKG_VERSION the binary reports. It is not the release tag or the workflow_dispatch input. The step also finalizes the release.
2. Commits. sentry-cli releases set-commits <release> --commit zaflun/lumio@<sha> uses the GitHub integration of the Sentry org. Sentry computes the range from the previous release; a release without a predecessor gets the last 20 commits. A shipped commit whose message contains Fixes <SHORT-ID> resolves that Sentry issue with the release. Sentry fetches the commits in the background, so the step waits (up to 120 s) until the release reports them before it moves to the next app. Back-to-back fetches for all apps of one run left [email protected] with 0 commits. Every app's release gets the same commits, and Sentry resolves a Fixes issue in the last release that received the commit. That is why api, which owns the issues, is always processed last.
3. Debug file. It splits the debug info off the app's unstripped binary (objcopy --only-keep-debug --compress-debug-sections=zlib) and uploads that debug companion with sentry-cli debug-files upload. The companion has the binary's GNU build-id and therefore the same debug id. The unstripped api binary is about 1 GB, the companion only a fraction of it. Symbolication then shows file and line for Rust frames.

No deploy is reported. The workflow pushes images and moves the beta/latest channel tags; it does not deploy. Staging pulls :beta on the host (dev-stack/staging/verified-pull.sh). The SDK's environment already tells staging and production events apart.

Apps without a Sentry project are skipped with a warning. Every Rust app in RUST_APPS has a project today. A new app needs a project named exactly like the app (the service name).

Upload size limit. sentry-cli uploads debug files in 8 MiB chunks, several per request; the server's GET /api/0/organizations/<org>/chunk-upload/ reports chunkSize and maxRequestSize (32 MiB). Every proxy in front of Sentry must accept request bodies of that size. A proxy with nginx's default client_max_body_size 1m answers 413 Request Entity Too Large, and the step logs debug-files upload failed.

Where the debug info comes from. --config=release (.bazelrc) adds -Cdebuginfo=line-tables-only -Cstrip=none through @rules_rust//rust/settings:extra_rustc_flag. rules_rust appends these after its opt defaults (-Cdebuginfo=0 -Cstrip=debuginfo), so they win. Line tables are the smallest level that still maps an address to file and line, inlined frames included. The rust_binary therefore keeps its line tables. Each app's image_layer packs :<app>_stripped instead: the strip_debug rule (misc/strip_debug.bzl) runs the C++ toolchain's objcopy --strip-debug. That removes the DWARF sections and keeps the symbol table, the GNU build-id note and .eh_frame. The image binary has the same Sentry debug id as the uploaded file, and in-process backtraces keep their function names. The debug info never enters the image. CI (fastbuild) is not affected: the flags only live in --config=release, and the _stripped targets are manual.

Token and settings. The step reads three repository settings:

Setting	Kind	Value
SENTRY_RELEASE_TOKEN	secret	A Sentry Organization Auth Token (organization settings, Organization Tokens). Its only scope is org:ci: create releases, set commits, upload debug files. It cannot read issues or events.
SENTRY_URL	variable	the base URL of the Sentry instance
SENTRY_ORG	variable	the org slug

The token reaches sentry-cli only as the SENTRY_AUTH_TOKEN environment variable of that step, never in argv or the log. The triage token the agents use (event:read and friends) has no release scope and is not used here.

Fail-open. The step has continue-on-error: true and always exits 0. A missing secret or variable, an unreachable Sentry, a missing project or a failed upload each log a ::warning and the release still publishes its images. sentry-cli is a pinned GitHub release binary checked against its SHA-256 (SENTRY_CLI_VERSION / SENTRY_CLI_SHA256 in the step). When you bump it, take the digest from the release asset (gh api repos/getsentry/sentry-cli/releases/tags/<version>, field digest).

Checking a release. The step log prints <release>: <n> commits per app. In Sentry, Releases → <app>@<version> shows the commit count. On a new event from that release, the Rust frames carry filename and lineno, and Settings → Projects → <app> → Debug Files lists the uploaded binary.

Sentry is telemetry: no audit event, no permission.