Testing, Preview & Incident Response

Composer needs tighter review rigour than an existing internal precedent already requires, at higher stakes since this is client-facing — and today neither a preview-before-publish step nor a plan for a live failure exists. The core defence is one mechanism reused two ways: render-and-compare visual regression, triggered only where the person responsible for a change isn't already watching the result live.

Two things are called "engine" around this brief, and this page needs both: Composer's mutation engine (the Change[] fold, rebase, enforcement — Composer's own code) and the Resolver Engine (the rendering pipeline, a separate delivery). Each is named in full below.

The review-rigour precedent

Composer is squarely the category of work an existing quality-risk precedent in this repo (NEO-1668, the Entain Shell Template idea doc) warns needs tighter review than routine frontend work — it records that an adjacent template-configuration feature surfaced real quality problems with AI-assisted UI work — at higher stakes than that precedent's own case: client-facing, not internal admin.

The two incident classes

Two genuinely different failure classes, needing different fixes:

Class	What it looks like	Fix
A single Project going bad	A bad rebase or a bad Element edit breaks one live BSR render; everything else stays fine.	Revert by pointer reassignment (below), with the existing render-failure signal wired to a shared channel so someone hears about it (below). What stays open is the leaf: a campaign fork still pinned to the bad version isn't moved by the Project's own revert (Open questions).
A defect in code, shipped by deploy	Either Composer's mutation engine or the Resolver Engine's rendering code, not caused by any Composer edit; potentially hits every Project at once.	Reverting one Project's version does nothing. Either a fast code rollback, or a kill switch falling back to a non-Composer render path — and that second option only exists if the schema-fork question (Identity, Versions & Schema Versioning) lands on "separate." If Composer extends the engine's format in place, there is no fallback path, only a forward fix via code rollback.

Decided: reverting a Project is a pointer reassignment

The Project identity's currentVersionId gets reassigned to a prior frozen version — not a full new fork through the normal commit and approval cycle, since incident-response speed matters most here. It still carries a lightweight audit trail, who reverted and when, reusing the same committedBy / committedAt-style fields designed for ordinary versions, applied to the revert action itself. A revert is one of the lifecycle operations outside Change[] (The Change Model). What it does not do on its own is retarget a campaign fork pinned to the bad version — see Open questions.

Decided: Blair's go/no-go authority extends to code-level rollback

One clear owner rather than fragmenting authority, given staffing is already ad hoc (Process & Organisation). The content-side consequence of a rollback — committed versions ahead of Composer's deployed transformer chain fail loudly — is designed in Identity, Versions & Schema Versioning.

Preview means both

Decided

Some form of fast static mockup and some form of real render both need to exist — not one standing in for the other. The mockup shows during active editing; the real render triggers on save or commit (or an explicit "preview real render" action), never live on every keystroke.

The real render is asynchronous and never blocks the save. Nobody hitting save can wait on a Remotion render, so the save completes and the render lands after it. That has a consequence worth stating rather than discovering: a render that comes back bad arrives after the version is already committed, so the response is the ordinary incident path — surface it on the item, revert if it matters — not a rollback of the save. The mockup is what an author actually judges by while working, which is why the triggers below deliberately exclude a user's own edits: the safety argument rests on the mockup plus this post-commit render, not on a blocking gate.

Visual regression: one mechanism, two triggers

The primary defence against both incident classes is the same underlying mechanism, reused two ways, not two systems — the "one way to do a thing" law applied to testing. Render stills of a document and compare them pixel by pixel against a prior render of the same document, with a tolerance rather than a strict zero-difference bar. Remotion's render path is CPU-only and has a documented history of subpixel text-rendering artifacts, so a strict bar risks flagging noise; the engine brief's own PR gate makes the same call, diffing with a tolerance rather than demanding equality. The exact threshold is uncalibrated (Open questions).

This only ever runs where the person responsible for the change isn't already watching the result live. A user editing a real Scene, Element, or Project sees a mockup as they work and the real render on save or commit (above); an automated diff of their own direct edit would re-check something they see for themselves at commit. The two places this adds something a human's own eyes can't cover:

Trigger	Compares	Catches	Blocking?
Picking up an upstream version — a fork-level rebase onto a newer base, or a saved diff containing a rebaseInstance — where the fork is a renderable document: a Scene, a Project, or a campaign fork	The rebased piece, under whichever comparison rule the open question on Fork, Rebase & Conflict Resolution settles	Content arriving from somewhere the person doing the rebase wasn't watching live — a version bump elsewhere rippling into a visual change here	Yes — gates the mandatory Approve Migration step
A PR against rendering or mutation-engine code — the Resolver Engine's pipeline, or Composer's own resolver, rebase, and enforcement code	A dedicated feature-exercising fixture document, rendered in full at a tunable frame stride; the fixture, stride, and tolerance spec is the engine's own, in the engine brief's shared Test plan section	A defect in the code itself — the second incident class, which the rebase trigger can't catch, since changed code has no real Project to diff against	Yes — gates merging the PR

flowchart LR
    subgraph never["Never triggers a diff"]
        LE["a user's own live edit<br/>(mockup while editing, real render at commit)"]
    end
    RB["rebase of a fork"] --> RC["render the rebased piece"] --> AM["Approve Migration"]
    PR["PR against rendering or<br/>mutation-engine code"] --> FX["render the full fixture document<br/>tunable frame stride"] --> MG["merge gate"]
    RC --> TOL["compare with tolerance"]
    FX --> TOL

Element and Overlay rebases are scored structurally, not visually

Neither altitude is renderable on its own. An Element carries no size, no duration, and — when shared across brands — no single brand-style binding; an Overlay carries no duration either, since its span is derived from the Scene instances of whichever Project places it (Overlays). In both cases any synthetic box or duration chosen for the render would decide the verdict. A draft at either altitude is therefore scored on structural signals alone — did each replayed Change still find its target — and the visual consequence is caught one level up, at the first Scene or Project rebase that picks the new version up, which is a real render of a real document. Approve Migration still gates these rebases; only the render half is absent.

The rebase trigger defends against the single-Project class; the PR gate is the concrete answer to the code-defect class.

The Resolver Engine has its own gate; sharing the implementation is Composer's ask

The engine brief carries a PR gate of exactly this shape as a required, blocking CI gate on its own code, and declares it independent of any Composer-side rebase mechanism. That the two are one render-and-compare implementation rather than two is Composer's position, listed as an unreconciled ask in The Resolver Engine Dependency.

The rebase trigger is also the concrete mechanism behind the confidence score the fork model commits to (Fork, Rebase & Conflict Resolution). And it gives a partial answer to a Change-model risk — individually valid Changes composing into a visually broken end state — because that risk only materialises at the point two independent things combine, which is exactly what a rebase is; neither editor could have seen the collision on their own screen. That is why this belongs at the rebase trigger and not as a check on every ordinary edit; what it leaves uncovered is a single fork's own individually-valid-but-jointly-bad Changes, which no rebase would ever surface.

Decided: authorship at two levels, not redundant

Every Change carries who authored it and when, and every committed version separately carries who committed it and when. Change[] is a compacted diff, so field-level authorship only ever answers "who last touched this in this fork." Version-level authorship doesn't have that limitation, since every version is frozen, and it's the more useful granularity for incident work: "who committed the version that broke this" and "which version do we revert to" are both answered there.

Decided: baseline stills live in S3

Confirmed rather than presumed, matching every other asset in this system.

Testing Composer's own engine

Everything above is about catching a visual regression in rendered content. Composer's mutation engine — the Change[] fold and compaction, the rebase conflict-resolution and confidence-scoring algorithm, and permission enforcement — is just code, and needs the same unit and integration test coverage as anything else in this codebase: resolveDocument folding two Changes to the same instance correctly, compaction dropping an edit that returns a target to its base value, the confidence-scoring rule producing the right verdict for a given conflict shape, a Change at a given rung being accepted or rejected, with no rendering involved.

Decided: an ordinary CI merge gate

The same treatment any other code in this codebase gets, not a softer expectation.

Notification and detection

Render success or failure already has real, if silent, infrastructure: a dead-letter-queue handler (render-dlq.ts) marks a failed render in the database, and a PostHog-based observability layer captures typed render_failed / qc_failed events — but neither notifies a human. No render output quality monitoring exists today; the only render inspection tool found in this repository is a manual, on-demand still-render script with no diffing and no CI wiring.

Decided: wire the existing failure signal to a shared channel now

Route that already-captured signal to a shared incident channel immediately — a cheap, separate fix from anything else on this page. Given staffing is ad hoc by design, a named on-call rotation doesn't fit; a single shared channel any detection path posts into is the honest fit for how this team operates.

Risks

• Whether a purely rebase- and PR-triggered check misses some class of same-session, self-inflicted visual regression that a human editing quickly doesn't notice at commit, despite this page's reasoning that they would. Not knowable until real usage.
• Whether Remotion's CPU-only rendering turns out noisy enough to make a useful tolerance threshold hard to find, versus being a minor calibration exercise.

Open questions

• The tolerance threshold itself — how much pixel difference counts as noise, and whether it needs to vary by region of the frame or can be a single global number. Not calibrated against real renders.
• How many fixture documents the PR gate needs, whether one comprehensive fixture is enough or feature interactions need their own, and the frame stride — deliberately tunable, pending real render-time data.
• What a Project revert does for forks pinned to the reverted-away version. Live renders come from campaign forks pinning their own version; moving the Project's currentVersionId back to v2 leaves a campaign on v3 rendering v3, and it is not "behind" v2, so the staleness badge never flags it. Whether revert must also retarget the leaf forks that render — which for a campaign means the fork-level rebase of its own base, not rebaseInstance (The Change Model) — and whether the badge needs a "diverged" state for a version no longer on currentVersionId's lineage, is not decided; it also needs the on-demand reverse-reference query (Fork, Rebase & Conflict Resolution).
• Whether deploying a new major transformer runs render-and-compare over a sample of real live Projects, since a silent cross-major upcast changes every un-migrated live render at once with no rebase gate firing (Identity, Versions & Schema Versioning → Risks).
• How this connects to, or supersedes, the separate retail-studio-template-testing effort. Its current state could not be confirmed against this repository checkout, so whether it already does some of this is genuinely unclear, not just unconnected.
• Whether NEO-1657's advisory migration-time visual diff is the same mechanism (Identity, Versions & Schema Versioning).