Cues

A cue is a named, symbolic time landmark an animation can address instead of a raw frame number or a scene-relative offset. Two capabilities live here, deliberately sequenced: the addressing primitive (cheap, ships early) and the full cue/retiming system (a real author-facing entity, genuinely more design work, ships later) — bundled as one feature because they're the same underlying concept at different levels of completeness, not two unrelated systems.

The addressing primitive

The addressing primitive (auto-derived scene-edge cues, real at-addressing Zod) is scoped as an early, cheap slice of this feature — pulled forward specifically so templates authored against this delivery don't need their animations rewritten once the full cue system (authored mid-scene cues, hint-driven authoring, retiming) lands. The wiring terminals go in the wall now; the actual wiring is still later work within this same feature, not a different one.

Why this is cheap: most of the mechanism already exists

animationStartSchema (zod.ts) already resolves a time reference through an indirection layer, not a raw frame number directly — one of its existing forms, {rel, anchor, sceneId}, already names a scene's start/end symbolically. sceneAnchorResolver() (timeline.ts) already builds exactly a (sceneId, anchor) → frame lookup table from sceneStartFrames()'s scene-duration accumulation — functionally a cue table already, just not named that and currently restricted to overlay layers only (checkOverlays in validate.ts). interpolateAnimatedProps/continuousDriftAt (interpreter/animation.ts) only ever consume an already-resolved plain frame number — the actual animation runtime math needs zero changes.

Cues stay auto-derived at this stage, not an authoring concept yet. Every scene implicitly owns exactly two cues, start and end. There is no cues: Cue[] array on the document and no Builder authoring surface — a real, author-facing cue entity (mid-scene, hint-driven, campaign-editor authored) is this feature's own later work, not a different one.

The addressing shape itself is real Zod now, not illustrative-only:

const animationAtSchema = z.discriminatedUnion('mode', [
  z.object({ mode: z.literal('cueOffset'), cue: z.string(), offsetFrames: z.number() }),
  z.object({ mode: z.literal('cueFraction'), fromCue: z.string(), toCue: z.string(), fraction: z.number() }),
])

cueOffset resolves to cueFrame + offsetFrames. cueFraction lerps between two cues' resolved frames — no stretch-factor math needed yet, since cues are static scene edges until the full retiming system lands.

Widens today's existing scene-anchor addressing, doesn't replace it wholesale

animationStartSchema already has a {rel, anchor, sceneId} form that names a scene's start/end symbolically — but it's restricted to overlay layers only, because an ordinary in-scene layer never needed to reference a scene other than its own. at-addressing removes that restriction: any layer's animation can reference any scene's cue. A layer's own-lifecycle addressing ({rel, anchor: start|end}, relative to itself) is a different, orthogonal concept and is untouched by this.

Cue ids are scene-scoped; a reference says which scene

Opening cue references to every layer, not just overlays, is a bigger reference surface than exists today, and it gets more involved once the full cue system adds authored mid-scene cues: an overlay (a "floating scene" spanning several underlying scenes) sees every cue across the whole span it covers, so a reference needs to say which scene's cue it means. The answer is in the reference string, not in a global cue namespace:

Form	Example	Resolves to	Valid from
<cueId>	start	The named cue of the referencing layer's own scene.	In-scene layers only.
<sceneId>.<cueId>	scene-offer.start	The named cue of a specific scene, wherever the reference lives.	Anywhere.
[<n>].<cueId>	[0].end	The named cue of the nth scene under the overlay's own span, counting from the first scene the overlay overlaps.	Overlay layers only.

Cue ids live inside their scene (start, end, and later beat-2, price-reveal, …), so two scenes can both have a start and neither is renamed when scenes are reordered. The qualified form is the one that never moves: scene-offer.start means that scene's start whatever sits around it. The span-relative form is the one that should move: an overlay authored against [0].start and [1].end keeps meaning "the first scene under me starts, the second one ends" when the overlay is dragged to a different point in the timeline or extended over another scene, with no keyframe touched. Neither scene ids nor cue ids may contain ., [, or ] (validate.ts), so the string splits unambiguously, and a plain string stays the JSON shape rather than a nested object because it reads like a path and an author or a model can write it without a schema in front of them.

This also settles how animationAtSchema relates to animationStartSchema: at-addressing replaces the overlay-only {rel, anchor, sceneId} form outright under schemaVersion: 2 (it's the <sceneId>.start|end case, spelled longer), and coexists with the own-lifecycle {rel, anchor: start|end} form, which references the layer itself and isn't a cue at all.

Resolve-stage cue map: reuses sceneAnchorResolver()'s existing scene-boundary lookup, exposed as the resolved cue-id-to-frame-number map (Foundations → Pipeline & Architecture). interpolateAnimatedProps/continuousDriftAt need no changes at all — they only ever consume an already-resolved plain frame number, regardless of how it was addressed.

Validation: a dangling-cue-reference check, structurally simpler than pin's validation (Positioning) — cues never reference each other, so there's no cycle or depth-cap concern, only "does this cue id name a real scene."

The full cue system (later, more design uncertainty)

Moves off today's three fixed animation archetypes toward a general per-property keyframe timeline, plus a cue/retiming system for stretching a whole creative to a different voiceover length. The keyframe-timeline half of this work is documented under Keyframe Animation — this page covers only the cue/retiming half.

Inherits the addressing primitive above, doesn't build it fresh. The at-addressing shape (cueOffset/cueFraction) and the resolve-stage cue map already exist by this point, built against auto-derived scene-edge cues. What this later work actually adds on top: a real, author-facing cue entity (not just implicit scene edges), the hint/word-matching authoring flow, and the retiming/stretch-factor math that makes cues move non-linearly instead of staying fixed. Every at reference already authored against a scene-edge cue keeps resolving once this lands — that's the point of building the primitive early.

Where cue resolution lives in the pipeline: per Foundations' stage names (resolve→measure→solve→animate→place), cue resolution turns authored time references into concrete frame numbers — frame-invariant but not spatial — and needs to settle before anything spatial runs. That's resolve, which already does other frame-invariant precompute, not solve, which is specifically the frame-invariant spatial stage and doesn't otherwise deal with time.

The cue system, modelled directly on the aerender-engine's own retiming.jsx — the genuinely portable part is the pure math, not AE's own imperative plumbing (markers, timeRemap keyframes, AVLayer manipulation), none of which survives the port:

Component	Ported from	Role
getLayerRetimeRegions	aerender-engine retiming.jsx	Breaks a layer's timeline into protected versus stretchy spans.
calculateStretchFactor	aerender-engine retiming.jsx	Source-queue-to-target-queue stretch mapping, with safe and hard minimum ramping.
autoFindSplits	packages/retail-studio-schema/src/scene-splits.ts	Fuzzy word/phrase matching to find cue-hint candidates. Already generic and reusable — takes arbitrary target strings, not scene-specific ones.
confirmSplits / ConfirmRetailSplitsDto	apps/api-v1/src/retail-studio/voiceover/	The confirm-and-persist path. Currently hard-coded to scenes.length - 1 splits writing into scene durations; needs new, parallel plumbing for cues specifically.

Cues are authored against a template's reference timing (the reference voiceover it's built against), not purely word-boundary structural, since real reference ads already exist to author against. At campaign/render time, cues retime proportionally against the real per-campaign voiceover duration; the author can manually override a cue afterward, auto-resolve then human confirms, the same pattern already used for scene splits.

Manual cue overrides need their own mutation path for this delivery

There's no formal change-tracking/mutation model in this delivery's scope, so overrides need to slot into whatever document-mutation mechanism TemplateBuilder already uses today, not wait on a system that isn't being built alongside this.

The cue schema entity itself is a genuinely new thing, distinct from sceneFramesSchema's existing key/sub frames, which are presentation markers for thumbnail selection, not retiming anchors — don't conflate the two when this gets designed as real Zod. Expected to be reasonably straightforward once reached, since the aerender-engine gives a real reference for the desired behaviour even though the specifics of the port differ.

Risks

• This feature's later work (the full cue system) has real design uncertainty of its own, distinct from the well-grounded addressing-primitive work above: whether protected-versus-stretchy timeline regions are needed at all is explicitly unresolved, not just undesigned (AE's own retiming model forced that shape because AE's layer/timeline model left no other option; Retail Studio owns its own render pipeline and isn't bound by that constraint — which sounds like it should make this easier but actually removes the ready-made answer AE's constraints provided, needs designing from first principles), and the actual cue schema entity doesn't exist yet as real Zod. Treat this half's own estimate as less certain than the addressing primitive's.

Open questions

• Whether protected-versus-stretchy timeline regions are even needed at all — genuinely unresolved, not just undesigned (see Risks above).

Done when

• The addressing primitive is done when an ordinary in-scene layer's animation can address { mode: 'cueOffset', cue: 'scene-2.start', offsetFrames: -10 } and resolve correctly via the same underlying scene-boundary math sceneAnchorResolver already provides; the resolved cue map is available to solveScene/animateScene per Foundations → Pipeline & Architecture; a dangling cue reference fails validation with a friendly message, matching the reserved-mode precedent.
• The full cue system is done when the cue confirm/override flow round-trips through the same auto-resolve-then-human-confirms pattern scene splits already use (see Keyframe Animation → Done When for the retiming-survival criteria, which belongs to that feature since it's about keyframe tracks specifically).