ℹ️ TL;DR
The technical companion to part one. If your web app has a Playwright end-to-end suite, you already own most of a training-video studio. This post walks through what it takes to turn one into the other: a small helper vocabulary, an injected overlay, honest fixtures, and a build pipeline, with the lessons I picked up at each step.

Part one told the story; this post is the how. The pitch, one more time for the people who came straight here: I generate silent, captioned training videos for two web applications by driving them with browser automation, and when an interface changes I rebuild the affected videos with one command. If you want to see the output first, the two libraries live at locallygrown.net/docs/videos and refreshingspaces.life/help/videos. Everything below is what you’d need to know to do the same for your own project.

One insight starts it

Playwright can record video of everything that happens in a browser context. Turn that option on, and every test you already have becomes a screen recording. That’s the whole foundation, and it’s why this took an evening rather than a month: an end-to-end test and a walkthrough video are the same sequence of navigations and interactions. A test asserts at each step. A walkthrough narrates at each step. Everything else is presentation.

So the first thing to build isn’t video infrastructure at all. It’s a flow: a name, a viewport, a starting URL, and an async function that performs the steps. Flows are data, which means the generator can run one, several, or all of them, and your video library has a single source of truth you can read like a table of contents.

export type Flow = {
	title: string;
	description: string;
	/** Per-role storageState path (playwright/.auth/<role>.json) — reused from
	 * the e2e global-setup so recordings start already signed in. Omit for an
	 * anonymous (signed-out) walkthrough. */
	storageState?: string;
	/** Routes to pre-compile on an unrecorded page before recording, so the
	 * cold vite dev-server compile of the first page isn't recorded as dead
	 * air. List every route the flow navigates to. */
	warmup?: string[];
	/** Optional fixture work on an unrecorded page (full speed, no captions). */
	prepare?: (page: Page) => Promise<void>;
	/** Recording viewport. 'desktop' uses 1280×800 + a desktop UA + centered,
	 * larger captions; omit (or 'mobile') for the default phone size. */
	device?: 'mobile' | 'desktop';
	record: (page: Page) => Promise<void>;
};

A flow itself reads like stage directions:

'01-place-an-order': {
	title: 'Placing your first order',
	description: 'Browse your market, add fresh items to your basket, and reserve them for pickup.',
	username: 'e2e-customer', // Casey Customer
	warmup: ['/market', '/cart/review'],
	async record(page) {
		await goto(page, '/market');
		await say(page, "Welcome to your market — let's place an order.");

		// Skim down to the fresh items, the way you would on your phone.
		await scrollBy(page, 620);
		await say(page, "Scroll through what's fresh from your local growers.");
		await scrollBy(page, 380);

		// First item — tap() smooth-scrolls it to center before the finger
		// lands. Anchor the cart write so the basket persists before we leave.
		const tomatoes = productCard(page, 'Heirloom Tomatoes');
		await say(page, 'Found something you like? Tap Add to Order.');
		let flushed = page.waitForResponse(
			(r) => r.url().includes('/api/cart/batch-update') && r.request().method() === 'POST'
		);
		await tap(page, tomatoes.getByRole('button', { name: 'Add to Order' }));
		await flushed;
		await say(page, "It's reserved — add as much as you like, from any grower.");

		// … eggs from a second grower, the basket review, and Reserve Now
		// follow the same shape …
	},
},

One configuration rule I’ll hand you up front, because I learned it by squinting at blurry output: the recording size must exactly equal the viewport size. The comment in the harness says it with the authority of experience:

const context = await browser.newContext({
	...contextOpts,
	// recordVideo.size MUST equal the viewport 1:1 — a larger canvas
	// letterboxes the page into a corner instead of scaling up.
	recordVideo: { dir: OUT_DIR, size: viewport },
});

A vocabulary of four helpers

Nearly everything the flows do goes through four helpers, and building these well is most of the work.

The navigation helper goes to a URL and waits for the app to hydrate rather than merely load. A SvelteKit page is visible before it’s interactive, and early on I had recordings that clicked during that gap: the video shows a confident tap and then nothing, which reads as broken software. Waiting for hydration is the difference. Mine watches for an attribute the app sets when it’s ready:

/** Navigate and wait for client hydration (the app sets body[data-hydrated]). */
export async function goto(page: Page, path: string) {
	await page.goto(`${BASE}${path}`);
	await page.locator('body[data-hydrated]').waitFor({ timeout: 30_000 });
}

The caption helper puts a sentence on screen and holds it long enough to read. My first version used a flat delay, and watching those drafts taught me that a flat delay is wrong in both directions: short captions overstay and long captions flash past. Hold time now scales with word count, and each caption is also logged with a timestamp for the narration cue sheet:

/** Show a caption and hold it on screen long enough to comfortably read
 * before anything moves. Hold time scales with length (a flat delay left
 * longer lines flashing past); pass readMs to override. */
export async function say(page: Page, text: string, readMs?: number) {
	const words = text.trim().split(/\s+/).length;
	const hold = readMs ?? Math.max(2800, 1000 + words * 360);
	narration.push({ atMs: Date.now() - recordingStart, text });
	await page.evaluate((t) => {
		const el = document.getElementById('tv-caption');
		if (el) {
			// Reset to the bottom each caption — a previous tap may have dodged
			// it to the top; the next caption should read from its home spot.
			el.classList.remove('top');
			el.textContent = t;
			el.classList.add('show');
		}
	}, text);
	await pause(page, hold);
}

The tap helper is where the humanity lives. It smooth-scrolls the target toward the center of the viewport with an eased animation, then glides a synthetic finger to it over two dozen interpolated steps, pulses, and clicks. The glide exists because my first recordings jumped from place to place, the way automation naturally moves, and watching them I had the same reaction every time: it feels like a machine. Easing fixed it, and I never had that reaction again. Nobody else saw a single video until I was happy with them, which is its own small lesson — you are the first audience for your generated material, and if it makes you wince, that’s a signal. Sending your material out into the world sight unseen guarantees someone else will be wincing on your behalf.

export async function tap(page: Page, locator: Locator) {
	// Bring the target into view within whatever scrolls it — a modal's own
	// overflow container, not just the window (smoothCenter only scrolls the
	// window). Without this, a button below the fold of a tall modal is never
	// reachable and the tap lands nowhere.
	await locator.scrollIntoViewIfNeeded().catch(() => {});
	// Smooth-scroll the target toward center (a human glide), then read its
	// post-scroll box so the finger lands accurately.
	await smoothCenter(page, locator);
	const box = await locator.boundingBox();
	if (!box) throw new Error('tap target not visible');
	const viewportHeight = page.viewportSize()?.height ?? VIEWPORT.height;
	const nearBottom = box.y + box.height / 2 > viewportHeight - 150;
	await page.evaluate(
		(toTop) => document.getElementById('tv-caption')?.classList.toggle('top', toTop),
		nearBottom
	);
	await page.mouse.move(box.x + box.width / 2, box.y + box.height / 2, { steps: 24 });
	await pause(page, 260);
	await page.mouse.down();
	await pause(page, 90);
	await page.mouse.up();
}

The typing helper selects a field’s contents before it types, character by character at human speed. The select-first rule arrived by way of a video that proudly displayed “36.5012.00” after typing a new amount into a prefilled field. Your users clear fields without thinking about it; your automation won’t unless you tell it to.

/** Tap a field, then type at a human pace — select-all first to overwrite any
 * prefilled value. */
export async function type(page: Page, locator: Locator, text: string) {
	await tap(page, locator);
	await page.keyboard.press('ControlOrMeta+a');
	await page.keyboard.type(text, { delay: 45 });
}

An overlay that isn’t part of your app

The caption bar and the finger are injected DOM: a hundred-odd lines of markup, CSS, and script added to every recorded page at load time, riding on top of the untouched application. This matters more than it might seem. Your app needs zero knowledge that it’s being filmed, which means no test-mode conditionals bleeding into production code, and the overlay works identically across every page and, it turned out, across two entirely different applications.

The finger is a ring that shadows the mouse and pulses on every press; the caption bar is a fixed element the say() helper talks to:

#tv-cursor { position: fixed; z-index: 2147483646; width: 28px; height: 28px;
	border-radius: 50%; background: rgba(58,107,53,0.30);
	border: 2px solid rgba(58,107,53,0.85); pointer-events: none;
	transform: translate(-50%,-50%); left: -100px; top: -100px; }
#tv-cursor.tap { animation: tv-tap 0.4s ease-out; }
@keyframes tv-tap { 0% { transform: translate(-50%,-50%) scale(1); }
	40% { transform: translate(-50%,-50%) scale(1.7); }
	100% { transform: translate(-50%,-50%) scale(1); } }

addEventListener('mousemove', (e) => {
	cursor.style.left = e.clientX + 'px';
	cursor.style.top = e.clientY + 'px';
}, true);
addEventListener('mousedown', () => {
	cursor.classList.remove('tap');
	void cursor.offsetWidth;
	cursor.classList.add('tap');
}, true);

The overlay taught me a fun small lesson on adaptability. The caption bar sits at the bottom of the viewport, which is exactly where mobile interfaces put their primary buttons, and several early recordings tapped buttons hidden behind the caption. Now, when a tap target sits in the bottom portion of the screen, the caption hops to the top for that beat — you can see it in tap() above, in the nearBottom check. It’s four lines of logic, and it’s the difference between a video you trust and a video that covers the control it’s teaching.

Fixtures are writing

The flows run against the same seeded fixtures your test suite already uses, and sign-in happens through the same stored authentication state, which is why every video opens already signed in instead of spending ten seconds on a login form.

But video asks more of fixtures than tests do, and this is the part I’d underline for anyone building one of these. A test only needs data that’s valid. A video needs data that’s believable, because the camera lingers on it. LocallyGrown’s “meet your growers” walkthrough is only as good as the fictional farm it features, and my seeded farm turned out to have a one-line bio, an empty photo frame, and a “uses synthetic chemicals” badge on a market that bans them. I spent an evening hour writing great fixtures that show off the system and that farm properly, such as a three-paragraph story and a coherent certification and a photo, and it changed how real every frame feels. Budget time for fixture writing. It’s writing. That farm was the first of many; the fixtures kept growing until the demo data told the story of a whole small community.

Two disciplines go with it. Before recording anything for sharing, verify that every table your camera visits holds only fictional data, and make that a check the system runs rather than a thing you remember, because development databases accumulate real imports over time and my own had one. And seed richly enough to unlock every feature you’re teaching, since plenty of interface doesn’t render until settings exist: a home-base address for the one-tap mileage button, payment handles for the tap-to-pay buttons. I use a separate test database that gets torn down and rebuilt from scratch every run, but if you use a single database shared between development and testing, this can become a thorny problem.

One more fixture-adjacent choice: each recording also emits a timecoded narration cue sheet, caption text mapped to timestamps. Mine were meant for a learning exercise in AI voiceover that I ended up not wanting to get into just yet, but they cost nothing to keep producing, and they’d make captions translatable or narratable later without touching a single flow.

From recording to serving

Playwright records webm; the web wants mp4. The generator transcodes with ffmpeg as its last step, and the files are all gitignored, because they’re build artifacts. The flow catalog is the source; the videos are what it compiles to today.

// webm → mp4 (yuv420p for Safari/iOS, +faststart so playback starts before
// the file finishes downloading). Best-effort: keep the webm regardless.
execFileSync(
	'ffmpeg',
	['-y', '-i', webm, '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-movflags', '+faststart', mp4],
	{ stdio: 'pipe' }
);

Serving them had two lessons waiting, both one-line fixes. My ignore entry read training/, which also matched src/routes/training/, the route that serves the videos, so production returned 404 for every video while my laptop happily played them all. One day I’ll have gitignore syntax correctly stored in my brain. The anchored pattern /training/ fixed it, and the commit title is the honest summary: “Ship the /training/[file] route the gitignore swallowed.” Then Cloudflare, which fronts the app, kept serving old copies after regeneration, because the route sends a one-day cache header and re-uploading a file changes nothing an edge cache cares about. Versioned URLs fixed that one: every source renders as /training/{video.file}?v={VIDEO_VERSION}, and the version bumps after each upload. If mutable files live behind an edge cache, the URL has to change when the bytes do.

Regeneration reviews, by the way, don’t require watching every video end to end. For a change confined to one screen, I can extract a single frame at the timestamp where that screen is on camera, and that answers the only question that matters.

Scaling from one user to three audiences

The prototype was a single eleven-hundred-line file, phone-only, built in an evening for an app with one user. Porting it to LocallyGrown the next night is what forced real structure: the flow catalog, harness, overlay, and orchestration each became a module, and flows gained a device dimension, because shoppers use phones while market managers use laptops, so most flows record twice at different viewports.

Desktop mode had a subtle surprise for me. The app needed to know it was rendering at desktop size before first paint, and the obvious approach, setting an attribute on the root element from an init script, silently did nothing. The comment in the harness explains why, better than I could from memory:

// Let the overlay adapt captions (centered + larger) for desktop recordings.
// Set a window global rather than an attribute: init scripts run before the
// HTML is parsed, and the parser replaces the initial <html> element, so an
// attribute set here is silently discarded. The overlay installer runs at
// DOMContentLoaded and stamps data-tv-device from this global.
if (desktop) {
	await page.addInitScript(() => {
		(window as unknown as { __tvDevice?: string }).__tvDevice = 'desktop';
	});
}

The two-device pipeline also taught me the project’s most transferable operational lesson. My first version recorded the phone pass, reseeded the database, and reused the still-running dev server for the desktop pass, and that server held stale module state and stale row IDs. Sometimes the run crashed; sometimes it completed and produced videos of stale data, which is quieter and therefore worse. Each device pass now gets its own full server lifecycle: start, seed, record, stop. And the fix got better on review — I always have AI coding assistants do several review passes over their own work, and it caught that the stop step returned silently when the port never freed, letting a wedged server survive into the next pass. It now escalates to a hard kill and throws if the port is still listening. A cleanup step that always reports success is a specific example of a surprisingly common gotcha: never trust anything that always reports success.

A walkthrough I couldn’t record

I’ll close with the one that got away, because it’s always nice to have a puzzle I can come back to.

The LocallyGrown walkthrough for adding a photo to a product produced a video in which the photo never appears. The flow ran cleanly, the form saved, and the recording shows a save with no photo, because that’s what actually happened.

Probing it layer by layer: Playwright’s setInputFiles does exactly what it documents. After the call, the input’s files property contains the file, and a native change event fires. The application’s handler simply never runs. The app is Svelte 5, which delegates common events to a single root listener rather than attaching handlers per element, and that delegation doesn’t invoke the component’s onchange for this synthetic selection. App state never updates, the preview never renders, and the form honestly submits without a photo.

This is a known class of behavior, not a novel bug: sveltejs/svelte issue 12147 documents the same shape, where programmatically dispatched events with native names reach handlers under the legacy on:change syntax and stop reaching them under the delegated onchange syntax. A real person operating a real file picker generates a trusted, browser-initiated event, which delegation handles normally, so no actual grower has ever hit this. The failure exists only for synthetic events. A recording rig is made of synthetic events.

Every workaround I considered was a form of lying. Dispatch a hand-rolled event, call the handler directly, patch the state from outside: each produces a video of something the application doesn’t do. So the flow is gone from the catalog and the written photo guide remains. The commit message stands as the summary of record: “A recorded demo can’t show a real photo upload… Real growers using a real file picker are unaffected — there’s just no way to film it truthfully.”

I find I like what this says about the whole approach. The system’s entire value is that it refuses to fake anything: real app, real fixtures, real interactions, and a recording of whatever truly happens. The same property that makes the videos trustworthy is the one that made this flow unrecordable, for now. If you build one of these, you’ll eventually meet your own version of this walkthrough, and I’d encourage you to let it win.

What two evenings bought

The part I keep marveling at is how little of this system I actually had to build. Playwright already knew how to record video. The e2e suite already knew how to seed a believable market and sign in as anyone. ffmpeg already knew how to make Safari happy. The new code is a thin layer of glue and taste: four helpers, an overlay, and a catalog of flows that read like stage directions. That’s why the first version took an evening and the port took another one, and it’s a low-risk reason to try this yourself if you have a test suite sitting there: the expensive parts are already on your machine.

What it bought is out of proportion to that effort. Both applications are better documented than anything I’ve shipped in twenty-plus years of building software, and more approachable too. A stranger landing on LocallyGrown can watch themselves place an order before they’ve read a word, and Carol can answer “how do I do that again?” in under a minute, standing in a client’s kitchen.

And my favorite part of the whole project was the least automated one. Writing the fixtures grew into writing a world: farmers bringing what they grow to a common market, neighbors filling baskets, recipes traveling from one kitchen to another, meals made for family and friends. What started as believable data in a database became an aspirational story about a community, and every video is a little window into it. The machine records what it sees, and it fell to me to make what it sees worth looking at. I could happily do more of that.