How do you test whether a machine understands the saddest scene ever filmed?
At the end of All Quiet on the Western Front (1930), a soldier reaches from his trench towards a butterfly. A sniper fires. The hand goes still. There is no dialogue and almost no action, and yet the scene is one of the most reliably devastating ninety seconds in cinema. If a measurement system reads it as "a man is shot", the system has missed the point of the scene, and arguably the film. I am a co-founder at FishDog, and over the past weeks we calibrated Screening Room, our synthetic audience product, against twelve scenes like this one. This article sets out the method, the results, and the limitations, in that order.
What we claim, and what we do not
Two framing points before the method, stated plainly.
First, Screening Room's viewers are synthetic. A panel of constructed personas watches a video and produces a beat-by-beat emotional read: engagement, an emotion mix, and each viewer's own words at each moment. The results are synthetic and directional. They are designed to complement human research, not to replace it.
Second, "calibration" in this article means something specific and limited. We tested the system against scenes whose emotional readings are culturally documented: written about, taught, and agreed upon for decades. That is a form of ground truth, but it is not the same as validation against measured human panels, which is a separate programme we run and have not completed. When that work reaches its bar we will publish it. Until then, nothing in this article should be read as a claim of human equivalence.
The method: pre-registered readings, standardized measurement
The design has three rules.
The expected reading is fixed before the test. For each scene we wrote down, in advance, the emotional trajectory a competent viewer is documented to experience: for the butterfly ending, grief arriving through a small gesture; for the Odessa Steps, mounting horror organised by editing; for the skeleton finale of House on Haunted Hill, dread building to a jolt and releasing into delighted laughter. A test whose pass condition is written after the result is not a test.
The measurement is standardized. Every film is screened to the same panel of twelve synthetic viewers, with identical settings, through the production system that customers use. Eleven films use a 60 to 120 second cut centred on the famous scene; House on Haunted Hill was screened at full feature length (75 minutes), which is the same measurement applied to a longer runtime. Because the panel and settings never change, results are comparable across films, and any film can be re-run.
Everything is preserved and published. Raw model outputs are retained for every run, so any reading can be re-audited, and each film's result links to its full public report. A measurement tool that cannot show you its failures is not a measurement tool.
The materials: twelve famous scenes
We chose public-domain films (1902 to 1968) for two reasons: the scenes are rights-safe to publish, and their emotional readings have decades of documentation behind them. The expected readings below were written before any panel run executed.
Film | Year | The scene | Expected reading (pre-registered) |
|---|---|---|---|
A Trip to the Moon | 1902 | The capsule strikes the Man in the Moon's eye | Anticipation resolving into delighted comic surprise |
The Kid | 1921 | The child is taken; the rooftop chase | Distress and desperation; attachment under threat |
Nosferatu | 1922 | The shadow on the staircase; the sacrifice | Mounting dread; fatal resolve held to first light |
Safety Last! | 1923 | The clock hang | Vertiginous suspense sustained through comedy |
The Phantom of the Opera | 1925 | The unmasking | Curiosity punctured by shock and recoil |
Battleship Potemkin | 1925 | The Odessa Steps | Horror and helplessness, driven by editing |
The Gold Rush | 1925 | The boot dinner | Absurd comedy over genuine desperation |
The General | 1926 | The bridge collapse | Spectacle and shock; the cost made visible |
Steamboat Bill, Jr. | 1928 | The falling facade | Held breath; relief arriving as astonishment |
All Quiet on the Western Front | 1930 | The butterfly ending | Grief through a small gesture; futility |
House on Haunted Hill | 1959 | The skeleton finale | Dread to jolt to knowing laughter |
Night of the Living Dead | 1968 | The ending | Hope extinguished; grim injustice |
The instrument: what is actually measured
Screening Room's Emotional Cadence measurement produces, for each synthetic viewer, a per-beat record across the runtime: engagement, excitement, interest, boredom, a mix across eight named emotions, and a one-line reaction in the viewer's own words. The film-level system beneath it watches every scene at full attention, builds a structured memory of the film, and then verifies the most important claims by re-watching the relevant scenes before anything is reported.
Three disciplines in that pipeline matter for calibration, because each one exists as the result of a documented failure.
Endings must be re-watched, not inferred. In an early prototype, every synthetic viewer confidently described the protagonist of Night of the Living Dead being overwhelmed by the ghouls. That is not the ending; he survives the night and is shot by the posse at dawn. The system had fabricated narrative closure. The fix was structural: re-watch verification of a film's final scenes is mandatory, and the synthesis stage is prohibited from asserting fates the footage does not show.
Emotion must survive summarisation. When we probed the pipeline with the butterfly ending, the model saw the moment but the summarisation step lost it: the emotional peak survived perception and died in compression. The fix was a dedicated emotional channel in the film memory, kept separate from factual plot fields, in which every entry must cite the visible evidence for the feeling it claims. After the fix, the film memory carried the moment explicitly, including the butterfly and the stilled hand.
The model's self-reports are not trusted. In a full-film test, the synthesis stage fabricated emotional entries for two segments whose data had been rejected as malformed, and then reported perfect coverage statistics for itself. Its arithmetic was internally consistent, so schema checks could not catch it. The fix was server-side enforcement: every emotional entry is filtered against an authoritative record of which segments actually produced valid data, and coverage is computed by our code, not accepted from the model. In the re-test the model attempted the same fabrication again, and the system caught and removed it. We regard that re-test as the strongest single piece of evidence in this programme: the control was demonstrated against a live failure, not a simulated one.
One further discipline is worth stating because it initially looked like a failure. Asked to read Ellen's sacrifice in Nosferatu from a two-minute clip, the system reported dread and terror but declined to call it a willing sacrifice. On review, the refusal was correct: the sacrifice reading depends on an earlier scene outside the clip, and the system's evidence rules forbid asserting what it has not seen. With the full film in memory, it produced the sacrifice reading, grounded in the film's own setup. A system that guesses plausibly is precisely what we are trying not to build.
Results: twelve scenes, twelve public reports
We scored each scene on a fixed standard. A match means the dominant measured emotions at the famous beat correspond to the pre-registered reading and the trajectory moves in the documented direction. A partial match means the direction is right but the engagement peak landed at a different beat, or a named component of the expected reading failed to dominate. A miss means the panel's reading contradicts the documented one.
The panel's measured reading matched the pre-registered expected reading in nine of twelve scenes, partially matched in three, and contradicted it in none. The three partial matches are Steamboat Bill, Jr., Night of the Living Dead, and House on Haunted Hill, and each diverged in a specific, explainable way, set out on its card below and in the limitations section. The strongest single readings were Battleship Potemkin's Odessa Steps, which produced the highest single-beat engagement of the programme with dread and sadness dominant, and The General's bridge collapse, which produced its run's engagement peak and its surprise maximum in the same beat, seconds after viewers had been admiring how solidly the bridge was built.
Each card below pairs the pre-registered expected reading with what the panel actually measured, a strip of the panel's engagement across the scene with the peak marked, the dominant emotions at the beat, and one viewer's words at that moment. Every card links to its full public report.
A Trip to the Moon
1902 · The capsule strikes the Man in the Moon's eye
- Expected
- Anticipation resolving into delighted comic surprise
- Measured
- Amusement rose to 53% of the panel's emotion mix as the capsule struck, the film's amusement maximum. The image read as comic delight; the engagement peak came later, at the lunar landing.
See the full panel reportPutting a rocket right in the moon's eye - that's something.
Synthetic viewer, at the beat
The Kid
1921 · The child is taken; the rooftop chase
- Expected
- Distress and desperation; attachment under threat
- Measured
- Sadness surged to 60% of the emotion mix at the boy's outstretched arms, the run's engagement peak. Tension carried the chase on either side of it.
See the full panel reportLord, that look on his face, it just makes your gut twist.
Synthetic viewer, at the beat
Nosferatu
1922 · The shadow on the staircase; the sacrifice
- Expected
- Mounting dread; fatal resolve held to first light
- Measured
- Dread built through the bedroom scene to the run's engagement peak, at 45% of the mix, and stayed the dominant reading to the final frame.
See the full panel reportIt's just as I figured, that thing is right over him now.
Synthetic viewer, at the beat
Safety Last!
1923 · The clock hang
- Expected
- Vertiginous suspense sustained through comedy
- Measured
- Tension was the dominant reading in 84 of 110 beats and climbed all the way to the cut's final seconds at the clock face. Amusement kept surfacing through the climb.
See the full panel reportHe's still holding on, but I can't imagine his grip lasts much longer.
Synthetic viewer, at the beat
The Phantom of the Opera
1925 · The unmasking
- Expected
- Curiosity punctured by shock and recoil
- Measured
- Curiosity collapsed at the reveal and dread took over at 48% of the mix, holding through the recoil. The jolt registered as sustained dread rather than momentary surprise.
See the full panel reportWell, she still looks like she's seen a ghost, doesn't she?
Synthetic viewer, at the beat
Battleship Potemkin
1925 · The Odessa Steps
- Expected
- Horror and helplessness, driven by editing
- Measured
- Dread and sadness dominated from the opening seconds, and the sequence produced the highest single-beat engagement of all twelve films, 89.6 out of 100.
See the full panel reportThis is just awful, seeing that mother and child right there in the line of fire. It just makes my stomach clench.
Synthetic viewer, at the beat
The Gold Rush
1925 · The boot dinner
- Expected
- Absurd comedy over genuine desperation
- Measured
- Amusement was the most common dominant emotion across the run, while individual viewers read the desperation underneath as tension and dread at the same beats.
See the full panel reportWell, didn't think it was literally a shoe. That's a rough meal.
Synthetic viewer, at the beat
The General
1926 · The bridge collapse
- Expected
- Spectacle and shock; the cost made visible
- Measured
- The collapse produced the run's engagement peak and its surprise maximum in the same beat. Seconds earlier, viewers had been admiring how solidly the bridge was built.
See the full panel reportWell, that bridge didn't hold up forever after all, did it?
Synthetic viewer, at the beat
Steamboat Bill, Jr.
1928 · The falling facade
- Expected
- Held breath; relief arriving as astonishment
- Measured
- The facade fall itself read as curiosity with only a flicker of surprise in the wide shot. The panel's held-breath tension peak arrived about half a minute later, inside the wrecked house.
See the full panel reportWell, now something's really going on - what in the world is all that stuff flying around the yard?
Synthetic viewer, at the beat
All Quiet on the Western Front
1930 · The butterfly ending
- Expected
- Grief through a small gesture; futility
- Measured
- Sadness climbed to 56% of the mix at the run's engagement peak, as the film cuts from the stilled hand to the march of the fallen.
See the full panel reportThat's the cost. A whole lot of good men gone.
Synthetic viewer, at the beat
House on Haunted Hill
1959 · The skeleton finale (full film)
- Expected
- Dread to jolt to knowing laughter
- Measured
- At the puppeteer reveal the film's amusement maximum landed exactly where pre-registered, with all three sampled viewers reading the skeleton gag as comedy. The film's overall engagement peak came at 57:58, though, not at the skeleton's rise.
See the full panel reportWell, that's a funny way to spend your time, fiddling with a skeleton on strings.
Synthetic viewer, at the beat
Night of the Living Dead
1968 · The ending
- Expected
- Hope extinguished; grim injustice
- Measured
- Engagement peaked in the exact beat of the shot, with surprise entering the mix. The grief and injustice reading did not dominate: it depends on the whole night, which a 90-second cut cannot carry.
See the full panel reportThat was a mighty loud bang, and I suppose it means those guns weren't just for show anymore.
Synthetic viewer, at the beat
The full-length run: House on Haunted Hill (1959)
House on Haunted Hill was measured across its full 75-minute runtime: twelve viewers, 1,464 of 1,464 checkpoints, the same instrument the short cuts used. The pre-registered reading for the skeleton finale was dread building to a jolt and releasing into knowing laughter, with the engagement peak expected at the skeleton's rise. The finale's shape arrived as written: tension and dread carry the vat sequence, and the puppeteer reveal produced the film's amusement maximum, with all three sampled viewers reading the gag as comedy. The peak-location expectation was wrong, though: the film's overall engagement peak (88.8) came at 57:58, during the buildup in the darkened house, while the finale sequence carried a top-five peak (87.2). We scored it a partial match and publish it as measured: the full House on Haunted Hill report is public, beat by beat.
Limitations
These are the boundaries of what this calibration establishes, stated with the same precision as the results.
Synthetic, directional. The viewers are synthetic personas. Consistency with documented readings of famous scenes does not establish equivalence with human panels. That validation is a separate, ongoing programme with its own pre-registered bar, and it has not yet been met.
Documented consensus is not a measured baseline. Our expected readings are drawn from decades of critical documentation, which makes them unusually stable ground truth, but they are qualitative. A scene's documented reading tells you the direction and shape of the expected response, not its magnitude.
Panel size. Twelve viewers per film is the product's standard panel, chosen for comparability with real usage rather than statistical power. This calibration tests whether the measurement reads documented beats correctly, not whether twelve viewers predict a population.
Scene windows. Eleven of twelve tests use 60 to 120 second cuts. Emotional readings that depend on full-film context are only testable at full length, which is why the refusal case above matters and why one film was run at feature length. Night of the Living Dead's partial match is the same boundary measured: the grim-injustice reading of its ending depends on the whole night the 90-second cut cannot carry.
Audio. Eight of the twelve films are silent-era pictures whose source prints carry no audio stream, so their panels watched silent. For a silent-era film this is close to the historical viewing condition, but it is a measurement difference and we record it per film. The Phantom of the Opera's print carries a score and was screened with sound, as were All Quiet on the Western Front (the 1930 sound version), House on Haunted Hill, and Night of the Living Dead. A source without audio would have been rejected for All Quiet, by a rule fixed before the run.
Completion. Across the twelve reports, 13,761 of 13,764 checkpoints completed (99.98%). Nine reports carry every checkpoint. Three carry a single missing one-second beat for a single viewer: Battleship Potemkin at 1,439 of 1,440, The General at 1,079 of 1,080, and All Quiet on the Western Front at 959 of 960. Each was a one-off provider error that our retry logic did not classify as retryable. We accepted those runs with this disclosure rather than re-running them for cosmetic completeness, and the missed classifications are logged defects with fixes pending. A measurement run that only reports its clean completions is marketing.
Run recovery. Nine of the eleven new panel runs stalled once mid-run on the same queue defect and required a single operator-invoked requeue of unstarted work before completing. No completed data was altered by any recovery. The defect and its fix are tracked in our engineering record.
Modern viewers, old films. The panel is population-true to today's viewers, not to 1902's. Stretches of connective footage in the oldest films read as boredom (a third of A Trip to the Moon's beats), and Steamboat Bill, Jr.'s facade fall read as confusion in the distant wide shot rather than astonishment. We treat that as the instrument being honest about its panel rather than a defect of the panel.
A known open defect. In the final segment of Nosferatu, the system read Ellen's death, which the film conveys through silent-era dramatic convention rather than explicit statement, as a recovery. The perception layer asserted the wrong resolution instead of recording the fate as ambiguous. This is logged, published in our defect record, and the candidate fix is that uncertain character fates must be recorded as unresolved rather than resolved in either direction. We publish the defect because a calibration exercise that only reports its passes is marketing.
What this establishes
The segmented verdict, as plainly as I can put it.
If the question is "does Screening Room's emotional measurement read documented emotional beats consistently, reproducibly, and with published evidence", this calibration answers it: the panel's reading matched the pre-registered documented reading in nine of twelve scenes, partially matched with published divergences in the other three, and contradicted none. If the question is "does a synthetic panel match a human panel", this calibration does not answer it, and we have not claimed otherwise; that programme is running and will be published against its own pre-registered bar. And if the question is "should you trust a measurement system that cannot show you its failures", the answer is no, which is why the failures are documented in this article and all twelve reports are public.
The twelve reports are linked from their cards above, one per scene. The full-length example, House on Haunted Hill (1959) with twelve synthetic viewers across 75 minutes, is the place to start: House on Haunted Hill (1959) Screening Room report.