All case studies
Case study Medical device reporting

Cascadia Early Warning.

Which product codes deserve a closer look? Cascadia Early Warning forecasts monthly FDA device-report volume, compares it with what arrived, and applies a fixed review rule. The retrospective results show where the method holds up and why three of seven codes remain unsuitable for automated review.

The results Counts of reports, not rates
The measured results, each figure generated from the module's frozen tables
ResultMeaning
DSQ November 2026 outlook: about 736 reports, 80% range 376 to 906, three months aheadAt three months ahead, ranges like it held 20 of 22 locked-test months; a forecast, not yet scored
Candidate selected for 2 of 7 codesAdded model complexity earned use selectively on the development test
DSQ average miss about 246 versus 265 reportsThe 2024 to 2025 test showed a modest improvement over the trailing mean
Three codes below the 70% coverage floorThe demonstration disables their review rules rather than presenting all lanes as usable
One retained episode after retrospective filteringIllustrative queue output, with seven ungated episodes disclosed separately
None of 24 Class I recall initiations preceded by a queue episode startNo demonstrated recall-prediction result

Coverage by code

Chart description: coverage by code

  • Coverage: The share of 24 test months inside each model's 80% range.
  • Thresholds: The shaded band is 60% to 95%; review is disabled below 70%; both were specified before the test.
  • Models: Row labels identify the candidate or trailing mean used for each code.
The data: coverage by code
The data: 80% range coverage in the locked test by code, model in use (M-04)
CodeDeviceModel in use80% coverageMonthsRule enabled
NIKDefibrillator, Automatic Implantable Cardioverter, With Cardiac Resynchronization (Crt-D)trailing three-month mean100.0%24yes
LWSImplantable Cardioverter Defibrillator (Non-Crt)trailing three-month mean95.8%24yes
DSQVentricular (Assist) BypassETS candidate91.7%24yes
DSPSystem, Balloon, Intra-Aortic And Controltrailing three-month mean70.8%24yes
NPTAortic Valve, Prosthesis, Percutaneously DeliveredETS candidate58.3%24no
OZDTemporary Non-Roller Type Left Heart Support Blood Pumptrailing three-month mean41.7%24no
PYXTemporary Non-Roller Type Right Heart Support Blood Pumptrailing three-month mean33.3%24no

01 / The review decision

Which product codes deserve a closer look this month

The decision is a quality team's monthly choice of where to look first: which product codes are reporting as expected and which are not. This build's contribution is testing report-volume forecasts and a repeatable review rule on public data: a forecast of each code's reports with a stated range, a held-out test of whether ranges like it held, and a fixed rule that turns departures into a review queue.

  • The cohort. Seven cardiovascular product codes spanning Classes II and III, read from openFDA's device-event endpoint: 649,080 distinct reports, 649,079 after a private firm-list exclusion removed 1, counted per code per receipt month over 896 code-months (a report filed under two codes counts once in each).
  • Summary reports. NPT is on FDA's list of product codes eligible for voluntary malfunction summary reporting, and DSP, DSQ, LWS, NIK, OZD and PYX are not. 4,179 of NPT's 72,845 reports are summary reports standing for 149,569 events, so its count can move with reporting practice and is not a count of events.
  • Counts, not rates. Report counts are not incidence rates, event rates or measures of device safety, as FDA's own documentation says and the module page quotes. No manufacturer or brand is named, by decision.

02 / The measured results

What the test found

  • The outlook. For DSQ (Ventricular (Assist) Bypass), about 736 reports in November 2026, three months ahead, with an 80% range of 376 to 906; at that horizon ranges like it held 20 of 22 locked-test months. September 2026, one month ahead, is an elapsed-period estimate and is not yet in the source.
  • The test. Rolling forecasts were fitted only through each origin and evaluated using the October 2026 snapshot. Model selection used target months through December 2023; its decision was recorded in October 2026 before the retained test results were generated. Historical public-data vintages were not reconstructed.
  • The experiment history. Model selection stayed fixed. A calibration-history defect was corrected after the first test run; point forecasts were unchanged, and the corrected interval results are the ones reported.
  • What was fixed in advance. The registered model form, the promotion rule and the review thresholds were committed before the first forecast row existed. The data snapshot was retrieved in the same session, so the rule was registered alongside the data rather than before it.
  • The locked test for DSQ. At one month ahead, the candidate's 80% range held 22 of 24 months, and its average miss was 246 reports a month against the trailing mean's 265.
  • Did the ranges hold? At one month ahead, across the seven codes, they held between 33% and 100% of locked-test months: two inside the band written before the test, two above it, and three below the 70% floor, where the review rule is switched off.
  • The review rule. A month is flagged when the observed count exceeds the 80% upper bound and is at least five reports above the point forecast. Two consecutive flagged months open an episode. As a retrospective, filtered demonstration it opened one episode where it is on, and would open seven if it ran in every code; each code's rule was switched on from its coverage in the 2024-01 to 2025-12 locked test, the first 24 of the 32 months it is then applied to.
  • Recall context. Of the 24 Class I recall initiations in the evaluated span, thirteen in codes whose rule is off, 0 were preceded by the start of a queue episode within the eighteen months before; the rule that flags every month would have preceded 23. Association only: no recall-prediction result is claimed.

03 / Decisions and trade-offs

The choices that shaped the result, and what each one cost

Claude Code implemented the approved plan, and AI agents ran the simulated chart reviews. Each choice below is recorded in the decision record, most with its counterfactual.

  • Counting reports, not events or rates. The target is distinct reports per code per receipt month, the only number here that can be certified; it says nothing about events, which summary reports and missing event dates blur, or about rates, which need a denominator the source does not carry (D1).
  • A fixed candidate and a promotion bar. One ETS form was fixed before any test and replaced the better of two simple baselines only where its development-period error was at least 10% lower without a worse interval score; it earned use on two of seven codes, and the trailing three-month mean runs the rest (D6, D8).
  • Ranges from the model's own errors. Each 80% range comes from the 36 latest errors of the same model and horizon, not from the model's assumptions; it is honest about past misses and slow to follow a series that changes level, which is how three codes' ranges fell short (D7, D16).
  • Keeping the band where it was registered. When five of seven codes landed outside the 60% to 95% band written before the test, neither the band nor the ranges were moved; the three codes below 70% stay on the page with their review rule off (D16).
  • A two-month trigger with a five-report floor. A month is flagged only above its 80% range and at least five reports over the point, and an episode needs two flagged months in a row, so a single receipt pile-up does not open one; the cost is that an episode opens a month after its first flagged month at the earliest, and a one-month departure opens none (D9).
  • Leading with three months ahead. The page leads with November 2026 at three months ahead because September 2026, one month ahead, had already ended when the forecast was issued; promotion was judged at one month ahead, and at three months ranges like November's held 20 of 22 locked-test months (D10, D19).
  • A private exclusion with a public receipt. Reports whose manufacturer or brand fields match a private list are removed whole, 1 of 649,080 here, and the count and the list's hash are published; a reader can check the receipt but not the list (D3).

04 / How the evidence was checked

What was checked, and what was not

  • Two paths. A separately written path, DuckDB SQL over the staged FDA pages plus its own Python arithmetic, recomputes the monthly and lag-matched counts, the distinct report totals, every forecast point and range, the scores, the review episodes, and the queue with and without its coverage gate as this page publishes it, and they must agree before anything is published. It does not refit the ETS model: candidate points are recomputed from the model's exported states, and ranges and scores from the published points. The recall timeline is checked only against fourteen independently verified Class I events, and the cohort gate, the exclusion receipt, the reports without an event date, the promotion decision, the pre-registered recall count, the summary-report composition, the remedial-recall counts and the outlook's twenty dots are carried from the build without a second derivation.
  • Checks that can fail. The domain gate runs 25 checks and re-runs them against deliberately corrupted copies of their input in 55 scenarios; the validation report records that every one tripped. Golden fixtures, computed outside both engines and committed before either existed, pass on both paths.
  • The chart review. Four AI agents reviewed rendered charts using simulated reviewer personas. They were instructed to use only the supplied images; reviewer isolation was not independently verified. Their findings exposed a concrete ambiguity: disabled review lanes still appeared to contain active episodes.

Links

  • Method and receipts: how the snapshot was frozen and hashed, how the second path is built, and how the model's states are exported for it.
  • Build repository: the pre-registration, the frozen harness, both derivation paths, the validation report and the chart review, each with its commit.

05 / The next test

What a quality manager does with a flag, and what is tested next

The intended workflow, for a quality manager whose code is flagged:

  1. Open the underlying reports for the flagged months and read their reporting context: the report types, the filers, and any batch or summary filing.
  2. Record a review disposition, including when nothing is found.
  3. Do not read an episode as a device-safety conclusion. It is a departure from expected reporting volume, and nothing more.

The live edge. The live edge has run 3 times; the last run, at 2026-10-13 23:38 UTC, reported status ok. It stores source vintages and scores a forecast when the target month first appears in the source. Live months are counted on a different basis from the frozen ones: raw count by receipt date as seen at the vintage, with no exclusion applied.

What is tested next. Two separate steps. The shuffled-dates reference tests whether the observed timing of queue episodes before Class I recall initiations differs from a chance reference, the same count recomputed over many random shufflings of the dates; until it runs, the pre-registered recall count means association only, and even then it would test timing, not prediction. Prospective scoring is a separate next step: each frozen forecast is scored as its month first appears in the source, beginning with September 2026 and November 2026.

Tech stack

PythonDuckDBpandasstatsmodelsopenFDAApache EChartsTailwind CSSPlaywrightStatic HTML / JSGit
Open build

The evidence is on the module page.

Every figure here comes from the same build as the module page, which sets out the forecast, the test, the review queue and the checks with their charts and tables.