Evidence & Audit | Agentic SWMM

11documented evidence paths

4,067tests reported in v0.9.4

SHA256byte-identical solver output

11words in verified NL prompt

New in v0.9.4 · Kelowna, BC

A complete run, with the model attached

One sentence requested downtown Kelowna for 1–4 November 2023. After one approval, the runtime fetched the model from SWMMCanada, ran EPA SWMM 5.2.4, audited the artifacts, and wrote a Word report. The public case includes the model and the two-turn conversation.

From the published case	Recorded result
Storm system simulated	411 derived subcatchments · 889 nodes · 832 conduits · 76 outfalls
Peak at principal outfall OUT_N467	0.130 m³/s
System peak outflow across all outfalls	0.706 m³/s
Continuity error	−0.078% runoff · −1.022% routing
Artifact audit	Pass · 3 checks · 0 failures

Kelowna rainfall and outfall hydrograph at OUT_N467 during 1–4 November 2023 — Kelowna’s principal outfall peaks at 0.130 m³/s. The rainfall station recorded 6.1 mm over the requested days; SWMM applied 5.200 mm inside the simulation window, which ends at 00:00 on 4 November. This is an uncalibrated first pass.

What still needs checking: upstream QA flags 10 uphill conduits and 5 subcatchments whose outlet routing exceeds 50 m. No observed-flow series supports the model. The audit verifies the run’s artifacts and recorded checks; design use still needs geometry review and calibration.

Inspect the study-area map and its network counts

Downtown Kelowna study area showing derived subcatchments and both storm and sanitary networks over a LiDAR surface — The map shows the whole upstream bundle: 1,465 conduits and 88 outfalls across storm and sanitary networks. Only the storm system (832 conduits, 76 outfalls) is simulated. Subcatchments are derived using the upstream junction-and-street method, with medium confidence.

Full case & source artifacts ↗ Download the Word report Inspect the runnable model

Read the session transcript and the SWMM report excerpts behind the numbers.

Validation snapshot

The repository includes runnable benchmarks and research previews with different evidence boundaries. Each row states what the benchmark actually demonstrates and where its evidence stops — these boundaries are deliberate and verification-first.

Release evidence: the v0.9.4 changelog reports 4,067 tests and sessions across five Canadian cities. The published Kelowna and Victoria cases document workflow behaviour and its limits. These software checks do not establish catchment predictive accuracy.

Benchmark / Preview	What it shows	Evidence boundary
Information-loss-guided subcatchment partition	QGIS-to-Agentic SWMM preprocessing using entropy and fuzzy-similarity concepts from Zhang & Valeo's Journal of Hydrology paper	GIS preprocessing concept, not a calibrated SWMM performance claim
Raw GeoPackage-to-INP benchmark	Public TUFLOW GeoPackage layers converted into SWMM-ready artifacts, QA, and audit	Structured raw GIS path, not arbitrary CAD/GIS recognition
Prepared-input SWMM benchmark	External 40-subcatchment Tecnopolo model execution, plotting, and direct `swmm5` comparison. v0.7.1 re-verification: `model.out` SHA256 unchanged across the v0.7.0 → v0.7.1 minor revision, with an 11-word natural-language prompt now sufficient to drive the full run-audit-plot chain end-to-end.	Prepared INP validation path
Cross-environment byte-identical reproducibility	A natural-language prompt drives the `aiswmm` chain to the same byte-identical `model.out` as bare `swmm5`, across macOS and Docker	SWMM execution-layer reproducibility, not agentic workflow reproducibility
Cross-session memory autonomously activated	On a real production natural-language run, the LLM planner consulted prior session history without any user instruction to do so — see the v0.7.1 cross-session memory evidence	Historical v0.7.1 evidence that memory retrieval shapes a real run. It does not establish memory accuracy across projects; v0.9.4 adds failure-memory and parameter-record retrieval.
Prior Monte Carlo uncertainty smoke	Tecnopolo HORTON parameter perturbation and hydrograph envelope preview	Prior uncertainty smoke, not calibration
Optional INP-derived raw adapter benchmark	Raw-like inputs extracted from a public SWMM fixture and rebuilt through the modular path	Adapter handoff check, not greenfield watershed generation
SWMMCanada upstream adapter	Natural-language dispatch requests a real published municipal storm network through `fetch_swmm_from_canada` — 35 Canadian cities, or open-data synthesis anywhere else in Canada — unpacks the bundle into `00_raw/`, renders a study-area map, and retains the archive and service identifiers in the run artifacts	Upstream retrieval and provenance path; network quality remains bounded by the selected municipal source data
Case study — one prompt to a client deliverable	A single English sentence fetched the real downtown Victoria, BC storm network (423 subcatchments, 325 nodes, 307 conduits), ran SWMM, audited it, screened it against the design rulebook, plotted the hydrograph, and exported a Word report with figures embedded. Continuity: −0.098% runoff, 0.371% routing. Prompt, numbers, and every artifact	End-to-end workflow conduct on a real network; the model is an uncalibrated first pass, and the design review says so (1 pass, 2 fail, 4 warnings, 4 needs-data)
Case study — the run that refused to report results	The same Victoria AOI on a Windows 11 ARM laptop, over a week that turned out to be dry. The run reported a 6.111% routing continuity error and a flooding volume against zero rainfall, named both as disqualifying rather than presenting them as findings, and proposed the design storm that would settle it. Read the case	Evidence that the verification layer blocks a bad run; not a demonstration of a working model for that period
Kelowna — one sentence, one approval	Fetch, run, audit, and Word report on 411 subcatchments and 832 storm conduits. The committed INP supports offline solver reproduction; the case record above links its figures, transcript, report, and source artifacts.	Uncalibrated first pass. Numerical checks and an artifact audit pass coexist with uphill conduits and long outlet-routing warnings.

Downtown Victoria study area fetched from SWMMCanada: DEM hillshade, subcatchments, conduits and outfalls — Downtown Victoria study area, rendered from the SWMMCanada bundle before the first simulation — DEM hillshade, subcatchments, conduits, and outfalls. Every Canada run starts with its inputs and its geography on the record.

Network map of the downtown Victoria storm system fetched from SWMMCanada — The fetched storm system drawn from the model itself — 423 subcatchments, 325 nodes, and 307 conduits of the real downtown Victoria municipal network. What the agent ran is what the map shows.

Rainfall and runoff at node DOF007021 in the downtown Victoria storm network — Rainfall–runoff at the peak-inflow node of the real Victoria storm network, 1–4 November 2023 (50.0 mm total). Peak node inflow 0.033 m³/s at 05:00. An uncalibrated first pass on a real municipal network, not a calibrated performance claim.

Information-loss-guided subcatchment partition using entropy and fuzzy-similarity preprocessing — Information-loss-guided subcatchment partition — entropy and fuzzy-similarity preprocessing of QGIS layers into SWMM-ready subcatchments. A GIS preprocessing concept, not a calibrated SWMM performance claim.

Tecnopolo Monte Carlo hydrograph uncertainty envelope from HORTON parameter perturbation — Tecnopolo Monte Carlo hydrograph uncertainty envelope — HORTON parameter perturbation producing a flow envelope preview. A prior uncertainty smoke, not calibration.

Tecnopolo rainfall-runoff plot at node J22 for the January 1994 event — Tecnopolo rainfall-runoff — rainfall and runoff at node J22 for the prepared 40-subcatchment Tecnopolo model along the prepared INP validation path.

Audit and research memory

The audit layer consolidates artifacts, QA checks, and metric provenance into an Obsidian-compatible experiment note. This example catches a recorded peak-flow value that does not match the value re-parsed from the SWMM report source section.

Experiment audit comparison showing a peak-flow provenance mismatch — Experiment audit comparison — a peak-flow provenance mismatch surfaced when the recorded value is re-parsed against the SWMM report source section.

In v0.9.4, failure memory loads at session start and relevant parameter records reach the planner. Re-audits preserve the prior record, and audited runs are protected from in-place agent edits by default. Modelling-memory summaries can propose skill changes, which still require human review and benchmark verification.

Validation evidence Experiment audit framework