End-to-end testing of the AST engine

[MoezLabiadh]

End-to-end testing of the AST engine

Closing issue #160

Runs the full pipeline: AOI → registry → adapter → operator → results → spatial output - against
real registries over live BCGW, and reports timing.

Full per-dataset results (summary and feature sheets for every run) are shared in the project folder
in W drive. This note contains only the high-level numbers.

---

What this branch adds

scripts/orchestrator_smoke.py. Runs the orchestrator over one or more real registry YAMLs for
one AOI and reports how long everything took. It prints a run summary and writes a spreadsheet with
one row per analysis.

• The AOI is either a file (shapefile, KML, GeoJSON…) or a Crown tenure parcel looked up in Tantalis
  by file number / disposition / parcel.
• Saving the spatial (GeoPackages) is optional (--spatial-out), so a run can be timed with and
  without it.
• Credentials come from environment variables only.

Fixed in this branch

Those minor issues were detected and fixed while working on the e2e testing:

One bad definition query no longer stops a whole registry build. A definition query the parser
cannot read used to raise while the dataset was being validated and abort the entire spreadsheet.
The dataset is now logged by name, counted, and skipped, so the rest of the build continues. the
same rule the orchestrator already follows when it maps registries to tasks.

Two registries no longer overwrite each other's spatial output. The saved features were written
to <folder>/<operator>/<dataset name>.gpkg, Registries can carry the same dataset name twice: both
wrote to one file, the second silently replaced the first. The path is now
<folder>/<registry>/<operator>/<dataset name>.gpkg.

This last fix is to the spatial-output code from the results-publisher (#163), so the commit was
made on that branch rather than this one.

---

Results

Three regions, each run twice: once with the spatial output saved, once without.

Tested against the demanding regions. Northeast and Cariboo carry the highest dataset counts and
the heaviest file-based datasets. Also added West Coast which sits in the mid-range in terms of the
size of file datasets.

AOIs used in the e2e tests were based on larger-than-typical Crown tenure parcels, roughly 10-20 ha
(top 20% of crown parcel sizes).

	West Coast	Northeast	Cariboo
Analyses per run	299	357	292
Run time (spatial export OFF)	3.8 min	6.6 min	4.3 min
Run time (spatial export ON)	3.6 min	5.9 min	3.7 min
Mean per dataset	0.73 s	0.99 s	0.76 s
Median per dataset	0.37 s	0.30 s	0.33 s
95th percentile	1.5 s	2.9 s	1.9 s
Peak memory	185 MB	207 MB	185 MB
Spatial files written	55 (18.5 MB)	90 (37.6 MB)	89 (45.9 MB)
Time spent writing them	1.4 s (0.6%)	1.9 s (0.5%)	2.1 s (1.0%)

The run with the spatial output on finished faster than the one without, in all three regions -
which looks wrong, since it does more work. It is not a measurement error: running the same job
multiple times varies by 8 to 40 seconds for reasons outside the tool, mainly the general load
on BCGW, the CPU load on the execution machine and bandwidth for fetching datasets from netwrok
drive.

---

What the numbers say

A few datasets dominate; most are nearly free. The mean is two to three times the median in
every region. Most datasets finish in under half a second, while single datasets take 20 to 70
seconds. This is important for the potential parallelization: work should be handed out as workers
become free rather than split evenly across them: an even split leaves most workers idle while one
finishes the slow dataset.

BCGW datasets are much faster than file datasets - about 4x faster each. Across all three
regions:

	BCGW (Oracle)	Local / network files
Datasets read	830	118
Median per dataset	0.33 s	1.34 s
Slowest one	25.6 s	68.3 s

This is the SDO push-down doing its job. For a BCGW dataset the spatial filter runs inside the
database against a spatial index, and only the matching features travel back. A file dataset has to
be opened over a network share and read by the client, and it only avoids a full scan if the file
has a usable spatial index of its own.

But BCGW still dominates the clock, because there are so many more of them.

Nearly all of the time is fetching the data, not analysing it. The read was timed separately
from the operator work. Measured on the runs without the spatial output, so nothing else is mixed
in:

	West Coast	Northeast	Cariboo
Fetching the data	225.9 s	395.9 s	254.9 s
Overlay / proximity / adjacency calculations	0.5 s	0.5 s	0.6 s
Share spent fetching	99.8%	99.9%	99.8%

Half a second of actual analysis in a six-minute run, and it holds for BCGW and file datasets alike.
Anything we do to make runs faster has to target getting the data - there is nothing to win in
the geometry/spatial work.

What this means for parallelization. : Heavily depends on the format:

• File datasets are the straightforward win. They are few, individually slow, and fully
  independent, so several can run at once with no change to how the engine is built: in Northeast
  that is over half the run time coming from a fifth of the datasets.
• BCGW datasets could be parallelized, but the benefit is limited. The current implementation
  shares a single database connection and cursor across the entire run, and a cursor can execute
  only one query at a time. Supporting concurrent queries would require a pool of database
  connections, which would change the existing design where the caller creates and passes in a
  single connection. While adding connection pooling should be feasible, the expected performance
  gains are relatively small, as most BCGW queries already complete in less than half a second.

The practical order then is to parallelise the file reads first, since it needs no change to the
current engine's contract, and treat connection pooling for BCGW as a separate decision.

Saving the spatial output is cheap enough to leave on by default. Under one percent of run time

• few seconds to write 55 to 90 GeoPackages.

Memory is not a constraint, and that matters for the API. Peak memory was measured in the e2e
test. A run holds one dataset's features at a time and drops them as soon as they are saved, so
memory stays flat no matter how many datasets are in the registry: 185 to 207 MB across runs
covering 292 to 357 datasets. For deployment that means a job can be sized at roughly 250-500 MB
rather than scaled to the size of the registry, and the API can plan concurrency per job. The one
thing to watch is that parallelising inside a job holds several datasets' features at once, so
memory then grows with the number of workers.

[wburt]

I am questioning if we want to complete the registration process if hydration of a dataset fails.

[MoezLabiadh]

You're right, and it has already happened. cariboo.yaml was built with 62 datasets from a 64row spreadsheet (2 datasets had broken paths), and nothing in the yaml says they are missing. I only spotted it by chance.

I copied the skip-and-continue rule from the orchestrator, but that's safe there because a failed dataset still shows up in the results with its error. At build time it just quietly comes out short, and everything after that trusts the registry.

Suggestion: keep collecting all the failures in one pass but don't write a registry that looks complete when it isn't: either stop the build at the end and list what was skipped, or record the skipped datasets in the yaml. Worth noting this applies to the enrichment too, not just the hydration one I added

[wburt]

I agree

[wburt]

Looks great! the non-spatial time difference would be something to look into later on. Probably make a bigger difference when results are prepped for mapping and stored in s3. The times look good will be interesting to see if Oracle caching these queries is effecting our timing results. Regardless <10 minutes seems positive.