[Feature]: Lambda delay table geneator #4

Activity

1. SomethingNew71

   added

   enhancementNew feature or requestNew feature or request

   on Dec 26, 2025
2. SomethingNew71 commented on Jul 16, 2026

   

   SomethingNew71

   on Jul 16, 2026

   CollaboratorAuthor

   More actions

   A design doc covering this feature is now up: docs/plans/2026-07-16-tuning-table-generators.md

   This and #3 share ~80% of their machinery, so the doc designs one table-generation framework in src/analysis/tables/ (channel-role mapping with auto-suggestion via the existing normalization system, event detection primitives, 2D RPM × load binning, heatmap results view with per-cell counts/confidence, CSV + clipboard export, multi-log accumulation) with lambda delay and accel enrichment as pluggable analyzers on top.

   Lambda delay approach (this issue):

   • Detect injector PW steps (relative derivative threshold, default 8% within 150 ms), gated on steady RPM/load, no overlapping events, and fuel-cut regions excluded.
   • Measure delay to the first lambda deflection in the expected direction — threshold scaled to measured sensor noise (3σ), crossing time linearly interpolated between samples so a 20 Hz log can still resolve sub-sample delays.
   • Bin by RPM × load at the step instant; median per cell with MAD spread; per-cell confidence (Empty/Low/Medium/High) from count + dispersion. Sparse cells stay blank — no interpolation of fabricated values.
   • Multiple logs fold into one accumulating table; a rejection breakdown ("39 rejected: 30 unsteady, 9 no response") tells you what driving to log next.
   • Windowed cross-correlation between PW and lambda ships later as an alternative mode for steady-cruise logs without crisp steps.

   Validation targets: exampleLogs/haltech/2025-07-18_0215pm_Log1118.csv (RPM / Manifold Pressure / Injector 1 On Time / Wideband O2 Overall / Target Lambda) and exampleLogs/speeduino/speeduino.mlg (RPM / MAP / PW / AFR), plus synthetic-signal unit tests with known ground-truth delays at 5–50 Hz sample rates.

   Phasing: framework + lambda delay ship first (Phase 1); accel enrichment (#3) follows as Phase 2 reusing the same framework. Full algorithm details, tunable thresholds, data structures, and UI flow are in the doc.
3. SomethingNew71 commented on Sep 17, 2026

   

   SomethingNew71

   on Sep 17, 2026

   CollaboratorAuthor

   More actions

   Design review against 2.14.1 + bolstered plan (2026-09-17)

   The 2026-07-16 design doc (commit 78dbf4a, branch claude/inspiring-lamport-fc57c1, never merged) was re-verified against current main (2.14.1, 59 commits later). The architecture holds: sibling TableAnalyzer trait in src/analysis/tables/, channel roles with auto-suggestion, RPM × load accumulator with median/MAD/confidence, egui window with heatmap + event inspector, CSV + clipboard export. What follows is the record of what no longer holds, the design additions, and the implementation plan.

   Decisions: 2.15.0 ships Phase 1 + Phase 2 (this issue + #3), with MCP tools, as two PRs — PR 1 core (framework + both analyzers + normalization + fixture tests, no UI), PR 2 surface (window, export, presets, MCP, docs, version bump).

   A. Verification findings (doc vs main)

   #	Doc claim	Reality on main	Consequence
   A1	Reuse AnalysisResult	AnalysisResult.values is strictly 1-D per timestamp (src/analysis/mod.rs:65).	Sibling trait + new table result type confirmed.
   A2	Median/MAD helpers exist	None anywhere in src/; only median_filter (filters.rs:350) and a median field in DescriptiveStats.	Write median, mad, robust_sigma in tables/stats.rs.
   A3	Median sample interval	No sample-rate helper exists; two fragile ad-hoc estimators (ipc/handler.rs:862, filters.rs:454).	Write median_interval + effective_update_interval.
   A4	Reuse ParamDef/ParamType::Channel	Private to src/ui/analysis_panel.rs:29-43; no enum/choice variant.	Table window renders its own param grid with the same egui idioms.
   A5	Reuse scatter_plot.rs heatmap	Closer precedent is src/ui/histogram.rs (painter rect_filled per cell, hover, click, get_aaa_text_color, TSV clipboard copy at :2270, 2-D bin math calculate_data_bin at :64). All private and entangled with painting. There are already four duplicate heat ramps and five duplicate binning loops.	Extract binning into tables/binning.rs (pub) and have histogram call it. Use colormap::sample(Viridis) for the new view; do not add a fifth ramp.
   A6	speeduino.mlg is a validation target	9.3 s, 139 records, engine off (RPM 0).	Channel-naming fixture only. Replace with rusEFI + Link + MegaSquirt.
   A7	Haltech Injector 1 On Time auto-suggests via normalization	Does not normalize (exact-match map, no entry). Also unmapped: Throttle Position Derivative, Decel Cut State, Clutch State, Speeduino TPS DOT/DFCO/PW2, rusEFI Fuel: Last inj pulse width.	Auto-suggestion must be category + unit + name-heuristic driven, plus new built-in entries.
   A8	Haltech values are clean	Haltech writes an i32 sentinel family (-2147483617, …637, …630, …638, …643) for "no reading": 3 rows in HALTECH_SMALL (Undriven Wheel Speed), ~250 k cells in the 88 MB sibling. Target Lambda is ×1000 int; Injector 1 On Time is µs and includes injector dead time.	Invalid-sample masking + plausibility ranges are mandatory. Dead-time param.
   A9	Haltech small fixture gives ground-truth delays	It is a throttle-blip log: RPM 630→3500, MAP 26→98 kPa in 17 s. Nothing is steady.	Rejection-breakdown fixture (and the primary AE fixture for #3). Ground truth comes from synthetic tests.
   A10	OECUA spec metadata is reliable for lambda target	haltech-nsp.adapter.yaml lists "Lambda Target" under both lambda and afr_target (last-write-wins). romraider-csv files a PW channel under injector_duty.	Spec category/unit is a hint; confirm unit by data. Upstream spec fix filed separately.
   A11	Persist mapping via eframe storage like computed_library	ComputedChannelLibrary persists to its own JSON file (computed.rs:214-296).	Follow that pattern → table_presets.json.
   A12	(not in doc)	UltraLog now has an MCP server (23 tools) with a 512 KiB response guard.	Expose generators as MCP tools.
   A13	EcuType as preset key	Not Serialize.	Key by EcuType::name().
   A14	Tools panel is localized	tools_panel.rs strings are hardcoded.	New window strings under table_gen.* in all 15 i18n/*.yaml.
   A15	MCP port 52385	Code says 52453 (mcp/server.rs:53); CLAUDE.md and mcp/mod.rs are stale.	Fix in passing.

   B. Bolstered design (deltas to the doc)

   B1. Dead time first, rise time second. Per event store dead_time_ms (interpolated first crossing of max(k·σ, min_delta) in the expected direction — primary table value), t63_ms (time to 63 % of settled deflection — secondary view), response_magnitude, noise_sigma. Noise: σ = 1.4826 × MAD(Δλ at update instants) / √2 — differences only between distinct-value update instants, never raw log samples (sample-and-hold makes 90 % of raw first differences exactly 0 and collapses the threshold); √2 corrects for differencing two iid samples. A zero-variance baseline (sensor not ready, flat 1.000/14.7) rejects as InvalidSamples. Wiki note: the measured value includes ≈½ engine cycle of injection-scheduling bias plus ≈1 cycle to the exhaust; that is what a closed-loop delay table should contain, so it is reported, not subtracted.

   B2. Auto-suggestion: three-tier scoring. Roles Rpm, Load (+LoadKind::{Map,Tps}), PulseWidth, Lambda, optional LambdaTarget, FuelCut, ClosedLoopState, CoolantTemp, Clutch. Score per channel: normalization hit (100) → OECUA spec category + unit (60–70) → name heuristics (40: on time, pulse width, inj.*pw, dfco, decel cut, o2 control state, ego, clutch, coolant…) → data-plausibility veto (RPM median 300–12000, MAP 15–400, λ 0.5–1.6 or AFR 7–24 which also sets FuelMixtureUnit). Names containing overall|avg|average score −10 so a per-bank sensor wins ties (averages of sensors with different transport delays smear the rise). Ties within 10 points → ⚠ in the UI. New built-in normalization entries: Injector 1 On Time, Injection Stage 1 Average Injection Time, Injection Effective PW, Injection Actual PW, Fuel: Last inj pulse width, Base PW, INJ Duration(ms) → Pulse Width; TPS DOT, Throttle Position Derivative, TPS Delta → new canonical TPS Rate.

   B3. Invalid-sample masking (load-bearing). Mask non-finite, |v| ≥ 2^31 − 64, and out-of-band samples per role before any math. A window with > 10 % masked lambda samples → InvalidSamples. mask_invalid(times, values) also enforces alignment: Log::get_channel_data is a filter_map that drops ragged rows, so a column can be shorter than times (the require_aligned contract in CLAUDE.md); ragged → ComputationError, never a GUI-thread panic.

   B4. Effective update rate + stair-step interpolation (load-bearing). CAN widebands are often sample-and-hold at 10–20 Hz inside a 50–500 Hz log. Compute effective_update_interval per mapped channel (median spacing of distinct-value changes); interpolate crossings between update instants, not adjacent samples; derate quality when update interval > 0.25 × expected delay; warn "Lambda updates at 10 Hz inside a 100 Hz log — resolution ±50 ms".

   B5. Event source + gating profiles. PW steps only in v1, relative test on PW − injector_deadtime_ms (new param, default 0; a 10 % bump at 3 ms with 1 ms dead time reads as ≈6.7 % and would miss an 8 % threshold). Profiles: Strict (default; trim-bump logs: RPM ±200, load ±8 kPa/±5 %, overlap 600 ms) and Relaxed (driving logs: widens only RPM ±400 and overlap 400 ms; load band unchanged because a 15 kPa rise is a tip-in whose lean-first response measures wall-wetting; events record their profile and any relaxed event caps the cell at Medium). Fuel-cut-exit events are cut from 2.15.0 — on DFCO exit the sensor is pegged lean with slow pump-cell recovery, walls are dry, and rusEFI/MS taper fuel back; the "delay" would be recovery + wall-wetting. Phase 3 with that caveat.

   B6. Closed-loop contamination gate. When a ClosedLoopState/correction role is mapped (Haltech O2 Control State/O2 Control Bank 1 Short Term Fuel Trim, Speeduino Gego, rusEFI Fuel: Total correction — rusEFI has no dedicated STFT channel), reject events where the correction moved > 1 % within ±step_window_ms (ClosedLoopActive). Unmapped → run report warns "no closed-loop role mapped; results assume open loop". Wiki pre-flight: open loop, manual trim bumps, warm engine.

   B7. RejectReason enum. Unsteady, Overlap, NoResponse, WrongDirection, FuelCut, LowPw, ClosedLoopActive, ColdEngine, Clutch, InvalidSamples, OutOfAxis. Run report = counts per reason ("41 steps found · 39 rejected: 30 unsteady, 9 no response"). Every event, accepted or rejected, stays in the inspector with its reason.

   B8. Output units. Cell = ms. Export/clipboard also offer engine cycles (ms × RPM_center / 120 000) and ignition events (cycles × cylinders / 2, new cylinders param) because MS3 / AEM Infinity / MoTeC M1 / Link / Emerald differ.

   B9. Statistics. Median, MAD, count, confidence (Empty / Low n<3 or MAD/median>0.5 / Medium / High n≥8 and ≤0.25). No interpolation of empty cells.

   B10. Axes. Defaults from the 1st–99th percentile of masked RPM/load, rounded to 250/500 rpm and 10 kPa / 10 %, editable as a comma list; lower-edge inclusive; grid capped at 64 × 64 (keeps the MCP payload ≪ 512 KiB).

   B11. Performance. Read columns via LoadedFile::get_channel_column (cached column-major), not LogDataAccess (re-transposes per call). O(n); the 88 MB Haltech log runs well under a second, so v1 is synchronous like run_analyzer, with computation_time_ms in the report.

   B12. Validation fixtures (replaces the doc's list).

   Fixture	Rate	Role
   exampleLogs/rusefi/rusefilog.mlg (+ Log1.mlg)	100 Hz	Primary real-log test: Fuel: Last inj pulse width, Lambda, Fuel: target lambda, RPM, MAP, dfcoActive, Fuel: TPS AE Active.
   exampleLogs/link/ECU Log 2024-03-22 11;20;32 am.llg5	500 Hz	Best resolution: Injection Effective PW, Lambda Avg, Engine Speed, MAP. Verify Lambda Avg is non-zero past t=0 first.
   exampleLogs/haltech/2025-07-18_0215pm_Log1118.csv	50 Hz	Auto-suggestion regression, rejection breakdown (strict ≈ all rejected, relaxed > 0), primary AE fixture.
   exampleLogs/haltech/2025-03-06_0937pm_Logs658to874.csv (88 MB)	50 Hz	Sentinel masking at scale + performance; gated on example_file_exists.
   exampleLogs/megasquirt/2026-04-12_12.49.36.mlg	15 Hz	668 s engine-running log; Duty Cycle as PW, Lambda, TPS DOT, DFCO; low-rate warning path.
   exampleLogs/speeduino/speeduino.mlg	—	Channel-naming fixture only.

   B13. Shared helpers to create (all pub — PR 1 has no consumers and CI runs clippy --all-targets -D warnings): tables/stats.rs (median, mad, robust_sigma_diff, percentile, median_interval, effective_update_interval), tables/binning.rs (AxisSpec, bin_index, TableGrid, CellStats, Confidence, uniform_bin replacing histogram's private calculate_data_bin). Reuse as-is: time_derivative, median_filter, FuelMixtureUnit + detect_fuel_mixture_unit, colormap::sample, get_aaa_text_color (→ pub(crate)), analytics::track_export, toasts, set_jump_to_time. Do not reuse find_record_at_time (hard-codes files.first()). ChannelRole, ChannelMapping, AxisSpec, Confidence, RejectReason derive serde (+ Hash/Eq on unit enums) for the tagged IPC enums and presets file. All params go through AnalyzerConfig in PR 1 so PR 2 adds no analyzer signature change.

   B14. UI. egui::Window in src/ui/table_generator.rs using the analysis-panel idiom, opened from a new "📊 Table Generators" section in the tools panel, rendered from the modal block in app.rs; uses selected_file. States: Setup (roles with ⚠, axes, params, profile) → Results (Viridis heatmap, value text, count badge by confidence, hover tooltip, click → inspector with jump-to-time: find the tab whose file_index matches the event's file, activate it, then set_jump_to_time; disabled if the log is unloaded; toolbar Add current file / Remove log… / Reset / measure selector / units / Export CSV / Copy for paste) → Empty/error with rejection breakdown.

   B15. MCP + IPC. IpcCommand::GenerateTable { generator, file_id, mapping?, axes?, params?, profile }, ResponseData::Table { … cells, counts, mad, confidence, rejected: HashMap<String, usize>, warnings, resolved mapping }. Tools generate_lambda_delay_table, generate_accel_enrich_table, get_table_events { generator, file_id, max_events } (capped like FindPeaks, with total_events/truncated). const _ cap assert like the existing ones; roundtrip + coverage tests.

   B16. Persistence. Mapping presets in src/table_presets.rs following ComputedChannelLibrary exactly → table_presets.json, keyed by (generator_id, EcuType::name()), seeded built-ins for Haltech / Speeduino / rusEFI / Link / MegaSquirt / ME; no UserSettings change. Accumulator is session-only, events keyed by a per-load nonce (never by file index — index-keyed cache contract — and never by bare file name: every rusEFI install has a Log1.mlg).

   C. Implementation plan

   PR 1 — feature/tuning-tables-core (branched off origin/main; cherry-pick 78dbf4a for the doc, then revise it with this record)

   1. docs(analysis): revise tuning-table design doc against 2.14.1
   2. feat(analysis): add table-generation framework — binning, stats, channel roles, event primitives (src/analysis/tables/{mod,stats,binning,channel_map,events}.rs; histogram switches to binning::uniform_bin)
   3. feat(analysis): add lambda delay table generator (lambda_delay.rs + #[cfg(test)] synthetic.rs with in-file xorshift + Box-Muller — no rand, there are no dev-dependencies)
   4. feat(analysis): add acceleration enrichment table generator (accel_enrich.rs, see [Feature]: Acceleration enrichment table Generator #3)
   5. feat(normalize): map injector on-time and TPS-rate channel names
   6. test(analysis): tuning-table fixtures against rusEFI, Link, Haltech, MegaSquirt logs (tests/core/table_generator_tests.rs)

   PR 2 — feature/tuning-tables-ui (after PR 1 merges)

   1. feat(ui): add table generator window for lambda delay and accel enrichment (+ i18n × 15)
   2. feat(export): CSV and clipboard export for generated tables (hand-written CSV: value / count / MAD grids, # headers, blank empties, "exclude Low" toggle; bare TSV to clipboard)
   3. feat(analysis): persist table generator channel mappings per ECU
   4. feat(mcp): expose tuning-table generators and their event lists as tools
   5. chore(release): bump to 2.15.0 and add tuning table generators to the site (CLAUDE.md contracts section + port fix, README, wiki Tuning-Table-Generators.md + User-Guide + sidebar, docs/index.html cards + sitemap)

   D. Verification

   • CI parity: cargo fmt --all -- --check, cargo clippy --all-targets --all-features -- -D warnings, cargo test --all-features, cargo doc --no-deps --all-features.
   • Synthetic ground truth: PW step + first-order lambda response λ(t) = λ₀ + Δ(1 − e^{−(t−t₀−d)/τ}), sample-and-hold at update rate u, Gaussian σ. Sweep d ∈ {60, 120, 250, 500} ms, τ ∈ {30, 80} ms, log rate ∈ {5, 10, 20, 50, 100} Hz, u ∈ {log rate, 10 Hz}, σ ∈ {0, 0.005, 0.02 λ}. Assert dead-time error ≤ max(½ update interval, 10 ms); the u = 10 Hz in 100 Hz, σ = 0.005 case proves the update-instant noise estimator; 5 Hz refuses; AFR input = λ input; RPM ramp → Unsteady; no response → NoResponse; sentinels / flat baseline → InvalidSamples; ragged log → error, no panic; falling steps reversed; dead-time-adjusted step passes when injector_deadtime_ms is set.
   • Binning/stats: lower-edge inclusive; tiers; accumulate two logs then remove one = single-log result.
   • Real logs: rusEFI events > 0 with medians 20–1000 ms; Haltech auto-suggestion picks RPM, Manifold Pressure, Injector 1 On Time, a wideband (⚠ Overall vs 1), Target Lambda, Decel Cut State, Clutch State; strict rejects ≥ 90 % on Haltech; MegaSquirt emits the low-rate warning.
   • Manual GUI, MCP (64 × 64 stays under the guard; truncation flag), presets round-trip, and lambda→AE composition in one session.
4. SomethingNew71

   added a commit that references this issue on Sep 18, 2026

   feat(analysis): lambda delay and acceleration enrichment table genera…

   ...

   1c2305f
5. SomethingNew71 commented on Sep 19, 2026

   

   SomethingNew71

   on Sep 19, 2026

   CollaboratorAuthor

   More actions

   Shipped in v2.15.0.

   Lambda Delay is a top-level tool beside Log Viewer, Scatter Plots and Histogram (⌘4 / Ctrl+4). It finds injector pulse-width steps, measures the delay to the wideband response, and bins the result by RPM × load with per-cell sample counts and confidence. Setup (auto-detected channel roles, axes, parameters) is in the Tool Properties panel; the heatmap, rejection breakdown and event inspector with jump-to-time fill the center. Several logs fold into one table. Export as CSV, clipboard, PNG or PDF in milliseconds, engine cycles or ignition events.

   What landed from the plan above:

   • Framework, both generators, normalization additions, fixture tests — feat(analysis): lambda delay and acceleration enrichment table generators (#3, #4) #90 (@Barneyjm)
   • Promotion to top-level tools, PNG/PDF export, release — feat(tables): promote lambda delay and accel enrichment to top-level tools, add PNG/PDF export, release 2.15.0 #91
   • Release PR — Release v2.15.0 — lambda delay and acceleration enrichment table generators #92
   • Wiki: Tuning Table Generators

   Not in this release, deferred to Phase 3: MCP tools for the generators, on-disk per-ECU mapping presets, and the cross-correlation mode for steady-cruise logs. Open a new issue if any of those matter to you.
6. SomethingNew71

   closed this as completedon Sep 19, 2026