[go] Velocity-coherent tracking system and LiDAR documentation reorganisation#391
Open
ddol wants to merge 83 commits into
Open
[go] Velocity-coherent tracking system and LiDAR documentation reorganisation#391ddol wants to merge 83 commits into
ddol wants to merge 83 commits into
Conversation
…dmap Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
…usions, coverage) Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
…y coloring, occlusion markers) Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
…caffolding, add track point cloud export Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
…P replay Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
…onger placeholder) Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
…lement corresponding handler
…ound point issues Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
… during PCAP replay
…eground point encoding issues
… better data handling
…AP handling for background seeding
… counts in status UI
… and ComputeQualityMetrics Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
* Initial plan * [ai][docs] add velocity-coherent foreground extraction design document Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [ai][docs] fix code review feedback in velocity-coherent design doc Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [ai][go] implement Phase 1 velocity estimation and update design doc Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [ai][go] implement velocity-coherent foreground extraction phases 2-5 Co-authored-by: ddol <445312+ddol@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
… update cleanup response
…database size analysis
…ocity-coherent tracks
…ntegration spec (#168) * Initial plan * Initial plan * [ai][docs] Add comprehensive 7DOF schema compatibility analysis Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [ai][docs] Add PR split recommendation for 7DOF migration Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [ai][docs] Add executive summary for 7DOF migration plan Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [ai][docs] Add navigation README for 7DOF migration documentation Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [ai][docs] Add current vs 7DOF feature comparison table Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [ai][docs] Replace 5 docs with 2 focused plans: static pose alignment (current) and motion capture architecture (future) Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [ai][docs] update static plan for 7-variable 3D bounding boxes (x,y,z + L,W,H + heading) Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [ai][docs] Refocus static plan on Phase 1: Hesai PCAP → 7DOF tracks, align with av-lidar-integration-plan.md schema Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [ai][docs] Add Phase 1 implementation checklist: 4 PRs for Hesai → 7DOF tracks with UI visualization Co-authored-by: ddol <445312+ddol@users.noreply.github.com> * [docs] formatting --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ddol <445312+ddol@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| ws.writeJSONError(w, http.StatusForbidden, err.Error()) | ||
| return | ||
| } | ||
| if err := os.MkdirAll(absDir, 0o755); err != nil { |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
Copilot Autofix
AI 3 months ago
Copilot could not generate an autofix suggestion
Copilot could not generate an autofix suggestion for this alert. Try pushing a new commit or if the problem persists contact support.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a velocity-coherent (VC) tracking pipeline alongside the existing background-subtraction tracker, expands PCAP replay/forwarding tooling, introduces track/cluster quality metrics and DB introspection endpoints, and reorganises/extends LiDAR documentation.
Changes:
- Add VC tracking storage/querying, dual-pipeline algorithm switching, and web UI support for rendering VC tracks.
- Introduce track/cluster “quality metrics” persisted to SQLite (plus related tests and schema/migrations).
- Enhance PCAP replay (including real-time replay under
pcaptag) and add database stats + snapshot cleanup helpers.
Reviewed changes
Copilot reviewed 73 out of 78 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| web/src/routes/lidar/tracks/+page.svelte | Fetches and toggles display of VC tracks on the tracks page. |
| web/src/lib/types/lidar.ts | Extends Track type with optional quality metrics. |
| web/src/lib/components/lidar/MapPane.svelte | Renders VC tracks with a distinct style; adds quality/occlusion visual overlays. |
| web/src/lib/api.ts | Adds getVCTracks() API client and adapts response into Track. |
| web/package.json | Bumps web package version to 0.4.0-pre4. |
| internal/security/pathvalidation.go | Adds CLI output-path validation helper and filename sanitiser. |
| internal/lidar/velocity_estimation_test.go | Adds unit tests for velocity estimation primitives. |
| internal/lidar/velocity_coherent_tracking_test.go | Adds tests for VC long-tail + sparse continuation logic. |
| internal/lidar/velocity_coherent_tracker_test.go | Adds tests for VC tracker lifecycle and concurrency. |
| internal/lidar/velocity_coherent_clustering_test.go | Adds tests for 6D clustering and spatial index. |
| internal/lidar/vc_track_store_test.go | Adds tests for VC track persistence (tracks + observations). |
| internal/lidar/vc_track_store.go | Implements VC track/cluster persistence and querying utilities. |
| internal/lidar/tracking_test.go | Adds tests for track quality metrics computation. |
| internal/lidar/tracking.go | Adds track quality metrics fields + computation on TrackedObject. |
| internal/lidar/track_store_test.go | Updates test schema to include new quality/cluster metric columns. |
| internal/lidar/track_store.go | Persists new cluster/track quality metrics; computes metrics before insert/update. |
| internal/lidar/track_export.go | Adds track point-cloud export scaffolding and Pandar40P packet encoder stub. |
| internal/lidar/quality_test.go | Adds tests for run stats, training filters, and quality scoring. |
| internal/lidar/network/pcap_stub.go | Updates ReadPCAPFile signature to accept an optional forwarder. |
| internal/lidar/network/pcap_realtime_stub.go | Adds stub for real-time PCAP replay when pcap tag is off. |
| internal/lidar/network/pcap_realtime.go | Adds real-time PCAP replay (timing-respecting) under pcap tag. |
| internal/lidar/network/pcap.go | Adds optional packet forwarding during PCAP replay. |
| internal/lidar/network/forwarder.go | Switches forwarder logging to lidar.Debugf. |
| internal/lidar/monitor/status.html | Extends monitor UI: PCAP speed controls, algorithm selection, export/cleanup links. |
| internal/lidar/monitor/algorithm_api_test.go | Adds tests for tracking algorithm config/stats/VC endpoints. |
| internal/lidar/foreground_snapshot.go | Adds foreground-only ASC export with path validation. |
| internal/lidar/foreground.go | Adds warmup gating + post-settle alpha behaviour for background/foreground processing. |
| internal/lidar/dual_pipeline_test.go | Adds tests for dual extraction pipeline behaviour and concurrency. |
| internal/lidar/docs/pcap-split-tool.md | Formatting/whitespace normalisation in docs excerpt. |
| internal/lidar/docs/pcap-analysis-mode.md | Adds documentation for “PCAP analysis mode” workflow and endpoints. |
| internal/lidar/docs/lidar_sidecar_overview.md | Updates overview metadata and formatting. |
| internal/lidar/docs/lidar-tracking-integration.md | Adds tracking integration status doc. |
| internal/lidar/docs/lidar-foreground-tracking-status.md | Adds consolidated foreground tracking/export investigation doc. |
| internal/lidar/docs/lidar-background-grid-standards.md | Adds standards comparison + export recommendations doc. |
| internal/lidar/docs/foreground_tracking_plan.md | Adds cross-links to new VC/foreground status docs. |
| internal/lidar/clustering_test.go | Adds aspect-ratio degenerate-shape tests. |
| internal/lidar/clustering.go | Computes/persists cluster density + aspect ratio metrics. |
| internal/lidar/background.go | Adds warmup/post-settle params + setters; adjusts EMA update alpha behaviour. |
| internal/lidar/arena.go | Extends WorldCluster with quality metrics fields. |
| internal/db/schema_consistency_test.go | Switches schema normalisation to shared helper. |
| internal/db/schema.sql | Updates canonical schema with new metrics fields and VC tracking tables. |
| internal/db/migrations/000013_create_velocity_coherent_tracks.up.sql | Adds VC tables, merge log, algorithm config log. |
| internal/db/migrations/000013_create_velocity_coherent_tracks.down.sql | Drops VC tables and related indexes. |
| internal/db/migrations/000012_add_track_quality_metrics.up.sql | Adds track/cluster quality columns + analysis run stats field; adds quality index. |
| internal/db/migrations/000012_add_track_quality_metrics.down.sql | Rebuilds tables to remove quality columns (SQLite rollback pattern). |
| internal/db/migrations/000001_original_schema.up.sql | Adjusts original schema migration notes/table creation ordering. |
| internal/db/migrations/000001_original_schema.down.sql | Adjusts rollback semantics consistent with bootstrap migration. |
| internal/db/migrations/000000_bootstrap.up.sql | Introduces migration 0 to create the original data table. |
| internal/db/migrations/000000_bootstrap.down.sql | Drops the data table for migration 0 rollback. |
| internal/db/migrate.go | Enables NoTxWrap for sqlite migrate driver; enhances SQL normalisation. |
| internal/db/db.go | Adds DB stats API helpers + background snapshot dedupe helpers; exposes /api/db_stats. |
| internal/api/server.go | Adds /api/db_stats endpoint and validates report temp config path. |
| go.mod | Promotes go-echarts v2.6.7 to a direct dependency. |
| docs/package.json | Bumps docs package version to 0.4.0-pre4. |
| cmd/tools/pcap-analyze/main.go | Uses updated PCAP reader signature; wires runtime clustering params; parser init changes. |
| cmd/sweep/main.go | Validates output paths for sweep CSV outputs. |
| cmd/radar/radar.go | Adds foreground forwarder support; adds algorithm selection + dual pipeline wiring; updates dev args semantics. |
| cmd/deploy/monitor.go | Adds API-first DB stats collection with SSH fallback; adds JSON decode helpers. |
| cmd/deploy/main.go | Bumps deploy tool version to 0.4.0-pre4. |
| Makefile | Bumps repo VERSION to 0.4.0-pre4 and updates dev-go-lidar target flags. |
Comment on lines
+126
to
+133
| var b strings.Builder | ||
| // Limit resulting filename length to avoid overly long paths | ||
| const maxLen = 128 | ||
| lastUnderscore := false | ||
| for _, r := range s { | ||
| if len(b.String()) >= maxLen { | ||
| break | ||
| } |
Comment on lines
+610
to
+659
| func (track *TrackedObject) ComputeQualityMetrics() { | ||
| // Track length: Sum of Euclidean distances between consecutive positions | ||
| track.TrackLengthMeters = 0 | ||
| if len(track.History) > 1 { | ||
| for i := 1; i < len(track.History); i++ { | ||
| dx := track.History[i].X - track.History[i-1].X | ||
| dy := track.History[i].Y - track.History[i-1].Y | ||
| track.TrackLengthMeters += float32(math.Sqrt(float64(dx*dx + dy*dy))) | ||
| } | ||
| } | ||
|
|
||
| // Track duration: Total lifetime in seconds | ||
| if track.LastUnixNanos > track.FirstUnixNanos { | ||
| track.TrackDurationSecs = float32(track.LastUnixNanos-track.FirstUnixNanos) / 1e9 | ||
| } | ||
|
|
||
| // Occlusion count: Count gaps in observations (>200ms = missed frame at ~10Hz) | ||
| const occlusionThresholdNanos = 200_000_000 // 200ms | ||
| track.OcclusionCount = 0 | ||
| track.MaxOcclusionFrames = 0 | ||
|
|
||
| if len(track.History) > 1 { | ||
| for i := 1; i < len(track.History); i++ { | ||
| gap := track.History[i].Timestamp - track.History[i-1].Timestamp | ||
| if gap > occlusionThresholdNanos { | ||
| track.OcclusionCount++ | ||
| // Estimate frames at 10Hz | ||
| gapFrames := int(gap / 100_000_000) // 100ms per frame | ||
| if gapFrames > track.MaxOcclusionFrames { | ||
| track.MaxOcclusionFrames = gapFrames | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // Spatial coverage: Ratio of observed area to theoretical max | ||
| // This is a simplified metric - more sophisticated versions could track | ||
| // actual point cloud coverage within the bounding box | ||
| if track.ObservationCount > 0 { | ||
| // Estimate coverage as (observations / theoretical_max_observations) | ||
| // At 10Hz, theoretical max = duration * 10 | ||
| theoreticalMax := track.TrackDurationSecs * 10 | ||
| if theoreticalMax > 0 { | ||
| track.SpatialCoverage = float32(track.ObservationCount) / theoreticalMax | ||
| // Clamp to [0, 1] | ||
| if track.SpatialCoverage > 1.0 { | ||
| track.SpatialCoverage = 1.0 | ||
| } | ||
| } | ||
| } |
Comment on lines
88
to
+92
| // InsertTrack inserts a new track into the database. | ||
| func InsertTrack(db *sql.DB, track *TrackedObject, worldFrame string) error { | ||
| // Ensure quality metrics are computed | ||
| track.ComputeQualityMetrics() | ||
|
|
Comment on lines
146
to
+150
| // UpdateTrack updates an existing track in the database. | ||
| func UpdateTrack(db *sql.DB, track *TrackedObject, worldFrame string) error { | ||
| // Ensure quality metrics are computed | ||
| track.ComputeQualityMetrics() | ||
|
|
Comment on lines
+235
to
+248
| var track VelocityCoherentTrack | ||
| var stateStr string | ||
| var objectClass, objectConfidence sql.NullString | ||
|
|
||
| err := rows.Scan( | ||
| &track.TrackID, &track.SensorID, &stateStr, | ||
| &track.FirstUnixNanos, &track.LastUnixNanos, &track.ObservationCount, &track.Hits, &track.Misses, | ||
| &track.AvgSpeedMps, &track.PeakSpeedMps, | ||
| &track.VelocityConfidence, &track.VelocityConsistency, | ||
| &track.BoundingBoxLengthAvg, &track.BoundingBoxWidthAvg, &track.BoundingBoxHeightAvg, | ||
| &track.HeightP95Max, &track.IntensityMeanAvg, | ||
| &track.MinPointsObserved, &track.SparseFrameCount, | ||
| &objectClass, &objectConfidence, | ||
| ) |
| // Initialize parser | ||
| parserConfig := parse.LoadEmbeddedPandar40PConfig() | ||
| parser := parse.NewPandar40PParser(parserConfig) | ||
| parserConfig, _ := parse.LoadEmbeddedPandar40PConfig() |
Comment on lines
+94
to
+106
| // Calculate delay since last packet (scaled by speed multiplier) | ||
| delay := captureTime.Sub(lastPacketTime) | ||
| scaledDelay := time.Duration(float64(delay) / config.SpeedMultiplier) | ||
|
|
||
| // Wait for scaled delay to maintain timing | ||
| if scaledDelay > 0 { | ||
| select { | ||
| case <-ctx.Done(): | ||
| return ctx.Err() | ||
| case <-time.After(scaledDelay): | ||
| // Continue | ||
| } | ||
| } |
Comment on lines
116
to
+120
| // Create sqlite driver instance | ||
| driver, err := sqlite.WithInstance(db.DB, &sqlite.Config{}) | ||
| // Set NoTxWrap to avoid wrapping migrations in transactions (SQLite handles this) | ||
| driver, err := sqlite.WithInstance(db.DB, &sqlite.Config{ | ||
| NoTxWrap: true, | ||
| }) |
ddol
added a commit
that referenced
this pull request
Mar 23, 2026
ddol
added a commit
that referenced
this pull request
Mar 23, 2026
ddol
added a commit
that referenced
this pull request
Mar 24, 2026
ddol
added a commit
that referenced
this pull request
Mar 24, 2026
* [docs] Refactor structural hygiene plan: update implementation status and fix JSON tag inconsistencies * [go] Enhance error handling in radar and lidar processing: add error checks for setting ring elevations and background parameters * [js] Refactor Event and RawRadarStats interfaces: update property names to camelCase for consistency * [go] Fix metric key casing in cosine correction test: change "MaxSpeed" to "max_speed" for consistency * [go] Fix casing for radar event API keys: change "magnitude", "uptime", and "speed" to "Magnitude", "Uptime", and "Speed" for consistency * Fix casing for EventAPI keys: change "Magnitude", "Uptime", and "Speed" to "magnitude", "uptime", and "speed" for consistency * [docs] Standardize JSON tags for EventAPI and RadarObjectsRollupRow to snake_case * [docs][ai] backlog audit: PR refs, landed items, open PR links - Add [#422] [#418] [#416] [#419] [#413] to 7 Complete items missing PR refs - Move Go god file splitting Phase 1 ([#412] [#417]) to Complete - Add [#421] [#425] [#391] [#390] [#392] to open items with active PRs * [docs] sort completed * [docs] sort backlog * [docs] backlog: reorder completed items for clarity and consistency * [docs][ai] backlog 0.5.x merge update: move #420 to Complete and add #424 in Complete * [docs][ai] backlog PR ref audit: fix misattributed and dead PR tags - Remove [#421] LiDAR tracks table consolidation from v0.5.0 open (already landed in #419, correctly in Complete section) - Remove [#392] tag from unpopulated data remediation (PR stale/abandoned) - Remove [#389] VRLOG frame coalescing from open (landed via #381); add to Complete as [#381] - Remove [#393] SSE item; replace with untagged SSE backpressure item (subscriber buffer landed in #380; full backpressure outstanding) - Remove [#382] tag from sweep workers open item (plan doc merged, implementation outstanding); add [#382] to Complete - Remove [#387] duplicate from v0.6.0 (plan merged, implementation is #429 already in v0.5.0); add [#387] to Complete * [ver] bump version to 0.5.0-pre25 across all relevant files * [sh] add version-bump script and update Makefile for version management * [docs] add backlog item for displaying runtime version and git SHA in settings UI * [docs] update backlog item for v0.7.0 - unify frontend terminology * [docs] compelte backlog item * [docs] update implementation snapshot and findings in go codebase structural hygiene plan * [docs] update Go Codebase Structural Hygiene Plan to reflect current implementation status and outstanding tasks * [docs] add template for project planning documentation * [docs] add binary size reduction plan to address excessive binary size * [js][make] enhance build environment setup to include versioning information * [docs] add clean-web target to remove build artifacts * [docs] update binary size reduction plan and add governance document
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full velocity-coherent tracking system, dual pipeline, PCAP real-time replay, foreground packet forwarding, DB stats/cleanup endpoints, algorithm comparison API, and documentation reorganisation.
Key Changes
Velocity-Coherent Tracking
Network and Replay
API and Database
Documentation
Stats
76 commits, 78 files changed, +14,676 -331
Note: This is a very large, stale branch. Many changes may conflict with main. Consider cherry-picking novel pieces rather than merging wholesale.