feat(new source): add Windows Event Log source (cherry-pick #24305 + fork fixes)#128
Open
ajayshekar-s1 wants to merge 7 commits into
Open
feat(new source): add Windows Event Log source (cherry-pick #24305 + fork fixes)#128ajayshekar-s1 wants to merge 7 commits into
ajayshekar-s1 wants to merge 7 commits into
Conversation
* feat(source): add Windows Event Log source Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(source): add Windows Event Log unit and integration tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci(source): add Windows Event Log integration workflow Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(source): add Windows Event Log changelog entry Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(source): cleanup and address review feedback for Windows Event Log Style and formatting cleanup across all source files: consistent log message punctuation, hoisted imports to module level, rustfmt single-line struct variants, and removed redundant blank lines. Maintainer review feedback: - Emit keywords as hex string (0x...) to preserve unsigned bitmask - Propagate flush_bookmarks() set_batch error instead of swallowing - Remove 5 placeholder tests that inflated coverage - Strengthen truncation test with real assertions - Add missing fields to Vector namespace schema (level_value, provider_guid, version, qualifiers, string_inserts, event_data, user_data, task_name, opcode_name, keyword_names) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(source): event ID filter perf fix, tests, and docs for windows_event_log Performance: build_xpath_query() now auto-generates XPath from only_event_ids (e.g. *[System[EventID=4624 or EventID=4625]]) so the Windows API filters at the source. Added early pre-filter in the drain loop after parse_system_section to discard non-matching events before expensive metadata/message calls. Includes 4096-char length guard that falls back to wildcard for very large ID lists. Tests: XPath generation unit tests, multi-filter interaction tests, config validation boundary tests, integration test exercising XPath generation without explicit event_query, and fixes for four pre-existing test assertion errors. Docs: markdown page and changelog fragment. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(source): promote windows_event_log error events to error level and run fmt The `vdev check events` linter requires InternalEvent types with "Error" in their name to log at error level. Promote WindowsEventLogParseError, WindowsEventLogQueryError, and WindowsEventLogBookmarkError from warn! to error! to satisfy this requirement. Also widen the max_event_age integration test timing margins (2s→5s sleep, 1s→3s threshold) to reduce flakiness on slow CI, remove the redundant integration test README, and apply rustfmt formatting fixes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(source): split config/runtime layers for cross-platform doc generation Move the windows_event_log config layer to compile on all platforms so `make generate-component-docs` works on Linux/macOS. Runtime code stays behind #[cfg(windows)] and build() returns an error on non-Windows. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(source): address PR review feedback for windows_event_log Simplify changelog to single-line summary since generated docs cover details. Add required CI check pattern (paths-filter + status gate) for Windows integration workflow. Replace repeated #[cfg(windows)] with cfg_if! blocks in mod.rs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(source): address CI failures and review feedback for windows_event_log - Fix field cap guard in parse_section: use OR instead of AND so the loop stops when either named_data or inserts reaches MAX_FIELDS - Fix channel fallback in build_event: use the subscription channel parameter when the parsed system channel is empty - Fix extract_message_from_event_data: check string_inserts (unnamed <Data> elements from eventcreate) before falling back to generic msg - Fix checkpoint resume test: allow small overlap (up to batch_size) caused by timeout-based shutdown racing between send_batch/finalize - Fix rejected ack test: also search string_inserts for marker event in case EvtFormatMessage is unavailable on the CI runner Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tests): isolate windows_event_log integration tests with per-test source names Each test now uses a unique provider name (e.g. VT_stress, VT_backlog) for eventcreate and XPath filtering, preventing cross-test pollution when tests run in parallel on multi-core CI runners. Also serializes test execution via --test-threads=1 in the Makefile target. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tests): fix checkpoint resume test assertion for per-test source isolation With per-test source names, phase 1 only receives 1 event (our test event), so the old `second_run.len() < first_count` assertion fails when first_count==1 and the second run correctly gets 1 new event. Replace the count-based assertion with content-based checks: verify phase1 event is NOT redelivered and phase2 event IS present. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(windows_event_log): remove --test-threads=1 to verify parallel isolation Per-test source names should be sufficient to prevent cross-test pollution. Removing serialization to prove tests pass in parallel. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tests): use dedicated log channel for resubscribe-after-clear test test_resubscribe_after_log_clear was running `wevtutil cl Application` which nukes the entire Application log, destroying events that other parallel tests depend on. This caused test_rejected_ack to get 0 events in phase 2 when running in parallel. Fix: create a temporary custom log channel (VectorTestResub) via PowerShell New-EventLog, subscribe to that, clear only that channel, and clean up via Drop guard. Application log is never touched. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tests): fix 4 failing windows_event_log unit tests - Parser tests: populate enriched metadata fields (opcode_name, task_name, keyword_names) in test events since the parser copies these directly rather than resolving them - Fallback message test: clear string_inserts so the parser reaches the fallback code path instead of returning the first string insert - Bookmark test: change assertion to verify graceful fallback behavior since hand-crafted bookmark XML is not valid for the Windows API Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
…event lost wakeup (#25195) * fix(windows_event_log): fix pre-drain ResetEvent race and add lost-wakeup regression tests The Windows Event Log service signals the pull-mode wait handle via SetEvent each time a new matching event is recorded. Because the handle is manual-reset, any SetEvent that fires between the last EvtNext call and the post-drain ResetEvent is silently lost — the subscription then hangs until the next OS event arrives (vectordotdev/vector#25194). Fix: reset the handle *before* entering the drain loop. Signals raised during the drain are preserved because SetEvent on an already-signaled handle is a no-op. Re-arm (SetEvent) on early exits so the next pull_events revisits the channel without waiting for a fresh OS notification: - budget exhaustion - bookmark failure mid-batch - transient EvtNext error Regression tests: - test_pull_events_preserves_setevent_during_drain: installs DRAIN_STEP_HOOK to fire SetEvent mid-drain and asserts wait_for_events_blocking returns EventsAvailable, not Timeout. - test_speculative_pull_recovers_without_signal: manually clears the channel signal via ResetEvent, confirms wait times out, then asserts pull_events still returns events — proving the speculative timeout pull in mod.rs self-heals independently of signal state. Also: comment re-subscription break paths (ERROR_EVT_QUERY_RESULT_STALE and INVALID_POSITION) noting the speculative pull as a safety net if the re-subbed channel does not immediately re-signal; add serialization note to DRAIN_STEP_HOOK. * fix(windows_event_log): add speculative timeout pull, deduplicate processing, fix error handling Four related improvements to mod.rs: 1. Speculative pull on WaitResult::Timeout: call pull_events on every timeout cycle as a belt-and-suspenders self-heal. EvtNext returns ERROR_NO_MORE_ITEMS immediately on an empty channel (near-zero cost). If events are recovered a warning is emitted. Guarantees recovery within one timeout period regardless of the root cause of the lost wakeup. 2. Extract with_subscription_blocking helper: wraps the spawn_blocking ownership-transfer pattern (move subscription in, run blocking fn, return subscription + result). All three blocking calls (wait, normal pull, speculative pull) now use this helper instead of inlining spawn_blocking. 3. Extract process_event_batch helper: the parse/emit/send_batch/finalize sequence was duplicated verbatim between the EventsAvailable arm and the speculative-timeout arm. Extracted into a shared free async function. Rate limiting is applied consistently in both paths via the helper. 4. Fix error-handling asymmetry: the speculative pull Err branch previously only logged warn! and continued, so a non-recoverable error (access denied, channel not found) would spam warnings indefinitely. Now mirrors the EventsAvailable path: emit WindowsEventLogQueryError, break on non-recoverable errors, apply exponential backoff on recoverable ones. * test(windows_event_log source): harden lost-wakeup regression tests against flakiness Address two independent flakiness sources in the #25194 regression tests so the suite is stable on real Windows CI runners. test_pull_events_preserves_setevent_during_drain: - Replaced a 1000ms blocking wait with an immediate 0ms poll after pull_events returns, so the check measures only the reset/preserve behavior of pull_events and is not contaminated by unrelated Windows system events signaling the handle during a nonzero wait window. - Keyed the DRAIN_STEP_HOOK fire-once to the subscription's own signal handle so concurrent pull_events calls from other tests can't flip the hook first and SetEvent the wrong handle. test_speculative_pull_recovers_without_signal: - Same 500ms→0ms poll change, opposite direction: real events arriving during the wait would have re-signaled the manually-cleared handle and flipped the expected Timeout into a real signal result. - Seed a deterministic record via 'eventcreate' before subscription creation so the non-empty-events assertion is independent of whatever backlog the runner happens to have. Freshly provisioned images can have an empty Application log, which would otherwise cause pull_events(100) to legitimately return empty and false-fail the test. * fix(windows_event_log): implement Sync for subscription types to satisfy Send bound on source future EventLogSubscription and ChannelSubscription had unsafe impl Send but no Sync impl. Since &T: Send requires T: Sync, process_event_batch holding &EventLogSubscription across an .await made the entire source future !Send, causing a compile error (ICE + future-not-Send) in release builds. All mutation on these types requires &mut self; &self methods are read-only or delegate to already-Sync types (RateLimiter). The underlying Windows kernel handles are safe for concurrent access. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(changelog): simplify windows_event_log fix fragment to user-focused one-liner Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(windows_event_log): prioritize shutdown signal * fix(windows_event_log): lighten speculative timeout pulls --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
* fix(windows_event_log): add standard source metadata * chore(changelog): add windows_event_log metadata fix note
… tasks Adapted from upstream vectordotdev/vector 29b089da (#25521). The fork does not have the crate::spawn_in_current_span() helper, so the two tokio::spawn sites in windows_event_log/mod.rs are wrapped with .in_current_span() directly to inherit the component tracing span. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…0.58, quick-xml) Adds the windows 0.58 dependency tree required by the sources-windows_event_log feature. Generated via cargo, not hand-edited. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Windows Event Log source's mod.rs uses cfg_if::cfg_if! blocks, but the vector crate only declared cfg-if in [workspace.dependencies], not in its own [dependencies]. Upstream's vector crate already depended on cfg-if, so the squash-merged source PR did not add it — but this fork did not, causing E0433 "unresolved module cfg_if" when building with sources-windows_event_log. Verified with: cargo check -p vector --no-default-features --features sources-windows_event_log Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cherry-picked upstream source assumed a newer edition/toolchain and vector_lib API than this fork provides (edition 2021, rustc 1.88). Cross- compiling to x86_64-pc-windows-gnu surfaced 9 errors in the cfg(windows) code that the host build could not see: - Enable dep:lru in the sources-windows_event_log feature. The vector crate declares lru as optional; the feature never activated it (E0432). - Replace vector_lib::NamedInternalEvent derive (absent in this fork's vector_lib) with manual InternalEvent::name() impls returning the event name, matching the fork's convention (E0432). - Rewrite 5 let-chains (if let X && cond) as nested conditionals; let-chains require edition 2024 but the crate is edition 2021 (E0658). Verified: cargo check --no-default-features --features sources-windows_event_log --target x86_64-pc-windows-gnu now finishes cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cherry-picks the upstream Windows Event Log source (vectordotdev/vector PR #24305, commit
891643ed) into the fork, along with two upstream follow-up bug fixes and one adapted observability fix, plusthe changes required to make it build on this fork's toolchain.
Commits
feat(new source)windows_event_logsource (source, tests, CI workflow, docs). Conflicts resolved: kept fork'swindows-service 0.7.0+wefentries;sources/base/; kept fork'sinternal_events/mod.rslayout.fix: prevent permanent freeze from lost wakeupfix: add source metadatasource_type/timestamp metadata to emitted events.fix: propagate component span into spawned tasksspawn_in_current_span(); used.in_current_span()).chore(deps): update Cargo.lockwindows 0.58dependency tree.fix(deps): add cfg-if dependencycfg_if!; fork'svectorcrate only had it in[workspace.dependencies].fix: make it compile on fork's toolchain/editiondep:lru; replace absentNamedInternalEventderive with manualInternalEvent::name(); rewrite 5 let-chains as nested conditionalsValidation
cargo check --no-default-features --features sources-windows_event_log(host) — clean.x86_64-pc-windows-gnu, mingw-w64) withsources-windows_event_log+sinks-console+transforms-remap— clean.Caveats / follow-ups
make test-integration-windows-event-log).Securitychannel requires Administrator/LocalSystem privileges at runtime.