Skip to content

perf: Skip BigQuery schema diff for unchanged payloads#3589

Draft
ruslandoga wants to merge 3 commits into
Logflare:mainfrom
ruslandoga:ruslandoga+conductor/ruslandoga-conductor/schema-builder-change-flag
Draft

perf: Skip BigQuery schema diff for unchanged payloads#3589
ruslandoga wants to merge 3 commits into
Logflare:mainfrom
ruslandoga:ruslandoga+conductor/ruslandoga-conductor/schema-builder-change-flag

Conversation

@ruslandoga

@ruslandoga ruslandoga commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

NOT YET READY FOR REVIEW


This PR:

  • Adds SchemaBuilder.build_table_schema_with_change/2 so callers can reuse the existing schema and know whether a payload changed it.
  • Updates BigQuery schema planning to use that change flag instead of rebuilding and then doing a full schema equality diff for unchanged payloads.
  • Preserves build_table_schema/2 behavior and adds focused tests for changed and unchanged cases.
  • Persists the branch profiler benchmark snapshot showing the reduced noop work.

@ruslandoga ruslandoga changed the title Skip BigQuery schema diff for unchanged payloads perf: Skip BigQuery schema diff for unchanged payloads Jun 11, 2026
@ruslandoga

ruslandoga commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author
$ MIX_ENV=test LABEL="schema-builder-change-flag-vs-baseline" MACHINE="m2-local" mix run test/profiling/bq_schema_update_internals_bench.exs

##### With input edge log #####
Name             ips        average  deviation         median         99th %
update      105.52 K        9.48 μs   ±451.54%           9 μs       18.33 μs
noop         81.61 K       12.25 μs   ±273.77%       11.67 μs       18.04 μs

Memory usage statistics:
update        17.01 KB
noop          11.51 KB

Reduction count statistics:
update          1.85 K
noop            1.12 K

##### With input lists #####
Name             ips        average  deviation         median         99th %
noop        280.31 K        3.57 μs   ±451.53%        3.29 μs        6.71 μs
update      263.12 K        3.80 μs   ±442.34%        3.54 μs        5.33 μs

Memory usage statistics:
noop           4.61 KB
update         7.04 KB

Reduction count statistics:
noop               464
update             824

##### With input scalars #####
Name             ips        average  deviation         median         99th %
noop        208.36 K        4.80 μs   ±259.13%        4.50 μs        8.63 μs
update      206.18 K        4.85 μs   ±272.51%        4.50 μs       12.67 μs

Memory usage statistics:
noop           5.45 KB
update         8.23 KB

Reduction count statistics:
noop               532
update             939

Delta vs 14d2094d0 (2026-06-02T13:02:23.013962Z):
  noop / edge log
    ips:  34.2K ips → 81.6K ips  (+138.8%)
    wall: 26.58 → 11.67 µs  (-56.1%)
    mem:  35.7 KB → 11.5 KB  (-67.8%)
    reds: 2.9K → 1.1K  (-61.7%)
  noop / lists
    ips:  120.6K ips → 280.3K ips  (+132.4%)
    wall: 7.42 → 3.29 µs  (-55.7%)
    mem:  13.4 KB → 4.6 KB  (-65.5%)
    reds: 1.3K → 464  (-65.1%)
  noop / scalars
    ips:  83.2K ips → 208.4K ips  (+150.4%)
    wall: 10.54 → 4.5 µs  (-57.3%)
    mem:  17.2 KB → 5.4 KB  (-68.4%)
    reds: 1.5K → 532  (-65.4%)
  update / edge log
    ips:  29.9K ips → 105.5K ips  (+252.8%)
    wall: 32.08 → 9.0 µs  (-71.9%)
    mem:  40.7 KB → 17.0 KB  (-58.2%)
    reds: 3.2K → 1.8K  (-42.4%)
  update / lists
    ips:  109.1K ips → 263.1K ips  (+141.2%)
    wall: 8.17 → 3.54 µs  (-56.7%)
    mem:  14.5 KB → 7.0 KB  (-51.5%)
    reds: 1.4K → 824  (-42.3%)
  update / scalars
    ips:  75.0K ips → 206.2K ips  (+174.8%)
    wall: 11.83 → 4.5 µs  (-62.0%)
    mem:  18.5 KB → 8.2 KB  (-55.4%)
    reds: 1.6K → 939  (-40.7%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant