Skip to content

feat: allow disabling dm-crypt workqueues for encrypted volumes - #1563

Open
lukasmetzner wants to merge 2 commits into
mainfrom
feat/luks-disable-workqueues
Open

lukasmetzner wants to merge 2 commits into
mainfrom
feat/luks-disable-workqueues

Conversation

@lukasmetzner

@lukasmetzner lukasmetzner commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Add the optional encryption secret keys encryption-disable-read-workqueue and encryption-disable-write-workqueue. When set, the corresponding --perf-no_read_workqueue and --perf-no_write_workqueue flags are passed to luksOpen, reducing I/O latency on encrypted volumes. Requires Linux 5.9+.

When work queues are disabled, the tail latency is significantly better. CPU usage moves from kernel threads into the pod's own cgroups, but the overall usage does not change. Users have to account for this when using tight Kubernetes CPU limits. Benchmarks: #1563 (comment)

Enable both keys in the encrypted kubernetes e2e run.

Closes #1535

@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.25%. Comparing base (748235a) to head (4a6cdf2).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1563      +/-   ##
==========================================
+ Coverage   82.17%   83.25%   +1.07%     
==========================================
  Files          16       16              
  Lines        1470     1505      +35     
==========================================
+ Hits         1208     1253      +45     
+ Misses        262      252      -10     
Flag Coverage Δ
e2e-controller 65.11% <94.59%> (+2.73%) ⬆️
e2e-node 65.11% <94.59%> (+2.73%) ⬆️
integration 3.69% <0.00%> (-0.13%) ⬇️
unit 57.34% <72.97%> (+0.47%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@lukasmetzner lukasmetzner changed the title feat: allow disabling dm-crypt workqueues for encrypted volumes wip: feat: allow disabling dm-crypt workqueues for encrypted volumes Nov 13, 2026
@lukasmetzner
lukasmetzner force-pushed the feat/luks-disable-workqueues branch from d37192d to a54649b Compare November 16, 2026 07:54
@lukasmetzner

Copy link
Copy Markdown
Member Author

Benchmark results

fio was used as a benchmarking tool.

fio config used for testing.
    [global]
    ioengine=libaio
    direct=1               ; bypass page cache so you measure the disk, not RAM
    time_based
    runtime=60
    ramp_time=5
    size=4G
    directory=/data
    filename=fio-testfile
    group_reporting
    stonewall              ; run each job one after another, not in parallel

    [rand-read-4k]
    rw=randread
    bs=4k
    iodepth=64
    numjobs=4

    [rand-write-4k]
    rw=randwrite
    bs=4k
    iodepth=64
    numjobs=4

    [seq-read-1m]
    rw=read
    bs=1M
    iodepth=16
    numjobs=1

    [seq-write-1m]
    rw=write
    bs=1M
    iodepth=16
    numjobs=1

    [mixed-70r-30w-4k]
    rw=randrw
    rwmixread=70
    bs=4k
    iodepth=32
    numjobs=4

    [latency-qd1-4k]
    rw=randread
    bs=4k
    iodepth=1
    numjobs=1

Each run (fast and slow) used its own new Hetzner Cloud volume (6 volumes total). Parallel runs (fast/slow) were scheduled with preferred pod anti-affinity on kubernetes.io/hostname.

Ranges overlap means the ranges share values, so some slow runs beat some fast runs.

Metric (slow → fast) Pair 1 Pair 2 Pair 3 Ranges overlap?
rand-write p99 158 → 63 ms 169 → 61 ms 154 → 64 ms No
rand-write p99.9 242 → 77 ms 325 → 75 ms 283 → 88 ms No
mixed write p99.99 36 → 165 ms 173 → 98 ms 171 → 57 ms Yes
seq-write fio CPU (sys) 1.3 → 44.0% 1.5 → 44.1% 1.3 → 40.8% No
rand-read p99 181 → 53 ms 135 → 53 ms 129 → 55 ms No
seq-write p99.9 267 → 271 ms 261 → 287 ms 287 → 292 ms Yes
QD1 read IOPS 828 → 792 876 → 821 841 → 795 No (barely)

Summary

  • Throughput: unchanged. Both hit the Hetzner limits (~7,500 IOPS / ~300 MB/s).
  • Gain: random-write tail latency is ~2.5–4× lower, and the random-read p99 tail is ~2.5× lower.
  • CPU: encryption moves from kernel threads into the pod's own cgroup (~0.45 core at 300 MB/s of writes). Write-heavy pods need CPU headroom; tight limits can throttle I/O.
  • Cons: QD1 read IOPS ~5% lower, and seq-write p99.9 slightly higher.

Raw fio output:

results-fast-1.txt
results-slow-1.txt

results-fast-2.txt
results-slow-2.txt

results-fast-3.txt
results-slow-3.txt

@lukasmetzner
lukasmetzner force-pushed the feat/luks-disable-workqueues branch 3 times, most recently from f72df2b to bb8d7f7 Compare November 16, 2026 13:18
Add the optional encryption secret keys encryption-disable-read-workqueue
and encryption-disable-write-workqueue. When set, the corresponding
--perf-no_read_workqueue and --perf-no_write_workqueue flags are passed to
luksOpen, reducing I/O latency on encrypted volumes. Requires Linux 5.9+.

Enable both keys in the encrypted kubernetes e2e run.

Closes #1535
@lukasmetzner
lukasmetzner force-pushed the feat/luks-disable-workqueues branch from bb8d7f7 to d994684 Compare November 16, 2026 13:19
@lukasmetzner
lukasmetzner force-pushed the feat/luks-disable-workqueues branch from d994684 to 4a6cdf2 Compare November 16, 2026 13:24
@lukasmetzner lukasmetzner changed the title wip: feat: allow disabling dm-crypt workqueues for encrypted volumes feat: allow disabling dm-crypt workqueues for encrypted volumes Nov 18, 2026
@lukasmetzner
lukasmetzner marked this pull request as ready for review November 18, 2026 11:43
@lukasmetzner
lukasmetzner requested a review from a team as a code owner November 18, 2026 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Encrypted volumes: allow opening LUKS devices without dm-crypt workqueues (--perf-no_read_workqueue --perf-no_write_workqueue)

1 participant