Skip to content

Get rid of a variable shift on the tcmalloc alloc fast path. - #621

Merged
copybara-service[bot] merged 1 commit into
masterfrom
test_973845047
Sep 1, 2026
Merged

Get rid of a variable shift on the tcmalloc alloc fast path.#621
copybara-service[bot] merged 1 commit into
masterfrom
test_973845047

Conversation

@copybara-service

Copy link
Copy Markdown

Get rid of a variable shift on the tcmalloc alloc fast path.

When converting sizes into table indexes (to lookup further into
the size class), the fast path does a shift by 3 and the slow path
a shift by 7. Clang joins the two paths into a variable-shift path,
which is very slow on certain x86 CPUs. Since this is on the hot path
for us, add a dummy asm() statement to make sure the two are properly
separated; it also gets rid of some move instructions and register
pressure, but this is probably less important.

Skylake microbenchmarks:

name CYCLES/op CYCLES/op vs base
BM_new_sized_delete/1 17.04 ± 0% 15.07 ± 0% -11.55% (p=0.000 n=20)
BM_new_sized_delete/8 17.05 ± 0% 15.08 ± 0% -11.57% (p=0.000 n=20)
BM_new_sized_delete/64 17.16 ± 0% 15.19 ± 0% -11.44% (p=0.000 n=20)
BM_new_sized_delete_fence/1 49.09 ± 0% 46.14 ± 0% -6.01% (p=0.000 n=20)
BM_new_sized_delete_fence/8 49.10 ± 0% 46.15 ± 0% -6.02% (p=0.000 n=20)
BM_new_sized_delete_fence/64 49.19 ± 0% 46.16 ± 0% -6.17% (p=0.000 n=20)

Milan is less dramatic (shr reg, cl is particularly bad on Skylake),
but still a win:

name CYCLES/op CYCLES/op vs base
BM_new_sized_delete/1 15.01 ± 7% 15.01 ± 7% ~ (p=0.100 n=20)
BM_new_sized_delete/8 14.51 ± 3% 15.01 ± 7% ~ (p=0.212 n=20)
BM_new_sized_delete/64 15.05 ± 6% 15.07 ± 7% ~ (p=0.310 n=20)
BM_new_sized_delete_fence/1 41.01 ± 0% 40.01 ± 0% -2.44% (p=0.000 n=20)
BM_new_sized_delete_fence/8 41.02 ± 0% 40.02 ± 0% -2.44% (p=0.000 n=20)
BM_new_sized_delete_fence/64 41.09 ± 0% 40.09 ± 0% -2.45% (p=0.000 n=20)

When converting sizes into table indexes (to lookup further into
the size class), the fast path does a shift by 3 and the slow path
a shift by 7. Clang joins the two paths into a variable-shift path,
which is very slow on certain x86 CPUs. Since this is on the hot path
for us, add a dummy asm() statement to make sure the two are properly
separated; it also gets rid of some move instructions and register
pressure, but this is probably less important.

Skylake microbenchmarks:

name                              CYCLES/op     CYCLES/op        vs base
BM_new_sized_delete/1               17.04 ±  0%    15.07 ±  0%  -11.55% (p=0.000 n=20)
BM_new_sized_delete/8               17.05 ±  0%    15.08 ±  0%  -11.57% (p=0.000 n=20)
BM_new_sized_delete/64              17.16 ±  0%    15.19 ±  0%  -11.44% (p=0.000 n=20)
BM_new_sized_delete_fence/1         49.09 ±  0%    46.14 ±  0%   -6.01% (p=0.000 n=20)
BM_new_sized_delete_fence/8         49.10 ±  0%    46.15 ±  0%   -6.02% (p=0.000 n=20)
BM_new_sized_delete_fence/64        49.19 ±  0%    46.16 ±  0%   -6.17% (p=0.000 n=20)

Milan is less dramatic (shr reg, cl is particularly bad on Skylake),
but still a win:

name                               CYCLES/op    CYCLES/op        vs base
BM_new_sized_delete/1               15.01 ±  7%    15.01 ±  7%        ~ (p=0.100 n=20)
BM_new_sized_delete/8               14.51 ±  3%    15.01 ±  7%        ~ (p=0.212 n=20)
BM_new_sized_delete/64              15.05 ±  6%    15.07 ±  7%        ~ (p=0.310 n=20)
BM_new_sized_delete_fence/1         41.01 ±  0%    40.01 ±  0%   -2.44% (p=0.000 n=20)
BM_new_sized_delete_fence/8         41.02 ±  0%    40.02 ±  0%   -2.44% (p=0.000 n=20)
BM_new_sized_delete_fence/64        41.09 ±  0%    40.09 ±  0%   -2.45% (p=0.000 n=20)

PiperOrigin-RevId: 974576006
@copybara-service
copybara-service Bot merged commit 967e0ca into master Sep 1, 2026
3 checks passed
@copybara-service
copybara-service Bot deleted the test_973845047 branch September 1, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant