Skip to content

Freeze per-connection and per-poll span tags instead of rebuilding them (tag api - part 2) - #13141

Draft
dougqh wants to merge 6 commits into
dougqh/settagsfrom-lock-coarseningfrom
dougqh/memoized-tag-contributor
Draft

dougqh wants to merge 6 commits into
dougqh/settagsfrom-lock-coarseningfrom
dougqh/memoized-tag-contributor

Conversation

@dougqh

@dougqh dougqh commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Adds FrozenTags, a cached form of setTagsFrom: run a TagExtractor / TagContributor once against a source that outlives its spans, freeze the tags it writes, and apply them to each span with span.setTagsFrom(frozen) instead of re-reading the source. Two first users:

  • netty 4.1 client — peer tags (peer.hostname, peer.ipv4/ipv6, peer.port) are frozen once per connection channel (HTTP/2 streams share their parent's). Previously every request rebuilt the IP string (InetAddress.getHostAddress()) and re-resolved the hostname.
  • Kafka 0.11 / 3.8 consumer — destination, group, cluster id and bootstrap servers are frozen once per topic per poll (one-entry cache in the poll's iterator; records come out partition by partition).

Commits, in order:

  1. Move BaseDecorator.onPeerConnection's InetAddress handling into PeerAddress / PeerSocketAddress extractors (no caching yet).
  2. FrozenTags + AgentSpan.setTagsFrom(FrozenTags). On DDSpan it reuses the prototype seed plan: with the dense store, known tags that need no interception are copied in at once.
  3. netty 4.1 client freezes peer tags per connection.
  4. A partial bulk seed: frozen tags often overlap tags the span already has (an HTTP client sets peer.hostname/peer.port from the URL in onURI, then the connection's tags overwrite them). Non-overlapping tags are copied in bulk; the overlapping ones go through the interceptor, as setTag does.
  5. Kafka consumer tags frozen per topic per poll.
  6. Review fixes: replay recorded writes in order (and keep removals); no reverse DNS under the tag lock; key the netty cache on the address object.

Motivation

APMLP-2176: many tag values are constant per framework object but get rebuilt — often as new objects — on every span. Instrumentation stays simple (a plain extractor over the framework object); the core records, freezes and applies.

Behavior

No intended behavior change, including ordering and interception:

  • FrozenTags records each write (same setter, same value, in order). Applying it replays them exactly; the bulk copy of their net result is used only where it's provably the same: no write needs the interceptor (incl. split-by-tags on custom tags), and none removes a tag or sets an empty value. Tags already on the span are overwritten through the interceptor.
  • netty keeps today's double write (URL-derived peer tags, then the connection's) and its final values. The cache is keyed on the remote address object the tags were frozen from, so a channel whose address changes, or a server-accepted child channel, still gets its own address's tags.
  • BaseDecorator.onPeerConnection calls the extractors directly rather than through setTagsFrom, so a reverse DNS lookup never runs under the span's tag lock.
  • The recorder rejects getTag and span-level setters (they can't be replayed against another span).

Additional Notes

Tests

  • FrozenTagsTest, DDSpanFrozenTagsTest + DDSpanFrozenTagsDenseForkedTest (both stores; pins which path each shape takes, incl. the HTTP-client overlap shape, recorded removals, and split-by-tags write order vs. live writes), new partial-seed cases in TagStoresTest, PeerAddressTest, KafkaConsumerTagsTest.
  • netty 4.1 client (V0/V1) and Kafka 0.11/3.8 client suites pass.

Jira ticket: APMLP-2176

🤖 Generated with Claude Code

dougqh and others added 6 commits October 30, 2026 14:52
Move BaseDecorator.onPeerConnection's InetAddress / InetSocketAddress
handling into PeerAddress and PeerSocketAddress extractors, applied with
setTagsFrom so the span's tag lock is taken once per call. This is the
first step toward freezing the extracted tags once per long-lived address
instead of rebuilding them, and the new IP string, on every span.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FrozenTags.of(source, extractor) records the tags an extractor or
contributor writes, once, into a frozen TagMap. span.setTagsFrom(frozen)
then applies them as if the writes were made again, without reading the
source. On DDSpan this reuses the prototype seed plan: with the dense
store, known tags that need no interception and aren't already on the
span are copied in at once; otherwise they go one at a time through the
interceptor, as setTag does.

The recorder rejects reads of the span and span-level setters, since
neither can be replayed against a different span.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The client handler rebuilt peer.hostname / peer.ipv4 / peer.port from the
channel's remote address on every request, allocating a new IP string
each time. The address is fixed for the connection's life, so freeze the
tags on first use, keep them on the connection channel (HTTP/2 streams
share their parent's), and apply them with setTagsFrom(FrozenTags).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frozen tags often overlap tags the span already has: an HTTP client sets
peer.hostname and peer.port from the URL, then the connection's peer
tags overwrite them. Falling back to tag-by-tag for the whole set made
that common shape lose the bulk copy.

Add a partial IdStore.seedAbsent that appends the source tags whose slot
bits are clear and hands the ones that may be present to the caller,
which sets them through the interceptor as setTag does. The result is
the same as the recorded writes; only the overlapping tags pay for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each consumed record's span got messaging.destination.name, kafka.group,
messaging.kafka.cluster.id and messaging.kafka.bootstrap.servers set one
by one, though all four are the same for every record of a topic in a
poll. The poll's iterator now freezes them per topic (records come out
partition by partition, so one cached topic suffices) and applies them
with setTagsFrom(FrozenTags), at the same point in onConsume as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fixes, keeping FrozenTags exactly equivalent to the live writes:

- FrozenTags records each write (same setter, same value) and replays
  them in order on every tag-by-tag path. Split-by-tags is last-write-
  wins, so replaying in store order could pick a different service. A
  recorded null or empty value now replays as written instead of being
  dropped, and keeps the set off the bulk path, as does a custom tag that
  split-by-tags routes.
- BaseDecorator.onPeerConnection calls the peer extractors directly:
  through setTagsFrom, the hostname's reverse DNS lookup ran under the
  span's tag lock.
- netty reads the address from the request's channel as before, shares
  the connection's frozen tags only when the parent reports the same
  address object (HTTP/2 streams, not a server channel), and refreezes
  when the address changes.
- Rename the partial seed to IdStore.seedPartial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added comp: core Tracer core tag: no release notes Changes to exclude from release notes type: refactoring inst: kafka Kafka instrumentation inst: netty Netty instrumentation tag: ai generated Largely based on code generated by an AI or LLM labels Oct 30, 2026
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 72.40%
• Overall Coverage: 59.86% (+0.03%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3ddcf66 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.76 s 14.66 s [-0.1%; +1.4%] (no difference)
startup:insecure-bank:tracing:Agent 13.62 s 13.64 s [-0.9%; +0.6%] (no difference)
startup:petclinic:appsec:Agent 17.07 s 16.41 s [-0.5%; +8.5%] (no difference)
startup:petclinic:iast:Agent 16.91 s 16.95 s [-1.0%; +0.5%] (no difference)
startup:petclinic:profiling:Agent 16.59 s 16.84 s [-2.6%; -0.4%] (maybe better)
startup:petclinic:sca:Agent 16.56 s 16.88 s [-6.3%; +2.6%] (no difference)
startup:petclinic:tracing:Agent 16.11 s 16.13 s [-1.2%; +0.9%] (no difference)

Commit: 3ddcf664 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@pr-commenter

pr-commenter Bot commented Oct 30, 2026

Copy link
Copy Markdown

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch dougqh/settagsfrom-lock-coarsening dougqh/memoized-tag-contributor
git_commit_date 1793141804 1793394419
git_commit_sha 0d2a9e0 3ddcf66
See matching parameters
Baseline Candidate
ci_job_date 1793395595 1793395595
ci_job_id 2198467220 2198467220
ci_pipeline_id 149115302 149115302
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-vd8of0up 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-vd8of0up 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 2 metrics, 0 unstable metrics.

scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce worse
[-74116.552op/s; -18929.163op/s] or [-4.028%; -1.029%]
See unchanged results
scenario Δ mean throughput
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce same

@pr-commenter

pr-commenter Bot commented Oct 30, 2026

Copy link
Copy Markdown

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch dougqh/settagsfrom-lock-coarsening dougqh/memoized-tag-contributor
git_commit_date 1793141804 1793394419
git_commit_sha 0d2a9e0 3ddcf66
See matching parameters
Baseline Candidate
ci_job_date 1793395650 1793395650
ci_job_id 2198467222 2198467222
ci_pipeline_id 149115302 149115302
cpu_model Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-0xjtcbxv 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-0xjtcbxv 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume same

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core inst: kafka Kafka instrumentation inst: netty Netty instrumentation tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant