Enable Memory & CPU Profiling for df_engine by sapatrjv · Pull Request #2420 · open-telemetry/otel-arrow

sapatrjv · 2026-03-24T04:49:53Z

Change Summary

PR to enable CPU & Memory profiling for df_engine.

What issue does this PR close?

Fixes #2296
Fixes #2297
Fixes #2298
Fixes #2299

:)

How are these changes tested?

Are there any user-facing changes?

N/A

codecov · 2026-03-24T04:54:06Z

Codecov Report

❌ Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.10%. Comparing base (baac519) to head (7d2a092).
⚠️ Report is 2 commits behind head on main.

❌ Your patch status has failed because the patch coverage (0.00%) is below the target coverage (70.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2420      +/-   ##
==========================================
- Coverage   88.10%   88.10%   -0.01%     
==========================================
  Files         633      633              
  Lines      237176   237191      +15     
==========================================
- Hits       208975   208967       -8     
- Misses      27677    27700      +23     
  Partials      524      524

Components	Coverage Δ
otap-dataflow	`89.78% <0.00%> (-0.02%)`	⬇️
query_abstraction	`80.61% <ø> (ø)`
query_engine	`90.72% <ø> (ø)`
otel-arrow-go	`52.45% <ø> (ø)`
quiver	`92.27% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…sapatrjv/otel-arrow into dev/sapatr/memprofilingwithdhat

lalitb · 2026-03-24T05:12:44Z

 sysinfo.workspace = true
+dhat = "0.3.3"
+ctrlc = "3"
+once_cell = "1"


Could of points here

These 3 crates should be optional and enabled only when dhat-heap feature is enabled.

Since this crate already targets Rust 1.87, could we use std::sync::LazyLock here instead of adding once_cell just for this static?

This reverts commit d23af0e.

cijothomas · 2026-03-25T06:20:35Z

Does this closes the issue #2304 ?

lalitb · 2026-03-25T07:35:11Z

Does this closes the issue #2304 ?

I don’t think this closes #2304. That issue is about One-Collect support for profiling other processes, while this PR seems focused on CPU/memory profiling for df_engine itself.

jmacd · 2026-03-25T16:47:55Z

The issue this fixes: #2297. I modified the PR description.

lquerel

I think there's a piece of code that needs to be put back, and a potential follow-up PR to track in a GitHub issue.

…sapatrjv/otel-arrow into dev/sapatr/memprofilingwithdhat

reyang · 2026-04-03T07:17:15Z

+
+> [!NOTE]
+> In this command, all default features are disabled.
+> Use specific flags to enable individual features.


Suggested change

> Use specific flags to enable individual features.

Use specific flags to enable individual features.

In README,md file similar format usage is there. Here is an example.

Example:

Here is the rendered output:

@sapatrjv which one should we use?

Copilot response on it:

Copilot response on it:

and what's your take?

In my opinion, both approaches will work in this scenario. It will differ when a sentence needs to appear on a separate line. In this case add a blank line with > and then include the next sentence.

jmacd · 2026-04-03T15:53:58Z

I had to approve the workflows. @sapatrjv, please accept the invitation that was sent after open-telemetry/community#3352.

…sapatrjv/otel-arrow into dev/sapatr/memprofilingwithdhat

cijothomas · 2026-04-15T01:16:00Z

+
+> [!NOTE]
+> `dhat` needs a clean shutdown to generate `dhat-heap.json` file.
+> In `df_engine` this can be done manually with Ctrl+C.


does Ctrl+C do a clean shutdown? The engine is not yet configured to react to it...

It does not handle clean shutdown with Ctrl+C. Here is PR #2325 that you own to handle it.

Until that PR (or another PR which adds this) is merged, lets not mention that engine can do clean shutdown with Ctrl+C

@sapatrjv

…default malloc selection (#2695) # Change Summary Enables clippy in the conditional for Jemalloc in src/main.rs, with @sapatrjv. ## What issue does this PR close? Introduced in #2420. Adds `--all-features --workspace -- -D warnings` to the clippy step in `cargo xtask check`. https://cloud-native.slack.com/archives/C08RRSJR7FD/p1776408892537689 ## How are these changes tested? Ran `cargo xtask clippy` on Ubuntu, not sure this actually works in Windows. ## Are there any user-facing changes? No

Memory & CPU Profiling.

c294b16

sapatrjv requested a review from a team as a code owner March 24, 2026 04:49

github-project-automation Bot added this to OTel-Arrow Mar 24, 2026

github-actions Bot added the rust Pull requests that update Rust code label Mar 24, 2026

sapatrjv added 2 commits March 23, 2026 21:55

Enable Memory & CPU profiling for df_engine.

062e1fb

Merge branch 'main' into dev/sapatr/memprofilingwithdhat

f79dffd

lalitb reviewed Mar 24, 2026

View reviewed changes

Comment thread rust/otap-dataflow/src/main.rs Outdated

sapatrjv added 2 commits March 23, 2026 22:11

Enable Memory & CPU Profiling for df_engine

1ab21be

Merge branch 'dev/sapatr/memprofilingwithdhat' of https://github.com/…

8c6daa6

…sapatrjv/otel-arrow into dev/sapatr/memprofilingwithdhat

lalitb reviewed Mar 24, 2026

View reviewed changes

sapatrjv added 6 commits March 23, 2026 22:13

Enable Memory & CPU Profiling for df_engine

d23af0e

Revert "Enable Memory & CPU Profiling for df_engine"

ad71e6b

This reverts commit d23af0e.

Enable Memory & CPU Profiling for df_engine

7482d31

Enable Memory & CPU Profiling for df_engine

aebde81

Enable Memory & CPU Profiling for df_engine

cc9e748

Merge branch 'main' into dev/sapatr/memprofilingwithdhat

9222922

jmacd reviewed Mar 25, 2026

View reviewed changes

Comment thread rust/otap-dataflow/src/main.rs Outdated

lalitb reviewed Mar 25, 2026

View reviewed changes

Comment thread rust/otap-dataflow/src/main.rs Outdated

Enable Memory & CPU Profiling for df_engine

6568f0d

lquerel previously requested changes Mar 26, 2026

View reviewed changes

Comment thread rust/otap-dataflow/src/main.rs

Comment thread rust/otap-dataflow/src/main.rs Outdated

sapatrjv added 6 commits March 30, 2026 12:12

Enable Memory & CPU Profiling for df_engine.

a87c8fe

Enable Memory & CPU Profiling for df_engine.

157f5ab

Enable Memory & CPU Profiling for df_engine

d73b21e

Merge branch 'main' into dev/sapatr/memprofilingwithdhat

eed136f

Enable Memory & CPU Profiling for df_engine

34c417c

Merge branch 'dev/sapatr/memprofilingwithdhat' of https://github.com/…

0ad845f

…sapatrjv/otel-arrow into dev/sapatr/memprofilingwithdhat

Enable Memory & CPU Profiling for df_engine

e533701

lalitb reviewed Apr 3, 2026

View reviewed changes

Comment thread rust/otap-dataflow/PROFILING.md

sapatrjv added 3 commits April 2, 2026 21:18

Enable Memory & CPU Profiling for df_engine

0358fa0

Enable Memory & CPU Profiling for df_engine

06111a1

Enable Memory & CPU Profiling for df_engine

ebdf8d4

reyang reviewed Apr 3, 2026

View reviewed changes

Comment thread rust/otap-dataflow/PROFILING.md

jmacd enabled auto-merge April 3, 2026 15:52

Enable Memory & CPU Profiling for df_engine

cd8bf90

auto-merge was automatically disabled April 3, 2026 17:51
Head branch was pushed to by a user without write access

jmacd enabled auto-merge April 3, 2026 19:54

jmacd marked this pull request as draft April 9, 2026 16:03

auto-merge was automatically disabled April 9, 2026 16:03
Pull request was converted to draft

sapatrjv and others added 5 commits April 14, 2026 15:43

Enable Memory & CPU Profiling for df_engine.

1b4185f

Merge branch 'main' into dev/sapatr/memprofilingwithdhat

600d336

Merge branch 'main' into dev/sapatr/memprofilingwithdhat

a6dad10

Enable Memory & CPU Profiling for df_engine

658e151

Merge branch 'dev/sapatr/memprofilingwithdhat' of https://github.com/…

285883f

…sapatrjv/otel-arrow into dev/sapatr/memprofilingwithdhat

cijothomas reviewed Apr 15, 2026

View reviewed changes

sapatrjv and others added 3 commits April 14, 2026 19:42

Remove code coverage on profiling path.

787bfa5

Remove code coverage on profiling path.

7c81041

Merge branch 'main' into dev/sapatr/memprofilingwithdhat

7d2a092

jmacd marked this pull request as ready for review April 15, 2026 16:42

jmacd added this pull request to the merge queue Apr 15, 2026

Merged via the queue into open-telemetry:main with commit 60ceaa2 Apr 15, 2026
76 of 79 checks passed

github-project-automation Bot moved this to Done in OTel-Arrow Apr 15, 2026

jmacd mentioned this pull request Apr 17, 2026

cargo xtask check: synchronize rust-ci.yaml and clippy settings, fix default malloc selection; avoid openss-sys in Windows build #2695

Merged

	> Use specific flags to enable individual features.
	Use specific flags to enable individual features.

Conversation

sapatrjv commented Mar 24, 2026 • edited by jmacd Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Summary

What issue does this PR close?

How are these changes tested?

Are there any user-facing changes?

Uh oh!

codecov Bot commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cijothomas commented Mar 25, 2026

Uh oh!

lalitb commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmacd commented Mar 25, 2026

Uh oh!

Uh oh!

Uh oh!

lquerel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sapatrjv Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jmacd commented Apr 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sapatrjv Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sapatrjv commented Mar 24, 2026 •

edited by jmacd

Loading

codecov Bot commented Mar 24, 2026 •

edited

Loading

lalitb commented Mar 25, 2026 •

edited

Loading

sapatrjv Apr 3, 2026 •

edited

Loading

sapatrjv Apr 15, 2026 •

edited

Loading