Skip to content

feat: Adding a connection pool to reduce the time spent opening connection to Postgres#4851

Open
Ma77Ball wants to merge 11 commits intoapache:mainfrom
Ma77Ball:fix/connection-pool
Open

feat: Adding a connection pool to reduce the time spent opening connection to Postgres#4851
Ma77Ball wants to merge 11 commits intoapache:mainfrom
Ma77Ball:fix/connection-pool

Conversation

@Ma77Ball
Copy link
Copy Markdown
Contributor

@Ma77Ball Ma77Ball commented May 3, 2026

What changes were proposed in this PR?

This PR adds a HikariCP connection pool to SqlServer.scala so that jOOQ queries borrow pre-authenticated connections from a pool instead of opening a new TCP connection and performing SCRAM-SHA-256 authentication on every database call.

What changed in SqlServer.scala:

  • Added a HikariConfig with maximumPoolSize=10, minimumIdle=2, connectionTimeout=30s, idleTimeout=10min, maxLifetime=30min (chosen to stay below typical PostgreSQL idle and load-balancer connection-reaping windows)
  • Initialized a HikariDataSource and built the jOOQ DSLContext via DSL.using(dataSource, SQL_DIALECT) so queries draw from the pool
  • Added a close() method to shut down the pool on application or test teardown, and updated clearInstance() to call it, so test classes that replace the singleton don't leak pool threads
  • Added a documented warning against caching the DSLContext in a val or lazy val. MockTexeraDB swaps the singleton between test classes, and a cached context would hold a stale reference to a dead pool

Any related issues, documentation, discussions?

closes: #4852

How was this PR tested?

  • Existing unit and integration tests pass — the pool-backed DSLContext is a drop-in replacement for the previous direct-driver context
  • Verified the MockTexeraDB flow still works across test classes by running the full test suite and confirming no "Connection refused" failures from stale contexts

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude opus 4.6 (Anthropic) in compliance with ASF

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file fix common labels May 3, 2026
@Ma77Ball Ma77Ball changed the title Adding a connection pool to reduce the time spent opening connection to Postgres refactor: Adding a connection pool to reduce the time spent opening connection to Postgres May 3, 2026
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 42.17%. Comparing base (056e39d) to head (c37711b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #4851   +/-   ##
=========================================
  Coverage     42.16%   42.17%           
  Complexity     2180     2180           
=========================================
  Files           980      980           
  Lines         36305    36297    -8     
  Branches       3783     3783           
=========================================
  Hits          15309    15309           
+ Misses        20068    20060    -8     
  Partials        928      928           
Flag Coverage Δ
access-control-service 39.53% <ø> (ø)
computing-unit-managing-service 0.00% <ø> (ø)
config-service 0.00% <ø> (ø)
file-service 33.24% <ø> (ø)
workflow-compiling-service 47.72% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Contributor

@Yicong-Huang Yicong-Huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Left inline comments

Comment thread common/dao/src/main/scala/org/apache/texera/dao/SqlServer.scala
Comment thread common/dao/src/main/scala/org/apache/texera/dao/SqlServer.scala Outdated
Comment thread common/dao/build.sbt Outdated
Comment thread common/dao/src/main/scala/org/apache/texera/dao/SqlServer.scala Outdated
@Yicong-Huang
Copy link
Copy Markdown
Contributor

please also change the title. this is not a refactor. we are adding a new feature.

@Ma77Ball Ma77Ball changed the title refactor: Adding a connection pool to reduce the time spent opening connection to Postgres feat: Adding a connection pool to reduce the time spent opening connection to Postgres May 4, 2026
@github-actions github-actions Bot added the platform Non-amber Scala service paths label May 4, 2026
@Ma77Ball Ma77Ball requested a review from Yicong-Huang May 5, 2026 00:03
Comment thread common/dao/build.sbt Outdated
@Ma77Ball Ma77Ball requested a review from Yicong-Huang May 7, 2026 01:59
Copy link
Copy Markdown
Contributor

@Yicong-Huang Yicong-Huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks

@Yicong-Huang Yicong-Huang enabled auto-merge (squash) May 7, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common dependencies Pull requests that update a dependency file fix platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add HikariCP connection pool to SqlServer

3 participants