tokio-rt-worker panic causing crash w/o restarting Kubernetes Pod

[gl-jkeys]

PgDog version
e.g., v0.1.34 or a41e41d1

Helm chart v0.58

Description
pgdog issues a tokio panic, but then failed to restart the affected pod. This left our environment in a bad state, and required a manual restart of the underlying Kubernetes Pod to remediate.

Logs
Attach trace logs, if the issue can be reproduced locally. To get trace logs, set RUST_LOG=trace, e.g.:

thread 'tokio-rt-worker' (7) panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/time/entry.rs:181:13:
mark_pending called when the timer entry is in an invalid state

Configuration

pgdog.toml:
----
[general]
port                      =   5432
workers                   =   2
default_pool_size         =   10
log_format                =   "text"
log_level                 =   "info"
pooler_mode               =   "transaction"
healthcheck_interval      =   30_000
idle_healthcheck_interval =   30_000
idle_healthcheck_delay    =   5_000
healthcheck_timeout       =   5_000
rollback_timeout          =   5_000
load_balancing_strategy   =   "round_robin"
read_write_strategy       =   "conservative"
tls_client_required       =   false
tls_verify                =   "prefer"
shutdown_timeout          =   60_000
prepared_statements       =   "extended"
passthrough_auth          =   "disabled"
connect_timeout           =   5_000
connect_attempts          =   1
connect_attempt_delay     =   0
checkout_timeout          =   5_000
dry_run                   =   false
client_idle_timeout       =   60000
mirror_queue              =   128
mirror_exposure           =   1.0
auth_type                 =   "scram"
cross_shard_disabled      =   false
openmetrics_port = 9090
openmetrics_namespace = "pgdog_"
server_lifetime = 86400000
log_connections = true
log_disconnections = true
expanded_explain = false

[levkk]

Hello! Please upgrade to latest PgDog version - I think this might be a bug in Tokio. We've seen something like this before in #1052. The latest version of PgDog also uses the latest version of Tokio.

On the issue of restart: it's not usually necessary; Tokio will re-spawn the thread and continue to serve traffic. I suspect the bad state in your case came from the applications misbehaving when their requests get interrupted, but I can't be sure. It would be good to have more details on what actually happened, e.g.,:

1. Couldn't create new connections to pgdog
2. Existing clients crashed

Thanks!

[gl-jkeys]

Thanks for the quick response as always! We will upgrade ASAP.

Regarding the bad state: we saw this panic and, after it happened, all requests from every client began to fail with: connect ETIMEDOUT (NodeJS applications). That seems to indicate that the requests are being issued correctly, but the TCP packets are being swallowed upstream of the client?

[levkk]

That's a bad one. I suspect Tokio landed in an unrecoverable state. Typically, a panic inside a thread is fine, Tokio just spawns another one, but a bug in the runtime can certainly stop all requests from flowing through.

Let me know if this happens again after you upgrade. Let's keep this issue open in the meantime.

[gl-jkeys]

Quick update. So far so good on the update to Helm chart 0.68. We have not observed any additional tokio panics since then.

I will keep this issue open a while longer, but so far, it does appear that the more recent version is stable for us.

(Side question: I noticed that the chart uses the latest tag for the pgdog repo. Does that mean the Helm chart installs are not idempotent, in the sense of which pgdog version gets installed?)

[levkk]

(Side question: I noticed that the chart uses the latest tag for the pgdog repo. Does that mean the Helm chart installs are not idempotent, in the sense of which pgdog version gets installed?)

Our chart? No, it uses appVersion: https://github.com/pgdogdev/helm/blob/main/Chart.yaml#L4. Fwiw, you should just tag the image you want like this:

image:
   tag: v0.1.48

[gl-jkeys]

Ah okay, thanks. I must've missed that. We will definitely pin to the current version we're on once we've validated its stability.

Thanks again for all your help!!

[levkk]

Anytime!