Enable PostgreSQL as the default database for rhdh-local - #324
NiallTwomey2 wants to merge 11 commits into
Conversation
| $TOOL exec -e PGPASSWORD=postgres db \ | ||
| psql -U postgres -d postgres -c '\l' |
There was a problem hiding this comment.
Run postgreSQL inside the db container.
| if [ -z "$plugin_dbs" ]; then | ||
| echo "ERROR: Postgres is up but no backstage_plugin_* databases were found" >&2 | ||
| echo "RHDH is probably using SQLite, or never migrated plugins." >&2 | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
If there are no backstage plugins inside $plugin_dbs, then issue an error message and exit 1 as a failure
Scenario: Postgres could be running inside the container, but RHDH never created plugin databases, due to databases never being migrated or SQLite is still marked as the default database.
| if ! echo "$plugin_dbs" | grep -qx 'backstage_plugin_catalog'; then | ||
| echo "ERROR: missing required database backstage_plugin_catalog" >&2 | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
The backstage plugin catalog must exist within the backstage plugins pulled from "$plugin_dbs", if this is not the case, this will result in an error
| env: | ||
| TOOL: ${{ inputs.container_tool }} | ||
| run: | | ||
| set -euo pipefail |
There was a problem hiding this comment.
Ensures that if there are any failures in the bash script, it exits immediately
|
|
||
| plugin_dbs=$($TOOL exec -e PGPASSWORD=postgres db \ | ||
| psql -U postgres -d postgres -Atc \ | ||
| "SELECT datname FROM pg_database WHERE datname LIKE 'backstage_plugin_%' ORDER BY 1;") |
There was a problem hiding this comment.
Querying to find existing PostgreSQL column for datname within the system catalog pg_database.
List the database names that begin with backstage_plugin_.
| environment: | ||
| - HTTP_PROXY=http://proxy:3128 | ||
| - HTTPS_PROXY=http://proxy:3128 | ||
| - NO_PROXY=localhost,127.0.0.1,db |
There was a problem hiding this comment.
With Postgres now being the default database, RHDH connects to PostgreSQL via port 5432 / TCP port.
Without NO_PROXY, node traffic still sees HTTP_PROXY and can try to access port 5432 through Squid (Squid only operates with HTTP not TCP for PostgreSQL), as a result database connections to PostgreSQL may fail under timeouts.
With NO_PROXY as the default, RHDH connects traffic to db via the Compose service on TCP port 5432.
There was a problem hiding this comment.
| - NO_PROXY=localhost,127.0.0.1,db | |
| - NO_PROXY=localhost,127.0.0.1 |
I don't think this is needed. Instead, I would suggest adding db to the internal_net network instead, so it is reachable from rhdh and rhdh won't need to bypass the proxy at all for this. This would actually mimic a real corporate setup: the DB is usually only accessed only from the internal network.
| volumes: | ||
| - "postgresqldata:/var/lib/pgsql/data" |
There was a problem hiding this comment.
postgresqldata volume holds PostgreSQL's data files on the volume disk.
Data and files information is retained to the postgresqldata volume, even after the overall db container is deleted.
compose down --volumes will remove the volume and the data.
There was a problem hiding this comment.
| volumes: | |
| - "postgresqldata:/var/lib/pgsql/data" | |
| volumes: | |
| - "/var/lib/pgsql/data" |
One of the use cases of RHDH Local is to test different versions of RHDH, so this might mean switching from a new version to an older one. And since RHDH currently has issues with downgrades, a named volume might be an issue. The compose-with-db.yaml handles this using an anonymous volume and it makes sense in this context:
rhdh-local/compose-with-db.yaml
Line 17 in ea366db
| healthcheck: | ||
| test: ["CMD", "pg_isready", "-U", "postgres"] | ||
| interval: 5s | ||
| timeout: 5s | ||
| retries: 5 |
There was a problem hiding this comment.
Every 5 seconds, the live healthcheck, inside the container will check to see if the status of PostgreSQL accepting connections is still successful, which indicates that the database service is healthy for RHDH.
| install-dynamic-plugins: | ||
| condition: service_completed_successfully | ||
| db: | ||
| condition: service_healthy |
There was a problem hiding this comment.
RHDH depends on the healthy status connection of the db service, otherwise it will not run, this comes from the healthcheck from the db service.
| @@ -51,6 +51,23 @@ fi | |||
| # Loaded after the patched default config (SQLite) and before user local config. | |||
| EXTRA_CONFIGS="" | |||
| if [[ "${WITH_POSTGRES:-}" == "true" ]]; then | |||
There was a problem hiding this comment.
Waits for PostgreSQL to accept TCP connections
| PG_HOST="${POSTGRES_HOST:-db}" | ||
| PG_PORT="${POSTGRES_PORT:-5432}" | ||
| PG_TIMEOUT="${POSTGRES_WAIT_TIMEOUT:-60}" |
There was a problem hiding this comment.
Specifying the database (PostgreSQL), the port - 5432, and the timeout wait time for connection - 60
rm3l
left a comment
There was a problem hiding this comment.
@NiallTwomey2 Looks like CI is failing on this PR. Could you please take a look?
|
@rm3l The CI has passed. Podman does not cause the CI checks to fail anymore. |
| echo "[$(date)] RHDH is ready" | ||
| curl -i --insecure http://localhost:7007 | ||
|
|
||
| - name: Assert Backstage plugin databases exist in PostgreSQL |
There was a problem hiding this comment.
Why do we need this assertion? IMO this is already guaranteed/tested by RHDH/Backstage, so no need to test this here again. And this test might be fragile if later on we decide to change the pluginDivisionMode or if we change the default backstage_plugin_ DB prefix.
If your goal is to make sure we are using PostgreSQL OOTB, I think you can just verify that the effective RHDH configuration has the right info to connect to PostgreSQL. But tbh, I don't think this is needed.
There was a problem hiding this comment.
I agree, that assertion is an unneeded fail safe, that makes sense to leave this assertion out.
This assertion was to address the failure of the database, caused by missing backstage plugin databases in a container - RHIDP-16884
| echo "ERROR: Postgres is up but no backstage_plugin_* databases were found" >&2 | ||
| echo "RHDH is probably using SQLite, or never migrated plugins." >&2 |
There was a problem hiding this comment.
it doesn't necessarily mean that we are using SQLite; depending on the config, we might still be using PostgreSQL but with all plugins sharing a single database.
| # comment out the following 'database' section to use the PostgreSQL database | ||
| database: | ||
| client: better-sqlite3 | ||
| connection: ":memory:" |
There was a problem hiding this comment.
The goal is to make PostgreSQL the default, so the default app-config should reflect that IMO.
|
|
||
| db: | ||
| container_name: db | ||
| image: "${POSTGRES_IMAGE:-registry.redhat.io/rhel10/postgresql-18:latest}" # dclint disable-line service-image-require-explicit-tag |
There was a problem hiding this comment.
| image: "${POSTGRES_IMAGE:-registry.redhat.io/rhel10/postgresql-18:latest}" # dclint disable-line service-image-require-explicit-tag | |
| image: "${POSTGRES_IMAGE:-quay.io/fedora/postgresql-18:latest}" # dclint disable-line service-image-require-explicit-tag |
The default DB image should be the public one that doesn't require any auth, and then we should document how people can use a commercially-supported version of it from registry.redhat.io.
| ``` | ||
|
|
||
| > **Warning:** If you already run optional Postgres and have a persisted `/var/lib/pgsql/data` volume from an **older major** image, do **not** only bump `POSTGRES_IMAGE` (or the default image major). Follow [Upgrading PostgreSQL](#upgrading-postgresql) first so the volume is upgraded safely. | ||
| You do not need [`compose-with-db.yaml`](https://github.com/redhat-developer/rhdh-local/blob/HEAD/compose-with-db.yaml) or [`app-config.local.yaml`](https://github.com/redhat-developer/rhdh-local/blob/HEAD/configs/app-config/app-config.local.example.yaml) to use Postgres. That overlay remains for CI and back-compat; a normal start does not use it. |
There was a problem hiding this comment.
You can remove the compose-with-db.yaml file.
| You do not need [`compose-with-db.yaml`](https://github.com/redhat-developer/rhdh-local/blob/HEAD/compose-with-db.yaml) or [`app-config.local.yaml`](https://github.com/redhat-developer/rhdh-local/blob/HEAD/configs/app-config/app-config.local.example.yaml) to use Postgres. That overlay remains for CI and back-compat; a normal start does not use it. | ||
|
|
||
| 1. Login to container registry with *Red Hat Login* credentials to use `postgresql` image | ||
| `compose.yaml` sets `WITH_POSTGRES=true` on `rhdh`. On startup, RHDH loads [`app-config.db.yaml`](https://github.com/redhat-developer/rhdh-local/blob/HEAD/configs/app-config/app-config.db.yaml) (`client: pg`) after the SQLite block in [`app-config.yaml`](https://github.com/redhat-developer/rhdh-local/blob/HEAD/configs/app-config/app-config.yaml). |
There was a problem hiding this comment.
We can simplify this by removing the logic in the script handling WITH_POSTGRES since the default app-config will already contain the right info to connect to PostgreSQL.
| The examples below use `podman` and `podman compose`. If you use Docker, replace `podman` with `docker` (for example `docker login`, `docker compose`, `docker exec`). | ||
|
|
||
| 2. Start RHDH with the optional Postgres overlay [`compose-with-db.yaml`](https://github.com/redhat-developer/rhdh-local/blob/main/compose-with-db.yaml). Note that the order of the YAML files is important: | ||
| > **NOTE**: The default image is [`registry.redhat.io/rhel10/postgresql-18`](https://catalog.redhat.com/en/software/containers/rhel10/postgresql-18/6942a60aab9edd836017e3d0). That registry needs a [Red Hat Login](https://access.redhat.com/RegistryAuthentication#getting-a-red-hat-login-2) (`podman login registry.redhat.io`). To skip login, set `POSTGRES_IMAGE` in `.env` (for example `quay.io/fedora/postgresql-18:latest`). |
There was a problem hiding this comment.
quay.io/fedora/postgresql-18:latest should be the default, otherwise it means that users will require authentication to start using RHDH Local, which would be an issue.
| volumes: | ||
| - "postgresqldata:/var/lib/pgsql/data" |
There was a problem hiding this comment.
| volumes: | |
| - "postgresqldata:/var/lib/pgsql/data" | |
| volumes: | |
| - "/var/lib/pgsql/data" |
One of the use cases of RHDH Local is to test different versions of RHDH, so this might mean switching from a new version to an older one. And since RHDH currently has issues with downgrades, a named volume might be an issue. The compose-with-db.yaml handles this using an anonymous volume and it makes sense in this context:
rhdh-local/compose-with-db.yaml
Line 17 in ea366db
Fortune-Ndlovu
left a comment
There was a problem hiding this comment.
Thanks for the RHIDP-16882 work. The overall approach looks good: Postgres in the default compose file, pg client in app-config, startup wait, and NO_PROXY for db behind the corporate proxy overlay.
Before merge, please align the PR description with the code, add (or explicitly defer) the CI check for plugin databases, and pin the default Postgres image by digest instead of :latest.
Details are on the inline comments.
|
|
||
| db: | ||
| container_name: db | ||
| image: "${POSTGRES_IMAGE:-quay.io/fedora/postgresql-18:latest}" # dclint disable-line service-image-require-explicit-tag |
There was a problem hiding this comment.
Please default to an image digest (for example quay.io/fedora/postgresql-18@sha256:...) instead of :latest. The tag can change between pulls, which makes local runs and CI less predictable. Users who want the newest image can still set POSTGRES_IMAGE in .env.
| # The default corporate proxy image is located on registry.redhat.io, which requires authentication. | ||
| CORPORATE_PROXY_IMAGE: docker.io/ubuntu/squid:latest | ||
| # The default Postgres image is located on registry.redhat.io, which requires authentication. | ||
| POSTGRES_IMAGE: quay.io/fedora/postgresql-18:latest |
There was a problem hiding this comment.
Use the same digest pin here as in compose.yaml so CI always runs against a known Postgres build. Right now :latest can drift without a code change.
| The examples below use `podman` and `podman compose`. If you use Docker, replace `podman` with `docker` (for example `docker login`, `docker compose`, `docker exec`). | ||
|
|
||
| 1. Login to container registry with *Red Hat Login* credentials to use `postgresql` image | ||
| > **NOTE**: The default image is [`quay.io/fedora/postgresql-18:latest`](https://quay.io/repository/fedora/postgresql-18). No registry login is required for a normal start. To use the commercially supported image, set `POSTGRES_IMAGE=registry.redhat.io/rhel10/postgresql-18:latest` in `.env` and [log in to `registry.redhat.io`](https://access.redhat.com/RegistryAuthentication#getting-a-red-hat-login-2) (`podman login registry.redhat.io`). |
There was a problem hiding this comment.
Once the default image is digest-pinned in compose, update this note to match (digest in the repo default, :latest only as an optional override in .env).
| podman compose -f compose.yaml -f compose-with-db.yaml up -d | ||
| ``` | ||
| - `podman compose stop` / `start` (or `restart`) keep the volume and catalog data. | ||
| - `podman compose down` then `podman compose up` creates a new empty volume. Use this when switching RHDH versions. |
There was a problem hiding this comment.
This says compose down then up creates a new empty Postgres volume. The PR description "How to test" section says podman compose down && podman compose up -d keeps data. Please make those two places agree so users know when catalog data is kept or wiped.
There was a problem hiding this comment.
Thanks for raising this.
Just to clarify:
-
podman compose down && podman compose upContainers are removed anddownwipes the previous volume and removes thedbcontainer. The anonymous volume is no longer attached to the running container.
upattaches a new empty volume to the container and creates a new database container for postgreSQL. -
podman compose stopthenpodman compose start (or restart), maintains the same containers, the same anonymous volume. Catalog and plugin data is retained. -
podman compose down --volumesthenpodman compose up, the volume is wiped, as well as PostgreSQL and the named volumes (plugins. RAG) are deleted.
I now have made edits to the PR description to highlight this.
| sleep 2 | ||
| elapsed=$((elapsed + 2)) | ||
| done | ||
| echo "PostgreSQL at reachable at ${PG_HOST}:${PG_PORT}" |
There was a problem hiding this comment.
Small typo: "PostgreSQL at reachable" should be "PostgreSQL is reachable".
| @@ -1,36 +0,0 @@ | |||
| # This Compose file is not usable on its own. | |||
There was a problem hiding this comment.
This overlay is removed in the PR, which matches moving db into compose.yaml. Please update the PR description: it still mentions keeping compose-with-db.yaml, WITH_POSTGRES, and app-config.db.yaml. That will confuse reviewers and anyone tracing RHIDP-16882.
| host: ${POSTGRES_HOST} | ||
| port: ${POSTGRES_PORT} | ||
| user: ${POSTGRES_USER} | ||
| password: ${POSTGRES_PASSWORD} |
There was a problem hiding this comment.
Fine for local dev. Worth a short note in the Postgres guide that catalog data now lives in Postgres on disk (until volumes are removed), and that default postgres credentials are not for production.
| env: | | ||
| CORPORATE_PROXY_IMAGE=docker.io/ubuntu/squid:latest | ||
| POSTGRES_IMAGE=quay.io/fedora/postgresql-18:latest | ||
| POSTGRES_IMAGE=quay.io/fedora/postgresql-18@sha256:9ff34b7dd85956cb5cfbaeff193c66ddc4f410aceb20848330d6432b18a99bd5 |
There was a problem hiding this comment.
| POSTGRES_IMAGE=quay.io/fedora/postgresql-18@sha256:9ff34b7dd85956cb5cfbaeff193c66ddc4f410aceb20848330d6432b18a99bd5 | |
| POSTGRES_IMAGE=quay.io/fedora/postgresql-18:latest |
I think it is fine to use latest here. The image name specifies postgresql-18, so it shouldn't cause inadvertent major upgrades of the DB, by contract. Or, if we want to use digests, we should make sure this is monitored by Renovate, otherwise it will likely fall behind.
I would suggest keeping latest for now and maybe capturing a followup ticket to handle this consistently, as this is not the single image that might be impacted.
| repo: context.repo.repo, | ||
| }); | ||
|
|
||
| CORPORATE_PROXY_IMAGE: docker.io/ubuntu/squid:latest | ||
| # The default Postgres image is located on registry.redhat.io, which requires authentication. | ||
| POSTGRES_IMAGE: quay.io/fedora/postgresql-18:latest | ||
| POSTGRES_IMAGE: quay.io/fedora/postgresql-18@sha256:9ff34b7dd85956cb5cfbaeff193c66ddc4f410aceb20848330d6432b18a99bd5 |
There was a problem hiding this comment.
| POSTGRES_IMAGE: quay.io/fedora/postgresql-18@sha256:9ff34b7dd85956cb5cfbaeff193c66ddc4f410aceb20848330d6432b18a99bd5 | |
| POSTGRES_IMAGE: quay.io/fedora/postgresql-18:latest |
Same suggestion. latest is fine.
| CORPORATE_PROXY_IMAGE: docker.io/ubuntu/squid:latest | ||
| # The default Postgres image is located on registry.redhat.io, which requires authentication. | ||
| POSTGRES_IMAGE: quay.io/fedora/postgresql-18:latest | ||
| POSTGRES_IMAGE: quay.io/fedora/postgresql-18@sha256:9ff34b7dd85956cb5cfbaeff193c66ddc4f410aceb20848330d6432b18a99bd5 |
There was a problem hiding this comment.
| POSTGRES_IMAGE: quay.io/fedora/postgresql-18@sha256:9ff34b7dd85956cb5cfbaeff193c66ddc4f410aceb20848330d6432b18a99bd5 | |
| POSTGRES_IMAGE: quay.io/fedora/postgresql-18:latest |
Same suggestion: latest is fine.
| You can combine this with other overlays the same way. For example, with the [corporate proxy](corporate-proxy-setup-sim.md) setup: | ||
| - `podman compose stop` / `start` (or `restart`) keep the volume and catalog data. | ||
| - `podman compose down` then `podman compose up` creates a new empty volume, catalog and plugin data is wiped. Use this when switching RHDH versions. | ||
| - `podman compose down --volumes` also deletes other Compose volumes (plugins, RAG, Postgres, and so on). |
There was a problem hiding this comment.
| - `podman compose down --volumes` also deletes other Compose volumes (plugins, RAG, Postgres, and so on). | |
| - `podman compose down --volumes` also deletes other Compose volumes (plugins, Postgres, and so on). |
RAG is going to be removed soon.
| environment: | ||
| - HTTP_PROXY=http://proxy:3128 | ||
| - HTTPS_PROXY=http://proxy:3128 | ||
| - NO_PROXY=localhost,127.0.0.1,db |
There was a problem hiding this comment.
| - NO_PROXY=localhost,127.0.0.1,db | |
| - NO_PROXY=localhost,127.0.0.1 |
The install-dynamic-plugins service does not need to reach the DB.
| echo "Waiting for Postgres at ${PG_HOST}:${PG_PORT} ..." | ||
| until bash -c "exec 3<>/dev/tcp/${PG_HOST}/${PG_PORT}" 2>/dev/null; do | ||
| if [ "${elapsed}" -ge "${PG_TIMEOUT}" ]; then | ||
| echo "Timed out waiting for PostgreSQL at ${PG_HOST}:${PG_PORT} after ${PG_TIMEOUT}s" >&2 | ||
| exit 1 | ||
| fi | ||
| echo "PostgreSQL is not reachable yet, retrying..." | ||
| sleep 2 | ||
| elapsed=$((elapsed + 2)) | ||
| done | ||
| echo "PostgreSQL is reachable at ${PG_HOST}:${PG_PORT}" |
There was a problem hiding this comment.
Please fix the indentation here, as well as the SonarCloud suggestion to Use '[[' instead of '[' for conditional tests. The '[[' construct is safer and more feature-rich.
| environment: | ||
| - HTTP_PROXY=http://proxy:3128 | ||
| - HTTPS_PROXY=http://proxy:3128 | ||
| - NO_PROXY=localhost,127.0.0.1,db |
There was a problem hiding this comment.
| - NO_PROXY=localhost,127.0.0.1,db | |
| - NO_PROXY=localhost,127.0.0.1 |
I don't think this is needed. Instead, I would suggest adding db to the internal_net network instead, so it is reachable from rhdh and rhdh won't need to bypass the proxy at all for this. This would actually mimic a real corporate setup: the DB is usually only accessed only from the internal network.
|
|
We never wanted PostgreSQL to be on by default so we could keep the resource overheads low. Why do we need this now? |
|
@benwilcock In memory |



Description
Enable PostgreSQL as the default database for RHDH Local (RHIDP-16882 / RHIDP-16883).
dbis incompose.yaml, image${POSTGRES_IMAGE:-quay.io/fedora/postgresql-18:latest}/var/lib/pgsql/datacompose stop/start/restartkeep data;compose downthencompose upstarts a new empty volume (RHDH does not support DB downgrades)rhdhdepends_on: db(healthy)app-config.yamlusesclient: pgandPOSTGRES_*fromdefault.envWITH_POSTGRES,app-config.db.yaml,compose-with-db.yamlwait-for-plugins-and-start.shwaits for$POSTGRES_HOST:$POSTGRES_PORTbefore starting nodeNO_PROXY=localhost,127.0.0.1app-config.local.yamloverridePOSTGRES_IMAGE=registry.redhat.io/rhel10/postgresql-18:latestin.envplus registry loginWhich issue(s) does this PR fix or relate to
Relates to RHIDP-16882
Relates to RHIDP-16883
Follow up issue RHIDP-16884 - CI check that the effective RHDH config is client: pg (Postgres, not SQLite) after the stack is healthy.
PR acceptance criteria
How to test changes / Special notes to the reviewer
Postgres uses an anonymous volume at
/var/lib/pgsql/data.podman compose stopthenstart(orrestart)podman compose downthenpodman compose up -dpodman compose down --volumesthenpodman compose up -d