Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 24 additions & 5 deletions docs/operating/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,6 @@ sudo ./scripts/setup-observability.sh

Set retention in the SigNoz UI, under Settings > Workspace > Retention Controls.

> [!IMPORTANT]
> `casting.yaml` is generated only once. To change a running install, edit `observability/signoz/casting.yaml` and run `foundryctl cast -f observability/signoz/casting.yaml` again. It recreates only the `signoz` container; Postgres and ClickHouse keep their data.

## Uptime Kuma monitors

Open Uptime Kuma through its domain, or through an SSH tunnel at <http://localhost:3001>:
Expand Down Expand Up @@ -123,14 +120,36 @@ sudo ./manage.sh restart

To turn it off, blank `OTEL_EXPORTER_OTLP_ENDPOINT` and restart.

## Changing the SigNoz configuration

`observability/signoz/casting.yaml` is generated only once, so updating the Publisher doesn't change a running SigNoz install. To change it, edit `casting.yaml`, then apply it from the install directory:

```bash
sudo foundryctl cast -f observability/signoz/casting.yaml
sudo docker compose \
-f pours/deployment/compose.yaml \
-f observability/signoz/docker-compose.override.yml \
up -d --force-recreate
```

`cast` alone starts the containers without the override's memory limits and hardening. `--force-recreate` also reloads the config files `cast` rewrote, which Compose doesn't detect. Postgres and ClickHouse keep their data.

## Upgrading SigNoz

1. Check the [upgrade path](https://signoz.io/upgrade-path/) for versions you must upgrade through first, and read the migration notes for each release you cross.
2. Back up the metastore: `sudo sh -c 'docker exec relaysms-publisher-signoz-metastore-postgres-0 pg_dump -U signoz signoz > /root/signoz-metastore-$(date +%F).sql'`.
3. Install the foundryctl version that `scripts/setup-observability.sh` sets in `FOUNDRY_VERSION`: `curl -fsSL https://signoz.io/foundry.sh | sudo FOUNDRY_VERSION=<VERSION> FOUNDRY_INSTALL_DIR=/usr/local/bin bash`.
4. Copy the `image:` lines under `signoz` and `ingester` from `observability/signoz/casting.yaml.template` into `casting.yaml`, adding them if they're missing.
5. Apply it as in [Changing the SigNoz configuration](#changing-the-signoz-configuration), then check the version under Settings.

## Setting up by hand

### SigNoz

Needs Docker Engine with the Compose plugin. Set `<PASSWORD>` to 12 or more characters, with upper and lower case, a number and a symbol.
Needs Docker Engine with the Compose plugin. Set `<VERSION>` to the `FOUNDRY_VERSION` in `scripts/setup-observability.sh`, and `<PASSWORD>` to 12 or more characters, with upper and lower case, a number and a symbol.

```bash
curl -fsSL https://signoz.io/foundry.sh | FOUNDRY_INSTALL_DIR=/usr/local/bin bash
curl -fsSL https://signoz.io/foundry.sh | sudo FOUNDRY_VERSION=<VERSION> FOUNDRY_INSTALL_DIR=/usr/local/bin bash

sed \
-e "s/__POSTGRES_DB__/signoz/" -e "s/__POSTGRES_USER__/signoz/" \
Expand Down
4 changes: 4 additions & 0 deletions observability/signoz/casting.yaml.template
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,11 @@ spec:
POSTGRES_PASSWORD: __POSTGRES_PASSWORD__
telemetrykeeper:
kind: clickhousekeeper
# Keep the ingester on the collector release that matches this SigNoz
# release. Foundry runs the schema migrator from the ingester image.
signoz:
spec:
image: signoz/signoz:v0.151.0
env:
SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST: "__SMTP_SMARTHOST__"
SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__USERNAME: "__SMTP_USERNAME__"
Expand All @@ -33,6 +36,7 @@ spec:
SIGNOZ_ALERTMANAGER_SIGNOZ_EXTERNAL__URL: "__SIGNOZ_EXTERNAL_URL__"
ingester:
spec:
image: signoz/signoz-otel-collector:v0.144.16
config:
data:
# Points the collector's opamp client at the SigNoz backend.
Expand Down
3 changes: 2 additions & 1 deletion scripts/setup-observability.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ SIGNOZ_SMTP_PORT="587"
SIGNOZ_SMTP_USERNAME=""
SIGNOZ_SMTP_PASSWORD=""
SIGNOZ_SMTP_FROM=""
FOUNDRY_VERSION="v0.5.0"

usage() {
cat <<'EOF'
Expand Down Expand Up @@ -240,7 +241,7 @@ main() {

if ! command -v foundryctl &>/dev/null; then
log "Installing foundryctl"
curl -fsSL https://signoz.io/foundry.sh | FOUNDRY_INSTALL_DIR=/usr/local/bin bash
curl -fsSL https://signoz.io/foundry.sh | FOUNDRY_VERSION="$FOUNDRY_VERSION" FOUNDRY_INSTALL_DIR=/usr/local/bin bash
fi

signoz_external_url="http://localhost:$SIGNOZ_PORT"
Expand Down
Loading