Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .github/actions/setup/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,6 @@ runs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends libsqlcipher-dev libmagic1

# A hit skips pip entirely. The venv links to its interpreter, so the key
# has the exact Python version.
- uses: actions/cache@v6
id: venv-cache
with:
Expand All @@ -40,7 +38,6 @@ runs:
shell: bash
run: echo "sha=$(git rev-parse HEAD:lib_relaysms_payload_specs)" >> "$GITHUB_OUTPUT"

# The bindings don't depend on the Python version, so every job shares them.
- uses: actions/cache@v6
id: specs-cache
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/check-requirements.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
python-version: ${{ matrix.python-version }}
cache: pip

# Dry run so conflicts fail the PR without needing the native build deps.
# Resolves the requirements without building them.
- name: Resolve dependencies
run: |
pip install --dry-run --disable-pip-version-check \
Expand Down
6 changes: 2 additions & 4 deletions .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
tests:
name: Tests (Python ${{ matrix.python-version }})
runs-on: ubuntu-latest
# Ubuntu 24.04 LTS and Debian 13. The coverage job covers 3.14, the Docker image's.
# The Python versions of Ubuntu 24.04 and Debian 13. The coverage job runs 3.14, the Docker image's.
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -105,16 +105,14 @@ jobs:
with:
fetch-depth: 0

# Adapters pin SDK releases while the Publisher runs sdk/ as it is, so every
# change to it needs a version that can be tagged.
- name: Require a version bump for SDK changes
env:
BASE: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
git cat-file -e "$BASE^{commit}" 2>/dev/null || exit 0
git diff --quiet "$BASE...HEAD" -- sdk/src && exit 0
git diff "$BASE...HEAD" -- sdk/pyproject.toml | grep -q '^+version = ' && exit 0
echo "::error file=sdk/pyproject.toml::sdk/src changed; bump version here."
echo "::error file=sdk/pyproject.toml::sdk/src changed without a new version here."
exit 1

deploy-staging:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
steps:
- uses: actions/checkout@v7

# Local files and anchors only, so an external site being down can't fail a PR.
# Checks local files and anchors, not external sites.
- name: Check Markdown links
uses: lycheeverse/lychee-action@v2
with:
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@ jobs:
with:
python-version: "3.12"

# The e2e tests only need pytest, not the app's test environment.
- name: Install pytest
run: |
python -m venv venv
Expand Down
6 changes: 2 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -129,11 +129,9 @@ venv/
# Written by install.sh into the install directory
.instance-name

# Runtime data: adapters, their venvs and assets, gateway client imports
# Runtime data: adapters' code, venvs, config and state
/data/
# Runtime data an install kept here before data/ existed. It can hold adapter
# credentials and gateway client numbers, so it stays ignored until
# scripts/migrate-runtime-data.sh moves it.
# Where older installs kept runtime data. It can hold adapter credentials and gateway client numbers.
/platforms/
/gateway_clients/
ENV/
Expand Down
407 changes: 210 additions & 197 deletions CONTRIBUTING.md

Large diffs are not rendered by default.

11 changes: 4 additions & 7 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.14.7-slim-bookworm AS builder
FROM python:3.15.0-slim-bookworm AS builder

WORKDIR /publisher

Expand All @@ -19,8 +19,8 @@ RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
| sh -s -- -y --no-modify-path
ENV PATH="/root/.cargo/bin:${PATH}"

# Outside /publisher, so a source edit doesn't recopy it in the final stage.
# The link puts it where make and scripts/run.sh look for it.
# The venv lives outside /publisher, linked to where make and scripts/run.sh
# look for it.
RUN python3 -m venv /venv && ln -s /venv venv
COPY requirements.txt requirements-observability.txt ./
COPY sdk ./sdk
Expand All @@ -29,17 +29,14 @@ RUN --mount=type=cache,sharing=locked,target=/root/.cache/pip \
-r requirements-observability.txt

COPY . .
# Submodule URL is SSH-based; rewrite to HTTPS since no SSH key is
# available in the build environment (same fix install.sh applies).
# The cargo caches outlive the layer, so source edits don't redo the Rust build.
RUN --mount=type=cache,sharing=locked,target=/root/.cargo/registry \
--mount=type=cache,sharing=locked,target=/root/.cargo/git \
--mount=type=cache,sharing=locked,target=/publisher/lib_relaysms_payload_specs/target \
git config --global url."https://github.com/".insteadOf "git@github.com:" \
&& make build


FROM python:3.14.7-slim-bookworm
FROM python:3.15.0-slim-bookworm

WORKDIR /publisher

Expand Down
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,11 @@ coverage:
test-sdk:
$(PYTHON) -m pytest -c sdk/pyproject.toml sdk/tests sdk/examples $(PYTEST_ARGS)

## Migrate and query SQLite, SQLCipher, and Postgres, MySQL and MariaDB in podman.
## Migrate and query SQLite and SQLCipher, and Postgres, MySQL and MariaDB in podman.
test-dialects:
$(PYTHON) -m pytest tests/dialects $(PYTEST_ARGS)

## Install and update in a systemd container with podman (slow, needs network).
## Install, update and uninstall in systemd containers with podman (slow, needs network).
test-e2e:
$(PYTHON) -m pytest --noconftest tests/e2e $(PYTEST_ARGS)

Expand Down
20 changes: 3 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,27 +15,13 @@ curl -fsSL https://raw.githubusercontent.com/smswithoutborders/RelaySMS-Publishe

## Documentation

The [manual](docs/README.md) starts with:

- [Quick start](docs/getting-started/quick-start.md)
- [Settings reference](docs/configuration/settings.md)
- [Managing adapters](docs/platforms/manage.md) and the [provider guides](docs/platforms/manage.md#provider-guides)
- [Connecting a relay](docs/gateway-clients/connect-a-relay.md)
- [Troubleshooting](docs/operating/troubleshooting.md)

`sudo ./manage.sh docs <DOMAIN>` serves the manual from your own server.
Read the [manual](docs/README.md). Its [Where to start](docs/README.md#where-to-start) table points to the page for each task.

## Contributing

[CONTRIBUTING.md](CONTRIBUTING.md) covers setup, running the services locally, testing and the workflow. To start:

```bash
git clone --recurse-submodules https://github.com/smswithoutborders/RelaySMS-Publisher.git
cd RelaySMS-Publisher
make setup
```
Read [CONTRIBUTING.md](CONTRIBUTING.md) to set up a development checkout and open a pull request.

Adapters are built with the [adapter SDK](sdk/README.md).
To add a platform, see [Writing an adapter](docs/developers/writing-an-adapter.md).

## Security

Expand Down
6 changes: 3 additions & 3 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

Don't open a public issue for a vulnerability. Email **<developers@smswithoutborders.com>** with:

- a description of the issue and its impact;
- the affected versions or commits;
- steps to reproduce it.
- a description of the issue and its impact
- the affected versions or commits
- the steps to reproduce it

This covers the Publisher, its adapter SDK in `sdk/` and the platform adapters in the [smswithoutborders](https://github.com/smswithoutborders) organization.
2 changes: 1 addition & 1 deletion alembic.ini
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# path to migration scripts
script_location = %(here)s/migrations

# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# template used to generate migration file names. The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
# for all available tokens
Expand Down
15 changes: 7 additions & 8 deletions deploy/nginx/relaysms-publisher-nginx.conf.template
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,12 @@
# Installed to /etc/nginx/sites-available/<domain>.conf by install.sh, with
# __SERVER_NAME__, __REST_PORT__ and __GRPC_PORT__ substituted from .env.
#
# Upstream, map and zone names are global in nginx, so they carry the REST
# port to keep instances on one host apart.
# Upstream, map and zone names end in the REST port, one set per instance on
# a host.
#
# Defines only the plain-HTTP (port 80) block. Run `certbot --nginx -d
# <domain>` afterwards (install.sh offers to do this) to add the HTTPS block
# and redirect; certbot copies every directive here into it, so add
# hardening here rather than to the generated 443 block.
# Defines only the port 80 block. `certbot --nginx -d <domain>` adds the HTTPS
# block and redirect, copying every directive here into it. Add hardening here,
# not to the generated 443 block.

# Rate-limit logins and requests carrying Basic credentials.
limit_req_zone $binary_remote_addr zone=publisher_login___REST_PORT__:10m rate=5r/m;
Expand Down Expand Up @@ -178,8 +177,8 @@ server {
return 494 '{"error": "Request headers are too large."}';
}

# proxy_set_header above doesn't apply to grpc_pass; repeated here via
# grpc_set_header so the gRPC backend also gets the real client IP, etc.
# grpc_pass ignores proxy_set_header: the gRPC locations set the client IP
# headers with grpc_set_header.
location /publisher.v3.Publisher {
grpc_pass grpc://publisher_grpc___REST_PORT__;
grpc_set_header Host $host;
Expand Down
3 changes: 1 addition & 2 deletions deploy/systemd/relaysms-publisher-rest.service
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,7 @@ WorkingDirectory=/opt/relaysms/relaysms-publisher
EnvironmentFile=/opt/relaysms/relaysms-publisher/.env
Environment=OTEL_SERVICE_NAME=relaysms-publisher-rest
Environment=PATH=/opt/relaysms/relaysms-publisher/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# uvicorn keeps replacing workers that fail on startup, so systemd would never see
# the failure. Importing the app once first makes a config error fail the unit.
# Fails the unit on a config error before uvicorn starts its workers.
ExecStartPre=/opt/relaysms/relaysms-publisher/venv/bin/python3 -c "import publisher.api.rest.app"
ExecStart=/opt/relaysms/relaysms-publisher/scripts/otel-wrap.sh /opt/relaysms/relaysms-publisher/venv/bin/python3 -m uvicorn publisher.api.rest.app:app --host "${HOST}" --port "${PORT}" --workers 4 --proxy-headers --forwarded-allow-ips "*"
Restart=on-failure
Expand Down
4 changes: 1 addition & 3 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,7 @@ services:
container_name: relaysms-publisher
volumes:
- ./data:/publisher/data
# HOST/GRPC_HOST default to 127.0.0.1 in .env, which would make the
# container unreachable from outside. Override regardless of what .env
# has, since inside a container 0.0.0.0 is the only sensible bind.
# Listens on every interface in the container, whatever .env sets.
environment:
HOST: 0.0.0.0
GRPC_HOST: 0.0.0.0
Expand Down
3 changes: 0 additions & 3 deletions docker-entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,5 @@ export WORKERS="${WORKERS:-4}"
export GRPC_HOST="${GRPC_HOST:-0.0.0.0}"

python3 -m alembic upgrade head
# Registers adapters installed before the registry moved to the database, and
# moves their files out of their code directories.
python3 -m publisher platforms import

exec "$SCRIPT_DIR/scripts/run.sh"
7 changes: 1 addition & 6 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,6 @@ Publishes to online platforms, such as Gmail, from content people send by SMS, s
## How it works

```mermaid
---
config:
flowchart:
curve: linear
---
flowchart TB
app["RelaySMS app"]
gateway["Gateway client"]
Expand Down Expand Up @@ -72,5 +67,5 @@ The [glossary](reference/glossary.md) defines the rest, and [Architecture](expla

## Getting help

- Ask questions and report bugs in the [issue tracker](https://github.com/smswithoutborders/RelaySMS-Publisher/issues); [CONTRIBUTING.md](https://github.com/smswithoutborders/RelaySMS-Publisher/blob/main/CONTRIBUTING.md#ways-to-contribute) says what a useful report has.
- Ask questions and report bugs in the [issue tracker](https://github.com/smswithoutborders/RelaySMS-Publisher/issues). [CONTRIBUTING.md](https://github.com/smswithoutborders/RelaySMS-Publisher/blob/main/CONTRIBUTING.md#where-to-start) says what a useful report has.
- Report vulnerabilities privately, as [SECURITY.md](https://github.com/smswithoutborders/RelaySMS-Publisher/blob/main/SECURITY.md) describes.
2 changes: 1 addition & 1 deletion docs/configuration/encryption-keys.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Encryption keys

Three keys in `.env` protect the data. The installer and `make .env` generate them. Each is 64 hex characters; `openssl rand -hex 32` makes one, and `manage.sh check` reports a missing or malformed key.
Three keys in `.env` protect the data. The installer and `make .env` generate them. Each is 64 hex characters, which `openssl rand -hex 32` generates. `manage.sh check` reports a missing or malformed key.

| Setting | Protects | Used |
| --- | --- | --- |
Expand Down
8 changes: 5 additions & 3 deletions docs/configuration/offline-publishing.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
# Offline publishing

Offline payloads let people send email through [RelaySMS Mail](../platforms/relaysms-mail.md) without linking an account; linked accounts need none of this ([settings](settings.md#offline-publishing)). The app encrypts an offline payload to the server's identity key, and the Publisher publishes it through the `rmail` adapter, which must be installed. Anyone who can reach an allowed protocol can send one.
Offline payloads let people send email through [RelaySMS Mail](../platforms/relaysms-mail.md) without linking an account. Payloads from linked accounts don't depend on these [settings](settings.md#offline-publishing). The app encrypts an offline payload to the server's identity key, and the Publisher publishes it through the `rmail` adapter, which must be installed. Anyone who can reach an allowed protocol can send one.

Each payload is tagged with the protocol it arrived on:

- `https`: [REST `POST /v1/publications`](../reference/rest-api.md#publishing)
- `smtp`: the [SMTP transport](smtp-transport.md)

- An offline payload from a protocol not in `OFFLINE_PUBLISH_ALLOWED_PROTOCOLS` is discarded.
## Rules

- An offline payload from a protocol not in `OFFLINE_PUBLISH_ALLOWED_PROTOCOLS` is discarded. An empty list allows every protocol.
- An offline payload over `https` must carry `OFFLINE_PUBLISH_SHARED_SECRET` in the request's `tag` field, or it's discarded.
- Leaving `OFFLINE_PUBLISH_ALLOWED_PROTOCOLS` empty, or listing `https`, without the secret fails `manage.sh check`, and the services refuse to start.
- A discarded payload is logged and recorded as `failed` in the publication stats, with reason `protocol_not_allowed`, `offline_tag_missing` or `offline_tag_invalid`. `POST /v1/publications` still returns success, since the check runs in the queue.
- A discarded payload is logged and recorded as `rejected` in the publication stats, with reason `protocol_not_allowed`, `offline_tag_missing` or `offline_tag_invalid`. `POST /v1/publications` still returns success, since the check runs in the queue.

## Allowing https

Expand Down
20 changes: 10 additions & 10 deletions docs/configuration/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Settings live in `.env` in the install directory. `template.env` lists each one with its default.

- `sudo ./manage.sh check` reports missing or invalid settings; each service refuses to start on an error in the settings it uses.
- `sudo ./manage.sh check` reports missing or invalid settings. Each service refuses to start on an error in the settings it uses.
- Restart the services after a change: `sudo ./manage.sh restart`.
- After an update, `sudo ./sync-env.sh` adds the settings from `template.env` that `.env` is missing, and shows the change before writing it.
- On Docker, check with `docker compose exec publisher python -m publisher config check` and apply a change with `docker compose up -d`.
Expand All @@ -17,9 +17,9 @@ Settings live in `.env` in the install directory. `template.env` lists each one
| `GRPC_HOST` | `127.0.0.1` | gRPC address. Docker always binds `0.0.0.0`. |
| `GRPC_PORT` | `6000` | gRPC port |
| `GRPC_MAX_WORKERS` | `10` | gRPC worker threads |
| `GRPC_TLS_ENABLED` | `false` | Serves TLS on `GRPC_PORT` itself; leave off behind nginx ([details](../installation/reverse-proxy.md#grpc-tls-without-nginx)) |
| `GRPC_TLS_ENABLED` | `false` | Serves TLS on `GRPC_PORT` itself. Leave it off behind nginx ([details](../installation/reverse-proxy.md#grpc-tls-without-nginx)) |
| `GRPC_TLS_CERT_FILE`, `GRPC_TLS_KEY_FILE` | None | Certificate and key. Required when `GRPC_TLS_ENABLED=true`. |
| `NONCE_TTL_SECONDS` | `600` | How far a gRPC request's timestamp may be from the server's clock; nonces are remembered this long to refuse replays |
| `NONCE_TTL_SECONDS` | `600` | How far a gRPC request's timestamp may be from the server's clock, and how long nonces are remembered to refuse replays |

## Database

Expand All @@ -29,14 +29,14 @@ See [Database](../installation/database.md) and [Encryption keys](encryption-key
| --- | --- | --- |
| `DATABASE_DIALECT` | `sqlite` | `sqlite`, `mysql` (MySQL and MariaDB) or `postgres` |
| `SQLITE_DATABASE_PATH` | `data/relaysms.db` | SQLite database file |
| `MYSQL_HOST` | `127.0.0.1` | MySQL server |
| `MYSQL_HOST` | `127.0.0.1` (`localhost` if unset) | MySQL server |
| `MYSQL_PORT` | `3306` | MySQL port |
| `MYSQL_USER`, `MYSQL_PASSWORD` | None | Required for `mysql` |
| `MYSQL_DATABASE` | `relaysms` | Required for `mysql`; created if it doesn't exist |
| `POSTGRES_HOST` | `127.0.0.1` | PostgreSQL server |
| `MYSQL_DATABASE` | `relaysms` | Required for `mysql`. Created if it doesn't exist. |
| `POSTGRES_HOST` | `127.0.0.1` (`localhost` if unset) | PostgreSQL server |
| `POSTGRES_PORT` | `5432` | PostgreSQL port |
| `POSTGRES_USER`, `POSTGRES_PASSWORD` | None | Required for `postgres` |
| `POSTGRES_DATABASE` | `relaysms` | Required for `postgres`; created if it doesn't exist |
| `POSTGRES_DATABASE` | `relaysms` | Required for `postgres`. Created if it doesn't exist. |
| `DATABASE_ENCRYPTION_ENABLED` | `true` (`false` if unset) | Encrypts the SQLite file with SQLCipher. No effect on MySQL or PostgreSQL. |
| `DATABASE_ENCRYPTION_KEY` | None | Required for SQLite when encryption is on |
| `DATABASE_FIELD_ENCRYPTION_ENABLED` | `true` (`false` if unset) | Encrypts linked accounts' platform tokens |
Expand Down Expand Up @@ -80,7 +80,7 @@ See [Offline publishing](offline-publishing.md).

| Setting | Default | Description |
| --- | --- | --- |
| `OFFLINE_PUBLISH_ALLOWED_PROTOCOLS` | `smtp` (empty if unset) | Comma-separated protocols whose offline payloads publish: `https`, `smtp`. Empty allows both. |
| `OFFLINE_PUBLISH_ALLOWED_PROTOCOLS` | `smtp` (empty if unset) | Comma-separated protocols whose offline payloads publish, such as `smtp,https`. Empty allows every protocol. |
| `OFFLINE_PUBLISH_SHARED_SECRET` | None | 64 hex characters. Required when `https` is allowed. |

## SMTP transport
Expand Down Expand Up @@ -109,7 +109,7 @@ See [Managing adapters](../platforms/manage.md#files). Relative paths are from t
| `PLATFORMS_ADAPTERS_VENV_DIR` | `data/platforms/venvs` | Directory for adapters' virtualenvs |
| `PLATFORMS_ADAPTERS_CONFIG_DIR` | `data/platforms/config` | Directory for adapters' `credentials.json` files |
| `PLATFORMS_ADAPTERS_STATE_DIR` | `data/platforms/state` | Directory for files adapters write, such as databases |
| `PLATFORMS_GITHUB_ORGS` | None | Comma-separated GitHub orgs that administrators may install and update adapters from over the REST API. Empty refuses REST installs; `publisher.sh platforms` is unaffected. |
| `PLATFORMS_GITHUB_ORGS` | None | Comma-separated GitHub orgs that administrators may install and update adapters from over the REST API. Empty refuses REST installs, but not `publisher.sh platforms`. |

## REST API access

Expand All @@ -136,6 +136,6 @@ See [Observability](../operating/observability.md).
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` | `grpc` or `http/protobuf` |
| `OTEL_TRACES_SAMPLER` | `parentbased_traceidratio` | Which traces to keep |
| `OTEL_TRACES_SAMPLER_ARG` | `1.0` | Fraction of traces kept, `0` to `1` |
| `OTEL_LOGS_EXPORTER` | `otlp` | `otlp` sends logs to the collector; `none` doesn't |
| `OTEL_LOGS_EXPORTER` | `otlp` | `otlp` sends logs to the collector, and `none` turns log export off |
| `OTEL_PYTHON_LOG_CORRELATION` | `true` | Adds trace and span IDs to log records |
| `UPTIME_KUMA_WORKER_PUSH_URL`, `UPTIME_KUMA_SMTP_PUSH_URL` | None | Uptime Kuma push URLs for the worker and email listener heartbeats. Blank turns them off. |
Loading
Loading