Network: dynamic DNS, own CA, Let's Encrypt

.github/workflows/build.yml

@@ -48,6 +48,8 @@ jobs:
         run: tests/test-bootlog.sh
       - name: Power schedule (shareos-power)
         run: LD_LIBRARY_PATH="$HOME/hostroot/lib" tests/test-power.sh "$HOME/hostroot/bin"
+      - name: Dynamic DNS (shareos-ddns, all providers with a fake curl)
+        run: LD_LIBRARY_PATH="$HOME/hostroot/lib" tests/test-ddns.sh "$HOME/hostroot/bin"
       - name: Zabbix agent controller (settings, disk/RAID items)
         run: LD_LIBRARY_PATH="$HOME/hostroot/lib" tests/test-zabbix-items.sh "$HOME/hostroot/bin"
       - name: SMART evaluation (disk health)
@@ -57,6 +59,7 @@ jobs:
           tests/check-ucode-order.py src/shareos-core/usr/share/rpcd/ucode/shareos.uc
           tests/check-ucode-order.py src/shareos-core/usr/sbin/shareos-apps
           tests/check-ucode-order.py src/shareos-core/usr/sbin/shareos-notify
+          tests/check-ucode-order.py src/shareos-core/usr/sbin/shareos-ddns
           tests/check-ucode-regex.py
           tests/check-acl.py
           tests/check-i18n.py
@@ -81,7 +84,7 @@ jobs:
           sudo apt-get update
           sudo apt-get install -y --no-install-recommends \
             build-essential bc bison flex cpio rsync unzip file wget \
-            libssl-dev libncurses-dev python3
+            libssl-dev libncurses-dev python3 qemu-user-static
 
       - name: Cache downloads
         uses: actions/cache@v4
@@ -96,6 +99,15 @@ jobs:
       - name: Test writing .bin files (rollback to Alt-F)
         run: tests/test-flash-bin.sh output/${{ matrix.board }}/host/bin
 
+      - name: Own certificate authority (shareos-ca with the firmware's mbedtls and uci in qemu)
+        run: |
+          T=$PWD/output/${{ matrix.board }}/target W=$(mktemp -d)
+          for p in gen_key cert_write cert_app uci; do
+            f=$T/usr/bin/$p; [ -e "$f" ] || f=$T/sbin/$p
+            printf '#!/bin/sh\nexec qemu-arm-static -L %s %s "$@"\n' "$T" "$f" > $W/$p; chmod 755 $W/$p
+          done
+          tests/test-ca.sh $W $W
+
       - name: Initramfs load address of every model
         run: tests/check-initrd-address.sh output/${{ matrix.board }}/images
 

CHANGELOG.md

@@ -4,6 +4,14 @@ Changes per release (newest first). The section of a version is added to its
 GitHub release automatically when the release is built. Test versions are
 pre-releases for test devices only.
 
+## 0.2.0-beta.18
+
+- **DynDNS, own certificate authority and Let's Encrypt** (System → Netzwerk).
+  - *DynDNS*: DuckDNS, dynv6, Cloudflare, No-IP or any dyndns2 address (Strato, IONOS, …), IPv4 and/or IPv6 (for lines without their own IPv4 address: DS-Lite, fibre, mobile). The box reports its public address every 5 minutes when it changed, otherwise once a day; a failure becomes a message. Token and password go to curl through stdin, not the command line, and are never sent back to the browser. `shareos-ddns`, `tests/test-ddns.sh` (24 checks with a fake curl).
+  - *Own CA*: a certificate authority on the box (EC P-256, 10 years) issues certificates for the box and other devices in the home network (825 days, server authentication, wildcards like `*.home.lan`, IP addresses); its root certificate is installed once on PCs and phones, after that there are no certificate warnings. The box's own certificate follows name and IP address and is renewed 30 days before it runs out. The CA is part of the settings backup. `shareos-ca` (mbedtls of the firmware), `tests/test-ca.sh` (18 checks, in CI with the firmware's programs in qemu).
+  - *Let's Encrypt* for the DynDNS name with DuckDNS or Cloudflare: the check runs over DNS (TXT record at the provider), no port has to be open; renewed daily when less than 30 days are left. `shareos-acme` with uacme (105 KB, mbedtls), test mode with the staging CA.
+  - `shareos-cert` no longer replaces a certificate of the own CA with a self-signed one when the IP address changes (it only renews its own self-signed one).
+
 ## 0.2.0-beta.17
 
 - **SMB 3 with signing about three times as fast: the crypto engine computes the signatures** ([#94](https://github.com/UniqueDroid/ShareOS/issues/94)). Windows 11 signs every SMB connection, and on the 800 MHz Kirkwood the signature limited writing: measured on a DNS-320L, about 35 MB/s without signing (the disks), only 4–10 MB/s with SMB 3.1.1 – ksmbd computes AES-CMAC on the CPU, where AES manages about 12 MB/s. The Kirkwood has a crypto engine (Marvell CESA) that does AES-CBC at about 38 MB/s with DMA, but ksmbd could not use it. A kernel patch (`0001-ksmbd-cmac-via-cbc`) computes the AES-CMAC of messages from 8 KiB through `cbc(aes)` – the engine – (CMAC is a CBC-MAC; the last block and the subkeys on the CPU), checks the first result against the old way and falls back to it on any error. Measured: **19–23 MB/s** signed, the CPU load of ksmbd went from about 64 % to under 10 %. `shareos-shares` loads `marvell-cesa` before ksmbd; module parameter `ksmbd.cbc_cmac=0` switches it off. Tested before on a UML kernel with Samba's smbclient (signing required, checksums equal). Tried as well and dropped: AES-GMAC signing (GHASH in C on ARMv5 is not faster, 3–8 MB/s).

CLAUDE.md

@@ -6,7 +6,7 @@
 - Tags/releases are created by Jan; prepare the release text and the `CHANGELOG.md` section.
 - **Every release updates the docs** – see `docs/RELEASE.md` (CHANGELOG, README status/features, TESTEN/TESTING, ENTWICKLUNG/DEVELOPMENT).
 - Photos for the docs: strip EXIF/GPS and blur serial-number stickers before committing.
-- Tests: `tests/` (host: test-update, test-net, test-flash-bin, test-boards, test-raid-wait, test-smart, test-bootlog, test-power, test-notify, test-zabbix-items, check-acl.py, check-i18n.py; ARM emulator with sudo: run-stack-qemu --check, test-users, test-storage, test-cloud-paperless, test-apps, test-firstinstall, test-transmission, test-media, test-dlna, test-vpn, test-tailscale, test-autobrr, test-zabbix – these seven need `make sysapps-pack`, test-snapshots, test-sysperms). UI work: `tests/mock-ubus.py`; app SDK in Chromium: `node tests/sdk-storage.js` (Playwright). ucode plugin: `tests/check-ucode-order.py` (no hoisting in ucode).
+- Tests: `tests/` (host: test-update, test-net, test-flash-bin, test-boards, test-raid-wait, test-smart, test-bootlog, test-power, test-notify, test-zabbix-items, test-ddns, test-ca (mbedtls/uci of the firmware through qemu-arm-static, see build.yml), check-acl.py, check-i18n.py; ARM emulator with sudo: run-stack-qemu --check, test-users, test-storage, test-cloud-paperless, test-apps, test-firstinstall, test-transmission, test-media, test-dlna, test-vpn, test-tailscale, test-autobrr, test-zabbix – these seven need `make sysapps-pack`, test-snapshots, test-sysperms). UI work: `tests/mock-ubus.py`; app SDK in Chromium: `node tests/sdk-storage.js` (Playwright). ucode plugin: `tests/check-ucode-order.py` (no hoisting in ucode).
 - Never `rm -rf` a chroot that still has mounts below it (it would delete the host's /dev).
 - Apps live in the separate repository UniqueDroid/ShareOS-Apps.
 - New UI texts: add them to `i18n/en.json` and to every `tools/i18n/<code>.tsv` (new numbers at the end), then `tools/i18n/build-lang.py`; `tests/check-i18n.py` must pass.

README.md

@@ -113,7 +113,8 @@ Version in brackets = first release with the feature.
 - First installation from the Alt-F or D-Link web interface: the `.bin` fetches the system from a USB stick or from GitHub (beta.7; tested on real hardware, step by step: [SWITCHING.md](docs/SWITCHING.md) · [UMSTIEG.md](docs/UMSTIEG.md))
 - Back to Alt-F (or any D-Link `.bin`) from the firmware page, checked and read back (beta.7)
 - First-run assistant, network settings with automatic rollback, name via mDNS (`<name>.local`), fallback address `192.168.0.32` without DHCP
-- HTTPS with a self-signed certificate that follows name and IP, own certificate possible
+- HTTPS with a self-signed certificate that follows name and IP, own certificate possible; own certificate authority for the home network and Let's Encrypt over DNS (0.2.0-beta.18)
+- Dynamic DNS: DuckDNS, dynv6, Cloudflare, No-IP or any dyndns2 address, IPv4 and IPv6 (0.2.0-beta.18)
 - Power schedule: switch off and on per weekday (switching on through the wake-up alarm of the DNS-320L/320 rev B/327L; 0.2.0-beta.15; switching off works from 0.2.0-beta.17, the overview shows the date of the next switch-on)
 - Start log: the logs of the last 3 starts are kept on the flash (System → Protokolle → *Start-Protokoll*), so a box that was unreachable after a start can be looked into afterwards (0.2.0-beta.15)
 - Reset button: access reset (5–10 s: network, root password, certificate) or factory reset (≥ 15 s), USB button removes USB drives (beta.9) or copies them into a dated folder (0.2.0-beta.12); the buttons work again on the DNS-320L and DNS-325 (0.2.0-beta.10)

configs/shareos_dns320l_defconfig

@@ -73,6 +73,7 @@ BR2_PACKAGE_LIBCURL=y
 BR2_PACKAGE_LIBCURL_CURL=y
 BR2_PACKAGE_LIBCURL_MBEDTLS=y
 BR2_PACKAGE_CA_CERTIFICATES=y
+BR2_PACKAGE_UACME=y
 
 # --- Images ---
 # Full system as squashfs (goes into the UBI volume "rootfs"); the small

package/shareos-core/shareos-core.mk

@@ -89,6 +89,10 @@ define SHAREOS_CORE_INSTALL_TARGET_CMDS
 	$(INSTALL) -D -m 0755 $(@D)/etc/init.d/rcS $(TARGET_DIR)/etc/init.d/rcS
 	$(INSTALL) -D -m 0755 $(@D)/usr/sbin/shareos-bootlog $(TARGET_DIR)/usr/sbin/shareos-bootlog
 	$(INSTALL) -D -m 0755 $(@D)/usr/sbin/shareos-power $(TARGET_DIR)/usr/sbin/shareos-power
+	$(INSTALL) -D -m 0755 $(@D)/usr/sbin/shareos-ddns $(TARGET_DIR)/usr/sbin/shareos-ddns
+	$(INSTALL) -D -m 0755 $(@D)/usr/sbin/shareos-ca $(TARGET_DIR)/usr/sbin/shareos-ca
+	$(INSTALL) -D -m 0755 $(@D)/usr/sbin/shareos-acme $(TARGET_DIR)/usr/sbin/shareos-acme
+	$(INSTALL) -D -m 0755 $(@D)/usr/libexec/shareos-acme-hook $(TARGET_DIR)/usr/libexec/shareos-acme-hook
 	$(INSTALL) -D -m 0755 $(@D)/usr/sbin/shareos-users $(TARGET_DIR)/usr/sbin/shareos-users
 	$(INSTALL) -D -m 0755 $(@D)/usr/sbin/shareos-shares $(TARGET_DIR)/usr/sbin/shareos-shares
 	$(INSTALL) -D -m 0755 $(@D)/usr/sbin/shareos-quota $(TARGET_DIR)/usr/sbin/shareos-quota

src/shareos-core/usr/libexec/shareos-acme-hook

@@ -0,0 +1,29 @@
+#!/bin/sh
+#
+# uacme hook of shareos-acme: answers only the DNS check (dns-01) through the
+# TXT record at the DynDNS provider (shareos-ddns).
+#
+#   shareos-acme-hook begin|done|failed <type> <ident> <token> <key_auth>
+#
+# uacme hands over the TXT value for dns-01 as key_auth. Exit 1 declines a
+# check type (http-01, tls-alpn-01: no port of the box is open to the internet).
+#
+PATH=/usr/sbin:/usr/bin:/sbin:/bin
+method=$1 type=$2 ident=$3 auth=$5
+[ "$type" = dns-01 ] || exit 1
+name="_acme-challenge.${ident#\*.}"
+case "$method" in
+	begin)
+		/usr/sbin/shareos-ddns txt-set "$name" "$auth" >/dev/null || exit 1
+		# wait until a public resolver sees the value (at most about 2 minutes)
+		for i in 1 2 3 4 5 6 7 8 9 10 11 12; do
+			sleep 10
+			nslookup -type=txt "$name" 1.1.1.1 2>/dev/null | grep -qF -- "$auth" && break
+		done
+		sleep 5
+		exit 0 ;;
+	done|failed)
+		/usr/sbin/shareos-ddns txt-clear "$name" "$auth" >/dev/null
+		exit 0 ;;
+esac
+exit 1

src/shareos-core/usr/sbin/shareos-acme

@@ -0,0 +1,132 @@
+#!/bin/sh
+#
+# shareos-acme – certificate from Let's Encrypt for the DynDNS name
+#
+#   shareos-acme issue [staging]   get a certificate for the DynDNS name
+#                                  (shareos.ddns.host) and install it for the
+#                                  web interface; staging: the test CA
+#   shareos-acme renew             renew it when less than 30 days are left
+#                                  (cron, daily) and install the new one
+#   shareos-acme status            JSON: name, valid until, last error
+#   shareos-acme off               back to the self-signed certificate
+#
+# The check runs over DNS (DNS-01): the TXT record _acme-challenge.<name> is
+# set at the DynDNS provider (shareos-ddns txt-set) – no port has to be open
+# to the internet. Works with DuckDNS and Cloudflare. uacme keeps the account
+# and the certificates in /etc/shareos/acme. Settings: UCI shareos.acme
+# (email: optional, for expiry mails of Let's Encrypt; staging '1').
+#
+PATH=/usr/sbin:/usr/bin:/sbin:/bin
+DIR=/etc/shareos/acme
+LOG=/tmp/shareos-acme.log
+HOOK=/usr/libexec/shareos-acme-hook
+
+die() { echo "shareos-acme: $*" >&2; echo "$*" > /tmp/shareos-acme.error; exit 1; }
+host() { uci -q get shareos.ddns.host | tr A-Z a-z; }
+name() {
+	h=$(host)
+	[ "$(uci -q get shareos.ddns.provider)" = duckdns ] && ! echo "$h" | grep -q '\.duckdns\.org$' && h="$h.duckdns.org"
+	echo "$h"
+}
+uacme_run() {
+	s=; [ "$(uci -q get shareos.acme.staging)" = 1 ] && s=-s
+	uacme -v -c "$DIR" -t EC -b 256 $s -h "$HOOK" "$@" >>$LOG 2>&1
+}
+restart_web() { (sleep 1; killall uhttpd) >/dev/null 2>&1 & }
+
+check() {
+	command -v uacme >/dev/null || die "uacme missing in this firmware"
+	[ "$(uci -q get shareos.ddns.enabled)" = 1 ] || die "switch on DynDNS first"
+	case "$(uci -q get shareos.ddns.provider)" in
+		duckdns|cloudflare) ;;
+		*) die "Let's Encrypt over DNS needs DuckDNS or Cloudflare as DynDNS provider" ;;
+	esac
+	[ -n "$(host)" ] || die "no DynDNS name"
+}
+
+account() {
+	[ -s "$DIR/private/key.pem" ] && return 0
+	mkdir -p "$DIR"; chmod 700 "$DIR"
+	m=$(uci -q get shareos.acme.email)
+	uacme_run -y new $m || die "Let's Encrypt account could not be created (see $LOG)"
+}
+
+install() {
+	n=$(name)
+	[ -s "$DIR/$n/cert.pem" ] && [ -s "$DIR/private/$n/key.pem" ] || die "no certificate from Let's Encrypt"
+	/usr/sbin/shareos-cert install "$DIR/$n/cert.pem" "$DIR/private/$n/key.pem" >/dev/null || die "the web server did not take the certificate"
+	uci -q get shareos.web >/dev/null || uci -q set shareos.web=web
+	uci -q get shareos.acme >/dev/null || uci -q set shareos.acme=acme
+	uci -q set shareos.web.source=acme
+	uci -q set shareos.acme.name="$n"
+	uci -q commit shareos
+	/usr/sbin/shareos-cron sync >/dev/null 2>&1
+	restart_web
+}
+
+issue() {
+	check
+	: > $LOG
+	rm -f /tmp/shareos-acme.error
+	uci -q get shareos.acme >/dev/null || uci -q set shareos.acme=acme
+	if [ "$1" = staging ]; then uci -q set shareos.acme.staging=1; else uci -q delete shareos.acme.staging; fi
+	uci -q commit shareos
+	# the account belongs to one CA: a new one when switching between test and real
+	ca=$( [ "$1" = staging ] && echo staging || echo prod )
+	[ "$(cat "$DIR/ca" 2>/dev/null)" = "$ca" ] || rm -rf "$DIR"
+	account
+	echo "$ca" > "$DIR/ca"
+	uacme_run -f issue "$(name)"
+	rc=$?
+	[ $rc = 0 ] || die "Let's Encrypt did not issue the certificate (see $LOG)"
+	install
+	logger -t shareos-acme "certificate for $(name) issued" 2>/dev/null
+	echo "certificate for $(name) issued and installed"
+}
+
+renew() {
+	[ "$(uci -q get shareos.web.source)" = acme ] || return 0
+	check
+	: > $LOG
+	uacme_run -d 30 issue "$(name)"
+	case $? in
+		0) install; rm -f /tmp/shareos-acme.error
+		   /usr/sbin/shareos-notify clear acme-error >/dev/null 2>&1
+		   logger -t shareos-acme "certificate for $(name) renewed" 2>/dev/null; echo renewed ;;
+		1) echo "still valid" ;;
+		*) echo "renewal failed" > /tmp/shareos-acme.error
+		   /usr/sbin/shareos-notify send warn network acme-error "Let's-Encrypt-Zertifikat nicht erneuert" \
+			"Let's Encrypt certificate not renewed" "Neuer Versuch morgen. Protokoll: $LOG" "Next try tomorrow. Log: $LOG" >/dev/null 2>&1
+		   exit 1 ;;
+	esac
+}
+
+status() {
+	n=$(uci -q get shareos.acme.name); on=false
+	[ "$(uci -q get shareos.web.source)" = acme ] && on=true
+	u=
+	[ -n "$n" ] && [ -s "$DIR/$n/cert.pem" ] && u=$( (mbedtls_cert_app mode=file filename="$DIR/$n/cert.pem" 2>/dev/null \
+		|| cert_app mode=file filename="$DIR/$n/cert.pem" 2>/dev/null) | sed -n 's/^ *expires on *: *//p' | head -1)
+	e=$(cat /tmp/shareos-acme.error 2>/dev/null | head -1 | sed 's/\\/\\\\/g;s/"/\\"/g')
+	staging=false; [ "$(uci -q get shareos.acme.staging)" = 1 ] && staging=true
+	printf '{"active":%s,"name":"%s","until":"%s","staging":%s,"email":"%s","error":"%s"}\n' \
+		$on "$n" "$u" $staging "$(uci -q get shareos.acme.email)" "$e"
+}
+
+off() {
+	[ "$(uci -q get shareos.web.source)" = acme ] || { echo "not active"; return; }
+	/usr/sbin/shareos-cert selfsigned >/dev/null || die "self-signed certificate failed"
+	uci -q delete shareos.web.source
+	uci -q commit shareos
+	/usr/sbin/shareos-cron sync >/dev/null 2>&1
+	restart_web
+	echo "back to the self-signed certificate"
+}
+
+case "$1" in
+	issue) issue "$2" ;;
+	renew) renew ;;
+	status) status ;;
+	off) off ;;
+	*) sed -n '3,17s/^# \{0,1\}//p' "$0"; exit 2 ;;
+esac