Repository navigation
0.47.0 upgrade advisory: database ownership startup failure and security hardening #834
matteius
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
If an upgrade to LightNVR 0.47.0 leaves an existing installation restarting with the following messages, check the ownership of its persistent database files:
We encountered this on two existing Kubernetes deployments and restored both on 0.47.0, without rolling back or deleting the database. This is an upgrade compatibility issue introduced by stricter private-file handling.
What happens, and why
The official container runs LightNVR as UID:GID 10001:10001. Our existing volumes contained database files owned by root:10001, with group write access. That allowed the application to write the database, but did not allow it to change the file's permissions. Kubernetes
fsGroup: 10001supplies group access; it does not make UID 10001 the file owner.In 0.47.0, writable database startup requires the database and existing SQLite
-wal,-shm, and-journalsidecars to have private permissions (0600). The application opens and validates each file, then callsfchmod. If that fails, startup stops before the HTTP server starts. The check also rejects symlinks, non-regular files, and files with multiple hard links, so ownership is not the only possible cause of this message. See the private database startup implementation.The reason is that database contents and journals can include camera credentials and authentication material. Permission hardening must cover existing files as well as newly created ones: specifying
0600when creating a file does not tighten an existing file's permissions. Refusing writable startup prevents the application from continuing to write sensitive data into files it cannot protect. This was introduced in the credential transport and private storage hardening change.There are two misleading symptoms worth recognizing:
Completedbefore Kubernetes put it intoCrashLoopBackOff. That was not a successful recorder startup. Increasing readiness/startup timeouts will not fix this ownership failure.Recovery for existing installations
With LightNVR stopped, inspect numeric ownership and modes on the configured database path and its sidecars. Make those files owned by the actual LightNVR runtime UID/GID, with
0600file permissions, and ensure the database and backup directories are owned and writable by that identity. We used0700for those directories. Adapt the UID/GID for custom images or native installations.Preserve the database and its WAL/journal files together; deleting sidecars can discard uncheckpointed data. Do not work around this by making files world-writable or running the recorder permanently as root. If the files are already correctly owned, investigate links, read-only mounts, or storage that does not support the required ownership/permission operations.
For our Kubernetes deployments, we added the following init container under
spec.template.spec.initContainers. It repairs metadata before LightNVR starts, including on subsequent pod replacements. The recorder itself continues to run unprivileged.Adapt
name: recordingsandsubPath: databaseto your PVC layout. This example assumes that subdirectory contains only LightNVR's database, sidecars, and database backups. Merge it with existing init containers; retain the deployment's existing volumes and application containers.This changes ownership and modes only within the dedicated database directory. It does not rewrite database contents or traverse the recordings directory, and it does not repair unsupported symlink/hard-link layouts. Where cluster policy disallows a root init container, have the storage administrator provision or repair the volume with the correct ownership before starting LightNVR. Keep the change in your deployment source so a controller or later redeploy does not remove it.
Both repaired deployments completed their rollouts with zero container restarts, and
/api/health?liveness=truereturnedhealthy: true. The larger database then spent about 51 seconds applying migration 0092's legacy-detection-label index. That was actual migration work after the permissions issue was resolved; allow sufficient startup-probe time for your database size.Other hardening in 0.47.0
This release also strengthens several related boundaries:
The ownership requirement is intentional; the failure to accommodate legacy root-owned volumes is the upgrade rough edge. The deployment repair above is verified for the two affected installations, not an automatic migration included in the 0.47.0 image.
See the 0.47.0 release notes and full changes since 0.46.3. If you encounter the same startup message, share the image version, runtime UID/GID, storage type, and numeric ownership/modes of the database files—please omit credentials and database contents.
All reactions