Overhaul local development workflow

[runleveldev]

Testing this PR

Once merged to main, all the latest references will work, until then it's best to build the required containers locally.

cd images
docker buildx bake --set manager.tags=ghcr.io/mieweb/opensource-server/manager:latest
cd ..
docker compose up -d --wait --build

Then the documentation should be accessible at https://localhost and the manager UI at https://manager.localhost.

Copilot Summary

This pull request introduces a comprehensive overhaul of the development and deployment workflow for the Proxmox-based LXC container hosting platform. The changes add a full Docker Compose-based local development stack that simulates a Proxmox cluster, automates image and dependency management, and clarifies documentation. In addition, there are improvements to cookie handling for authentication, service initialization, and infrastructure Dockerfiles.

Development and Deployment Workflow Overhaul:

• New Docker Compose Stack for Local Development:

  • Adds a new compose.yml at the repository root that orchestrates services for a local Proxmox cluster, manager, documentation, and supporting tasks, automating image pulls, dependency installation, and manager bootstrapping.
  • Removes the old create-a-container/compose.yml in favor of the new unified stack.
  • Provides a new Proxmox VE Docker image (images/proxmox-ve) with supporting configuration, systemd units, and health checks. [1] [2] [3] [4] [5]

• Documentation and Onboarding Improvements:

  • Rewrites the README.md to point users to a structured documentation site, clarify the repository layout, and provide clear getting started instructions for different audiences.

Authentication and Service Initialization:

• Improved Session Cookie Logic:

  • Refactors session cookie generation to comply with RFC 6265, ensuring correct domain scoping for subdomains, IP addresses, and localhost, and making the secure flag track the actual request protocol. [1] [2] [3]

• Systemd and Service Startup Enhancements:

  • Updates job-runner.service to depend on the container-creator initialization and to use a shared environment file for configuration. [1] [2]

Infrastructure and Image Improvements:

• Agent and Proxmox Image Updates:
  • Adds an auth cache directory for NGINX forward authentication and updates the bundled uv version for documentation building in the agent Dockerfile. [1] [2]

---

Development and Deployment Workflow Overhaul

• Added a new root-level compose.yml that sets up a local Proxmox cluster, manager, documentation builder, and supporting services for seamless local development and testing.
• Removed the old create-a-container/compose.yml in favor of the new unified stack.
• Introduced a new Proxmox VE Docker image (images/proxmox-ve) with systemd units, health checks, network configuration, and repo setup for realistic cluster simulation. [1] [2] [3] [4] [5]

Documentation and Onboarding Improvements

• Rewrote README.md to direct users to the documentation site, clarify project structure, and provide clear onboarding for different use cases.

Authentication and Service Initialization

• Improved session cookie logic in server.js to handle domain scoping and secure flag based on protocol, ensuring compatibility with subdomains, IPs, and localhost. [1] [2] [3]
• Updated job-runner.service to depend on initialization and use a shared environment file for configuration. [1] [2]

Infrastructure and Image Improvements

• Added auth cache directory for NGINX forward authentication and updated the documentation build tool version in the agent Dockerfile. [1] [2]

[cmyers-mieweb]

Attempted a local docker compose to test, but getting an error on agent step 11/12

 => [agent 10/12] COPY . /opt/opensource-server                                                                                                                        1.3s
 => ERROR [agent 11/12] RUN cd /opt/opensource-server     && make install-pull-config                                                                                  1.0s
------
 > [agent 11/12] RUN cd /opt/opensource-server     && make install-pull-config:
0.560 cd pull-config && bash install.sh
0.567 install.sh: line 2: $'\r': command not found
: invalid option namee 3: set: pipefail
0.568 make: *** [Makefile:26: install-pull-config] Error 2
------
Dockerfile:53
--------------------
  52 |     COPY . /opt/opensource-server
  53 | >>> RUN cd /opt/opensource-server \
  54 | >>>     && make install-pull-config
  55 |     
--------------------
ERROR: target manager: failed to solve: process "/bin/sh -c cd /opt/opensource-server     && make install-pull-config" did not complete successfully: exit code: 2
PS C:\Users\cmyers\Documents\GitHub\opensource-server\images> 

going to attempt another test

[cmyers-mieweb]

docker compose up -d --wait --build

Was able to move install.sh to LF from CRLF and got a build ready to compose, on launch it looks like a proxmox dependency is failing to start.

#20 [proxmox] resolving provenance for metadata file
[+] up 22/22s
 ✔ Image node:24-trixie-slim                                Pulled                                                                                                     21.7s
 ✔ Image astral/uv:0.11.14-trixie-slim                      Pulled                                                                                                     14.2s
 ✔ Image curlimages/curl:latest                             Pulled                                                                                                      6.6s
 ✔ Image ghcr.io/mieweb/opensource-server/proxmox-ve:latest Built                                                                                                       6.8s
 ✔ Network opensource-server_default                        Created                                                                                                     0.1s
 ✔ Volume opensource-server_local                           Created                                                                                                     0.0s
 ✔ Container opensource-server-pull-image-1                 Exited                                                                                                    120.8s
 ✔ Container opensource-server-node-1                       Exited                                                                                                    192.7s
 ✔ Container opensource-server-zensical-1                   Started                                                                                                     1.7s
 ✘ Container opensource-server-proxmox-1                    Error dependency proxmox failed to start                                                                  388.8s
 ✔ Container opensource-server-bootstrap-manager-1          Created                                                                                                     0.1s
dependency failed to start: container opensource-server-proxmox-1 is unhealthy