From b4f4dc29524cce985e9ad18978993b38cc19846a Mon Sep 17 00:00:00 2001 From: John Watson Date: Wed, 22 Jul 2026 17:41:54 +0100 Subject: [PATCH 1/7] Migrate guides from FCP Development guide --- docs/guides/caching.md | 134 +++++++++++++++++++++ docs/guides/docker_guidance.md | 203 ++++++++++++++++++++++++++++++++ docs/guides/sonarqube_ide.md | 35 ++++++ docs/processes/pull_requests.md | 116 ++++++++++++++++++ mkdocs.yml | 6 + 5 files changed, 494 insertions(+) create mode 100644 docs/guides/caching.md create mode 100644 docs/guides/sonarqube_ide.md diff --git a/docs/guides/caching.md b/docs/guides/caching.md new file mode 100644 index 00000000..b5f4b762 --- /dev/null +++ b/docs/guides/caching.md @@ -0,0 +1,134 @@ +# Caching + +## Redis + +When an external cache is needed, Redis is the recommended choice. For local development, a [Docker image](https://hub.docker.com/_/redis) can be used. + +## Hapi.js server-side caching + +[Hapi caching](https://hapi.dev/tutorials/caching/) is described in the official documentation. The following guide describes how server-side Redis caching can be added to a Hapi-based service. + +Hapi server-side caching uses the [catbox](https://hapi.dev/module/catbox/) interface to abstract away the underlying caching technology being used (e.g. memory, Redis, Memcached). + +There are three main concepts to Hapi server-side caching: + +* The cache **strategy** (or provider): the underlying caching technology being employed. [catbox-redis](https://github.com/hapijs/catbox-redis) is the Redis adapter for catbox. +* The cache **client**: a low-level cache abstraction, initialised using a cache strategy. Hapi initialises an in-memory cache client by default, and you can create additional cache clients using the same or different strategies. +* The cache **policy**: a higher-level cache abstraction that sets a policy on the storage within the cache (e.g. expiry times). The cache policy also provides additional segmentation within the cache client. Typically the cache policy is how you interact with cache values via the `get` and `set` methods. + +### Configuring the default cache client + +Hapi initialises an in-memory cache client by default. You can make the default cache client use a different strategy. For example, the following uses Redis or memory depending on configuration: + +```javascript +const catbox = config.useRedis ? require('@hapi/catbox-redis') : require('@hapi/catbox-memory') +const catboxOptions = config.useRedis + ? { + host: process.env.REDIS_HOSTNAME, + port: process.env.REDIS_PORT, + password: process.env.REDIS_PASSWORD, + partition: process.env.REDIS_PARTITION, + tls: process.env.NODE_ENV === 'production' ? {} : undefined + } + : {} + +const server = hapi.server({ + port: config.port, + cache: [{ + provider: { + constructor: catbox, + options: catboxOptions + } + }] +}) +``` + +### Configuring additional cache clients + +Additional cache clients can be created when initialising the Hapi server by adding new definitions to the `cache` array. Additional caches must be given a `name`. For example, the following creates a new Redis cache client called `session`: + +```javascript +const catbox = require('@hapi/catbox-redis') +const catboxOptions = { + host: process.env.REDIS_HOSTNAME, + port: process.env.REDIS_PORT, + password: process.env.REDIS_PASSWORD, + partition: process.env.REDIS_PARTITION, + tls: process.env.NODE_ENV === 'production' ? {} : undefined +} + +const server = hapi.server({ + port: config.port, + cache: [{ + name: 'session', + provider: { + constructor: catbox, + options: catboxOptions + } + }] +}) +``` + +This example creates two cache clients: the default in-memory cache client and a new cache client called `session` that uses Redis. + +Hapi will always use the default in-memory cache client unless you specify the `name` when using it (either directly or via the cache policy). + +### Creating and using a cache policy + +The cache policy is typically how you interact with the cache. See the [catbox policy documentation](https://hapi.dev/module/catbox/api/?v=11.1.1#policy) for details on how to set and get data. + +To create a cache policy using a segment within the default cache client: + +```javascript +const myCache = server.cache({ + expiresIn: 36000, + segment: 'mySegment' +}) +``` + +To create a cache policy using a segment within a named cache client: + +```javascript +const myCache = server.cache({ + cache: 'session', + expiresIn: 36000, + segment: 'mySegment' +}) +``` + +## Integration with yar session cookies + +[Hapi yar](https://hapi.dev/module/yar/) is a plugin that adds unauthenticated session support (state across multiple browser requests) to Hapi. By default it tries to fit session data into a session cookie, but will use server-side storage via the Hapi cache interface if the session data exceeds the maximum cookie size. + +Combining Hapi yar with Redis caching allows multiple replicas of a web server to share server-side user session data. + +Example configuration using the default cache client: + +```javascript +server.register({ + plugin: require('@hapi/yar'), + options: { + cache: { + expiresIn: 36000 + }, + maxCookieSize: 0 + } +}) +``` + +Setting `maxCookieSize` to `0` forces all session data to be stored server-side. + +Example configuration using a named cache client: + +```javascript +server.register({ + plugin: require('@hapi/yar'), + options: { + cache: { + cache: 'session', + expiresIn: 36000 + }, + maxCookieSize: 0 + } +}) +``` diff --git a/docs/guides/docker_guidance.md b/docs/guides/docker_guidance.md index c5bddd7e..c48b0898 100644 --- a/docs/guides/docker_guidance.md +++ b/docs/guides/docker_guidance.md @@ -2,6 +2,12 @@ A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably across multiple environments. Docker is a tool to build and run these containers. +## Local development principles + +- Teams must not constrain their local development setup to a specific device or operating system to maintain developer mobility and agility. +- Local development should maximise the use of emulation and avoid tight coupling to cloud services where possible. +- Repositories should include full instructions for anyone to be able to easily run the service locally. + ## More information [Docker introduction on docker.com](https://www.docker.com/resources/what-container) @@ -350,3 +356,200 @@ docker() { fi } ``` + +## Local development workflow + +This section describes conventions for using Docker Compose to support an efficient local development workflow including hot reload, test-driven development, and debugging. + +### Compose file conventions + +Separate Compose files by concern to allow different combinations for different workflows: + +| File | Purpose | +|------|---------| +| `docker-compose.yaml` | Production image build and core service definitions. No port or volume bindings. | +| `docker-compose.override.yaml` | Local development overrides: port bindings, volume mounts, watch mode. Applied automatically by `docker compose up`. | +| `docker-compose.debug.yaml` | Starts the application waiting for a debugger before executing code. | +| `docker-compose.test.yaml` | Runs tests. No port bindings (avoids conflicts in CI). | +| `docker-compose.test.watch.yaml` | Override for `docker-compose.test.yaml` to run tests in watch mode for TDD. | +| `docker-compose.test.debug.yaml` | Runs tests in watch mode with debugger support. | + +### Hot reload with watch mode + +The development image should run the application in watch mode so that code changes are automatically picked up without rebuilding. For Node.js services, [nodemon](https://www.npmjs.com/package/nodemon) is used to restart the application when files change. + +This requires binding the application source as a volume in `docker-compose.override.yaml`: + +```yaml +volumes: + - ./app:/home/node/app +``` + +### Avoiding port conflicts + +When running multiple services locally, each must bind to a unique host port. Map container ports to different host ports per service: + +```yaml +# service 1 docker-compose.override.yaml +ports: + - "3000:3000" + - "9229:9229" + +# service 2 docker-compose.override.yaml +ports: + - "3001:3000" + - "9230:9229" +``` + +The same applies to dependency containers (databases, caches). + +### package.json scripts + +To support the above workflow, configure scripts in `package.json`: + +```json +"scripts": { + "test": "jest --runInBand --forceExit", + "test:watch": "jest --coverage=false --onlyChanged --watch --runInBand", + "test:debug": "node --inspect-brk=0.0.0.0 ./node_modules/jest/bin/jest.js --coverage=false --onlyChanged --watch --runInBand --no-cache", + "start:watch": "nodemon --inspect=0.0.0.0 --ext js --legacy-watch app/index.js", + "start:debug": "nodemon --inspect-brk=0.0.0.0 --ext js --legacy-watch app/index.js" +} +``` + +- `test` — runs all tests sequentially with coverage. Used in CI. +- `test:watch` — runs only changed tests in watch mode. Primary TDD workflow. +- `test:debug` — watch mode with `--inspect-brk` to wait for debugger attachment. +- `start:watch` — runs the app with nodemon for automatic restart on changes. +- `start:debug` — same as watch but waits for debugger before executing. + +For Jest watch to detect which files have changed, the `.git` directory must be mounted to the test container: + +```yaml +volumes: + - ./.git:/home/node/.git +``` + +### Debugging in VS Code + +Add the following debug configurations to `.vscode/launch.json` (excluded from source control). + +#### Attach to a running container + +```json +{ + "name": "Docker: Attach", + "type": "node", + "request": "attach", + "restart": true, + "port": 9229, + "remoteRoot": "/home/node", + "skipFiles": [ + "/**", + "**/node_modules/**" + ] +} +``` + +`restart: true` ensures the debugger reattaches when nodemon restarts the app. + +#### Attach to Jest tests in debug mode + +```json +{ + "name": "Docker: Jest Attach", + "type": "node", + "request": "attach", + "port": 9229, + "restart": true, + "timeout": 10000, + "remoteRoot": "/home/node", + "disableOptimisticBPs": true, + "continueOnAttach": true, + "skipFiles": [ + "/**", + "**/node_modules/**" + ] +} +``` + +`disableOptimisticBPs` is required because Jest copies test files before execution, which can cause breakpoint mapping issues. + +#### Launch container in debug mode via VS Code task + +```json +{ + "name": "Docker: Attach Launch", + "type": "node", + "request": "attach", + "remoteRoot": "/home/node", + "restart": true, + "port": 9229, + "skipFiles": [ + "/**", + "**/node_modules/**" + ], + "preLaunchTask": "compose-debug-up", + "postDebugTask": "compose-debug-down" +} +``` + +With supporting `.vscode/tasks.json`: + +```json +{ + "version": "2.0.0", + "tasks": [ + { + "label": "compose-debug-up", + "type": "shell", + "command": "docker compose -f docker-compose.yaml -f docker-compose.override.yaml -f docker-compose.debug.yaml up -d" + }, + { + "label": "compose-debug-down", + "type": "shell", + "command": "docker compose -f docker-compose.yaml -f docker-compose.override.yaml -f docker-compose.debug.yaml down" + } + ] +} +``` + +### Debugging .NET in a Linux container + +.NET services developed in Linux containers require the `vsdbg` remote debugger, which is included in the Defra .NET development base image. + +#### VS Code + +```json +{ + "name": ".NET Core Docker Attach", + "type": "coreclr", + "request": "attach", + "processId": "${command:pickRemoteProcess}", + "pipeTransport": { + "pipeProgram": "docker", + "pipeArgs": ["exec", "-i", "my-service-container"], + "debuggerPath": "/vsdbg/vsdbg", + "pipeCwd": "${workspaceRoot}", + "quoteArgs": false + }, + "sourceFileMap": { + "/home/dotnet": "${workspaceFolder}" + } +} +``` + +#### Visual Studio + +Visual Studio does not integrate with the WSL filesystem, so WSL users must clone the repository in Windows to debug using Visual Studio. Ensure the following git configuration is set to preserve line endings: + +```bash +git config --global core.autocrlf input +``` + +1. Start the container with `docker-compose up --build` +2. In Visual Studio, select `Debug -> Attach to process` +3. Select `Docker (Linux Container)` for connection type +4. Enter the container name in connection target +5. Select the process matching the running application +6. Select `Managed (.NET Core for Unix)` code type diff --git a/docs/guides/sonarqube_ide.md b/docs/guides/sonarqube_ide.md new file mode 100644 index 00000000..1da62e89 --- /dev/null +++ b/docs/guides/sonarqube_ide.md @@ -0,0 +1,35 @@ +# SonarQube for IDE + +[SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/) is an IDE extension that identifies code quality issues as you code, providing immediate feedback before committing. + +All Defra projects are required to set up their repositories within the [SonarQube Cloud Defra organisation](https://sonarcloud.io/organizations/defra/projects). Using the IDE extension in connected mode ensures you get the same rules and quality gates locally as in CI. + +## Dependencies + +- Java Runtime Environment v21+ + +With Ubuntu (including WSL), install the open source JRE: + +```bash +sudo apt-get install openjdk-21-jre +``` + +## VS Code setup + +1. Install the [SonarQube for IDE](https://marketplace.visualstudio.com/items?itemName=SonarSource.sonarlint-vscode) extension. + +2. Set the JRE location in VS Code settings: + + ```json + { + "sonarlint.ls.javaHome": "/usr/lib/jvm/java-21-openjdk-amd64" + } + ``` + +This gives you Sonar code analysis using default quality gates for supported languages. + +## Connected mode + +Connected mode binds the extension to your actual project in SonarQube Cloud. This ensures the IDE uses the same rules, quality gates, and exclusions as the SonarQube project configured for your repository. + +Follow the [connected mode setup documentation](https://docs.sonarsource.com/sonarqube-for-ide/vs-code/team-features/connected-mode-setup/#connection-setup) to connect your SonarQube for IDE extension to the SonarQube Cloud project. diff --git a/docs/processes/pull_requests.md b/docs/processes/pull_requests.md index ae340a42..f53e1579 100644 --- a/docs/processes/pull_requests.md +++ b/docs/processes/pull_requests.md @@ -97,3 +97,119 @@ The simplest way to do this is in the GitHub UI. Within the PR the merge button When you click it GitHub will present a box which contains the combined text from all the commit messages in the PR. Use this opportunity to reword the content to a single commit message (the smaller the change the simpler this is to do!) When done ensure you delete the branch. Again GitHub will present this option in the UI immediately after merging so make use of it then. + +## Reviewing a pull request + +When code reviews are conducted to a high standard, they provide a valuable learning opportunity for the author, reviewer and any observers of the review process. + +At least one reviewer must approve a pull request before it can be merged. + +### Tone of code review comments + +The tone of communications is extremely important in fostering an inclusive, collaborative atmosphere within teams. + +Remember that your colleagues put a lot of effort into their work and may feel offended by harsh criticism, particularly if you make assumptions or imply a lack of thought. Approach code reviews with kindness and empathy, assuming the best intentions and applauding good solutions as well as making suggestions for improvement. + +Comments should be used to give praise for good solutions and to point out potential improvements. They should not be used to criticise your colleagues or make strongly opinionated statements. Always be mindful of your tone, considering how others might perceive your comments, and be explicit about when a comment is non-blocking or unimportant. + +#### Be constructive + +Don't use harsh language or criticise without making constructive suggestions. +Do suggest alternative approaches and explain your reasoning. + +#### Be specific + +Don't make vague statements about the changes as a whole. +Do point out specific issues and offer specific ideas for how the changes can be improved. + +#### Avoid strong opinions + +Don't make strong, opinionated statements or dictate specific changes. +Do ask open-ended questions and keep an open mind about alternative solutions and reasoning that you may not have thought of. + +### Scope of a code review + +Code reviews should focus on what is being changed and whether the change is appropriate. The scope will be stated in the acceptance criteria of the ticket. + +Expanding the scope of a pull request at the review stage is not acceptable. It is generally more valuable to swiftly conclude a piece of work that the team has prioritised than to opportunistically seek additional changes, such as refactoring related code. That said, it can be useful to comment on refactoring opportunities without blocking the pull request. + +### What to look for + +#### Are the changes focused? + +Are the changes focused on a specific issue, referenced in the pull request description? If the changes go beyond the intended scope, should they be broken up to make the code review more manageable? + +#### Maintainability + +Is all new code extensible and easy for other developers to understand? Does it follow common design patterns? Look for unnecessary complexity and remember that this is subjective so take care not to be overly critical or opinionated. + +#### Duplication + +Is there duplication within new code or between new and existing code? Could an existing abstraction be reused or should a new abstraction be created? + +Identifying useful abstractions at the review stage may indicate that there wasn't enough collaboration before coding began. See this as a trigger to review the team's ways of working, rather than blocking a pull request unnecessarily. + +#### Reusability + +Have any new abstractions been introduced? Are they sufficiently reusable? If other parts of the system could be updated to use the new abstractions, consider suggesting this in a non-blocking comment. + +#### Impact on other parts of the system + +Will the changes have knock-on effects or otherwise necessitate changes to other parts of the system? + +#### Unit test coverage + +Is all new code covered by detailed unit tests? Have any edge cases been missed? Don't rely on metrics such as code coverage. Inspect the code thoroughly and ensure that the tests contain appropriate assertions to confirm that all the intended functionality works as expected. + +#### Integration tests + +Have integration tests been added to cover all changes to functionality? + +### Concluding a review + +When concluding a review, there are three options: + +1. Comment +2. Approve the PR +3. Block the PR by requesting changes + +#### When to comment + +You should conclude with a comment when your review asks questions which need answering before you can determine whether the pull request is acceptable. If you haven't proposed a solution to a specific problem in the pull request, it's generally better to leave a neutral review than to block the PR with a request for change. + +#### When to approve + +You should approve a pull request when you have confirmed that: + +1. it meets the objectives set out in its description +2. it doesn't introduce new defects or code that is hard to maintain +3. all new and modified code has thorough unit test coverage +4. integration tests have been added where appropriate +5. there are no unresolved questions or comments against the pull request + +Occasionally, it may be prudent to accept a pull request which does not meet all of the above requirements, such as to resolve an urgent issue with the live product. Such cases must always be agreed between the product owner, author(s) and reviewer(s). + +If comments or questions on a pull request have been addressed elsewhere (e.g. face-to-face or on Slack), ensure that the outcome is recorded in comment replies so that it is visible to anyone looking back at the pull request in future. + +Use the Resolve button to make it clear which of your concerns have been addressed and which still need attention. A pull request should only be approved after all reviewers have explicitly indicated that each of their comments have been resolved. + +Sometimes, a reviewer may become unavailable after commenting on a pull request. When that happens, a second reviewer may accept the pull request without resolving the first reviewer's comments, as long as the author and second reviewer agree that they believe all legitimate concerns have been addressed. + +#### When to request changes + +You should request changes to a pull request if any of the following are true: + +1. it creates a defect in the product +2. it exacerbates an existing defect +3. it doesn't meet the objectives set out in its description +4. it would make the product more difficult to maintain +5. new or modified code lacks thorough unit tests +6. required integration tests have not been included + +#### When **not** to request changes + +> Perfection is the enemy of good + +A pull request does not need to be perfect to be good enough. Often, there are many solutions to a problem and one which is not the best is still good enough to meet current needs. + +Make suggestions for improvement as comments without explicitly approving or rejecting the pull request. This way, your suggestions can open dialogue with the author about how important your suggestions are compared to other work you could each move on to. diff --git a/mkdocs.yml b/mkdocs.yml index 23c817a5..e934eae3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -56,10 +56,15 @@ nav: - Resolving GitHub security alerts: processes/github_security_alerts.md - Guides: - General guidance: guides/README.md + - Caching: guides/caching.md - Choosing between a mono repo and a multi repo: guides/mono_or_multi_repo.md - Choosing packages: guides/choosing_packages.md + - Cookie banner: guides/cookies-banner.md + - Defra Identity: guides/defra-id.md - Developer workflows: guides/developer_workflows.md - Docker guidance: guides/docker_guidance.md + - Entra: guides/entra.md + - GitHub Advanced Security: guides/github_advanced_security.md - Java auto-format with Eclipse: guides/java_auto_format_eclipse.md - Kubernetes: guides/kubernetes.md - Managing application credentials: guides/application_credentials.md @@ -69,6 +74,7 @@ nav: - PL/SQL auto-format with TOAD: guides/plsql_auto_format_toad.md - SQL Prompt: guides/sql_prompt_tool.md - Shared libraries: guides/shared_libraries.md + - SonarQube for IDE: guides/sonarqube_ide.md - Style guide for standards: guides/style_guide_for_standards.md - Training: guides/training.md - Using AWS session manager: guides/aws_session_manager.md From 7fa24bd5c92523ea06e190c70fc2f49facc8b883 Mon Sep 17 00:00:00 2001 From: John Watson Date: Wed, 22 Jul 2026 18:01:53 +0100 Subject: [PATCH 2/7] Update caching guide title to reflect server-side caching with Hapi.js --- docs/guides/caching.md | 121 ++++++++++++++++++++++++++--------------- mkdocs.yml | 2 +- 2 files changed, 79 insertions(+), 44 deletions(-) diff --git a/docs/guides/caching.md b/docs/guides/caching.md index b5f4b762..bc07acc1 100644 --- a/docs/guides/caching.md +++ b/docs/guides/caching.md @@ -1,4 +1,4 @@ -# Caching +# Server-Side Caching with Hapi.js ## Redis @@ -6,38 +6,41 @@ When an external cache is needed, Redis is the recommended choice. For local dev ## Hapi.js server-side caching -[Hapi caching](https://hapi.dev/tutorials/caching/) is described in the official documentation. The following guide describes how server-side Redis caching can be added to a Hapi-based service. +Hapi server-side caching uses the [catbox](https://hapi.dev/module/catbox/) interface to abstract the underlying caching technology (e.g. memory, Redis). -Hapi server-side caching uses the [catbox](https://hapi.dev/module/catbox/) interface to abstract away the underlying caching technology being used (e.g. memory, Redis, Memcached). +There are three main concepts: -There are three main concepts to Hapi server-side caching: - -* The cache **strategy** (or provider): the underlying caching technology being employed. [catbox-redis](https://github.com/hapijs/catbox-redis) is the Redis adapter for catbox. -* The cache **client**: a low-level cache abstraction, initialised using a cache strategy. Hapi initialises an in-memory cache client by default, and you can create additional cache clients using the same or different strategies. -* The cache **policy**: a higher-level cache abstraction that sets a policy on the storage within the cache (e.g. expiry times). The cache policy also provides additional segmentation within the cache client. Typically the cache policy is how you interact with cache values via the `get` and `set` methods. +* The cache **strategy** (or provider): the underlying caching technology. [`@hapi/catbox-redis`](https://github.com/hapijs/catbox-redis) is the Redis adapter for catbox. +* The cache **client**: a low-level cache abstraction, initialised using a strategy. Hapi initialises an in-memory cache client by default, and you can create additional cache clients using the same or different strategies. +* The cache **policy**: a higher-level cache abstraction that sets a policy on storage within the cache (e.g. expiry times). The cache policy provides additional segmentation within the cache client. Typically the cache policy is how you interact with cache values via the `get` and `set` methods. ### Configuring the default cache client Hapi initialises an in-memory cache client by default. You can make the default cache client use a different strategy. For example, the following uses Redis or memory depending on configuration: ```javascript -const catbox = config.useRedis ? require('@hapi/catbox-redis') : require('@hapi/catbox-memory') -const catboxOptions = config.useRedis +import Hapi from '@hapi/hapi' +import { Engine as CatboxRedis } from '@hapi/catbox-redis' +import { Engine as CatboxMemory } from '@hapi/catbox-memory' +import config from './config.js' + +const CacheEngine = config.useRedis ? CatboxRedis : CatboxMemory +const cacheOptions = config.useRedis ? { - host: process.env.REDIS_HOSTNAME, - port: process.env.REDIS_PORT, - password: process.env.REDIS_PASSWORD, - partition: process.env.REDIS_PARTITION, - tls: process.env.NODE_ENV === 'production' ? {} : undefined + host: config.cache.host, + port: config.cache.port, + password: config.cache.password, + partition: config.cache.partition, + tls: config.isProd ? {} : undefined } : {} -const server = hapi.server({ +const server = Hapi.server({ port: config.port, cache: [{ provider: { - constructor: catbox, - options: catboxOptions + constructor: CacheEngine, + options: cacheOptions } }] }) @@ -48,22 +51,23 @@ const server = hapi.server({ Additional cache clients can be created when initialising the Hapi server by adding new definitions to the `cache` array. Additional caches must be given a `name`. For example, the following creates a new Redis cache client called `session`: ```javascript -const catbox = require('@hapi/catbox-redis') -const catboxOptions = { - host: process.env.REDIS_HOSTNAME, - port: process.env.REDIS_PORT, - password: process.env.REDIS_PASSWORD, - partition: process.env.REDIS_PARTITION, - tls: process.env.NODE_ENV === 'production' ? {} : undefined -} - -const server = hapi.server({ +import Hapi from '@hapi/hapi' +import { Engine as CatboxRedis } from '@hapi/catbox-redis' +import config from './config.js' + +const server = Hapi.server({ port: config.port, cache: [{ name: 'session', provider: { - constructor: catbox, - options: catboxOptions + constructor: CatboxRedis, + options: { + host: config.cache.host, + port: config.cache.port, + password: config.cache.password, + partition: config.cache.partition, + tls: config.isProd ? {} : undefined + } } }] }) @@ -75,13 +79,13 @@ Hapi will always use the default in-memory cache client unless you specify the ` ### Creating and using a cache policy -The cache policy is typically how you interact with the cache. See the [catbox policy documentation](https://hapi.dev/module/catbox/api/?v=11.1.1#policy) for details on how to set and get data. +The cache policy is typically how you interact with the cache. See the [catbox policy documentation](https://hapi.dev/module/catbox/api/) for details on how to set and get data. To create a cache policy using a segment within the default cache client: ```javascript const myCache = server.cache({ - expiresIn: 36000, + expiresIn: config.cache.ttl, segment: 'mySegment' }) ``` @@ -91,27 +95,33 @@ To create a cache policy using a segment within a named cache client: ```javascript const myCache = server.cache({ cache: 'session', - expiresIn: 36000, + expiresIn: config.cache.ttl, segment: 'mySegment' }) ``` ## Integration with yar session cookies -[Hapi yar](https://hapi.dev/module/yar/) is a plugin that adds unauthenticated session support (state across multiple browser requests) to Hapi. By default it tries to fit session data into a session cookie, but will use server-side storage via the Hapi cache interface if the session data exceeds the maximum cookie size. +[`@hapi/yar`](https://hapi.dev/module/yar/) is a plugin that adds unauthenticated session support (state across multiple browser requests) to Hapi. By default it tries to fit session data into a session cookie, but will use server-side storage via the Hapi cache interface if the session data exceeds the maximum cookie size. -Combining Hapi yar with Redis caching allows multiple replicas of a web server to share server-side user session data. +Combining yar with Redis caching allows multiple replicas of a web server to share server-side user session data. Example configuration using the default cache client: ```javascript -server.register({ - plugin: require('@hapi/yar'), +import Yar from '@hapi/yar' + +await server.register({ + plugin: Yar, options: { + maxCookieSize: 0, cache: { - expiresIn: 36000 + expiresIn: config.cache.ttl }, - maxCookieSize: 0 + cookieOptions: { + password: config.cookie.password, + isSecure: config.isProd + } } }) ``` @@ -121,14 +131,39 @@ Setting `maxCookieSize` to `0` forces all session data to be stored server-side. Example configuration using a named cache client: ```javascript -server.register({ - plugin: require('@hapi/yar'), +import Yar from '@hapi/yar' + +await server.register({ + plugin: Yar, options: { + maxCookieSize: 0, cache: { cache: 'session', - expiresIn: 36000 + expiresIn: config.cache.ttl }, - maxCookieSize: 0 + cookieOptions: { + password: config.cookie.password, + isSecure: config.isProd + } } }) ``` + +## catbox-redis connection options + +`@hapi/catbox-redis` v7 uses [ioredis](https://github.com/redis/ioredis) under the hood. The following options are supported: + +| Option | Description | +| --- | --- | +| `host` | Redis server hostname (default: `127.0.0.1`) | +| `port` | Redis server port (default: `6379`) | +| `password` | Authentication password | +| `db` | Database number | +| `partition` | Key prefix for cache segmentation | +| `tls` | TLS configuration object (pass `{}` for default TLS settings) | +| `url` | Redis connection URL (alternative to `host`/`port`) | +| `client` | Pre-configured ioredis instance (must expose a `status` property set to `'ready'`) | +| `sentinels` | Array of `{ host, port }` sentinel addresses | +| `sentinelName` | Sentinel master name (required with `sentinels`) | + +Only one connection method should be used: `host`/`port`, `url`, or `client`. diff --git a/mkdocs.yml b/mkdocs.yml index e934eae3..070996da 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -56,7 +56,7 @@ nav: - Resolving GitHub security alerts: processes/github_security_alerts.md - Guides: - General guidance: guides/README.md - - Caching: guides/caching.md + - Server-Side Caching with Hapi.js: guides/caching.md - Choosing between a mono repo and a multi repo: guides/mono_or_multi_repo.md - Choosing packages: guides/choosing_packages.md - Cookie banner: guides/cookies-banner.md From 22fefdd6edf9a10cab39a216b99663b0586efac2 Mon Sep 17 00:00:00 2001 From: John Watson Date: Thu, 23 Jul 2026 08:52:47 +0100 Subject: [PATCH 3/7] Iterate docker guidance --- docs/guides/docker_guidance.md | 623 +++++++++++++-------------------- 1 file changed, 247 insertions(+), 376 deletions(-) diff --git a/docs/guides/docker_guidance.md b/docs/guides/docker_guidance.md index c48b0898..72ecb112 100644 --- a/docs/guides/docker_guidance.md +++ b/docs/guides/docker_guidance.md @@ -1,471 +1,339 @@ # Docker guidance -A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably across multiple environments. Docker is a tool to build and run these containers. +A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably across multiple environments. Docker is a tool to build and run these containers. + +Docker remains the default way to package and run Defra services, both in the pipeline and for local development. This guide covers building images, composing local environments, running tests, and debugging. It reflects current tooling: Docker Compose v2 (`docker compose`), Node.js 24, and .NET 10. ## Local development principles - Teams must not constrain their local development setup to a specific device or operating system to maintain developer mobility and agility. - Local development should maximise the use of emulation and avoid tight coupling to cloud services where possible. - Repositories should include full instructions for anyone to be able to easily run the service locally. +- Keep the local setup lean. Favour a single Compose file per repository over many overlapping variants, and prefer built-in tooling (for example Node's native watch and `--env-file`) over extra dependencies. ## More information + [Docker introduction on docker.com](https://www.docker.com/resources/what-container) ## Terminology -`Dockerfile` - set of instructions for building a docker image -`Image` - a constructed set of layered docker instructions -`Container` - a running instance of an image -## Multi stage builds -Dockerfiles should implement multi stage builds to allow different build stages to be targeted for specific purposes. For example, a final production image does not need all the unit test files and a unit test running image would use a different running command than the application. +`Dockerfile` - set of instructions for building a Docker image +`Image` - a constructed set of layered Docker instructions +`Container` - a running instance of an image +`Compose file` - a `compose.yaml` file describing how to build and run one or more services -Below is an example multi stage build which is intended to use the Defra Node.js base image. +> **Docker Compose v1 has reached end of life.** Use the Compose v2 plugin, invoked as `docker compose` (with a space), not the standalone `docker-compose` binary. The canonical Compose filename is `compose.yaml`, and the top-level `version:` key is obsolete and should be omitted. -``` -ARG PARENT_VERSION=1.0.0-node12.16.0 -ARG PORT=3000 -ARG PORT_DEBUG=9229 +## Base images -# Development -FROM defradigital/node-development:${PARENT_VERSION} AS development -ARG PARENT_VERSION -ARG REGISTRY -LABEL uk.gov.defra.parent-image=defradigital/node-development:${PARENT_VERSION} -ARG PORT -ENV PORT ${PORT} -ARG PORT_DEBUG -EXPOSE ${PORT} ${PORT_DEBUG} -COPY --chown=node:node package*.json ./ -RUN npm install -COPY --chown=node:node app/ ./app/ -RUN npm run build -CMD [ "npm", "run", "start:watch" ] +Defra publishes hardened base images that provide a non-root user, CA certificates, and the debugging tooling needed for local development. Build on these rather than the raw upstream images: -# Production -FROM defradigital/node:${PARENT_VERSION} AS production -ARG PARENT_VERSION -ARG REGISTRY -LABEL uk.gov.defra.parent-image=defradigital/node:${PARENT_VERSION} -ARG PORT -ENV PORT ${PORT} -EXPOSE ${PORT} -COPY --from=development /home/node/app/ ./app/ -COPY --from=development /home/node/package*.json ./ -RUN npm ci -CMD [ "node", "app" ] -``` - -## Docker Compose guidance - -### Use override files to reduce duplication -Additional settings can be applied to a docker compose file by using override files. - -Override files can be applied by listing the files after the `docker-compose` command with the `-f` parameter, i.e. - -`docker-compose -f docker-compose.yaml -f docker-compose.override.yaml up` +- [defra-docker-node](https://github.com/DEFRA/defra-docker-node) - Node.js (`defradigital/node` and `defradigital/node-development`) +- [defra-docker-dotnetcore](https://github.com/DEFRA/defra-docker-dotnetcore) - .NET (`defradigital/dotnetcore` and `defradigital/dotnetcore-development`) -Note that the above is equivalent to running the command: +Always pin the base image version. Never depend on `latest`, as an unpinned tag makes builds non-reproducible and can pull in unexpected changes. -`docker-compose up` - -as calling `docker-compose` without specifying any files will run `docker-compose` with any available `docker-compose.yaml` and `docker-compose.override.yaml` files in the executing directory. - -Note however that: +## Multi stage builds -`docker-compose up -f docker-compose.yaml` +Dockerfiles should implement multi stage builds so that different stages can be targeted for specific purposes. A production image does not need dev dependencies, test files, or a watch command, whereas a development image does. -will **not** apply the docker `docker-compose.override.yaml` file, only the file specified. +The example below uses the Defra Node.js base image. It has two stages: `development` (used locally, with dev dependencies and hot reload) and `production` (the lean deployable artifact). -One use case is for running tests in CI - common settings can be put into the base `docker-compose.yaml` file, while changes to the command and containers needed in local development can be placed in override files. +```dockerfile +ARG PARENT_VERSION=3.1.1-node24.18.0 +ARG PORT=3000 +ARG PORT_DEBUG=9229 -The below example demonstrates changing the command and container name for testing: +FROM defradigital/node-development:${PARENT_VERSION} AS development -`docker-compose.yaml` +ENV TZ="Europe/London" -``` -version: '3.4' -services: - ffc-demo-service: - build: . - image: ffc-demo-service - container_name: ffc-demo-service - environment: - DEMO_API: http://demo-api +ARG PORT +ARG PORT_DEBUG +ENV PORT=${PORT} +EXPOSE ${PORT} ${PORT_DEBUG} -volumes: - node_modules: {} +COPY --chown=node:node package*.json ./ +RUN npm ci +COPY --chown=node:node . . -``` +CMD [ "npm", "run", "dev" ] -`docker-compose.test.yaml` -``` -version: '3.4' -services: - ffc-demo-service: - command: npm run test - container_name: ffc-demo-service-test -``` +FROM defradigital/node:${PARENT_VERSION} AS production -The tests can be run by providing the `docker-compose.test.yaml` file with a `-f` parameter: +ENV TZ="Europe/London" -`docker-compose up -f docker-compose.yaml -f docker-compose.test.yaml` +# Add curl for the CDP platform health check +USER root +RUN apk add --no-cache curl -It is also recommended not to expose any ports through Docker Compose used in CI as they may conflict with other ports already in use in the build agent. +COPY --from=development --chown=root:root /home/node/package*.json ./ +COPY --from=development --chown=root:root /home/node/app/ ./app/ -Further documentation on docker-compose can be found at https://docs.docker.com/compose/reference/overview/#specifying-multiple-compose-files. +RUN npm ci --omit=dev -### Use projects to provide unique volumes and networks -To avoid conflicts when running different permutations of docker files, projects should be specified to segregate the volumes and networks. +# Remove write permissions from application files +RUN chmod -R a-w /home/node -This can be achieved with the `-p` switch when calling docker compose on the command line. +USER node - i.e. to start the service +ARG PORT +ENV PORT=${PORT} +EXPOSE ${PORT} -`docker-compose -p ffc-demo-service -f docker-compose.yaml up` +CMD [ "node", "app" ] +``` -and to run the tests +Notes on this example: -`docker-compose -p ffc-demo-service-test -f docker-compose.yaml -f docker-compose.test.yaml up` +- **Pin the base image** with `ARG PARENT_VERSION` and use the same version for both stages. `3.1.1-node24.18.0` is the current Node 24 (LTS) Defra base at the time of writing. Check [defra-docker-node](https://github.com/DEFRA/defra-docker-node) for the latest. +- **Use `npm ci`, not `npm install`.** `npm ci` installs exactly what is in `package-lock.json`, giving reproducible builds. Use `npm ci --omit=dev` in production to exclude dev dependencies. +- **No `LABEL uk.gov.defra...` lines are needed.** The base images already carry their provenance labels. +- **Set `ENV TZ`** so container timestamps match the expected timezone. -### Use environment variables to guarantee unique projects and containers -When running through CI, a combination of the `-p` switch and environment variables can be used to ensure each build and test has unique project and container names. This will prevent conflicts with other build pipelines when using tools such as a single node Jenkins. +Front-end services that build client-side assets (for example with Vite) should add an intermediate build stage between `development` and `production` that runs the asset build, and copy the built output into the production stage. -For example using Jenkins, the following compose files can be started via: +## Security best practices -`docker-compose -p ffc-demo-service-$PR_NUMBER-$BUILD_NUMBER -f docker-compose.yaml up` +### Run as a non-root user -and tested with +Containers must run as a non-root user. The Defra base images provide a `node` user (and a `dotnet` user for .NET). Switch to it with `USER node` before the container's `CMD` runs so the process has the least privilege it needs. -`docker-compose -p ffc-demo-service-test-$PR_NUMBER-$BUILD_NUMBER -f docker-compose.yaml -f docker-compose.test.yaml up` +### File ownership and write permissions -using `PR_NUMBER` and `BUILD_NUMBER` environment variables to isolate build tasks. +Security scanners such as SonarQube flag application files that the running user can write to (see [SonarSource rule S6504](https://rules.sonarsource.com/docker/type/Security%20Hotspot/RSPEC-6504/)). A running process should not be able to modify its own application code, as this reduces the impact of a compromised process. -`docker-compose.yaml` +`COPY --chown=node:node` makes the running `node` user the owner, which grants write access. `COPY` also preserves the source file permissions, so changing ownership alone does not reliably remove write access. To guarantee read-only application files in the **production** stage: -``` -version: '3.4' -services: - ffc-demo-service: - build: . - image: ffc-demo-service - container_name: ffc-demo-service-${PR_NUMBER}-${BUILD_NUMBER} -volumes: - node_modules: {} +1. Copy files as `root` with `COPY --chown=root:root`. +2. Explicitly remove write permissions with `RUN chmod -R a-w /home/node`. +3. Switch to the non-root user with `USER node`. -``` +Because the `node` user neither owns the files nor has write permission, the application code is read-only at runtime. This is preferable to `chmod 755`, which still leaves the owner able to write. -`docker-compose.test.yaml` -``` -version: '3.4' -services: - ffc-demo-service: - command: npm run test - container_name: ffc-demo-service-test-${PR_NUMBER}-${BUILD_NUMBER} +```dockerfile +COPY --from=development --chown=root:root /home/node/app/ ./app/ +RUN npm ci --omit=dev +RUN chmod -R a-w /home/node +USER node ``` -### Composing multiple repositories for local development -For scenarios where multiple containers need to be created across multiple repositories, it might be advantageous to create a "development" repo. - -The development repository would: - -- clone all necessary repositories -- builds images from Dockerfiles in each repository by referencing Docker Compose files in those repositories -- run containers based on those images in a single Docker network by referencing Docker Compose files in those repositories -- run single containers for any shared dependencies across repositories such as message queues or databases +> **Apply read-only ownership to the production stage only.** In the `development` stage keep `COPY --chown=node:node` and do **not** run `chmod -R a-w`. Watch mode, tests, and coverage reports all need to write to the container filesystem, so a read-only development image breaks the inner loop. Scanners may flag the development stage for the missing `chmod`; that is acceptable for a local-only image. -To facilitate this, each repository with a potentially shared dependency will need its Docker Compose override files to be setup in such a way that dependency containers can be isolated. This will allow those repository services to run both in isolation and as part of wider service depending on development needs. +> **Some processes legitimately need to write at runtime.** A service might write to a mounted `tmp` directory or a cache such as `node_modules/.cache`. Where this is required, define and secure those specific writable locations (for example a dedicated mounted volume) rather than making the whole application tree writable. Consider whether your service has this need before applying blanket read-only permissions. -For example, let's say we have two repositories, **ServiceA** and **ServiceB**. **ServiceA** communicates with **ServiceB** via an ActiveMQ message queue. **ServiceB** has a PostgreSQL database. +### Scan images for vulnerabilities -**ServiceA**'s Docker Compose files could be structured as follows. +Scan built images for known vulnerabilities as part of the pipeline. The Defra base image repositories use tools such as [Trivy](https://github.com/aquasecurity/trivy) and [Grype](https://github.com/anchore/grype); [Snyk](https://snyk.io/) is also used across Defra. Keep base images current so that upstream security fixes are picked up. -`docker-compose.yaml` - builds image and runs **ServiceA** -`docker-compose.override.yaml` - runs Artemis ActiveMQ container -`docker-compose.link.yaml` - runs **ServiceA** in a named Docker network +## Docker Compose -**ServiceB**'s Docker Compose files could be structured as follows. +Use a single `compose.yaml` per repository. Older services split configuration across many files (`docker-compose.override.yaml`, `docker-compose.test.yaml`, `docker-compose.test.watch.yaml`, and so on), which drift out of sync and are hard to reason about. Compose v2 profiles remove the need for most of these. -`docker-compose.yaml` - builds image and runs **ServiceB** and PostgreSQL container -`docker-compose.override.yaml` - runs Artemis ActiveMQ container -`docker-compose.link.yaml` - runs **ServiceB** in a named Docker network +### One file with profiles -**ServiceA** and **ServiceB** can be run in isolation by running the following commands in each repository. +Put the application service behind a profile so that `docker compose up` starts only the backing dependencies, and the full stack starts on demand: -`docker-compose build` -`docker-compose up` - -The development repository would contain the following. - -`docker-compose.yaml` - runs Artemis ActiveMQ container in named Docker network +```yaml +services: + my-service: + profiles: ["app"] + build: + context: . + target: development + ports: + - "3000:3000" + - "9229:9229" + env_file: + - .env + environment: + REDIS_HOST: redis + depends_on: + redis: + condition: service_healthy + volumes: + - ./app:/home/node/app + networks: + - my-network -A script which would run the following commands: + redis: + image: redis + ports: + - "6379:6379" + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 3s + retries: 5 + networks: + - my-network +networks: + my-network: + driver: bridge + name: my-network ``` -if [ -z "$(docker network ls --filter name=^NETWORK_NAME$ --format={{.Name}})" ]; then - docker network create NETWORK_NAME -fi -docker-compose up -docker-compose -f path/to/ServiceA/docker-compose.yaml -f path/to/ServiceA/docker-compose.link.yaml up --detach -docker-compose -f path/to/ServiceB/docker-compose.yaml -f path/to/ServiceB/docker-compose.link.yaml up --detach -``` - -#### Avoiding docker-compose.yaml in the development repository -If it is preferred to avoid the need for an additional `docker-compose.yaml` file in the development repository itself, an alternative approach would be to explicity declare the shared resources are not started in subsequent `override` files in the start up script. -For example: - -``` -if [ -z "$(docker network ls --filter name=^NETWORK_NAME$ --format={{.Name}})" ]; then - docker network create NETWORK_NAME -fi -docker-compose up -docker-compose -f path/to/ServiceA/docker-compose.yaml -f path/to/ServiceA/docker-compose.override.yaml -f path/to/ServiceA/docker-compose.link.yaml up --detach -docker-compose -f path/to/ServiceB/docker-compose.yaml -f path/to/ServiceB/docker-compose.override.yaml -f path/to/ServiceB/docker-compose.link.yaml up --detach --scale SERVICE_NAME=0 -``` +Start dependencies only, or the whole stack: -### Binding volumes to container -To aide local development, the local workspace can be bound to a Docker volume. This allows code changes to be automatically picked up within the container without the need to rebuild the image or restart the container. - -To best support this, workspaces should be structured so it is simple to determine which files should be bound to Docker volumes as it would not be appropriate to bind everything. For example, it would not be beneficial to bind `node_modules` or a `README`. - -Example of Docker compose file with volume binding. - -``` -volumes: - - ./app/:/home/node/app/ - - ./test/:/home/node/test/ - - ./test-output/:/home/node/test-output/ - - ./package.json:/home/node/package.json +```bash +docker compose up -d # start dependencies only (Redis here) +docker compose --profile app up -d # start dependencies and the application ``` -Changes to any of the directories listed above would automatically be picked up in the running container. +Gating the app behind `profiles: ["app"]` supports the common local workflow of running the app itself on the host (or in your IDE) while its dependencies run in containers, and still lets an orchestration repo bring up the whole stack with `--profile app`. -Binding also allows developers to take advantage of file watching in testing applications. Changes made to code locally will automatically be reflected in the running container supporting a TDD approach. +### Layer environment variables -### .dockerignore -A `.dockerignore` file is a way of preventing local files being copied into an image during build. +Use `env_file` for developer-supplied values and `environment` for the few values that must differ inside Docker (typically service hostnames). The `environment` block takes precedence over `env_file`: -For example, if a repository contains the following files. - -``` -app/index.js -app/config.js -node_modules -index.js -README.md -LICENCE -Dockerfile +```yaml + env_file: + - .env # e.g. REDIS_HOST=localhost for host-native dev + environment: + REDIS_HOST: redis # inside Docker the dependency is on its service name ``` -The `Dockerfile` in this repository includes the following layer which would copy all local files to the container. +This removes the old pattern of duplicating every variable across a base file and an override file. Keep `.env` out of source control and out of images by listing it in both `.gitignore` and `.dockerignore`, and commit a `.env.example` instead. -``` -COPY . . -``` +### Health checks and start ordering -When the image is built then all files in the repository are copied to the image. In this scenario, it is not ideal for performance and disk space reasons to copy the `node_modules`, `LICENCE`, `Dockerfile` or `README.md` to the image. +Give each dependency a `healthcheck` and make the app `depends_on` it with `condition: service_healthy`. Without this, the app can start before the dependency is ready and fail to connect: -To prevent this a `.dockerignore` file should be added with the following content. - -``` -node_modules -Dockerfile -LICENCE -README.md +```yaml + postgres: + image: postgres:16.6 + environment: + POSTGRES_DB: my_database + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres -d my_database"] + interval: 10s + timeout: 10s + retries: 5 ``` -### Container and image names using Docker Compose -If an image name or container name is not specified in a Docker Compose file, then Docker Compose will determine it's own based on the service name. This can result in duplication in the name and unpredictabilty in futher container interaction. +### Set image and container names -#### Set image and container name +If you do not set an image or container name, Compose derives one from the project and service names, which can be unpredictable. Set them explicitly where you need to reference the container later: -``` -version: '3.7' +```yaml services: my-service: image: my-service container_name: my-service ``` -### Preserving database volumes during test runs -In many scenarios it is beneficial to utilise Docker to run local integration tests against a containerised dependency such as a database or message broker. +### Preserving database volumes -These tests would typically write and delete data during test execution. In order to prevent this impacting on local development data and still avoid duplication in Docker Compose definitions, volumes should be declared separate to the database definition. +Integration tests that run against a containerised database write and delete data. To keep test data separate from local development data, declare the persistent volume only where you want persistence rather than in a shared base definition. For most repositories, prefer [Testcontainers](#running-tests) for integration tests, which gives each run a fresh, isolated database with no volume management at all. -For example, if you have the following Docker Compose files +### .dockerignore -- `docker-compose.yaml` - base definition used in all scenarios -- `docker-compose.override.yaml` - applied when running locally only -- `docker-compose.test.yaml` - applied when running tests only +A `.dockerignore` file prevents local files being copied into an image during build. This keeps images small and avoids copying artifacts such as `node_modules`, local `.env` files, and test files. -Then using a Postgres image as an example each definition should contain the following. +For a typical Node.js service: -#### docker-compose.yaml ``` -version: '3.7' -services: - my-postgres-service: - image: postgres:11.4-alpine - environment: - POSTGRES_DB: my_database - POSTGRES_PASSWORD: postgres - POSTGRES_USERNAME: postgres -``` - -#### docker-compose.override.yaml -``` -version: '3.7' -services: - ffc-demo-claim-postgres: - ports: - - "5432:5432" - volumes: - - postgres_data:/var/lib/postgresql/data - -volumes: - postgres_data: {} +node_modules +Dockerfile +.dockerignore +.git +.env +coverage +**/*.test.js +LICENCE +README.md ``` -Then volume and port bindings are only used during local development and any local tests runs will not impact development data. +## Local development workflow -### Windows Git Bash +The recommended inner loop runs the application with hot reload while its dependencies run in containers. Running the app itself on the host (rather than in a container) gives the fastest feedback and the simplest debugging, because there is no rebuild or bind-mount sync on each change and the debugger attaches to a local process. -There is an issue where Git Bash may not correctly interpret volume paths when running Docker Compose on Windows. +For a repeatable, cross-repository approach to this model (host-native app, containerised dependencies, and Testcontainers for integration tests), see the [local development refactoring playbook](https://github.com/johnwatson484/local-dev-refactoring). -To avoid this issue, the following snippet should be added to the `.bashrc` file in the home directory of the user running Git Bash. +### Hot reload with Node's native watch -```bash -# --- Make Docker work nicely in Git Bash --- +Node 24 has built-in watch mode, so a separate tool such as nodemon is no longer needed. Use `node --watch` (restart on change) and `--watch-path` to scope what triggers a restart: -# Prevent MSYS from mangling paths -__docker_env() { - MSYS_NO_PATHCONV=1 MSYS2_ARG_CONV_EXCL='*' "$@" -} - -# Wrapper for docker.exe -docker() { - local needs_winpty= - for a in "$@"; do - [[ "$a" == "-it" || "$a" == "-i" || "$a" == "-t" ]] && needs_winpty=1 - done - - if [[ -n "$needs_winpty" ]] && command -v winpty >/dev/null 2>&1; then - __docker_env winpty docker.exe "$@" - else - __docker_env docker.exe "$@" - fi +```json +"scripts": { + "dev": "node --watch --watch-path=./app --env-file-if-exists=.env app", + "dev:debug": "node --watch --watch-path=./app --inspect --env-file-if-exists=.env app", + "start": "node app", + "test": "node --test", + "test:watch": "node --test --watch" } ``` -## Local development workflow - -This section describes conventions for using Docker Compose to support an efficient local development workflow including hot reload, test-driven development, and debugging. - -### Compose file conventions - -Separate Compose files by concern to allow different combinations for different workflows: - -| File | Purpose | -|------|---------| -| `docker-compose.yaml` | Production image build and core service definitions. No port or volume bindings. | -| `docker-compose.override.yaml` | Local development overrides: port bindings, volume mounts, watch mode. Applied automatically by `docker compose up`. | -| `docker-compose.debug.yaml` | Starts the application waiting for a debugger before executing code. | -| `docker-compose.test.yaml` | Runs tests. No port bindings (avoids conflicts in CI). | -| `docker-compose.test.watch.yaml` | Override for `docker-compose.test.yaml` to run tests in watch mode for TDD. | -| `docker-compose.test.debug.yaml` | Runs tests in watch mode with debugger support. | +`--env-file-if-exists=.env` loads a local `.env` when present and is a no-op when it is absent, so the same script works locally and in deployed environments where platform environment variables are provided directly. Use the `-if-exists` variant so a missing file does not error. -### Hot reload with watch mode - -The development image should run the application in watch mode so that code changes are automatically picked up without rebuilding. For Node.js services, [nodemon](https://www.npmjs.com/package/nodemon) is used to restart the application when files change. - -This requires binding the application source as a volume in `docker-compose.override.yaml`: +If you run the app inside a container instead, mount the source you want watched as a volume in the Compose file so changes are picked up without a rebuild: ```yaml volumes: - ./app:/home/node/app ``` +Bind only what needs watching. It is not beneficial to bind `node_modules` or a `README`. + ### Avoiding port conflicts -When running multiple services locally, each must bind to a unique host port. Map container ports to different host ports per service: +When running multiple services locally, each must bind to a unique host port. Map container ports to different host ports per service, and do the same for dependency containers: ```yaml -# service 1 docker-compose.override.yaml +# service 1 ports: - "3000:3000" - "9229:9229" -# service 2 docker-compose.override.yaml +# service 2 ports: - "3001:3000" - "9230:9229" ``` -The same applies to dependency containers (databases, caches). +Do not expose ports on containers used only in CI, as they may conflict with ports already in use on the build agent. -### package.json scripts +## Running tests -To support the above workflow, configure scripts in `package.json`: +Prefer running unit tests on the host for speed. For integration tests that need real infrastructure (a database, cache, or message broker), use [Testcontainers](https://testcontainers.com/) to start that infrastructure programmatically from within the test process. This replaces the older approach of maintaining separate `docker-compose.test*.yaml` files. -```json -"scripts": { - "test": "jest --runInBand --forceExit", - "test:watch": "jest --coverage=false --onlyChanged --watch --runInBand", - "test:debug": "node --inspect-brk=0.0.0.0 ./node_modules/jest/bin/jest.js --coverage=false --onlyChanged --watch --runInBand --no-cache", - "start:watch": "nodemon --inspect=0.0.0.0 --ext js --legacy-watch app/index.js", - "start:debug": "nodemon --inspect-brk=0.0.0.0 --ext js --legacy-watch app/index.js" -} -``` +Testcontainers gives each test run a fresh, isolated dependency, and the same code path runs locally and in CI (GitHub-hosted runners provide a Docker daemon). Because there are no test-specific Compose files to keep in sync, and no shared volumes to reset, tests are both simpler and more reliable. -- `test` — runs all tests sequentially with coverage. Used in CI. -- `test:watch` — runs only changed tests in watch mode. Primary TDD workflow. -- `test:debug` — watch mode with `--inspect-brk` to wait for debugger attachment. -- `start:watch` — runs the app with nodemon for automatic restart on changes. -- `start:debug` — same as watch but waits for debugger before executing. +```js +import { GenericContainer, Wait } from 'testcontainers' -For Jest watch to detect which files have changed, the `.git` directory must be mounted to the test container: +const redis = await new GenericContainer('redis') + .withExposedPorts(6379) + .withWaitStrategy(Wait.forLogMessage('Ready to accept connections')) + .start() -```yaml -volumes: - - ./.git:/home/node/.git +process.env.REDIS_HOST = redis.getHost() +process.env.REDIS_PORT = String(redis.getMappedPort(6379)) ``` -### Debugging in VS Code +Tests need a running Docker daemon but do not need `docker compose up` first. -Add the following debug configurations to `.vscode/launch.json` (excluded from source control). +## Debugging in VS Code -#### Attach to a running container +The simplest option is to run the app on the host and debug it directly with a normal launch configuration (inspector bound to `127.0.0.1`). Where you need to debug the process running inside a container, use an attach configuration. -```json -{ - "name": "Docker: Attach", - "type": "node", - "request": "attach", - "restart": true, - "port": 9229, - "remoteRoot": "/home/node", - "skipFiles": [ - "/**", - "**/node_modules/**" - ] -} -``` +Add debug configurations to `.vscode/launch.json`. -`restart: true` ensures the debugger reattaches when nodemon restarts the app. +### Attach to a Node process in a running container -#### Attach to Jest tests in debug mode +The container must run the app with the inspector enabled (for example `node --inspect=0.0.0.0` on the debug port exposed in the Compose file). ```json { - "name": "Docker: Jest Attach", + "name": "Docker: Attach", "type": "node", "request": "attach", - "port": 9229, "restart": true, - "timeout": 10000, + "port": 9229, "remoteRoot": "/home/node", - "disableOptimisticBPs": true, - "continueOnAttach": true, "skipFiles": [ "/**", "**/node_modules/**" @@ -473,52 +341,20 @@ Add the following debug configurations to `.vscode/launch.json` (excluded from s } ``` -`disableOptimisticBPs` is required because Jest copies test files before execution, which can cause breakpoint mapping issues. +`restart: true` reattaches the debugger when watch mode restarts the app. -#### Launch container in debug mode via VS Code task +When running several services together, give each a unique host debug port (for example 9229, 9230, 9231) mapped to the container's inspector port, so you can attach to more than one at a time. -```json -{ - "name": "Docker: Attach Launch", - "type": "node", - "request": "attach", - "remoteRoot": "/home/node", - "restart": true, - "port": 9229, - "skipFiles": [ - "/**", - "**/node_modules/**" - ], - "preLaunchTask": "compose-debug-up", - "postDebugTask": "compose-debug-down" -} -``` +## Debugging .NET in a Linux container -With supporting `.vscode/tasks.json`: +.NET services running in Linux containers are debugged with the `vsdbg` remote debugger. `vsdbg` is not part of the .NET SDK, so it must be present in the image. The Defra .NET development base image ([defra-docker-dotnetcore](https://github.com/DEFRA/defra-docker-dotnetcore)) already installs it (at `/vsdbg`), so services built on `defradigital/dotnetcore-development` do not need to add it. This remains the case for .NET 10. If you build on the plain Microsoft SDK image instead, install it yourself in the development stage: -```json -{ - "version": "2.0.0", - "tasks": [ - { - "label": "compose-debug-up", - "type": "shell", - "command": "docker compose -f docker-compose.yaml -f docker-compose.override.yaml -f docker-compose.debug.yaml up -d" - }, - { - "label": "compose-debug-down", - "type": "shell", - "command": "docker compose -f docker-compose.yaml -f docker-compose.override.yaml -f docker-compose.debug.yaml down" - } - ] -} +```dockerfile +ADD https://aka.ms/getvsdbgsh /tmp/getvsdbgsh +RUN /bin/sh /tmp/getvsdbgsh -v latest -l /vsdbg && rm /tmp/getvsdbgsh ``` -### Debugging .NET in a Linux container - -.NET services developed in Linux containers require the `vsdbg` remote debugger, which is included in the Defra .NET development base image. - -#### VS Code +### VS Code ```json { @@ -539,17 +375,52 @@ With supporting `.vscode/tasks.json`: } ``` -#### Visual Studio +### Visual Studio -Visual Studio does not integrate with the WSL filesystem, so WSL users must clone the repository in Windows to debug using Visual Studio. Ensure the following git configuration is set to preserve line endings: +Visual Studio does not integrate with the WSL filesystem, so WSL users must clone the repository in Windows to debug using Visual Studio. Set the following git configuration to preserve line endings: ```bash git config --global core.autocrlf input ``` -1. Start the container with `docker-compose up --build` -2. In Visual Studio, select `Debug -> Attach to process` -3. Select `Docker (Linux Container)` for connection type -4. Enter the container name in connection target -5. Select the process matching the running application -6. Select `Managed (.NET Core for Unix)` code type +1. Start the container with `docker compose up --build`. +2. In Visual Studio, select `Debug -> Attach to process`. +3. Select `Docker (Linux Container)` for connection type. +4. Enter the container name in connection target. +5. Select the process matching the running application. +6. Select `Managed (.NET Core for Unix)` code type. + +## Windows Git Bash + +Git Bash may not correctly interpret volume paths when running Docker Compose on Windows. To avoid this, add the following to the `.bashrc` in the home directory of the user running Git Bash: + +```bash +# --- Make Docker work nicely in Git Bash --- + +# Prevent MSYS from mangling paths +__docker_env() { + MSYS_NO_PATHCONV=1 MSYS2_ARG_CONV_EXCL='*' "$@" +} + +# Wrapper for docker.exe +docker() { + local needs_winpty= + for a in "$@"; do + [[ "$a" == "-it" || "$a" == "-i" || "$a" == "-t" ]] && needs_winpty=1 + done + + if [[ -n "$needs_winpty" ]] && command -v winpty >/dev/null 2>&1; then + __docker_env winpty docker.exe "$@" + else + __docker_env docker.exe "$@" + fi +} +``` + +## References + +- [defra-docker-node](https://github.com/DEFRA/defra-docker-node) - Defra Node.js base images +- [defra-docker-dotnetcore](https://github.com/DEFRA/defra-docker-dotnetcore) - Defra .NET base images +- [local development refactoring playbook](https://github.com/johnwatson484/local-dev-refactoring) - host-native inner loop with Testcontainers +- [Docker Compose documentation](https://docs.docker.com/compose/) +- [Testcontainers](https://testcontainers.com/) From 4cc9a4f439fd593576990355554ddfb52942c589 Mon Sep 17 00:00:00 2001 From: John Watson Date: Thu, 23 Jul 2026 09:47:42 +0100 Subject: [PATCH 4/7] Update docker --- docs/guides/docker_guidance.md | 23 +++++++---------------- mkdocs.yml | 2 +- 2 files changed, 8 insertions(+), 17 deletions(-) diff --git a/docs/guides/docker_guidance.md b/docs/guides/docker_guidance.md index 72ecb112..9459438a 100644 --- a/docs/guides/docker_guidance.md +++ b/docs/guides/docker_guidance.md @@ -1,9 +1,7 @@ -# Docker guidance +# Docker A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably across multiple environments. Docker is a tool to build and run these containers. -Docker remains the default way to package and run Defra services, both in the pipeline and for local development. This guide covers building images, composing local environments, running tests, and debugging. It reflects current tooling: Docker Compose v2 (`docker compose`), Node.js 24, and .NET 10. - ## Local development principles - Teams must not constrain their local development setup to a specific device or operating system to maintain developer mobility and agility. @@ -26,7 +24,11 @@ Docker remains the default way to package and run Defra services, both in the pi ## Base images -Defra publishes hardened base images that provide a non-root user, CA certificates, and the debugging tooling needed for local development. Build on these rather than the raw upstream images: +Defra publishes hardened base images that provide a non-root user, CA certificates, and the debugging tooling needed for local development. + +These images are scanned for vulnerabilities daily using [Trivy](https://github.com/aquasecurity/trivy) and [Grype](https://github.com/anchore/grype). + +Build on these rather than the raw upstream images: - [defra-docker-node](https://github.com/DEFRA/defra-docker-node) - Node.js (`defradigital/node` and `defradigital/node-development`) - [defra-docker-dotnetcore](https://github.com/DEFRA/defra-docker-dotnetcore) - .NET (`defradigital/dotnetcore` and `defradigital/dotnetcore-development`) @@ -63,9 +65,7 @@ FROM defradigital/node:${PARENT_VERSION} AS production ENV TZ="Europe/London" -# Add curl for the CDP platform health check USER root -RUN apk add --no-cache curl COPY --from=development --chown=root:root /home/node/package*.json ./ COPY --from=development --chown=root:root /home/node/app/ ./app/ @@ -88,11 +88,8 @@ Notes on this example: - **Pin the base image** with `ARG PARENT_VERSION` and use the same version for both stages. `3.1.1-node24.18.0` is the current Node 24 (LTS) Defra base at the time of writing. Check [defra-docker-node](https://github.com/DEFRA/defra-docker-node) for the latest. - **Use `npm ci`, not `npm install`.** `npm ci` installs exactly what is in `package-lock.json`, giving reproducible builds. Use `npm ci --omit=dev` in production to exclude dev dependencies. -- **No `LABEL uk.gov.defra...` lines are needed.** The base images already carry their provenance labels. - **Set `ENV TZ`** so container timestamps match the expected timezone. -Front-end services that build client-side assets (for example with Vite) should add an intermediate build stage between `development` and `production` that runs the asset build, and copy the built output into the production stage. - ## Security best practices ### Run as a non-root user @@ -122,13 +119,9 @@ USER node > **Some processes legitimately need to write at runtime.** A service might write to a mounted `tmp` directory or a cache such as `node_modules/.cache`. Where this is required, define and secure those specific writable locations (for example a dedicated mounted volume) rather than making the whole application tree writable. Consider whether your service has this need before applying blanket read-only permissions. -### Scan images for vulnerabilities - -Scan built images for known vulnerabilities as part of the pipeline. The Defra base image repositories use tools such as [Trivy](https://github.com/aquasecurity/trivy) and [Grype](https://github.com/anchore/grype); [Snyk](https://snyk.io/) is also used across Defra. Keep base images current so that upstream security fixes are picked up. - ## Docker Compose -Use a single `compose.yaml` per repository. Older services split configuration across many files (`docker-compose.override.yaml`, `docker-compose.test.yaml`, `docker-compose.test.watch.yaml`, and so on), which drift out of sync and are hard to reason about. Compose v2 profiles remove the need for most of these. +Use a single `compose.yaml` per repository. Older services may split configuration across many files (`docker-compose.override.yaml`, `docker-compose.test.yaml`, `docker-compose.test.watch.yaml`, and so on), which drift out of sync and are hard to reason about. Compose v2 profiles remove the need for most of these. ### One file with profiles @@ -251,8 +244,6 @@ README.md The recommended inner loop runs the application with hot reload while its dependencies run in containers. Running the app itself on the host (rather than in a container) gives the fastest feedback and the simplest debugging, because there is no rebuild or bind-mount sync on each change and the debugger attaches to a local process. -For a repeatable, cross-repository approach to this model (host-native app, containerised dependencies, and Testcontainers for integration tests), see the [local development refactoring playbook](https://github.com/johnwatson484/local-dev-refactoring). - ### Hot reload with Node's native watch Node 24 has built-in watch mode, so a separate tool such as nodemon is no longer needed. Use `node --watch` (restart on change) and `--watch-path` to scope what triggers a restart: diff --git a/mkdocs.yml b/mkdocs.yml index 070996da..68dd0fc6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -62,7 +62,7 @@ nav: - Cookie banner: guides/cookies-banner.md - Defra Identity: guides/defra-id.md - Developer workflows: guides/developer_workflows.md - - Docker guidance: guides/docker_guidance.md + - Docker: guides/docker_guidance.md - Entra: guides/entra.md - GitHub Advanced Security: guides/github_advanced_security.md - Java auto-format with Eclipse: guides/java_auto_format_eclipse.md From 03b9e63fe8f970c21f85ea63c007527f50912525 Mon Sep 17 00:00:00 2001 From: John Watson Date: Fri, 7 Aug 2026 13:13:46 +0100 Subject: [PATCH 5/7] Revert docker guidance changes (covered by #140) --- docs/guides/docker_guidance.md | 503 ++++++++++++++------------------- mkdocs.yml | 2 +- 2 files changed, 220 insertions(+), 285 deletions(-) diff --git a/docs/guides/docker_guidance.md b/docs/guides/docker_guidance.md index 9459438a..c5bddd7e 100644 --- a/docs/guides/docker_guidance.md +++ b/docs/guides/docker_guidance.md @@ -1,389 +1,332 @@ -# Docker +# Docker guidance -A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably across multiple environments. Docker is a tool to build and run these containers. - -## Local development principles - -- Teams must not constrain their local development setup to a specific device or operating system to maintain developer mobility and agility. -- Local development should maximise the use of emulation and avoid tight coupling to cloud services where possible. -- Repositories should include full instructions for anyone to be able to easily run the service locally. -- Keep the local setup lean. Favour a single Compose file per repository over many overlapping variants, and prefer built-in tooling (for example Node's native watch and `--env-file`) over extra dependencies. +A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably across multiple environments. Docker is a tool to build and run these containers. ## More information - [Docker introduction on docker.com](https://www.docker.com/resources/what-container) ## Terminology - -`Dockerfile` - set of instructions for building a Docker image -`Image` - a constructed set of layered Docker instructions -`Container` - a running instance of an image -`Compose file` - a `compose.yaml` file describing how to build and run one or more services - -> **Docker Compose v1 has reached end of life.** Use the Compose v2 plugin, invoked as `docker compose` (with a space), not the standalone `docker-compose` binary. The canonical Compose filename is `compose.yaml`, and the top-level `version:` key is obsolete and should be omitted. - -## Base images - -Defra publishes hardened base images that provide a non-root user, CA certificates, and the debugging tooling needed for local development. - -These images are scanned for vulnerabilities daily using [Trivy](https://github.com/aquasecurity/trivy) and [Grype](https://github.com/anchore/grype). - -Build on these rather than the raw upstream images: - -- [defra-docker-node](https://github.com/DEFRA/defra-docker-node) - Node.js (`defradigital/node` and `defradigital/node-development`) -- [defra-docker-dotnetcore](https://github.com/DEFRA/defra-docker-dotnetcore) - .NET (`defradigital/dotnetcore` and `defradigital/dotnetcore-development`) - -Always pin the base image version. Never depend on `latest`, as an unpinned tag makes builds non-reproducible and can pull in unexpected changes. +`Dockerfile` - set of instructions for building a docker image +`Image` - a constructed set of layered docker instructions +`Container` - a running instance of an image ## Multi stage builds +Dockerfiles should implement multi stage builds to allow different build stages to be targeted for specific purposes. For example, a final production image does not need all the unit test files and a unit test running image would use a different running command than the application. -Dockerfiles should implement multi stage builds so that different stages can be targeted for specific purposes. A production image does not need dev dependencies, test files, or a watch command, whereas a development image does. - -The example below uses the Defra Node.js base image. It has two stages: `development` (used locally, with dev dependencies and hot reload) and `production` (the lean deployable artifact). +Below is an example multi stage build which is intended to use the Defra Node.js base image. -```dockerfile -ARG PARENT_VERSION=3.1.1-node24.18.0 +``` +ARG PARENT_VERSION=1.0.0-node12.16.0 ARG PORT=3000 ARG PORT_DEBUG=9229 +# Development FROM defradigital/node-development:${PARENT_VERSION} AS development - -ENV TZ="Europe/London" - +ARG PARENT_VERSION +ARG REGISTRY +LABEL uk.gov.defra.parent-image=defradigital/node-development:${PARENT_VERSION} ARG PORT +ENV PORT ${PORT} ARG PORT_DEBUG -ENV PORT=${PORT} EXPOSE ${PORT} ${PORT_DEBUG} - COPY --chown=node:node package*.json ./ -RUN npm ci -COPY --chown=node:node . . - -CMD [ "npm", "run", "dev" ] +RUN npm install +COPY --chown=node:node app/ ./app/ +RUN npm run build +CMD [ "npm", "run", "start:watch" ] +# Production FROM defradigital/node:${PARENT_VERSION} AS production +ARG PARENT_VERSION +ARG REGISTRY +LABEL uk.gov.defra.parent-image=defradigital/node:${PARENT_VERSION} +ARG PORT +ENV PORT ${PORT} +EXPOSE ${PORT} +COPY --from=development /home/node/app/ ./app/ +COPY --from=development /home/node/package*.json ./ +RUN npm ci +CMD [ "node", "app" ] +``` -ENV TZ="Europe/London" - -USER root - -COPY --from=development --chown=root:root /home/node/package*.json ./ -COPY --from=development --chown=root:root /home/node/app/ ./app/ +## Docker Compose guidance -RUN npm ci --omit=dev +### Use override files to reduce duplication +Additional settings can be applied to a docker compose file by using override files. -# Remove write permissions from application files -RUN chmod -R a-w /home/node +Override files can be applied by listing the files after the `docker-compose` command with the `-f` parameter, i.e. -USER node +`docker-compose -f docker-compose.yaml -f docker-compose.override.yaml up` -ARG PORT -ENV PORT=${PORT} -EXPOSE ${PORT} +Note that the above is equivalent to running the command: -CMD [ "node", "app" ] -``` +`docker-compose up` -Notes on this example: +as calling `docker-compose` without specifying any files will run `docker-compose` with any available `docker-compose.yaml` and `docker-compose.override.yaml` files in the executing directory. -- **Pin the base image** with `ARG PARENT_VERSION` and use the same version for both stages. `3.1.1-node24.18.0` is the current Node 24 (LTS) Defra base at the time of writing. Check [defra-docker-node](https://github.com/DEFRA/defra-docker-node) for the latest. -- **Use `npm ci`, not `npm install`.** `npm ci` installs exactly what is in `package-lock.json`, giving reproducible builds. Use `npm ci --omit=dev` in production to exclude dev dependencies. -- **Set `ENV TZ`** so container timestamps match the expected timezone. +Note however that: -## Security best practices +`docker-compose up -f docker-compose.yaml` -### Run as a non-root user +will **not** apply the docker `docker-compose.override.yaml` file, only the file specified. -Containers must run as a non-root user. The Defra base images provide a `node` user (and a `dotnet` user for .NET). Switch to it with `USER node` before the container's `CMD` runs so the process has the least privilege it needs. +One use case is for running tests in CI - common settings can be put into the base `docker-compose.yaml` file, while changes to the command and containers needed in local development can be placed in override files. -### File ownership and write permissions +The below example demonstrates changing the command and container name for testing: -Security scanners such as SonarQube flag application files that the running user can write to (see [SonarSource rule S6504](https://rules.sonarsource.com/docker/type/Security%20Hotspot/RSPEC-6504/)). A running process should not be able to modify its own application code, as this reduces the impact of a compromised process. +`docker-compose.yaml` -`COPY --chown=node:node` makes the running `node` user the owner, which grants write access. `COPY` also preserves the source file permissions, so changing ownership alone does not reliably remove write access. To guarantee read-only application files in the **production** stage: +``` +version: '3.4' +services: + ffc-demo-service: + build: . + image: ffc-demo-service + container_name: ffc-demo-service + environment: + DEMO_API: http://demo-api -1. Copy files as `root` with `COPY --chown=root:root`. -2. Explicitly remove write permissions with `RUN chmod -R a-w /home/node`. -3. Switch to the non-root user with `USER node`. +volumes: + node_modules: {} -Because the `node` user neither owns the files nor has write permission, the application code is read-only at runtime. This is preferable to `chmod 755`, which still leaves the owner able to write. +``` -```dockerfile -COPY --from=development --chown=root:root /home/node/app/ ./app/ -RUN npm ci --omit=dev -RUN chmod -R a-w /home/node -USER node +`docker-compose.test.yaml` +``` +version: '3.4' +services: + ffc-demo-service: + command: npm run test + container_name: ffc-demo-service-test ``` -> **Apply read-only ownership to the production stage only.** In the `development` stage keep `COPY --chown=node:node` and do **not** run `chmod -R a-w`. Watch mode, tests, and coverage reports all need to write to the container filesystem, so a read-only development image breaks the inner loop. Scanners may flag the development stage for the missing `chmod`; that is acceptable for a local-only image. +The tests can be run by providing the `docker-compose.test.yaml` file with a `-f` parameter: -> **Some processes legitimately need to write at runtime.** A service might write to a mounted `tmp` directory or a cache such as `node_modules/.cache`. Where this is required, define and secure those specific writable locations (for example a dedicated mounted volume) rather than making the whole application tree writable. Consider whether your service has this need before applying blanket read-only permissions. +`docker-compose up -f docker-compose.yaml -f docker-compose.test.yaml` -## Docker Compose +It is also recommended not to expose any ports through Docker Compose used in CI as they may conflict with other ports already in use in the build agent. -Use a single `compose.yaml` per repository. Older services may split configuration across many files (`docker-compose.override.yaml`, `docker-compose.test.yaml`, `docker-compose.test.watch.yaml`, and so on), which drift out of sync and are hard to reason about. Compose v2 profiles remove the need for most of these. +Further documentation on docker-compose can be found at https://docs.docker.com/compose/reference/overview/#specifying-multiple-compose-files. -### One file with profiles +### Use projects to provide unique volumes and networks +To avoid conflicts when running different permutations of docker files, projects should be specified to segregate the volumes and networks. -Put the application service behind a profile so that `docker compose up` starts only the backing dependencies, and the full stack starts on demand: +This can be achieved with the `-p` switch when calling docker compose on the command line. -```yaml -services: - my-service: - profiles: ["app"] - build: - context: . - target: development - ports: - - "3000:3000" - - "9229:9229" - env_file: - - .env - environment: - REDIS_HOST: redis - depends_on: - redis: - condition: service_healthy - volumes: - - ./app:/home/node/app - networks: - - my-network + i.e. to start the service - redis: - image: redis - ports: - - "6379:6379" - healthcheck: - test: ["CMD", "redis-cli", "ping"] - interval: 10s - timeout: 3s - retries: 5 - networks: - - my-network - -networks: - my-network: - driver: bridge - name: my-network -``` +`docker-compose -p ffc-demo-service -f docker-compose.yaml up` -Start dependencies only, or the whole stack: +and to run the tests -```bash -docker compose up -d # start dependencies only (Redis here) -docker compose --profile app up -d # start dependencies and the application -``` +`docker-compose -p ffc-demo-service-test -f docker-compose.yaml -f docker-compose.test.yaml up` -Gating the app behind `profiles: ["app"]` supports the common local workflow of running the app itself on the host (or in your IDE) while its dependencies run in containers, and still lets an orchestration repo bring up the whole stack with `--profile app`. +### Use environment variables to guarantee unique projects and containers +When running through CI, a combination of the `-p` switch and environment variables can be used to ensure each build and test has unique project and container names. This will prevent conflicts with other build pipelines when using tools such as a single node Jenkins. -### Layer environment variables +For example using Jenkins, the following compose files can be started via: -Use `env_file` for developer-supplied values and `environment` for the few values that must differ inside Docker (typically service hostnames). The `environment` block takes precedence over `env_file`: +`docker-compose -p ffc-demo-service-$PR_NUMBER-$BUILD_NUMBER -f docker-compose.yaml up` -```yaml - env_file: - - .env # e.g. REDIS_HOST=localhost for host-native dev - environment: - REDIS_HOST: redis # inside Docker the dependency is on its service name -``` +and tested with -This removes the old pattern of duplicating every variable across a base file and an override file. Keep `.env` out of source control and out of images by listing it in both `.gitignore` and `.dockerignore`, and commit a `.env.example` instead. +`docker-compose -p ffc-demo-service-test-$PR_NUMBER-$BUILD_NUMBER -f docker-compose.yaml -f docker-compose.test.yaml up` -### Health checks and start ordering +using `PR_NUMBER` and `BUILD_NUMBER` environment variables to isolate build tasks. -Give each dependency a `healthcheck` and make the app `depends_on` it with `condition: service_healthy`. Without this, the app can start before the dependency is ready and fail to connect: +`docker-compose.yaml` -```yaml - postgres: - image: postgres:16.6 - environment: - POSTGRES_DB: my_database - POSTGRES_USER: postgres - POSTGRES_PASSWORD: postgres - healthcheck: - test: ["CMD-SHELL", "pg_isready -U postgres -d my_database"] - interval: 10s - timeout: 10s - retries: 5 ``` +version: '3.4' +services: + ffc-demo-service: + build: . + image: ffc-demo-service + container_name: ffc-demo-service-${PR_NUMBER}-${BUILD_NUMBER} +volumes: + node_modules: {} -### Set image and container names - -If you do not set an image or container name, Compose derives one from the project and service names, which can be unpredictable. Set them explicitly where you need to reference the container later: +``` -```yaml +`docker-compose.test.yaml` +``` +version: '3.4' services: - my-service: - image: my-service - container_name: my-service + ffc-demo-service: + command: npm run test + container_name: ffc-demo-service-test-${PR_NUMBER}-${BUILD_NUMBER} ``` -### Preserving database volumes +### Composing multiple repositories for local development +For scenarios where multiple containers need to be created across multiple repositories, it might be advantageous to create a "development" repo. -Integration tests that run against a containerised database write and delete data. To keep test data separate from local development data, declare the persistent volume only where you want persistence rather than in a shared base definition. For most repositories, prefer [Testcontainers](#running-tests) for integration tests, which gives each run a fresh, isolated database with no volume management at all. +The development repository would: -### .dockerignore +- clone all necessary repositories +- builds images from Dockerfiles in each repository by referencing Docker Compose files in those repositories +- run containers based on those images in a single Docker network by referencing Docker Compose files in those repositories +- run single containers for any shared dependencies across repositories such as message queues or databases -A `.dockerignore` file prevents local files being copied into an image during build. This keeps images small and avoids copying artifacts such as `node_modules`, local `.env` files, and test files. +To facilitate this, each repository with a potentially shared dependency will need its Docker Compose override files to be setup in such a way that dependency containers can be isolated. This will allow those repository services to run both in isolation and as part of wider service depending on development needs. -For a typical Node.js service: +For example, let's say we have two repositories, **ServiceA** and **ServiceB**. **ServiceA** communicates with **ServiceB** via an ActiveMQ message queue. **ServiceB** has a PostgreSQL database. -``` -node_modules -Dockerfile -.dockerignore -.git -.env -coverage -**/*.test.js -LICENCE -README.md -``` +**ServiceA**'s Docker Compose files could be structured as follows. -## Local development workflow +`docker-compose.yaml` - builds image and runs **ServiceA** +`docker-compose.override.yaml` - runs Artemis ActiveMQ container +`docker-compose.link.yaml` - runs **ServiceA** in a named Docker network -The recommended inner loop runs the application with hot reload while its dependencies run in containers. Running the app itself on the host (rather than in a container) gives the fastest feedback and the simplest debugging, because there is no rebuild or bind-mount sync on each change and the debugger attaches to a local process. +**ServiceB**'s Docker Compose files could be structured as follows. -### Hot reload with Node's native watch +`docker-compose.yaml` - builds image and runs **ServiceB** and PostgreSQL container +`docker-compose.override.yaml` - runs Artemis ActiveMQ container +`docker-compose.link.yaml` - runs **ServiceB** in a named Docker network -Node 24 has built-in watch mode, so a separate tool such as nodemon is no longer needed. Use `node --watch` (restart on change) and `--watch-path` to scope what triggers a restart: +**ServiceA** and **ServiceB** can be run in isolation by running the following commands in each repository. -```json -"scripts": { - "dev": "node --watch --watch-path=./app --env-file-if-exists=.env app", - "dev:debug": "node --watch --watch-path=./app --inspect --env-file-if-exists=.env app", - "start": "node app", - "test": "node --test", - "test:watch": "node --test --watch" -} -``` +`docker-compose build` +`docker-compose up` -`--env-file-if-exists=.env` loads a local `.env` when present and is a no-op when it is absent, so the same script works locally and in deployed environments where platform environment variables are provided directly. Use the `-if-exists` variant so a missing file does not error. +The development repository would contain the following. -If you run the app inside a container instead, mount the source you want watched as a volume in the Compose file so changes are picked up without a rebuild: +`docker-compose.yaml` - runs Artemis ActiveMQ container in named Docker network + +A script which would run the following commands: -```yaml -volumes: - - ./app:/home/node/app +``` +if [ -z "$(docker network ls --filter name=^NETWORK_NAME$ --format={{.Name}})" ]; then + docker network create NETWORK_NAME +fi +docker-compose up +docker-compose -f path/to/ServiceA/docker-compose.yaml -f path/to/ServiceA/docker-compose.link.yaml up --detach +docker-compose -f path/to/ServiceB/docker-compose.yaml -f path/to/ServiceB/docker-compose.link.yaml up --detach ``` -Bind only what needs watching. It is not beneficial to bind `node_modules` or a `README`. +#### Avoiding docker-compose.yaml in the development repository +If it is preferred to avoid the need for an additional `docker-compose.yaml` file in the development repository itself, an alternative approach would be to explicity declare the shared resources are not started in subsequent `override` files in the start up script. -### Avoiding port conflicts +For example: -When running multiple services locally, each must bind to a unique host port. Map container ports to different host ports per service, and do the same for dependency containers: +``` +if [ -z "$(docker network ls --filter name=^NETWORK_NAME$ --format={{.Name}})" ]; then + docker network create NETWORK_NAME +fi +docker-compose up +docker-compose -f path/to/ServiceA/docker-compose.yaml -f path/to/ServiceA/docker-compose.override.yaml -f path/to/ServiceA/docker-compose.link.yaml up --detach +docker-compose -f path/to/ServiceB/docker-compose.yaml -f path/to/ServiceB/docker-compose.override.yaml -f path/to/ServiceB/docker-compose.link.yaml up --detach --scale SERVICE_NAME=0 +``` -```yaml -# service 1 -ports: - - "3000:3000" - - "9229:9229" +### Binding volumes to container +To aide local development, the local workspace can be bound to a Docker volume. This allows code changes to be automatically picked up within the container without the need to rebuild the image or restart the container. -# service 2 -ports: - - "3001:3000" - - "9230:9229" -``` +To best support this, workspaces should be structured so it is simple to determine which files should be bound to Docker volumes as it would not be appropriate to bind everything. For example, it would not be beneficial to bind `node_modules` or a `README`. -Do not expose ports on containers used only in CI, as they may conflict with ports already in use on the build agent. +Example of Docker compose file with volume binding. -## Running tests +``` +volumes: + - ./app/:/home/node/app/ + - ./test/:/home/node/test/ + - ./test-output/:/home/node/test-output/ + - ./package.json:/home/node/package.json +``` -Prefer running unit tests on the host for speed. For integration tests that need real infrastructure (a database, cache, or message broker), use [Testcontainers](https://testcontainers.com/) to start that infrastructure programmatically from within the test process. This replaces the older approach of maintaining separate `docker-compose.test*.yaml` files. +Changes to any of the directories listed above would automatically be picked up in the running container. -Testcontainers gives each test run a fresh, isolated dependency, and the same code path runs locally and in CI (GitHub-hosted runners provide a Docker daemon). Because there are no test-specific Compose files to keep in sync, and no shared volumes to reset, tests are both simpler and more reliable. +Binding also allows developers to take advantage of file watching in testing applications. Changes made to code locally will automatically be reflected in the running container supporting a TDD approach. -```js -import { GenericContainer, Wait } from 'testcontainers' +### .dockerignore +A `.dockerignore` file is a way of preventing local files being copied into an image during build. -const redis = await new GenericContainer('redis') - .withExposedPorts(6379) - .withWaitStrategy(Wait.forLogMessage('Ready to accept connections')) - .start() +For example, if a repository contains the following files. -process.env.REDIS_HOST = redis.getHost() -process.env.REDIS_PORT = String(redis.getMappedPort(6379)) +``` +app/index.js +app/config.js +node_modules +index.js +README.md +LICENCE +Dockerfile ``` -Tests need a running Docker daemon but do not need `docker compose up` first. +The `Dockerfile` in this repository includes the following layer which would copy all local files to the container. -## Debugging in VS Code +``` +COPY . . +``` -The simplest option is to run the app on the host and debug it directly with a normal launch configuration (inspector bound to `127.0.0.1`). Where you need to debug the process running inside a container, use an attach configuration. +When the image is built then all files in the repository are copied to the image. In this scenario, it is not ideal for performance and disk space reasons to copy the `node_modules`, `LICENCE`, `Dockerfile` or `README.md` to the image. -Add debug configurations to `.vscode/launch.json`. +To prevent this a `.dockerignore` file should be added with the following content. -### Attach to a Node process in a running container +``` +node_modules +Dockerfile +LICENCE +README.md +``` -The container must run the app with the inspector enabled (for example `node --inspect=0.0.0.0` on the debug port exposed in the Compose file). +### Container and image names using Docker Compose +If an image name or container name is not specified in a Docker Compose file, then Docker Compose will determine it's own based on the service name. This can result in duplication in the name and unpredictabilty in futher container interaction. -```json -{ - "name": "Docker: Attach", - "type": "node", - "request": "attach", - "restart": true, - "port": 9229, - "remoteRoot": "/home/node", - "skipFiles": [ - "/**", - "**/node_modules/**" - ] -} +#### Set image and container name + +``` +version: '3.7' +services: + my-service: + image: my-service + container_name: my-service ``` -`restart: true` reattaches the debugger when watch mode restarts the app. +### Preserving database volumes during test runs +In many scenarios it is beneficial to utilise Docker to run local integration tests against a containerised dependency such as a database or message broker. -When running several services together, give each a unique host debug port (for example 9229, 9230, 9231) mapped to the container's inspector port, so you can attach to more than one at a time. +These tests would typically write and delete data during test execution. In order to prevent this impacting on local development data and still avoid duplication in Docker Compose definitions, volumes should be declared separate to the database definition. -## Debugging .NET in a Linux container +For example, if you have the following Docker Compose files -.NET services running in Linux containers are debugged with the `vsdbg` remote debugger. `vsdbg` is not part of the .NET SDK, so it must be present in the image. The Defra .NET development base image ([defra-docker-dotnetcore](https://github.com/DEFRA/defra-docker-dotnetcore)) already installs it (at `/vsdbg`), so services built on `defradigital/dotnetcore-development` do not need to add it. This remains the case for .NET 10. If you build on the plain Microsoft SDK image instead, install it yourself in the development stage: +- `docker-compose.yaml` - base definition used in all scenarios +- `docker-compose.override.yaml` - applied when running locally only +- `docker-compose.test.yaml` - applied when running tests only -```dockerfile -ADD https://aka.ms/getvsdbgsh /tmp/getvsdbgsh -RUN /bin/sh /tmp/getvsdbgsh -v latest -l /vsdbg && rm /tmp/getvsdbgsh -``` +Then using a Postgres image as an example each definition should contain the following. -### VS Code - -```json -{ - "name": ".NET Core Docker Attach", - "type": "coreclr", - "request": "attach", - "processId": "${command:pickRemoteProcess}", - "pipeTransport": { - "pipeProgram": "docker", - "pipeArgs": ["exec", "-i", "my-service-container"], - "debuggerPath": "/vsdbg/vsdbg", - "pipeCwd": "${workspaceRoot}", - "quoteArgs": false - }, - "sourceFileMap": { - "/home/dotnet": "${workspaceFolder}" - } -} +#### docker-compose.yaml +``` +version: '3.7' +services: + my-postgres-service: + image: postgres:11.4-alpine + environment: + POSTGRES_DB: my_database + POSTGRES_PASSWORD: postgres + POSTGRES_USERNAME: postgres ``` -### Visual Studio - -Visual Studio does not integrate with the WSL filesystem, so WSL users must clone the repository in Windows to debug using Visual Studio. Set the following git configuration to preserve line endings: +#### docker-compose.override.yaml +``` +version: '3.7' +services: + ffc-demo-claim-postgres: + ports: + - "5432:5432" + volumes: + - postgres_data:/var/lib/postgresql/data -```bash -git config --global core.autocrlf input +volumes: + postgres_data: {} ``` -1. Start the container with `docker compose up --build`. -2. In Visual Studio, select `Debug -> Attach to process`. -3. Select `Docker (Linux Container)` for connection type. -4. Enter the container name in connection target. -5. Select the process matching the running application. -6. Select `Managed (.NET Core for Unix)` code type. +Then volume and port bindings are only used during local development and any local tests runs will not impact development data. -## Windows Git Bash +### Windows Git Bash -Git Bash may not correctly interpret volume paths when running Docker Compose on Windows. To avoid this, add the following to the `.bashrc` in the home directory of the user running Git Bash: +There is an issue where Git Bash may not correctly interpret volume paths when running Docker Compose on Windows. + +To avoid this issue, the following snippet should be added to the `.bashrc` file in the home directory of the user running Git Bash. ```bash # --- Make Docker work nicely in Git Bash --- @@ -407,11 +350,3 @@ docker() { fi } ``` - -## References - -- [defra-docker-node](https://github.com/DEFRA/defra-docker-node) - Defra Node.js base images -- [defra-docker-dotnetcore](https://github.com/DEFRA/defra-docker-dotnetcore) - Defra .NET base images -- [local development refactoring playbook](https://github.com/johnwatson484/local-dev-refactoring) - host-native inner loop with Testcontainers -- [Docker Compose documentation](https://docs.docker.com/compose/) -- [Testcontainers](https://testcontainers.com/) diff --git a/mkdocs.yml b/mkdocs.yml index 68dd0fc6..070996da 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -62,7 +62,7 @@ nav: - Cookie banner: guides/cookies-banner.md - Defra Identity: guides/defra-id.md - Developer workflows: guides/developer_workflows.md - - Docker: guides/docker_guidance.md + - Docker guidance: guides/docker_guidance.md - Entra: guides/entra.md - GitHub Advanced Security: guides/github_advanced_security.md - Java auto-format with Eclipse: guides/java_auto_format_eclipse.md From c0f7e5a7ed3b0a3563d6fdf50986fbe5eedd2422 Mon Sep 17 00:00:00 2001 From: John Watson Date: Fri, 7 Aug 2026 13:25:08 +0100 Subject: [PATCH 6/7] Update Sonar --- docs/guides/sonarqube_ide.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/guides/sonarqube_ide.md b/docs/guides/sonarqube_ide.md index 1da62e89..26c59a60 100644 --- a/docs/guides/sonarqube_ide.md +++ b/docs/guides/sonarqube_ide.md @@ -2,7 +2,14 @@ [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/) is an IDE extension that identifies code quality issues as you code, providing immediate feedback before committing. -All Defra projects are required to set up their repositories within the [SonarQube Cloud Defra organisation](https://sonarcloud.io/organizations/defra/projects). Using the IDE extension in connected mode ensures you get the same rules and quality gates locally as in CI. +Running analysis locally, rather than waiting for a CI pipeline or a reviewer to flag issues, means you can: + +- fix bugs, code smells and security vulnerabilities as you introduce them, when the context is still fresh in your mind +- avoid pull requests being blocked or sent back for issues that a quick local check would have caught +- spend review time on design and logic, rather than on issues a linter could have found +- learn the reasoning behind each rule from inline explanations, rather than just being told something is wrong + +All Defra projects are required to set up their repositories within the [SonarQube Cloud Defra organisation](https://sonarcloud.io/organizations/defra/projects). Using the IDE extension in connected mode ensures you get the same rules and quality gates locally as in CI, so there are no surprises when your code reaches the pipeline. ## Dependencies @@ -20,11 +27,11 @@ sudo apt-get install openjdk-21-jre 2. Set the JRE location in VS Code settings: - ```json - { - "sonarlint.ls.javaHome": "/usr/lib/jvm/java-21-openjdk-amd64" - } - ``` +```json +{ + "sonarlint.ls.javaHome": "/usr/lib/jvm/java-21-openjdk-amd64" +} +``` This gives you Sonar code analysis using default quality gates for supported languages. From 6b69760ba1f724e4f1ed27732bc6167a7d87e54f Mon Sep 17 00:00:00 2001 From: John Watson Date: Tue, 25 Aug 2026 15:02:31 +0100 Subject: [PATCH 7/7] Address review comments --- docs/processes/pull_requests.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/processes/pull_requests.md b/docs/processes/pull_requests.md index f53e1579..919ab2a7 100644 --- a/docs/processes/pull_requests.md +++ b/docs/processes/pull_requests.md @@ -80,12 +80,14 @@ It's on you to keep your branch up to date with with your main development branc git rebase origin/main ``` -## Get it looked at +## Get it reviewed When you're finished and have pushed your last commit request someone to review it. If there are multiple members on your team and all could review, feel free to request them all. The key thing is at least one other person should review the PR before it is merged. You and the reviewer will then work to confirm the changes are OK. Once the reviewer is happy they need to **approve it**. +When code reviews are conducted to a high standard, they provide a valuable learning opportunity for the author, reviewer and any observers of the review process. + ## Completing the PR Once approved to complete the PR you'll need to `squash` your commits down to one. @@ -98,11 +100,7 @@ When you click it GitHub will present a box which contains the combined text fro When done ensure you delete the branch. Again GitHub will present this option in the UI immediately after merging so make use of it then. -## Reviewing a pull request - -When code reviews are conducted to a high standard, they provide a valuable learning opportunity for the author, reviewer and any observers of the review process. - -At least one reviewer must approve a pull request before it can be merged. +## Tips for reviewing PRs ### Tone of code review comments