docs/src: Add 2021-07-17 postmortem

Author: the-maldridge

docs/src/SUMMARY.md

@@ -33,3 +33,4 @@
   - [Image Building](./upkeep/images.md)
 - [Postmortems](./postmortem/index.md)
   - [2021-06-06: a-hel-fi Failure](./postmortem/2021-06-06.md)
+  - [2021-07-17: package signing issues](./postmortem/2021-07-17.md)

docs/src/postmortem/2021-07-17.md

@@ -0,0 +1,114 @@
+# Post Mortem 2021-06-06
+
+## Incident summary
+
+For approximately 2 days the Nomad managed tasks on a-fsn-de were
+unavailable, including repository management tasks.  Resolution was
+and continues to be hampered by a distinct and ongoing outage.
+
+### Ammended 2021-07-31
+
+CAS service was restored on 2021-07-22, any point below that mentions
+and ongoing outage was written while the incident was ongoing.
+
+## Leadup
+
+At this time we do not know what caused the reboot of a-fsn-de,
+however we do know that a parallel and ongoing incident was occurring
+at the same time.  Void uses a centralized authentication service (CAS) to
+manage access to our machines, and like many secure services this one
+relies on TLS certificates.  This certificate expired without being
+noticed, which prevented what would have otherwise been a quick
+recovery of logging in and bouncing some services.
+
+Additionally, when the CAS is unavailable, we maintain a break-glass
+login capability for a handful of extremely trusted maintainers (1:1
+with the people that have access to the package signing key).  This
+access was discovered to be impaired by one developer's key missing
+entirely, and one developer having failed to rotate their key.  The
+third developer was on vacation but was able to log in and rectify the
+keying problem.
+
+## Fault
+
+The Nomad outage was caused by an unexpected restart of a-fsn-de.
+When Nomad hosts reboot there is a known defect in that runit may
+bring up the services in a race that results in Nomad not being usable
+until services are restarted in a specific order.
+
+The unavailability of the CAS system means that we still cannot log in
+to all hosts normally.
+
+The issues with the break-glass keys caused the recovery of both the
+specific Nomad host and the CAS server to be slow.
+
+## Impact
+
+Publicly Visible:
+
+  * Repository signing unavailable
+  * Builds not completing normally
+  * musl and aarch64 appeared to lag behind glibc
+
+Internally Visible:
+
+  * CAS logins not available
+  * Detailed failure logs not available in Grafana (requires CAS login)
+  * Couldn't make control requests to Nomad (requires Vault CAS integration)
+
+## Detection
+
+The Nomad failure was detected by external observation that an update
+to the `less` package was not signed, and was preventing installs
+from proceeding.
+
+The CAS failure was discovered a few days before the signing issue but
+was deemed non-critical as it was an inconvenience that could be fixed
+within a week.
+
+## Response
+
+The ability to merge changes in GitHub was restricted to prevent new
+builds from running that might further complicate recovery efforts.
+
+@the-maldridge and @Gottox were recalled from vacation to recover
+access to the system.
+
+## Recovery
+
+@Gottox used break-glass access to both restore break-glass access for
+@the-maldridge and @duncaen, and to restart the stuck signing process.
+
+
+## What went well
+
+  * Excellent internal communication kept everyone in the loop as to
+    what was broken, what was being done to fix it, and who was
+    responsible for taking action.
+
+  * Break-glass access, once used, did work effectively.
+
+## What could be done better
+
+  * External communication was not great.  A post went up on
+    voidlinux.org, but no twitter notification was made, and the post
+    was not widely shared on our other channels.
+
+  * Break glass connectivity existed, but did not work.
+
+  * Initially recalling critical team members from vacation was an
+    ad-hoc process.
+
+## Lessons learned
+
+  * Having the capability isn't enough.  Break glass needs to be
+    regularly tested to be effective.
+
+  * For foundational infrastructure that has very infrequent updates,
+    such as long lived TLS certificates, we should ensure multiple
+    people are aware of the expiry date, and make use of multiple
+    calendars to ensure critical life-cycle events are not missed.
+
+## Timeline
+
+No timeline is provided for this incident.