ansible: Improve Nomad reliability

This commit improves the reliability of Nomad on machines that have
just restarted.  It incorporates a suggestion from @ericonr to change
the unbound config to permit consul to come up independently of
unbound, and a change to the Nomad service to ensure that all of
Nomad's dependencies are available when starting.

One major issue remains, that is tracked as hashicorp/nomad#11272
wherein its possible for system tasks that have static port
allocations to fail to reschedule if there's dangling state on a node
after a reboot.  This is likely going to require a patch in Nomad
itself to resolve.

Author: the-maldridge

ansible/roles/nomad/files/run

@@ -2,5 +2,16 @@
 
 modprobe bridge
 
+# We need consul and unbound first
+sv check consul >/dev/null || exit 1
+sv check unbound >/dev/null || exit 1
+
+# We're racing DNS setup here because this *has* to be up before Nomad
+# starts in order for it to get fingerprinted.
+while ! getent hosts active.vault.service.consul ; do
+    sleep 5
+    unbound-control flush_negative
+done
+
 exec 2>&1
 exec nomad agent -config /etc/nomad/

ansible/roles/unbound/templates/unbound.conf.j2

@@ -16,6 +16,7 @@ server:
 
   do-not-query-localhost: no
   extended-statistics: yes
+  infra-keep-probing: yes
 
 {% if unbound_stub_consul|default(false) %}
   private-domain: consul.