Skip to content

Fix sync ironic KeyError for fake-hardware nodes - #2903

Merged
berendt merged 2 commits into
mainfrom
sync-ironic-fake-hardware
Nov 19, 2026
Merged

berendt merged 2 commits into
mainfrom
sync-ironic-fake-hardware

Conversation

@ideaship

@ideaship ideaship commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

osism sync ironic fails on every new node whose ironic_parameters.driver is fake-hardware, the mode metalbox's scripts/devel/fake-ironic-mode.sh switches to:

Creating baremetal node for device-007
Could not fully synchronize device device-007 with ironic: 'fake-hardware'

The node is created in enroll and left there, without instance_info/extra, so nothing can be deployed on it.

Cause

After computing the driver_info diff, the sync drops the driver's password key from it, because Ironic never returns the password. That key is looked up with driver_params[driver]["password"], and driver_params only lists ipmi and redfish, so any other driver raises KeyError. This became reachable on the first sync once new nodes started being created as stubs (only driver and automated_clean) followed by the regular update procedure: the stub has no driver_info, so there is always a diff. Before that, a fake-hardware node only hit the lookup if its driver_info changed between syncs.

Changes

  1. Mask secrets in sync ironic update output. The live "Updating baremetal node … with {node_updates}" message printed the updates verbatim, with only the driver password removed. It now goes through mask_secrets with the device's secret template values, exactly as the dry-run output already does. The payload sent to Ironic is unchanged. This comes first because the fix below lets the sync print updates for drivers it used to crash on: a driver without a driver_params entry that carries a password (e.g. ilo_password) keeps it in the diff.
  2. Fix sync ironic KeyError for fake-hardware nodes. Both password lookups (live sync and dry-run update branch) use .get(), so a driver without an entry has no password key to drop.

Known limitations

The live output now has the same masking as the dry-run output, no stricter. These limitations of mask_secrets and the display code apply to both and are not changed here:

  • Secrets are recognised by key name (*password*, *secret*, ironic_osism_*) and by the values of secret template variables. Credentials under other names (e.g. ipmi_hex_kg_key, snmp_community) are printed.
  • Values in extra are JSON strings, so secret keys inside them (e.g. in the vault-decrypted frr_parameters) are not seen by key-based masking.
  • Secret values are replaced one at a time; a secret containing another may be only partly masked, and an empty secret value garbles the output.

Testing

  • New unit tests: masking of secret keys and of secret template values in the update output; sync_ironic passing template variables through; a fake-hardware node on the create path (with the real deep_compare) reaching baremetal_node_update; the dry-run update branch for fake-hardware; an unmapped driver's password masked in the output.
  • Full unit suite passes locally.
  • Not yet run against a live fake-mode Ironic.

🤖 Generated with Claude Code

Pass the node updates through mask_secrets before printing them in the
"Updating baremetal node" task output of _sync_ironic_device, as the
dry-run path already does. sync_ironic now hands each device's template
variables to _sync_ironic_device, so secret values rendered into other
strings are masked as well. The attributes sent to Ironic are unchanged;
mask_secrets works on a deep copy.

Until now only the dry-run output was masked. The live sync printed the
node updates verbatim, with only the driver password removed (it is
dropped from the diff because Ironic never returns it). Any value under
a key containing "password" or "secret" or starting with
"ironic_osism_" in instance_info, extra or properties went to the task
output in clear text, as did a secret rendered into a non-secret value
such as kernel parameters.

The collection of secret values from template variables moves out of
_sync_ironic_device_dry_run into _collect_secret_values, shared by both
paths. template_vars is an optional argument of _sync_ironic_device;
without it, masking falls back to the key-based rules.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
Look up the driver's password key in driver_params with .get() at both
places where the ironic sync compares driver_info, so a driver without
an entry is treated as having no password key. This affects the real
sync (_sync_ironic_device) and the dry-run update branch
(_sync_ironic_device_dry_run).

driver_params only lists ipmi and redfish. After computing the
driver_info diff, the sync drops the password key from it because
Ironic never returns the password. That key was looked up with
driver_params[driver]["password"], which raises KeyError for any other
driver, e.g. fake-hardware. The other uses of driver_params in this
module already check membership first.

The sync now creates a new node as a stub with only driver and
automated_clean, then always runs the update procedure. The stub has no
driver_info, so every new fake-hardware node produces a driver_info
diff and fails on its first sync:

  Could not fully synchronize device node1 with ironic: 'fake-hardware'

The node is left in enroll without instance_info or extra and is never
moved to manageable/available, so nothing can be deployed on it. A
config-only workaround does not exist: Ironic's redfish hardware type
has no fake power interface, so a node without a real BMC needs the
fake-hardware driver. This is the mode metalbox's fake Ironic setup
uses.

For a driver without a driver_params entry that does carry a password
in driver_info (e.g. ilo_password), the password now stays in the
driver_info diff. It is sent to Ironic on every sync, which is harmless,
and it appears in the "Updating baremetal node" output, where the
masking of that output hides it.

Add regression tests for the create path, using the real deep_compare
so the stub node's missing driver_info produces the diff, and for the
dry-run update branch. A further test checks that an unmapped driver's
password is masked in the update output.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship ideaship self-assigned this Nov 19, 2026
@ideaship
ideaship marked this pull request as ready for review November 19, 2026 14:03
@ideaship
ideaship requested a review from berendt November 19, 2026 14:03

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 4 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="osism/tasks/conductor/ironic.py" line_range="408" />
<code_context>
             request_id,
-            f"Updating baremetal node for {device.name} with {node_updates}\n",
+            f"Updating baremetal node for {device.name} with "
+            f"{mask_secrets(node_updates, secret_values=secret_values)}\n",
         )
         # NOTE: Pop target_raid_config from node_attributes since they cannot be set using node updates
</code_context>
<issue_to_address>
**Secrets in extra fields are logged**

When a serialized `extra` field contains a secret under a secret-named key and its value is not a collected template secret, `mask_secrets` treats the serialized value as an ordinary string and does not inspect its keys, so the update message prints the secret in clear text.

Inspect serialized `extra` values and redact values under secret-named keys before logging the update.

Also at `osism/tasks/conductor/ironic.py:409`.
</issue_to_address>

### Comment 2
<location path="osism/tasks/conductor/ironic.py" line_range="408" />
<code_context>
             request_id,
-            f"Updating baremetal node for {device.name} with {node_updates}\n",
+            f"Updating baremetal node for {device.name} with "
+            f"{mask_secrets(node_updates, secret_values=secret_values)}\n",
         )
         # NOTE: Pop target_raid_config from node_attributes since they cannot be set using node updates
</code_context>
<issue_to_address>
**Unmapped credentials appear in task output**

When an unmapped driver has a credential in `driver_info` under a key not recognized by the masking rules, such as `ilo_token`, `mask_secrets` does not recognize the credential key and `_collect_secret_values` does not contain its value, so the update message prints the credential in clear text.

Mask credentials using driver-specific secret metadata or extend the masking policy to cover credential fields beyond the current key patterns.
</issue_to_address>

### Comment 3
<location path="osism/tasks/conductor/ironic.py" line_range="409" />
<code_context>
-            f"Updating baremetal node for {device.name} with {node_updates}\n",
+            f"Updating baremetal node for {device.name} with "
+            f"{mask_secrets(node_updates, secret_values=secret_values)}\n",
         )
         # NOTE: Pop target_raid_config from node_attributes since they cannot be set using node updates
         target_raid_config = node_attributes.pop("target_raid_config", None)
</code_context>
<issue_to_address>
**Overlapping secrets remain visible**

When two collected secret values overlap and the shorter value is replaced first, `mask_secrets` replaces values one at a time, so it masks only the prefix of the longer secret and leaves its suffix visible in the update output.

Replace longer secrets before shorter ones, or otherwise ensure overlapping values are fully masked.

Also at `osism/tasks/conductor/ironic.py:408`.
</issue_to_address>

### Comment 4
<location path="osism/tasks/conductor/ironic.py" line_range="364" />
<code_context>
+            or "secret" in key.lower()
+            or key.lower().startswith("ironic_osism_")
+        ):
+            secret_values.add(value)
+    return secret_values
+
</code_context>
<issue_to_address>
**Empty secrets garble update output**

When a password, secret, or `ironic_osism_*` template variable has an empty value, `_collect_secret_values` adds the empty string, so `mask_secrets` applies `str.replace("", "xxx")` and inserts `xxx` between every character in the logged update.

Exclude empty values when collecting secrets for masking.

Also at `osism/tasks/conductor/ironic.py:365`.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 4 findings to address first, and the sync can now proceed for drivers without a password entry in driver_params, so incorrect node attributes may be written to Ironic instead of the sync stopping with a KeyError. Those stored settings can be corrected by a later update, but reverting this code alone would not undo them.

Blocking findings: osism/tasks/conductor/ironic.py:408, osism/tasks/conductor/ironic.py:408, osism/tasks/conductor/ironic.py:409, osism/tasks/conductor/ironic.py:364


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread osism/tasks/conductor/ironic.py Resolved
Comment thread osism/tasks/conductor/ironic.py Resolved
Comment thread osism/tasks/conductor/ironic.py Resolved
Comment thread osism/tasks/conductor/ironic.py Resolved
@ideaship
ideaship marked this pull request as draft November 19, 2026 14:17
@ideaship
ideaship force-pushed the sync-ironic-fake-hardware branch 2 times, most recently from fd18b5a to 18500ba Compare November 19, 2026 15:08
@ideaship

Copy link
Copy Markdown
Contributor Author

Secrets in extra fields / Overlapping secrets / Empty secrets: Valid, but these are existing limitations of the shared mask_secrets and the dry-run output, which this PR doesn't change. The live output now gets the same masking as the dry-run, no stricter. Listed under "Known limitations" in the description.

Unmapped credentials: Same: masking works by key name, and widening the patterns is a separate change.

@ideaship
ideaship marked this pull request as ready for review November 19, 2026 15:11
@berendt
berendt merged commit db0bfde into main Nov 19, 2026
4 checks passed
@berendt
berendt deleted the sync-ironic-fake-hardware branch November 19, 2026 18:14
@github-project-automation github-project-automation Bot moved this from New to Done in Human Board Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants