Skip to content

Full ESPHome API 1.18: HA state subscriptions, linked wizard inputs, dial-back, async IR/RF - #45

Merged
bbangert merged 15 commits into
mainfrom
feat/api-1-18-full
Oct 21, 2026
Merged

bbangert merged 15 commits into
mainfrom
feat/api-1-18-full

Conversation

@bbangert

Copy link
Copy Markdown
Owner
  • Noise server hello carries the MAC as 12 lowercase hex digits (DeviceConfig.hello_mac/1), as firmware does; DeviceInfoResponse.mac_address keeps the configured form
  • IR/RF transmits run off the connection in Espex.InfraredQueue (own :one_for_all subtree with its Task.Supervisor): one pending reply per key with supersede, one call at a time per adapter module, a call 30 s past its air time (capped at 8 min) answered false and killed. Clients below 1.18 get no reply and queue at most one frame per key; unknown keys are refused; the 30 s air-time refusal is gone
  • New optional Espex.HomeAssistantStates adapter (:home_assistant_states) with Espex.HomeAssistantStates.Subscription: SubscribeHomeAssistantStatesRequest walks the table, HomeAssistantStateResponse calls handle_state/3, new entries go to subscribed clients. Table owned by Espex.Server, synced copy per connection
  • Espex.subscribe_home_assistant_state/2 and Espex.unsubscribe_home_assistant_state/2
  • Espex.Wizard.Input gains links: a linked input's value fills subscription entries, resent to every subscribed client when it changes
  • Dial-back: new :outgoing_connection supervisor option and Espex.Dialer (own subtree with its connection DynamicSupervisor and Task.Supervisor). Upstream timing (delay, 5–300 s backoff with jitter, 5 s precondition retry), dialed socket run through Espex.Connection via ThousandIsland's ready message, the dialed connection sends its hello first, target learned from outgoing_connection_target hellos over Noise
  • New optional Espex.OutgoingTargetStore adapter (:outgoing_target_store) for the learned target; DeviceInfoResponse.api_outgoing_connection_supported set while dial-back is configured
  • DeviceConfig inspect hides the PSK; Connection.format_status/1 hides the Noise session keys
  • Guides for API 1.15, 1.16, 1.17 and 1.18 in a new "API versions" docs group; README, Espex and Espex.Supervisor docs updated
  • thousand_island floor raised to ~> 1.5

🤖 Generated with Claude Code

bbangert and others added 9 commits October 20, 2026 16:16
ESPHome firmware puts the MAC in the server hello as 12 lowercase hex
digits, and aioesphomeapi's dial-back listener refuses any other form.
DeviceConfig.hello_mac/1 derives that form from the configured MAC;
DeviceInfoResponse.mac_address keeps the configured form, as firmware
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
transmit_raw/3 used to block the connection for the frame's air time.
It now runs in a task under a new Espex.InfraredQueue, which applies
the firmware's reply rules:

- One pending reply per entity key. A newer request on the key, from
  any client, answers the older one false at once; a queued older
  request is dropped, a running one finishes unanswered.
- One call at a time per adapter module; different modules run in
  parallel. A request finishes on the adapter its connection started
  with.
- A call not back 30 s past the frame's air time (counted up to 8 min)
  is answered false and killed, and the next frame starts.
- A requester that disconnects loses its replies; its frames still go
  out.

Behaviour changes: clients below API 1.18 get no completion reply, as
from firmware; a key the adapter did not advertise is refused; frames
longer than 30 s on air are no longer refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new Espex.HomeAssistantStates adapter (key :home_assistant_states)
lists the entities the device wants from Home Assistant and receives
their states:

- SubscribeHomeAssistantStatesRequest is answered with one
  SubscribeHomeAssistantStateResponse per table entry; a second request
  walks the table again.
- HomeAssistantStateResponse calls handle_state/3 once when it matches
  an entry on entity id and attribute, inline and rescued.
- Espex.subscribe_home_assistant_state/2 and
  unsubscribe_home_assistant_state/2 change the table at runtime.
  Subscribed clients are sent only entries new to the table; removals
  send nothing, as there is no unsubscribe on the wire.

Espex.Server holds the table and broadcasts each change; connections
keep a copy, so matching a state never leaves the connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An Espex.Wizard.Input can carry links: subscriptions without an
entity id that the input's value fills in, like ESPHome's homeassistant
sensors linked to a wizard input. Setting the input replaces its
entries in the subscription table, and every client that subscribed to
Home Assistant states, the sender included, is sent the new ones.

Setting the value the input already has sends nothing, where firmware
resends and so doubles Home Assistant's listener. Links stay on the
device: the wizard JSON has no field for them, as upstream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new :outgoing_connection supervisor option makes the node dial Home
Assistant's dial-back listener (port 6054 by default), as ESPHome's
outgoing_connection does for networks where Home Assistant cannot reach
the node:

- Espex.Dialer follows upstream's state machine: first dial after
  delay, no dial while a target client or the dialed connection is up,
  5 s backoff doubling to 300 s with ±20 % jitter for failed dials, and
  a 5 s recheck while there is no target or no Noise key.
- The dialed socket runs through Espex.Connection via ThousandIsland's
  documented ready message (Espex.Dialer.Adopt). The connection sends
  its Noise hello first and stays the responder.
- Without a configured host, a client whose hello sets
  outgoing_connection_target over Noise becomes the target; its address
  is remembered and written through the new Espex.OutgoingTargetStore
  adapter (:outgoing_target_store).
- DeviceInfoResponse sets api_outgoing_connection_supported while
  dial-back is configured.
- The options are checked at start, raising ArgumentError.

thousand_island's floor rises to 1.5: older releases have no
HandlerConfig.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each guide says what the version added, what espex implements and
through which adapters and callbacks, gives a usage example, and lists
upgrade notes. They sit in their own "API versions" group in the docs.
DeviceConfig.api_version_minor/0 now points at them, and the README
lists the 1.18 features and the two new behaviours.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Backoff, air time and deadline arithmetic now guard their inputs as
integers, so their results are integer milliseconds as their specs say,
and outgoing_invalid!/1 is marked as never returning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A dial that dies mid-handoff no longer leaves a connection waiting
  for a socket in the one dial slot: the dialer sweeps untracked
  connections after a crashed dial and on :max_children.
- A dialed connection that says hello and dies before its dial's result
  arrives keeps the delay wait instead of escalating the backoff.
- A client below API 1.18 queues at most one IR/RF frame per key: a
  newer unanswered request drops a queued one.
- Throws and exits from OutgoingTargetStore.store_target/1 and
  HomeAssistantStates.subscriptions/0 are logged like raises instead of
  taking the dialer or server down.
- The dialed server hello returns a failed send instead of crashing,
  DeviceConfig's inspect hides the PSK, and Connection.format_status/1
  hides the Noise session keys from crash reports.
- Server table changes are explicit calls, and the IR queue shares one
  slot-release helper; an overdue unanswered transmit logs "stopped".
- Docs: wizard links outlive the client that set them; each new value
  adds a Home Assistant listener; states are not checked against the
  sender; a provisioning client can become the dial-back target.
- Tests for the in-flight dial cancel, the crashed and full dial paths,
  the dial result race, the IR start failure, a wizard swap dropping
  links, no re-seed for the same adapter, and a throwing
  subscriptions/0; waits replace state reads that raced a DOWN.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the repo's code-review skill (.github/skills/code-review):

- The dialer owns the dial handoff. Its task only opens the TCP
  connection and gives the socket to the dialer, which starts the
  connection, hands it the socket and monitors it in one callback. A
  dial that dies mid-handoff can no longer leave a connection holding
  the one slot, so the orphan sweep and the dial-result race guard are
  gone. A dropped dial is not killed; its late socket is closed.
- The dialer and its two supervisors form a :one_for_all subtree with
  the dialer last, so it starts after them (no startup retry) and stops
  first (no false "crashed" log on shutdown). Any fault still takes the
  subtree down and restarts the listener with it.
- The IR queue and its Task.Supervisor form their own :one_for_all
  subtree, task supervisor first. A queue crash now restarts only IR:
  connections answer false while it is gone instead of all dropping,
  and the queue stops before its tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Learned-target recovery and stale wizard subscription updates remain unresolved.

2 open issues
What changed in this PR

Expands Espex’s ESPHome API support with Home Assistant state subscriptions, linked wizard inputs, dial-back connections, and asynchronous IR/RF transmission.

Changes:

  • Adds subscription management, wizard links, and optional dial-back adapters.
  • Moves IR/RF transmission into a supervised queue and redacts Noise secrets.
  • Adds integration tests and API version guides; raises the ThousandIsland minimum.
File Description
test/​support/​noise_client.ex Adds a Noise test client.
test/​support/​fake_adapters.ex Adds feature-specific test adapters.
test/​espex/​wizard_test.exs Tests link validation and serialization.
test/​espex/​server_state_test.exs Updates adapter defaults.
test/​espex/​outgoing_connection_test.exs Tests encrypted dial-back sessions.
test/​espex/​infrared_queue_test.exs Tests queue scheduling and completion.
test/​espex/​home_assistant_states/​table_test.exs Tests subscription ownership.
test/​espex/​home_assistant_states/​subscription_test.exs Tests subscription validation.
test/​espex/​home_assistant_states_test.exs Tests state subscriptions over TCP.
test/​espex/​encrypted_integration_test.exs Tests redaction and hello MAC formatting.
test/​espex/​dispatch_test.exs Tests new protocol dispatch behavior.
test/​espex/​dialer/​helpers_test.exs Tests backoff and address parsing.
test/​espex/​dialer_test.exs Tests dialer lifecycle and configuration.
test/​espex/​device_config_test.exs Tests MAC normalization.
test/​espex/​connection_state_test.exs Updates adapter defaults.
test/​espex/​api_1_18_test.exs Tests asynchronous transmission and wizard links.
README.md Summarizes features and links guides.
mix.exs Raises dependency minimum and groups guides.
lib/​espex/​wizard/​input.ex Defines and documents input links.
lib/​espex/​wizard/​document.ex Validates and normalizes links.
lib/​espex/​supervisor.ex Wires queue and dialer subtrees.
lib/​espex/​server.ex Owns and broadcasts subscription changes.
lib/​espex/​server_state.ex Stores subscription tables and adapters.
lib/​espex/​outgoing_target_store.ex Defines target persistence behavior.
lib/​espex/​infrared_queue.ex Implements supervised transmission scheduling.
lib/​espex/​infrared_proxy.ex Documents asynchronous transmission semantics.
lib/​espex/​home_assistant_states/​table.ex Tracks subscription ownership.
lib/​espex/​home_assistant_states/​subscription.ex Defines validated subscriptions.
lib/​espex/​home_assistant_states.ex Defines state delivery behavior.
lib/​espex/​dispatch.ex Handles subscriptions, links, and dial-back messages.
lib/​espex/​dialer/​target.ex Parses dial-back addresses.
lib/​espex/​dialer/​backoff.ex Implements retry timing.
lib/​espex/​dialer/​adopt.ex Hands outbound sockets to connection handlers.
lib/​espex/​dialer.ex Manages dialing and learned targets.
lib/​espex/​device_config.ex Hides PSKs and normalizes hello MACs.
lib/​espex/​connection.ex Integrates new features and secret redaction.
lib/​espex/​connection_state.ex Tracks subscriptions and dial-back state.
lib/​espex.ex Exposes runtime subscription APIs.
guides/​api/​1.18.md Documents features and upgrade behavior.
guides/​api/​1.17.md Explains serial ownership and port modes.
guides/​api/​1.16.md Explains proxy acknowledgements.
guides/​api/​1.15.md Explains device capabilities.

Review effort: Balanced

Comment thread lib/espex/dialer.ex Outdated
Comment thread lib/espex/server.ex Outdated
From Copilot's review and the Argus check on #45:

- The dialer stores the target it learned, and the one its store last
  accepted, in Espex.Server, and a restarted dialer starts from them, so
  a dialer fault no longer drops a learned target back to the boot seed.
- Wizard links go through Server.set_wizard_links/4 with the
  connection's wizard module. A connection accepted before
  update_adapters/2 swapped the wizard can no longer put the old
  document's links back.
- The IR queue and dialer subtrees are Espex.InfraredQueue.Supervisor
  and Espex.Dialer.Supervisor. Argus reads their task supervisor, a
  runtime name, as the top-level one; mix.exs ignores that inferred
  finding for the two call sites, with the reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Overlapping IR/RF requests leak process monitors in the shared transmit queue.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Prevent leaked monitors when updating existing pending transmit keys

lib/​espex/​infrared_queue.ex:216

Map.update/4 eagerly evaluates its default argument even when pid is already present. When one client has pending transmits on two different keys, this creates another process monitor but discards its reference. Releasing both requests removes only the retained monitor, so repeated overlapping requests accumulate monitors in the shared queue until the client disconnects. Create the monitor only in the missing-key branch.

2 resolved since last review

Review effort: Balanced

Map.update/4 evaluated its Process.monitor/1 default even when the
requester already had a pending transmit, so every overlapping request
added a monitor the queue never removed. The monitor is now created
only for a requester the queue does not hold one for. From Copilot's
review on #45.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Subscription ordering remains unresolved, and socket adoption with coupled process recovery needs final human validation.

2 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Allow MapSet subscriptions in the request typespec

lib/​espex/​home_assistant_states/​table.ex:29

Dispatch.handle_request/2 passes state.ha_subscriptions, a MapSet, here when handling SubscribeHomeAssistantStatesRequest. The implementation accepts it, but this list-only typespec excludes that valid caller and contradicts ConnectionState.t(). Include the MapSet input type so the contract matches actual usage.

Review effort: Balanced

Comment thread lib/espex/connection.ex Outdated
Comment thread test/espex/infrared_queue_test.exs
From Copilot's review on #45:

- After Server.set_wizard_links/4 returns, the connection applies the
  table broadcasts already in its mailbox, its own included, in order.
  A state for the new entity, or a subscribe request, buffered behind
  the input set in the same read now sees the new links, as on
  firmware, instead of the old table.
- The IR monitor test syncs with the queue before reading its monitors:
  the reply goes out before the monitor is dropped.
- Table.sort/1's spec takes the MapSet Dispatch passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

An unresolved queue issue remains, and socket adoption and restart behavior require human validation.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Drop older queued frames when superseding with a 1.18 request

lib/​espex/​infrared_queue.ex:157

A 1.18 request leaves older queued, unanswered frames for the same key and adapter intact because they have no entry in state.slots. For example, while another transmit occupies the adapter, enqueue a legacy frame for key K followed by a 1.18 frame for K: both transmit instead of dropping the older queued frame. Requests whose clients disconnected have the same problem. Remove matching queued unowned frames before checking the reply slot, while leaving running frames alone. Add a mixed-version regression test for this sequence.

2 resolved since last review

Review effort: Balanced

A 1.18 request superseded only the key's slot owner, so a queued frame
nobody waited on (from a client below 1.18, or one that disconnected)
still transmitted ahead of it. Every newer request for a key now drops
queued unanswered frames for that key and adapter; running frames are
left alone. From Copilot's review on #45.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Socket adoption and coupled restart behaviour require human validation, alongside the unresolved target-store cache issue.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Store swaps leave stale target cache and lose dial-back targets

lib/​espex/​dialer.ex:273

After a target is saved through StoreA, Espex.update_adapters/2 can replace :outgoing_target_store with StoreB without changing stored_target. Subsequent target hellos return here before looking up StoreB, so it never receives the address while the IP stays unchanged. A reboot seeded from StoreB can then lose the learned dial-back target. Include the store identity in the successful-write cache and the copy retained by Server, or invalidate both when the adapter changes. Add a regression test that swaps stores with an unchanged target and checks persistence, including after a dialer restart.

Review effort: Balanced

The dialer skipped the write when the address matched the last one
stored, whichever store took it, so after update_adapters/2 swapped
the :outgoing_target_store the new store never got an unchanged
address. The dialer, and the copy Espex.Server keeps for a restarted
dialer, now record {store module, address}, and a write is skipped
only when the current store already holds the address. From Copilot's
review on #45.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bbangert
bbangert requested a balanced review from Copilot October 21, 2026 02:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved concurrency issues and the new supervision boundaries and outbound socket handoff require human validation.

2 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Checkpoint dialer state before killing after store notification

test/​espex/​dialer_test.exs:138

The store notification is sent before learn/2 checkpoints the target in Espex.Server. Killing the dialer after receiving that notification can interrupt the checkpoint, so the restarted dialer restores the original 127.0.0.2 seed and this test fails depending on scheduling. Call :sys.get_state(dialer) before the kill to wait for the hello callback to finish.

Review effort: Balanced

Comment thread lib/espex/infrared_queue.ex Outdated
Comment thread test/espex/dialer_test.exs
A transmit task reports its result just before it exits, and the queue
started the lane's next frame at once. The task's exit and that start
reach the task supervisor from different processes, so at the
supervisor's limit the start could find the finished task still
holding a slot and fail the next frame with :max_children. The queue
now terminates the finished task synchronously first. The two dialer
restart tests wait for the hello callback to finish before killing the
dialer. From Copilot's review on #45.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Outbound socket adoption and supervision recovery paths need human validation of lifecycle behavior and dependency compatibility.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@bbangert
bbangert merged commit 95f8275 into main Oct 21, 2026
8 checks passed
@bbangert
bbangert deleted the feat/api-1-18-full branch October 21, 2026 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants