Fix VRF sub-device discovery: defer + concurrent subList, robust parsing - #507
Conversation
RobHofmann
added
help wanted
|
I need some testers with VRF devices for this one! |
* Add integration code to python analysis so that autocomplete works for the new tools * Use VRF local discovery as in #507 * Remove unnecessary check for sub_devices lists length * Prevent one VRF gateway not connecting from breaking full discovery process Also properly use the discover timeout for getting the subdevices * Properly assign VRF subunits a name * Use the retireved key for sub-units * Reduce time waster discovering-subdevices when errors occur --------- Co-authored-by: Pedro Monteiro <p-monteiro@users.noreply.github.com>
|
Hi @vellad1, @Ilya-Draigor and @ItayGo — you all run VRF setups. @RobHofmann needs a VRF owner to validate this PR. Could any of you give it a quick test and report back? Thanks! |
Two problems made multi-split (VRF) gateways discover unreliably: 1. Sub-device (subList) fetching ran inline inside the UDP receive loop. The subList query has its own retries/timeouts, so it blocked the loop and ate the discovery time budget, causing other devices' broadcast replies to be dropped (e.g. finding 3 of 4 units). Gateways are now collected during the loop and queried afterwards, concurrently via asyncio.gather, keeping discovery fast and complete. 2. A gateway was only treated as such when subCnt > 1, so single-unit gateways were missed. Any subCnt > 0 is now handled as a gateway. get_subunits_list is rewritten to send subList unencrypted and handle both response shapes seen in the wild: the sub-unit list at the top level, or wrapped in a pack encrypted with the gateway's bound device key (bind first, then decrypt). A small self-contained _fetch_subunits_raw helper does the raw UDP exchange since the response has no pack to feed through FetchResult.
973cfe1 to
cae8692
Compare
The gateway only answers a subList query when it is encrypted, and it encrypts
the *response* with a different key depending on the request envelope:
- i:0 / t:pack -> response encrypted with the gateway's bound device key.
- i:1 / t:subList (classic app form) -> response encrypted with the generic
key, like scan/bind.
Different WiFi-module firmwares answer different forms, and some gateways
return a slightly different subset of units in each form. get_subunits_list now
binds once, queries both forms, and unions the results by MAC. This handles
both module families and never returns fewer units than either form alone.
Verified on a GR-Gcloud V3.2.M gateway: device-key form returns 4 units,
generic-key form returns 3, merged result is the full 4, alongside a standalone
non-VRF unit discovered normally.
cae8692 to
90a1bdc
Compare
When several gateways are probed concurrently (asyncio.gather), the unconnected UDP sockets could receive datagrams intended for another in-flight request - observed in the wild as one gateway's bind call returning a different gateway's key. Connecting the socket to the target peer makes the kernel drop datagrams from any other address, correlating each reply to its request. Applied to both FetchResult and the subList helper. FetchResult is only ever called with unicast device IPs, so this does not affect broadcast discovery.
|
@RobHofmann #507 is now validated end-to-end on real hardware across three testers and two different WiFi-module generations:
Two findings from testing shaped the final approach:
Also included: deferred + concurrent sub-device fetch (so discovery no longer drops other devices' replies), and connecting the UDP sockets to their target so concurrent gateway probes don't receive each other's responses. #507 is |
|
@RobHofmann summary across the VRF series — all three PRs are now validated on real hardware by VRF owners:
The three are independent and additive, but together they make VRF/multi-split work end-to-end. Testing surfaced real module variance that #507 now handles (different sub-enumeration commands — Given the confirmations from multiple VRF owners, would you be open to reviewing/merging the set? Happy to rebase, split, or adjust anything you'd like. Thanks! |
Older gateway WiFi modules (e.g. W06, hid 362001067012+U-W06AV30.bin, ver V1.1.0.0) do not answer the subList command at all - they enumerate sub-units via a 'subDev' command wrapped in a device-key pack, returning the same subList-shaped response. Captured from a real W06 gateway. get_subunits_list now also issues the subDev form (with uid:0) and unions its result with the subList forms. Confirmed on a GR-Gcloud V3.2.M gateway the subDev form returns the full unit list too (device-key=4, generic-key=3, subDev=4, merged=4), so it is a good universal path and adds no regression.
a5038a3 to
626a7da
Compare
|
https://github.com/RobHofmann/HomeAssistant-GreeClimateComponent/releases/tag/4.0.7 Thanks for contributing guys! amazing work! I will do another PR towards the 5.0-dev branch for the future version with the same knowledge! Will ask you guys to help me test that one too later. |
|
@meirlo a follow-up question on the generic-key form, raised by @p-monteiro while porting this to 5.0 (#522). The request pack is encrypted with the device key but the reply comes back in the generic key. On my standalone unit, Could you run this read-only script against your GR-Gcloud? It sends one bind (the gateway returns its existing key) and three subList queries, and no commands.
probe_vrf_sublist_keys.py"""Which key does a Gree VRF gateway expect in a subList request with i=1?
Usage: python probe_vrf_sublist_keys.py <gateway-ip> <gateway-mac>
Needs: pip install pycryptodome
Read-only. It sends one bind (the gateway returns its existing key) and a few
subList queries. It sends no command, so nothing on the units changes.
"""
import base64
import json
import socket
import sys
from Crypto.Cipher import AES
HOST, MAC, PORT = sys.argv[1], sys.argv[2].replace(":", "").lower(), 7000
GENERIC = b"a3K8Bx%2r8Y7#xDh"
RANDOM = b"0123456789abcdef"
def pad(s):
n = 16 - len(s) % 16
return s + chr(n) * n
def enc(key, obj):
raw = AES.new(key, AES.MODE_ECB).encrypt(pad(json.dumps(obj)).encode())
return base64.b64encode(raw).decode()
def dec(key, b64):
try:
t = AES.new(key, AES.MODE_ECB).decrypt(base64.b64decode(b64)).decode("utf-8", "ignore")
return json.loads(t[: t.rindex("}") + 1])
except Exception:
return None
def send(payload, tries=3):
for _ in range(tries):
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.settimeout(2)
try:
s.sendto(json.dumps(payload).encode(), (HOST, PORT))
return json.loads(s.recvfrom(64000)[0])
except socket.timeout:
pass
finally:
s.close()
return None
def env(t, i, pack):
return {"cid": "app", "i": i, "pack": pack, "t": t, "tcid": MAC, "uid": 0}
def report(label, resp, key):
if resp is None:
print(f"{label}: no answer")
return
pack = resp.get("pack", "")
for name, k in (("generic", GENERIC), ("device", key)):
body = dec(k, pack) if pack else None
if body is not None:
n = len(body.get("list", [])) if isinstance(body.get("list"), list) else "no list"
print(f"{label}: answer readable with the {name} key, units={n}")
return
print(f"{label}: answer, but readable with neither key")
bind = send(env("pack", 1, enc(GENERIC, {"mac": MAC, "t": "bind", "uid": 0})))
if bind is None or dec(GENERIC, bind["pack"]) is None:
sys.exit("The bind got no usable answer. Check the IP and MAC.")
KEY = dec(GENERIC, bind["pack"])["key"].encode()
print("bind: ok")
inner = {"mac": MAC, "i": 1}
report("A subList i=1, pack with device key ", send(env("subList", 1, enc(KEY, inner))), KEY)
report("B subList i=1, pack with generic key", send(env("subList", 1, enc(GENERIC, inner))), KEY)
report("C subList i=1, pack with random key ", send(env("subList", 1, enc(RANDOM, inner))), KEY)If C (random key) also answers, the pack is ignored and we can use the generic key both ways. Thanks! |
|
Ran the probe against my GR-Gcloud ( All three request keys — device, generic, and even a random key (case C) — get the same reply, decryptable with the generic key, One thing worth carrying over: on my gateway this generic-key form returns a partial list (3 of 4), while the device-key Thanks for the clean read-only probe — made this easy to answer. |
|
Thanks for the input man! 4.0.10 has went out! Also, working on the VRF PR towards the 5.0-dev right now. will be available shortly. |
…until confirmed (#522) * VRF gateways: ask all sub-device list forms, hold sent values until confirmed Port of the VRF fixes that landed on master in #507 and #508. #454 needs no port: 5.0 already gives each indoor unit its own device. Sub-device list: a gateway answers one or more of three request forms, depending on its WiFi module firmware (device key, generic key, subDev). The old single hybrid form was never confirmed on hardware. V1 gateways now get all three forms and the lists are joined by MAC; V2 gets the device key form. The generic key form is encrypted with the device key but answered with the generic key, so request_json() takes an optional response_cipher. Gateways from a scan are handled side by side. On the GR-Gcloud V3.2.M gateway from #507 the forms gave 4, 3 and 4 units, joined 4. Stale state: a gateway answers from a cache for a few seconds after a command, so the read right after it showed the old value again. The device state now holds sent values until the device reports them, for at most 8 s. The coordinator polls again every 2 s while values are held. A standalone unit confirms on the first read and gets no extra poll. Suite: 247 passed (227 before). * Only hold sent values for VRF sub-units The stale cache was only seen on VRF gateways. For a standalone unit a hold has a cost: when the unit corrects a value it cannot take, for example an unsupported swing mode, it reports its own value, and the hold kept the refused value on screen for up to 8 s. Standalone units now behave as they did before the hold. To find out whether a standalone unit or the MQTT transport caches too, every device now logs at debug level when the read right after a command does not report the sent value, with the transport and whether it is a sub-unit. Suite: 249 passed. * Send the generic-key sub-device list request with the generic key The generic-key form is answered with the generic key, but its request used the bound device key, so transport.request_json() needed a separate response_cipher for this one case. A probe on a GR-Gcloud V3.2.M gateway (PR 507) showed the gateway ignores the request pack of this form: device, generic and random keys all got the same answer, 3 units, readable with the generic key. The request now uses the generic key too, and response_cipher is removed again. The form stays in the union, because it adds units on some gateways.
|
@meirlo @ItayGo @Ilya-Draigor @vellad1 the VRF work is now ported to the 5.0 line, and I'd like your help to test it on real hardware. What is in
Before you start: 5.0 is an alpha ( Install: download the 5.0-dev branch, copy What would help most:
Thanks again for all the help on the 4.x side! |
|
Sure I'll give it a try. |
|
Yes and no. It kind of depends on whether you are using YAML or the config flow for the integration. Config flow: 5.0-dev will automatically migrate the 4.x config into the 5.0 integration and remove the 4.x config. YAML: It will give you the new YAML through the portal, which you have to copy in. Then you simply can let them co-exist (or comment out the YAML from 4.x for the testing period). |
3 participants
Problem
On multi-split (VRF) setups — one WiFi gateway fronting several indoor units — discovery was unreliable:
subList) fetching ran inline inside the UDP receive loop. That query has its own retries/timeouts, so it blocked the loop and consumed the discovery time budget while other devices' broadcast replies were still arriving. Result: intermittently finding e.g. 3 of 4 units.subCnt > 1, so a gateway reporting exactly one indoor unit was never expanded.subListnever returned usable data. The old code sent an unencrypted/generic-key query and pushed the reply throughFetchResult, which failed on real gateways.What the gateway actually expects
The gateway only answers a
subListquery when it is encrypted, and it encrypts the response with a different key depending on the request envelope:i:0/t:"pack"→ response encrypted with the gateway's bound device key (e.g. GR-Gcloud firmwareV3.2.M).i:1/t:"subList"(the classic Gree-app form) → response encrypted with the generic key, like scan/bind. Some modules only answer this form.Different firmwares answer different forms, and some gateways even return a slightly different subset of units in each form.
Changes
asyncio.gather). Keeps discovery fast and stops it dropping other devices' replies.subCnt > 0as a gateway.get_subunits_list: bind to the gateway, query both encrypted forms, and union the results by MAC. This handles both module families and never returns fewer units than either form alone. v2 (GCM) modules use the device-key form.Scope
gree_protocol.pyonly. Standalone (non-VRF) devices report nosubCntand never enter the gateway path — verified on a mixed setup (VRF gateway + standalone unit).Related
Part of a small VRF series. Complements #454 (per-unit
device_info/unique_id), which makes each discovered sub-unit its own HA device. This PR is what makes all the sub-units reliably get discovered.Testing
Validated on a GR-Gcloud
V3.2.Mgateway (v1/ECB) fronting 4 indoor units (subMAC@gatewayMAC) plus a standalone non-VRF unit:The device-key form returned all 4 units; the generic-key form returned a subset of 3; the union is the full 4, with the standalone unit added normally. A second GR-Gcloud gateway (another tester) discovers its units the same way.