Page the model-services probe past an ACL-emptied first page

[david-siqi-liu]

What did you change, and why?

ucode configure's model-service probe (_probe_ai_gateway_v3) listed /api/2.1/unity-catalog/model-services?page_size=1 and decided accessibility from that single page. Unity Catalog applies page_size before ACL filtering, so an early page can come back empty with only a next_page_token while accessible model services remain behind the cursor.

On the ai-devtools gateway workspace this made configure print:

✔ Unity AI Gateway detected
  Model service: reachable, no accessible model services returned; check USE CATALOG on system, and USE SCHEMA and EXECUTE on system.ai

even though the caller had 42 accessible model services and full grants. The hint sent users to fix Unity Catalog permissions that were already correct.

Fix

Page the model-services probe instead of trusting one page. _probe_ai_gateway_v3 now follows next_page_token (page size 50, bounded by a page cap) and resolves to one of four outcomes, tracked by a new conclusive flag on GatewayProbe:

• an accessible model service is found on any page: resource_available;
• the listing is walked to the end (no cursor) with nothing accessible: reachable, confirmed empty;
• the first call fails: unreachable, with the reason;
• a later page errors, or the page cap is hit with a cursor still pending: reachable but conclusive=False (the listing was not walked to a verdict, so we do not claim it is empty).

Don't let an unavailable fallback bury a reachable gateway. probe_unity_gateway_capabilities returns the reachable-but-inconclusive probe rather than raising "neither endpoint available": the model-services API answered, so the gateway is enabled even though the walk did not finish.

Missing-OAuth-scope 403s route to re-login, not to a grant hint — and only after the legacy fallback is attempted, since the model-service and legacy-endpoint APIs require different scopes. The classifier matches the OAuth-token wording specifically, so a PAT's permission 403 (which re-login cannot fix) is not misrouted and instead falls through to the grant guidance.

Trim the happy-path output. When an accessible model service is found, configure closes the spinner with ✔ Unity AI Gateway connected and drops the redundant model-service detail line. It prints a single warning line only when no model service was detected but the gateway is still usable (or the result is inconclusive); a hard failure still raises.

Also removed the now-unused empty_hint parameter from _gateway_probe_result.

Related: ES-2185388

How do you know it works?

Verified the built probe against three live workspaces:

• ai-devtools gateway workspace (the reported case): the probe now resolves to an accessible model service, and configure closes the spinner with a success line:

  GatewayProbe: GatewayProbe(reachable=True, detail='reachable, accessible model service returned', resource_available=True, conclusive=True)
  --- what `ucode configure` prints for the gateway step ---
  ✔ Unity AI Gateway connected
  

• CSP-enabled workspace with no model services (v3 reachable but genuinely empty; legacy endpoints disabled for CSP): still hard-fails with an actionable error rather than a false success:

  RuntimeError: Databricks Unity AI Gateway is not enabled on this workspace: neither model
  services (reachable, no accessible model services returned; check USE CATALOG on system, and
  USE SCHEMA and EXECUTE on system.ai) nor legacy endpoints (HTTP 404 Not Found:
  {"error_code":"FEATURE_DISABLED","message":"AI Gateway legacy endpoint is not available for
  CSP-enabled workspaces."...}) are available. See https://docs.databricks.com/aws/en/ai-gateway/overview-beta
  

• A scope-limited token: probes the legacy fallback first, then returns re-login guidance instead of the UC-grant message.

Tests:

• empty-first-page-plus-cursor finds an accessible service on a later page (the ES-2185388 repro);
• exhausting the cursor reports confirmed-empty;
• a later-page error and page-cap exhaustion are reachable-but-inconclusive, not empty;
• an inconclusive probe does not hard-fail when the legacy endpoint is unavailable;
• a scope 403 succeeds when the legacy endpoint is reachable, and routes to re-login when it is not;
• the scope classifier matches the OAuth-token wording but not a PAT permission 403;
• the happy path prints the success line without the model-service detail, and a warning line is printed only when no model service is detected.

test_databricks.py + test_cli.py pass (538). ruff check and ruff format --check clean.

[lilly-luo]

RuntimeError: Databricks Unity AI Gateway is not enabled on this workspace: neither model
services (reachable, no accessible model services returned; check USE CATALOG on system, and
USE SCHEMA and EXECUTE on system.ai) nor legacy endpoints (HTTP 404 Not Found:
{"error_code":"FEATURE_DISABLED","message":"AI Gateway legacy endpoint is not available for
CSP-enabled workspaces."...}) are available. See https://docs.databricks.com/aws/en/ai-gateway/overview-beta

this error message isn't very user friendly. Can we have it say something like this? Just remove the part about legacy endpoints. OK to do in follow up sine this won't affect Isaac

Databricks Unity AI Gateway is not enabled for this workspace.

No accessible AI Gateway model services were found. This may mean that
AI Gateway is not enabled, or that you do not have the required permissions.

If you believe AI Gateway is enabled, verify that you have:
  - USE CATALOG on system
  - USE SCHEMA on system.ai
  - EXECUTE on system.ai

See: https://docs.databricks.com/aws/en/ai-gateway/overview-beta

[david-siqi-liu]

RuntimeError: Databricks Unity AI Gateway is not enabled on this workspace: neither model
services (reachable, no accessible model services returned; check USE CATALOG on system, and
USE SCHEMA and EXECUTE on system.ai) nor legacy endpoints (HTTP 404 Not Found:
{"error_code":"FEATURE_DISABLED","message":"AI Gateway legacy endpoint is not available for
CSP-enabled workspaces."...}) are available. See https://docs.databricks.com/aws/en/ai-gateway/overview-beta

this error message isn't very user friendly. Can we have it say something like this? Just remove the part about legacy endpoints. OK to do in follow up sine this won't affect Isaac

Databricks Unity AI Gateway is not enabled for this workspace.

No accessible AI Gateway model services were found. This may mean that
AI Gateway is not enabled, or that you do not have the required permissions.

If you believe AI Gateway is enabled, verify that you have:
  - USE CATALOG on system
  - USE SCHEMA on system.ai
  - EXECUTE on system.ai

See: https://docs.databricks.com/aws/en/ai-gateway/overview-beta

I don't think we can remove the part about legacy endpoints, as they are still valid use cases. Otherwise, we should only be probing for v3 and just not use v2.