Repository navigation
Conversation
The WebXR client already reads immersiveMode, referenceSpace, replayPacing and showRecordingControls from its query string, but --setup-oob had no way to set them: the operator had to open the bookmark and edit the URL by hand, which is exactly what an unattended run cannot do. Follow the existing TELEOP_CLIENT_* convention (codec, panelHiddenAtStart) rather than adding a new mechanism. All four default to unset, so a URL built without them is byte-identical to today's. immersiveMode matters beyond replay: a PICO 4 Ultra rejects immersive-ar with NotSupportedError, so a headset that needs vr currently cannot be driven by --setup-oob at all. Signed-off-by: Emily Ma <yuhanm@nvidia.com>
A PICO 4 Ultra exposes one weblayer_devtools_remote_<pid> socket per browser process. Prefix priority cannot separate them, so discovery returned whichever came first in /proc/net/unix - observed driving a new-tab page while the teleop tab sat on the second socket. Nothing reports it: CDP connects, the tab list is non-empty, and CONNECT is simply never clicked. Probe only the sockets that tie at the top of the priority order, and keep the one with an oobEnable= tab. Prefix order still decides on its own whenever it can, so the single-socket and mixed-prefix paths are unchanged and cost no extra adb calls. Finding none is the normal pre-am-start state and still returns the best-ranked socket. Signed-off-by: Emily Ma <yuhanm@nvidia.com>
--setup-oob could open the page and click CONNECT, but the client's recording file input can only be filled through the browser's own filesystem, so a run still began with someone picking a file in the headset. TELEOP_OOB_RECORDING pushes the file over adb and selects it via DOM.setFileInputFiles, between page-ready and the CONNECT click. Before rather than after: the input is rendered from first paint by RecorderProvider, so waiting buys nothing and means reaching into a page the operator is already immersed in. Unset - the default - leaves the flow untouched. Starting the replay itself stays with the operator. Signed-off-by: Emily Ma <yuhanm@nvidia.com>
Five additions to the environment table in the OOB reference, placed next to the TELEOP_CLIENT_* and TELEOP_OOB_* entries they extend. Signed-off-by: Emily Ma <yuhanm@nvidia.com>
A benchmark replays the same recording the same way on every run, which until now meant editing the headset URL by hand - something an automated run cannot do, and the one step that kept --setup-oob from being usable unattended. --oob-reference-space pin the XR reference space --oob-replay-pacing frame-paced replay, independent of render speed --oob-immersive-mode vr, for headsets that reject immersive-ar --oob-recording PATH to push, or 'latest' from the headset --oob-recording-controls show the in-XR replay buttons Each is the command-line spelling of a TELEOP_* variable the OOB code already reads, so the flag is the only new surface; omit them all and the URL is the one built today. They require --setup-oob, which is what opens the page they describe, and 'service start' passes them through to the detached 'run'. Selecting the recording is as far as this goes: the operator still starts the replay. Signed-off-by: Emily Ma <yuhanm@nvidia.com>
Connecting retries until the application has an XR session, and each attempt re-opens the page and re-selects the file. A sim that spends ten minutes loading its scene therefore pushed the same recording dozens of times. Key the cache on the resolved path, size and mtime so a re-recorded file of the same name still lands. 'latest' is left uncached: it is one adb shell ls, and re-reading it means a recording made while waiting is picked up. Signed-off-by: Emily Ma <yuhanm@nvidia.com>
run_oob_connect forwards the CDP port before the page exists, and _find_and_click_teleop_tab then polls only that socket. On a PICO the browser runs several WebLayer processes, and _discover_devtools_socket tells them apart by asking which one holds an oobEnable= tab - but _close_stale_teleop_tabs has just closed exactly those, so on a cold start every candidate answers empty and the tie falls back to rank order. When that guess is the new-tab process, the loop watches a socket the page never appears on and only gives up at the deadline. Re-run discovery every third pass of the wait loop and move the forward when it names a different socket. Once am start has painted the page the oobEnable= probe matches for real, so the loop corrects itself within a few seconds. On the headset this took cold starts that timed out after 60s down to about 3s, across browser restarts that changed the socket each time. Signed-off-by: Emily Ma <yuhanm@nvidia.com>
The service already knows how to hold headset setup back: defer_oob_connect gates both the OOB lifecycle and the relay of headset signalling until connect_headset() is called. Only code that embeds the service can reach it, though. `service run --setup-oob` always starts the lifecycle as soon as the proxy listens, so when the OpenXR application starts after the service - the usual order, since the app needs the runtime first - the headset enters XR and connects to a runtime with nothing to stream, then cycles through the client's attach and warm-up timeouts and the lifecycle's retries until the app arrives. Expose the gate as --defer-oob-connect, and open it with GET /api/oob/v1/connect. The endpoint authenticates like the other hub endpoints, keeps their no-CORS response, is idempotent, and answers 409 on a service that was not started deferred rather than pretending to have done something. GET because the proxy's HTTP layer only parses GET; /api/oob/v1/config already writes through it the same way. Without the flag nothing changes: no gate is created and setup starts at once, as before. `service start` passes the flag through to the detached run. Signed-off-by: Emily Ma <yuhanm@nvidia.com>
On PICO, headset_browser_package() prefers com.pico.browser.overseas when it is installed. That package is a regular app, and PICO OS intercepts a VR launch from a regular app that does not declare hand tracking: while the headset is in hand-tracking mode it shows "This app requires the use of controllers for interaction" and only enters VR once someone answers. Nobody is there to answer on an unattended run, so CONNECT is dispatched and the XR session never starts. com.pico.browser ships as a system app, which the same check skips. Both are shells over the system WebLayer, so the page behaves the same; on a PICO 4 Ultra in hand-tracking mode the system browser entered VR 1s after CONNECT with no dialog, where the overseas one waited on it. TELEOP_HEADSET_BROWSER_PACKAGE already overrides the choice. Give it the same command-line spelling as the other --oob-* client flags, and document the variable, which the environment table did not list. The per-vendor default is unchanged. Signed-off-by: Emily Ma <yuhanm@nvidia.com>
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
We benchmark teleoperation by replaying a recorded session, so every run needs
the same input, started the same way, with nobody in the headset. #1307 makes
the web client replay that way; this PR is the host side. With
--setup-oob,the service can now take a run from launch to a streaming XR session with the
recording already selected, unattended.
Everything is in
src/python/isaaccapture/cloudxr/, its tests andoob_teleop_control.rst. There are no web-client changes: the page is drivenonly through URL parameters and controls that the client on
mainalready has,so this does not depend on #1307. Without the new flags, behaviour is unchanged.
Choose the page a run starts from.
--oob-reference-space,--oob-replay-pacing,--oob-immersive-mode,--oob-recording PATH|latestand
--oob-recording-controlsare the command-line spelling ofTELEOP_*variables, forwarded to the headset URL. A recording is selected in the
client's file input before CONNECT, and pushed to the headset once per run
rather than once per connect attempt. Starting the replay stays an operator
action.
Connect the headset after the application, not before. The service
already had a
defer_oob_connectgate, but only code embedding it couldreach it, so
service runalways connected the headset at startup, beforethe OpenXR app that needs the runtime first had started. The headset then
sits on a runtime with nothing to stream and cycles through the client's
timeouts and the lifecycle's retries.
--defer-oob-connectexposes the gateand
GET /api/oob/v1/connectopens it. The endpoint authenticates like theother hub endpoints (fix(cloudxr): require a control token on the OOB teleop hub #1316), sends no CORS headers (fix(cloudxr): stop serving the OOB hub API cross-origin and reject oversized hub frames before parsing #1317), is idempotent,
and answers 409 on a service that was not started deferred.
Find the teleop tab on PICO. The PICO browser runs several WebLayer
processes. The DevTools socket is now chosen by which process actually
holds the teleop tab, and re-scanned while waiting for the page, because on
a cold start the stale-tab cleanup has just closed the tabs that would tell
them apart. Before this, the wait watched the wrong socket about half the
time and gave up at the 60 s deadline.
--oob-browser-package. PICO OS holds a VR launch from an app that isneither a system app nor declares hand tracking behind a "This app requires
the use of controllers" dialog whenever the headset is in hand-tracking
mode.
com.pico.browser.overseas, which is preferred by default, is such anapp;
com.pico.browserships as a system app and is not held. The flagspells the existing
TELEOP_HEADSET_BROWSER_PACKAGEoverride. The defaultis unchanged.
A typical unattended run:
python -m isaaccapture.cloudxr.service run --accept-eula --setup-oob --host-client \ --defer-oob-connect \ --oob-reference-space local --oob-replay-pacing frame --oob-immersive-mode vr \ --oob-recording latest --oob-recording-controls \ --oob-browser-package com.pico.browser # start the application; once it has its OpenXR session: curl -k -H "X-Control-Token: $CONTROL_TOKEN" https://localhost:48322/api/oob/v1/connect#1318 moves the control token into the URL fragment and also touches
_service.pyandwss.py; whichever lands second will need a small rebase.Type of change
Testing
pytest tests/python/core/cloudxr: 684 passed. The 2 failures, intest_wss_oob_lifecycle.py, fail the same way on unmodifiedmainin thisenvironment; they need the native build, as fix(cloudxr): require a control token on the OOB teleop hub #1316 also notes.
pre-commit run --from-ref main --to-ref HEAD: all hooks pass.196ecd5da(Isaac Sim 6.0.1):--defer-oob-connect, the service left the headset untouched until/api/oob/v1/connect. It then opened the page, selected the latestrecording and clicked CONNECT. The XR session was active 1 s after the
click, and the lifecycle confirmed the stream 32 s after the release.
--oob-browser-package com.pico.browser.With the default overseas browser, the same run stopped at the controller
dialog.
socket changed each time. The page was found within about 3 s on every
restart.
is on a slow cross-subnet Wi-Fi link that does not sustain the stream.
Checklist
SKIP=check-copyright-year pre-commit run --all-filesgit commit -s) per the DCO