Skip to content

feat(camera): read camera intrinsics + pose off the WebXR view (table stays fallback)#419

Draft
salmanmkc wants to merge 2 commits into
google:mainfrom
salmanmkc:feat/xr-camera-params
Draft

feat(camera): read camera intrinsics + pose off the WebXR view (table stays fallback)#419
salmanmkc wants to merge 2 commits into
google:mainfrom
salmanmkc:feat/xr-camera-params

Conversation

@salmanmkc

Copy link
Copy Markdown
Contributor

follow-up to the thread on #268. right now the device camera's intrinsics and pose come from a static per-device table (DEVICE_CAMERA_PARAMETERS) keyed by targetDevice, which we pick from the user agent. that's fragile: galaxy xr and aura share the same android xr user agent, so the UA can't always tell them apart.

but webxr already hands us the real numbers. the raw camera access spec puts the camera's intrinsics on XRView.projectionMatrix and its pose on XRView.transform, and updateXRCamera is already holding that view to read the camera image off view.camera, it just wasn't reading those two fields.

so this reads them off the view we already have. while raw camera access is streaming, updateXRCamera captures view.projectionMatrix (clip-from-view) and view.transform.matrix (world-from-view) into the device camera, in the same reference space we pass to getViewerPose (and that three renders the scene in). then getDeviceCameraClipFromView / getDeviceCameraWorldFromView prefer those when present, so the device reports its own camera math whatever device it is.

the DEVICE_CAMERA_PARAMETERS table stays as the fallback for the getUserMedia path (older Quest with no XRView) and the simulator, and targetDevice stays as a manual override. the flag resets when camera access drops so we fall back cleanly.

added unit tests for the selection (prefer the view params when captured, otherwise the table) and for isDeviceCameraPoseAvailable picking up the new source. lint, tests and build are clean.

honestly not 100% sure it lines up cleanly on every device, so it only kicks in on the WebXR raw-camera path and leaves the table untouched as the fallback. worth confirming on a real galaxy xr / quest before leaning on it.

…a access

While streaming raw camera frames, read the XRView's projectionMatrix
(clip-from-view) and transform.matrix (world-from-view) into the device camera,
in the same reference space three renders in. Reset the flag when camera access
is lost so consumers fall back cleanly.
getDeviceCameraClipFromView/WorldFromView now use the intrinsics/pose captured
off the XRView when available, so the device reports its own camera math instead
of a static per-device profile. The DEVICE_CAMERA_PARAMETERS table stays as the
fallback for the getUserMedia path and the simulator, and targetDevice remains a
manual override.
@salmanmkc salmanmkc marked this pull request as draft June 26, 2026 14:50
@salmanmkc

salmanmkc commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

this only kicks in on the webxr raw-camera path and leaves the  DEVICE_CAMERA_PARAMETERS  table as the │ fallback, so the getUserMedia path and the simulator are unchanged. i haven't been able to validate the numbers on │ real hardware yet though, so it'd be good to confirm the intrinsics + pose line up on a galaxy xr and an aura before │ leaning on it. those two share the android xr user agent, which is the whole reason for reading off the view instead of the device table.

@dli7319

dli7319 commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

I haven't tested, but I suspect it likely won't work on Galaxy XR or Aura since it's not exposed at the Android or C++ api level afaik.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants