feat(camera): read camera intrinsics + pose off the WebXR view (table stays fallback)#419
Draft
salmanmkc wants to merge 2 commits into
Draft
feat(camera): read camera intrinsics + pose off the WebXR view (table stays fallback)#419salmanmkc wants to merge 2 commits into
salmanmkc wants to merge 2 commits into
Conversation
…a access While streaming raw camera frames, read the XRView's projectionMatrix (clip-from-view) and transform.matrix (world-from-view) into the device camera, in the same reference space three renders in. Reset the flag when camera access is lost so consumers fall back cleanly.
getDeviceCameraClipFromView/WorldFromView now use the intrinsics/pose captured off the XRView when available, so the device reports its own camera math instead of a static per-device profile. The DEVICE_CAMERA_PARAMETERS table stays as the fallback for the getUserMedia path and the simulator, and targetDevice remains a manual override.
Contributor
Author
|
this only kicks in on the webxr raw-camera path and leaves the DEVICE_CAMERA_PARAMETERS table as the │ fallback, so the getUserMedia path and the simulator are unchanged. i haven't been able to validate the numbers on │ real hardware yet though, so it'd be good to confirm the intrinsics + pose line up on a galaxy xr and an aura before │ leaning on it. those two share the android xr user agent, which is the whole reason for reading off the view instead of the device table. |
Collaborator
|
I haven't tested, but I suspect it likely won't work on Galaxy XR or Aura since it's not exposed at the Android or C++ api level afaik. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
follow-up to the thread on #268. right now the device camera's intrinsics and pose come from a static per-device table (
DEVICE_CAMERA_PARAMETERS) keyed bytargetDevice, which we pick from the user agent. that's fragile: galaxy xr and aura share the same android xr user agent, so the UA can't always tell them apart.but webxr already hands us the real numbers. the raw camera access spec puts the camera's intrinsics on
XRView.projectionMatrixand its pose onXRView.transform, andupdateXRCamerais already holding that view to read the camera image offview.camera, it just wasn't reading those two fields.so this reads them off the view we already have. while raw camera access is streaming,
updateXRCameracapturesview.projectionMatrix(clip-from-view) andview.transform.matrix(world-from-view) into the device camera, in the same reference space we pass togetViewerPose(and that three renders the scene in). thengetDeviceCameraClipFromView/getDeviceCameraWorldFromViewprefer those when present, so the device reports its own camera math whatever device it is.the
DEVICE_CAMERA_PARAMETERStable stays as the fallback for the getUserMedia path (older Quest with no XRView) and the simulator, andtargetDevicestays as a manual override. the flag resets when camera access drops so we fall back cleanly.added unit tests for the selection (prefer the view params when captured, otherwise the table) and for
isDeviceCameraPoseAvailablepicking up the new source. lint, tests and build are clean.honestly not 100% sure it lines up cleanly on every device, so it only kicks in on the WebXR raw-camera path and leaves the table untouched as the fallback. worth confirming on a real galaxy xr / quest before leaning on it.