Skip to content

DeveloperGuide.md

Latest commit

 

History

History
184 lines (125 loc) · 11.5 KB

DeveloperGuide.md

File metadata and controls

184 lines (125 loc) · 11.5 KB

Information for developers

This Slicer extension is in active development. The API may change from version to version without notice.

Build instructions

  • Build the extension against the newly built Slicer using the SuperBuild system.
  • To start Slicer from a build tree and ensure the extension is properly loaded, consider running the SlicerWithVirtualReality launcher. For more details, see here.

CMake build options

The top-level CMakeLists.txt exposes three backend options:

CMake option Default (Windows) Default (macOS) Description
SlicerVirtualReality_HAS_OPENVR_SUPPORT ON OFF Build the OpenVR XR backend
SlicerVirtualReality_HAS_OPENXR_SUPPORT ON OFF Build the OpenXR XR backend
SlicerVirtualReality_HAS_OPENXRREMOTING_SUPPORT ON OFF Build OpenXR Remoting support (HoloLens 2)

OpenXR Remoting is automatically disabled if SlicerVirtualReality_HAS_OPENXR_SUPPORT is OFF. It is only supported on Windows.

Key classes

Class Location Description
vtkMRMLVirtualRealityViewNode VirtualReality/MRML/ MRML node holding all VR view settings (backend, magnification, controller transforms, etc.)
vtkSlicerVirtualRealityLogic VirtualReality/Logic/ Main logic class: activates/deactivates VR, manages the active view node, and sets up button bindings
qMRMLVirtualRealityView VirtualReality/Widgets/ Qt widget that owns the VTK render window and interactor for the VR view
vtkVirtualRealityViewInteractorObserver VirtualReality/MRMLDM/ Bridges VTK VR interactor events to Slicer displayable managers
vtkVirtualRealityViewInteractorStyleDelegate VirtualReality/MRMLDM/ Shared delegate implementing scene/object grab and gesture logic for both OpenVR and OpenXR styles
vtkVirtualRealityComplexGestureRecognizer VirtualReality/MRMLDM/ Slicer-specific two-controller gesture recognition (translate/rotate/scale)
vtkMRMLVirtualRealityViewDisplayableManagerFactory VirtualReality/MRMLDM/ Singleton factory that registers displayable managers for the VR view

Mapping of Controller Action to VTK event

The mapping process consists of two main steps:

  1. Parsing the vtk_open<vr|xr>_actions.json action manifest file to link controller-specific interaction paths with generic event paths. This file references controller-specific binding files, usually named vtk_open<vr|xr>_binding_<vendor_name>.json, where each controller interaction path is associated with a VTK-specific event path.

  2. Assigning a VTK event path to either a VTK event or a std::function. This association of a VTK event path involving a single controller with a VTK event is carried out in vtkOpen<VR|XR>InteractorStyle::SetupActions().

Action Manifest File

The controller interaction paths are specific to each backend:

As of Slicer@c7fe8657c, the provided vtk_open<vr|xr>_actions.json and vtk_open<vr|xr>_binding_<vendor_name>.json files in the vtkRenderingOpenVR and vtkRenderingOpenXR VTK modules are as follow:

OpenVR OpenXR
Action manifest url url
- HP Motion Controller url url
- HTC Vive Controller url url
- Microsoft Hand Interaction url
- Oculus Touch url url
- Valve Knuckles url url
- Khronos Simple Controller1 url

These files serve as essential references for mapping controller actions to VTK events.

Mapping of VTK event path

The association of VTK event paths to VTK events hardcoded in each VTK modules is as follow:

Action Identifier Differences Between Backends

OpenVR and OpenXR use different formats for action identifiers when calling vtkVRRenderWindowInteractor::AddAction():

Backend Format Example
OpenVR Full action path /actions/vtk/in/TriggerAction
OpenXR Action name only (lowercase) triggeraction

The vtkSlicerVirtualRealityLogic::SetGestureButton*() helpers detect the active backend at runtime using vtkOpenXRRenderWindowInteractor class name and apply the correct identifier automatically. Additionally, OpenXR grip/squeeze button bindings are inconsistent across controllers (the right grip is typically bound to positionprop, while the left grip may be bound to complexgestureaction or not bound at all), so both identifiers are registered when using OpenXR.

The default button configuration (set in qMRMLVirtualRealityViewPrivate::createRenderWindow()) is:

  • Trigger button: grab objects and world
  • Grip button: complex gesture (translate/rotate/scale scene)

Complex Gesture Support

Recognition of complex gesture events commences when the two controller buttons mapped to the ComplexGesture action are pressed.

The SlicerVirtualReality implements its own heuristic by specializing the HandleComplexGestureEvents() and RecognizeComplexGesture() in the vtkVirtualRealityComplexGestureRecognizer class.

Limitations:

  • The selected controller buttons are exclusively mapped to the ComplexGesture action and cannot be associated with a regular action.

  • To workaround an OpenVR specific limitation, each button expected to be involved in the complex gesture needs to be respectively associated with /actions/vtk/in/ComplexGestureAction and /actions/vtk/in/ComplexGestureAction_Event2.

Useful Python Snippets

Activate virtual reality view:

import logging
import slicer

def isXRBackendInitialized():
    """Determine if XR backend has been initialized."""
    vrLogic = slicer.modules.virtualreality.logic()
    return vrLogic.GetVirtualRealityActive() if vrLogic else False

def vrCamera():
    # Get VR module widget
    if not isXRBackendInitialized():
        return None
    # Get VR camera
    vrViewWidget = slicer.modules.virtualreality.viewWidget()
    if vrViewWidget is None:
      return None
    rendererCollection = vrViewWidget.renderWindow().GetRenderers()
    if rendererCollection.GetNumberOfItems() < 1:
        logging.error('Unable to access VR renderers')
        return None
    return rendererCollection.GetItemAsObject(0).GetActiveCamera()


assert isXRBackendInitialized() is False
assert vrCamera() is None

vrLogic = slicer.modules.virtualreality.logic()
vrLogic.SetVirtualRealityActive(True)

assert isXRBackendInitialized() is True
assert vrCamera() is not None

Set virtual reality view background color to black:

color = [0,0,0]
vrView=getNode('VirtualRealityView')
vrView.SetBackgroundColor(color)
vrView.SetBackgroundColor2(color)

Set whether a node can be selected/grabbed/moved:

nodeLocked.SetSelectable(0)
nodeMovable.SetSelectable(1)

Related VTK modules

Footnotes

  1. https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#_khronos_simple_controller_profile ↩