LightlyStudio Plugins

---

Lightly Studio Plugins

A collection of installable plugins that extend the base functionality of Lightly Studio.

Each plugin in this repository is packaged independently, installs in a single command, and is auto-discovered by Lightly Studio via Python entry points.

SAM3 Segmentation Plugin

Each plugin entry below includes the exact copy-paste install command. After installation, the plugin is available in Lightly Studio automatically.

Available Plugins

• BBox auto propagation nano tracker
  Propagates boxes from one annotated video frame to other frames in the same video.

  Details

  If triggered from a frame, all bounding box annotations on that frame are propagated. If triggered from an annotation, only the selected annotation is propagated.

  • Scope: video only, within a single video
  • Entry points: frame or annotation
  • Controls: forward and backward propagation windows in seconds
  • Tradeoff: uses OpenCV NanoTracker, which is lightweight and fast on many machines but less robust on difficult motion, occlusion, or scale changes
  • Maintainer: Lightly
  • Install: pip install git+https://github.com/lightly-ai/lightly-studio-plugins.git#subdirectory=plugins/bbox_auto_propagation_nano_tracker/

• SAM3 Segmentation
  Segments all instances matching a text prompt in a single image or across the current view.

  Details

  This is designed for dataset-wide prompt-based labeling workflows with class-like prompts such as person, car, or dog.

  • Scope: single image or images in the current view
  • Input: text prompt
  • Output: segmentation masks
  • Labels: the prompt text is used as the annotation class name
  • Requirement: Hugging Face access to facebook/sam3
  • Maintainer: Lightly
  • Install: pip install git+https://github.com/lightly-ai/lightly-studio-plugins.git#subdirectory=plugins/sam3_segmentation/

• LightlyTrain object detection inference
  Runs LightlyTrain object detection inference on one image or the current view for auto-labeling.

  Details

  You can use built-in LightlyTrain models for quick bootstrapping or provide a path to your own LightlyTrain checkpoint.

  • Scope: single image or images in the current view
  • Input: LightlyTrain model name or local path to a LightlyTrain checkpoint
  • Output: object detection annotations
  • Labels: class labels are read from the loaded model and created in the dataset if they do not exist yet
  • Recommended models: dinov3/convnext-large-ltdetr-coco for best performance, dinov3/vits16-ltdetr-coco for a speed/quality balance, picodet-l-coco for resource-constrained environments
  • Maintainer: Lightly
  • Install: pip install git+https://github.com/lightly-ai/lightly-studio-plugins.git#subdirectory=plugins/lightly_train_object_detection_inference/

Contributing Plugins

1. Create a new directory under plugins/:

   plugins/my_plugin/
   ├── pyproject.toml
   └── src/lightly_plugins_my_plugin/
       ├── __init__.py
       └── operator.py
   

2. Register your operator class via entry points in pyproject.toml:

   [project.entry-points."lightly_studio.plugins"]
   my_plugin = "lightly_plugins_my_plugin.operator:MyPluginOperator"

3. Update README.md and plugins.toml.

4. Install: pip install -e plugins/my_plugin