LSODeviceLinkMismatch / LSONoStableLocalVolumePath

[DanielFroehlich]

After Updateing to 4.22 #390 we see a lot of warnings like

LSODeviceLinkMismatch

Local Storage operator has observed that pv local-pv-d60306a3 is using a symlink
that is not optimum for stability between reboots and OS upgrades. Consider updating
LocalVolumeDeviceLink policy to preferredSymlink to allow LSO to manage the symlink.
[View details](https://console-openshift-console.apps.isar.coe.muc.redhat.com/monitoring/alerts/3734418817?managed_cluster=2997ae23-5631-49ad-8f0b-1dcf4e528e00&prometheus=openshift-monitoring%2Fk8s&severity=warning&alertname=LSODeviceLinkMismatch&container=kube-rbac-proxy&endpoint=metrics&instance=10.129.25.50%3A9393&job=local-storage-diskmaker-metrics&name=local-pv-d60306a3&namespace=openshift-local-storage&persistent_volume=local-pv-d60306a3&pod=diskmaker-manager-2pjlt&policy=None&service=local-storage-diskmaker-metrics)

and

LSONoStableLocalVolumePath

Local Storage operator has observed that pv local-pv-a35797a9 is using a symlink
that is not optimum for stability between reboots and OS upgrades. Consider updating
LocalVolumeDeviceLink policy to preferredSymlink to allow LSO to manage the symlink.
[View details](https://console-openshift-console.apps.isar.coe.muc.redhat.com/monitoring/alerts/3734418817?managed_cluster=2997ae23-5631-49ad-8f0b-1dcf4e528e00&prometheus=openshift-monitoring%2Fk8s&severity=warning&alertname=LSODeviceLinkMismatch&container=kube-rbac-proxy&endpoint=metrics&instance=10.131.24.163%3A9393&job=local-storage-diskmaker-metrics&name=local-pv-a35797a9&namespace=openshift-local-storage&persistent_volume=local-pv-a35797a9&pod=diskmaker-manager-krqnc&policy=None&service=local-storage-diskmaker-metrics)

We probably need to review our LSO config

[DanielFroehlich]

Last time I touched LSO config I severly messed ODF up @rbo can you handle this one please?

[rbo]

Puh, looks like that's a new alert. Nothing found about LSODeviceLinkMismatch in access.redhat.com.

[rbo]

Let's analyze

Local Storage operator has observed that pv local-pv-a35797a9 is using a symlink

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: local-volume-provisioner-storm6
    storage.openshift.com/device-name: sdt
  creationTimestamp: "2026-05-24T17:30:34Z"
  finalizers:
  - storage.openshift.com/lso-symlink-deleter
  - kubernetes.io/pv-protection
  labels:
    kubernetes.io/hostname: storm6
    storage.openshift.com/owner-kind: LocalVolumeSet
    storage.openshift.com/owner-name: storm-nvme
    storage.openshift.com/owner-namespace: openshift-local-storage
  name: local-pv-a35797a9
  resourceVersion: "4099548501"
  uid: 5ce8e040-02c4-4561-95f8-627c13726615
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 120Gi
  local:
    path: /mnt/local-storage/storm-nvme/sdt
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - storm6
  persistentVolumeReclaimPolicy: Delete
  storageClassName: storm-nvme
  volumeMode: Block
status:
  lastPhaseTransitionTime: "2026-05-24T17:30:34Z"
  phase: Available

The PV is not in use:

%oc get pvc -A | grep local-pv-a35797a9
%

Take a look at the node:

% oc debug node/storm6
Starting pod/storm6-debug-7h5wn ...
To use host binaries, run `chroot /host`. Instead, if you need to access host namespaces, run `nsenter -a -t 1`.
Pod IP: 10.32.105.6
If you don't see a command prompt, try pressing enter.
sh-5.1# chroot /host
sh-5.1# ls -la /mnt/local-storage/storm-nvme/sdt
lrwxrwxrwx. 1 root root 8 May 24 17:30 /mnt/local-storage/storm-nvme/sdt -> /dev/sdt
sh-5.1# ls -la /dev/sdt
ls: cannot access '/dev/sdt': No such file or directory
sh-5.1# find /mnt/
/mnt/
sh-5.1#

🤔

[rbo]

% oc get pv | grep ^local
local-pv-188a319                           1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-2-data-4jz2sc                                  storm-nvme                                   <unset>                          180d
local-pv-21219d59                          1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-0-data-24cn9t                                  storm-nvme                                   <unset>                          180d
local-pv-439319f6                          505Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          180d
local-pv-44cecd25                          1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-0-data-1wshg2                                  storm-nvme                                   <unset>                          190d
local-pv-5635b501                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          43d
local-pv-58b7d090                          1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-2-data-3nkm85                                  storm-nvme                                   <unset>                          180d
local-pv-5cfdb058                          1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-2-data-0qktwm                                  storm-nvme                                   <unset>                          190d
local-pv-69226f55                          1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-1-data-1r5lv8                                  storm-nvme                                   <unset>                          190d
local-pv-7eeaa6e5                          1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-2-data-1qrbmr                                  storm-nvme                                   <unset>                          190d
local-pv-97dcd0b0                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          180d
local-pv-9d084e0d                          1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-1-data-2bxpmw                                  storm-nvme                                   <unset>                          180d
local-pv-a35797a9                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          44d
local-pv-a81b0e59                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          180d
local-pv-b990b6f7                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          96d
local-pv-c79cbd7d                          1Gi          RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          111d
local-pv-c7edc3f6                          50Gi         RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          89d
local-pv-c870926                           1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-0-data-3cpmtf                                  storm-nvme                                   <unset>                          180d
local-pv-cc46c9ca                          1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-1-data-0ctcwz                                  storm-nvme                                   <unset>                          190d
local-pv-d60306a3                          100Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          10h
local-pv-daf6a31c                          1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-0-data-0pzklx                                  storm-nvme                                   <unset>                          190d
local-pv-e0fdc20c                          1Gi          RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          111d
local-pv-e2f04032                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          180d
local-pv-e30e647b                          10Gi         RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          132d
local-pv-fdd90b12                          1863Gi       RWO            Delete           Bound       openshift-storage/ocs-deviceset-storm-nvme-nvme-1-data-3lrvpc                                  storm-nvme                                   <unset>                          180d

The size of the PVs is strange. Looks like local volume found some LVMs disks?!

[rbo]

% oc get pv | grep ^local | grep  Available
local-pv-439319f6                          505Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          180d
local-pv-5635b501                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          43d
local-pv-97dcd0b0                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          180d
local-pv-a35797a9                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          44d
local-pv-a81b0e59                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          180d
local-pv-b990b6f7                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          96d
local-pv-c79cbd7d                          1Gi          RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          111d
local-pv-c7edc3f6                          50Gi         RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          89d
local-pv-d60306a3                          100Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          10h
local-pv-e0fdc20c                          1Gi          RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          111d
local-pv-e2f04032                          120Gi        RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          180d
local-pv-e30e647b                          10Gi         RWO            Delete           Available                                                                                                  storm-nvme                                   <unset>                          132d

Let's check all PV's and delete the PV if disk is not available.

• local-pv-439319f6, 505Gi, storm6
• local-pv-5635b501, 120Gi, storm6
• local-pv-97dcd0b0, 120Gi, storm6
• local-pv-a35797a9, 120Gi, storm6
• local-pv-a81b0e59, 120Gi, storm6
• local-pv-b990b6f7, 120Gi, storm5
• local-pv-c79cbd7d, 1Gi , storm5
• local-pv-c7edc3f6, 50Gi , storm5
• local-pv-d60306a3, 100Gi, storm3
• local-pv-e0fdc20c, 1Gi , storm5
• local-pv-e2f04032, 120Gi, storm6
• local-pv-e30e647b, 10Gi , storm5

[rbo]

local-pv-439319f6

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: local-volume-provisioner-storm6
    storage.openshift.com/device-name: sde
  creationTimestamp: "2026-01-08T15:24:47Z"
  finalizers:
  - storage.openshift.com/lso-symlink-deleter
  - kubernetes.io/pv-protection
  labels:
    kubernetes.io/hostname: storm6
    storage.openshift.com/owner-kind: LocalVolumeSet
    storage.openshift.com/owner-name: storm-nvme
    storage.openshift.com/owner-namespace: openshift-local-storage
  name: local-pv-439319f6
  resourceVersion: "3545412731"
  uid: 0a7020bb-014f-4d26-babf-5e1d5a6760f1
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 505Gi
  local:
    path: /mnt/local-storage/storm-nvme/sde
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - storm6
  persistentVolumeReclaimPolicy: Delete
  storageClassName: storm-nvme
  volumeMode: Block
status:
  lastPhaseTransitionTime: "2026-01-08T15:24:47Z"
  phase: Available

sh-5.1# hostname
storm6
sh-5.1# ls -la /mnt/local-storage/storm-nvme/sde
lrwxrwxrwx. 1 root root 8 Jan  8 15:24 /mnt/local-storage/storm-nvme/sde -> /dev/sde
sh-5.1# ls -la /dev/sde
brw-rw----. 1 root disk 8, 64 Jul  7 11:57 /dev/sde
sh-5.1# lsblk | grep sde
sde                                                         8:64   0   120G  0 disk
sh-5.1#
sh-5.1# lsblk  | grep sde
sde                                                         8:64   0   120G  0 disk
sh-5.1# dmesg | grep sde
[11487.322386] sd 11:0:0:1: [sde] 251658240 512-byte logical blocks: (129 GB/120 GiB)
[11487.322389] sd 11:0:0:1: [sde] 4096-byte physical blocks
[11487.322551] sd 11:0:0:1: [sde] Write Protect is off
[11487.322553] sd 11:0:0:1: [sde] Mode Sense: aa 00 00 08
[11487.323117] sd 11:0:0:1: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[11487.323681] sd 11:0:0:1: [sde] Preferred minimum I/O size 4096 bytes
[11487.323683] sd 11:0:0:1: [sde] Optimal transfer size 65536 bytes
[11487.379731]  sde: sde1 sde2 sde3 sde4
[11487.380095] sd 11:0:0:1: [sde] Attached SCSI disk
sh-5.1#
sh-5.1# find /dev/disk/ -ls | grep sde
     7094      0 lrwxrwxrwx   1 root     root            9 Jul  7 11:57 /dev/disk/by-diskseq/81 -> ../../sde
     7092      0 lrwxrwxrwx   1 root     root            9 Jul  7 11:57 /dev/disk/by-path/ip-10.32.97.31:3260-iscsi-iqn.1992-08.com.netapp:sn.c75b807ffc3c11ec829000a0987cd31a:vs.16-lun-1 -> ../../sde
sh-5.1#

=> That's a local iSCSI disk from the NetApp!

Why is the localvolumeset detecting "remote" disks?

[rbo]

Let's be more specific in LocalVolumeSet and allow only the specific NVME models:

storm6

sh-5.1# lsblk -o NAME,VENDOR,MODEL,ROTA,SIZE | grep ^nvme
nvme0n1                                                            INTEL SSDPEDMD020T4                      0   1.8T
nvme1n1                                                            INTEL SSDPEDMD020T4D HHHL NVMe 2000GB    0   1.8T
nvme2n1                                                            INTEL SSDPEDMD020T4D HHHL NVMe 2000GB    0   1.8T
nvme3n1                                                            INTEL SSDPEDMD020T4D HHHL NVMe 2000GB    0   1.8T
sh-5.1#

storm3

sh-5.1# lsblk -o NAME,VENDOR,MODEL,ROTA,SIZE | grep ^nvme
nvme1n1                                          INTEL SSDPEDMD020T4                      0   1.8T
nvme2n1                                          INTEL SSDPEDMD020T4D HHHL NVMe 2000GB    0   1.8T
nvme0n1                                          INTEL SSDPEDMD020T4D HHHL NVMe 2000GB    0   1.8T
nvme3n1                                          INTEL SSDPEDMD020T4                      0   1.8T
sh-5.1#

storm5

sh-5.1# lsblk -o NAME,VENDOR,MODEL,ROTA,SIZE | grep ^nvme
nvme2n1                                                            INTEL SSDPEDMD020T4D HHHL NVMe 2000GB    0   1.8T
nvme3n1                                                            INTEL SSDPEDMD020T4D HHHL NVMe 2000GB    0   1.8T
nvme0n1                                                            INTEL SSDPEDMD020T4D HHHL NVMe 2000GB    0   1.8T
nvme1n1                                                            INTEL SSDPEDMD020T4D HHHL NVMe 2000GB    0   1.8T
sh-5.1#

[rbo]

Added to LocalVolumeSet:

  deviceInclusionSpec:
    models:
    - "INTEL SSDPEDMD020T4"
    - "INTEL SSDPEDMD020T4D HHHL NVMe 2000GB"

apiVersion: local.storage.openshift.io/v1alpha1
kind: LocalVolumeSet
metadata:
  creationTimestamp: "2025-12-29T13:26:28Z"
  finalizers:
  - storage.openshift.com/local-volume-protection
  generation: 2
  name: storm-nvme
  namespace: openshift-local-storage
  resourceVersion: "4289571376"
  uid: fba7bcb6-d484-4d4a-a4d5-c22277991028
spec:
  deviceInclusionSpec:
    deviceMechanicalProperties:
    - NonRotational
    deviceTypes:
    - disk
    minSize: 1Gi
    models:
    - INTEL SSDPEDMD020T4
    - INTEL SSDPEDMD020T4D HHHL NVMe 2000GB
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: kubernetes.io/hostname
        operator: In
        values:
        - storm3
        - storm5
        - storm6
  storageClassName: storm-nvme
  tolerations:
  - effect: NoSchedule
    key: node.ocs.openshift.io/storage
    operator: Equal
    value: "true"
  volumeMode: Block
status:
  conditions:
  - lastTransitionTime: "2025-12-29T13:26:28Z"
    message: 'DiskMaker: Available'
    status: "True"
    type: DaemonSetsAvailable
  - lastTransitionTime: "2025-12-29T13:26:28Z"
    message: Operator reconciled successfully.
    status: "True"
    type: Available
  observedGeneration: 2
  totalProvisionedDeviceCount: 24

[rbo]

Deleted all local pv's there are available:

% oc get pv | grep ^local | grep  Available
%

[rbo]

Only the real NVME's are left:

% oc get pv -L kubernetes.io/hostname -l storage.openshift.com/owner-name=storm-nvme
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                           STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE    HOSTNAME
local-pv-188a319    1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-2-data-4jz2sc   storm-nvme     <unset>                          180d   storm3
local-pv-21219d59   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-0-data-24cn9t   storm-nvme     <unset>                          180d   storm6
local-pv-44cecd25   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-0-data-1wshg2   storm-nvme     <unset>                          190d   storm3
local-pv-58b7d090   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-2-data-3nkm85   storm-nvme     <unset>                          180d   storm5
local-pv-5cfdb058   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-2-data-0qktwm   storm-nvme     <unset>                          190d   storm5
local-pv-69226f55   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-1-data-1r5lv8   storm-nvme     <unset>                          190d   storm6
local-pv-7eeaa6e5   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-2-data-1qrbmr   storm-nvme     <unset>                          190d   storm6
local-pv-9d084e0d   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-1-data-2bxpmw   storm-nvme     <unset>                          180d   storm5
local-pv-c870926    1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-0-data-3cpmtf   storm-nvme     <unset>                          180d   storm3
local-pv-cc46c9ca   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-1-data-0ctcwz   storm-nvme     <unset>                          190d   storm3
local-pv-daf6a31c   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-0-data-0pzklx   storm-nvme     <unset>                          190d   storm5
local-pv-fdd90b12   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-storm-nvme-nvme-1-data-3lrvpc   storm-nvme     <unset>                          180d   storm6

[rbo]

Let's double check tomorrow if the alerts are gone

[rbo]

Alerts still exist.