hpe-csi-driver crash - threading bug?

[megakid]

Had this occur a few times (maybe since we enabled trace logging to track down another issue but cannot be sure):

This is v2.4.1 with trace logging enabled:

> kubectl logs hpe-csi-controller-789559697d-hvfxs hpe-csi-driver --previous 
...
time="2024-03-07T14:03:16Z" level=debug msg="About to decode the error response &{0x5033c0 0xc001507040 0x863940} into destination interface" file="client.go:231"
time="2024-03-07T14:03:16Z" level=trace msg="Found Volume &{ID:0654316e079d129ca30000000000000000000cdb7c Name:pvc-89832d9c-90f9-46b4-83dc-db503d924398 Size:107374182400 Description:investmentcontrol for ci (sql-eq-ci-tca-042-19295746-investmentcontrol) InUse:false Published:false BaseSnapID:0454316e079d129ca300000000000e3a6e00070f65 ParentVolID: Clone:false Config:map[clone_of:pvc-95661875-d0ee-4d79-8005-1be4479875a2 create_snapshot:false dedupe_enabled:true destroy_on_delete:true encrypted:true folder:k8s-onprem-tnd-investment limit_iops:-1 limit_mbps:-1 performance_policy:default pool:default sync_on_detach:false target_scope:volume thick:false] Metadata:[] SerialNumber: AccessProtocol: Iqn: Iqns:[] DiscoveryIP: DiscoveryIPs:[] MountPoint: Status:map[] Chap:<nil> Networks:[] ConnectionMode: LunID: TargetScope: IscsiSessions:[] FcSessions:[] VolumeGroupId: SecondaryArrayDetails: UsedBytes:0 FreeBytes:0 EncryptionKey:}" file="driver.go:457"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< GetVolume" file="driver.go:458"
time="2024-03-07T14:03:16Z" level=trace msg="Found Volume pvc-89832d9c-90f9-46b4-83dc-db503d924398 with ID 0654316e079d129ca30000000000000000000cdb7c" file="controller_server.go:654"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> GetStorageProvider" file="driver.go:348"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> CreateCredentials" file="storage_provider.go:65"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< CreateCredentials" file="storage_provider.go:125"
time="2024-03-07T14:03:16Z" level=trace msg="Storage provider already exists. Returning it." file="driver.go:368"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< GetStorageProvider" file="driver.go:369"
time="2024-03-07T14:03:16Z" level=info msg="About to delete volume 0654316e079d129ca30000000000000000000cdb7c with force=false" file="controller_server.go:673"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> DeleteVolume, id: 0654316e079d129ca30000000000000000000cdb7c, force: false" file="container_storage_provider.go:484"
time="2024-03-07T14:03:16Z" level=trace msg="About to invoke CSP request for backend 10.2.10.160" file="container_storage_provider.go:153"
time="2024-03-07T14:03:16Z" level=trace msg="Header: {x-auth-token : *****}\n" file="client.go:166"
time="2024-03-07T14:03:16Z" level=trace msg="Header: {x-array-ip : 10.2.10.160}\n" file="client.go:168"
time="2024-03-07T14:03:16Z" level=trace msg="Request: action=DELETE path=http://nimble-csp-svc:8080/containers/v1/volumes/0654316e079d129ca30000000000000000000cdb7c?force=false" file="client.go:173"
time="2024-03-07T14:03:16Z" level=info msg="GRPC call: /csi.v1.Controller/DeleteVolume" file="utils.go:69"
time="2024-03-07T14:03:16Z" level=info msg="GRPC request: {\"secrets\":\"***stripped***\",\"volume_id\":\"0654316e079d129ca30000000000000000000cdb7b\"}" file="utils.go:70"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> DeleteVolume" file="controller_server.go:608"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> HandleDuplicateRequest, key: DeleteVolume:0654316e079d129ca30000000000000000000cdb7b" file="driver.go:510"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> GetRequest, key: DeleteVolume:0654316e079d129ca30000000000000000000cdb7b" file="driver.go:567"
time="2024-03-07T14:03:16Z" level=trace msg="Acquiring mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7b" file="concurrent.go:44"
time="2024-03-07T14:03:16Z" level=trace msg="Releasing mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7b" file="concurrent.go:67"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< GetRequest" file="driver.go:574"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> AddRequest, key: DeleteVolume:0654316e079d129ca30000000000000000000cdb7b, value: true" file="driver.go:579"
time="2024-03-07T14:03:16Z" level=trace msg="Acquiring mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7b" file="concurrent.go:44"
time="2024-03-07T14:03:16Z" level=trace msg="Print RequestCache: map[ControllerUnpublishVolume:0654316e079d129ca30000000000000000000cdb7a:b2407086-6b79-f702-1653-402998f6db25:true DeleteVolume:0654316e079d129ca30000000000000000000cdb7b:true DeleteVolume:0654316e079d129ca30000000000000000000cdb7c:true]" file="driver.go:587"
time="2024-03-07T14:03:16Z" level=trace msg="Successfully inserted an entry with key DeleteVolume:0654316e079d129ca30000000000000000000cdb7b to the cache map" file="driver.go:588"
time="2024-03-07T14:03:16Z" level=trace msg="Releasing mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7b" file="concurrent.go:67"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< AddRequest" file="driver.go:589"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< HandleDuplicateRequest" file="driver.go:527"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> deleteVolume, volumeID: 0654316e079d129ca30000000000000000000cdb7b, force: false" file="controller_server.go:631"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> getPVByNFSLabel with key nfs-parent-volume-id value 0654316e079d129ca30000000000000000000cdb7b" file="nfs.go:535"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< getPVByNFSLabel" file="nfs.go:549"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> GetVolume, ID: 0654316e079d129ca30000000000000000000cdb7b" file="driver.go:419"
time="2024-03-07T14:03:16Z" level=trace msg="Secrets are provided. Checking with this particular storage provider." file="driver.go:426"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> GetStorageProvider" file="driver.go:348"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> CreateCredentials" file="storage_provider.go:65"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< CreateCredentials" file="storage_provider.go:125"
time="2024-03-07T14:03:16Z" level=trace msg="Storage provider already exists. Returning it." file="driver.go:368"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< GetStorageProvider" file="driver.go:369"
time="2024-03-07T14:03:16Z" level=trace msg="About to invoke CSP request for backend 10.2.10.160" file="container_storage_provider.go:153"
time="2024-03-07T14:03:16Z" level=trace msg="Header: {x-auth-token : *****}\n" file="client.go:166"
time="2024-03-07T14:03:16Z" level=trace msg="Header: {x-array-ip : 10.2.10.160}\n" file="client.go:168"
time="2024-03-07T14:03:16Z" level=trace msg="Request: action=GET path=http://nimble-csp-svc:8080/containers/v1/volumes/0654316e079d129ca30000000000000000000cdb7b" file="client.go:173"
time="2024-03-07T14:03:16Z" level=trace msg="response: 204 No Content, length=0" file="client.go:224"
time="2024-03-07T14:03:16Z" level=debug msg="Received a null reader. That is not expected." file="client.go:245"
time="2024-03-07T14:03:16Z" level=trace msg="Replying with status code: 204" file="container_storage_provider.go:180"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< controllerUnpublishVolume" file="controller_server.go:1032"
time="2024-03-07T14:03:16Z" level=trace msg=">>>>> ClearRequest, key: ControllerUnpublishVolume:0654316e079d129ca30000000000000000000cdb7a:b2407086-6b79-f702-1653-402998f6db25" file="driver.go:593"
time="2024-03-07T14:03:16Z" level=trace msg="Acquiring mutex lock for ControllerUnpublishVolume:0654316e079d129ca30000000000000000000cdb7a:b2407086-6b79-f702-1653-402998f6db25" file="concurrent.go:44"
time="2024-03-07T14:03:16Z" level=trace msg="Print RequestCache: map[DeleteVolume:0654316e079d129ca30000000000000000000cdb7b:true DeleteVolume:0654316e079d129ca30000000000000000000cdb7c:true]" file="driver.go:613"
time="2024-03-07T14:03:16Z" level=trace msg="Successfully removed an entry with key ControllerUnpublishVolume:0654316e079d129ca30000000000000000000cdb7a:b2407086-6b79-f702-1653-402998f6db25 from the cache map" file="driver.go:614"
time="2024-03-07T14:03:16Z" level=trace msg="Releasing mutex lock for ControllerUnpublishVolume:0654316e079d129ca30000000000000000000cdb7a:b2407086-6b79-f702-1653-402998f6db25" file="concurrent.go:67"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< ClearRequest" file="driver.go:615"
time="2024-03-07T14:03:16Z" level=trace msg="<<<<< ControllerUnpublishVolume" file="controller_server.go:980"
time="2024-03-07T14:03:16Z" level=info msg="GRPC response: {}" file="utils.go:75"
time="2024-03-07T14:03:17Z" level=info msg="GRPC call: /csi.v1.Controller/DeleteVolume" file="utils.go:69"
time="2024-03-07T14:03:17Z" level=info msg="GRPC request: {\"secrets\":\"***stripped***\",\"volume_id\":\"0654316e079d129ca30000000000000000000cdb7a\"}" file="utils.go:70"
time="2024-03-07T14:03:17Z" level=trace msg=">>>>> DeleteVolume" file="controller_server.go:608"
time="2024-03-07T14:03:17Z" level=trace msg=">>>>> HandleDuplicateRequest, key: DeleteVolume:0654316e079d129ca30000000000000000000cdb7a" file="driver.go:510"
time="2024-03-07T14:03:17Z" level=trace msg=">>>>> GetRequest, key: DeleteVolume:0654316e079d129ca30000000000000000000cdb7a" file="driver.go:567"
time="2024-03-07T14:03:17Z" level=trace msg="Acquiring mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7a" file="concurrent.go:44"
time="2024-03-07T14:03:17Z" level=trace msg="Releasing mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7a" file="concurrent.go:67"
time="2024-03-07T14:03:17Z" level=trace msg="<<<<< GetRequest" file="driver.go:574"
time="2024-03-07T14:03:17Z" level=trace msg=">>>>> AddRequest, key: DeleteVolume:0654316e079d129ca30000000000000000000cdb7a, value: true" file="driver.go:579"
time="2024-03-07T14:03:17Z" level=trace msg="Acquiring mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7a" file="concurrent.go:44"
time="2024-03-07T14:03:17Z" level=trace msg="Print RequestCache: map[DeleteVolume:0654316e079d129ca30000000000000000000cdb7a:true DeleteVolume:0654316e079d129ca30000000000000000000cdb7b:true DeleteVolume:0654316e079d129ca30000000000000000000cdb7c:true]" file="driver.go:587"
time="2024-03-07T14:03:17Z" level=trace msg="Successfully inserted an entry with key DeleteVolume:0654316e079d129ca30000000000000000000cdb7a to the cache map" file="driver.go:588"
time="2024-03-07T14:03:17Z" level=trace msg="Releasing mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7a" file="concurrent.go:67"
time="2024-03-07T14:03:17Z" level=trace msg="<<<<< AddRequest" file="driver.go:589"
time="2024-03-07T14:03:17Z" level=trace msg="<<<<< HandleDuplicateRequest" file="driver.go:527"
time="2024-03-07T14:03:17Z" level=trace msg=">>>>> deleteVolume, volumeID: 0654316e079d129ca30000000000000000000cdb7a, force: false" file="controller_server.go:631"
time="2024-03-07T14:03:17Z" level=trace msg=">>>>> getPVByNFSLabel with key nfs-parent-volume-id value 0654316e079d129ca30000000000000000000cdb7a" file="nfs.go:535"
time="2024-03-07T14:03:17Z" level=trace msg="<<<<< getPVByNFSLabel" file="nfs.go:549"
time="2024-03-07T14:03:17Z" level=trace msg=">>>>> GetVolume, ID: 0654316e079d129ca30000000000000000000cdb7a" file="driver.go:419"
time="2024-03-07T14:03:17Z" level=trace msg="Secrets are provided. Checking with this particular storage provider." file="driver.go:426"
time="2024-03-07T14:03:17Z" level=trace msg=">>>>> GetStorageProvider" file="driver.go:348"
time="2024-03-07T14:03:17Z" level=trace msg=">>>>> CreateCredentials" file="storage_provider.go:65"
time="2024-03-07T14:03:17Z" level=trace msg="<<<<< CreateCredentials" file="storage_provider.go:125"
time="2024-03-07T14:03:17Z" level=trace msg="Storage provider already exists. Returning it." file="driver.go:368"
time="2024-03-07T14:03:17Z" level=trace msg="<<<<< GetStorageProvider" file="driver.go:369"
time="2024-03-07T14:03:17Z" level=trace msg="About to invoke CSP request for backend 10.2.10.160" file="container_storage_provider.go:153"
time="2024-03-07T14:03:17Z" level=trace msg="Header: {x-auth-token : *****}\n" file="client.go:166"
time="2024-03-07T14:03:17Z" level=trace msg="Header: {x-array-ip : 10.2.10.160}\n" file="client.go:168"
time="2024-03-07T14:03:17Z" level=trace msg="Request: action=GET path=http://nimble-csp-svc:8080/containers/v1/volumes/0654316e079d129ca30000000000000000000cdb7a" file="client.go:173"
time="2024-03-07T14:03:18Z" level=trace msg="response: 200 OK, length=616" file="client.go:224"
time="2024-03-07T14:03:18Z" level=debug msg="About to decode the error response &{0x5033c0 0xc0015076c0 0x863940} into destination interface" file="client.go:231"
time="2024-03-07T14:03:18Z" level=trace msg="Found Volume &{ID:0654316e079d129ca30000000000000000000cdb7b Name:pvc-5ac1ba6b-c123-42bb-a92a-b9b80c7171a5 Size:9895604649984 Description:pnl for ci (sql-eq-ci-tca-042-19295746-pnl) InUse:false Published:false BaseSnapID:0454316e079d129ca300000000000e3a7b00070f6f ParentVolID: Clone:false Config:map[clone_of:pvc-2fce87ae-9f9c-4b01-997b-a1a12e4ede73 create_snapshot:false dedupe_enabled:true destroy_on_delete:true encrypted:true folder:k8s-onprem-tnd-investment limit_iops:-1 limit_mbps:-1 performance_policy:default pool:default sync_on_detach:false target_scope:volume thick:false] Metadata:[] SerialNumber: AccessProtocol: Iqn: Iqns:[] DiscoveryIP: DiscoveryIPs:[] MountPoint: Status:map[] Chap:<nil> Networks:[] ConnectionMode: LunID: TargetScope: IscsiSessions:[] FcSessions:[] VolumeGroupId: SecondaryArrayDetails: UsedBytes:0 FreeBytes:0 EncryptionKey:}" file="driver.go:457"
time="2024-03-07T14:03:18Z" level=trace msg="<<<<< GetVolume" file="driver.go:458"
time="2024-03-07T14:03:18Z" level=trace msg="Found Volume pvc-5ac1ba6b-c123-42bb-a92a-b9b80c7171a5 with ID 0654316e079d129ca30000000000000000000cdb7b" file="controller_server.go:654"
time="2024-03-07T14:03:18Z" level=trace msg=">>>>> GetStorageProvider" file="driver.go:348"
time="2024-03-07T14:03:18Z" level=trace msg=">>>>> CreateCredentials" file="storage_provider.go:65"
time="2024-03-07T14:03:18Z" level=trace msg="<<<<< CreateCredentials" file="storage_provider.go:125"
time="2024-03-07T14:03:18Z" level=trace msg="Storage provider already exists. Returning it." file="driver.go:368"
time="2024-03-07T14:03:18Z" level=trace msg="<<<<< GetStorageProvider" file="driver.go:369"
time="2024-03-07T14:03:18Z" level=info msg="About to delete volume 0654316e079d129ca30000000000000000000cdb7b with force=false" file="controller_server.go:673"
time="2024-03-07T14:03:18Z" level=trace msg=">>>>> DeleteVolume, id: 0654316e079d129ca30000000000000000000cdb7b, force: false" file="container_storage_provider.go:484"
time="2024-03-07T14:03:18Z" level=trace msg="About to invoke CSP request for backend 10.2.10.160" file="container_storage_provider.go:153"
time="2024-03-07T14:03:18Z" level=trace msg="Header: {x-auth-token : *****}\n" file="client.go:166"
time="2024-03-07T14:03:18Z" level=trace msg="Header: {x-array-ip : 10.2.10.160}\n" file="client.go:168"
time="2024-03-07T14:03:18Z" level=trace msg="Request: action=DELETE path=http://nimble-csp-svc:8080/containers/v1/volumes/0654316e079d129ca30000000000000000000cdb7b?force=false" file="client.go:173"
time="2024-03-07T14:03:18Z" level=trace msg="response: 200 OK, length=628" file="client.go:224"
time="2024-03-07T14:03:18Z" level=debug msg="About to decode the error response &{0x5033c0 0xc00138e3c0 0x863940} into destination interface" file="client.go:231"
time="2024-03-07T14:03:18Z" level=trace msg="Found Volume &{ID:0654316e079d129ca30000000000000000000cdb7a Name:pvc-0bed9987-72ec-49c6-9038-eeb1b33523a1 Size:1099511627776 Description:positions for ci (sql-eq-ci-tca-042-19295746-positions) InUse:false Published:false BaseSnapID:0454316e079d129ca300000000000e3a6800070f59 ParentVolID: Clone:false Config:map[clone_of:pvc-83b0e1da-245e-4bf6-967d-a8a758bbc364 create_snapshot:false dedupe_enabled:true destroy_on_delete:true encrypted:true folder:k8s-onprem-tnd-investment limit_iops:-1 limit_mbps:-1 performance_policy:default pool:default sync_on_detach:false target_scope:volume thick:false] Metadata:[] SerialNumber: AccessProtocol: Iqn: Iqns:[] DiscoveryIP: DiscoveryIPs:[] MountPoint: Status:map[] Chap:<nil> Networks:[] ConnectionMode: LunID: TargetScope: IscsiSessions:[] FcSessions:[] VolumeGroupId: SecondaryArrayDetails: UsedBytes:0 FreeBytes:0 EncryptionKey:}" file="driver.go:457"
time="2024-03-07T14:03:18Z" level=trace msg="<<<<< GetVolume" file="driver.go:458"
time="2024-03-07T14:03:18Z" level=trace msg="Found Volume pvc-0bed9987-72ec-49c6-9038-eeb1b33523a1 with ID 0654316e079d129ca30000000000000000000cdb7a" file="controller_server.go:654"
time="2024-03-07T14:03:18Z" level=trace msg=">>>>> GetStorageProvider" file="driver.go:348"
time="2024-03-07T14:03:18Z" level=trace msg=">>>>> CreateCredentials" file="storage_provider.go:65"
time="2024-03-07T14:03:18Z" level=trace msg="<<<<< CreateCredentials" file="storage_provider.go:125"
time="2024-03-07T14:03:18Z" level=trace msg="Storage provider already exists. Returning it." file="driver.go:368"
time="2024-03-07T14:03:18Z" level=trace msg="<<<<< GetStorageProvider" file="driver.go:369"
time="2024-03-07T14:03:18Z" level=info msg="About to delete volume 0654316e079d129ca30000000000000000000cdb7a with force=false" file="controller_server.go:673"
time="2024-03-07T14:03:18Z" level=trace msg=">>>>> DeleteVolume, id: 0654316e079d129ca30000000000000000000cdb7a, force: false" file="container_storage_provider.go:484"
time="2024-03-07T14:03:18Z" level=trace msg="About to invoke CSP request for backend 10.2.10.160" file="container_storage_provider.go:153"
time="2024-03-07T14:03:18Z" level=trace msg="Header: {x-auth-token : *****}\n" file="client.go:166"
time="2024-03-07T14:03:18Z" level=trace msg="Header: {x-array-ip : 10.2.10.160}\n" file="client.go:168"
time="2024-03-07T14:03:18Z" level=trace msg="Request: action=DELETE path=http://nimble-csp-svc:8080/containers/v1/volumes/0654316e079d129ca30000000000000000000cdb7a?force=false" file="client.go:173"
time="2024-03-07T14:03:18Z" level=trace msg="response: 204 No Content, length=0" file="client.go:224"
time="2024-03-07T14:03:18Z" level=debug msg="Received a null reader. That is not expected." file="client.go:245"
time="2024-03-07T14:03:18Z" level=trace msg="Replying with status code: 204" file="container_storage_provider.go:180"
time="2024-03-07T14:03:18Z" level=trace msg="<<<<< DeleteVolume" file="container_storage_provider.go:503"
time="2024-03-07T14:03:18Z" level=trace msg="DB service disabled" file="driver.go:674"
time="2024-03-07T14:03:18Z" level=trace msg="<<<<< deleteVolume" file="controller_server.go:684"
time="2024-03-07T14:03:18Z" level=trace msg=">>>>> ClearRequest, key: DeleteVolume:0654316e079d129ca30000000000000000000cdb7c" file="driver.go:593"
time="2024-03-07T14:03:18Z" level=trace msg="Acquiring mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7c" file="concurrent.go:44"
time="2024-03-07T14:03:18Z" level=trace msg="Print RequestCache: map[DeleteVolume:0654316e079d129ca30000000000000000000cdb7a:true DeleteVolume:0654316e079d129ca30000000000000000000cdb7b:true]" file="driver.go:613"
time="2024-03-07T14:03:18Z" level=trace msg="Successfully removed an entry with key DeleteVolume:0654316e079d129ca30000000000000000000cdb7c from the cache map" file="driver.go:614"
time="2024-03-07T14:03:18Z" level=trace msg="Releasing mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7c" file="concurrent.go:67"
time="2024-03-07T14:03:18Z" level=trace msg="<<<<< ClearRequest" file="driver.go:615"
time="2024-03-07T14:03:18Z" level=trace msg="<<<<< DeleteVolume" file="controller_server.go:627"
time="2024-03-07T14:03:18Z" level=info msg="GRPC response: {}" file="utils.go:75"
time="2024-03-07T14:03:19Z" level=trace msg="response: 204 No Content, length=0" file="client.go:224"
time="2024-03-07T14:03:19Z" level=debug msg="Received a null reader. That is not expected." file="client.go:245"
time="2024-03-07T14:03:19Z" level=trace msg="response: 204 No Content, length=0" file="client.go:224"
time="2024-03-07T14:03:19Z" level=trace msg="Replying with status code: 204" file="container_storage_provider.go:180"
time="2024-03-07T14:03:19Z" level=debug msg="Received a null reader. That is not expected." file="client.go:245"
time="2024-03-07T14:03:19Z" level=trace msg="<<<<< DeleteVolume" file="container_storage_provider.go:503"
time="2024-03-07T14:03:19Z" level=trace msg="Replying with status code: 204" file="container_storage_provider.go:180"
time="2024-03-07T14:03:19Z" level=trace msg="DB service disabled" file="driver.go:674"
time="2024-03-07T14:03:19Z" level=trace msg="<<<<< DeleteVolume" file="container_storage_provider.go:503"
time="2024-03-07T14:03:19Z" level=trace msg="<<<<< deleteVolume" file="controller_server.go:684"
time="2024-03-07T14:03:19Z" level=trace msg="DB service disabled" file="driver.go:674"
time="2024-03-07T14:03:19Z" level=trace msg="<<<<< deleteVolume" file="controller_server.go:684"
time="2024-03-07T14:03:19Z" level=trace msg=">>>>> ClearRequest, key: DeleteVolume:0654316e079d129ca30000000000000000000cdb7b" file="driver.go:593"
time="2024-03-07T14:03:19Z" level=trace msg=">>>>> ClearRequest, key: DeleteVolume:0654316e079d129ca30000000000000000000cdb7a" file="driver.go:593"
time="2024-03-07T14:03:19Z" level=trace msg="Acquiring mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7a" file="concurrent.go:44"
time="2024-03-07T14:03:19Z" level=trace msg="Acquiring mutex lock for DeleteVolume:0654316e079d129ca30000000000000000000cdb7b" file="concurrent.go:44"
fatal error: concurrent map iteration and map write

goroutine 147304 [running]:
reflect.mapiternext(0x4a56ef?)
        /usr/lib/golang/src/runtime/map.go:1380 +0x19
reflect.(*MapIter).Next(0xc0006512c8?)
        /usr/lib/golang/src/reflect/value.go:1924 +0x7e
internal/fmtsort.Sort({0x19c7860?, 0xc000308ae0?, 0x110?})
        /usr/lib/golang/src/internal/fmtsort/sort.go:62 +0x1f0
fmt.(*pp).printValue(0xc0012c01a0, {0x19c7860?, 0xc000308ae0?, 0xc000256000?}, 0x76, 0x0)
        /usr/lib/golang/src/fmt/print.go:816 +0x986
fmt.(*pp).printArg(0xc0012c01a0, {0x19c7860?, 0xc000308ae0}, 0x76)
        /usr/lib/golang/src/fmt/print.go:759 +0x756
fmt.(*pp).doPrintf(0xc0012c01a0, {0x1c7885e, 0x16}, {0xc000651800?, 0x1, 0x1})
        /usr/lib/golang/src/fmt/print.go:1077 +0x387
fmt.Sprintf({0x1c7885e, 0x16}, {0xc000651800, 0x1, 0x1})
        /usr/lib/golang/src/fmt/print.go:239 +0x59
github.com/sirupsen/logrus.(*Entry).Logf(0xc0010609a0, 0x6, {0x1c7885e?, 0x19130a0?}, {0xc000651800?, 0x19130a0?, 0xc000f66b20?})
        /usr/src/hpe-csi-driver/vendor/github.com/sirupsen/logrus/entry.go:338 +0x49
github.com/sirupsen/logrus.(*Entry).Tracef(...)
        /usr/src/hpe-csi-driver/vendor/github.com/sirupsen/logrus/entry.go:343
github.com/hpe-storage/common-host-libs/logger.Tracef({0x1c7885e, 0x16}, {0xc000651800, 0x1, 0x1})
        /usr/src/hpe-csi-driver/vendor/github.com/hpe-storage/common-host-libs/logger/logger.go:556 +0x55
github.com/hpe-storage/csi-driver/pkg/driver.(*Driver).ClearRequest(0xc00026a300, {0xc000f36380, 0x37})
        /usr/src/hpe-csi-driver/pkg/driver/driver.go:613 +0x34f
github.com/hpe-storage/csi-driver/pkg/driver.(*Driver).DeleteVolume(0xc00026a300, {0x1c6ff37?, 0x40de67?}, 0xc0015071c0)
        /usr/src/hpe-csi-driver/pkg/driver/controller_server.go:627 +0x454
github.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1({0x1f04248, 0xc00155a930}, {0x1b4fb20?, 0xc0015071c0})
        /usr/src/hpe-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5696 +0x78
github.com/hpe-storage/csi-driver/pkg/driver.logGRPC({0x1f04248, 0xc00155a930}, {0x1b4fb20?, 0xc0015071c0?}, 0xc000d11ab8?, 0xc0006ecbe8)
        /usr/src/hpe-csi-driver/pkg/driver/utils.go:71 +0x127
github.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler({0x1c58d60?, 0xc00026a300}, {0x1f04248, 0xc00155a930}, 0xc00111a420, 0x1d6d1f0)
        /usr/src/hpe-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5698 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0006dc000, {0x1f0a5a0, 0xc000a60000}, 0xc000f94c00, 0xc0006d41e0, 0x2c9c5f8, 0x0)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:1024 +0xd5e
google.golang.org/grpc.(*Server).handleStream(0xc0006dc000, {0x1f0a5a0, 0xc000a60000}, 0xc000f94c00, 0x0)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:1313 +0xa25
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:722 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:720 +0xea

goroutine 1 [chan receive, 182 minutes]:
main.csiCliHandler(0x1c82d51?)
        /usr/src/hpe-csi-driver/cmd/csi-driver/csi-driver.go:173 +0x767
main.glob..func1(0x2c9e620?, {0xc000252440, 0x0, 0x4})
        /usr/src/hpe-csi-driver/cmd/csi-driver/csi-driver.go:61 +0x1c5
github.com/spf13/cobra.(*Command).execute(0x2c9e620, {0xc0000500b0, 0x4, 0x4})
        /usr/src/hpe-csi-driver/vendor/github.com/spf13/cobra/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x2c9e620)
        /usr/src/hpe-csi-driver/vendor/github.com/spf13/cobra/command.go:1040 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /usr/src/hpe-csi-driver/vendor/github.com/spf13/cobra/command.go:968
main.Main()
        /usr/src/hpe-csi-driver/cmd/csi-driver/csi-driver.go:184 +0x25
main.main()
        /usr/src/hpe-csi-driver/cmd/csi-driver/csi-driver.go:192 +0x17

goroutine 135 [chan receive, 2 minutes]:
gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun(0xc00026c120)
        /usr/src/hpe-csi-driver/vendor/gopkg.in/natefinch/lumberjack.v2/lumberjack.go:379 +0x45
created by gopkg.in/natefinch/lumberjack%2ev2.(*Logger).mill.func1
        /usr/src/hpe-csi-driver/vendor/gopkg.in/natefinch/lumberjack.v2/lumberjack.go:390 +0x96

goroutine 136 [chan receive, 182 minutes]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc000528c30)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116

goroutine 137 [chan receive, 182 minutes]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/record/event.go:307 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/record/event.go:305 +0x13d

goroutine 138 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0003f9578, 0x283d)
        /usr/lib/golang/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc00042a9c0?)
        /usr/lib/golang/src/sync/cond.go:70 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0003f9550, 0xc000382140)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:571 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0006b01b0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:184 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xcc1365?, {0x1eec0a0, 0xc00069b350}, 0x1, 0xc000334540)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006b0218?, 0x3b9aca00, 0x0, 0x0?, 0x7faa421097c0?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*controller).Run(0xc0006b01b0, 0xc000334540)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:155 +0x2c6
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0003643c0, 0xc0001fff10?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/shared_informer.go:441 +0x538
created by github.com/hpe-storage/csi-driver/pkg/flavor/kubernetes.NewKubernetesFlavor
        /usr/src/hpe-csi-driver/pkg/flavor/kubernetes/flavor.go:123 +0x5ae

goroutine 139 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0002d3db8, 0x228)
        /usr/lib/golang/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc000e4cf40?)
        /usr/lib/golang/src/sync/cond.go:70 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0002d3d90, 0xc0003a5880)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:571 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0003063f0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:184 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xcc1365?, {0x1eec0a0, 0xc0003e5440}, 0x1, 0xc0003345a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000306458?, 0x3b9aca00, 0x0, 0x80?, 0x7faa421097c0?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*controller).Run(0xc0003063f0, 0xc0003345a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:155 +0x2c6
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000364460, 0x0?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/shared_informer.go:441 +0x538
created by github.com/hpe-storage/csi-driver/pkg/flavor/kubernetes.NewKubernetesFlavor
        /usr/src/hpe-csi-driver/pkg/flavor/kubernetes/flavor.go:141 +0x814

goroutine 107 [chan receive, 182 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc00027e690, 0x0?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/shared_informer.go:667 +0x45
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 108 [chan receive, 182 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x28
created by k8s.io/client-go/tools/cache.(*controller).Run
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:130 +0xbe

goroutine 177 [IO wait, 182 minutes]:
internal/poll.runtime_pollWait(0x7faa42388be8, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0006be000?, 0x20?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0006be000)
        /usr/lib/golang/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc0006be000)
        /usr/lib/golang/src/net/fd_unix.go:172 +0x35
net.(*UnixListener).accept(0x443240?)
        /usr/lib/golang/src/net/unixsock_posix.go:172 +0x1c
net.(*UnixListener).Accept(0xc0006d4030)
        /usr/lib/golang/src/net/unixsock.go:260 +0x3d
google.golang.org/grpc.(*Server).Serve(0xc0006dc000, {0x1f03600, 0xc0006d4030})
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:597 +0x362
github.com/hpe-storage/csi-driver/pkg/driver.(*nonBlockingGRPCServer).serve(0xc000624000, {0x7ffca32edc5d?, 0xc00047a530?}, {0x1ef8350?, 0xc00026a300}, {0x1f0dfa0?, 0xc00026a300}, {0x0?, 0x0})
        /usr/src/hpe-csi-driver/pkg/driver/server.go:88 +0x3ef
created by github.com/hpe-storage/csi-driver/pkg/driver.(*nonBlockingGRPCServer).Start
        /usr/src/hpe-csi-driver/pkg/driver/server.go:37 +0x145

goroutine 161 [chan receive, 182 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc00027e700, 0xc0002a9e60?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/shared_informer.go:667 +0x45
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 162 [chan receive, 182 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x28
created by k8s.io/client-go/tools/cache.(*controller).Run
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:130 +0xbe

goroutine 109 [select]:
k8s.io/client-go/tools/cache.watchHandler({0xa?, 0x0?, 0x2cb7040?}, {0x1ef32b8?, 0xc000ccf080}, {0x7faa421097c0, 0xc0003f9550}, {0x1f17e10?, 0x1c34020}, 0x0, ...)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:493 +0x1b8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0003dc000, 0xc000334540)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:329 +0x62c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:223 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000056dc0?, {0x1eec080, 0xc0006ab590}, 0x1, 0xc000334540)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0003dc000, 0xc000334540)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:222 +0x185
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 144 [select, 2 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:273 +0x11d
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:267 +0x25f

goroutine 163 [select]:
k8s.io/client-go/tools/cache.watchHandler({0xa?, 0x0?, 0x2cb7040?}, {0x1ef32b8?, 0xc000fb0280}, {0x7faa421097c0, 0xc0002d3d90}, {0x1f17e10?, 0x1c19740}, 0x0, ...)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:493 +0x1b8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0005460f0, 0xc0003345a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:329 +0x62c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:223 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00054e0c0?, {0x1eec080, 0xc000858230}, 0x1, 0xc0003345a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0005460f0, 0xc0003345a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:222 +0x185
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 156 [select, 2 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:273 +0x11d
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:267 +0x25f

goroutine 142 [syscall, 182 minutes]:
os/signal.signal_recv()
        /usr/lib/golang/src/runtime/sigqueue.go:152 +0x2f
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
        /usr/lib/golang/src/os/signal/signal.go:151 +0x2a

goroutine 178 [select]:
github.com/hpe-storage/csi-driver/pkg/monitor.(*Monitor).monitorPod.func1()
        /usr/src/hpe-csi-driver/pkg/monitor/monitor.go:104 +0x71
created by github.com/hpe-storage/csi-driver/pkg/monitor.(*Monitor).monitorPod
        /usr/src/hpe-csi-driver/pkg/monitor/monitor.go:102 +0x153

goroutine 210 [IO wait]:
internal/poll.runtime_pollWait(0x7faa42388cd8, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000697680?, 0xc000b6e000?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000697680, {0xc000b6e000, 0xa000, 0xa000})
        /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc000697680, {0xc000b6e000?, 0xc000b6e005?, 0xe11?})
        /usr/lib/golang/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc00047a308, {0xc000b6e000?, 0x41756d?, 0xc00024c8b0?})
        /usr/lib/golang/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc000286e88, {0xc000b6e000?, 0xc000286e88?, 0x0?})
        /usr/lib/golang/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc00024c990, {0x1ee84c0, 0xc000286e88})
        /usr/lib/golang/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc00024c700, {0x1eec5c0?, 0xc00047a308}, 0xa000?)
        /usr/lib/golang/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc00024c700, 0x0)
        /usr/lib/golang/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/lib/golang/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc00024c700, {0xc000b0f000, 0x1000, 0xc000a68143?})
        /usr/lib/golang/src/crypto/tls/conn.go:1316 +0x16f
bufio.(*Reader).Read(0xc0006b7740, {0xc0006e23c0, 0x9, 0xc0006f9d38?})
        /usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x1ee82c0, 0xc0006b7740}, {0xc0006e23c0, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0006e23c0?, 0x9?, 0xc000000000?}, {0x1ee82c0?, 0xc0006b7740?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0006e2380)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:498 +0x95
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0006f9f98)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:2275 +0x12e
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0006dc180)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:2170 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:821 +0xc1f

goroutine 160 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0003381e0, 0x1)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0006222a0)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Server.func2()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:296 +0xce
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:293 +0x1493

goroutine 241 [select, 62 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000bdc000)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:968 +0x233
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:302 +0x14d8

goroutine 242 [IO wait]:
internal/poll.runtime_pollWait(0x7faa42388dc8, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000254300?, 0xc000bc4000?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000254300, {0xc000bc4000, 0x8000, 0x8000})
        /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc000254300, {0xc000bc4000?, 0x60100000000?, 0x8?})
        /usr/lib/golang/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000624010, {0xc000bc4000?, 0xc000e02b60?, 0xc000a14d00?})
        /usr/lib/golang/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000596840, {0xc000280040, 0x9, 0xc000a14d28?})
        /usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x1ee82c0, 0xc000596840}, {0xc000280040, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc000280040?, 0x9?, 0xc0020068e8?}, {0x1ee82c0?, 0xc000596840?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000280000)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000bdc000, 0x7073222c7d5d7d22?, 0x6361227b3a226365?)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:458 +0x177
google.golang.org/grpc.(*Server).serveStreams(0xc0006dc000, {0x1f0a5a0?, 0xc000bdc000})
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:718 +0x114
google.golang.org/grpc.(*Server).handleRawConn.func1()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:679 +0x34
created by google.golang.org/grpc.(*Server).handleRawConn
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:678 +0x3f5

goroutine 197 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000618320, 0x1)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000596b40)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Server.func2()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:296 +0xce
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:293 +0x1493

goroutine 198 [select, 62 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000a60000)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:968 +0x233
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:302 +0x14d8

goroutine 199 [IO wait]:
internal/poll.runtime_pollWait(0x7faa42388af8, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0002d4480?, 0xc000a46000?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0002d4480, {0xc000a46000, 0x8000, 0x8000})
        /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc0002d4480, {0xc000a46000?, 0x60100000000?, 0x8?})
        /usr/lib/golang/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000520028, {0xc000a46000?, 0xc00023e9c0?, 0xc000319d00?})
        /usr/lib/golang/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000a22180, {0xc0006e2040, 0x9, 0xc000319d28?})
        /usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x1ee82c0, 0xc000a22180}, {0xc0006e2040, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0006e2040?, 0x9?, 0xc00158ac18?}, {0x1ee82c0?, 0xc000a22180?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0006e2000)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000a60000, 0x0?, 0x0?)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:458 +0x177
google.golang.org/grpc.(*Server).serveStreams(0xc0006dc000, {0x1f0a5a0?, 0xc000a60000})
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:718 +0x114
google.golang.org/grpc.(*Server).handleRawConn.func1()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:679 +0x34
created by google.golang.org/grpc.(*Server).handleRawConn
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:678 +0x3f5

goroutine 534 [select, 1 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0012de050, 0x1)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000e14a20)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Server.func2()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:296 +0xce
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:293 +0x1493

goroutine 535 [select, 62 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000e9e780)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:968 +0x233
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:302 +0x14d8

goroutine 536 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7faa42388558, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc00185a100?, 0xc0009a0000?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00185a100, {0xc0009a0000, 0x8000, 0x8000})
        /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc00185a100, {0xc0009a0000?, 0xff?, 0xc000963c98?})
        /usr/lib/golang/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000624008, {0xc0009a0000?, 0x6?, 0xc000963d00?})
        /usr/lib/golang/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00026c2a0, {0xc000948040, 0x9, 0xc000963d28?})
        /usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x1ee82c0, 0xc00026c2a0}, {0xc000948040, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc000948040?, 0x9?, 0xc001d51ab8?}, {0x1ee82c0?, 0xc00026c2a0?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000948000)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000e9e780, 0xc0008717b0?, 0xc000871780?)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:458 +0x177
google.golang.org/grpc.(*Server).serveStreams(0xc0006dc000, {0x1f0a5a0?, 0xc000e9e780})
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:718 +0x114
google.golang.org/grpc.(*Server).handleRawConn.func1()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:679 +0x34
created by google.golang.org/grpc.(*Server).handleRawConn
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:678 +0x3f5

goroutine 144755 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000b65848, 0x6)
        /usr/lib/golang/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc001971dd0?)
        /usr/lib/golang/src/sync/cond.go:70 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc000b65830, {0xc001217001, 0xdff, 0xdff})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc001217001?, 0x2?, 0x1c668ae?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:2558 +0x75
encoding/json.(*Decoder).refill(0xc000da0b40)
        /usr/lib/golang/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc000da0b40)
        /usr/lib/golang/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc000da0b40, {0x19f26e0, 0xc001862600})
        /usr/lib/golang/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000c03290, {0xc00083ca00, 0x1000, 0x1500})
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x1a5
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc001a46f50, 0xc000bb0cb0?, {0x1ef3128, 0xc000b57f80})
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000b6d220)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000fb0280)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdc
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 139349 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001a299c8, 0x245)
        /usr/lib/golang/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc001529050?)
        /usr/lib/golang/src/sync/cond.go:70 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc001a299b0, {0xc000c1c001, 0x1dff, 0x1dff})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000c1c001?, 0x2?, 0x1c668ae?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:2558 +0x75
encoding/json.(*Decoder).refill(0xc0007e9cc0)
        /usr/lib/golang/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc0007e9cc0)
        /usr/lib/golang/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc0007e9cc0, {0x19f26e0, 0xc0002873e0})
        /usr/lib/golang/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000f79cb0, {0xc001342000, 0x1000, 0x1500})
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x1a5
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000f0cd20, 0xc001064000?, {0x1ef3128, 0xc001507bc0})
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0006b8880)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000ccf080)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdc
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 144754 [select, 2 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000b65800, 0xc000e8f500)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1464 +0xb27
golang.org/x/net/http2.(*clientStream).doRequest(0x8a0126?, 0xc001659560?)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1326 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1232 +0x34a

goroutine 139348 [select, 5 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc001a29980, 0xc000f95100)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1464 +0xb27
golang.org/x/net/http2.(*clientStream).doRequest(0x8a0186?, 0xc00183cfc0?)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1326 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1232 +0x34a

goroutine 147331 [runnable]:
internal/fmtsort.Sort({0x19c7860?, 0xc000308ae0?, 0x110?})
        /usr/lib/golang/src/internal/fmtsort/sort.go:66 +0x393
fmt.(*pp).printValue(0xc000414680, {0x19c7860?, 0xc000308ae0?, 0xc000f38000?}, 0x76, 0x0)
        /usr/lib/golang/src/fmt/print.go:816 +0x986
fmt.(*pp).printArg(0xc000414680, {0x19c7860?, 0xc000308ae0}, 0x76)
        /usr/lib/golang/src/fmt/print.go:759 +0x756
fmt.(*pp).doPrintf(0xc000414680, {0x1c7885e, 0x16}, {0xc000655800?, 0x1, 0x1})
        /usr/lib/golang/src/fmt/print.go:1077 +0x387
fmt.Sprintf({0x1c7885e, 0x16}, {0xc000655800, 0x1, 0x1})
        /usr/lib/golang/src/fmt/print.go:239 +0x59
github.com/sirupsen/logrus.(*Entry).Logf(0xc000f393b0, 0x6, {0x1c7885e?, 0x19130a0?}, {0xc000655800?, 0x19130a0?, 0xc000f66900?})
        /usr/src/hpe-csi-driver/vendor/github.com/sirupsen/logrus/entry.go:338 +0x49
github.com/sirupsen/logrus.(*Entry).Tracef(...)
        /usr/src/hpe-csi-driver/vendor/github.com/sirupsen/logrus/entry.go:343
github.com/hpe-storage/common-host-libs/logger.Tracef({0x1c7885e, 0x16}, {0xc000655800, 0x1, 0x1})
        /usr/src/hpe-csi-driver/vendor/github.com/hpe-storage/common-host-libs/logger/logger.go:556 +0x55
github.com/hpe-storage/csi-driver/pkg/driver.(*Driver).ClearRequest(0xc00026a300, {0xc000fcbe80, 0x37})
        /usr/src/hpe-csi-driver/pkg/driver/driver.go:613 +0x34f
github.com/hpe-storage/csi-driver/pkg/driver.(*Driver).DeleteVolume(0xc00026a300, {0x1c6ff37?, 0x40de67?}, 0xc000efdf80)
        /usr/src/hpe-csi-driver/pkg/driver/controller_server.go:627 +0x454
github.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1({0x1f04248, 0xc001638210}, {0x1b4fb20?, 0xc000efdf80})
        /usr/src/hpe-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5696 +0x78
github.com/hpe-storage/csi-driver/pkg/driver.logGRPC({0x1f04248, 0xc001638210}, {0x1b4fb20?, 0xc000efdf80?}, 0xc000313ab8?, 0xc000542f78)
        /usr/src/hpe-csi-driver/pkg/driver/utils.go:71 +0x127
github.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler({0x1c58d60?, 0xc00026a300}, {0x1f04248, 0xc001638210}, 0xc001066d80, 0x1d6d1f0)
        /usr/src/hpe-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5698 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0006dc000, {0x1f0a5a0, 0xc000a60000}, 0xc000d57200, 0xc0006d41e0, 0x2c9c5f8, 0x0)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:1024 +0xd5e
google.golang.org/grpc.(*Server).handleStream(0xc0006dc000, {0x1f0a5a0, 0xc000a60000}, 0xc000d57200, 0x0)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:1313 +0xa25
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:722 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:720 +0xea

[megakid]

Got the same behaviour on a per-node CSI driver (same container so not surprising) also.

[megakid]

This is caused by a threading bug that only happens when trace logging is enabled:

Possible fix (will PR if I get time):

[datamattsson]

Thanks for identifying this. I'll bring it up with the team.

[megakid]

This looks more serious actually.

I'm getting issues without trace logging, caused by multiple delete() calls across more than 1 key at the same time:

time="2024-03-13T17:12:03Z" level=info msg="GRPC call: /csi.v1.Controller/DeleteVolume" file="utils.go:69"
time="2024-03-13T17:12:03Z" level=info msg="GRPC request: {\"secrets\":\"***stripped***\",\"volume_id\":\"0654316e079d129ca30000000000000000000d3249\"}" file="utils.go:70"
time="2024-03-13T17:12:03Z" level=info msg="GRPC call: /csi.v1.Controller/DeleteVolume" file="utils.go:69"
time="2024-03-13T17:12:03Z" level=info msg="GRPC request: {\"secrets\":\"***stripped***\",\"volume_id\":\"0654316e079d129ca30000000000000000000d3247\"}" file="utils.go:70"
time="2024-03-13T17:12:03Z" level=info msg="GRPC response: {\"publish_context\":{\"accessProtocol\":\"iscsi\",\"discoveryIps\":\"10.2.80.160\",\"fsCreateOptions\":\"\",\"fsMode\":\"\",\"fsOwner\":\"\",\"fsType\":\"xfs\",\"lunId\":\"0\",\"readOnly\":\"false\",\"serialNumber\":\"86f6f5a91a0a3c5f6c9ce900a4f223c9\",\"targetNames\":\"iqn.2007-11.com.nimblestorage:pvc-60101f03-e63a-4191-a402-57b6d9434f19-v54316e079d129ca3.000d329c.c923f2a4\",\"targetScope\":\"volume\",\"volumeAccessMode\":\"mount\"}}" file="utils.go:75"
time="2024-03-13T17:12:03Z" level=info msg="About to delete volume 0654316e079d129ca30000000000000000000d3249 with force=false" file="controller_server.go:673"
time="2024-03-13T17:12:03Z" level=info msg="About to delete volume 0654316e079d129ca30000000000000000000d3247 with force=false" file="controller_server.go:673"
time="2024-03-13T17:12:03Z" level=info msg="GRPC response: {}" file="utils.go:75"
fatal error: concurrent map writes

goroutine 29364 [running]:
github.com/hpe-storage/csi-driver/pkg/driver.(*Driver).ClearRequest(0xc000a22300, {0xc000057780, 0x37})
        /usr/src/hpe-csi-driver/pkg/driver/driver.go:612 +0x30b
github.com/hpe-storage/csi-driver/pkg/driver.(*Driver).DeleteVolume(0xc000a22300, {0x1c6ff37?, 0x40de67?}, 0xc0019ae0c0)
        /usr/src/hpe-csi-driver/pkg/driver/controller_server.go:627 +0x454
github.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1({0x1f04248, 0xc000e70bd0}, {0x1b4fb20?, 0xc0019ae0c0})
        /usr/src/hpe-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5696 +0x78
github.com/hpe-storage/csi-driver/pkg/driver.logGRPC({0x1f04248, 0xc000e70bd0}, {0x1b4fb20?, 0xc0019ae0c0?}, 0xc001559ab8?, 0xc0013a88a0)
        /usr/src/hpe-csi-driver/pkg/driver/utils.go:71 +0x127
github.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler({0x1c58d60?, 0xc000a22300}, {0x1f04248, 0xc000e70bd0}, 0xc001ab4900, 0x1d6d1f0)
        /usr/src/hpe-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5698 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000002900, {0x1f0a5a0, 0xc000ab0c00}, 0xc0011b4500, 0xc0005b21e0, 0x2c9c5f8, 0x0)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:1024 +0xd5e
google.golang.org/grpc.(*Server).handleStream(0xc000002900, {0x1f0a5a0, 0xc000ab0c00}, 0xc0011b4500, 0x0)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:1313 +0xa25
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:722 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:720 +0xea

goroutine 1 [chan receive, 32 minutes]:
main.csiCliHandler(0x1c82d51?)
        /usr/src/hpe-csi-driver/cmd/csi-driver/csi-driver.go:173 +0x767
main.glob..func1(0x2c9e620?, {0xc0002b3f40, 0x0, 0x4})
        /usr/src/hpe-csi-driver/cmd/csi-driver/csi-driver.go:61 +0x1c5
github.com/spf13/cobra.(*Command).execute(0x2c9e620, {0xc0002b4010, 0x4, 0x4})
        /usr/src/hpe-csi-driver/vendor/github.com/spf13/cobra/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x2c9e620)
        /usr/src/hpe-csi-driver/vendor/github.com/spf13/cobra/command.go:1040 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /usr/src/hpe-csi-driver/vendor/github.com/spf13/cobra/command.go:968
main.Main()
        /usr/src/hpe-csi-driver/cmd/csi-driver/csi-driver.go:184 +0x25
main.main()
        /usr/src/hpe-csi-driver/cmd/csi-driver/csi-driver.go:192 +0x17

goroutine 140 [chan receive, 32 minutes]:
gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun(0xc000a24120)
        /usr/src/hpe-csi-driver/vendor/gopkg.in/natefinch/lumberjack.v2/lumberjack.go:379 +0x45
created by gopkg.in/natefinch/lumberjack%2ev2.(*Logger).mill.func1
        /usr/src/hpe-csi-driver/vendor/gopkg.in/natefinch/lumberjack.v2/lumberjack.go:390 +0x96

goroutine 141 [chan receive, 32 minutes]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0003fe0a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116

goroutine 142 [chan receive, 32 minutes]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/record/event.go:307 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/record/event.go:305 +0x13d

goroutine 143 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000532028, 0x5d5)
        /usr/lib/golang/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc0013a34e0?)
        /usr/lib/golang/src/sync/cond.go:70 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000532000, 0xc000534010)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:571 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000516090)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:184 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xcc1365?, {0x1eec0a0, 0xc00047ade0}, 0x1, 0xc000a42540)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0005160f8?, 0x3b9aca00, 0x0, 0x0?, 0x7fc0595bac18?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*controller).Run(0xc000516090, 0xc000a42540)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:155 +0x2c6
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000a48fa0, 0x0?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/shared_informer.go:441 +0x538
created by github.com/hpe-storage/csi-driver/pkg/flavor/kubernetes.NewKubernetesFlavor
        /usr/src/hpe-csi-driver/pkg/flavor/kubernetes/flavor.go:123 +0x5ae

goroutine 144 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0004a7418, 0x86)
        /usr/lib/golang/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc0013a3060?)
        /usr/lib/golang/src/sync/cond.go:70 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0004a73f0, 0xc000238700)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:571 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0003805a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:184 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xcc1365?, {0x1eec0a0, 0xc0004b3c20}, 0x1, 0xc000a425a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000380608?, 0x3b9aca00, 0x0, 0x80?, 0x7fc0595bac18?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*controller).Run(0xc0003805a0, 0xc000a425a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:155 +0x2c6
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000a49040, 0x0?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/shared_informer.go:441 +0x538
created by github.com/hpe-storage/csi-driver/pkg/flavor/kubernetes.NewKubernetesFlavor
        /usr/src/hpe-csi-driver/pkg/flavor/kubernetes/flavor.go:141 +0x814

goroutine 193 [IO wait, 32 minutes]:
internal/poll.runtime_pollWait(0x7fc05988fdc8, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc00059c000?, 0x20?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00059c000)
        /usr/lib/golang/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc00059c000)
        /usr/lib/golang/src/net/fd_unix.go:172 +0x35
net.(*UnixListener).accept(0x443240?)
        /usr/lib/golang/src/net/unixsock_posix.go:172 +0x1c
net.(*UnixListener).Accept(0xc0005b2030)
        /usr/lib/golang/src/net/unixsock.go:260 +0x3d
google.golang.org/grpc.(*Server).Serve(0xc000002900, {0x1f03600, 0xc0005b2030})
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:597 +0x362
github.com/hpe-storage/csi-driver/pkg/driver.(*nonBlockingGRPCServer).serve(0xc000244000, {0x7ffc53285c5e?, 0xc000a24120?}, {0x1ef8350?, 0xc000a22300}, {0x1f0dfa0?, 0xc000a22300}, {0x0?, 0x0})
        /usr/src/hpe-csi-driver/pkg/driver/server.go:88 +0x3ef
created by github.com/hpe-storage/csi-driver/pkg/driver.(*nonBlockingGRPCServer).Start
        /usr/src/hpe-csi-driver/pkg/driver/server.go:37 +0x145

goroutine 48 [chan receive, 32 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000a3a5b0, 0x0?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/shared_informer.go:667 +0x45
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 177 [chan receive, 32 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x28
created by k8s.io/client-go/tools/cache.(*controller).Run
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:130 +0xbe

goroutine 194 [select]:
github.com/hpe-storage/csi-driver/pkg/monitor.(*Monitor).monitorPod.func1()
        /usr/src/hpe-csi-driver/pkg/monitor/monitor.go:104 +0x71
created by github.com/hpe-storage/csi-driver/pkg/monitor.(*Monitor).monitorPod
        /usr/src/hpe-csi-driver/pkg/monitor/monitor.go:102 +0x153

goroutine 146 [chan receive, 32 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000a3a620, 0x0?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/shared_informer.go:667 +0x45
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 147 [chan receive, 32 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x28
created by k8s.io/client-go/tools/cache.(*controller).Run
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/controller.go:130 +0xbe

goroutine 163 [syscall, 32 minutes]:
os/signal.signal_recv()
        /usr/lib/golang/src/runtime/sigqueue.go:152 +0x2f
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
        /usr/lib/golang/src/os/signal/signal.go:151 +0x2a

goroutine 148 [select]:
k8s.io/client-go/tools/cache.watchHandler({0xa?, 0x0?, 0x2cb7040?}, {0x1ef32b8?, 0xc000a413c0}, {0x7fc0595bac18, 0xc0004a73f0}, {0x1f17e10?, 0x1c19740}, 0x0, ...)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:493 +0x1b8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0004560f0, 0xc000a425a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:329 +0x62c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:223 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00098bec8?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00088d780?, {0x1eec080, 0xc000051ae0}, 0x1, 0xc000a425a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0004560f0, 0xc000a425a0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:222 +0x185
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 276 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:273 +0x11d
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:267 +0x25f

goroutine 178 [select]:
k8s.io/client-go/tools/cache.watchHandler({0xa?, 0x0?, 0x2cb7040?}, {0x1ef32b8?, 0xc0018ba9c0}, {0x7fc0595bac18, 0xc000532000}, {0x1f17e10?, 0x1c34020}, 0x0, ...)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:493 +0x1b8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc00054a000, 0xc000a42540)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:329 +0x62c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:223 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000555ec8?)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000056980?, {0x1eec080, 0xc0002b41e0}, 0x1, 0xc000a42540)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc00054a000, 0xc000a42540)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:222 +0x185
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:58 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

goroutine 264 [IO wait]:
internal/poll.runtime_pollWait(0x7fc05988fbe8, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc00025e400?, 0xc00099e000?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00025e400, {0xc00099e000, 0xa000, 0xa000})
        /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc00025e400, {0xc00099e000?, 0xc00099e005?, 0x83?})
        /usr/lib/golang/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0000127c8, {0xc00099e000?, 0x7ee483?, 0xc0006b4fb0?})
        /usr/lib/golang/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc00072ee10, {0xc00099e000?, 0xc00072ee10?, 0x0?})
        /usr/lib/golang/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc0006b5090, {0x1ee84c0, 0xc00072ee10})
        /usr/lib/golang/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc0006b4e00, {0x1eec5c0?, 0xc0000127c8}, 0xa000?)
        /usr/lib/golang/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc0006b4e00, 0x0)
        /usr/lib/golang/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/lib/golang/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc0006b4e00, {0xc000cc5000, 0x1000, 0x0?})
        /usr/lib/golang/src/crypto/tls/conn.go:1316 +0x16f
bufio.(*Reader).Read(0xc000594720, {0xc000896ba0, 0x9, 0xc001044988?})
        /usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x1ee82c0, 0xc000594720}, {0xc000896ba0, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc000896ba0?, 0x9?, 0xc000000000?}, {0x1ee82c0?, 0xc000594720?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000896b60)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:498 +0x95
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000269f98)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:2275 +0x12e
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0006b9080)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:2170 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:821 +0xc1f

goroutine 210 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00050c280, 0x1)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000458ba0)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Server.func2()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:296 +0xce
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:293 +0x1493

goroutine 211 [select, 32 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000002a80)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:968 +0x233
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:302 +0x14d8

goroutine 212 [IO wait]:
internal/poll.runtime_pollWait(0x7fc05988faf8, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc00059c200?, 0xc0005d4000?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00059c200, {0xc0005d4000, 0x8000, 0x8000})
        /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc00059c200, {0xc0005d4000?, 0x60100000000?, 0x8?})
        /usr/lib/golang/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0005a0010, {0xc0005d4000?, 0xc0004eb380?, 0xc000246d00?})
        /usr/lib/golang/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0005940c0, {0xc0005be120, 0x9, 0xc000246d28?})
        /usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x1ee82c0, 0xc0005940c0}, {0xc0005be120, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0005be120?, 0x9?, 0xc000b8b3c8?}, {0x1ee82c0?, 0xc0005940c0?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005be0e0)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000002a80, 0x0?, 0x0?)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:458 +0x177
google.golang.org/grpc.(*Server).serveStreams(0xc000002900, {0x1f0a5a0?, 0xc000002a80})
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:718 +0x114
google.golang.org/grpc.(*Server).handleRawConn.func1()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:679 +0x34
created by google.golang.org/grpc.(*Server).handleRawConn
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:678 +0x3f5

goroutine 234 [select, 32 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000511bd0, 0x1)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000242d20)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Server.func2()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:296 +0xce
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:293 +0x1493

goroutine 153 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0002b4e60, 0x1)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000a24240)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Server.func2()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:296 +0xce
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:293 +0x1493

goroutine 154 [select, 32 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000ab0c00)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:968 +0x233
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:302 +0x14d8

goroutine 155 [IO wait]:
internal/poll.runtime_pollWait(0x7fc05988fa08, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc00002f080?, 0xc000a14000?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00002f080, {0xc000a14000, 0x8000, 0x8000})
        /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc00002f080, {0xc000a14000?, 0x1060100000000?, 0x8?})
        /usr/lib/golang/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000244700, {0xc000a14000?, 0x0?, 0xc000249cf8?})
        /usr/lib/golang/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0002426c0, {0xc0002dd0e0, 0x9, 0x1c3e5c8b4f5?})
        /usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x1ee82c0, 0xc0002426c0}, {0xc0002dd0e0, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0002dd0e0?, 0x9?, 0xc0002b4eb0?}, {0x1ee82c0?, 0xc0002426c0?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002dd0a0)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000ab0c00, 0x0?, 0x0?)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:458 +0x177
google.golang.org/grpc.(*Server).serveStreams(0xc000002900, {0x1f0a5a0?, 0xc000ab0c00})
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:718 +0x114
google.golang.org/grpc.(*Server).handleRawConn.func1()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:679 +0x34
created by google.golang.org/grpc.(*Server).handleRawConn
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:678 +0x3f5

goroutine 229 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0005102d0, 0x1)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000458fc0)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Server.func2()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:296 +0xce
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:293 +0x1493

goroutine 230 [select, 32 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0006b8c00)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:968 +0x233
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:302 +0x14d8

goroutine 231 [IO wait]:
internal/poll.runtime_pollWait(0x7fc05988f918, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0006b0480?, 0xc0006c0000?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0006b0480, {0xc0006c0000, 0x8000, 0x8000})
        /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc0006b0480, {0xc0006c0000?, 0x60100000000?, 0x8?})
        /usr/lib/golang/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000012578, {0xc0006c0000?, 0xc000683ba0?, 0xc00024bd00?})
        /usr/lib/golang/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000458f60, {0xc0005bee40, 0x9, 0xc00024bd28?})
        /usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x1ee82c0, 0xc000458f60}, {0xc0005bee40, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0005bee40?, 0x9?, 0xc0018c3608?}, {0x1ee82c0?, 0xc000458f60?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005bee00)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0006b8c00, 0x0?, 0x0?)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:458 +0x177
google.golang.org/grpc.(*Server).serveStreams(0xc000002900, {0x1f0a5a0?, 0xc0006b8c00})
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:718 +0x114
google.golang.org/grpc.(*Server).handleRawConn.func1()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:679 +0x34
created by google.golang.org/grpc.(*Server).handleRawConn
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:678 +0x3f5

goroutine 290 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:273 +0x11d
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/tools/cache/reflector.go:267 +0x25f

goroutine 235 [select, 32 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0006b8d80)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:968 +0x233
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:302 +0x14d8

goroutine 236 [IO wait, 32 minutes]:
internal/poll.runtime_pollWait(0x7fc05988f828, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0006b0700?, 0xc0006e6000?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0006b0700, {0xc0006e6000, 0x8000, 0x8000})
        /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc0006b0700, {0xc0006e6000?, 0x60100000000?, 0x8?})
        /usr/lib/golang/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000012590, {0xc0006e6000?, 0xc000682680?, 0xc00024cd00?})
        /usr/lib/golang/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000459140, {0xc0005bf7e0, 0x9, 0xc00024cd28?})
        /usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x1ee82c0, 0xc000459140}, {0xc0005bf7e0, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0005bf7e0?, 0x9?, 0xc00005ae88?}, {0x1ee82c0?, 0xc000459140?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005bf7a0)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0006b8d80, 0x0?, 0x0?)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/internal/transport/http2_server.go:458 +0x177
google.golang.org/grpc.(*Server).serveStreams(0xc000002900, {0x1f0a5a0?, 0xc0006b8d80})
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:718 +0x114
google.golang.org/grpc.(*Server).handleRawConn.func1()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:679 +0x34
created by google.golang.org/grpc.(*Server).handleRawConn
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:678 +0x3f5

goroutine 28730 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0021e4780, 0xc0011b4b00)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1464 +0xb27
golang.org/x/net/http2.(*clientStream).doRequest(0x8a0126?, 0xc000e7cfc0?)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1326 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1232 +0x34a

goroutine 25976 [select, 4 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc001123200, 0xc0006ab000)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1464 +0xb27
golang.org/x/net/http2.(*clientStream).doRequest(0x8a0126?, 0xc00110fb00?)
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1326 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:1232 +0x34a

goroutine 28731 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0021e47c8, 0x26)
        /usr/lib/golang/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc000f5ad20?)
        /usr/lib/golang/src/sync/cond.go:70 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc0021e47b0, {0xc000db6001, 0x1dff, 0x1dff})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000db6001?, 0x2?, 0x1c668ae?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:2558 +0x75
encoding/json.(*Decoder).refill(0xc001450780)
        /usr/lib/golang/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc001450780)
        /usr/lib/golang/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc001450780, {0x19f26e0, 0xc000c10bd0})
        /usr/lib/golang/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc001944f60, {0xc0001e4000, 0x1000, 0x1500})
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x1a5
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000ef34f0, 0xc001188000?, {0x1ef3128, 0xc001608340})
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000ed25c0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0018ba9c0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdc
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 29400 [runnable]:
github.com/hpe-storage/common-host-libs/concurrent.(*MapMutex).Unlock(0xc000a38400, {0xc000b8f040, 0x37})
        /usr/src/hpe-csi-driver/vendor/github.com/hpe-storage/common-host-libs/concurrent/concurrent.go:79 +0x174
github.com/hpe-storage/csi-driver/pkg/driver.(*Driver).ClearRequest(0xc000a22300, {0xc000b8f040, 0x37})
        /usr/src/hpe-csi-driver/pkg/driver/driver.go:615 +0x3ab
github.com/hpe-storage/csi-driver/pkg/driver.(*Driver).DeleteVolume(0xc000a22300, {0x1c6ff37?, 0x40de67?}, 0xc000e50540)
        /usr/src/hpe-csi-driver/pkg/driver/controller_server.go:627 +0x454
github.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1({0x1f04248, 0xc0013ce600}, {0x1b4fb20?, 0xc000e50540})
        /usr/src/hpe-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5696 +0x78
github.com/hpe-storage/csi-driver/pkg/driver.logGRPC({0x1f04248, 0xc0013ce600}, {0x1b4fb20?, 0xc000e50540?}, 0xc00155cab8?, 0xc00072ec18)
        /usr/src/hpe-csi-driver/pkg/driver/utils.go:71 +0x127
github.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler({0x1c58d60?, 0xc000a22300}, {0x1f04248, 0xc0013ce600}, 0xc0015ab0e0, 0x1d6d1f0)
        /usr/src/hpe-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5698 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000002900, {0x1f0a5a0, 0xc000ab0c00}, 0xc000260e00, 0xc0005b21e0, 0x2c9c5f8, 0x0)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:1024 +0xd5e
google.golang.org/grpc.(*Server).handleStream(0xc000002900, {0x1f0a5a0, 0xc000ab0c00}, 0xc000260e00, 0x0)
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:1313 +0xa25
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:722 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /usr/src/hpe-csi-driver/vendor/google.golang.org/grpc/server.go:720 +0xea

goroutine 25977 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001123248, 0x13)
        /usr/lib/golang/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc001945a10?)
        /usr/lib/golang/src/sync/cond.go:70 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc001123230, {0xc000941001, 0xdff, 0xdff})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000941001?, 0x2?, 0x1c668ae?})
        /usr/src/hpe-csi-driver/vendor/golang.org/x/net/http2/transport.go:2558 +0x75
encoding/json.(*Decoder).refill(0xc0012c48c0)
        /usr/lib/golang/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc0012c48c0)
        /usr/lib/golang/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc0012c48c0, {0x19f26e0, 0xc000df78c0})
        /usr/lib/golang/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0012fb020, {0xc000a1d500, 0x1000, 0x1500})
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x1a5
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000c0fb30, 0xc000cbc000?, {0x1ef3128, 0xc000ff5140})
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000597e00)
        /usr/src/hpe-csi-driver/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000a413c0)
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdc
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /usr/src/hpe-csi-driver/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

I don't think it makes sense to lock on a per-key basis for this?

[dileepds]

We will look into this issue. As I remember the lock is on per request ID , but will check any failures are there in the lock creation part.