Auto failover, when one 3par is down

[krakazyabra]

Hello. I've deployed 3par-primera-csp for 1.18 .

I've faced with problem: when one of two 3pars device is down (no iscsi link), the volume stops to export on host.

My config was described here

I have pvc in k8s, on 3par side there are 2 volumes (one original and second replicated). This volume exports to node, I can see it in multipath. But when I'm shutting down one 3par, I'm waiting, that the volume still will be exporting through remaining available ports.

[datamattsson]

@wdurairaj can you take a look at this or involve someone who can?

[wdurairaj]

this is documented as a known limitation and there is a workaround around how to make the POD to come in running state in this page -- https://github.com/hpe-storage/csi-driver/blob/master/release-notes/v1.3.0.md

this is the workaround suggested there

• It is recommended to edit backend in primary secret (using kubectl apply -f secret) to make it point to secondary array ip

[krakazyabra]

Hello @wdurairaj
I was talking not about PVC, but about more lower level - iscsi device on node. if one array is down, on node there should be some switching to active ghost running device.
This is my multipath -ll

root@m5c25:/home/ep192# multipath -ll -v2
mpathq (360002ac0000000000000003700019d4a) dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 2:0:0:0 sda 8:0  active ready running
| `- 3:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 4:0:0:0 sdc 8:32 active ghost running

And I expect, when I disconnect sda and sdb, sdc device will become primary.
But I get

root@m5c25:/home/ep192# multipath -ll -v2
mpathq (360002ac0000000000000003700019d4a) dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=enabled
| |- 2:0:0:0 sda 8:0  failed faulty running
| `- 3:0:0:0 sdb 8:16 failed faulty running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 4:0:0:0 sdc 8:32 failed ghost running

My miltipath config is

root@m5c25:/home/ep192# cat /etc/multipath.conf 
defaults {
    user_friendly_names yes
    find_multipaths     no
    uxsock_timeout      10000
}
blacklist {
    devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
    devnode "^hd[a-z]"
    device {
        vendor  ".*"
        product ".*"
    }
}
blacklist_exceptions {
    property "(ID_WWN|SCSI_IDENT_.*|ID_SERIAL)"
    device {
        vendor  "Nimble"
        product "Server"
    }
    device {
        product "VV"
        vendor  "3PARdata"
    }
    device {
        vendor  "TrueNAS"
        product "iSCSI Disk"
    }
}
devices {
    device {
        vendor               "Nimble"
        rr_weight            uniform
        rr_min_io            100
        hardware_handler     "1 alua"
        rr_min_io_rq         1
        prio                 alua
        dev_loss_tmo         infinity
        fast_io_fail_tmo     5
        no_path_retry        18
        failback             immediate
        path_selector        "round-robin 0"
        product              "Server"
        path_checker         tur
        path_grouping_policy group_by_prio
        features             0
    }
    device {
        uid_attribute        ID_SERIAL
        vendor               "TrueNAS"
        product              "iSCSI Disk"
        path_grouping_policy group_by_prio
        path_selector        "queue-length 0"
        hardware_handler     "1 alua"
        rr_weight            priorities
    }
}

[krakazyabra]

I could manually make failover.
from 3par02 there were 2 connections, from 3par01 - one
multipath showed me

mpathb (360002ac0000000000000005300019d4a) dm-1 ##,##
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
mpatha (360002ac0000000000000005c00019d4a) dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 3:0:0:0 sdb 8:16 active ready running
| `- 2:0:0:0 sda 8:0  active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 4:0:0:0 sdc 8:32 active ghost running

(sda and sdb - from 3par02, sdc from 3par01)

I shutted down all ports on 3par02 (remote copy + iscsi). I expected, that device sdc will automatically become active ready running. But it get another status: failed ghost running.
Then in SSMC I went to Remote Copy Groups, and click on Failover button. In the same second, volume from 3par01 became active and I could access it from pod. All ports from 3par02 were down.

So, in manual mode, I could solve the task. But in my opinion, such failovering should do csi-driver

[wdurairaj]

• Failover is not done by the CSI driver, but this driver provides a way to replicate the data between arrays
• Automatic failover is ideally done by another product called HPE Quorum Witness which monitores the health of the primary/secondary array and initiates failover. There is a community blog around this peer persistence replication that we do in the CSI driver which talks in detail about this. You can read in some Youtube videos as well. This is the preferred mechanism for doing failover. Whatever is mentioned in the previous steps is a manual failover procedure using tools like SSMC/CLI.

[krakazyabra]

Hello, @wdurairaj
I checked everything and fount, that Auto failover was not enabled for RMC. Now it is enable and I made some tests:
Disabled secondary array first (physically, disconnected iSCSI, RC and mgmt cable in 10 seconds), waited for 1-2 mins and disconnected master array. I simulated outage.
Then I connected master back (second array was still disconnected), but on the node my volume became ro (read-only). I couldn't find the way, how to make it rw again. Probably that should be done by hpe-csi-node daemonset.

My second question is: if I manually logout from the iscsi session iscsiadm --mode node --targetname <iqn> --portal <ip:port> --logout and delete the session iscsiadm -m node -T <iqn> -p <ip:port> -u, how is it possible to restore connection back (from 3par volume is exporting)

Thanks in advance.