Could not create Target in configFS #159
Comments
|
In waiting, I tried CentOS 8 which was an even bigger bust. I wiped that clean and tried again with Fedora 31. Same darn error "Could not create Target in configFS". Anyone?? |
|
I figure the type of HBA would be good to know. They're a Cavium QLogic BR-1860 QLE2662 which is supported from all I've read. Here are some details.
|
I am going to investigate this. |
Note: it's not going to work in CentOS 8 because the Red Hat management decided to disable the qla2xxx target kernel module. If I remember correctly, there were concerns about the stability of that code. |
|
I was able to load the module and made everything look like I had it in CentOS 7 which let me get much further but still whenever looking at the structure via Any idea about the logging of that error so I can trace down the issue? FWIW, it's not working in CentOS 7, CentOS 8 or even Fedora 31. Actually 7 and 31 behave exactly the same. Also, @maurizio-lombardi I see you are a maintainer. I REALLY appreciate your help. I have HOURS into this and getting nowhere like this is literally driving me mad. |
Did you also test the default 3.10.0-* Centos7 kernel ? |
Thanks :) |
|
Yes. I used the default kernel first but it wouldn’t load qla2xxx with an error about invalid parameter. I then used modprobe-f and it said invalid key. Googling that said it was an unsigned module. I’ll wipe the machine and reinstall 7 again and see if the issue repeats itself. |
As I said, this is totally expected in Centos 8 because we removed the qla2xxx target mode entirely (in rtslib too). I am not sure about Centos 7 and Fedora 31, I will try to find out. |
|
I have a new CentOS 7 minimal install done with targetcli installed now. Created Rebooted to ensure everything loaded. Noticed tcm_qla2xxx did not load. Checked to make sure the mod is there. What is causing this? This is what had me try that later kernel since the tcm_qla2xxx would load. I checked configfs and it's mounted as it should be. dmesg reveals
|
This is very interesting! |
|
Ok, I found out the possible mistake: a missing check_stop_free pointer initialization in the RHEL/Centos 7 tcm_qla2xxx kernel module. |
|
I am almost certain that it's a regression introduced in RHEL 7.6, I am going to open a bugzilla against RHEL7. |
Any idea which kernel actually works so I can downgrade or is there a repo of testing kernels their fix may be applied against so I can get this going? This is why I went to that newer kernel elrepo since it seems that part of it was resolved. Should I try upgrading to that again so we can see what the problem is with targetcli while they research that module? |
Probably it could work with older kernels (version <= kernel-3.10.0-900.el7)
It's not a top priority for my team in Red Hat (we are focused on 3.10.0 kernels) but if you want to do that then go ahead, it may be interesting for the qla2xxx team. |
Sounds awesome! Let me know when ready.
I would think they could see what was done and implement the fix in the older kernel. Does the rtslib and/or targetcli do any logging any where to show why it fails? The message |
I think the problem is in the kernel, not in rtslib. rtslib only receives a error return code and prints an error message, it's not able to give you much details. |
|
Ok. I’ll wait to try out your test kernel. |
|
Hello, I have a test kernel for Centos 7 that fixes the module loading (check_stop_free error) |
|
Reboot... Had to make another entry to load tcm_qla2xxx on boot since it wasn't loaded. Ensuring everything is loaded. Confirmed target mode Looking for relevant information in dmesg Cool. Error is gone on the module. Trying targetcli No new messages in dmesg and nothing in /var/log/messages either. What's next? |
|
Ok thanks for testing, I am waiting for the qla2xxx maintainer to answer my email. |
|
It’s worth mentioning that while it does not say it in any of the logs, these cards are QLogic 2662. QLogic doesn’t offer drivers for RHEL7/CentOS 7 saying that they are supported in the qla2xxx module. |
|
That may be the reason why the driver refuses to create the target |
All the docs online say the qla2xxx series cards, which mine are, are supported in the latest kernels. At this point, it's as if the driver is silently failing. There's no feedback to indicate why. Why would the driver refuse to create the target if everything says it’s supported? Is it possible that the driver is misidentifying the cards? I wonder if the team will know of some specific bios setting I need to apply. It’ll be interesting to hear their feedback. |
|
Per this link: https://bugzilla.redhat.com/show_bug.cgi?id=1666377
Is that only for CentOS 8? Been googling for a while. Everything I've seen says the cards are supported. This is brutal. No errors reported anywhere. Just silent fail. |
Yes, for RHEL8 and Centos8. For those 2 operating systems qla2xxx target mode has been completely disabled. For RHEL7 and Centos7 it's expected to be available (but only as a technology preview, not fully supported)
I guess this is a problem within the qla2xxx driver. |
|
Well, since the one issue with the kernel you sent seems fixed (yay!), do you have another module with some debug code and logging that we can test to find out what's going on? |
I will prepare something to test |
|
Anything new from the team on the status of support for the card or a test module to see what's happening? In waiting, I installed FreeNAS again, set the server in target mode. The CentOS 7 initiator on one of my other servers didn't activate the qla2xxx module on boot. I then enabled it with modprobe. I tried putting an entry /etc/modules-load.d/qla2xxx.conf but the kernel still doesn't load it automatically. It doesn't recognize the target either. This is so strange. I'll be calling QLogic to ask about this. |
|
QLogic has no number but I opened a case. Any feedback from the qla2xxx team as to why it's not properly recognizing the card? |
|
Swapped out those cards. Put in QLE2564 which work with FreeNAS fine so I'm sure they're good with CentOS 7. This is new...
|
|
Couldn't relaunch targetcli. Rebooted server, tried again and same error. Suggestions? |
|
[Errno 13] Permission denied: '/sys/kernel/config/target/qla2xxx/21:00:00:24:ff:67:44:0c/tpgt_1/enable' Hmmm, this not a bug in targetcli. I am going to ping again the qla2xxx team |
|
Ok. Did they ever respond the previous time? While waiting on them, I plan to use CLVM to manage writes to the SAN. Is it possible to mount the target block device on the target server, aside from the initiators (for performing maintenance etc)? |
Yes, the guy I contacted told me he was going to do a test but he never got back with the results.
It's possible, but you have to be sure that the filesystem is not mounted by an initiator, otherwise you will end up corrupting the filesystem. |
If the target server is also in the same pool for CLVM, that should not be an issue. When I was originally testing on CentOS 7, the kernel/targetcli complained when I had the volume mounted. I had to unmount it just to get as far as I did for you to see that same error repeatedly. I can tell you that whatever changes done to the (tcm_) qla2xxx drivers in Fedora work correctly with the qle2564 FC cards and targetcli so the issue seems isolated now to CentOS. I tested this just to see how it worked on Fedora. Is there a way in targetcli to be more generic like wildcards for the FC ports and ACLs? I'm stuck at the ACLs because I don't yet know all the addresses of the initiators. FreeNAS seems to do this automatically somehow. On that platform once set up, it does all ports when set up and regardless of what initiator is plugged into which port, it just works. Can this be done somehow for target or ACLs through targetcli? Here's what I have so far in testing Fedora while waiting on your team for the CentOS issue. |
|
In waiting, I wiped the server clean and installed FreeNAS to determine if the hardware was fine. The block device shows up so I know the hardware is fine. I'll reinstall the old config but surely there's something missing in the config. No other ideas? |
|
OS Reinstalled
In targetcli
Went to initiator
Am I missing any steps? |
|
Seems ok, I suppose the FC switch is configured correctly. Is there any error message in dmesg? |
|
Connected p2p |
I've googled this particular issue and not really finding any good results on how to resolve it.
I installed targetcli via yum on CentOS 7 and this is the version in the repo:
# targetclitargetcli shell version 2.1.fb49Copyright 2011-2013 by Datera, Inc and others.For help on commands, type 'help'.I downloaded the .zip of the targetcli-fb version and tried it too from the scripts folder and same results.
Python:
# python --versionPython 2.7.5Libs:
Installed Packagespython-configshell.noarch 1:1.1.fb25-1.el7 @basepython-rtslib.noarch 2.1.fb69-3.el7 @basepython-rtslib-doc.noarch 2.1.fb69-3.el7 @baseThe error comes up when using create.
/> /qla2xxx create naa.xxxxxxxxxxxxxxxCould not create Target in configFSBelow is the info on my FC
/> qla2xxx/ infoFabric module name: qla2xxxConfigFS path: /sys/kernel/config/target/qla2xxxAllowed WWN types: naaAllowed WWNs list: naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxxFabric module features: aclsCorresponding kernel module: tcm_qla2xxxI have attached the log but don't see anything that really stands out. log.txt
I checked per the link http://www.linux-iscsi.org/wiki/Fibre_Channel#Enable_target_mode to ensure that I have initiator mode disabled
# cat /sys/module/qla2xxx/parameters/qlini_modedisabledThe kernel modules are loaded.
# lsmod | grep qla2xxxtcm_qla2xxx 32768 1target_core_mod 335872 4 tcm_qla2xxx,iscsi_target_modqla2xxx 634880 1 tcm_qla2xxxscsi_transport_fc 65536 3 bfa,qla2xxx,tcm_qla2xxxconfigfs is mounted:
# cat /proc/mounts | grep configfsconfigfs /sys/kernel/config configfs rw,relatime 0 0In the kernel config:
CONFIG_CONFIGFS_FS=yKernel:
4.4.207-1.el7.elrepo.x86_64Should I be able to create a directory in /sys/kernel/config/target/qla2xxxx?
# mkdir naa.10008c7cffc7ef01mkdir: cannot create directory ‘naa.xxxxxxxxxxxxxxx’: Invalid argumentI did notice that when I rebooted, there was no qla2xxx folder in /sys/kernel/config/target/ but after trying to use targetcli, it did make one so I'm not sure what else would cause it to fail.
Suggestions?
The text was updated successfully, but these errors were encountered: