-----Original Message----- From: Jonathan Teh <jonathan.teh(a)mayadata.io&gt; Sent: Thursday, April 1, 2021 2:49 AM To: spdk(a)lists.01.org Subject: [SPDK] NVMe Multipath subsystem controller IDs I'm looking at using Linux's native NVMe multipathing with SPDK's nvmf target where multiple SPDK instances are used. As the subsystem issues controller IDs in the range [1, 0xFFEF], the initial connection to each individual target ends up with controller ID 1, leading to a clash for the connection to the 2nd, and subsequent, targets requiring multiple connections to "cycle" to the non-clashing ID. This issue that was previously discussed in https://lists.01.org/hyperkitty/list/spdk@lists.01.org/thread/ADLA6IPHVGV5LW 2AXZUBWLCYQW3HGOND/ but I don't think a resolution was agreed upon.

One approach would be to limit the controller ID range per subsystem (which was also previously proposed) which was implemented for the Linux kernel NVMe target in 5.7 as https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit... 4a39d61f80fcd679debda11e1ca02b88d90e67e. Would you consider a pull request to implement a similar feature in SPDK? An external orchestrator could then assign the subsystem in each SPDK instance a non-overlapping controller ID range.

On 01/04/2021 22:57, Walker, Benjamin wrote: > Exposing the same subsystem from two nodes is still certainly problematic. I think all of the > issues I listed in that thread are still valid (reservations, discovery, etc.) A better solution is likely > namespaces shared across subsystems, but that is not in the NVMe specification today. Understood, we have solutions being considered for other issues (essentially something to orchestrate the multiple SPDK instances). > Could we make an API in include/spdk/nvmf.h to allow the user to register a function pointer for generating > the controller ids? Maybe just stick a function pointer into spdk_nvmf_target_opts. Then you can use whatever > algorithm you want. How does that sound? That's certainly another option and gives maximum flexibility though it means a standalone, unmodified SPDK can only used the built-in controller ID generation function, perhaps ruling out some quick experiments. I have already implemented the per-subsystem controller ID range in SPDK and it is a fairly small patch. It gives some flexibility without having the complexity of having to supply your own implementation of controller ID generation and maintaining the state of that per-subsystem. _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org

Hi Jonathan, Ben,

I'm not an expert in this area but supporting multiple SPDK target instances is also important to iSCSI. I see the value of per-subsystem controller ID range and it is already added to Linux target. But we have little auto test for multiple SPDK target instances. Could you enhance test/nvmf/target/multitarget.sh to verify the feature?

And do we have negative side effect if we support per-subsystem controller ID range?