On Wed, Aug 1, 2018, 23:59 Harris, James R <email@example.com> wrote:
Your gut is not fooling you. I think doing this in the context of an existing polled mode thread is a great idea. (But I’m glad that the spdk_allocate_thread() approach was able to work for you!)
Yeah, it works nice, albeit with some (anticipated) performance degradation.
On a separate note, I didn't notice spdk_mempool to provide an expected miltiple consumers/single producer flavor of the dpdk mempool_create (which I've utilized for inter-thread messages), and from my experiments with replacing spdk _pool_create with dpdk call (with F_SP_MC set) I saw a few percent improvement in IOPS against an malloc bdev.
That could be added as an extension to the SPDK env layer. So I assume in your case, only your extra thread is putting buffers back into the mempool?Exactly, as I've got a single thread managing shared state (that's to your next question below, with that state being, for instance, a logical-to-physical address translation or anything else that needs to be accessed by miltiple client threads) that receives messages and puts them back to the pool once processed.
You could just always pick the master thread. This is the thread that calls your module’s init() function.
That's not necessarily the most convenient option as one might then end up with all ST contexts being run by the same master thread (or that init thread might turn out to be just a typical external init thread that brings things up and then goes sleeping until needed).
Can you describe these ST contexts some more, and how and when they are created? Certainly, if you have a lot of them, and they are processing lots of messages, putting it all on one thread is not ideal.Imagine multiple vbdevs on the same system. I definitely would want to avoid pooling them all up on the same core.
Ideally, I'd like to be able to pick up a pinned thread on a chosen core, and attach to that one w/o any dependency on what thread has brought things up (unless explicitly told to pick up that init thread).
You may want to explore spdk_for_each_thread() – this would allow you to iterate through all of the threads in the SPDK application and you could build something based off of that.Thanks, that's a very good point.Regards,Andrey