On Wed, Aug 1, 2018, 20:55 Harris, James R <james.r.harris@intel.com> wrote:

Hi Andrey,


Your gut is not fooling you.  I think doing this in the context of an existing polled mode thread is a great idea.  (But I’m glad that the spdk_allocate_thread() approach was able to work for you!)

Yeah, it works nice, albeit with some (anticipated) performance degradation. 

On a separate note, I didn't notice spdk_mempool to provide an expected miltiple consumers/single producer flavor of the dpdk mempool_create (which I've utilized for inter-thread messages), and from my experiments with replacing spdk _pool_create with dpdk call (with F_SP_MC set) I saw a few percent improvement in IOPS against an malloc bdev.


You could just always pick the master thread.  This is the thread that calls your module’s init() function. 

That's not necessarily the most convenient option as one might then end up with all ST contexts being run by the same master thread (or that init thread might turn out to be just a typical external init thread that brings things up and then goes sleeping until needed). 

Ideally, I'd like to be able to pick up a pinned thread on a chosen core, and attach to that one w/o any dependency on what thread has brought things up (unless explicitly told to pick up that init thread).

You can call spdk_poller_register() from your init function to start pollers – and pick a polling frequency that makes sense for the expected rate of synchronization messages.  You can also save spdk_get_thread() to a global variable so that you can do spdk_thread_send_msg() calls to it from other threads – or maybe you create a multi-producer/single-consumer ring that other threads can enqueue to, and then the poller picks up those messages, executes them, and sends a message back to the calling thread.


There are a lot of cases in SPDK currently where we start up pollers on a specific thread at application start – i.e. RPC handling (lib/event/rpc.c) and iSCSI acceptors (lib/iscsi/acceptor.c) immediately come to mind.  The virtio bdev driver (lib/bdev/virtio/bdev_virtio_scsi.c) also uses this poller + ring concept to synchronize reset operations.

Thanks for the pointers, that's enough to chew upon.



Would this be sufficient for your use case?






From: SPDK <spdk-bounces@lists.01.org> on behalf of Andrey Kuzmin <andrey.v.kuzmin@gmail.com>
Reply-To: Storage Performance Development Kit <spdk@lists.01.org>
Date: Tuesday, July 31, 2018 at 2:31 AM
To: Storage Performance Development Kit <spdk@lists.01.org>
Subject: [SPDK] Single-threaded semantics in vbdev


Hi guys,


Working on a vbdev that maintains some global state and underlying data structures, I decided to explore an SPDKish alternative to the traditional synchronization, that of the single-threaded semantics where global data structures are being maintained by a dedicated thread, with requests being served by message passing to/from that thread. 


Coding such a thread against the spdk_allocate_thread() guidelines proved to be quite straightforward, and I've got it up and running. At the same time, I'm still unclear as to why I should be essentially duplicating the features of the pinned SPDK thread, with my gut feeling being that the very same single-threaded semantics could have been easily achieved by pinning my execution context to some randomly or round-robin chose pinned thread of the configured allowed cores mask.


If that gut feeling doesn't fool me, I will appreciate a pointer to any docs and/or examples showing how one can pin an arbitrary execution context inside a vbdev to an SPDK native thread (similar to app_start and friends, I believe), and do polling on that thread/ messaging to/from that thread similar to what a POSIX thread equipped with spdk thread envelope could do.

Thanks in advance,

SPDK mailing list