I just joined the SPDK email list after dabbling with the SPDK for a few weeks. I looked
through the SPDK email list’s archives and didn’t see any subjects focused on the topic of
order of operations in NVMe completions, hence my inaugural posting to this list begins
with a question about this. I did see and read through the thread in July 2016 (Re:
Callback passed to spdk_nvme_ns_cmd_read not being called sometimes) but it didn’t touch
on the specific question I have. Please accept my apologies in advance if I missed
something else related to NVMe completions when scanning through the SPDK email archives.
In the SPDK NVMe driver I noted that the order of operations in processing completions is
1. If a callback was provided, issue the callback.
2. Invoke nvme_free_request().
As the caller’s callback info was saved in the nvme_request struct allocated during the
I/O request, it appears logical that one would reference those respective callback fields
and invoke the callback, before freeing the request struct that contains them. However,
I’m curious to know if there’s any risk or possible concern for diverging from the
programming model if the following sequence is done instead:
1. Save off the req->cb_fn and req->cb_arg in new temporary pointers (local
variables on the stack of the completion function).
2. Invoke nvme_free_request() for the req.
3. If a callback was provided (as saved in the new temporary pointer), issue the
callback via the temporary saved function pointer and with the temporary ptr to the
I’m inquiring about this alteration for the scenario where in the I/O completion callback
code path, the application may (and likely will) wish to issue a new I/O. However, if
this is the first completion invoked after a maximum number of requests (based on
opts->io_queue_requests) have already been submitted, then the I/O callback’s attempt
to issue a new I/O will fail because there are no free requests available based on the
current implementation of the completion code. If on the other hand, the completion code
is altered to invoke nvme_free_request() before calling the applications’s I/O callback, a
request entry will be available to that application as it was just freed, thus enabling it
to issue a new I/O successfully.
I made such a change in my sandbox and this appeared to operate, but as I’m new to the
SPDK NVMe driver I’m eager to solicit feedback from the seasoned developers here to learn
if I’m overlooking something.