On Wed, Sep 19, 2018 at 9:13 PM John Barnard <john.barnard(a)broadcom.com>
wrote:
While testing our NVMf FC transport code, we ran into an issue with
an
spdk_ring we are using to share an FC resource between multiple pollers
running on a particular FC port (i.e. a multi-producer, multi-consumer
scenario). We were seeing corruption of the ring and the resources causing
the IO's to fail. When we debugged this problem it came down to the
difference in the rte_ring calls being made but SPDK enqueue/dequeue.
Here's the code snippet from lib/env_dpdk/env.c (with the highlighted
difference):
size_t
spdk_ring_enqueue(struct spdk_ring *ring, void **objs, size_t count)
{
int rc;
#if RTE_VERSION < RTE_VERSION_NUM(17, 5, 0, 0)
rc = rte_ring_mp_enqueue_bulk((struct rte_ring *)ring, objs, count);
if (rc == 0) {
return count;
}
return 0;
#else
rc = rte_ring_mp_enqueue_bulk((struct rte_ring *)ring, objs, count, NULL);
return rc;
#endif
}
size_t
spdk_ring_dequeue(struct spdk_ring *ring, void **objs, size_t count)
{
#if RTE_VERSION < RTE_VERSION_NUM(17, 5, 0, 0)
return rte_ring_sc_dequeue_burst((struct rte_ring *)ring, objs, count);
#else
return rte_ring_sc_dequeue_burst((struct rte_ring *)ring, objs, count,
NULL);
#endif
}
It seems that the spdk_ring_enqueue function calls
rte_ring_mp_enqueue_bulk(), while spdk_ring_dequeue function calls
rte_ring_sc_dequeue_burst(). When we change it to call
ret_ring_mp_dequeue_burst(), the problem went away. So my question is, why
this difference in the rte calls made by the SPDK? Is this a bug or on
purpose? Also, we noticed that there is no ring create flag (in
include/spdk/env.h) for multi-producer, multi-consumer (i.e.
SPDK_RING_TYPE_MP_MC), although it doesn't seem to matter if we pass the
SPDK_RING_TYPE_MP_SC flag to spdk_ring_create() (i.e. it's the call to
rte_ring_mp_dequeue_burst that's critical). Is there a reason this flag
was left out?
The two issues above are interrelated. SPDK seemingly does not care for
MP/MC rings, so it neither defines the flag to create the respective ring
type, nor appreciates this ring type under ring_dequeue (for enqueue, it
uses MP flavor which works for SP case as well).
The definitive answer as to why belongs to the SPDK team, while my guess is
that rings in SPDK are primarily (and solely, to the best of my knowledge)
for passing messages between SPDK threads where MP/MC scenario does not
apply.
HTH,
Andrey
I'm not familiar with the history of these functions, but there
seems to be
an inconsistency here and it's causing a failure for us.
Thanks,
John Barnard
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk