Thanks for your reply.
Actually, I'm seeing difference in performance w.r.t Host read vs write commands, my
configuration details are -
1. Three CentOS Linux host (kernel - 3.10.0-957 ).
2. io tool - FIO
2. Three raid pool with 4 drives each, single volume carved out from every pool and
exposed to each initiator.
Note: 4 SSD drives pool is capable of reaching up to 2 M IOPS for read and write.
I see aggregate of 4K read workload from all three initiator is not going beyond 2.7 M
IOPS, network bandwidth is 100 Gbps, so we should be able to reach 3M IOPS.
However, if I change workload in one of the initiator from 4K read to 4K write, aggregate
IOPS touches 3M IOPS. This is the reason I was thinking may be in capsule data used in
Host write command is making the difference here. Please let me know your thoughts on
The reason I'm asking about the in-capsule data for Host read response is
From: SPDK [mailto:firstname.lastname@example.org] On Behalf Of Walker, Benjamin
Sent: Monday, June 24, 2019 2:11 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] RDMA in-line data transfer for host READ operations.
From: SPDK [mailto:email@example.com] On Behalf Of Kumar,
Sent: Thursday, June 20, 2019 1:04 PM
Subject: [SPDK] RDMA in-line data transfer for host READ operations.
I was looking at the NVMe Data-in and Data-out flow for NVMeOF RDMA
transport, looks like SPDK is capable of handling in-capsule data (for
Write operations from the host) by pre-posting in-capsule data buffer
SGL to RDMA recv buffers. But, looks like SPDK doesn't use in-line
data buffers while sending response to the NVMe read commands from the
host and eventually using RDMA write to send the data, can someone
please confirm whether my observation is correct or not?
In the NVMe-oF specification, there is no in-capsule data on the completion side. The
target simply posts the data as an RDMA write, then sends the completion immediately by
using an RDMA send. There wouldn't be any performance advantage to stuffing the data
into the RDMA send command.
At the RDMA level there are some things like IBV_SEND_INLINE that may help with
performance in some cases, but that's not in-capsule data.
SPDK mailing list
SPDK mailing list