Thank you for answering my question. Now I feel not that bad. I thought maby I was doing
something totally wrong.
I'll see if I can contribute.
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Verkamp, Daniel
Sent: Thursday, February 9, 2017 3:19 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Question for SGL Descriptor Programming with SPDK NVMF Target
The SPDK NVMe-oF target currently only supports a single SGL descriptor per capsule
(command). This is indicated in the Identify Controller data structure MSDBD field
(Maximum SGL Data Block Descriptors). I believe the Linux NVMe-oF target has the same
Multiple SGL descriptors per command could be implemented, but nobody is currently working
on this feature. Patches are welcome if you want to add it.
The correct way to represent multiple SGLs for the RDMA transport in particular (if it
were actually supported) would be:
- SGL1 (the one contained within the 64-byte SQE) would be a SGL Last Segment
descriptor (type 3h) with an Offset (subtype 1h) describing the segment of SGLs within the
in-capsule data (e.g. offset = ICDOFF, length = 16 * number of SGL Keyed Data Block
- Within the capsule, starting at ICDOFF (in-capsule data offset) (0 for SPDK),
would be a list of SGL Keyed Data Block descriptors (type 4h) with subtype Address (0h) or
Invalidate Key (Fh).
This is almost the same as your first example, except the Last Segment descriptor must be
an Offset subtype (the RDMA transport spec doesn’t support SGLs in host memory, only
From: SPDK [mailto:email@example.com] On Behalf Of Wenhua Liu
Sent: Thursday, February 9, 2017 3:23 PM
Subject: [SPDK] Question for SGL Descriptor Programming with SPDK NVMF Target
Hi SPDK experts,
I'm using SPDK NVMF target to test my client code and run into an issue when
transferring data described my multiple SGLs.
To better describe my issue, here I assume the data buffer is represented by variable
dataSGL which containts 3 SGL elements:
dataSGL: addr=0x1000, length=0x800
dataSGL: addr=0x3000, length=0x2200
dataSGL: addr=0x8000, length=0x1300
According to the NVMe over Fabrics speciciation (section 7.3.2), RDMA transport supports
SGL Data Block, SGL Last Segment, and Keyed SGL Data Block descriptors only.
The first time I used SGL Last Segment descriptor. I created a buffer of size
3*sizeof(NVME_SGL_DESCRIPTOR) which is 48 bytes and put the above 3 dataSGLs in the
buffer, set descriptor type to Keyed SGL Data Block for all and subtype to 0 (address),
filled in key. Then I set SQE.dptr.sgl1's descriptor type to SGL Last Segment and
subtype to 0 (address) and address field to the address of the buffer containing 3
dataSGLs. The size field is set to 48 indicating this Last SGL Segment contains 3 SGL
descriptors. When doing RDMA send, the above buffer was also sent to the target side as
part of RDMA send call. SPDK target received the command and failed it with error
message "Invalid NVMf I/O Command SGL: Type 0x3, Subtype 0x0". This is apparent
if looking at SPDK source code as it does not support SGL Last Segment descriptor.
The second time I used SGL Data Block descriptor. Comparing to the first case, the only
difference is in SQE.dptr.sgl1, I set descriptor type to SGL Data Block and subtype to 1
(address field contains offset) and address field to 0 (offset is 0). I was hoping the
target would interpret the 48 byte in-capsule data as 3 Keyed SGL descriptors. This time,
SPDK target said buffer too small. It expected a buffer with size (0x800+0x2200+0x1300),
but SQE.dptr.sgl1 indicated the buffer size is 48.
The third time I used Keyed SGL Data Block descriptor. According to the NVMe over Fabrics
spec (section 2.3.2), "In this case, there is one SGL descriptor that is contained
within the Submission Queue Entry at CMD.SGL1. The SGL descriptor is a Keyed SGL Data
Block descriptor. If more SGLs were needed to complete the command, the additional SGLs
would be contained in the command capsule.". Following this example, I put dataSGL
in SQE.dptr.sgl1 by setting its descriptor type to Keyed SGL Data Block, subtype to 0
(address), address to 0x1000, length to 0x800 and the key to some value. I then created a
buffer of size 2*sizeof(NVME_SGL_DESCRIPTOR) and put dataSGL and dataSGL into it the
same way I put dataSGL into SQE.dptr.sgl1. When doing RDMA send, this buffer was also
sent to the target as in-capsule data of size 32. SPDK target received it and failed it
again saying buffer too small. It expected a buffer with size(0x800+0x2200+0x1300), but
SQE.dptr.sgl1 indicated the buffer size is 0x800.
As I still have trouble connecting to other targets, I do know how other targets will
behave. I have a few questions hoping someone can answer/explain:
1. Of the above 3 approaches, is there one that is correct?
2. If none of them is correct, what is the right way prepare NVME SGL descriptors for the
data described by multiple SGLs?
3. How should multiple SGLs be sent to target using RDMA transport? Using in-capsule
4. When receiving a NVME command (and maybe with some in-capsule data), how does target
know that data is real data or SGLs?
5. Should a NVME target (RDMA based) support all 3 SGL descriptors or can target choose
what it wants?
6. How does SPDK NVMF target handles multiple command containing data described by
multiple SGLs? Which piece of code handles this?