We are building a setup which exposes a local non nvme SSD connected via SAS Megaraid
controller as a NVMeF device, using libaio bdev, I added the entire setup detail bellow.
When doing single threaded IO, everything is OK, but when running multithreaded IO
(vdbench) with data verification (-vr: verify via read after each write) we get IO
Looking at the IO errors, the problem seems to be that the data buffer given by the read
This is identified by vdbench which fills the buffer with a predefined pattern which
should not be in the buffer post read.
As written above, this does not happen when I do single threaded IO. In addition, if we
read the contents from the error LBA using dd, the contents are correct.
We ran the same vdbench workload directly on the disk without any issue, in addition, we
tried replacing the physical device with a RAM disk, and with a SATA device directly
connected (not via megaraid controller), and the problem does not recreate (i.e. no IO
Here is the vdbench error:
16:34:59.377 localhost-0: 16:34:59.376 op: read lun: /dev/nvme0n1 lba: 789422931968
0xB7CD460000 xfer: 131072 errno: BAD_READ_RETURN: 'Read was successful, but data
buffer contents not changed'
P.S. system information and configuration details:
The setup is: Supermicro server.
8xSSD’s aggregated to single RAID0 volume configured 64K stripe size with WT and DirectIO.
Presented to OS by megaraid module.
CentOS 7.6 + OFED-4.6-1.0.1
SPDK configuration script we are using:
/root/spdk/app/nvmf_tgt/nvmf_tgt -m 0xF000000 &
spdk/scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:jbof01 -a -s 000000000 -d
spdk/scripts/rpc.py construct_aio_bdev /dev/sdc megaraid
spdk/scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:jbof01 megaraid
spdk/scripts/rpc.py nvmf_create_transport -t RDMA -u 8192
spdk/scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:jbof01 -t rdma -a
172.20.95.222 -s 1023
nvme connect -t rdma -s 1023 -a 172.20.95.222 -n nqn.2016-06.io.spdk:jbof01
Vdbench input file:
** If we change the threads to 1, there is no issue.
Show replies by date