Our results were also pretty much align with your findings. But CPU utilization was way
less in SPDK target compared to Kernel mode target.
It would be nice to learn if anyone else has seen better results on kernel mode initiator
with SPDK target.
From: SPDK [mailto:firstname.lastname@example.org] On Behalf Of Rajat Maheshwari
Sent: Thursday, March 09, 2017 12:53 AM
Subject: Re: [SPDK] NVMeoF IO latency performance apps
We are using SPDK on target end, but kernel Initiator. We used fio, but found some
unexpected results. (latency figures almost similar for SPDK target & Kernel Target,
besides we are using NULL block devices too, so media latency is not involved here)
So, I would like to check for certain fio parameters, if in case we are missing
couple of samples from fio "read I/O" run (SPDK NVMe-oF target, Kernel host)
(fio command line:
fio --bs=4k --numjobs=16 --iodepth=4 --loops=1 --ioengine=libaio --direct=1 --invalidate=1
--fsync_on_close=1 --randrepeat=1 --norandommap --time_based --runtime=10
--filename=/dev/nvme0n1 --name=read-phase --rw=read )
slat (usec): min=1 , max=43 , avg= 3.61, stdev= 1.77
clat (usec): min=40 , max=414 , avg=233.02, stdev=28.75
lat (usec): min=47 , max=416 , avg=236.73, stdev=28.79
slat (usec): min=1 , max=41 , avg= 3.63, stdev= 1.77
clat (usec): min=71 , max=473 , avg=232.60, stdev=28.65
lat (usec): min=74 , max=476 , avg=236.33, stdev=28.70
slat (usec): min=1 , max=48 , avg= 3.64, stdev= 1.82
clat (usec): min=17 , max=477 , avg=233.10, stdev=28.87
lat (usec): min=21 , max=479 , avg=236.84, stdev=28.93
All above samples (infact every sample) shows maximum clat above 380us. Also, the average
clat is never below 230us.
We also compared it using kernel target (& Kernel host) and apparently we didn't
see much difference between two runs.
Max clat range was almost in the range (only 10-20usec higher than SDPK target), but
average clat was, almost, exactly the same (230-240us).
Is it really expected? (I believe, its not!)
So, are we missing any fio flag(s)?
On Mon, 2017-03-06 at 23:04 +0530, Rajat Maheshwari wrote:
Which all supported IO utilities are available for NVMe-over-Fabric
Also, does all these applications support parallel exploitation of
The majority of people use fio. Are you planning to use the SPDK NVMe-
oF target, the NVMe-oF host, or both? If you are using just the SPDK
NVMe-oF target with the kernel host, fio is an excellent tool.
SPDK mailing list
SPDK mailing list