Hi, Ziye and Feng:
I have found the performance degradation root cause. SPDK still keep good
performance of RWMix at 19.01 version. I found the performance degrade
cause by our application. In our application, we create 8 POSIX thread and
run at CPU vcore 22 ~ 29. There is a bug that our threads will busy and
don't know why it affect SPDK iSCSI performance (VCORE 4 ~ 9, 20, 21) with
null block device.
After fix the bug, our thread won't busy and SPDK iSCSI RWMix performance
with null block device back to normal.
(NOTE: all these vcores locate at CPU socket 0, I still feel wired that
performance got affected)
Thank you for supporting.
On Fri, Jun 14, 2019 at 9:34 AM Lego Lin <lego.lin(a)gmail.com> wrote:
Many thanks for your quick response. I will try it.
On Fri, Jun 14, 2019 at 9:33 AM Yang, Ziye <ziye.yang(a)intel.com> wrote:
> Hi Lego,
> Better to upgrade to 19.04 or 19.01.1. The issue you reported is similar
> to the issue reported by others through the following issue:
> I think that you observe the similar issue. So would you please read the
> link I provided.
> Best Regards
> Ziye Yang
> -----Original Message-----
> From: SPDK [mailto:firstname.lastname@example.org] On Behalf Of Lego Lin
> Sent: Friday, June 14, 2019 9:29 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Performance degrade of RWMix when upgrade SPDK to 19.01
> Hi, all:
> Our team build up our storage system base on SPDK. Recenly, we try to
> upgrade SPDK to 19.01, but we got performance degradation when upgrade to
> SPDK 19.01.
> I have compared our the reactor implementation in lib\event\reactor.c
> code base with different SPDK version (18.04, 18.07, 18.10, 19.01). The
> implementation is close to 18.04, a little bit different with 18.07, 18.10,
> but big difference with 19.01.
> The test environment as following:
> 1. hugemem size = 64G
> 2. Network environment: 4*10G and connect nic card directly without
> switch 3. HW info: 2.2GHz with 2 CPU socket with total 40 vcore 4. total
> memory: 256G 5. CPU assignment for SPDK: total vcore for SPDK, 8 vcore of
> CPU socket 0 for SPDK iSCSI 6. Use SPDK null block device to evaluate
> performance 7. We got the code base (include dpdk) at 2018/01 8. Create 16
> targets and 16 null block devices, mini connections per core = 2, each
> physical connections has 4 iSCSI connections. The max QD of iSCSI is 128.
> 9. FIO test pattern: T1(BS=4K, 100%Randwrite, QD=32), T2(BS=4K,
> 100%Randread, QD=32), T3(BS=4K, 70%RandRead30%RandWrite, QD=32)
> Our code base
> T1 (100%RandWrite): 1170 KIOPS
> T2 (100%RandWrite): 1189 KIOPS
> T3 (100%RandWrite): 941 KIOPS (Read), 403 KIOPS (Write)
> Merge reactor.c from 18.07 to our code base
> T3 (100%RandWrite): 948 KIOPS (Read), 406 KIOPS (Write)
> Merge reactor.c from 18.10 to our code base
> T3 (100%RandWrite): 933 KIOPS (Read), 400 KIOPS (Write)
> Our code base upgrade to SPDK 19.01
> T1 (100%RandWrite): 1150 KIOPS
> T2 (100%RandWrite): 1187 KIOPS
> T3 (100%RandWrite): 826 KIOPS (Read), 354 KIOPS (Write)
> The RWMix performance has obvious degrade.
> Can anyone give me some idea about possible root cause for RWMix
> performance tunning?
> Is it related to DPDK version or SPDK new reactor implementation?
> Thank you.
> SPDK mailing list
> SPDK mailing list