April 2018 - SPDK - Ml01.01.Org

Question regarding async event in target.

by Sreeni (Sreenivasa) Busam (Stellus)

Hello, I am looking at how to register async events from initiator to target and handle SSD controller errors in the NVMF target. From the code in NVMF, I see there is only 1 async request. Is it not possible to have multiple async requests registered from initiator and handled between initiator and target? Please let me know how to setup multiple requests at the same time. If you know how different controller errors can be handled, let me know. spdk_nvmf_session_async_event_request(struct spdk_nvmf_request *req) { struct spdk_nvmf_session *session = req->conn->sess; struct spdk_nvme_cpl *rsp = &req->rsp->nvme_cpl; assert(session->vcdata.aerl + 1 == 1); if (session->aer_req != NULL) { SPDK_TRACELOG(SPDK_TRACE_NVMF, "AERL exceeded\n"); rsp->status.sct = SPDK_NVME_SCT_COMMAND_SPECIFIC; rsp->status.sc = SPDK_NVME_SC_ASYNC_EVENT_REQUEST_LIMIT_EXCEEDED; return SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE; } session->aer_req = req; I appreciate your help. Thank you, Sreeni

4 years, 4 months

2
1
0 / 0

Comparing I/O Performance w/ and w/o FUA

by Fenggang Wu

Hi All, I am considering to test the performance impact of FUA (flushing the writeback cache inside the SSD for each write request) in the SPDK I/O performance. What is a recommended way to do so in SPDK? Correct me if I get it wrong. My guess is to use example/nvme/fio_plugin or example/nvme/perf. I think spck_bdev_flush() is the right function to call when we need to flush the SSD's writeback cache. I was looking for any flag in either fio_plugin or perf to trigger spdk_bdev_flush() but have not found it yet. So I suspect that I might not be on the right track. Thanks! -Fenggang

4 years, 4 months

4
5
0 / 0

About per client QoS (IOPS rate limiting)

by Cao, Gang

Hi all, There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on. At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side. Based on Paul's great stuff of the Passthru vbdev, I've updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev. https://review.gerrithub.io/#/c/406888/ vbdev: update and have a create_passthru_disk function https://review.gerrithub.io/#/c/406891/ vbdev: add the construct_passthru_bdev RPC method https://review.gerrithub.io/#/c/406977/ vbdev: make the passthru vbdev share the same underlying bdev The possible usage as following: [Take NVMe-oF target as example] 1. In the nvmf.conf file, # Configure the Passthru vbdev on same bdev like Malloc0 [Passthru] PT Malloc0 PT0 PT Malloc0 PT1 # The subsystem is configured with shareable bdev by Passthru vbdevs [Subsystem2] NQN nqn.2016-06.io.spdk:cnode2 Listen RDMA 192.168.2.21:4420 AllowAnyHost No Host nqn.2016-06.io.spdk:init SN SPDK00000000000002 Namespace PT0 >>>> This can be connected by one client Namespace PT1 >>>> This can be connected by another client # Assign different QoS IOPS limiting on Passthru vbdevs [QoS] Limit_IOPS PT0 20000 Limit_IOPS PT1 30000 2. Use RPC method to add the Passthru vbdev at runtime a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3 b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3 c. python ./scripts/rpc.py enable_bdev_qos PT3 20000 If there is any concern or comment, please feel free to let me know. Thanks, Gang

4 years, 4 months

4
15
0 / 0

vhost target failover?

by Huang Frank

Hi，all What's the failover mechism of vhost target? If the vhost target goes to crash, how to make sure the guest can process without affectting? Thanks, Frank. ________________________________ kinzent(a)hotmail.com

4 years, 4 months

2
3
0 / 0

Re: [SPDK] anyone ran the SPDK ( app/iscsi_tgt/iscsi_tgt ) with VPP?

by Isaac Otsiabah

Hi Tomasz, I configured vpp and I thought it was working because I can ping the initiator machine (192.168.2.10) on my lab private network. (On target Server): vpp# show int address TenGigabitEthernet82/0/0 (up): 192.168.2.20/24 local0 (dn): vpp# vpp# show int Name Idx State Counter Count TenGigabitEthernet82/0/0 1 up rx packets 58 rx bytes 5342 tx packets 58 tx bytes 5324 drops 71 (drops were expected because I did a ping to itself on 192.168.2.20) ip4 49 local0 0 down vpp# ping 192.168.2.10 64 bytes from 192.168.2.10: icmp_seq=1 ttl=64 time=.1229 ms 64 bytes from 192.168.2.10: icmp_seq=2 ttl=64 time=.0412 ms 64 bytes from 192.168.2.10: icmp_seq=3 ttl=64 time=.0733 ms 64 bytes from 192.168.2.10: icmp_seq=4 ttl=64 time=.0361 ms 64 bytes from 192.168.2.10: icmp_seq=5 ttl=64 time=.0332 ms Statistics: 5 sent, 5 received, 0% packet loss However, something is missing because i started the iscsi_tgt server (below) and executed add_portal_group RPC command to add a portal group and got a bind() error below. Iscsi.conf is attached (portal group and target sections are commented out). The iscsi_tgt binary was linked to the same vpp library that I built. [[email protected] spdk]# ./scripts/rpc.py add_portal_group 1 192.168.2.20:3260 Got JSON-RPC error response request: { "params": { "portals": [ { "host": "192.168.2.20", "port": "3260" } ], "tag": 1 }, "jsonrpc": "2.0", "method": "add_portal_group", "id": 1 } response: { "message": "Invalid parameters", "code": -32602 } [[email protected] spdk]# ./app/iscsi_tgt/iscsi_tgt -m 0x0101 Starting SPDK v18.04-pre / DPDK 18.02.0 initialization... [ DPDK EAL parameters: iscsi -c 0x0101 --file-prefix=spdk_pid8974 ] EAL: Detected 16 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Multi-process socket /var/run/.spdk_pid8974_unix EAL: Probing VFIO support... EAL: VFIO support initialized app.c: 443:spdk_app_start: *NOTICE*: Total cores available: 2 reactor.c: 645:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x3 reactor.c: 429:_spdk_reactor_run: *NOTICE*: Reactor started on core 8 on socket 1 reactor.c: 429:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0 copy_engine_ioat.c: 318:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme EAL: PCI device 0000:03:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme EAL: PCI device 0000:04:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme EAL: PCI device 0000:05:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme init_grp.c: 107:spdk_iscsi_init_grp_add_initiator: *WARNING*: Please use "ANY" instead of "ALL" init_grp.c: 108:spdk_iscsi_init_grp_add_initiator: *WARNING*: Converting "ALL" to "ANY" automatically posix.c: 221:spdk_posix_sock_create: *ERROR*: bind() failed, errno = 99 posix.c: 231:spdk_posix_sock_create: *ERROR*: IP address 192.168.2.20 not available. Verify IP address in config file and make sure setup script is run before starting spdk app. portal_grp.c: 190:spdk_iscsi_portal_open: *ERROR*: listen error 192.168.2.20.3260 iscsi_rpc.c: 969:spdk_rpc_add_portal_group: *ERROR*: portal_grp_open failed From: Zawadzki, Tomasz [mailto:[email protected]] Sent: Thursday, April 12, 2018 12:27 AM To: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com> Cc: Harris, James R <james.r.harris(a)intel.com>; Verkamp, Daniel <daniel.verkamp(a)intel.com>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com> Subject: RE: anyone ran the SPDK ( app/iscsi_tgt/iscsi_tgt ) with VPP? Hello Isaac, Are you using following patch ? (I suggest cherry picking it) https://review.gerrithub.io/#/c/389566/ SPDK iSCSI target can be started without specific interface to bind on, by not specifying any target nodes or portal groups. They can be added later via RPC http://www.spdk.io/doc/iscsi.html#iscsi_rpc. Please see https://github.com/spdk/spdk/blob/master/test/iscsi_tgt/lvol/iscsi.conf for example of minimal iSCSI config. Suggested flow of starting up applications is: 1. Unbind interfaces from kernel 2. Start VPP and configure the interface via vppctl 3. Start SPDK 4. Configure the iSCSI target via RPC, at this time it should be possible to use the interface configured in VPP Please note, there is some leeway here. The only requirement is having VPP app started before SPDK app. Interfaces in VPP can be created (like tap or veth) and configured at runtime, and are available for use in SPDK as well. Let me know if you have any questions. Tomek From: Isaac Otsiabah [mailto:[email protected]] Sent: Wednesday, April 11, 2018 8:47 PM To: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com<mailto:[email protected]>> Cc: Harris, James R <james.r.harris(a)intel.com<mailto:[email protected]>>; Verkamp, Daniel <daniel.verkamp(a)intel.com<mailto:[email protected]>>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com<mailto:[email protected]>> Subject: anyone ran the SPDK ( app/iscsi_tgt/iscsi_tgt ) with VPP? Hi Tomaz, Daniel and Jim, i am trying to test VPP so build the VPP on a Centos 7.4 (x86_64), build the SPDK and tried to run the ./app/iscsi_tgt/iscsi_tgt application. For VPP, first, I unbind the nick from the kernel as and start VPP application. ./usertools/dpdk-devbind.py -u 0000:07:00.0 vpp unix {cli-listen /run/vpp/cli.sock} Unbinding the nic takes down the interface, however, the ./app/iscsi_tgt/iscsi_tgt -m 0x101 application needs an interface to bind to during startup so it fails to start. The information at: "Running SPDK with VPP VPP application has to be started before SPDK iSCSI target, in order to enable usage of network interfaces. After SPDK iSCSI target initialization finishes, interfaces configured within VPP will be available to be configured as portal addresses. Please refer to Configuring iSCSI Target via RPC method<http://www.spdk.io/doc/iscsi.html#iscsi_rpc>." is not clear because the instructions at the "Configuring iSCSI Traget via RPC method" suggest the iscsi_tgt server is running for one to be able to execute the RPC commands but, how do I get the iscsi_tgt server running without an interface to bind on during its initialization? Please, can anyone of you help to explain how to run the SPDK iscsi_tgt application with VPP (for instance, what should change in iscsi.conf?) after unbinding the nic, how do I get the iscsi_tgt server to start without an interface to bind to, what address should be assigned to the Portal in iscsi.conf.. etc)? I would appreciate if anyone would help. Thank you. Isaac

4 years, 4 months

1
2
0 / 0

SPDK - multi process DPDK? Or single process?

by Sai Rajesh

HI, I'm relatively new to SPDK. Can someone help me answer some of my questions. In my experiment, I'm running "perf" application in a loop via script. #!/bin/bash count=1 while true; do echo $count ./perf -q 64 -s 131072 -w write -t 300 -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.22.55 trsvcid:4420' let count++ done When I do this way, after 1hour into runs, I hit "remote access error". https://github.com/spdk/spdk/issues/264/. As if someone has freed the memory while its still being used by RNIC. After debugging for long, I found that disabling ASLR made my script run for 3days without any issues. In that context, I have following questions 1. Do SPDK run as DPDK's single process or multi-process? 2. Why disabling ASLR worked for me? 3. When I allocate task buffer via spdk_dma_zmalloc() in perf application, does RNIC device save these memory mappings and refer them during completion processing? 4. In my failure case, with ASLR enabled, does this memory mapping changed over the fly and RNIC now, was not able to access task buffer? Thanks for your time and patience. Thanks, Rajesh.

4 years, 4 months

3
4
0 / 0

Trying to recover from one SPDK process crashing in a multi-process environment

by Cao, Gang

Hi all, This topic is regarding the usage of SPDK NVMe driver in the multi-process mode. There are some cases that any process (primary or secondary) could exit in the unexpectedly way to leave the allocated memory not released, the held lock also not released or even some severe problem like in the middle of the memory allocation. Some of these issues may have a way to solve and others may be difficult to solve. Would like to initiate some discussion on this topic to get more input, suggestions and comments. For example, we may need some mechanism to track the per process memory allocation through those memory related APIs (called DPDK's) so that these memory has the hint to release once its associated process exits. Some co-work will be covered by SPDK and DPDK memory management. Any feedback is welcomed. Thanks, Gang

4 years, 4 months

4
4
0 / 0

difference between the vhost-blk and vhost-scsi?

by Huang Frank

hi,ALL I find that the vhost-target can offer two type of devices: vhost-scsi and vhost-blk. What's the difference? Which one should I used? Thanks, Frank ________________________________ kinzent(a)hotmail.com

4 years, 4 months

2
1
0 / 0

nvmf/rdma: handling QP errors

by Philipp Skadorov

Hi there, Looking at the list of errors ibv_get_async_event offers - would it make sense to monitor them in the polling loops? Right now, per my understanding, the only way SPDK gets that things go wrong is: when initiator issues disconnect. Any thoughts? Regards, Philipp

4 years, 4 months

2
1
0 / 0

Paying More Attention to Nightly Build Status

by Luse, Paul E

All- As many of you know, there's been a lot of effort over the past months to beef up our CI infrastructure, restructure our test directories, refactor some test scripts, add new tests, etc. I wanted to take a minute to highlight the need for community ownership when it comes to code quality. Ultimately we all rely on those voted in as maintainers to assure that nothing gets checked into the database that doesn't meet with SPDK standards. However, we are working with a large, diverse and fast moving project where we are seeing the size of the community grow rapidly which is awesome as it underscores the value of the code base and continued contributions from all of you! At the same time, however, this growth requires all of us to do our part in monitoring the quality of the project. That means spending more time on reviews, asking questions, learning and essentially holding each other accountable for keeping SPDK on track. To that end, a few new links are available on the CI website https://ci.spdk.io/ which is a link right off of http://www.spdk.io/. Please take a minute to review the info below and help keep SPDK code quality up by checking in on these links often, asking questions and helping debug as much as you can! Thanks!! Paul NIGHTLY BUILD STATUS ------------------------------- As most of you are aware, every patch goes through a battery of tests before it's committed to the repo. We need to keep the length of those tests to something reasonable in order to promote timely contributions so in addition to the per patch testing, every night a much longer set of tests is executed against the tip. Previously it was possible to find these test results but it wasn't super visible. Now it is! If you click on the "SPDK Test Pool Status" link off the CI page (https://ci.spdk.io/spdk/status/) you will now see the status of the last nightly build as shown below: [cid:[email protected]] NIGHTLY BUILD HISTORY --------------------------------- You can also click on the link in the upper right corner of that page called "Nightly Build" (https://ci.spdk.io/spdk/nightly_status.html) and see a history of recent nightly builds [cid:[email protected]]

4 years, 4 months

1
0
0 / 0

2022

2021

2020

2019

2018

2017

2016

SPDK April 2018