Question regarding async event in target.
by Sreeni (Sreenivasa) Busam (Stellus)
Hello,
I am looking at how to register async events from initiator to target and handle SSD controller errors in the NVMF target. From the code in NVMF, I see there is only 1 async request. Is it not possible to have multiple async requests registered from initiator and handled between initiator and target?
Please let me know how to setup multiple requests at the same time. If you know how different controller errors can be handled, let me know.
spdk_nvmf_session_async_event_request(struct spdk_nvmf_request *req)
{
struct spdk_nvmf_session *session = req->conn->sess;
struct spdk_nvme_cpl *rsp = &req->rsp->nvme_cpl;
assert(session->vcdata.aerl + 1 == 1);
if (session->aer_req != NULL) {
SPDK_TRACELOG(SPDK_TRACE_NVMF, "AERL exceeded\n");
rsp->status.sct = SPDK_NVME_SCT_COMMAND_SPECIFIC;
rsp->status.sc = SPDK_NVME_SC_ASYNC_EVENT_REQUEST_LIMIT_EXCEEDED;
return SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE;
}
session->aer_req = req;
I appreciate your help.
Thank you,
Sreeni
1 year, 11 months
Comparing I/O Performance w/ and w/o FUA
by Fenggang Wu
Hi All,
I am considering to test the performance impact of FUA (flushing the
writeback cache inside the SSD for each write request) in the SPDK I/O
performance. What is a recommended way to do so in SPDK?
Correct me if I get it wrong. My guess is to use example/nvme/fio_plugin or
example/nvme/perf. I think spck_bdev_flush() is the right function to call
when we need to flush the SSD's writeback cache. I was looking for any flag
in either fio_plugin or perf to trigger spdk_bdev_flush() but have not
found it yet. So I suspect that I might not be on the right track.
Thanks!
-Fenggang
1 year, 11 months
About per client QoS (IOPS rate limiting)
by Cao, Gang
Hi all,
There has been some work done to have SPDK support the QoS per bdev which means that each bdev has its assigned IOPS rate limiting. This QoS functionality is below the storage protocols like NVMe-oF, iSCSI and so on.
At the time of last PRC SPDK summit, there are some questions on the per client QoS (i.e., the NVMe-oF iniator / host) besides the per bdev QoS. Each client like NVMe-oF initiator can see its own IOPS rate limiting not matter whether the connected device is exclusive or shared at the target side.
Based on Paul's great stuff of the Passthru vbdev, I've updated through below three patches to achieve the per client QoS. Each client sees its own passthru vbdev and QoS assigned on this vbdev. These vbdev shares the same bdev. Currently it only supports shareable and read only vbdev on same bdev.
https://review.gerrithub.io/#/c/406888/ vbdev: update and have a create_passthru_disk function
https://review.gerrithub.io/#/c/406891/ vbdev: add the construct_passthru_bdev RPC method
https://review.gerrithub.io/#/c/406977/ vbdev: make the passthru vbdev share the same underlying bdev
The possible usage as following:
[Take NVMe-oF target as example]
1. In the nvmf.conf file,
# Configure the Passthru vbdev on same bdev like Malloc0
[Passthru]
PT Malloc0 PT0
PT Malloc0 PT1
# The subsystem is configured with shareable bdev by Passthru vbdevs
[Subsystem2]
NQN nqn.2016-06.io.spdk:cnode2
Listen RDMA 192.168.2.21:4420
AllowAnyHost No
Host nqn.2016-06.io.spdk:init
SN SPDK00000000000002
Namespace PT0 >>>> This can be connected by one client
Namespace PT1 >>>> This can be connected by another client
# Assign different QoS IOPS limiting on Passthru vbdevs
[QoS]
Limit_IOPS PT0 20000
Limit_IOPS PT1 30000
2. Use RPC method to add the Passthru vbdev at runtime
a. python ./scripts/rpc.py construct_passthru_bdev Malloc0 PT3
b. python ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 PT3
c. python ./scripts/rpc.py enable_bdev_qos PT3 20000
If there is any concern or comment, please feel free to let me know.
Thanks,
Gang
1 year, 11 months
vhost target failover?
by Huang Frank
Hi，all
What's the failover mechism of vhost target? If the vhost target goes to crash, how to make sure the guest can process without affectting?
Thanks,
Frank.
________________________________
kinzent(a)hotmail.com
1 year, 11 months
Re: [SPDK] anyone ran the SPDK ( app/iscsi_tgt/iscsi_tgt ) with VPP?
by Isaac Otsiabah
Hi Tomasz, I configured vpp and I thought it was working because I can ping the initiator machine (192.168.2.10) on my lab private network.
(On target Server):
vpp# show int address
TenGigabitEthernet82/0/0 (up):
192.168.2.20/24
local0 (dn):
vpp#
vpp# show int
Name Idx State Counter Count
TenGigabitEthernet82/0/0 1 up rx packets 58
rx bytes 5342
tx packets 58
tx bytes 5324
drops 71 (drops were expected because I did a ping to itself on 192.168.2.20)
ip4 49
local0 0 down
vpp# ping 192.168.2.10
64 bytes from 192.168.2.10: icmp_seq=1 ttl=64 time=.1229 ms
64 bytes from 192.168.2.10: icmp_seq=2 ttl=64 time=.0412 ms
64 bytes from 192.168.2.10: icmp_seq=3 ttl=64 time=.0733 ms
64 bytes from 192.168.2.10: icmp_seq=4 ttl=64 time=.0361 ms
64 bytes from 192.168.2.10: icmp_seq=5 ttl=64 time=.0332 ms
Statistics: 5 sent, 5 received, 0% packet loss
However, something is missing because i started the iscsi_tgt server (below) and executed add_portal_group RPC command to add a portal group and got a bind() error below. Iscsi.conf is attached (portal group and target sections are commented out). The iscsi_tgt binary was linked to the same vpp library that I built.
[root@spdk2 spdk]# ./scripts/rpc.py add_portal_group 1 192.168.2.20:3260
Got JSON-RPC error response
request:
{
"params": {
"portals": [
{
"host": "192.168.2.20",
"port": "3260"
}
],
"tag": 1
},
"jsonrpc": "2.0",
"method": "add_portal_group",
"id": 1
}
response:
{
"message": "Invalid parameters",
"code": -32602
}
[root@spdk2 spdk]# ./app/iscsi_tgt/iscsi_tgt -m 0x0101
Starting SPDK v18.04-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: iscsi -c 0x0101 --file-prefix=spdk_pid8974 ]
EAL: Detected 16 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Multi-process socket /var/run/.spdk_pid8974_unix
EAL: Probing VFIO support...
EAL: VFIO support initialized
app.c: 443:spdk_app_start: *NOTICE*: Total cores available: 2
reactor.c: 645:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x3
reactor.c: 429:_spdk_reactor_run: *NOTICE*: Reactor started on core 8 on socket 1
reactor.c: 429:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
copy_engine_ioat.c: 318:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled
EAL: PCI device 0000:02:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
init_grp.c: 107:spdk_iscsi_init_grp_add_initiator: *WARNING*: Please use "ANY" instead of "ALL"
init_grp.c: 108:spdk_iscsi_init_grp_add_initiator: *WARNING*: Converting "ALL" to "ANY" automatically
posix.c: 221:spdk_posix_sock_create: *ERROR*: bind() failed, errno = 99
posix.c: 231:spdk_posix_sock_create: *ERROR*: IP address 192.168.2.20 not available. Verify IP address in config file and make sure setup script is run before starting spdk app.
portal_grp.c: 190:spdk_iscsi_portal_open: *ERROR*: listen error 192.168.2.20.3260
iscsi_rpc.c: 969:spdk_rpc_add_portal_group: *ERROR*: portal_grp_open failed
From: Zawadzki, Tomasz [mailto:tomasz.zawadzki@intel.com]
Sent: Thursday, April 12, 2018 12:27 AM
To: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com>
Cc: Harris, James R <james.r.harris(a)intel.com>; Verkamp, Daniel <daniel.verkamp(a)intel.com>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com>
Subject: RE: anyone ran the SPDK ( app/iscsi_tgt/iscsi_tgt ) with VPP?
Hello Isaac,
Are you using following patch ? (I suggest cherry picking it)
https://review.gerrithub.io/#/c/389566/
SPDK iSCSI target can be started without specific interface to bind on, by not specifying any target nodes or portal groups. They can be added later via RPC http://www.spdk.io/doc/iscsi.html#iscsi_rpc.
Please see https://github.com/spdk/spdk/blob/master/test/iscsi_tgt/lvol/iscsi.conf for example of minimal iSCSI config.
Suggested flow of starting up applications is:
1. Unbind interfaces from kernel
2. Start VPP and configure the interface via vppctl
3. Start SPDK
4. Configure the iSCSI target via RPC, at this time it should be possible to use the interface configured in VPP
Please note, there is some leeway here. The only requirement is having VPP app started before SPDK app.
Interfaces in VPP can be created (like tap or veth) and configured at runtime, and are available for use in SPDK as well.
Let me know if you have any questions.
Tomek
From: Isaac Otsiabah [mailto:IOtsiabah@us.fujitsu.com]
Sent: Wednesday, April 11, 2018 8:47 PM
To: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com<mailto:tomasz.zawadzki@intel.com>>
Cc: Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris@intel.com>>; Verkamp, Daniel <daniel.verkamp(a)intel.com<mailto:daniel.verkamp@intel.com>>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com<mailto:PVonStamwitz@us.fujitsu.com>>
Subject: anyone ran the SPDK ( app/iscsi_tgt/iscsi_tgt ) with VPP?
Hi Tomaz, Daniel and Jim, i am trying to test VPP so build the VPP on a Centos 7.4 (x86_64), build the SPDK and tried to run the ./app/iscsi_tgt/iscsi_tgt application.
For VPP, first, I unbind the nick from the kernel as and start VPP application.
./usertools/dpdk-devbind.py -u 0000:07:00.0
vpp unix {cli-listen /run/vpp/cli.sock}
Unbinding the nic takes down the interface, however, the ./app/iscsi_tgt/iscsi_tgt -m 0x101 application needs an interface to bind to during startup so it fails to start. The information at:
"Running SPDK with VPP
VPP application has to be started before SPDK iSCSI target, in order to enable usage of network interfaces. After SPDK iSCSI target initialization finishes, interfaces configured within VPP will be available to be configured as portal addresses. Please refer to Configuring iSCSI Target via RPC method<http://www.spdk.io/doc/iscsi.html#iscsi_rpc>."
is not clear because the instructions at the "Configuring iSCSI Traget via RPC method" suggest the iscsi_tgt server is running for one to be able to execute the RPC commands but, how do I get the iscsi_tgt server running without an interface to bind on during its initialization?
Please, can anyone of you help to explain how to run the SPDK iscsi_tgt application with VPP (for instance, what should change in iscsi.conf?) after unbinding the nic, how do I get the iscsi_tgt server to start without an interface to bind to, what address should be assigned to the Portal in iscsi.conf.. etc)?
I would appreciate if anyone would help. Thank you.
Isaac
1 year, 11 months
SPDK - multi process DPDK? Or single process?
by Sai Rajesh
HI,
I'm relatively new to SPDK. Can someone help me answer some of my questions.
In my experiment, I'm running "perf" application in a loop via script.
#!/bin/bash
count=1
while true; do
echo $count
./perf -q 64 -s 131072 -w write -t 300 -r 'trtype:RDMA adrfam:IPv4
traddr:192.168.22.55 trsvcid:4420'
let count++
done
When I do this way, after 1hour into runs, I hit "remote access error".
https://github.com/spdk/spdk/issues/264/. As if someone has freed the
memory while its still being used by RNIC.
After debugging for long, I found that disabling ASLR made my script run
for 3days without any issues.
In that context, I have following questions
1. Do SPDK run as DPDK's single process or multi-process?
2. Why disabling ASLR worked for me?
3. When I allocate task buffer via spdk_dma_zmalloc() in perf application,
does RNIC device save these memory mappings and refer them during
completion processing?
4. In my failure case, with ASLR enabled, does this memory mapping changed
over the fly and RNIC now, was not able to access task buffer?
Thanks for your time and patience.
Thanks,
Rajesh.
1 year, 11 months
Trying to recover from one SPDK process crashing in a multi-process environment
by Cao, Gang
Hi all,
This topic is regarding the usage of SPDK NVMe driver in the multi-process mode. There are some cases that any process (primary or secondary) could exit in the unexpectedly way to leave the allocated memory not released, the held lock also not released or even some severe problem like in the middle of the memory allocation. Some of these issues may have a way to solve and others may be difficult to solve.
Would like to initiate some discussion on this topic to get more input, suggestions and comments. For example, we may need some mechanism to track the per process memory allocation through those memory related APIs (called DPDK's) so that these memory has the hint to release once its associated process exits. Some co-work will be covered by SPDK and DPDK memory management.
Any feedback is welcomed.
Thanks,
Gang
1 year, 12 months
difference between the vhost-blk and vhost-scsi?
by Huang Frank
hi,ALL
I find that the vhost-target can offer two type of devices: vhost-scsi and vhost-blk. What's the difference? Which one should I used?
Thanks,
Frank
________________________________
kinzent(a)hotmail.com
1 year, 12 months
nvmf/rdma: handling QP errors
by Philipp Skadorov
Hi there,
Looking at the list of errors ibv_get_async_event offers - would it make sense to monitor them in the polling loops?
Right now, per my understanding, the only way SPDK gets that things go wrong is: when initiator issues disconnect.
Any thoughts?
Regards,
Philipp
1 year, 12 months
Paying More Attention to Nightly Build Status
by Luse, Paul E
All-
As many of you know, there's been a lot of effort over the past months to beef up our CI infrastructure, restructure our test directories, refactor some test scripts, add new tests, etc. I wanted to take a minute to highlight the need for community ownership when it comes to code quality. Ultimately we all rely on those voted in as maintainers to assure that nothing gets checked into the database that doesn't meet with SPDK standards. However, we are working with a large, diverse and fast moving project where we are seeing the size of the community grow rapidly which is awesome as it underscores the value of the code base and continued contributions from all of you!
At the same time, however, this growth requires all of us to do our part in monitoring the quality of the project. That means spending more time on reviews, asking questions, learning and essentially holding each other accountable for keeping SPDK on track. To that end, a few new links are available on the CI website https://ci.spdk.io/ which is a link right off of http://www.spdk.io/.
Please take a minute to review the info below and help keep SPDK code quality up by checking in on these links often, asking questions and helping debug as much as you can!
Thanks!!
Paul
NIGHTLY BUILD STATUS
-------------------------------
As most of you are aware, every patch goes through a battery of tests before it's committed to the repo. We need to keep the length of those tests to something reasonable in order to promote timely contributions so in addition to the per patch testing, every night a much longer set of tests is executed against the tip. Previously it was possible to find these test results but it wasn't super visible. Now it is! If you click on the "SPDK Test Pool Status" link off the CI page (https://ci.spdk.io/spdk/status/) you will now see the status of the last nightly build as shown below:
[cid:image001.png@01D3CD91.7EF6ECA0]
NIGHTLY BUILD HISTORY
---------------------------------
You can also click on the link in the upper right corner of that page called "Nightly Build" (https://ci.spdk.io/spdk/nightly_status.html) and see a history of recent nightly builds
[cid:image002.png@01D3CD91.7EF6ECA0]
1 year, 12 months
