spdkcli.py in weird state
by Chuck Tuffli
Hi
Using the latest spdkcli.py from git, I can get it into a weird state where
the ll command only displays:
'uuid'
after adding an NVMe drive to a subsystem's namespace. The below process
works correctly if the configuration uses a Null bdev instead of an NVMe
bdev, but perhaps I'm doing something wrong.
$ sudo ./nvmf_tgt --wait-for-rpc
$ sudo -sH scripts/spdkcli.py
SPDK CLI v0.1
/> start_subsystem_init
/> cd /bdevs/nvme
/bdevs/nvme> create Nvme0 PCIe 0000:37:00.0
['Nvme0n1']
/> cd /nvmf/transport
/nvmf/transport> create rdma
/nvmf/transport> cd /nvmf/subsystem/
/nvmf/subsystem> create nqn.2016-06.io.spdk:cnode1 SPDK00000000000001 8 true
/nvmf/subsystem> cd nqn.2016-06.io.spdk:cnode1/namespaces
/nvmf/subsyst...e1/namespaces> create Nvme0n1
/nvmf/subsyst...e1/namespaces> cd /
/> ll
'uuid'
/> ll /bdevs/nvme/
o- nvme
.................................................................................................................
[Bdevs: 1]
o- Nvme0n1
..................................................................................................
[Size=1.5T, Claimed]
--chuck
3 years, 8 months
Tomorrow's community meeting is cancelled
by Harris, James R
Due to the US Thanksgiving holiday, tomorrow’s community meeting is cancelled. The next “Asia-Friendly” community meeting will be two weeks from tomorrow.
Thanks,
-Jim
3 years, 8 months
Jenkins CI Status Page
by Luse, Paul E
So I have a question that I should already know the answer to but don't :)
Seth/Karol/anyone, is this available for everyone to view/monitor yet?
Thanks!
Paul
3 years, 8 months
Create your own blog posts on https://spdk.io
by Walker, Benjamin
Hi everyone,
I'd like to take a moment to point out what I think is a really cool feature of
the SPDK project. The project website, https://spdk.io, is entirely open source
and anyone can propose patches. The process used is identical to contributing to
the main SPDK repository (outlined at https://spdk.io/development), except it
uses the spdk.github.io repository instead. You can follow along with the
regular development environment set up instructions, but change URLs as follows:
https://review.gerrithub.io/spdk/spdk ->
https://review.gerrithub.io/spdk/spdk.github.io
https://github.com/spdk/spdk -> https://github.com/spdk/spdk.github.io
The site is statically generated from markdown files into HTML using a tool
called Jekyll. It's the standard GitHub pages development process and is well
documented by GitHub - see https://pages.github.com/. We even do continuous
integration on patches proposed through GerritHub and it will build the site and
let you preview it before we merge. The only exception is the content under
https://spdk.io/doc, which is automatically generated by Doxygen once per day.
The source for that documentation is in the main SPDK repository under the doc/
directory.
One way that we hope people will contribute is to create blog posts. Yesterday,
we committed the first such blog post from a non-Intel contributor (Shuhei
Matsumoto from Hitachi posted about NVMe-oF performance). If anyone in the
community is interested in writing a blog post, we highly encourage you to do
so! Our recommendations are that blog posts:
1) Focus primarily on SPDK.
2) Are technical in nature.
3) Are accompanied by test methodology and reproduction steps, if benchmarks are
presented.
4) Are written in English. Reviewers will provide editorial feedback to assist
blog post authors in this area.
As with code patches, acceptance of blog posts is at the discretion of the
maintainers and follows the same review process.
Thanks,
Ben
3 years, 9 months
SPDK NVMe-oF TCP Transport
by Harris, James R
NVM Express has ratified TCP as a new transport for NVMe-oF! The announcement can be found here<https://nvmexpress.org/welcome-nvme-tcp-to-the-nvme-of-family-of-transports/>. Quoting from the NVM Express announcement:
“What’s really exciting about NVMe/TCP is that it enables efficient end-to-end NVMe operations between NVMe-oF host(s) and NVMe-oF controller devices interconnected by any standard IP network with excellent performance and latency characteristics. This allows large-scale data centers to utilize their existing ubiquitous Ethernet infrastructure with multi-layered switch topologies and traditional Ethernet network adapters. NVMe/TCP is designed to layer over existing software based TCP transport implementations as well as future hardware accelerated implementations.”
Ziye Yang from Intel has been actively working on SPDK support for NVMe-oF TCP, and has a patch<https://review.gerrithub.io/#/c/spdk/spdk/+/425191/> that is now available for testing. This patch adds NVMe-oF TCP support to both the SPDK NVMe-oF target as well as the SPDK NVMe host initiator. This support will likely be merged in the next couple of weeks, so we welcome any code review or testing feedback!
Thanks,
Jim Harris
3 years, 9 months
perf failure with RDMA
by Chuck Tuffli
Using the latest from git (5aace139), I'm able to run perf to NVMe drives
over PCIe and run the NVMe-oF target. The Linux kernel NoF driver can
successfully connect to the spdk target, but perf fails when connecting
over RDMA:
# uname -a
Linux nighthawk01 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux
# grep -i huge /proc/meminfo
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
HugePages_Total: 4096
HugePages_Free: 4096
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
# ./perf -q 4 -o 512 -w randread -r 'trtype:RDMA adrfam:IPV4
traddr:10.0.0.14 trsvcid:4420' -t 60
Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
--base-virtaddr=0x200000000000 --file-prefix=spdk_pid8083 ]
EAL: Detected 96 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
Initializing NVMe Controllers
nvme_rdma.c: 276:nvme_rdma_qpair_init: *ERROR*: rdma_create_qp failed
perf.dbg: rdam_create_qp: Cannot allocate memory
nvme_rdma.c: 812:nvme_rdma_qpair_connect: *ERROR*: nvme_rdma_qpair_init()
failed
nvme_rdma.c:1388:nvme_rdma_ctrlr_construct: *ERROR*: failed to create admin
qpair
nvme.c: 523:spdk_nvme_probe_internal: *ERROR*: NVMe ctrlr scan failed
spdk_nvme_probe() failed for transport address '10.0.0.14'
./perf: errors occured
# ping -c1 10.0.0.14
PING 10.0.0.14 (10.0.0.14) 56(84) bytes of data.
64 bytes from 10.0.0.14: icmp_seq=1 ttl=64 time=0.157 ms
--- 10.0.0.14 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.157/0.157/0.157/0.000 ms
#
Should this work? Am I missing a step? Note that I did modify perf only to
print the error returned from rdma_create_qp:
[598] git diff
diff --git a/lib/nvme/nvme_rdma.c b/lib/nvme/nvme_rdma.c
index 30560c17..fd688ee3 100644
--- a/lib/nvme/nvme_rdma.c
+++ b/lib/nvme/nvme_rdma.c
@@ -30,7 +30,7 @@
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
-
+/*XXX*/#include <err.h>
/*
* NVMe over RDMA transport
*/
@@ -274,6 +274,7 @@ nvme_rdma_qpair_init(struct nvme_rdma_qpair *rqpair)
rc = rdma_create_qp(rqpair->cm_id, rctrlr->pd, &attr);
if (rc) {
SPDK_ERRLOG("rdma_create_qp failed\n");
+ warn("rdam_create_qp");
return -1;
}
rctrlr->pd = rqpair->cm_id->qp->pd;
---chuck
3 years, 9 months
The disk hot remove function of SPDK
by Vincent
Hello all,
Recently we are trying the disk hot remove property of SPDK.
We have a counter to record the IO send out for a io channel
The roughly hot remove procedure in my code is
(1) when receive the disk hot remove call back from spdk, we stop sending IO
(2) Because we have a counter to record the IO send out for IO channel, we
wait all IOs call back(complete)
(3) close io channel
(4)close bdev desc
But sometime we crashed (the crash rate is about 1/10) , the call stack is
attached
crash in function nvme_free_request
void
nvme_free_request(struct nvme_request *req)
{
assert(req != NULL);
assert(req->num_children == 0);
assert(req->qpair != NULL);
STAILQ_INSERT_HEAD(&req->qpair->free_req, req, stailq);
<-------------this line
}
Does any one can give me a hint ??
Any suggestion is appreciated
Thank you in advance
--------------------------------------------------------------------------------------------------------------------------
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./smistor_iscsi_tgt -c
/usr/smistor/config/smistor_iscsi_perf.conf'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000414b87 in nvme_free_request ([email protected]=0x7fe4edbf3100)
at nvme.c:227
227 nvme.c: No such file or directory.
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.170-4.el7.x86_64
elfutils-libs-0.170-4.el7.x86_64 glibc-2.17-222.el7.x86_64
libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-13.el7.x86_64
libcap-2.22-9.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64
libuuid-2.23.2-52.el7.x86_64 lz4-1.7.5-2.el7.x86_64
numactl-libs-2.0.9-7.el7.x86_64 openssl-libs-1.0.2k-12.el7.x86_64
systemd-libs-219-57.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64
zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x0000000000414b87 in nvme_free_request ([email protected]=0x7fe4edbf3100)
at nvme.c:227
#1 0x0000000000412056 in nvme_pcie_qpair_complete_tracker
([email protected]=0x7fe4c5376ef8, tr=0x7fe4ca8ad000,
[email protected]=0x7fe4c8e0a840, [email protected]=true)
at nvme_pcie.c:1170
#2 0x0000000000413be0 in nvme_pcie_qpair_process_completions
([email protected]=0x7fe4c5376ef8, max_completions=64,
[email protected]=0) at nvme_pcie.c:2013
#3 0x0000000000415d7b in nvme_transport_qpair_process_completions
([email protected]=0x7fe4c5376ef8,
[email protected]=0) at nvme_transport.c:201
#4 0x000000000041449d in spdk_nvme_qpair_process_completions
(qpair=0x7fe4c5376ef8, [email protected]=0)
at nvme_qpair.c:368
#5 0x000000000040a289 in bdev_nvme_poll (arg=0x7fe08c0012a0) at
bdev_nvme.c:208
#6 0x0000000000499baa in _spdk_reactor_run (arg=0x6081dc0) at reactor.c:452
#7 0x00000000004a4284 in eal_thread_loop ()
#8 0x00007fe8fb276e25 in start_thread () from /lib64/libpthread.so.0
#9 0x00007fe8fafa0bad in clone () from /lib64/libc.so.6
3 years, 9 months
