Jenkins Test System Updates
by Luse, Paul E
Hi Karol,
In CH test pool, we have 3 systems setup to run crypto testing in CI based on the flag SPDK_TEST_CRYPTO and on those systems a few things are needed:
* Nasm of 2.12.2 or higher
* Ipsec module installed as root
* Some misc DPDK things enabled
The third is part of the patch so nothing needs to be done there however the first two need to be done via running pkgdep on the test machines. If the packaged nasm isn't of high enough version the build log will let you know. If ipsec isn't installed it will also let you know (see below from latest crypto patch w/Jenkins running BlobFS_autotest
To enable crypto you must first go to the intel-ipsec-mb directory and
run 'make' then 'sudo make install' then re-run this script.
15:09:05 $ trap - ERR
15:09:05 $ print_backtrace
15:09:05 $ [[ ehxBE =~ e ]]
15:09:05 $ local shell_options=ehxBE
15:09:05 $ set +x
========== Backtrace start: ==========
Here is a list of the failed tests and what needs to be done to get them working with the Jenkins test pool, we don't need crypto running everywhere so if you can enable it just on a few machines (block dev tests) that would be great:
BlobFS_autotest: missing install of ipsec, run pkgdep.sh
CentOS test: missing nasm or old, maybe just don't run on this OS
FreeBSD: same as above
NVME autotest: LLooks to be configured correctly but crypto patch failed for what looks like unrelated reasons: https://ci.spdk.io/spdk-jenkins/results/autotest-per-patch/builds/10910/a...
Ubuntu16: old nasm, maybe don't enable crypto testing on this one
Ubuntu17: needs ipsec insetall (pkgdep.sh) and this is probably a good candidate for crypto testing
3 years, 11 months
spdk_nvme_detach
by Nabarro, Tom
I'm having an issue where when I enumerate namespaces and controllers twice within a process, the second iteration fails to detach controller on cleanup [SPDK spdk_nvme_detach] (stalls on that call).
I'm using sample code from hello_world.c to replicate
Any ideas?
Tom Nabarro
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
3 years, 11 months
How to handle not contiguous data in end-to-end data protection and use ISA-L in SPDK
by 松本周平 / MATSUMOTO，SHUUHEI
Hi All,
I have started to investigate end-to-end data protection, i.e. DIF/DIX in SPDK.
I have a couple of things to get any feedback.
I'm focusing on not separated data and metadata but interleaved metadata, i.e. extended LBA first.
and I'm investigating to implement verification of DIF in the bdev layer.
SPDK support not contiguous data but it will be easier if SPDK can avoid not contiguous read of metadata, i.e. read two 4 bytes from different buffer and concatenate them.
Can we set any requirement to the alignment of the data? As long as I understand, NVMe don't set strict requirement to alignment/granularity.
Or do we have to support all pattern of splitted reads?
Or should we copy not contiguous data to contiguous dynamically allocated buffer and walk the latter?
Do you have any other idea?
Secondly, about CRC calculation, I observed that ISA-L (Intel Storage Application Library) has a hundred times better performance than current SPDK's function.
Should I use ISA-L by my own risk or can we integrate ISA-L into SPDK master?
Thanks,
Shuhei
3 years, 11 months
"GerritHub Configuration" in SPDK development page are not updated
by Kevin Wilson
Hello,
I tried to follow the "GerritHub Configuration" section in SPDK
development page: http://spdk.io/development/, but it seems that this section
is not updated.
I log in to https://github.com/spdk/spdk and enter my password.
There is indeed a "Profile" tab on the left side in settings
However, I don't see "HTTP Password" nor "Preferences" nor "My menu"
What I do see on the right side is:
Profile
Account
Emails
Notifications
Billings
SSH and GPG keys
....
....
Can it be updated to reflect the current github website?
Regards,
Kevin
3 years, 11 months
Required and optional parameters
by Kaminski, PawelX
Hi,
I noticed that in rpc construct_nvmf_subsystem following parameters are required:
1. Nqn
2. Listen
3. Hosts
But in file scripts/rpc/nvmf.py for method construct_nvmf_subsystem following arguments are required:
1. Nqn
2. Serial number
I would like to know which parameters should be required in spdkcli?
Regards,
Pawel
--------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
3 years, 11 months
Commit Message Guidelines
by Luse, Paul E
Hi All,
Just a quick reminder, we don't have super stringent guidelines for commit messages but what we do have is important :) Please be sure to follow the guidelines below, failing to include why your patch is needed (just describing what it does) makes it difficult for new reviews to understand your intent. Takes just an extra minute or so to add a sentence or two to help others provide valuable feedback.
Thanks!!
Paul
>From http://spdk.io/development/ :
Provide a clear commit message describing the purpose of the commit. Good commit messages provide a very brief summary of what the commit does in the title followed by a short paragraph providing context for the change. For example, what problem is being solved, how was it discovered and how this patch solves the problem.
3 years, 11 months
Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
by Harris, James R
I like Darek’s suggestions. I do think this is a DPDK bug though. Other DPDK calls like rte_malloc() work from a non-DPDK thread. Paul – can you dig into this further?
-Jim
On 9/7/18, 7:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
That's really good to know, thanks! I'm hardcoded to 0 now which is fine for the remainder of testing I think, let me know how it goes with get_mempolicy
Thx,
Paul
-----Original Message-----
From: SPDK [mailto:[email protected]] On Behalf Of Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 7:31 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I wasn't registered on the mailing list with this email before, so I registered just now and am resending this message.
D.
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 4:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf
HI Paul,
Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstraction for multi-threading. It uses standard pthreads for most of its work and that's why you're seeing socket id 4294967295 (unsigned -1). This value is simply uninitialized. To get socket id in a portable way, you might want to use get_mempolicy. In fact, we should probably use it somewhere within our SPDK APIs. I'll look into adding it tomorrow.
D.
-----Original Message-----
From: SPDK [mailto:[email protected]] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 3:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
3 years, 11 months
socket_id difference under fio plugin vs bdevio or bdevperf
by Luse, Paul E
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
3 years, 11 months
