October 2017 - SPDK - Ml01.01.Org

Re: [SPDK] Buffer I/O error on bigger block size running fio

by Harris, James R

Hi Victor, Could you provide a few more details? This will help the list to provide some ideas. 1) On the client, are you using the SPDK NVMe-oF initiator or the kernel initiator? 2) Can you provide the fio configuration file or command line? Just so we can have more specifics on “bigger block size”. 3) Any details on the HW setup – specifically details on the RDMA NIC (or if you’re using SW RoCE). Thanks, -Jim From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Victor Banh <victorb(a)mellanox.com> Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org> Date: Thursday, October 5, 2017 at 11:26 AM To: "spdk(a)lists.01.org" <spdk(a)lists.01.org> Subject: [SPDK] Buffer I/O error on bigger block size running fio Hi I have SPDK NVMeoF and keep getting error with bigger block size with fio on randwrite tests. I am using Ubuntu 16.04 with kernel version 4.12.0-041200-generic on target and client. The DPDK is 17.08 and SPDK is 17.07.1. Thanks Victor [46905.233553] perf: interrupt took too long (2503 > 2500), lowering kernel.perf_event_max_sample_rate to 79750 [48285.159186] blk_update_request: I/O error, dev nvme1n1, sector 2507351968 [48285.159207] blk_update_request: I/O error, dev nvme1n1, sector 1301294496 [48285.159226] blk_update_request: I/O error, dev nvme1n1, sector 1947371168 [48285.159239] blk_update_request: I/O error, dev nvme1n1, sector 1891797568 [48285.159252] blk_update_request: I/O error, dev nvme1n1, sector 10833824 [48285.159265] blk_update_request: I/O error, dev nvme1n1, sector 614937152 [48285.159277] blk_update_request: I/O error, dev nvme1n1, sector 1872305088 [48285.159290] blk_update_request: I/O error, dev nvme1n1, sector 1504491040 [48285.159299] blk_update_request: I/O error, dev nvme1n1, sector 1182136128 [48285.159308] blk_update_request: I/O error, dev nvme1n1, sector 1662985792 [48285.191185] nvme nvme1: Reconnecting in 10 seconds... [48285.191254] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191291] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191305] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191314] ldm_validate_partition_table(): Disk read failed. [48285.191320] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191327] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191335] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191342] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191347] Dev nvme1n1: unable to read RDB block 0 [48285.191353] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191360] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191375] Buffer I/O error on dev nvme1n1, logical block 3, async page read [48285.191389] nvme1n1: unable to read partition table [48285.223197] nvme1n1: detected capacity change from 1600321314816 to 0 [48289.623192] nvme1n1: detected capacity change from 0 to -65647705833078784 [48289.623411] ldm_validate_partition_table(): Disk read failed. [48289.623447] Dev nvme1n1: unable to read RDB block 0 [48289.623486] nvme1n1: unable to read partition table [48289.643305] ldm_validate_partition_table(): Disk read failed. [48289.643328] Dev nvme1n1: unable to read RDB block 0 [48289.643373] nvme1n1: unable to read partition table

4 years, 3 months

4
22
0 / 0

SPDK Trello board

by Shah, Prital B

All, I just want to highlight that we have a SPDK Trello board: https://trello.com/spdk for Roadmap discussion and current feature design discussions. 1) To add items to SPDK backlog : Please use this board : https://trello.com/b/P5xBO7UR/things-to-do 2) To discuss features design via individual feature boards. We are planning to do a discussion on JSON based configuration for SPDK and brainstorming on how we can make it easier for developers to configure SPDK. You're welcome to join our meeting at +1(916)356-2663. Choose bridge 5, Conference ID: 864687063 Meeting on 09/15/2017 at 02:30 PM MST (UTC-7), this Friday. Thanks Prital

4 years, 6 months

3
2
0 / 0

Question about create a simple RAID1 in SPDK

by Annan Chang 張安男

Hello all, I am planning to create a simple RAID1 in spdk. My original idea is to implement a new spdk bdev device that contain two physical devices. So I looked up the code of AIO and nvme bdev device. But I found that writing a new spdk bdev device that contain two physical nvme devices is difficult for me because of I am not familiar with nvme’s user space driver. I just found in the maillist, maybe use “vbdev” architecture is just right. But I cannot find any example of vbdev. Does anybody can give me the advice how to write a vbdev device the contain of two bdev nvme device ? Thank you very much. -- AnNan, Chang

4 years, 7 months

2
1
0 / 0

SPDK errors

by Santhebachalli Ganesh

Folks, My name is Ganesh, and I am working on NVEMoF performance metrics using SPDK (and kernel). I would appreciate your expert insights. I am observing errors when QD on perf is increased above >=64 most of the times. Sometimes, even for <=16 Errors are not consistent. Attached are some details. Please let me know if have any additional questions. Thanks. -Ganesh

4 years, 7 months

4
20
0 / 0

VTOPYHS Mapping

by Kumaraparameshwaran Rathnavel

Hi All, I was looking into the vtophys.c and I had the following query. I understand that this creates a mapping of 128TB Virtual Address Space where Physical Addresses are populated in the map. The number of entries for Top-Level Map should be 17 Bits(30-46) which is 128K 1GB entries. But we have created entries for 18bits. So what is the reason behind having extra entries. And also the second level page map would require 9 Bits (21-29) 512 entries of 2MB page for a single 1GB map. Why do we create 1024 entries. Please correct if my understanding is wrong. SHIFT_128TB 47 SHIFT_1GB 30 SHIFT_2MB 21 /* Translation of a single 2MB page. */ struct map_2mb { uint64_t translation_2mb; }; /* Second-level map table indexed by bits [21..29] of the virtual address. * Each entry contains the address translation or SPDK_VTOPHYS_ERROR for entries that haven't * been retrieved yet. */ struct map_1gb { struct map_2mb map[1ULL << (SHIFT_1GB - SHIFT_2MB + 1)]; uint16_t ref_count[1ULL << (SHIFT_1GB - SHIFT_2MB + 1)]; }; /* Top-level map table indexed by bits [30..46] of the virtual address. * Each entry points to a second-level map table or NULL. */ struct map_128tb { struct map_1gb *map[1ULL << (SHIFT_128TB - SHIFT_1GB + 1)]; }; Thanking You, Param.

4 years, 8 months

1
0
0 / 0

FW: SPDK, Blobstore & Swift

by Luse, Paul E

Here's today's Swift community meeting IRC chat about SPDK driven by Wewe's proposal around blobstore, thanks Wewe!! For those not wanting to read through the transcript, my takeaways are: * SSDs are not in high use in Swift mainly due to cost making any SW effort in optimizing for flash a relatively low priority for them * Where SSD are used (container storage) there are no significant performance issues That said: * The Swift project technical lead (notmyname) is always looking for new ways to differentiate and wants to keep an open dialogue, just not willing to endorse activities (merge any code into Swift related to this) to support the effort So at this point anyone is, of course, free to work on a proof of concept w/o a ton of help from the swift folks and, provided there's some compelling data, they'd surely reconsider. I can't speak for the SPDK maintainers but I think w/o Swift community interest using Swift as vehicle to introduce SPDK into a more object oriented system is probably not the best route to go. Let me know if there are any questions, these guys are always willing to talk. Thx Paul <notmyname> today peluse is back with us (yay) to talk about something he's been working on <peluse> rock n roll <notmyname> peluse: take it away <peluse> I'm thinking I should have typed some shit up in advance to avoid all the typos I'm about to introduce :) <peluse> anyways... <peluse> http://spdk.io <notmyname> #link http://spdk.io <peluse> is the URL as I mentioned before. Quick high level overview then I'll bring up a proposal someone in our community has made <peluse> that we haven't spent a whole lot of time thinking about TBH <notmyname> ok * andreas_s has quit (Ping timeout: 240 seconds) <peluse> Also, here's a SNIA talk I did last month about SPDK in general and one relevant component called blobstore https://www.snia.org/sites/default/files/SDC/2017/presentations/Solid_Sta... <peluse> So SPDK is a set of user space components that is all BSD licensed * andreas_s ([email protected]/ibm/x-tvmvoewqleavvoqo) has joined <peluse> its used in a whole bunch of ways but mainly by storage appliances to optimize SSD performance in what swift would call the storage node <peluse> FYI its in Ceph already but not the default driver <peluse> and when I say "it" I mean whatever component the system has chosen to take on, in Ceph its the user space polled mode NVMe driver <peluse> there are some basic perf marketing type hypes slides in that deck I pated in for anyone interested <peluse> pretty huge gains when you consider latency and CPU sensitive apps running with latest SSDs <notmyname> so the basic idea is a fast/efficient way to talk to fast storage media that might potentially be useful in swift's object server? <peluse> anyway, that's the real trick is that its all user space, direct access to HW, no INTs and no locking <peluse> yup <peluse> but there are a ton of compoennts, well not a ton, but a bunch that would not be relevant <timburke> could it be useful for the account/container servers, too, or are we just looking at object servers (and diskfile in particular)? <notmyname> what are the integration points. I doubt it's as simple as mmaping a file and your'e done <peluse> and some are lirbaries and some are applications. * baoli has quit (Remote host closed the connection) <peluse> I think since its SSD only (well not techncially but it wouldn't make sense to use on spinning media) most likelt container <rledisez> so we are talking of objec servers on SSD. is it a real use case? (i would think it's the target of ceph, very low latency) <peluse> if you used object servers there are probably some limitations wrt what we call blobstore <peluse> I'l get to the integration question in a sec <peluse> so, assuming a node takes on the user space NVMe driver and the driver talks directly to HW you can see there no kernel and no FS <peluse> so... unless the storage application talks in blocks it doesn't make much sense * TxGirlGeek has quit (Quit: My MacBook has gone to sleep. ZZZzzz...) <notmyname> ok * artom_ (~artom(a)205.233.59.73<mailto:[email protected]>) has joined <peluse> blobstore is SPDK's answer to this but its not a FS * artom_ has quit (Remote host closed the connection) <peluse> it's a super simple way for apps that don't talk blocks that can use a really simple file-ish object-ish like interface to take advantage of SPDK <peluse> so for example, RocksDB * artom_ (~artom(a)205.233.59.73<mailto:[email protected]>) has joined <peluse> in that slide deck I mention some work we did there to bolt blobstore up to RocksDB as a back end <notmyname> so ... as you know swift likes to be HW and driver agnostic. what does this tie in too? is it possible to write stuff in a way that works if you have fast media or not? <peluse> its that kind of idea that might makes sense for Swift * rcernin ([email protected]/redhat/x-twozdfdqzloeefcs) has joined * andreas_s has quit (Ping timeout: 260 seconds) * jungleboyj looks in late <notmyname> or is the idea that swift would engage spdk mode if it detects flash? <peluse> so there are lots of things that can be done there * andreas_s ([email protected]/ibm/x-asrkfmvdxkgqcbhv) has joined <peluse> but yeah I think anything more aggressive than NVMe only would not be worth it * artom has quit (Ping timeout: 255 seconds) <peluse> SPDK doesn't automateically do any of that kind of detection <peluse> so that would have to be considered <notmyname> that makes sense <notmyname> I could imagine swift detecting that <peluse> and blocstore itself is pretty immature, need to point that out. We just now added code to recover from a dirty shutdown if that gives you an idea <notmyname> ok, so tell me (us) more about the blobstore. would that be a diskfile thing? <peluse> so this whole thing would be a proof of concept type activity for sure <notmyname> how does this make rledisez's LOSF work awesomer? <peluse> so yeah, I think diskfile would make sense <peluse> but I don't rememeber the details there of course. my brain is pretty small :) <peluse> In that slide deck you can see a super simple example of the interface <peluse> blobstore bascially takes over an entire disk, writes its own private metadata and then the app create "blobs" and does basic LBA sized reads and writes to them <notmyname> ah, ok <peluse> it can't handle sub-LBA access (by design) <peluse> well, we can them pages in blobstore but they're 4K <notmyname> that sounds like a haystack-in-a-library thing. or something similar to what you're working on rledisez <rledisez> yes, blobstore would be what we call volume. and I guess it embed its own k/v indexation. so it looks similar in some ways <peluse> yeah, I think the integration effort w/Swift for production would be a decent sized lift but for a POC may be worth it provided, maybe for container SSDs, the latency and CPU usage bebenfit made sense <notmyname> peluse: is there any spdk component that could replace sqlite? eg some kv store that does transactions? <notmyname> eg to replace the container layer * awaugama has quit (Quit: Leaving) <peluse> rocksDB would be the closest match, using blobstore as a backing component <peluse> but that's really what Wewe's proposal was - to add a k/v interface on blobstore <notmyname> ah ok. so a 3rd part db that works with spdk <peluse> yeah, maybe that's the best first step <notmyname> any questions from anyone, so far? <peluse> I can't remember what sqlite guts look like, can you easily replace the storage engine as its called in like MariaDB, anyone know? * dprince has quit (Quit: leaving) <notmyname> no <peluse> yeah, OK didn't think so <notmyname> sqlite is "just" a DB library <tdasilva> dumb question from me, but can you explain the difference from spdk and the intel cas tech? * xyang1 has quit (Quit: xyang1) <notmyname> ^ not a dumb question <peluse> sure, good question <peluse> they are totally different for one thing <peluse> CAS is a caching project/product that works between an app and the FS. <peluse> SPDK is a whole bunch of stuff, but not caching layers. It has to be integrated with an application unless you use one of the things like the compiled iSCSI target * e0ne has quit (Quit: My MacBook Pro has gone to sleep. ZZZzzz...) <peluse> dunno if that's enough explanation - block cache vs library of stuff for integration, mainly polled mode device driver for NVMe <peluse> so Q for you guys, is there any urgency with container SSDs and latency and/or using a bunch of CPU? <tdasilva> peluse, so spdk provides performance improvements by substituting the FS and writing directly to block storage <rledisez> do you handle caching in bdev or blobstore? or do you assume the underlaying device is fast enought <peluse> tdasilva, yup <peluse> rledisez, there's no data caching at all right now <tdasilva> peluse: very similar to bluestore? <peluse> bdev is a layer for abstracting different types of block devices. For example we can have an NVMe at the bottom of the stack or a RAM disk and for layers above bdev they don't care. its super light wieght <peluse> tdasilva, yeah, bluestore and blobstore area lot alike but bluestore was done of course just for Ceph and I think is more mature/feature rich right now <peluse> but Sage mentioned in his keynote at SNIA SDC about looking at maybe using rocksdb w/blobstore at some point in the future (dont quote me though) <peluse> that would be in addition to bluestore as backing FS though, no isntead of <notmyname> peluse: what questions do you have for us? <tdasilva> peluse: ack, thanks <peluse> jsut the one above about pain points wrt latency and or CPU utilization around SSDs * esberglu has quit (Remote host closed the connection) <peluse> well, and if anyone is interested enough to work with someone from the SPDK community to try and see if there's some sort of proof of concept worth messing with here <notmyname> only pain points I've seen recently with the container layer is drive fullness and the contaienr replicator not having all the goodness we've added to the object replicator for when drives fill up * priteau has quit () <notmyname> rledisez: how about you? any latency or cpu issues on containers or accounts? <rledisez> peluse: from my experience, there is not really a pain point about storage speed on containers. having a lot of containers slo down some process (like replicator) as they need to scan all db. not sure yet if blobstore would help here <peluse> wen I say CPU util, there's more in that deck I referenced, using SPDK (nvme + blobstore) greatly reduces CPU utillization while at the same time greatly improving perf <peluse> so you get kinda a two fer one thing <peluse> so for containers you'll get more CPU utillization for other things happening on the storage node, and the IOs will be faster and more repsonsive <peluse> (or your money back) <notmyname> heh <rledisez> how can you measure that CPU usage related to kernel/fs. i don't think i see any, but i would like to check <rledisez> most of the cpu usage comes from replicator or container-server <peluse> There's a perf blog on spdk.io that may have some good info in it, honestly I haven't read it :( <peluse> but we have some folks in our comm that live for that kinda stuff so I can ask there and get back to y'all <peluse> rledisez, yeah unless used for object storage wouldn't help w/replicator <rledisez> if you have a magic command to get the cpu usage i would be interested (i guess it would be something related to perf command) * msimonin has quit (Quit: Leaving.) * msimonin (~Adium(a)tsv35-1-78-232-147-61.fbx.proxad.net<mailto:[email protected]>) has joined <notmyname> honestly, spdk sounds really cool. it seems like something that would be great for an all-flash future. (but I'm not sure if anyone deloying swift is there yet) <peluse> rledisez, yeah I dunno the details of the various measurements but the team has looked at every metric known to man using a variety of tools * msimonin has quit (Client Quit) * msimonin (~Adium(a)tsv35-1-78-232-147-61.fbx.proxad.net<mailto:[email protected]>) has joined <notmyname> peluse: do you have people in the spdk community who are interested in swift? if so, are they interested because they just want to integrate spdk everywhere or because they are using swift already? * msimonin has quit (Client Quit) * Alex_Staf (~astafeye(a)bzq-109-65-185-7.red.bezeqint.net<mailto:[email protected]>) has joined * msimonin (~Adium(a)tsv35-1-78-232-147-61.fbx.proxad.net<mailto:[email protected]>) has joined <peluse> Wewe is the only person I know that's brought it up and he wasn't able to get connected today due to network issues * VW_ (~vw(a)50.56.228.68<mailto:[email protected]>) has joined * msimonin has quit (Client Quit) <peluse> right now there's more demand on features/integration than there is anything else so I don't think the former is driving anyone * msimonin (~Adium(a)tsv35-1-78-232-147-61.fbx.proxad.net<mailto:[email protected]>) has joined <notmyname> ok <peluse> which is one of the reasons I wanted to chat w/you guys about this - if it doesn't make a lot of sense to investigate from your perspective we certainly have enough work on our plate :) * msimonin has quit (Client Quit) * msimonin (~Adium(a)tsv35-1-78-232-147-61.fbx.proxad.net<mailto:[email protected]>) has joined <peluse> that's all I got for ya, other questions? <notmyname> I think it makes sense when looking a few years into the future and preparing for that. it doesn't make sense from the sense that all of our current employers have a huge amount of stuff we need to do in swift way before we get to needing spdk * msimonin has quit (Client Quit) <peluse> yup yup <notmyname> (my opinion) * msimonin (~Adium(a)tsv35-1-78-232-147-61.fbx.proxad.net<mailto:[email protected]>) has joined <peluse> what is the current split of SSD usage, still mostly containers? <notmyname> definitely something I want to keep an eye on <notmyname> yeah <peluse> cool * msimonin has quit (Client Quit) <notmyname> flash still too expensive for interesting-sized object server deployments * msimonin (~Adium(a)tsv35-1-78-232-147-61.fbx.proxad.net<mailto:[email protected]>) has joined <peluse> makes sense <notmyname> people these days are going for bigger nodes. 80 10TB in a single chassis <notmyname> (and getting all the eww that implies) * msimonin has quit (Client Quit) <peluse> well, that's not to say nobody on this end will work on a proof of concept anyways and if so I'll encourage them to check in the Swift comm frequently of course... * msimonin (~Adium(a)tsv35-1-78-232-147-61.fbx.proxad.net<mailto:[email protected]>) has joined <rledisez> i like the idea, and we can surely share some stuff between LOSF/blobstore but i think that people looking for really low latency object store will check ceph as by its design/implem, it looks more suited <notmyname> let's move on so we can give m_kazuhiro appropriate time :-) <notmyname> peluse: that's great! <peluse> thanks for the time guys!! <notmyname> and thanks for stopping by to give an update * VW has quit (Ping timeout: 264 seconds) <peluse> my pleasure... ping me later if anyone has followup questions. take care! <notmyname> rledisez: I can get you in contact with peluse if you can't find him on IRC late

4 years, 8 months

1
0
0 / 0

[SPDk] blockv uses Swift as a target system to support k-v store

by We We

Hi, @[email protected] I had submitted the “ blockv uses Swift as a target system ” on https://trello.com/b/P5xBO7UR/things-to-do <https://trello.com/b/P5xBO7UR/things-to-do>, please visit it. Thx, Helloway

4 years, 8 months

2
7
0 / 0

ppc64le support

by Jonas Pfefferle1

Hi @all, I just pushed a set of patches for review to gerrithub wich introduces support for ppc64le architecture: https://review.gerrithub.io/#/c/383725/ https://review.gerrithub.io/#/c/383726/ https://review.gerrithub.io/#/c/383727/ https://review.gerrithub.io/#/c/383728/ This patchset fixes the build on ppc architectures and page size alignment of cmd and cpl rings. Since ppc has less strict memory ordering rules it also introduces a memory barrier when polling for completions. Note that this only runs against the latest DPDK master which includes fixes to the sPAPR iommu. Currently it also requires to recompile the kernel with 4KB page size to allow mapping around the MSI-X table in BAR0. We are working on a patch to allow mapping the MSI-X table when interrupt remapping is enabled (this will work with 64KB default page size on ppc). Let me know what you guys think. Regards, Jonas

4 years, 8 months

2
2
0 / 0

compile error in old linux kernel version

by liupan1234

Hi All, I met an error when I tried to compile spdk under linux kernel 2.6.32, and gcc version is 4.9.2: $make DPDK_DIR=../dpdk/build/ CC lib/bdev/bdev.o bdev.c:1: error: bad value (native) for -march= switch bdev.c:1: error: bad value (native) for -mtune= switch make[2]: *** [bdev.o] Error 1 make[1]: *** [bdev] Error 2 make: *** [lib] Error 2 bdev.c:1: error: bad value (native) for -march= switch Could you give me some help? Thanks very much! Pan

4 years, 8 months

4
12
0 / 0

Re: [SPDK] Unusual interface and implementation of SPDK network function for iSCSI target

by Harris, James R

Hi Shuhei, All of your suggestions make sense to me. Refactoring net/sock.c make sense. The primary goal of this abstraction was to support multiple network stacks. Currently it is just the kernel network stack, but in the future there could be a userspace TCP stack implementation as well. Regarding removing * and [*], I would be curious to know if anyone reading the mailing list is depending on * and [*] and if changing it would cause them any difficulty. Maybe we could still support these for one release, but print warning message that these will be deprecated in the future. Could you see how difficult this would be to implement? Thanks, -Jim From: SPDK <spdk-bounces(a)lists.01.org> on behalf of 松本周平 / MATSUMOTO，SHUUHEI <shuhei.matsumoto.xt(a)hitachi.com> Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org> Date: Thursday, October 19, 2017 at 8:05 PM To: Storage Performance Development Kit <spdk(a)lists.01.org> Subject: Re: [SPDK] Unusual interface and implementation of SPDK network function for iSCSI target Sorry for repeated self reply. 2) User can use “*” as 0.0.0.0 or INADDR_ANY for IPv4 and “[*]” as [::] or in6addr_any for IPV6. I’m not so confident of my expertise for networking, but I’ve never heard of this odd interface. I would like to propose deleting the code related with “*” and “[*]”. does not cause any apparent error or misunderstanding. I did not understand why this interface is implemented. Hence I asked but if this interface is convenient, of course this should be maintained. Thank you, Shuhei Matsumoto From: 松本周平 / MATSUMOTO，SHUUHEI Sent: Friday, October 20, 2017 10:46 AM To: Storage Performance Development Kit Subject: RE: Unusual interface and implementation of SPDK network function for iSCSI target I’m preparing patch and change message for each one. I apologize for your inconvenience until that. From: 松本周平 / MATSUMOTO，SHUUHEI Sent: Friday, October 20, 2017 10:28 AM To: Storage Performance Development Kit Subject: Unusual interface and implementation of SPDK network function for iSCSI target Hi, I’m not so confident of networking but as long as I looked into the code I have found that at least the following items make iSCSI target erroneous or difficult to understand. The customized socket interface of SPDK (lib/net/sock.c) is only used in the SPDK iSCSI target. Hence I think that now may be a good chance to refactor. Related with my pushed changes I would like to change my priority to the following: 1. the change https://review.gerrithub.io/#/c/381246/ 2. the following (I have not registered into GerritHub except a few.) 3. remaining my pushed changes. I appreciate any feedback and I hope these would make sense for you and more standardized implementation will make connecting SPDK iSCSI target to user space TCP/IP stack easier. Best Regards, Shuhei Matsumoto 1) spdk_sock_getaddr(sock, saddr, slen, caddr, clen) (in lib/net/sock.c) can return only IPv4 address correctly, because get_addr_str() does not take into account of IPv6. Hence current code may not work in IPv6 correctly. static int get_addr_str(struct sockaddr_in *paddr, char *host, size_t hlen) { uint8_t *pa; if (paddr == NULL || host == NULL) return -1; pa = (uint8_t *)&paddr->sin_addr.s_addr; snprintf(host, hlen, "%u.%u.%u.%u", pa[0], pa[1], pa[2], pa[3]); return 0; } 2) User can use “*” as 0.0.0.0 or INADDR_ANY for IPv4 and “[*]” as [::] or in6addr_any for IPV6. I’m not so confident of my expertise for networking, but I’ve never heard of this odd interface. I would like to propose deleting the code related with “*” and “[*]”. 3) Network portal (struct spdk_iscsi_portal) remember IP address-port pair not as struct sockaddr but only string, host and port. Hence iSCSI target do not know if each network portal is IPv4 or IPv6 and have to check “[“ and “]” manually. If we strip “[“ and “]” from the user input and then pass it to getaddrinfo(), we can know and remember if it is IPv4 or IPv6. It will be helpful and we can delete the helper function spdk_sock_is_ipv6/4() in lib/net/sock.c. 4) In the spdk_iscsi_tgt_node_access(), “ALL” means that initiator group allows ANY IP address-port pair or iSCSI name of initiators. However iSCSI target do not know ALL initiators beforehand. Hence ANY may be better than ALL. 5) spdk_sock_connect() is not used anywhere. Hence abstraction by spdk_sock_create(LISTEN or CONNECT) is not necessary in lib/net/sock.c. 6) spdk_iscsi_portal_grp_is_visible may not run connrectly in the following case: - an iSCSI target has PG1-IG1 and PG1-IG2 - an initiator has logined to PG1 of the target. - the initiator is allowed by not IG1 but IG2. -> However spdk_iscsi_portal_grp_is_visible() only check the first IG among the same PG, that is, only IG1. Hence in this case spdk_iscsi_portal_grp_is_visible() should return true but would return false. -> I think this is caused by PG-IG map array and PG-IG map tree will be better (https://review.gerrithub.io/#/c/379933/). 7) I found OK or NG in the comment in lib/iscsi/tgt_node.c. These are not proper English and localized one in Japan. Hence it may be better to change to Allow and Deny because these are for ACL. 8) initiator group allow empty netmask list as a normal configuration. However we cannot create such one. (https://review.gerrithub.io/#/c/382920/) From: SPDK [mailto:[email protected]] On Behalf Of Victor Banh Sent: Thursday, October 19, 2017 4:17 PM To: Storage Performance Development Kit; Harris, James R; Cao, Gang Subject: Re: [SPDK] Buffer I/O error on bigger block size running fio I am using Ubuntu 16.04 and kernel 4.12.X. ________________________________ From: Cao, Gang <gang.cao(a)intel.com> Sent: Wednesday, October 18, 2017 11:51:39 PM To: Victor Banh; Storage Performance Development Kit; Harris, James R Subject: RE: [SPDK] Buffer I/O error on bigger block size running fio [[email protected] gangcao]# uname -a Linux node4 4.10.1 #1 SMP Fri Mar 10 15:59:57 CST 2017 x86_64 x86_64 x86_64 GNU/Linux [[email protected] gangcao]# lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 7.2.1511 (Core) Release: 7.2.1511 Codename: Core From: Victor Banh [mailto:[email protected]] Sent: Thursday, October 19, 2017 2:14 PM To: Storage Performance Development Kit <spdk(a)lists.01.org>; Harris, James R <james.r.harris(a)intel.com>; Cao, Gang <gang.cao(a)intel.com> Subject: RE: [SPDK] Buffer I/O error on bigger block size running fio Can you give the OS version and kernel version again for target and client? I couldn't compile dkdp without installed the latest Kernel on 4.12.X kernel on Ubuntu 16.04. Thanks Victor ________________________________ From: Cao, Gang <gang.cao(a)intel.com<mailto:[email protected]>> Sent: Wednesday, October 18, 2017 10:59:19 PM To: Victor Banh; Storage Performance Development Kit; Harris, James R Subject: RE: [SPDK] Buffer I/O error on bigger block size running fio This server does not have OFED installed and it is in the loopback mode. Found another server also in the loopback mode with ConnectX-3 and have OFED installed. By the way, what SSD are you using? Maybe it is relating to the SSD? I’ve just run with 2048k for a short duration and seems no issue. Will run more time to see whether can hit this error. [[email protected] fio]# lspci | grep -i mell 08:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [[email protected] fio]# lsmod | grep -i mlx mlx4_ib 159744 0 ib_core 208896 15 ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,nvme_rdma,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,ib_srpt,ib_ucm,rdma_ucm,mlx4_ib mlx4_en 114688 0 mlx4_core 307200 2 mlx4_en,mlx4_ib ptp 20480 3 ixgbe,igb,mlx4_en [[email protected] fio]# ofed_info MLNX_OFED_LINUX-3.2-2.0.0.0 (OFED-3.2-2.0.0): ar_mgr: osm_plugins/ar_mgr/ar_mgr-1.0-0.30.ga1ea4b7.tar.gz cc_mgr: osm_plugins/cc_mgr/cc_mgr-1.0-0.29.ga1ea4b7.tar.gz dapl: dapl.git mlnx_ofed_3_1 commit c30fb6ce2cbc29d8ed4bde51437f7abb93378c78 dump_pr: osm_plugins/dump_pr//dump_pr-1.0-0.25.ga1ea4b7.tar.gz fabric-collector: fabric_collector//fabric-collector-1.1.0.MLNX20140410.51b267e.tar.gz fca: mlnx_ofed_fca/fca-2.5.2431-1.src.rpm hcoll: mlnx_ofed_hcol/hcoll-3.4.807-1.src.rpm ibacm: mlnx_ofed/ibacm.git mlnx_ofed_3_2 commit 15ad8c13bdebbe62edea0b7df030710b65c14f7f ibacm_ssa: mlnx_ofed_ssa/acm/ibacm_ssa-0.0.9.3.MLNX20151203.50eb579.tar.gz ibdump: sniffer/sniffer-4.0.0-2/ibdump/linux/ibdump-4.0.0-2.tgz ibsim: mlnx_ofed_ibsim/ibsim-0.6-0.8.g9d76581.tar.gz ibssa: mlnx_ofed_ssa/distrib/ibssa-0.0.9.3.MLNX20151203.50eb579.tar.gz ibutils: ofed-1.5.3-rpms/ibutils/ibutils-1.5.7.1-0.12.gdcaeae2.tar.gz ibutils2: ibutils2/ibutils2-2.1.1-0.76.MLNX20160222.gd366c7b.tar.gz infiniband-diags: mlnx_ofed_infiniband_diags/infiniband-diags-1.6.6.MLNX20151130.7f0213e.tar.gz infinipath-psm: mlnx_ofed_infinipath-psm/infinipath-psm-3.3-2_g6f42cdb_open.tar.gz iser: mlnx_ofed/mlnx_rdma.git mlnx_ofed_3_2_2 commit 378ff029c77bac76cd02a8b89c6f3109bbb11c3d isert: mlnx_ofed/mlnx_rdma.git mlnx_ofed_3_2_2 commit 378ff029c77bac76cd02a8b89c6f3109bbb11c3d kernel-mft: mlnx_ofed_mft/kernel-mft-4.3.0-25.src.rpm knem: knem.git mellanox-master commit f143ee19a575cd42a334422fa8bd329d671238db libibcm: mlnx_ofed/libibcm.git mlnx_ofed_3_0 commit d7d485df305e6536711485bd7e477668e77d8320 libibmad: mlnx_ofed_libibmad/libibmad-1.3.12.MLNX20151122.d140cb1.tar.gz libibprof: mlnx_ofed_libibprof/libibprof-1.1.22-1.src.rpm libibumad: mlnx_ofed_libibumad/libibumad-1.3.10.2.MLNX20150406.966500d.tar.gz libibverbs: mlnx_ofed/libibverbs.git mlnx_ofed_3_2_2 commit 217a77686f4861229f0e4b94485a13f024634caf libmlx4: mlnx_ofed/libmlx4.git mlnx_ofed_3_2_2 commit dda6d7ae1e6e3779a485ebdd0a882f4bcbd027a6 libmlx5: mlnx_ofed/libmlx5.git mlnx_ofed_3_2_2 commit d0c8645359e0f0aba0408b2d344f3b418d27019b libopensmssa: mlnx_ofed_ssa/plugin/libopensmssa-0.0.9.3.MLNX20151203.50eb579.tar.gz librdmacm: mlnx_ofed/librdmacm.git mlnx_ofed_3_2_2 commit 6bd430fed9e7b3d57a1876c040431ce7295c7703 libsdp: libsdp.git mlnx_ofed_3_0 commit fbd01dfff05f42d6b82506e7dbf4bc6b7e6a59a4 libvma: vma/source_rpms//libvma-7.0.14-0.src.rpm mlnx-ethtool: upstream/ethtool.git for-upstream commit ac0cf295abe0c0832f0711fed66ab9601c8b2513 mlnx-ofa_kernel: mlnx_ofed/mlnx_rdma.git mlnx_ofed_3_2_2 commit 378ff029c77bac76cd02a8b89c6f3109bbb11c3d mpi-selector: ofed-1.5.3-rpms/mpi-selector/mpi-selector-1.0.3-1.src.rpm mpitests: mlnx_ofed_mpitest/mpitests-3.2.17-e1c7f2f.src.rpm mstflint: mlnx_ofed_mstflint/mstflint-4.3.0-1.49.g9b9af70.tar.gz multiperf: mlnx_ofed_multiperf/multiperf-3.0-0.10.gda89e8c.tar.gz mvapich2: mlnx_ofed_mvapich2/mvapich2-2.2a-1.src.rpm mxm: mlnx_ofed_mxm/mxm-3.4.3079-1.src.rpm ofed-docs: docs.git mlnx_ofed-3.2 commit ea3386416f9f7130edd2c70fc3424cb2cda50f7d openmpi: mlnx_ofed_ompi_1.8/openmpi-1.10.3a1-1.src.rpm opensm: mlnx_ofed_opensm/opensm-4.6.1.MLNX20160112.774e977.tar.gz perftest: mlnx_ofed_perftest/perftest-3.0-0.18.gb464d59.tar.gz qperf: mlnx_ofed_qperf/qperf-0.4.9.tar.gz rds-tools: rds-tools.git mlnx_ofed_2_4 commit 299420ca25cf9996bc0748e3bc4b08748996ba49 sdpnetstat: sdpnetstat.git mlnx_ofed_3_0 commit 3cf409a7cc07e5c71f9640eddbb801ece21b4169 sockperf: sockperf/sockperf-2.7-43.git3ee62bd8107a.src.rpm srp: mlnx_ofed/mlnx_rdma.git mlnx_ofed_3_2_2 commit 378ff029c77bac76cd02a8b89c6f3109bbb11c3d srptools: srptools/srptools-1.0.2-12.src.rpm Installed Packages: ------------------- infiniband-diags librdmacm libmlx4 libibverbs-utils mpi-selector libibmad-devel sdpnetstat knem libibumad-devel libsdp mlnx-ethtool libibverbs-debuginfo mlnx-ofa_kernel-modules srp opensm mstflint cc_mgr libibmad libibverbs kernel-mft libibverbs-devel-static libibumad librdmacm-devel mlnx-ofa_kernel libsdp-devel ibutils2 mlnxofed-docs libibmad-static iser opensm-devel dump_pr libibumad-static rds-tools libmlx4-debuginfo mlnx-ofa_kernel-devel opensm-libs opensm-static ar_mgr dapl-devel-static infiniband-diags-compat libibverbs-devel ibsim [[email protected] fio]# ibstat CA 'mlx4_0' CA type: MT4103 Number of ports: 2 Firmware version: 2.35.5100 Hardware version: 0 Node GUID: 0x248a0703006090e0 System image GUID: 0x248a0703006090e0 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x04010000 Port GUID: 0x268a07fffe6090e0 Link layer: Ethernet Port 2: State: Active Physical state: LinkUp Rate: 40 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x04010000 Port GUID: 0x268a07fffe6090e1 Link layer: Ethernet [[email protected] fio]# ibv_devinfo hca_id: mlx4_0 transport: InfiniBand (0) fw_ver: 2.35.5100 node_guid: 248a:0703:0060:90e0 sys_image_guid: 248a:0703:0060:90e0 vendor_id: 0x02c9 vendor_part_id: 4103 hw_ver: 0x0 board_id: MT_1090111023 phys_port_cnt: 2 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet port: 2 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet From: Victor Banh [mailto:[email protected]] Sent: Thursday, October 19, 2017 12:34 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>>; Harris, James R <james.r.harris(a)intel.com<mailto:[email protected]>>; Cao, Gang <gang.cao(a)intel.com<mailto:[email protected]>> Subject: RE: [SPDK] Buffer I/O error on bigger block size running fio Do you install Mellanox OFED on the Target and Client server? Can you run ibstat on both servers? Thanks Victor ________________________________ From: Cao, Gang <gang.cao(a)intel.com<mailto:[email protected]>> Sent: Wednesday, October 18, 2017 8:59:28 PM To: Victor Banh; Storage Performance Development Kit; Harris, James R Subject: RE: [SPDK] Buffer I/O error on bigger block size running fio Hi Victor, I’ve just tried the SPDK v17.07.1 and DPDK v17.08. nvme version: 1.1.38.gfaab fio version: 3.1 Tried the 512k and 1024k IO size and there is no error. demsg information as following. So that there may be other difference here? Looks like you are using ConnectX-5 while I am using ConnectX-4? Other related information: [[email protected] fio]# lspci | grep -i mell 81:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] 81:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [[email protected] fio]# lsmod | grep -i mlx mlx5_ib 172032 0 ib_core 200704 15 ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,nvme_rdma,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,mlx5_ib,ib_srpt,ib_ucm,rdma_ucm mlx5_core 380928 1 mlx5_ib ptp 20480 3 ixgbe,igb,mlx5_core [[email protected] fio]# uname -a Linux node4 4.10.1 #1 SMP Fri Mar 10 15:59:57 CST 2017 x86_64 x86_64 x86_64 GNU/Linux [577707.543326] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.100.8:4420 [577730.854540] detected loopback device [577730.893761] nvme nvme0: creating 7 I/O queues. [577730.893797] detected loopback device [577730.898611] detected loopback device [577730.908917] detected loopback device [577730.919073] detected loopback device [577730.928922] detected loopback device [577730.938679] detected loopback device [577730.948365] detected loopback device [577731.146290] nvme nvme0: new ctrl: NQN "nqn.2016-06.io.spdk:cnode2", addr 192.168.100.8:4420 Thanks, Gang From: Victor Banh [mailto:[email protected]] Sent: Thursday, October 19, 2017 9:43 AM To: Cao, Gang <gang.cao(a)intel.com<mailto:[email protected]>>; Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>>; Harris, James R <james.r.harris(a)intel.com<mailto:[email protected]>> Subject: RE: [SPDK] Buffer I/O error on bigger block size running fio Hi Gang Any update? Do you see any error message from “dmesg” with 512k block size running fio? Thanks Victor From: Victor Banh Sent: Tuesday, October 17, 2017 7:37 PM To: 'Cao, Gang' <gang.cao(a)intel.com<mailto:[email protected]>>; Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>>; Harris, James R <james.r.harris(a)intel.com<mailto:[email protected]>> Subject: RE: [SPDK] Buffer I/O error on bigger block size running fio Hi Gang spdk-17.07.1 and dpdk-17.08 Thanks Victor From: Cao, Gang [mailto:[email protected]] Sent: Monday, October 16, 2017 8:51 PM To: Victor Banh <victorb(a)mellanox.com<mailto:[email protected]>>; Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>>; Harris, James R <james.r.harris(a)intel.com<mailto:[email protected]>> Subject: RE: [SPDK] Buffer I/O error on bigger block size running fio Hi Victor, Could you share us which version of SPDK you are using when seeing this error? Or maybe you can have a try with the latest SPDK code? Thanks, Gang From: Victor Banh [mailto:[email protected]] Sent: Tuesday, October 17, 2017 5:30 AM To: Cao, Gang <gang.cao(a)intel.com<mailto:[email protected]>>; Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>>; Harris, James R <james.r.harris(a)intel.com<mailto:[email protected]>> Subject: RE: [SPDK] Buffer I/O error on bigger block size running fio Hi Cao Do you see any message from dmesg? I tried this fio version and still saw these error message from dmesg. fio-3.1 [869053.218235] Buffer I/O error on dev nvme2n1, logical block 0, async page read [869053.218250] Buffer I/O error on dev nvme2n1, logical block 0, async page read [869053.218259] Buffer I/O error on dev nvme2n1, logical block 0, async page read [869053.218263] ldm_validate_partition_table(): Disk read failed. [869053.218269] Buffer I/O error on dev nvme2n1, logical block 0, async page read [869053.218277] Buffer I/O error on dev nvme2n1, logical block 0, async page read [869053.218285] Buffer I/O error on dev nvme2n1, logical block 0, async page read [869053.218292] Buffer I/O error on dev nvme2n1, logical block 0, async page read [869053.218296] Dev nvme2n1: unable to read RDB block 0 [869053.218303] Buffer I/O error on dev nvme2n1, logical block 0, async page read [869053.218311] Buffer I/O error on dev nvme2n1, logical block 0, async page read [869053.218323] Buffer I/O error on dev nvme2n1, logical block 3, async page read [869053.218338] nvme2n1: unable to read partition table [869053.246126] nvme2n1: detected capacity change from -62111005559226368 to -62042256479092736 [869053.246195] ldm_validate_partition_table(): Disk read failed. [869053.246217] Dev nvme2n1: unable to read RDB block 0 From: Cao, Gang [mailto:[email protected]] Sent: Monday, October 09, 2017 10:59 AM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>>; Harris, James R <james.r.harris(a)intel.com<mailto:[email protected]>> Cc: Victor Banh <victorb(a)mellanox.com<mailto:[email protected]>> Subject: RE: [SPDK] Buffer I/O error on bigger block size running fio Hi Victor, Thanks for your detailed information on the testing. I’ve tried the latest SPDK code and with latest fio-3.1-20-g132b and fio-2.19. It seems like no this kind of error. Could you share us which version of SPDK you are using when seeing this error? Or maybe you can have a try with the latest SPDK code? fio --bs=512k --numjobs=4 --iodepth=16 --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --time_based --runtime=60 --filename=/dev/nvme0n1 --name=read-phase --rw=randwrite read-phase: (g=0): rw=randwrite, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=16 ... fio-3.1-20-g132b Starting 4 processes Jobs: 4 (f=4): [w(4)][100.0%][r=0KiB/s,w=1592MiB/s][r=0,w=3183 IOPS][eta 00m:00s] read-phase: (groupid=0, jobs=1): err= 0: pid=46378: Tue Oct 10 01:23:39 2017 My NIC information: [[email protected] nvme-cli-gerrit]# lsmod | grep -i mlx mlx5_ib 172032 0 ib_core 200704 15 ib_iser,ib_cm,rdma_cm,ib_umad,ib_srp,nvme_rdma,ib_isert,ib_uverbs,rpcrdma,ib_ipoib,iw_cm,mlx5_ib,ib_srpt,ib_ucm,rdma_ucm mlx5_core 380928 1 mlx5_ib ptp 20480 3 ixgbe,igb,mlx5_core [[email protected] nvme-cli-gerrit]# lspci | grep -i mell 81:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] 81:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] From: SPDK [mailto:[email protected]] On Behalf Of Victor Banh Sent: Friday, October 6, 2017 2:41 PM To: Harris, James R <james.r.harris(a)intel.com<mailto:[email protected]>>; Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>> Subject: Re: [SPDK] Buffer I/O error on bigger block size running fio From: Harris, James R [mailto:[email protected]] Sent: Friday, October 06, 2017 2:32 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>> Cc: Victor Banh <victorb(a)mellanox.com<mailto:[email protected]>> Subject: Re: [SPDK] Buffer I/O error on bigger block size running fio (cc Victor) From: James Harris <james.r.harris(a)intel.com<mailto:[email protected]>> Date: Thursday, October 5, 2017 at 1:59 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>> Subject: Re: [SPDK] Buffer I/O error on bigger block size running fio Hi Victor, Could you provide a few more details? This will help the list to provide some ideas. 1) On the client, are you using the SPDK NVMe-oF initiator or the kernel initiator? Kernel initiator, run these commands on client server. modprobe mlx5_ib modprobe nvme-rdma nvme discover -t rdma -a 192.168.10.11 -s 4420 nvme connect -t rdma -n nqn.2016-06.io.spdk:nvme-subsystem-1 -a 192.168.10.11 -s 4420 2) Can you provide the fio configuration file or command line? Just so we can have more specifics on “bigger block size”. fio --bs=512k --numjobs=4 --iodepth=16 --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --time_based --runtime=60 --filename=/dev/nvme1n1 --name=read-phase --rw=randwrite 3) Any details on the HW setup – specifically details on the RDMA NIC (or if you’re using SW RoCE). Nvmf.conf on target server [Global] Comment "Global section" ReactorMask 0xff00 [Rpc] Enable No Listen 127.0.0.1 [Nvmf] MaxQueuesPerSession 8 MaxQueueDepth 128 [Subsystem1] NQN nqn.2016-06.io.spdk:nvme-subsystem-1 Core 9 Mode Direct Listen RDMA 192.168.10.11:4420 NVMe 0000:82:00.0 SN S2PMNAAH400039 It is RDMA NIC, ConnectX 5, Intel CPU Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15 Thanks, -Jim From: SPDK <spdk-bounces(a)lists.01.org<mailto:[email protected]>> on behalf of Victor Banh <victorb(a)mellanox.com<mailto:[email protected]>> Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>> Date: Thursday, October 5, 2017 at 11:26 AM To: "spdk(a)lists.01.org<mailto:[email protected]>" <spdk(a)lists.01.org<mailto:[email protected]>> Subject: [SPDK] Buffer I/O error on bigger block size running fio Hi I have SPDK NVMeoF and keep getting error with bigger block size with fio on randwrite tests. I am using Ubuntu 16.04 with kernel version 4.12.0-041200-generic on target and client. The DPDK is 17.08 and SPDK is 17.07.1. Thanks Victor [46905.233553] perf: interrupt took too long (2503 > 2500), lowering kernel.perf_event_max_sample_rate to 79750 [48285.159186] blk_update_request: I/O error, dev nvme1n1, sector 2507351968 [48285.159207] blk_update_request: I/O error, dev nvme1n1, sector 1301294496 [48285.159226] blk_update_request: I/O error, dev nvme1n1, sector 1947371168 [48285.159239] blk_update_request: I/O error, dev nvme1n1, sector 1891797568 [48285.159252] blk_update_request: I/O error, dev nvme1n1, sector 10833824 [48285.159265] blk_update_request: I/O error, dev nvme1n1, sector 614937152 [48285.159277] blk_update_request: I/O error, dev nvme1n1, sector 1872305088 [48285.159290] blk_update_request: I/O error, dev nvme1n1, sector 1504491040 [48285.159299] blk_update_request: I/O error, dev nvme1n1, sector 1182136128 [48285.159308] blk_update_request: I/O error, dev nvme1n1, sector 1662985792 [48285.191185] nvme nvme1: Reconnecting in 10 seconds... [48285.191254] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191291] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191305] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191314] ldm_validate_partition_table(): Disk read failed. [48285.191320] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191327] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191335] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191342] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191347] Dev nvme1n1: unable to read RDB block 0 [48285.191353] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191360] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191375] Buffer I/O error on dev nvme1n1, logical block 3, async page read [48285.191389] nvme1n1: unable to read partition table [48285.223197] nvme1n1: detected capacity change from 1600321314816 to 0 [48289.623192] nvme1n1: detected capacity change from 0 to -65647705833078784 [48289.623411] ldm_validate_partition_table(): Disk read failed. [48289.623447] Dev nvme1n1: unable to read RDB block 0 [48289.623486] nvme1n1: unable to read partition table [48289.643305] ldm_validate_partition_table(): Disk read failed. [48289.643328] Dev nvme1n1: unable to read RDB block 0 [48289.643373] nvme1n1: unable to read partition table

4 years, 8 months

2
1
0 / 0

2022

2021

2020

2019

2018

2017

2016

SPDK October 2017