Re: [SPDK] nvme_ctrlr.c:1224:nvme_ctrlr_process_init: ***ERROR*** Initialization timed out in state 3

On Apr 4, 2017, at 7:18 AM, Oza Oza <oza.oza(a)broadcom.com&gt; wrote: And I am using following command. fio --randrepeat=1 --ioengine=/home/oza/SPDK/spdk/examples/nvme/fio_plugin/fio_plugin --direct=1 --numjobs=1 --gtod_reduce=1 --name=readtest --filename=0000.01.00.00/1 --bs=4096 --iodepth=128 --size=4G --readwrite=read

Regards, Oza. -----Original Message----- From: Oza Oza [mailto:[email protected]] Sent: Tuesday, April 04, 2017 2:55 PM To: 'Storage Performance Development Kit' Subject: RE: [SPDK] nvme_ctrlr.c:1224:nvme_ctrlr_process_init: ***ERROR*** Initialization timed out in state 3 Hi Harris, Thanks for the response. for time-being, I am stuck with one after the other problem. Although problems are being resolved. Whatever I posted on 1st of April is resolved, and that exposed bug in our RC driver where I was not checking competition status. That is now solved. The controller timeout also is resolved by adding some delay after SPDK nvme_ctrlr.c And I think all the errors have nothing to do with ARMv8 porting, because mostly it compiled straightly, there are some minor patches. Which I will let you know, also at the same time let me know how do I contribute/discuss any change ? Now I am stuck here: let me know if you have clue. odepth=128 --size=4G --readwrite=read --filename=0000.01.00.00/1 --bs=4096 --i /home/oza/fio /home/oza/SPDK/spdk EAL: pci driver is being registered 0x1nreadtest: (g=0): rw=read, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=spdk_fio, iodepth=128 fio-2.17-29-gf0ac1 Starting 1 process Starting Intel(R) DPDK initialization ... [ DPDK EAL parameters: fio -c 1 --file-prefix=spdk_pid6177 --base-virtaddr=0x1000000000 --proc-type=auto ] EAL: Detected 8 lcore(s) EAL: Auto-detected process type: PRIMARY EAL: Probing VFIO support... EAL: VFIO support initialized EAL: cannot open /proc/self/numa_maps, consider that all memory is in socket_id 0 EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme EAL: using IOMMU type 1 (Type 1) EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=13 EAL: reg=0x2000 fd=13 cap_offset=0x50 EAL: Hotplug doesn't support vfio yet EAL: probe driver: 8086:953 spdk_nvme EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=16 EAL: reg=0x2000 fd=16 cap_offset=0x50 nvme_pcie.c: 994:nvme_pcie_qpair_complete_pending_admin_request:[ 1041.919073] pgd = ffff800975800000 ***ERROR*** the active process (pid 6184) is not found for this[ 1041.928312] [00000008] *pgd=00000009f5815003 controller. fio: pid=6184, got signal=11 readtest: (groupid=0, jobs=1): err= 0: pid=6184: Tue Apr 4 09:12:28 2017 read: IOPS=442k, BW=1727MiB/s (1811MB/s)(4096MiB/2372msec) cpu : usr=99.87%, sys=0.00%, ctx=238, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, > =64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > =64=0.1% issued rwt: total=1048576,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): READ: bw=1727MiB/s (1811MB/s), 1727MiB/s-1727MiB/s (1811MB/s-1811MB/s), io=4096MiB (4295MB), run=2372-2372msec Regards, Oza. -----Original Message----- From: SPDK [mailto:[email protected]] On Behalf Of Harris, James R Sent: Monday, April 03, 2017 9:12 PM To: Storage Performance Development Kit Subject: Re: [SPDK] nvme_ctrlr.c:1224:nvme_ctrlr_process_init: ***ERROR*** Initialization timed out in state 3 Hi Oza, > On Apr 3, 2017, at 2:01 AM, Oza Oza <oza.oza(a)broadcom.com&gt; wrote: > > I have ported SPDK for ARMv8. Great! Do you have patches that you can share? > And DPDK is compiled with version: 16.11.1 > > init controller is failing. Does this mean that the previous issue you reported on April 1 has been resolved? If so, can you share details? > > [email protected]:/home/oza/SPDK/spdk# > odepth=128 --size=4G --readwrite=read --filename=0000.01.00.00/1 > --bs=4096 --i /home/oza/fio /home/oza/SPDK/spdk > EAL: pci driver is being registered 0x1nreadtest: (g=0): rw=read, > bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=spdk_fio, > iodepth=128 > fio-2.17-29-gf0ac1 > Starting 1 process > Starting Intel(R) DPDK initialization ... > [ DPDK EAL parameters: fio -c 1 --file-prefix=spdk_pid6448 > --base-virtaddr=0x1000000000 --proc-type=auto ] > EAL: Detected 8 lcore(s) > EAL: Auto-detected process type: PRIMARY > EAL: Probing VFIO support... > EAL: VFIO support initialized > EAL: cannot open /proc/self/numa_maps, consider that all memory is in > socket_id 0 > EAL: PCI device 0000:01:00.0 on NUMA socket 0 > EAL: probe driver: 8086:953 spdk_nvme > EAL: using IOMMU type 1 (Type 1) > EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=13 > EAL: reg=0x2000 fd=13 cap_offset=0x50 > EAL: the msi-x bar number is 0 0x2000 0x200 > EAL: Hotplug doesn't support vfio yet > spdk_fio_setup() is being called > EAL: PCI device 0000:01:00.0 on NUMA socket 0 > EAL: probe driver: 8086:953 spdk_nvme > EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=16 > EAL: reg=0x2000 fd=16 cap_offset=0x50 > EAL: the msi-x bar number is 0 0x2000 0x200 > EAL: inside pci_vfio_write_config offset=4 > nvme_ctrlr.c:1224:nvme_ctrlr_process_init: ***ERROR*** Initialization > timed out in state 3 > nvme_ctrlr.c: 403:nvme_ctrlr_shutdown: ***ERROR*** did not shutdown > within > 5 seconds I think this is where seeing your ARMv8 patches would help. Then we could collaborate on if/how the ARMv8 changes you have made may be causing the initialization error. > EAL: Hotplug doesn't support vfio yet > EAL: PCI device 0000:01:00.0 on NUMA socket 0 > EAL: probe driver: 8086:953 spdk_nvme > EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=18 > EAL: reg=0x2000 fd=18 cap_offset=0x50 > EAL: the msi-x bar number is 0 0x2000 0x200 > EAL: Hotplug doesn't support vfio yet > EAL: Requested device 0000:01:00.0 cannot be used > spdk_nvme_probe() failed Thanks, -Jim _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk