May 2017 - SPDK - Ml01.01.Org

by Isaac Otsiabah

I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks. Isaac

2 years, 10 months

3
6
0 / 0

FW: VM oops while testing SPDK hotplug

by Isaac Otsiabah

Resending the last email. From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah Sent: Monday, May 15, 2017 5:09 PM To: Storage Performance Development Kit <spdk(a)lists.01.org> Subject: Re: [SPDK] VM crashes while testing SPDK hotplug Daniel, the attached hotplug.sh is modified to skip the VM creation and spdk copying sections. I ran it twice, and it oops the second time with the script output redirected to /tmp/hotplug.out4 file. The oops trace is similar to the one sent this morning. [root@host1k]# ./test/lib/nvme/hotplug.sh 2>&1 |tee /tmp/hotplug.out3 [root@host1]# ./test/lib/nvme/hotplug.sh 2>&1 |tee /tmp/hotplug.out4 Isaac From: Isaac Otsiabah Sent: Monday, May 15, 2017 1:08 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Cc: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:IOtsiabah@us.fujitsu.com>>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com<mailto:PVonStamwitz@us.fujitsu.com>>; Edward Yang <eyang(a)us.fujitsu.com<mailto:eyang@us.fujitsu.com>> Subject: RE: VM crashes while testing SPDK hotplug Daniel, i installed a Fedora 24 VM and test it. After running the test twice or more, the VM oops. Unlike the previous failure on Centos, this failure does not reboot but VM oops after two or more test run. My host is a Centos machine. I found the qemu-kvm which comes with the OS installation does not support nvme so I build qemu-system-x86_64 version 2.9. [root@host1 spdk]# /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 -version QEMU emulator version 2.9.0 Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers One observation on (although this is not the problem because I executed the scripts/setup.sh and the hotplug binary from vm console during appropriate breakpoints because local port 10022 was not responsive), the hotplug.h has the flag "-net user,hostfwd=tcp::10022-:22 \" to redirect gust ssh port 22 to host port 10022. However, qemu-system-x86_64 version 2.9 does not have this option but it has -netdev option but is is different. The qemu-system-86_64 man page on -netdev flag is as follows: -netdev user,id=str[,ipv4[=on|off]][,net=addr[/mask]][,host=addr] [,ipv6[=on|off]][,ipv6-net=addr[/int]][,ipv6-host=addr] [,restrict=on|off][,hostname=host][,dhcpstart=addr] [,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,tftp=dir] [,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]] configure a user mode network backend with ID 'str', its DHCP server and optional services It says hostfwd=rule and does not give detail of the rule. I used tcp so I specified it as -netdev user,id=hotplug,hostfwd=tcp::10022-:22 \ >From the host "netstat -an |egrep -I listen |less" I see local port 10022 is being listened on. I installed sshpass and tested this -netdev flag redirection with a simple sshpass command to the vm but got no response. Therefore, i bypassed executing scripts/setup.sh and the hotplug binary using sshpass command. So I can test it without executing setup.sh and the hotplug binary through sshpass on port 10022. The main issue is why does it oops after I run it 2 or more times? Isaac From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Verkamp, Daniel Sent: Tuesday, May 09, 2017 3:33 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] VM crashes while testing SPDK hotplug Hi Isaac, Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5. I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels). Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue? Thanks, -- Daniel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah Sent: Tuesday, May 9, 2017 2:11 PM To: spdk(a)lists.01.org<mailto:spdk@lists.01.org> Subject: [SPDK] VM crashes while testing SPDK hotplug I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks. Isaac

2 years, 10 months

1
0
0 / 0

FW: VM oops while testing SPDK hotplug

by Isaac Otsiabah

Daniel, I am not sure you saw the messages I sent this morning so I am resending them. Isaac From: Isaac Otsiabah Sent: Monday, May 15, 2017 1:08 PM To: Storage Performance Development Kit <spdk(a)lists.01.org> Cc: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com>; Edward Yang <eyang(a)us.fujitsu.com> Subject: RE: VM crashes while testing SPDK hotplug Daniel, i installed a Fedora 24 VM and test it. After running the test twice or more, the VM oops. Unlike the previous failure on Centos, this failure does not reboot but VM oops after two or more test run. My host is a Centos machine. I found the qemu-kvm which comes with the OS installation does not support nvme so I build qemu-system-x86_64 version 2.9. [root@host1 spdk]# /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 -version QEMU emulator version 2.9.0 Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers One observation on (although this is not the problem because I executed the scripts/setup.sh and the hotplug binary from vm console during appropriate breakpoints because local port 10022 was not responsive), the hotplug.h has the flag "-net user,hostfwd=tcp::10022-:22 \" to redirect gust ssh port 22 to host port 10022. However, qemu-system-x86_64 version 2.9 does not have this option but it has -netdev option but is is different. The qemu-system-86_64 man page on -netdev flag is as follows: -netdev user,id=str[,ipv4[=on|off]][,net=addr[/mask]][,host=addr] [,ipv6[=on|off]][,ipv6-net=addr[/int]][,ipv6-host=addr] [,restrict=on|off][,hostname=host][,dhcpstart=addr] [,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,tftp=dir] [,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]] configure a user mode network backend with ID 'str', its DHCP server and optional services It says hostfwd=rule and does not give detail of the rule. I used tcp so I specified it as -netdev user,id=hotplug,hostfwd=tcp::10022-:22 \ >From the host "netstat -an |egrep -I listen |less" I see local port 10022 is being listened on. I installed sshpass and tested this -netdev flag redirection with a simple sshpass command to the vm but got no response. Therefore, i bypassed executing scripts/setup.sh and the hotplug binary using sshpass command. So I can test it without executing setup.sh and the hotplug binary through sshpass on port 10022. The main issue is why does it oops after I run it 2 or more times? Isaac From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Verkamp, Daniel Sent: Tuesday, May 09, 2017 3:33 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] VM crashes while testing SPDK hotplug Hi Isaac, Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5. I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels). Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue? Thanks, -- Daniel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah Sent: Tuesday, May 9, 2017 2:11 PM To: spdk(a)lists.01.org<mailto:spdk@lists.01.org> Subject: [SPDK] VM crashes while testing SPDK hotplug I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks. Isaac

2 years, 10 months

1
0
0 / 0

FW: VM oops while testing SPDK hotplug

by Isaac Otsiabah

Forwarding this message I sent this morning. From: Isaac Otsiabah Sent: Monday, May 15, 2017 1:43 PM To: Storage Performance Development Kit <spdk(a)lists.01.org> Cc: Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com>; Edward Yang <eyang(a)us.fujitsu.com>; Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com> Subject: RE: VM crashes while testing SPDK hotplug Daniel, please, do you have an updated version of the hotplug.sh script you can share with us? I created the VM using this exact command on my Centos 7 host IMAGE=/home/fedora24/fedora24.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M FEDORA_ISO=/tmp/Fedora-Server-dvd-x86_64-24-1.2.iso /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -netdev user,id=hotplug,hostfwd=tcp::10022-:22 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid_fedora.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $FEDORA_ISO After the install is complete, I setup the guest IP address in /etc/sysconfig/network-scripts/ifcfg-ens3 and brings up the interface with ./ifup ens3 >From the VM, I clone spdk and build. Then I run the group of test in hotplug.sh skipping the VM creation and copying spdk to the VM sections. I mentioned the -netdev flag in my earlier email Isaac From: Isaac Otsiabah Sent: Monday, May 15, 2017 1:08 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Cc: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:IOtsiabah@us.fujitsu.com>>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com<mailto:PVonStamwitz@us.fujitsu.com>>; Edward Yang <eyang(a)us.fujitsu.com<mailto:eyang@us.fujitsu.com>> Subject: RE: VM crashes while testing SPDK hotplug Daniel, i installed a Fedora 24 VM and test it. After running the test twice or more, the VM oops. Unlike the previous failure on Centos, this failure does not reboot but VM oops after two or more test run. My host is a Centos machine. I found the qemu-kvm which comes with the OS installation does not support nvme so I build qemu-system-x86_64 version 2.9. [root@host1 spdk]# /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 -version QEMU emulator version 2.9.0 Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers One observation on (although this is not the problem because I executed the scripts/setup.sh and the hotplug binary from vm console during appropriate breakpoints because local port 10022 was not responsive), the hotplug.h has the flag "-net user,hostfwd=tcp::10022-:22 \" to redirect gust ssh port 22 to host port 10022. However, qemu-system-x86_64 version 2.9 does not have this option but it has -netdev option but is is different. The qemu-system-86_64 man page on -netdev flag is as follows: -netdev user,id=str[,ipv4[=on|off]][,net=addr[/mask]][,host=addr] [,ipv6[=on|off]][,ipv6-net=addr[/int]][,ipv6-host=addr] [,restrict=on|off][,hostname=host][,dhcpstart=addr] [,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,tftp=dir] [,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]] configure a user mode network backend with ID 'str', its DHCP server and optional services It says hostfwd=rule and does not give detail of the rule. I used tcp so I specified it as -netdev user,id=hotplug,hostfwd=tcp::10022-:22 \ >From the host "netstat -an |egrep -I listen |less" I see local port 10022 is being listened on. I installed sshpass and tested this -netdev flag redirection with a simple sshpass command to the vm but got no response. Therefore, i bypassed executing scripts/setup.sh and the hotplug binary using sshpass command. So I can test it without executing setup.sh and the hotplug binary through sshpass on port 10022. The main issue is why does it oops after I run it 2 or more times? Isaac From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Verkamp, Daniel Sent: Tuesday, May 09, 2017 3:33 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] VM crashes while testing SPDK hotplug Hi Isaac, Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5. I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels). Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue? Thanks, -- Daniel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah Sent: Tuesday, May 9, 2017 2:11 PM To: spdk(a)lists.01.org<mailto:spdk@lists.01.org> Subject: [SPDK] VM crashes while testing SPDK hotplug I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks. Isaac

2 years, 10 months

1
0
0 / 0

April Summit Presentations

by Stern, Jonathan

In case you wanted to review the presentations from the April 19-20 Summit, links have been posted both on spdk.io: http://www.spdk.io/news/2017/05/03/summit_presentations/ And via the official registration page (hover over the link and click "Download Presentation"): https://builders.intel.com/events2017/developer-summit-santaclara/event

2 years, 10 months

1
0
0 / 0

Any idea for SPDK to limit the data transfer speed

by nixun_992＠sina.com

Hi, Community: For one nvme device, if i config SPDK with two partitions, is there any way to limit the transfer speed to a fix speed? I know the kernel has cgroup limit and buffered io limit, if i want to implement this in SPDK, is there a way to archive this? Thanks, Xun

2 years, 11 months

4
3
0 / 0

invalid usermode virtual address 0xffff8f600000

by Oza Oza

Hi, Why do I get invalid user mode virtual address when I enabled DEBUG in SPDK/DPDK ? invalid usermode virtual address 0xffff8f600000 invalid usermode virtual address 0xffff0f800000 invalid usermode virtual address 0xffff0f400000 this is armv8. Regards, Oza.

2 years, 11 months

3
4
0 / 0

回复：Re: Any idea for SPDK to limit the data transfer speed

by nixun_992＠sina.com

Thank you changpeng, Get your way to limit the speed. But that seems to be NVMe only, not a common infrastruct for speed limit. Thanks, Xun --------------- Hi Xun, Very good question, but currently SPDK does not have such feature. One crude implementation: you can modify function bdev_nvme_poll: spdk_nvme_qpair_process_completions(qpair, 0), replace 0 with a different number such as 8. This means batch of I/O at a time instead of wrap around. From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of nixun_992(a)sina.com Sent: Sunday, May 7, 2017 3:17 PM To: spdk <spdk(a)lists.01.org> Subject: [SPDK] Any idea for SPDK to limit the data transfer speed Hi, Community: For one nvme device, if i config SPDK with two partitions, is there any way to limit the transfer speed to a fix speed? I know the kernel has cgroup limit and buffered io limit, if i want to implement this in SPDK, is there a way to archive this? Thanks, Xun _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

2 years, 11 months

1
0
0 / 0

Re: [SPDK] Any idea for SPDK to limit the data transfer speed

by nixun_992＠sina.com

Hi, Paul: Thanks for the advice, currently i am only using SPDK NVMe module, the environment is SPDK + VHOST-USER app, as a server, the client is the qemu. Several qemu clients are using the same vhost server attached nvme disk partitions, of cause, each one use a different partition. I am trying to limit a special qemu process's IO speed. is there a way to do it in vhost-user backend? Thanks, Xun >>>>>>>>>>>>>>>>>>>>>>>> Hi Xun, I’m pretty new to SPDK so I’m sure some others will jump in but first I wanted to make sure you saw Ben’s email on IRC (attached). We’re trying to channel more discussions to IRC as opposed to email but you’re, of course, welcome to continue to use the dist list if that’s more convenient for you. Given that SPDK apps are typically given a full core, is it the case that you are trying to implement a basic form of QoS? Also, are you just using the SPDK NVMe driver or are you using other modules as well? As much info as you can provide on your application and what problem you are trying to solve would be great! Thanks!! Paul From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of nixun_992(a)sina.com Sent: May 7, 2017 12:17 AM To: spdk <spdk(a)lists.01.org> Subject: [SPDK] Any idea for SPDK to limit the data transfer speed Hi, Community: For one nvme device, if i config SPDK with two partitions, is there any way to limit the transfer speed to a fix speed? I know the kernel has cgroup limit and buffered io limit, if i want to implement this in SPDK, is there a way to archive this? Thanks, Xun _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

2 years, 11 months

1
0
0 / 0

Fwd: [spdk/spdk] Master Pull request for DPDK and Posix abstraction changes. (#152)

by John Meneghini

Hi Ben. I agree that commits fbed823 <https://github.com/spdk/spdk/commit/fbed8236facac9833bfd2677b7d76df79cc1c360>, 1bce206 <https://github.com/spdk/spdk/commit/1bce2064db31d44d900f41bab52764dcf65320e8>, edf25e7 <https://github.com/spdk/spdk/commit/edf25e716239ec0051df88207e6f70e474177839> can be dropped. These were the result of the fact that we are using -Wno-pointer-arith and -Wno-pointer-sign when compiling with clang-3.9.1. We’ve changed this. Commit 6ab852f <https://github.com/spdk/spdk/commit/6ab852f9e28d39dc021acbb9a0f0951e6dc933c0> was the result of running lint. We’ve disabled the process that runs lint on this code so you can drop that SHA as well. I will review the new pull requests by the end of today. John Meneghini john.a.meneghini(a)gmail.com > Begin forwarded message: > > From: Ben Walker <notifications(a)github.com> > Subject: Re: [spdk/spdk] Master Pull request for DPDK and Posix abstraction changes. (#152) > Date: May 3, 2017 at 2:37:14 PM EDT > To: spdk/spdk <spdk(a)noreply.github.com> > Cc: John Meneghini <john.a.meneghini(a)gmail.com>, Author <author(a)noreply.github.com> > Reply-To: spdk/spdk <reply+00c54bdda6bf8a3c64ce5c17da0be4a955707c78620de88e92cf000000011521e55a92a169ce0d7675a5(a)reply.github.com> > > I have so far picked off the following: > > e2ecccb <https://github.com/spdk/spdk/commit/e2ecccb3423c01ac96a2073ab11c413f2e70f1bc> as pull request #153 <https://github.com/spdk/spdk/pull/153> > 32f27bb <https://github.com/spdk/spdk/commit/32f27bb80026e73b73abd7e7f0eb83e9af81e68a> as pull request #154 <https://github.com/spdk/spdk/pull/154> > 59072b9 <https://github.com/spdk/spdk/commit/59072b953699628d0447dcf8734473e26807bc26> as pull request #155 <https://github.com/spdk/spdk/pull/155> > Note that I made significant modifications to each of these commits as I picked them off. Please review on the relevant PRs. > > I also recommend we drop the following commits entirely: > fbed823 <https://github.com/spdk/spdk/commit/fbed8236facac9833bfd2677b7d76df79cc1c360>, 1bce206 <https://github.com/spdk/spdk/commit/1bce2064db31d44d900f41bab52764dcf65320e8>, edf25e7 <https://github.com/spdk/spdk/commit/edf25e716239ec0051df88207e6f70e474177839> - These are all adding casts but we're intentionally using the GNU C extension to treat void pointers as uint8_t pointers regarding pointer math. If you are unable to change your compiler settings to allow for this, let me know and we'll address it some other way. > > 6ab852f <https://github.com/spdk/spdk/commit/6ab852f9e28d39dc021acbb9a0f0951e6dc933c0> - As far as we can see, this is a false positive from your linter and not an actual problem. > > That accounts for 7 out of 31 of the commits so far. I'll continue to cherry pick and update here. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub <https://github.com/spdk/spdk/pull/152#issuecomment-298998284>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMVL3c0haArPauICl6fTTal...>. >

2 years, 11 months

2
2
0 / 0

2020

2019

2018

2017

2016

SPDK May 2017