VM crashes while testing SPDK hotplug
by Isaac Otsiabah
I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[[email protected]]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[[email protected]]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[[email protected]]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[[email protected]]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks.
Isaac
5 years, 3 months
FW: VM oops while testing SPDK hotplug
by Isaac Otsiabah
Resending the last email.
From: SPDK [mailto:[email protected]] On Behalf Of Isaac Otsiabah
Sent: Monday, May 15, 2017 5:09 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug
Daniel, the attached hotplug.sh is modified to skip the VM creation and spdk copying sections. I ran it twice, and it oops the second time with the script output redirected to /tmp/hotplug.out4 file. The oops trace is similar to the one sent this morning.
[[email protected]]# ./test/lib/nvme/hotplug.sh 2>&1 |tee /tmp/hotplug.out3
[[email protected]]# ./test/lib/nvme/hotplug.sh 2>&1 |tee /tmp/hotplug.out4
Isaac
From: Isaac Otsiabah
Sent: Monday, May 15, 2017 1:08 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>>
Cc: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:[email protected]>>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com<mailto:[email protected]>>; Edward Yang <eyang(a)us.fujitsu.com<mailto:[email protected]>>
Subject: RE: VM crashes while testing SPDK hotplug
Daniel, i installed a Fedora 24 VM and test it. After running the test twice or more, the VM oops. Unlike the previous failure on Centos, this failure does not reboot but VM oops after two or more test run. My host is a Centos machine. I found the qemu-kvm which comes with the OS installation does not support nvme so I build qemu-system-x86_64 version 2.9.
[[email protected] spdk]# /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 -version
QEMU emulator version 2.9.0
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
One observation on (although this is not the problem because I executed the scripts/setup.sh and the hotplug binary from vm console during appropriate breakpoints because local port 10022 was not responsive), the hotplug.h has the flag "-net user,hostfwd=tcp::10022-:22 \" to redirect gust ssh port 22 to host port 10022. However, qemu-system-x86_64 version 2.9 does not have this option but it has -netdev option but is is different. The qemu-system-86_64 man page on -netdev flag is as follows:
-netdev user,id=str[,ipv4[=on|off]][,net=addr[/mask]][,host=addr]
[,ipv6[=on|off]][,ipv6-net=addr[/int]][,ipv6-host=addr]
[,restrict=on|off][,hostname=host][,dhcpstart=addr]
[,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,tftp=dir]
[,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]]
configure a user mode network backend with ID 'str',
its DHCP server and optional services
It says hostfwd=rule and does not give detail of the rule. I used tcp so I specified it as
-netdev user,id=hotplug,hostfwd=tcp::10022-:22 \
>From the host "netstat -an |egrep -I listen |less" I see local port 10022 is being listened on. I installed sshpass and tested this -netdev flag redirection with a simple sshpass command to the vm but got no response. Therefore, i bypassed executing scripts/setup.sh and the hotplug binary using sshpass command.
So I can test it without executing setup.sh and the hotplug binary through sshpass on port 10022. The main issue is why does it oops after I run it 2 or more times?
Isaac
From: SPDK [mailto:[email protected]] On Behalf Of Verkamp, Daniel
Sent: Tuesday, May 09, 2017 3:33 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug
Hi Isaac,
Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5. I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels).
Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue?
Thanks,
-- Daniel
From: SPDK [mailto:[email protected]] On Behalf Of Isaac Otsiabah
Sent: Tuesday, May 9, 2017 2:11 PM
To: spdk(a)lists.01.org<mailto:[email protected]>
Subject: [SPDK] VM crashes while testing SPDK hotplug
I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[[email protected]]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[[email protected]]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[[email protected]]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[[email protected]]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks.
Isaac
5 years, 3 months
FW: VM oops while testing SPDK hotplug
by Isaac Otsiabah
Daniel, I am not sure you saw the messages I sent this morning so I am resending them.
Isaac
From: Isaac Otsiabah
Sent: Monday, May 15, 2017 1:08 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Cc: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com>; Edward Yang <eyang(a)us.fujitsu.com>
Subject: RE: VM crashes while testing SPDK hotplug
Daniel, i installed a Fedora 24 VM and test it. After running the test twice or more, the VM oops. Unlike the previous failure on Centos, this failure does not reboot but VM oops after two or more test run. My host is a Centos machine. I found the qemu-kvm which comes with the OS installation does not support nvme so I build qemu-system-x86_64 version 2.9.
[[email protected] spdk]# /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 -version
QEMU emulator version 2.9.0
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
One observation on (although this is not the problem because I executed the scripts/setup.sh and the hotplug binary from vm console during appropriate breakpoints because local port 10022 was not responsive), the hotplug.h has the flag "-net user,hostfwd=tcp::10022-:22 \" to redirect gust ssh port 22 to host port 10022. However, qemu-system-x86_64 version 2.9 does not have this option but it has -netdev option but is is different. The qemu-system-86_64 man page on -netdev flag is as follows:
-netdev user,id=str[,ipv4[=on|off]][,net=addr[/mask]][,host=addr]
[,ipv6[=on|off]][,ipv6-net=addr[/int]][,ipv6-host=addr]
[,restrict=on|off][,hostname=host][,dhcpstart=addr]
[,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,tftp=dir]
[,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]]
configure a user mode network backend with ID 'str',
its DHCP server and optional services
It says hostfwd=rule and does not give detail of the rule. I used tcp so I specified it as
-netdev user,id=hotplug,hostfwd=tcp::10022-:22 \
>From the host "netstat -an |egrep -I listen |less" I see local port 10022 is being listened on. I installed sshpass and tested this -netdev flag redirection with a simple sshpass command to the vm but got no response. Therefore, i bypassed executing scripts/setup.sh and the hotplug binary using sshpass command.
So I can test it without executing setup.sh and the hotplug binary through sshpass on port 10022. The main issue is why does it oops after I run it 2 or more times?
Isaac
From: SPDK [mailto:[email protected]] On Behalf Of Verkamp, Daniel
Sent: Tuesday, May 09, 2017 3:33 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug
Hi Isaac,
Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5. I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels).
Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue?
Thanks,
-- Daniel
From: SPDK [mailto:[email protected]] On Behalf Of Isaac Otsiabah
Sent: Tuesday, May 9, 2017 2:11 PM
To: spdk(a)lists.01.org<mailto:[email protected]>
Subject: [SPDK] VM crashes while testing SPDK hotplug
I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[[email protected]]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[[email protected]]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[[email protected]]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[[email protected]]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks.
Isaac
5 years, 3 months
FW: VM oops while testing SPDK hotplug
by Isaac Otsiabah
Forwarding this message I sent this morning.
From: Isaac Otsiabah
Sent: Monday, May 15, 2017 1:43 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Cc: Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com>; Edward Yang <eyang(a)us.fujitsu.com>; Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com>
Subject: RE: VM crashes while testing SPDK hotplug
Daniel, please, do you have an updated version of the hotplug.sh script you can share with us? I created the VM using this exact command on my Centos 7 host
IMAGE=/home/fedora24/fedora24.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
FEDORA_ISO=/tmp/Fedora-Server-dvd-x86_64-24-1.2.iso
/tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-netdev user,id=hotplug,hostfwd=tcp::10022-:22 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid_fedora.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $FEDORA_ISO
After the install is complete, I setup the guest IP address in /etc/sysconfig/network-scripts/ifcfg-ens3 and brings up the interface with ./ifup ens3
>From the VM, I clone spdk and build.
Then I run the group of test in hotplug.sh skipping the VM creation and copying spdk to the VM sections.
I mentioned the -netdev flag in my earlier email
Isaac
From: Isaac Otsiabah
Sent: Monday, May 15, 2017 1:08 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>>
Cc: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:[email protected]>>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com<mailto:[email protected]>>; Edward Yang <eyang(a)us.fujitsu.com<mailto:[email protected]>>
Subject: RE: VM crashes while testing SPDK hotplug
Daniel, i installed a Fedora 24 VM and test it. After running the test twice or more, the VM oops. Unlike the previous failure on Centos, this failure does not reboot but VM oops after two or more test run. My host is a Centos machine. I found the qemu-kvm which comes with the OS installation does not support nvme so I build qemu-system-x86_64 version 2.9.
[[email protected] spdk]# /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 -version
QEMU emulator version 2.9.0
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
One observation on (although this is not the problem because I executed the scripts/setup.sh and the hotplug binary from vm console during appropriate breakpoints because local port 10022 was not responsive), the hotplug.h has the flag "-net user,hostfwd=tcp::10022-:22 \" to redirect gust ssh port 22 to host port 10022. However, qemu-system-x86_64 version 2.9 does not have this option but it has -netdev option but is is different. The qemu-system-86_64 man page on -netdev flag is as follows:
-netdev user,id=str[,ipv4[=on|off]][,net=addr[/mask]][,host=addr]
[,ipv6[=on|off]][,ipv6-net=addr[/int]][,ipv6-host=addr]
[,restrict=on|off][,hostname=host][,dhcpstart=addr]
[,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,tftp=dir]
[,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]]
configure a user mode network backend with ID 'str',
its DHCP server and optional services
It says hostfwd=rule and does not give detail of the rule. I used tcp so I specified it as
-netdev user,id=hotplug,hostfwd=tcp::10022-:22 \
>From the host "netstat -an |egrep -I listen |less" I see local port 10022 is being listened on. I installed sshpass and tested this -netdev flag redirection with a simple sshpass command to the vm but got no response. Therefore, i bypassed executing scripts/setup.sh and the hotplug binary using sshpass command.
So I can test it without executing setup.sh and the hotplug binary through sshpass on port 10022. The main issue is why does it oops after I run it 2 or more times?
Isaac
From: SPDK [mailto:[email protected]] On Behalf Of Verkamp, Daniel
Sent: Tuesday, May 09, 2017 3:33 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:[email protected]>>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug
Hi Isaac,
Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5. I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels).
Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue?
Thanks,
-- Daniel
From: SPDK [mailto:[email protected]] On Behalf Of Isaac Otsiabah
Sent: Tuesday, May 9, 2017 2:11 PM
To: spdk(a)lists.01.org<mailto:[email protected]>
Subject: [SPDK] VM crashes while testing SPDK hotplug
I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[[email protected]]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[[email protected]]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[[email protected]]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[[email protected]]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks.
Isaac
5 years, 3 months
Any idea for SPDK to limit the data transfer speed
by nixun_992＠sina.com
Hi, Community:
For one nvme device, if i config SPDK with two partitions, is there any way to limit the transfer speed to a fix speed? I know the kernel has cgroup limit and buffered io limit, if i want to implement this in SPDK, is there a way to archive this?
Thanks,
Xun
5 years, 3 months
invalid usermode virtual address 0xffff8f600000
by Oza Oza
Hi,
Why do I get invalid user mode virtual address when I enabled DEBUG in
SPDK/DPDK ?
invalid usermode virtual address 0xffff8f600000
invalid usermode virtual address 0xffff0f800000
invalid usermode virtual address 0xffff0f400000
this is armv8.
Regards,
Oza.
5 years, 3 months
回复：Re: Any idea for SPDK to limit the data transfer speed
by nixun_992＠sina.com
Thank you changpeng, Get your way to limit the speed. But that seems to be NVMe only, not a common infrastruct for speed limit.
Thanks,
Xun
---------------
Hi Xun,
Very good question, but currently SPDK does
not have such feature.
One crude implementation: you can modify function
bdev_nvme_poll: spdk_nvme_qpair_process_completions(qpair, 0),
replace 0 with a different number such as
8. This means batch of I/O at a time instead of wrap around.
From:
SPDK [mailto:[email protected]] On Behalf Of
nixun_992(a)sina.com
Sent: Sunday, May 7, 2017 3:17 PM
To: spdk <spdk(a)lists.01.org>
Subject: [SPDK] Any idea for SPDK to limit the data transfer speed
Hi, Community:
For one
nvme device, if i
config SPDK with two partitions, is there any way to limit the transfer speed to a fix speed? I know the kernel has
cgroup limit and buffered io limit, if
i want to implement this in SPDK, is there a way to archive this?
Thanks,
Xun
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
5 years, 3 months
Re: [SPDK] Any idea for SPDK to limit the data transfer speed
by nixun_992＠sina.com
Hi, Paul:
Thanks for the advice, currently i am only using SPDK NVMe module, the environment is SPDK + VHOST-USER app, as a server, the client is the qemu. Several qemu clients are using the same vhost server attached nvme disk partitions, of cause, each one use a different partition. I am trying to limit a special qemu process's IO speed. is there a way to do it in vhost-user backend?
Thanks,
Xun
>>>>>>>>>>>>>>>>>>>>>>>>
Hi Xun,
I’m pretty new to SPDK so I’m sure some others will jump in but first I wanted to make sure you saw Ben’s email on IRC (attached). We’re trying to
channel more discussions to IRC as opposed to email but you’re, of course, welcome to continue to use the dist list if that’s more convenient for you.
Given that SPDK apps are typically given a full core, is it the case that you are trying to implement a basic form of QoS? Also, are you just using
the SPDK NVMe driver or are you using other modules as well? As much info as you can provide on your application and what problem you are trying to solve would be great!
Thanks!!
Paul
From: SPDK [mailto:[email protected]]
On Behalf Of nixun_992(a)sina.com
Sent: May 7, 2017 12:17 AM
To: spdk <spdk(a)lists.01.org>
Subject: [SPDK] Any idea for SPDK to limit the data transfer speed
Hi, Community:
For one nvme device, if i config SPDK with two partitions, is there any way to limit the transfer speed to a fix speed? I know the kernel has cgroup limit and buffered io limit, if i want to implement this in SPDK, is there a way to
archive this?
Thanks,
Xun
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
5 years, 3 months
Fwd: [spdk/spdk] Master Pull request for DPDK and Posix abstraction changes. (#152)
by John Meneghini
Hi Ben.
I agree that commits fbed823 <https://github.com/spdk/spdk/commit/fbed8236facac9833bfd2677b7d76df79cc1c360>, 1bce206 <https://github.com/spdk/spdk/commit/1bce2064db31d44d900f41bab52764dcf65320e8>, edf25e7 <https://github.com/spdk/spdk/commit/edf25e716239ec0051df88207e6f70e474177839> can be dropped. These were the result of the fact that we are using -Wno-pointer-arith and -Wno-pointer-sign when compiling with clang-3.9.1. We’ve changed this. Commit 6ab852f <https://github.com/spdk/spdk/commit/6ab852f9e28d39dc021acbb9a0f0951e6dc933c0> was the result of running lint. We’ve disabled the process that runs lint on this code so you can drop that SHA as well.
I will review the new pull requests by the end of today.
John Meneghini
john.a.meneghini(a)gmail.com
> Begin forwarded message:
>
> From: Ben Walker <notifications(a)github.com>
> Subject: Re: [spdk/spdk] Master Pull request for DPDK and Posix abstraction changes. (#152)
> Date: May 3, 2017 at 2:37:14 PM EDT
> To: spdk/spdk <spdk(a)noreply.github.com>
> Cc: John Meneghini <john.a.meneghini(a)gmail.com>, Author <author(a)noreply.github.com>
> Reply-To: spdk/spdk <reply+00c54bdda6bf8a3c64ce5c17da0be4a955707c78620de88e92cf000000011521e55a92a169ce0d7675a5(a)reply.github.com>
>
> I have so far picked off the following:
>
> e2ecccb <https://github.com/spdk/spdk/commit/e2ecccb3423c01ac96a2073ab11c413f2e70f1bc> as pull request #153 <https://github.com/spdk/spdk/pull/153>
> 32f27bb <https://github.com/spdk/spdk/commit/32f27bb80026e73b73abd7e7f0eb83e9af81e68a> as pull request #154 <https://github.com/spdk/spdk/pull/154>
> 59072b9 <https://github.com/spdk/spdk/commit/59072b953699628d0447dcf8734473e26807bc26> as pull request #155 <https://github.com/spdk/spdk/pull/155>
> Note that I made significant modifications to each of these commits as I picked them off. Please review on the relevant PRs.
>
> I also recommend we drop the following commits entirely:
> fbed823 <https://github.com/spdk/spdk/commit/fbed8236facac9833bfd2677b7d76df79cc1c360>, 1bce206 <https://github.com/spdk/spdk/commit/1bce2064db31d44d900f41bab52764dcf65320e8>, edf25e7 <https://github.com/spdk/spdk/commit/edf25e716239ec0051df88207e6f70e474177839> - These are all adding casts but we're intentionally using the GNU C extension to treat void pointers as uint8_t pointers regarding pointer math. If you are unable to change your compiler settings to allow for this, let me know and we'll address it some other way.
>
> 6ab852f <https://github.com/spdk/spdk/commit/6ab852f9e28d39dc021acbb9a0f0951e6dc933c0> - As far as we can see, this is a false positive from your linter and not an actual problem.
>
> That accounts for 7 out of 31 of the commits so far. I'll continue to cherry pick and update here.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub <https://github.com/spdk/spdk/pull/152#issuecomment-298998284>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMVL3c0haArPauICl6fTTal...>.
>
5 years, 3 months
