Got it. I was tracking the dpdk project, and focused on the branch that Daniel told me to work on. So would the correct process be to push the patch to the spdk/dpdk gerrithub and discuss the commit, or the other way around?

From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E <paul.e.luse@ intel.com> Sent: Tuesday, May 8, 2018 5:14:10 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Sure, I understand that. SPDK maintains a fork of dpdk in github at https://g ithub.com/spdk/dpdk for the purpose of temporarily putting things in that we need that can’t make it upstream yet for whatever reason – this is the submodule in the SPDK repo. I’m not saying that it makes sense for your patch to land there permanently but it does make sense to talk about it. No problem on not being on the call, we’ll get some input from the maintainers and others I’m sure… Thx Paul From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Tuesday, May 8, 2018 7:04 AM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] Building spdk on CentOS6 Hey Paul, The problem is that the patch is in dpdk and not in spdk, and this is the reason I offered to support it. The patch (bellow) is only a few lines long modifying the deprecated attribute definition, a modification to the config, and an extra compilation flag. All of the above are required in order to build dpdk 18.01 on CentOS6, there are no problems building spdk on CentOS6. I understood that you are already applying minor patches to dpdk, so question is, whether this patch can be added? Unfortunately due to some personal issues, I am unable to participate in today's meeting. Thanks, Shahar From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E <paul.e.luse@ intel.com> Sent: Tuesday, May 8, 2018 4:25:29 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Hi Shahar, Couple of quick things: * thanks for putting all this time and effort into trying to make this work! * in the community model, patches aren’t accepted based on one individual’s ability/desire to support them so appreciate the offer but in general any patch that is accepted becomes the responsibly of the community (mostly the maintainers) to support it long term. Otherwise we’d have what I like to call “the flea market model” where there’s a bunch of separate individuals supporting their own things with very little cohesiveness across everything J So, wrt next steps, if the patch isn’t a tremendous amount of effort I would suggest you go ahead and submit it. That’s the best way to get everyone on the same page wrt exactly what we’re talking about. If it is some significant effort then, as Pawel states, you can call into a community meeting (see email I sent out last night) or you can explain more details on this list. Thanks again!! Paul From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Wodkowski, PawelX Sent: Tuesday, May 8, 2018 5:57 AM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] Building spdk on CentOS6 I think someone will answer you on this mailing list but there is community meeting today too. You can raise this if you wish. Pawel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Tuesday, May 8, 2018 2:23 PM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] Building spdk on CentOS6 Hi, I attempted to push the gcc support patch to dpdk, and got rejected since they do not wish to support CentOS6. Would it be possible to manually add the patch to the supported dpdk version (currently 18.01). I would be happy to support CentOS6 issues. Thanks, Shahar From: Shahar Salzman Sent: Tuesday, April 24, 2018 3:40:51 PM To: Storage Performance Development Kit Cc: Ilan Steinberg; Ido Benda; Yael Shavit Subject: Re: Building spdk on CentOS6 OK, got everything to work. For DPDK, I had to to do the following: add a patch to make the new __rte_experimental attribute to work remove a bunch of modules via config which had some gnarly compilation errors, and are not needed for spdk use -fno-strict-aliasing in the EXTRA_CFLAGS to avoid a ton of strict aliasing errors. I tried spdk 18.01, and ran into some issues with the shared library build. As I saw that Daniel had already dealt with them, I tried cherry picking, but eventually used the master branch. I got these type of warnings: /usr/bin/ld: warning: librte_mbuf.so.3.1, needed by /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) /usr/bin/ld: warning: librte_ethdev.so.8.1, needed by /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) I added the following patch bellow to make them go away. I'll approach the dpdk community on the mailing list in order to get the __rte_experimental stuff to work. My final configuration is: CenrOS6.4 gcc 4.4.7-17 dpdk v18.02 spdk master What are the next steps? Shahar P.S. Here is the patch to remove the warnings above. commit d4ef744de1cc4928972b7042d1c90aff12db7867 Author: shahar salzman <shahar.salzman(a)kaminario.com&gt; Date: Tue Apr 24 14:31:52 2018 +0300 lib/env_dpdk: add required libraries diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk index 450043c..b46bfed 100644 --- a/lib/env_dpdk/env.mk +++ b/lib/env_dpdk/env.mk @@ -68,6 +68,18 @@ ifneq ($(wildcard $(DPDK_ABS_DIR)/lib/librte_malloc.*),) DPDK_LIB_LIST += rte_malloc endif +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_mbuf.*)) +DPDK_LIB_LIST += rte_mbuf +endif + +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_net.*)) +DPDK_LIB_LIST += rte_net +endif + +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_ethdev.*)) +DPDK_LIB_LIST += rte_ethdev +endif + # librte_pci and librte_bus_pci were added in DPDK 17.11. Link these libraries conditionally # based on their existence to maintain backward compatibility. ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_pci.*)) From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Shahar Salzman <shahar.sal zman(a)kaminario.com&gt; Sent: Tuesday, April 24, 2018 9:57 AM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Great, and thanks! I have been running into issues compiling dpdk 17.11 on CentOS6 (there seems to be a gcc issue that does not exist in other versions), I'll focus my efforts on dpdk 18.02. From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Verkamp, Daniel <daniel.ve rkamp(a)intel.com&gt; Sent: Monday, April 23, 2018 9:40:36 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Hi Shahar, I’ve pushed a patch for the SPDK master branch that should allow it to compile with DPDK 16.07: https://review.gerrithub.io/#/c/spdk/spdk/+/408743/ Gerrit Code Review review.gerrithub.io Keep in touch. Copyright © 2017 | GerritForge Ltd. info(a)gerritforge.com www.gerritforge.com<http://www.gerritforge.com> However, DPDK 16.07 is no longer supported upstream; I highly recommend that you update to a newer version. SPDK is currently tested with DPDK 18.02. The VFIO issue you mention has also been fixed on SPDK master, and it will be part of the SPDK v18.04 release. Thanks, -- Daniel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Sunday, April 22, 2018 6:41 AM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: [SPDK] Building spdk on CentOS6 Hi, Finally got to looking at support of spdk build on CentOS6, things look good, except for one issue. spdk is latest 18.01.x version, dpdk is 16.07 (+3 dpdk patches to allow compilation) and some minor patches (mainly some memory configuration stuff), kernel is a patched 4.9.6. build succeeded except for the usage of the dpdk function pci_vfio_is_enabled. I had to apply the patch bellow, removing the usage of this function and then compilation completed without any issues. It seems that I am missing some sort of dpdk configuration as I see that the function is built, but not packaged into the generated archive. I went back to square one and ran the instructions in http://www.spdk.io/doc/g etting_started.html, but I see no mention of dpdk there. Actually the ./configure requires it. My next step is to use a more recent dpdk, but shouldn't this work with my version? Am I missing some dpdk configuration? BTW, as we are not using vhost, on our 17.07 version we simply use CONFIG_VHOST=n in order to skip this, but I would be happier if we used a better solution. Shahar P.S. Here is the patch to remove use of this function: diff --git a/lib/env_dpdk/vtophys.c b/lib/env_dpdk/vtophys.c index 92aa256..f38929f 100644 --- a/lib/env_dpdk/vtophys.c +++ b/lib/env_dpdk/vtophys.c @@ -53,8 +53,10 @@ #define SPDK_VFIO_ENABLED 1 #include <linux/vfio.h> +#if 0 /* Internal DPDK function forward declaration */ int pci_vfio_is_enabled(void); +#endif struct spdk_vfio_dma_map { struct vfio_iommu_type1_dma_map map; @@ -341,9 +343,11 @@ spdk_vtophys_iommu_init(void) DIR *dir; struct dirent *d; +#if 0 if (!pci_vfio_is_enabled()) { return; } +#endif dir = opendir("/proc/self/fd"); if (!dir) { _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

Got it. I was tracking the dpdk project, and focused on the branch that Daniel told me to work on. So would the correct process be to push the patch to the spdk/dpdk gerrithub and discuss the commit, or the other way around?

From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Luse, Paul E <paul.e.luse@

Sent: Tuesday, May 8, 2018 5:14:10 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Sure, I understand that. SPDK maintains a fork of dpdk in github at https://g ithub.com/spdk/dpdk for the purpose of temporarily putting things in that we need that can’t make it upstream yet for whatever reason – this is the submodule in the SPDK repo. I’m not saying that it makes sense for your patch to land there permanently but it does make sense to talk about it. No problem on not being on the call, we’ll get some input from the maintainers and others I’m sure… Thx Paul From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Tuesday, May 8, 2018 7:04 AM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Building spdk on CentOS6 Hey Paul, The problem is that the patch is in dpdk and not in spdk, and this is the reason I offered to support it. The patch (bellow) is only a few lines long modifying the deprecated attribute definition, a modification to the config, and an extra compilation flag. All of the above are required in order to build dpdk 18.01 on CentOS6, there are no problems building spdk on CentOS6. I understood that you are already applying minor patches to dpdk, so question is, whether this patch can be added? Unfortunately due to some personal issues, I am unable to participate in today's meeting. Thanks, Shahar From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Luse, Paul E <paul.e.luse@

Sent: Tuesday, May 8, 2018 4:25:29 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Hi Shahar, Couple of quick things: * thanks for putting all this time and effort into trying to make this work! * in the community model, patches aren’t accepted based on one individual’s ability/desire to support them so appreciate the offer but in general any patch that is accepted becomes the responsibly of the community (mostly the maintainers) to support it long term. Otherwise we’d have what I like to call “the flea market model” where there’s a bunch of separate individuals supporting their own things with very little cohesiveness across everything J So, wrt next steps, if the patch isn’t a tremendous amount of effort I would suggest you go ahead and submit it. That’s the best way to get everyone on the same page wrt exactly what we’re talking about. If it is some significant effort then, as Pawel states, you can call into a community meeting (see email I sent out last night) or you can explain more details on this list. Thanks again!! Paul From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Wodkowski, PawelX Sent: Tuesday, May 8, 2018 5:57 AM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Building spdk on CentOS6 I think someone will answer you on this mailing list but there is community meeting today too. You can raise this if you wish. Pawel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Tuesday, May 8, 2018 2:23 PM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Building spdk on CentOS6 Hi, I attempted to push the gcc support patch to dpdk, and got rejected since they do not wish to support CentOS6. Would it be possible to manually add the patch to the supported dpdk version (currently 18.01). I would be happy to support CentOS6 issues. Thanks, Shahar From: Shahar Salzman Sent: Tuesday, April 24, 2018 3:40:51 PM To: Storage Performance Development Kit Cc: Ilan Steinberg; Ido Benda; Yael Shavit Subject: Re: Building spdk on CentOS6 OK, got everything to work. For DPDK, I had to to do the following: add a patch to make the new __rte_experimental attribute to work remove a bunch of modules via config which had some gnarly compilation errors, and are not needed for spdk use -fno-strict-aliasing in the EXTRA_CFLAGS to avoid a ton of strict aliasing errors. I tried spdk 18.01, and ran into some issues with the shared library build. As I saw that Daniel had already dealt with them, I tried cherry picking, but eventually used the master branch. I got these type of warnings: /usr/bin/ld: warning: librte_mbuf.so.3.1, needed by /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) /usr/bin/ld: warning: librte_ethdev.so.8.1, needed by /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) I added the following patch bellow to make them go away. I'll approach the dpdk community on the mailing list in order to get the __rte_experimental stuff to work. My final configuration is: CenrOS6.4 gcc 4.4.7-17 dpdk v18.02 spdk master What are the next steps? Shahar P.S. Here is the patch to remove the warnings above. commit d4ef744de1cc4928972b7042d1c90aff12db7867 Author: shahar salzman <shahar.salzman@kaminario.com<mailto:shahar.salzman@kaminario.com>> Date: Tue Apr 24 14:31:52 2018 +0300 lib/env_dpdk: add required libraries diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk index 450043c..b46bfed 100644 --- a/lib/env_dpdk/env.mk +++ b/lib/env_dpdk/env.mk @@ -68,6 +68,18 @@ ifneq ($(wildcard $(DPDK_ABS_DIR)/lib/librte_malloc.*),) DPDK_LIB_LIST += rte_malloc endif +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_mbuf.*)) +DPDK_LIB_LIST += rte_mbuf +endif + +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_net.*)) +DPDK_LIB_LIST += rte_net +endif + +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_ethdev.*)) +DPDK_LIB_LIST += rte_ethdev +endif + # librte_pci and librte_bus_pci were added in DPDK 17.11. Link these libraries conditionally # based on their existence to maintain backward compatibility. ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_pci.*)) From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Shahar Salzman <shahar.sal zman@kaminario.com<mailto:zman@kaminario.com>> Sent: Tuesday, April 24, 2018 9:57 AM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Great, and thanks! I have been running into issues compiling dpdk 17.11 on CentOS6 (there seems to be a gcc issue that does not exist in other versions), I'll focus my efforts on dpdk 18.02. From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Verkamp, Daniel <daniel.ve rkamp@intel.com<mailto:rkamp@intel.com>> Sent: Monday, April 23, 2018 9:40:36 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Hi Shahar, I’ve pushed a patch for the SPDK master branch that should allow it to compile with DPDK 16.07: https://review.gerrithub.io/#/c/spdk/spdk/+/408743/ Gerrit Code Review review.gerrithub.io Keep in touch. Copyright © 2017 | GerritForge Ltd. info@gerritforge.com<mailto:info@gerritforge.com> www.gerritforge.com<http://www.gerritforge.com> However, DPDK 16.07 is no longer supported upstream; I highly recommend that you update to a newer version. SPDK is currently tested with DPDK 18.02. The VFIO issue you mention has also been fixed on SPDK master, and it will be part of the SPDK v18.04 release. Thanks, -- Daniel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Sunday, April 22, 2018 6:41 AM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: [SPDK] Building spdk on CentOS6 Hi, Finally got to looking at support of spdk build on CentOS6, things look good, except for one issue. spdk is latest 18.01.x version, dpdk is 16.07 (+3 dpdk patches to allow compilation) and some minor patches (mainly some memory configuration stuff), kernel is a patched 4.9.6. build succeeded except for the usage of the dpdk function pci_vfio_is_enabled. I had to apply the patch bellow, removing the usage of this function and then compilation completed without any issues. It seems that I am missing some sort of dpdk configuration as I see that the function is built, but not packaged into the generated archive. I went back to square one and ran the instructions in http://www.spdk.io/doc/g etting_started.html, but I see no mention of dpdk there. Actually the ./configure requires it. My next step is to use a more recent dpdk, but shouldn't this work with my version? Am I missing some dpdk configuration? BTW, as we are not using vhost, on our 17.07 version we simply use CONFIG_VHOST=n in order to skip this, but I would be happier if we used a better solution. Shahar P.S. Here is the patch to remove use of this function: diff --git a/lib/env_dpdk/vtophys.c b/lib/env_dpdk/vtophys.c index 92aa256..f38929f 100644 --- a/lib/env_dpdk/vtophys.c +++ b/lib/env_dpdk/vtophys.c @@ -53,8 +53,10 @@ #define SPDK_VFIO_ENABLED 1 #include <linux/vfio.h> +#if 0 /* Internal DPDK function forward declaration */ int pci_vfio_is_enabled(void); +#endif struct spdk_vfio_dma_map { struct vfio_iommu_type1_dma_map map; @@ -341,9 +343,11 @@ spdk_vtophys_iommu_init(void) DIR *dir; struct dirent *d; +#if 0 if (!pci_vfio_is_enabled()) { return; } +#endif dir = opendir("/proc/self/fd"); if (!dir) { _______________________________________________ SPDK mailing list SPDK@lists.01.org<mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk

Got it. I was tracking the dpdk project, and focused on the branch that Daniel told me to work on. So would the correct process be to push the patch to the spdk/dpdk gerrithub and discuss the commit, or the other way around?

From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Luse, Paul E <paul.e.luse@

Sent: Tuesday, May 8, 2018 5:14:10 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Sure, I understand that. SPDK maintains a fork of dpdk in github at https://g ithub.com/spdk/dpdk for the purpose of temporarily putting things in that we need that can't make it upstream yet for whatever reason - this is the submodule in the SPDK repo. I'm not saying that it makes sense for your patch to land there permanently but it does make sense to talk about it. No problem on not being on the call, we'll get some input from the maintainers and others I'm sure... Thx Paul From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Tuesday, May 8, 2018 7:04 AM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Building spdk on CentOS6 Hey Paul, The problem is that the patch is in dpdk and not in spdk, and this is the reason I offered to support it. The patch (bellow) is only a few lines long modifying the deprecated attribute definition, a modification to the config, and an extra compilation flag. All of the above are required in order to build dpdk 18.01 on CentOS6, there are no problems building spdk on CentOS6. I understood that you are already applying minor patches to dpdk, so question is, whether this patch can be added? Unfortunately due to some personal issues, I am unable to participate in today's meeting. Thanks, Shahar From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Luse, Paul E <paul.e.luse@

Sent: Tuesday, May 8, 2018 4:25:29 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Hi Shahar, Couple of quick things: * thanks for putting all this time and effort into trying to make this work! * in the community model, patches aren't accepted based on one individual's ability/desire to support them so appreciate the offer but in general any patch that is accepted becomes the responsibly of the community (mostly the maintainers) to support it long term. Otherwise we'd have what I like to call "the flea market model" where there's a bunch of separate individuals supporting their own things with very little cohesiveness across everything J So, wrt next steps, if the patch isn't a tremendous amount of effort I would suggest you go ahead and submit it. That's the best way to get everyone on the same page wrt exactly what we're talking about. If it is some significant effort then, as Pawel states, you can call into a community meeting (see email I sent out last night) or you can explain more details on this list. Thanks again!! Paul From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Wodkowski, PawelX Sent: Tuesday, May 8, 2018 5:57 AM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Building spdk on CentOS6 I think someone will answer you on this mailing list but there is community meeting today too. You can raise this if you wish. Pawel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Tuesday, May 8, 2018 2:23 PM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Building spdk on CentOS6 Hi, I attempted to push the gcc support patch to dpdk, and got rejected since they do not wish to support CentOS6. Would it be possible to manually add the patch to the supported dpdk version (currently 18.01). I would be happy to support CentOS6 issues. Thanks, Shahar From: Shahar Salzman Sent: Tuesday, April 24, 2018 3:40:51 PM To: Storage Performance Development Kit Cc: Ilan Steinberg; Ido Benda; Yael Shavit Subject: Re: Building spdk on CentOS6 OK, got everything to work. For DPDK, I had to to do the following: add a patch to make the new __rte_experimental attribute to work remove a bunch of modules via config which had some gnarly compilation errors, and are not needed for spdk use -fno-strict-aliasing in the EXTRA_CFLAGS to avoid a ton of strict aliasing errors. I tried spdk 18.01, and ran into some issues with the shared library build. As I saw that Daniel had already dealt with them, I tried cherry picking, but eventually used the master branch. I got these type of warnings: /usr/bin/ld: warning: librte_mbuf.so.3.1, needed by /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) /usr/bin/ld: warning: librte_ethdev.so.8.1, needed by /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) I added the following patch bellow to make them go away. I'll approach the dpdk community on the mailing list in order to get the __rte_experimental stuff to work. My final configuration is: CenrOS6.4 gcc 4.4.7-17 dpdk v18.02 spdk master What are the next steps? Shahar P.S. Here is the patch to remove the warnings above. commit d4ef744de1cc4928972b7042d1c90aff12db7867 Author: shahar salzman <shahar.salzman@kaminario.com<mailto:shahar.salzman@kaminario.com>> Date: Tue Apr 24 14:31:52 2018 +0300 lib/env_dpdk: add required libraries diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk index 450043c..b46bfed 100644 --- a/lib/env_dpdk/env.mk +++ b/lib/env_dpdk/env.mk @@ -68,6 +68,18 @@ ifneq ($(wildcard $(DPDK_ABS_DIR)/lib/librte_malloc.*),) DPDK_LIB_LIST += rte_malloc endif +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_mbuf.*)) +DPDK_LIB_LIST += rte_mbuf +endif + +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_net.*)) +DPDK_LIB_LIST += rte_net +endif + +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_ethdev.*)) +DPDK_LIB_LIST += rte_ethdev +endif + # librte_pci and librte_bus_pci were added in DPDK 17.11. Link these libraries conditionally # based on their existence to maintain backward compatibility. ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_pci.*)) From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Shahar Salzman <shahar.sal zman@kaminario.com<mailto:zman@kaminario.com>> Sent: Tuesday, April 24, 2018 9:57 AM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Great, and thanks! I have been running into issues compiling dpdk 17.11 on CentOS6 (there seems to be a gcc issue that does not exist in other versions), I'll focus my efforts on dpdk 18.02. From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Verkamp, Daniel <daniel.ve rkamp@intel.com<mailto:rkamp@intel.com>> Sent: Monday, April 23, 2018 9:40:36 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Hi Shahar, I've pushed a patch for the SPDK master branch that should allow it to compile with DPDK 16.07: https://review.gerrithub.io/#/c/spdk/spdk/+/408743/ Gerrit Code Review review.gerrithub.io Keep in touch. Copyright (c) 2017 | GerritForge Ltd. info@gerritforge.com<mailto:info@gerritforge.com> www.gerritforge.com<http://www.gerritforge.com> However, DPDK 16.07 is no longer supported upstream; I highly recommend that you update to a newer version. SPDK is currently tested with DPDK 18.02. The VFIO issue you mention has also been fixed on SPDK master, and it will be part of the SPDK v18.04 release. Thanks, -- Daniel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Sunday, April 22, 2018 6:41 AM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: [SPDK] Building spdk on CentOS6 Hi, Finally got to looking at support of spdk build on CentOS6, things look good, except for one issue. spdk is latest 18.01.x version, dpdk is 16.07 (+3 dpdk patches to allow compilation) and some minor patches (mainly some memory configuration stuff), kernel is a patched 4.9.6. build succeeded except for the usage of the dpdk function pci_vfio_is_enabled. I had to apply the patch bellow, removing the usage of this function and then compilation completed without any issues. It seems that I am missing some sort of dpdk configuration as I see that the function is built, but not packaged into the generated archive. I went back to square one and ran the instructions in http://www.spdk.io/doc/g etting_started.html, but I see no mention of dpdk there. Actually the ./configure requires it. My next step is to use a more recent dpdk, but shouldn't this work with my version? Am I missing some dpdk configuration? BTW, as we are not using vhost, on our 17.07 version we simply use CONFIG_VHOST=n in order to skip this, but I would be happier if we used a better solution. Shahar P.S. Here is the patch to remove use of this function: diff --git a/lib/env_dpdk/vtophys.c b/lib/env_dpdk/vtophys.c index 92aa256..f38929f 100644 --- a/lib/env_dpdk/vtophys.c +++ b/lib/env_dpdk/vtophys.c @@ -53,8 +53,10 @@ #define SPDK_VFIO_ENABLED 1 #include <linux/vfio.h> +#if 0 /* Internal DPDK function forward declaration */ int pci_vfio_is_enabled(void); +#endif struct spdk_vfio_dma_map { struct vfio_iommu_type1_dma_map map; @@ -341,9 +343,11 @@ spdk_vtophys_iommu_init(void) DIR *dir; struct dirent *d; +#if 0 if (!pci_vfio_is_enabled()) { return; } +#endif dir = opendir("/proc/self/fd"); if (!dir) { _______________________________________________ SPDK mailing list SPDK@lists.01.org<mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk

Got it. I was tracking the dpdk project, and focused on the branch that Daniel told me to work on. So would the correct process be to push the patch to the spdk/dpdk gerrithub and discuss the commit, or the other way around?

From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Luse, Paul E <paul.e.luse@

Sent: Tuesday, May 8, 2018 5:14:10 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Sure, I understand that. SPDK maintains a fork of dpdk in github at https://g ithub.com/spdk/dpdk for the purpose of temporarily putting things in that we need that can't make it upstream yet for whatever reason - this is the submodule in the SPDK repo. I'm not saying that it makes sense for your patch to land there permanently but it does make sense to talk about it. No problem on not being on the call, we'll get some input from the maintainers and others I'm sure... Thx Paul From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Tuesday, May 8, 2018 7:04 AM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Building spdk on CentOS6 Hey Paul, The problem is that the patch is in dpdk and not in spdk, and this is the reason I offered to support it. The patch (bellow) is only a few lines long modifying the deprecated attribute definition, a modification to the config, and an extra compilation flag. All of the above are required in order to build dpdk 18.01 on CentOS6, there are no problems building spdk on CentOS6. I understood that you are already applying minor patches to dpdk, so question is, whether this patch can be added? Unfortunately due to some personal issues, I am unable to participate in today's meeting. Thanks, Shahar From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Luse, Paul E <paul.e.luse@

Sent: Tuesday, May 8, 2018 4:25:29 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Hi Shahar, Couple of quick things: * thanks for putting all this time and effort into trying to make this work! * in the community model, patches aren't accepted based on one individual's ability/desire to support them so appreciate the offer but in general any patch that is accepted becomes the responsibly of the community (mostly the maintainers) to support it long term. Otherwise we'd have what I like to call "the flea market model" where there's a bunch of separate individuals supporting their own things with very little cohesiveness across everything J So, wrt next steps, if the patch isn't a tremendous amount of effort I would suggest you go ahead and submit it. That's the best way to get everyone on the same page wrt exactly what we're talking about. If it is some significant effort then, as Pawel states, you can call into a community meeting (see email I sent out last night) or you can explain more details on this list. Thanks again!! Paul From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Wodkowski, PawelX Sent: Tuesday, May 8, 2018 5:57 AM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Building spdk on CentOS6 I think someone will answer you on this mailing list but there is community meeting today too. You can raise this if you wish. Pawel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Tuesday, May 8, 2018 2:23 PM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Building spdk on CentOS6 Hi, I attempted to push the gcc support patch to dpdk, and got rejected since they do not wish to support CentOS6. Would it be possible to manually add the patch to the supported dpdk version (currently 18.01). I would be happy to support CentOS6 issues. Thanks, Shahar From: Shahar Salzman Sent: Tuesday, April 24, 2018 3:40:51 PM To: Storage Performance Development Kit Cc: Ilan Steinberg; Ido Benda; Yael Shavit Subject: Re: Building spdk on CentOS6 OK, got everything to work. For DPDK, I had to to do the following: add a patch to make the new __rte_experimental attribute to work remove a bunch of modules via config which had some gnarly compilation errors, and are not needed for spdk use -fno-strict-aliasing in the EXTRA_CFLAGS to avoid a ton of strict aliasing errors. I tried spdk 18.01, and ran into some issues with the shared library build. As I saw that Daniel had already dealt with them, I tried cherry picking, but eventually used the master branch. I got these type of warnings: /usr/bin/ld: warning: librte_mbuf.so.3.1, needed by /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) /usr/bin/ld: warning: librte_ethdev.so.8.1, needed by /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) I added the following patch bellow to make them go away. I'll approach the dpdk community on the mailing list in order to get the __rte_experimental stuff to work. My final configuration is: CenrOS6.4 gcc 4.4.7-17 dpdk v18.02 spdk master What are the next steps? Shahar P.S. Here is the patch to remove the warnings above. commit d4ef744de1cc4928972b7042d1c90aff12db7867 Author: shahar salzman <shahar.salzman@kaminario.com<mailto:shahar.salzman@kaminario.com>> Date: Tue Apr 24 14:31:52 2018 +0300 lib/env_dpdk: add required libraries diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk index 450043c..b46bfed 100644 --- a/lib/env_dpdk/env.mk +++ b/lib/env_dpdk/env.mk @@ -68,6 +68,18 @@ ifneq ($(wildcard $(DPDK_ABS_DIR)/lib/librte_malloc.*),) DPDK_LIB_LIST += rte_malloc endif +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_mbuf.*)) +DPDK_LIB_LIST += rte_mbuf +endif + +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_net.*)) +DPDK_LIB_LIST += rte_net +endif + +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_ethdev.*)) +DPDK_LIB_LIST += rte_ethdev +endif + # librte_pci and librte_bus_pci were added in DPDK 17.11. Link these libraries conditionally # based on their existence to maintain backward compatibility. ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_pci.*)) From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Shahar Salzman <shahar.sal zman@kaminario.com<mailto:zman@kaminario.com>> Sent: Tuesday, April 24, 2018 9:57 AM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Great, and thanks! I have been running into issues compiling dpdk 17.11 on CentOS6 (there seems to be a gcc issue that does not exist in other versions), I'll focus my efforts on dpdk 18.02. From: SPDK <spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>> on behalf of Verkamp, Daniel <daniel.ve rkamp@intel.com<mailto:rkamp@intel.com>> Sent: Monday, April 23, 2018 9:40:36 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Building spdk on CentOS6 Hi Shahar, I've pushed a patch for the SPDK master branch that should allow it to compile with DPDK 16.07: https://review.gerrithub.io/#/c/spdk/spdk/+/408743/ Gerrit Code Review review.gerrithub.io Keep in touch. Copyright (c) 2017 | GerritForge Ltd. info@gerritforge.com<mailto:info@gerritforge.com> www.gerritforge.com<http://www.gerritforge.com> However, DPDK 16.07 is no longer supported upstream; I highly recommend that you update to a newer version. SPDK is currently tested with DPDK 18.02. The VFIO issue you mention has also been fixed on SPDK master, and it will be part of the SPDK v18.04 release. Thanks, -- Daniel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Shahar Salzman Sent: Sunday, April 22, 2018 6:41 AM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: [SPDK] Building spdk on CentOS6 Hi, Finally got to looking at support of spdk build on CentOS6, things look good, except for one issue. spdk is latest 18.01.x version, dpdk is 16.07 (+3 dpdk patches to allow compilation) and some minor patches (mainly some memory configuration stuff), kernel is a patched 4.9.6. build succeeded except for the usage of the dpdk function pci_vfio_is_enabled. I had to apply the patch bellow, removing the usage of this function and then compilation completed without any issues. It seems that I am missing some sort of dpdk configuration as I see that the function is built, but not packaged into the generated archive. I went back to square one and ran the instructions in http://www.spdk.io/doc/g etting_started.html, but I see no mention of dpdk there. Actually the ./configure requires it. My next step is to use a more recent dpdk, but shouldn't this work with my version? Am I missing some dpdk configuration? BTW, as we are not using vhost, on our 17.07 version we simply use CONFIG_VHOST=n in order to skip this, but I would be happier if we used a better solution. Shahar P.S. Here is the patch to remove use of this function: diff --git a/lib/env_dpdk/vtophys.c b/lib/env_dpdk/vtophys.c index 92aa256..f38929f 100644 --- a/lib/env_dpdk/vtophys.c +++ b/lib/env_dpdk/vtophys.c @@ -53,8 +53,10 @@ #define SPDK_VFIO_ENABLED 1 #include <linux/vfio.h> +#if 0 /* Internal DPDK function forward declaration */ int pci_vfio_is_enabled(void); +#endif struct spdk_vfio_dma_map { struct vfio_iommu_type1_dma_map map; @@ -341,9 +343,11 @@ spdk_vtophys_iommu_init(void) DIR *dir; struct dirent *d; +#if 0 if (!pci_vfio_is_enabled()) { return; } +#endif dir = opendir("/proc/self/fd"); if (!dir) { _______________________________________________ SPDK mailing list SPDK@lists.01.org<mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk

Hi Daniel, I ran the dpdk build with T=spdk-linuxapp-gcc and build works fine. Isn't config/common_spdk only for configuration options (no reference for this type of definition) ? I would have thought that the compilation flag should go in spdk/dpdkbuild/Makefile, or in one of the mk files? Shahar ------------------------------ *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Verkamp, Daniel < daniel.verkamp(a)intel.com&gt; *Sent:* Thursday, May 10, 2018 8:31:21 PM *To:* Storage Performance Development Kit *Subject:* Re: [SPDK] Building spdk on CentOS6 Hi Shahar, All of the config options you listed below are already disabled in the config that we use to build the DPDK subodule (see config/common_spdk in our DPDK branch). Can you confirm that this works for you? I think we can add -fno-strict-aliasing to the DPDK config/common_spdk file as well if this is necessary to make it work on RHEL 6 (I am surprised DPDK doesn’t already compile with this enabled). If you could upload a second patch for this, that would be great. Thanks, -- Daniel *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *Shahar Salzman *Sent:* Thursday, May 10, 2018 6:39 AM *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; *Subject:* Re: [SPDK] Building spdk on CentOS6 Hi Daniel, I updated the patch as you suggested. How do you want to approach the gcc flag and the modules that spdk does not need? If you are building dpdk from an external makefile (as we are doing), all of the above can be added to the makefile with CONFIG...=n and EXTRA_CFLAGS="-fno-strict-aliasing", or I can send you another dpdk patch fixing this within dpdk, so there is no change in the builder, and maybe there is another option. WDYT? Shahar ------------------------------ *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < paul.e.luse(a)intel.com&gt; *Sent:* Wednesday, May 9, 2018 8:19:33 PM *To:* Storage Performance Development Kit *Subject:* Re: [SPDK] Building spdk on CentOS6 That’s great, thanks Daniel! *From:* SPDK [mailto:spdk-bounces@lists.01.org <spdk-bounces(a)lists.01.org>] *On Behalf Of *Verkamp, Daniel *Sent:* Wednesday, May 9, 2018 9:43 AM *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; *Subject:* Re: [SPDK] Building spdk on CentOS6 I agree with Paul’s review; it looks like it shouldn’t be a big problem to carry a patch for this in our DPDK submodule, at least while CentOS 6 is still officially supported. (I posted a few review comments on the patch, but the general approach looks fine.) The Intel build pool had a CentOS 6 test machine in the pool until recently, and we should be able to resurrect it to provide test coverage again. Thanks, -- Daniel *From:* SPDK [mailto:spdk-bounces@lists.01.org <spdk-bounces(a)lists.01.org>] *On Behalf Of *Shahar Salzman *Sent:* Wednesday, May 9, 2018 6:12 AM *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; *Subject:* Re: [SPDK] Building spdk on CentOS6 Hi Paul, Thanks for the review. We are now working on CentOS7 support, but it is not complete yet, so we will be on CentOS6 about 6-12 months. Shahar ------------------------------ *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < paul.e.luse(a)intel.com&gt; *Sent:* Wednesday, May 9, 2018 3:51:23 PM *To:* Storage Performance Development Kit *Subject:* Re: [SPDK] Building spdk on CentOS6 Thanks for doing that! I put my 2 cents on the review, one more data point for the maintainers might help – help long do you believe your team will be on CentOS6? Don’t need a specific date but are we talking months, years, indefinitely? I think the other implication of accepting this in the community would be to spin up a CentOS6 VM for some basic testing for as long as we hold that patch in our DPDK fork which is not a huge thing but want to make sure that’s clear as well. If nothing else we’ll know if something else breaks it sooner than later and will need to decide, again, at that time if we pull it or if it’s a high enough priority for anyone to dig in and find a potential solution like you did here. Assuming, of course, that this one is accepted J Thanks, Paul *From:* SPDK [mailto:spdk-bounces@lists.01.org <spdk-bounces(a)lists.01.org>] *On Behalf Of *Shahar Salzman *Sent:* Wednesday, May 9, 2018 2:01 AM *To:* spdk(a)lists.01.org *Subject:* Re: [SPDK] Building spdk on CentOS6 Hi Ben, Pushed the fix to gerrithub - https://review.gerrithub.io/#/c/spdk/dpdk/+/410559/ We are currently using a "frozen" dpdk tree based version 16.07, but I am pushing to using the spdk fork and performing regular updates so if things break, I will attempt to fix them, and post on the list. The above fixes a blunt gcc API issue, but there are more subtle inner module issues which I have not investigated, simply removed the modules all together as they are not needed by spdk. I can set up a howto for users who may need CentOS6 support or we can modify the config (as these modules are not really needed for spdk). For the sake of clarity here are the modules "configed out" (as I mentioned they are not used by spdk): +CONFIG_RTE_LIBRTE_PMD_BOND=n +CONFIG_RTE_LIBRTE_CRYPTODEV=n +CONFIG_RTE_LIBRTE_SECURITY=n +CONFIG_RTE_LIBRTE_PMD_SW_EVENTDEV=n +CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF=n +CONFIG_RTE_LIBRTE_LPM=n +CONFIG_RTE_LIBRTE_FLOW_CLASSIFY=n +CONFIG_RTE_LIBRTE_TABLE=n +CONFIG_RTE_LIBRTE_PIPELINE=n +CONFIG_RTE_TEST_PMD=n +CONFIG_RTE_LIBRTE_PMD_TAP=n +CONFIG_RTE_LIBRTE_AVP_PMD=n Regarding the -fn-strict-aliasing issue, I'll see if I can do the same trick I did for the gcc alias in the Makefile, and push the fix for review. Shahar ------------------------------ *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Walker, Benjamin < benjamin.walker(a)intel.com&gt; *Sent:* Tuesday, May 8, 2018 7:49:44 PM *To:* spdk(a)lists.01.org *Subject:* Re: [SPDK] Building spdk on CentOS6 On Tue, 2018-05-08 at 14:34 +0000, Shahar Salzman wrote: > Got it. > > I was tracking the dpdk project, and focused on the branch that Daniel told me > to work on. > So would the correct process be to push the patch to the spdk/dpdk gerrithub > and discuss the commit, or the other way around? Yes - go ahead and push to spdk/dpdk on GerritHub so we can all review it. The SPDK project has never written out a policy on which versions of DPDK are supported for any given SPDK release. The code is always tested using the latest released DPDK, and then attempts to support older versions of DPDK are done in an ad hoc manner. Similarly, the project never explicitly states which operating systems are supported. Instead, we indicate which operating systems the code is regularly tested on (Fedora N and N-1, Ubuntu latest, Ubuntu LTS, CentOS 7, FreeBSD 11). I see that CentOS 6 is no longer receiving full updates and is in maintenance only mode. It will stop receiving even maintenance updates in 2020. I think that is indicative of the level of support SPDK should provide here. Given the above, if this patch can be applied to the DPDK fork and doesn't have any additional impact, I'm not opposed to applying it. That wouldn't, in my mind, mean that SPDK is committed to supporting CentOS 6 indefinitely. But certainly that support could continue while it is relatively easy and convenient. > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < paul.e.luse@ <paul.e.luse@%0b>> intel.com> > Sent: Tuesday, May 8, 2018 5:14:10 PM > To: Storage Performance Development Kit > Subject: Re: [SPDK] Building spdk on CentOS6 > > Sure, I understand that. SPDK maintains a fork of dpdk in github at https://g > ithub.com/spdk/dpdk for the purpose of temporarily putting things in that we > need that can’t make it upstream yet for whatever reason – this is the > submodule in the SPDK repo. I’m not saying that it makes sense for your patch > to land there permanently but it does make sense to talk about it. > > No problem on not being on the call, we’ll get some input from the maintainers > and others I’m sure… > > Thx > Paul > > From: SPDK [mailto:spdk-bounces@lists.01.org <spdk-bounces(a)lists.01.org>] On Behalf Of Shahar Salzman > Sent: Tuesday, May 8, 2018 7:04 AM > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > Subject: Re: [SPDK] Building spdk on CentOS6 > > Hey Paul, > > The problem is that the patch is in dpdk and not in spdk, and this is the > reason I offered to support it. > The patch (bellow) is only a few lines long modifying the deprecated attribute > definition, a modification to the config, and an extra compilation flag. > All of the above are required in order to build dpdk 18.01 on CentOS6, there > are no problems building spdk on CentOS6. > > I understood that you are already applying minor patches to dpdk, so question > is, whether this patch can be added? > > Unfortunately due to some personal issues, I am unable to participate in > today's meeting. > > Thanks, > Shahar > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < paul.e.luse@ <paul.e.luse@%0b>> intel.com> > Sent: Tuesday, May 8, 2018 4:25:29 PM > To: Storage Performance Development Kit > Subject: Re: [SPDK] Building spdk on CentOS6 > > Hi Shahar, > > Couple of quick things: > > * thanks for putting all this time and effort into trying to make this work! > * in the community model, patches aren’t accepted based on one individual’s > ability/desire to support them so appreciate the offer but in general any > patch that is accepted becomes the responsibly of the community (mostly the > maintainers) to support it long term. Otherwise we’d have what I like to call > “the flea market model” where there’s a bunch of separate individuals > supporting their own things with very little cohesiveness across everything J > > So, wrt next steps, if the patch isn’t a tremendous amount of effort I would > suggest you go ahead and submit it. That’s the best way to get everyone on the > same page wrt exactly what we’re talking about. If it is some significant > effort then, as Pawel states, you can call into a community meeting (see email > I sent out last night) or you can explain more details on this list. > > Thanks again!! > Paul > > From: SPDK [mailto:spdk-bounces@lists.01.org <spdk-bounces(a)lists.01.org>] On Behalf Of Wodkowski, PawelX > Sent: Tuesday, May 8, 2018 5:57 AM > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > Subject: Re: [SPDK] Building spdk on CentOS6 > > I think someone will answer you on this mailing list but there is community > meeting today too. You can raise this if you wish. > > Pawel > > From: SPDK [mailto:spdk-bounces@lists.01.org <spdk-bounces(a)lists.01.org>] On Behalf Of Shahar Salzman > Sent: Tuesday, May 8, 2018 2:23 PM > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > Subject: Re: [SPDK] Building spdk on CentOS6 > > Hi, > > I attempted to push the gcc support patch to dpdk, and got rejected since they > do not wish to support CentOS6. > Would it be possible to manually add the patch to the supported dpdk version > (currently 18.01). > I would be happy to support CentOS6 issues. > > Thanks, > Shahar > > From: Shahar Salzman > Sent: Tuesday, April 24, 2018 3:40:51 PM > To: Storage Performance Development Kit > Cc: Ilan Steinberg; Ido Benda; Yael Shavit > Subject: Re: Building spdk on CentOS6 > > OK, got everything to work. > > For DPDK, I had to to do the following: > add a patch to make the new __rte_experimental attribute to work > remove a bunch of modules via config which had some gnarly compilation errors, > and are not needed for spdk > use -fno-strict-aliasing in the EXTRA_CFLAGS to avoid a ton of strict aliasing > errors. > > I tried spdk 18.01, and ran into some issues with the shared library build. As > I saw that Daniel had already dealt with them, I tried cherry picking, but > eventually used the master branch. > I got these type of warnings: > > /usr/bin/ld: warning: librte_mbuf.so.3.1, needed by > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- > gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) > /usr/bin/ld: warning: librte_ethdev.so.8.1, needed by > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- > gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) > > I added the following patch bellow to make them go away. > > I'll approach the dpdk community on the mailing list in order to get the > __rte_experimental stuff to work. > My final configuration is: > CenrOS6.4 > gcc 4.4.7-17 > dpdk v18.02 > spdk master > > What are the next steps? > > Shahar > > P.S. Here is the patch to remove the warnings above. > commit d4ef744de1cc4928972b7042d1c90aff12db7867 > Author: shahar salzman <shahar.salzman(a)kaminario.com&gt; > Date: Tue Apr 24 14:31:52 2018 +0300 > > lib/env_dpdk: add required libraries > > diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk > index 450043c..b46bfed 100644 > --- a/lib/env_dpdk/env.mk > +++ b/lib/env_dpdk/env.mk > @@ -68,6 +68,18 @@ ifneq ($(wildcard $(DPDK_ABS_DIR)/lib/librte_malloc.*),) > DPDK_LIB_LIST += rte_malloc > endif > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_mbuf.*)) > +DPDK_LIB_LIST += rte_mbuf > +endif > + > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_net.*)) > +DPDK_LIB_LIST += rte_net > +endif > + > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_ethdev.*)) > +DPDK_LIB_LIST += rte_ethdev > +endif > + > # librte_pci and librte_bus_pci were added in DPDK 17.11. Link these > libraries conditionally > # based on their existence to maintain backward compatibility. > ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_pci.*)) > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Shahar Salzman <shahar.sal > zman(a)kaminario.com&gt; > Sent: Tuesday, April 24, 2018 9:57 AM > To: Storage Performance Development Kit > Subject: Re: [SPDK] Building spdk on CentOS6 > > Great, and thanks! > > I have been running into issues compiling dpdk 17.11 on CentOS6 (there seems > to be a gcc issue that does not exist in other versions), I'll focus my > efforts on dpdk 18.02. > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Verkamp, Daniel < daniel.ve > rkamp(a)intel.com&gt; > Sent: Monday, April 23, 2018 9:40:36 PM > To: Storage Performance Development Kit > Subject: Re: [SPDK] Building spdk on CentOS6 > > Hi Shahar, > > I’ve pushed a patch for the SPDK master branch that should allow it to compile > with DPDK 16.07: https://review.gerrithub.io/#/c/spdk/spdk/+/408743/ > > Gerrit Code Review > > review.gerrithub.io > > Keep in touch. Copyright © 2017 | GerritForge Ltd. info(a)gerritforge.com > www.gerritforge.com > > > However, DPDK 16.07 is no longer supported upstream; I highly recommend that > you update to a newer version. SPDK is currently tested with DPDK 18.02. > > The VFIO issue you mention has also been fixed on SPDK master, and it will be > part of the SPDK v18.04 release. > > Thanks, > -- Daniel > > From: SPDK [mailto:spdk-bounces@lists.01.org <spdk-bounces(a)lists.01.org>] On Behalf Of Shahar Salzman > Sent: Sunday, April 22, 2018 6:41 AM > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > Subject: [SPDK] Building spdk on CentOS6 > > Hi, > > Finally got to looking at support of spdk build on CentOS6, things look good, > except for one issue. > spdk is latest 18.01.x version, dpdk is 16.07 (+3 dpdk patches to allow > compilation) and some minor patches (mainly some memory configuration stuff), > kernel is a patched 4.9.6. > > build succeeded except for the usage of the dpdk function pci_vfio_is_enabled. > I had to apply the patch bellow, removing the usage of this function and then > compilation completed without any issues. > > It seems that I am missing some sort of dpdk configuration as I see that the > function is built, but not packaged into the generated archive. > > I went back to square one and ran the instructions in http://www.spdk.io/doc/g > etting_started.html, but I see no mention of dpdk there. Actually the > ./configure requires it. > > My next step is to use a more recent dpdk, but shouldn't this work with my > version? Am I missing some dpdk configuration? > BTW, as we are not using vhost, on our 17.07 version we simply use > CONFIG_VHOST=n in order to skip this, but I would be happier if we used a > better solution. > > Shahar > > P.S. Here is the patch to remove use of this function: > diff --git a/lib/env_dpdk/vtophys.c b/lib/env_dpdk/vtophys.c > index 92aa256..f38929f 100644 > --- a/lib/env_dpdk/vtophys.c > +++ b/lib/env_dpdk/vtophys.c > @@ -53,8 +53,10 @@ > #define SPDK_VFIO_ENABLED 1 > #include <linux/vfio.h> > > +#if 0 > /* Internal DPDK function forward declaration */ > int pci_vfio_is_enabled(void); > +#endif > > struct spdk_vfio_dma_map { > struct vfio_iommu_type1_dma_map map; > @@ -341,9 +343,11 @@ spdk_vtophys_iommu_init(void) > DIR *dir; > struct dirent *d; > > +#if 0 > if (!pci_vfio_is_enabled()) { > return; > } > +#endif > > dir = opendir("/proc/self/fd"); > if (!dir) { > > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

On Fri, 2018-08-24 at 15:00 -0700, Gyan Prakash wrote: > Hi all, > > I am trying to build spdk-18.07 with dpdk-18.05 on Centos 7.4 system with > kernel 4.13.9. > > make DPDK_DIR=./dpdk-18.05/x86_64-native-linuxapp-gcc CONFIG_RDMA=y > > and seeing following compilation failure. Can someone please tell me what I > am missing there? > > CC lib/vhost/vhost.o > In file included from vhost_internal.h:39:0, > from vhost.c:43: > rte_vhost/rte_vhost.h:47:24: fatal error: rte_config.h: No such file or > directory > #include <rte_config.h> > ^ > compilation terminated. > make[2]: *** [vhost.o] Error 1 > make[1]: *** [vhost] Error 2 > make: *** [lib] Error 2 This means that the build couldn't find your DPDK installation (I think). As a quick sanity test, can you try this sequence of commands: git clone https://github.com/spdk/spdk cd spdk git submodule update --init ./scripts/pkgdep.sh ./configure --with-rdma make ./test/unit/unittest.sh This sequence is taken from this page: http://www.spdk.io/doc/getting_started.html If you need to do something more advanced, like use a fork of DPDK, that's also possible but we should confirm that the simple case is working first. > > Thanks, > Gyan > > On Sun, May 13, 2018 at 1:45 AM Shahar Salzman < shahar.salzman(a)kaminario.com&gt; > wrote: > > > Hi Daniel, > > > > > > I ran the dpdk build with T=spdk-linuxapp-gcc and build works fine. > > > > Isn't config/common_spdk only for configuration options (no reference for > > this type of definition) ? I would have thought that the compilation flag > > should go in spdk/dpdkbuild/Makefile, or in one of the mk files? > > > > > > Shahar > > > > > > > > ------------------------------ > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Verkamp, Daniel < > > daniel.verkamp(a)intel.com&gt; > > *Sent:* Thursday, May 10, 2018 8:31:21 PM > > *To:* Storage Performance Development Kit > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > Hi Shahar, > > > > > > > > All of the config options you listed below are already disabled in the > > config that we use to build the DPDK subodule (see config/common_spdk in > > our DPDK branch). Can you confirm that this works for you? > > > > > > > > I think we can add -fno-strict-aliasing to the DPDK config/common_spdk > > file as well if this is necessary to make it work on RHEL 6 (I am surprised > > DPDK doesn’t already compile with this enabled). If you could upload a > > second patch for this, that would be great. > > > > > > > > Thanks, > > > > -- Daniel > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *Shahar > > Salzman > > *Sent:* Thursday, May 10, 2018 6:39 AM > > *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > Hi Daniel, > > > > > > > > I updated the patch as you suggested. > > > > How do you want to approach the gcc flag and the modules that spdk does > > not need? > > > > If you are building dpdk from an external makefile (as we are doing), all > > of the above can be added to the makefile with CONFIG...=n and > > EXTRA_CFLAGS="-fno-strict-aliasing", or I can send you another dpdk patch > > fixing this within dpdk, so there is no change in the builder, and maybe > > there is another option. > > > > > > > > WDYT? > > > > > > > > Shahar > > > > > > ------------------------------ > > > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > > paul.e.luse(a)intel.com&gt; > > *Sent:* Wednesday, May 9, 2018 8:19:33 PM > > *To:* Storage Performance Development Kit > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > That’s great, thanks Daniel! > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > > *On Behalf Of *Verkamp, Daniel > > *Sent:* Wednesday, May 9, 2018 9:43 AM > > *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > I agree with Paul’s review; it looks like it shouldn’t be a big problem to > > carry a patch for this in our DPDK submodule, at least while CentOS 6 is > > still officially supported. (I posted a few review comments on the patch, > > but the general approach looks fine.) > > > > > > > > The Intel build pool had a CentOS 6 test machine in the pool until > > recently, and we should be able to resurrect it to provide test coverage > > again. > > > > > > > > Thanks, > > > > -- Daniel > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > > *On Behalf Of *Shahar Salzman > > *Sent:* Wednesday, May 9, 2018 6:12 AM > > *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > Hi Paul, > > > > > > > > Thanks for the review. > > > > We are now working on CentOS7 support, but it is not complete yet, so we > > will be on CentOS6 about 6-12 months. > > > > > > > > Shahar > > ------------------------------ > > > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > > paul.e.luse(a)intel.com&gt; > > *Sent:* Wednesday, May 9, 2018 3:51:23 PM > > *To:* Storage Performance Development Kit > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > Thanks for doing that! I put my 2 cents on the review, one more data point > > for the maintainers might help – help long do you believe your team will be > > on CentOS6? Don’t need a specific date but are we talking months, years, > > indefinitely? > > > > > > > > I think the other implication of accepting this in the community would be > > to spin up a CentOS6 VM for some basic testing for as long as we hold that > > patch in our DPDK fork which is not a huge thing but want to make sure > > that’s clear as well. If nothing else we’ll know if something else breaks > > it sooner than later and will need to decide, again, at that time if we > > pull it or if it’s a high enough priority for anyone to dig in and find a > > potential solution like you did here. Assuming, of course, that this one > > is accepted J > > > > > > > > Thanks, > > > > Paul > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > > *On Behalf Of *Shahar Salzman > > *Sent:* Wednesday, May 9, 2018 2:01 AM > > *To:* spdk(a)lists.01.org > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > Hi Ben, > > > > > > > > Pushed the fix to gerrithub - > > https://review.gerrithub.io/#/c/spdk/dpdk/+/410559/ > > > > > > > > We are currently using a "frozen" dpdk tree based version 16.07, but I am > > pushing to using the spdk fork and performing regular updates so if things > > break, I will attempt to fix them, and post on the list. > > > > > > > > The above fixes a blunt gcc API issue, but there are more subtle inner > > module issues which I have not investigated, simply removed the modules all > > together as they are not needed by spdk. > > > > > > > > I can set up a howto for users who may need CentOS6 support or we can > > modify the config (as these modules are not really needed for spdk). For > > the sake of clarity here are the modules "configed out" (as I mentioned > > they are not used by spdk): > > > > +CONFIG_RTE_LIBRTE_PMD_BOND=n > > > > +CONFIG_RTE_LIBRTE_CRYPTODEV=n > > > > +CONFIG_RTE_LIBRTE_SECURITY=n > > > > +CONFIG_RTE_LIBRTE_PMD_SW_EVENTDEV=n > > > > +CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF=n > > > > +CONFIG_RTE_LIBRTE_LPM=n > > > > +CONFIG_RTE_LIBRTE_FLOW_CLASSIFY=n > > > > +CONFIG_RTE_LIBRTE_TABLE=n > > > > +CONFIG_RTE_LIBRTE_PIPELINE=n > > > > +CONFIG_RTE_TEST_PMD=n > > > > +CONFIG_RTE_LIBRTE_PMD_TAP=n > > > > +CONFIG_RTE_LIBRTE_AVP_PMD=n > > > > > > > > Regarding the -fn-strict-aliasing issue, I'll see if I can do the same > > trick I did for the gcc alias in the Makefile, and push the fix for review. > > > > > > > > Shahar > > > > > > ------------------------------ > > > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Walker, Benjamin < > > benjamin.walker(a)intel.com&gt; > > *Sent:* Tuesday, May 8, 2018 7:49:44 PM > > *To:* spdk(a)lists.01.org > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > On Tue, 2018-05-08 at 14:34 +0000, Shahar Salzman wrote: > > > Got it. > > > > > > I was tracking the dpdk project, and focused on the branch that Daniel > > > > told me > > > to work on. > > > So would the correct process be to push the patch to the spdk/dpdk > > > > gerrithub > > > and discuss the commit, or the other way around? > > > > Yes - go ahead and push to spdk/dpdk on GerritHub so we can all review it. > > The > > SPDK project has never written out a policy on which versions of DPDK are > > supported for any given SPDK release. The code is always tested using the > > latest > > released DPDK, and then attempts to support older versions of DPDK are > > done in > > an ad hoc manner. Similarly, the project never explicitly states which > > operating > > systems are supported. Instead, we indicate which operating systems the > > code is > > regularly tested on (Fedora N and N-1, Ubuntu latest, Ubuntu LTS, CentOS 7, > > FreeBSD 11). > > > > I see that CentOS 6 is no longer receiving full updates and is in > > maintenance > > only mode. It will stop receiving even maintenance updates in 2020. I > > think that > > is indicative of the level of support SPDK should provide here. > > > > Given the above, if this patch can be applied to the DPDK fork and doesn't > > have > > any additional impact, I'm not opposed to applying it. That wouldn't, in my > > mind, mean that SPDK is committed to supporting CentOS 6 indefinitely. But > > certainly that support could continue while it is relatively easy and > > convenient. > > > > > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > > > > paul.e.luse@ > > <paul.e.luse@%0b>> intel.com> > > > Sent: Tuesday, May 8, 2018 5:14:10 PM > > > To: Storage Performance Development Kit > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > Sure, I understand that. SPDK maintains a fork of dpdk in github at > > > > https://g > > > ithub.com/spdk/dpdk for the purpose of temporarily putting things in > > > > that we > > > need that can’t make it upstream yet for whatever reason – this is the > > > submodule in the SPDK repo. I’m not saying that it makes sense for your > > > > patch > > > to land there permanently but it does make sense to talk about it. > > > > > > No problem on not being on the call, we’ll get some input from the > > > > maintainers > > > and others I’m sure… > > > > > > Thx > > > Paul > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > > > > On Behalf Of Shahar Salzman > > > Sent: Tuesday, May 8, 2018 7:04 AM > > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > Hey Paul, > > > > > > The problem is that the patch is in dpdk and not in spdk, and this is the > > > reason I offered to support it. > > > The patch (bellow) is only a few lines long modifying the deprecated > > > > attribute > > > definition, a modification to the config, and an extra compilation flag. > > > All of the above are required in order to build dpdk 18.01 on CentOS6, > > > > there > > > are no problems building spdk on CentOS6. > > > > > > I understood that you are already applying minor patches to dpdk, so > > > > question > > > is, whether this patch can be added? > > > > > > Unfortunately due to some personal issues, I am unable to participate in > > > today's meeting. > > > > > > Thanks, > > > Shahar > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > > > > paul.e.luse@ > > <paul.e.luse@%0b>> intel.com> > > > Sent: Tuesday, May 8, 2018 4:25:29 PM > > > To: Storage Performance Development Kit > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > Hi Shahar, > > > > > > Couple of quick things: > > > > > > * thanks for putting all this time and effort into trying to make this > > > > work! > > > * in the community model, patches aren’t accepted based on one > > > > individual’s > > > ability/desire to support them so appreciate the offer but in general any > > > patch that is accepted becomes the responsibly of the community (mostly > > > > the > > > maintainers) to support it long term. Otherwise we’d have what I like > > > > to call > > > “the flea market model” where there’s a bunch of separate individuals > > > supporting their own things with very little cohesiveness across > > > > everything J > > > > > > So, wrt next steps, if the patch isn’t a tremendous amount of effort I > > > > would > > > suggest you go ahead and submit it. That’s the best way to get everyone > > > > on the > > > same page wrt exactly what we’re talking about. If it is some > > > > significant > > > effort then, as Pawel states, you can call into a community meeting (see > > > > email > > > I sent out last night) or you can explain more details on this list. > > > > > > Thanks again!! > > > Paul > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > > > > On Behalf Of Wodkowski, PawelX > > > Sent: Tuesday, May 8, 2018 5:57 AM > > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > I think someone will answer you on this mailing list but there is > > > > community > > > meeting today too. You can raise this if you wish. > > > > > > Pawel > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > > > > On Behalf Of Shahar Salzman > > > Sent: Tuesday, May 8, 2018 2:23 PM > > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > Hi, > > > > > > I attempted to push the gcc support patch to dpdk, and got rejected > > > > since they > > > do not wish to support CentOS6. > > > Would it be possible to manually add the patch to the supported dpdk > > > > version > > > (currently 18.01). > > > I would be happy to support CentOS6 issues. > > > > > > Thanks, > > > Shahar > > > > > > From: Shahar Salzman > > > Sent: Tuesday, April 24, 2018 3:40:51 PM > > > To: Storage Performance Development Kit > > > Cc: Ilan Steinberg; Ido Benda; Yael Shavit > > > Subject: Re: Building spdk on CentOS6 > > > > > > OK, got everything to work. > > > > > > For DPDK, I had to to do the following: > > > add a patch to make the new __rte_experimental attribute to work > > > remove a bunch of modules via config which had some gnarly compilation > > > > errors, > > > and are not needed for spdk > > > use -fno-strict-aliasing in the EXTRA_CFLAGS to avoid a ton of strict > > > > aliasing > > > errors. > > > > > > I tried spdk 18.01, and ran into some issues with the shared library > > > > build. As > > > I saw that Daniel had already dealt with them, I tried cherry picking, > > > > but > > > eventually used the master branch. > > > I got these type of warnings: > > > > > > /usr/bin/ld: warning: librte_mbuf.so.3.1, needed by > > > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- > > > gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) > > > /usr/bin/ld: warning: librte_ethdev.so.8.1, needed by > > > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- > > > gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) > > > > > > I added the following patch bellow to make them go away. > > > > > > I'll approach the dpdk community on the mailing list in order to get the > > > __rte_experimental stuff to work. > > > My final configuration is: > > > CenrOS6.4 > > > gcc 4.4.7-17 > > > dpdk v18.02 > > > spdk master > > > > > > What are the next steps? > > > > > > Shahar > > > > > > P.S. Here is the patch to remove the warnings above. > > > commit d4ef744de1cc4928972b7042d1c90aff12db7867 > > > Author: shahar salzman <shahar.salzman(a)kaminario.com&gt; > > > Date: Tue Apr 24 14:31:52 2018 +0300 > > > > > > lib/env_dpdk: add required libraries > > > > > > diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk > > > index 450043c..b46bfed 100644 > > > --- a/lib/env_dpdk/env.mk > > > +++ b/lib/env_dpdk/env.mk > > > @@ -68,6 +68,18 @@ ifneq ($(wildcard > > > > $(DPDK_ABS_DIR)/lib/librte_malloc.*),) > > > DPDK_LIB_LIST += rte_malloc > > > endif > > > > > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_mbuf.*)) > > > +DPDK_LIB_LIST += rte_mbuf > > > +endif > > > + > > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_net.*)) > > > +DPDK_LIB_LIST += rte_net > > > +endif > > > + > > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_ethdev.*)) > > > +DPDK_LIB_LIST += rte_ethdev > > > +endif > > > + > > > # librte_pci and librte_bus_pci were added in DPDK 17.11. Link these > > > libraries conditionally > > > # based on their existence to maintain backward compatibility. > > > ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_pci.*)) > > > > > > > > > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Shahar Salzman > > > > <shahar.sal > > > zman(a)kaminario.com&gt; > > > Sent: Tuesday, April 24, 2018 9:57 AM > > > To: Storage Performance Development Kit > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > Great, and thanks! > > > > > > I have been running into issues compiling dpdk 17.11 on CentOS6 (there > > > > seems > > > to be a gcc issue that does not exist in other versions), I'll focus my > > > efforts on dpdk 18.02. > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Verkamp, Daniel < > > > > daniel.ve > > > rkamp(a)intel.com&gt; > > > Sent: Monday, April 23, 2018 9:40:36 PM > > > To: Storage Performance Development Kit > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > Hi Shahar, > > > > > > I’ve pushed a patch for the SPDK master branch that should allow it to > > > > compile > > > with DPDK 16.07: https://review.gerrithub.io/#/c/spdk/spdk/+/408743/ > > > > > > Gerrit Code Review > > > > > > review.gerrithub.io > > > > > > Keep in touch. Copyright © 2017 | GerritForge Ltd. info(a)gerritforge.com > > > www.gerritforge.com > > > > > > > > > However, DPDK 16.07 is no longer supported upstream; I highly recommend > > > > that > > > you update to a newer version. SPDK is currently tested with DPDK 18.02. > > > > > > The VFIO issue you mention has also been fixed on SPDK master, and it > > > > will be > > > part of the SPDK v18.04 release. > > > > > > Thanks, > > > -- Daniel > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > > > > On Behalf Of Shahar Salzman > > > Sent: Sunday, April 22, 2018 6:41 AM > > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > > Subject: [SPDK] Building spdk on CentOS6 > > > > > > Hi, > > > > > > Finally got to looking at support of spdk build on CentOS6, things look > > > > good, > > > except for one issue. > > > spdk is latest 18.01.x version, dpdk is 16.07 (+3 dpdk patches to allow > > > compilation) and some minor patches (mainly some memory configuration > > > > stuff), > > > kernel is a patched 4.9.6. > > > > > > build succeeded except for the usage of the dpdk function > > > > pci_vfio_is_enabled. > > > I had to apply the patch bellow, removing the usage of this function and > > > > then > > > compilation completed without any issues. > > > > > > It seems that I am missing some sort of dpdk configuration as I see that > > > > the > > > function is built, but not packaged into the generated archive. > > > > > > I went back to square one and ran the instructions in > > > > http://www.spdk.io/doc/g > > > etting_started.html, but I see no mention of dpdk there. Actually the > > > ./configure requires it. > > > > > > My next step is to use a more recent dpdk, but shouldn't this work with > > > > my > > > version? Am I missing some dpdk configuration? > > > BTW, as we are not using vhost, on our 17.07 version we simply use > > > CONFIG_VHOST=n in order to skip this, but I would be happier if we used a > > > better solution. > > > > > > Shahar > > > > > > P.S. Here is the patch to remove use of this function: > > > diff --git a/lib/env_dpdk/vtophys.c b/lib/env_dpdk/vtophys.c > > > index 92aa256..f38929f 100644 > > > --- a/lib/env_dpdk/vtophys.c > > > +++ b/lib/env_dpdk/vtophys.c > > > @@ -53,8 +53,10 @@ > > > #define SPDK_VFIO_ENABLED 1 > > > #include <linux/vfio.h> > > > > > > +#if 0 > > > /* Internal DPDK function forward declaration */ > > > int pci_vfio_is_enabled(void); > > > +#endif > > > > > > struct spdk_vfio_dma_map { > > > struct vfio_iommu_type1_dma_map map; > > > @@ -341,9 +343,11 @@ spdk_vtophys_iommu_init(void) > > > DIR *dir; > > > struct dirent *d; > > > > > > +#if 0 > > > if (!pci_vfio_is_enabled()) { > > > return; > > > } > > > +#endif > > > > > > dir = opendir("/proc/self/fd"); > > > if (!dir) { > > > > > > > > > _______________________________________________ > > > SPDK mailing list > > > SPDK(a)lists.01.org > > > https://lists.01.org/mailman/listinfo/spdk > > > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > > > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

Hi Gyan, Thanks for the update. Those unit test error messages are expected. The unit tests are specifically testing error cases which generate error messages like you see here. The run summary at the bottom is the key output and shows 0 test failures. Thanks, -Jim On 8/24/18, 4:38 PM, "SPDK on behalf of Gyan Prakash" < spdk-bounces(a)lists.01.org on behalf of gyapra2016(a)gmail.com&gt; wrote: Benjamin, Tests passed with your suggested steps, but there are some vHost errors, not sure if it is false positive. Please see below: lapsed time = 0.000 seconds ++ uname -s + '[' Linux = Linux ']' + /root/spdk-aug24-2018/spdk/test/unit/lib/vhost/vhost.c/vhost_ut CUnit - A unit testing framework for C - Version 2.1-2 http://cunit.sourceforge.net/ Suite: vhost_suite Test: desc_to_iov .../root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: 454:spdk_vhost_vring_desc_to_iov: *ERROR*: SPDK_VHOST_IOVS_MAX(129) reached passed * Test: create_controller .../root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: 661:spdk_vhost_dev_register: *ERROR*: Can't register controller with no name/root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: 619:spdk_vhost_parse_core_mask: *ERROR*: no cpu is selected among reactor mask(=1)/root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: 673:spdk_vhost_dev_register: *ERROR*: cpumask 0x2 is invalid (app mask is 0x1)/root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: 686:spdk_vhost_dev_register: *ERROR*: Resulting socket path for controller xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: 679:spdk_vhost_dev_register: *ERROR*: vhost controller vdev_name_0 already exists.* passed Test: dev_find_by_vid ...passed Test: remove_controller .../root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: 772:spdk_vhost_dev_unregister:* *ERROR**: Controller vdev_name_0 has still valid connection. passed Run Summary: Type Total Ran Passed Failed Inactive suites 1 1 n/a 0 0 tests 4 4 4 0 0 asserts 53 53 53 0 n/a Elapsed time = 0.000 seconds + '[' no = yes ']' + set +x ===================== *All unit tests passed* ===================== WARN: lcov not installed or SPDK built without coverage! WARN: neither valgrind nor ASAN is enabled! Thanks for the help, Gyan On Fri, Aug 24, 2018 at 3:23 PM Walker, Benjamin < benjamin.walker(a)intel.com&gt; wrote: > On Fri, 2018-08-24 at 15:00 -0700, Gyan Prakash wrote: > > Hi all, > > > > I am trying to build spdk-18.07 with dpdk-18.05 on Centos 7.4 system with > > kernel 4.13.9. > > > > make DPDK_DIR=./dpdk-18.05/x86_64-native-linuxapp-gcc CONFIG_RDMA=y > > > > and seeing following compilation failure. Can someone please tell me > what I > > am missing there? > > > > CC lib/vhost/vhost.o > > In file included from vhost_internal.h:39:0, > > from vhost.c:43: > > rte_vhost/rte_vhost.h:47:24: fatal error: rte_config.h: No such file or > > directory > > #include <rte_config.h> > > ^ > > compilation terminated. > > make[2]: *** [vhost.o] Error 1 > > make[1]: *** [vhost] Error 2 > > make: *** [lib] Error 2 > > > This means that the build couldn't find your DPDK installation (I think). > As a > quick sanity test, can you try this sequence of commands: > > git clone https://github.com/spdk/spdk > cd spdk > git submodule update --init > ./scripts/pkgdep.sh > ./configure --with-rdma > make > ./test/unit/unittest.sh > > This sequence is taken from this page: > http://www.spdk.io/doc/getting_started.html > > If you need to do something more advanced, like use a fork of DPDK, that's > also > possible but we should confirm that the simple case is working first. > > > > > > Thanks, > > Gyan > > > > On Sun, May 13, 2018 at 1:45 AM Shahar Salzman < > shahar.salzman(a)kaminario.com&gt; > > wrote: > > > > > Hi Daniel, > > > > > > > > > I ran the dpdk build with T=spdk-linuxapp-gcc and build works fine. > > > > > > Isn't config/common_spdk only for configuration options (no reference > for > > > this type of definition) ? I would have thought that the compilation > flag > > > should go in spdk/dpdkbuild/Makefile, or in one of the mk files? > > > > > > > > > Shahar > > > > > > > > > > > > ------------------------------ > > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Verkamp, Daniel > < > > > daniel.verkamp(a)intel.com&gt; > > > *Sent:* Thursday, May 10, 2018 8:31:21 PM > > > *To:* Storage Performance Development Kit > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > Hi Shahar, > > > > > > > > > > > > All of the config options you listed below are already disabled in the > > > config that we use to build the DPDK subodule (see config/common_spdk > in > > > our DPDK branch). Can you confirm that this works for you? > > > > > > > > > > > > I think we can add -fno-strict-aliasing to the DPDK config/common_spdk > > > file as well if this is necessary to make it work on RHEL 6 (I am > surprised > > > DPDK doesn’t already compile with this enabled). If you could upload a > > > second patch for this, that would be great. > > > > > > > > > > > > Thanks, > > > > > > -- Daniel > > > > > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *Shahar > > > Salzman > > > *Sent:* Thursday, May 10, 2018 6:39 AM > > > *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > Hi Daniel, > > > > > > > > > > > > I updated the patch as you suggested. > > > > > > How do you want to approach the gcc flag and the modules that spdk does > > > not need? > > > > > > If you are building dpdk from an external makefile (as we are doing), > all > > > of the above can be added to the makefile with CONFIG...=n and > > > EXTRA_CFLAGS="-fno-strict-aliasing", or I can send you another dpdk > patch > > > fixing this within dpdk, so there is no change in the builder, and > maybe > > > there is another option. > > > > > > > > > > > > WDYT? > > > > > > > > > > > > Shahar > > > > > > > > > ------------------------------ > > > > > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > > > paul.e.luse(a)intel.com&gt; > > > *Sent:* Wednesday, May 9, 2018 8:19:33 PM > > > *To:* Storage Performance Development Kit > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > That’s great, thanks Daniel! > > > > > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < > spdk-bounces(a)lists.01.org>] > > > *On Behalf Of *Verkamp, Daniel > > > *Sent:* Wednesday, May 9, 2018 9:43 AM > > > *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > I agree with Paul’s review; it looks like it shouldn’t be a big > problem to > > > carry a patch for this in our DPDK submodule, at least while CentOS 6 > is > > > still officially supported. (I posted a few review comments on the > patch, > > > but the general approach looks fine.) > > > > > > > > > > > > The Intel build pool had a CentOS 6 test machine in the pool until > > > recently, and we should be able to resurrect it to provide test > coverage > > > again. > > > > > > > > > > > > Thanks, > > > > > > -- Daniel > > > > > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < > spdk-bounces(a)lists.01.org>] > > > *On Behalf Of *Shahar Salzman > > > *Sent:* Wednesday, May 9, 2018 6:12 AM > > > *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > Hi Paul, > > > > > > > > > > > > Thanks for the review. > > > > > > We are now working on CentOS7 support, but it is not complete yet, so > we > > > will be on CentOS6 about 6-12 months. > > > > > > > > > > > > Shahar > > > ------------------------------ > > > > > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > > > paul.e.luse(a)intel.com&gt; > > > *Sent:* Wednesday, May 9, 2018 3:51:23 PM > > > *To:* Storage Performance Development Kit > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > Thanks for doing that! I put my 2 cents on the review, one more data > point > > > for the maintainers might help – help long do you believe your team > will be > > > on CentOS6? Don’t need a specific date but are we talking months, > years, > > > indefinitely? > > > > > > > > > > > > I think the other implication of accepting this in the community would > be > > > to spin up a CentOS6 VM for some basic testing for as long as we hold > that > > > patch in our DPDK fork which is not a huge thing but want to make sure > > > that’s clear as well. If nothing else we’ll know if something else > breaks > > > it sooner than later and will need to decide, again, at that time if we > > > pull it or if it’s a high enough priority for anyone to dig in and > find a > > > potential solution like you did here. Assuming, of course, that this > one > > > is accepted J > > > > > > > > > > > > Thanks, > > > > > > Paul > > > > > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < > spdk-bounces(a)lists.01.org>] > > > *On Behalf Of *Shahar Salzman > > > *Sent:* Wednesday, May 9, 2018 2:01 AM > > > *To:* spdk(a)lists.01.org > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > Hi Ben, > > > > > > > > > > > > Pushed the fix to gerrithub - > > > https://review.gerrithub.io/#/c/spdk/dpdk/+/410559/ > > > > > > > > > > > > We are currently using a "frozen" dpdk tree based version 16.07, but I > am > > > pushing to using the spdk fork and performing regular updates so if > things > > > break, I will attempt to fix them, and post on the list. > > > > > > > > > > > > The above fixes a blunt gcc API issue, but there are more subtle inner > > > module issues which I have not investigated, simply removed the > modules all > > > together as they are not needed by spdk. > > > > > > > > > > > > I can set up a howto for users who may need CentOS6 support or we can > > > modify the config (as these modules are not really needed for spdk). > For > > > the sake of clarity here are the modules "configed out" (as I mentioned > > > they are not used by spdk): > > > > > > +CONFIG_RTE_LIBRTE_PMD_BOND=n > > > > > > +CONFIG_RTE_LIBRTE_CRYPTODEV=n > > > > > > +CONFIG_RTE_LIBRTE_SECURITY=n > > > > > > +CONFIG_RTE_LIBRTE_PMD_SW_EVENTDEV=n > > > > > > +CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF=n > > > > > > +CONFIG_RTE_LIBRTE_LPM=n > > > > > > +CONFIG_RTE_LIBRTE_FLOW_CLASSIFY=n > > > > > > +CONFIG_RTE_LIBRTE_TABLE=n > > > > > > +CONFIG_RTE_LIBRTE_PIPELINE=n > > > > > > +CONFIG_RTE_TEST_PMD=n > > > > > > +CONFIG_RTE_LIBRTE_PMD_TAP=n > > > > > > +CONFIG_RTE_LIBRTE_AVP_PMD=n > > > > > > > > > > > > Regarding the -fn-strict-aliasing issue, I'll see if I can do the same > > > trick I did for the gcc alias in the Makefile, and push the fix for > review. > > > > > > > > > > > > Shahar > > > > > > > > > ------------------------------ > > > > > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Walker, > Benjamin < > > > benjamin.walker(a)intel.com&gt; > > > *Sent:* Tuesday, May 8, 2018 7:49:44 PM > > > *To:* spdk(a)lists.01.org > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > On Tue, 2018-05-08 at 14:34 +0000, Shahar Salzman wrote: > > > > Got it. > > > > > > > > I was tracking the dpdk project, and focused on the branch that > Daniel > > > > > > told me > > > > to work on. > > > > So would the correct process be to push the patch to the spdk/dpdk > > > > > > gerrithub > > > > and discuss the commit, or the other way around? > > > > > > Yes - go ahead and push to spdk/dpdk on GerritHub so we can all review > it. > > > The > > > SPDK project has never written out a policy on which versions of DPDK > are > > > supported for any given SPDK release. The code is always tested using > the > > > latest > > > released DPDK, and then attempts to support older versions of DPDK are > > > done in > > > an ad hoc manner. Similarly, the project never explicitly states which > > > operating > > > systems are supported. Instead, we indicate which operating systems the > > > code is > > > regularly tested on (Fedora N and N-1, Ubuntu latest, Ubuntu LTS, > CentOS 7, > > > FreeBSD 11). > > > > > > I see that CentOS 6 is no longer receiving full updates and is in > > > maintenance > > > only mode. It will stop receiving even maintenance updates in 2020. I > > > think that > > > is indicative of the level of support SPDK should provide here. > > > > > > Given the above, if this patch can be applied to the DPDK fork and > doesn't > > > have > > > any additional impact, I'm not opposed to applying it. That wouldn't, > in my > > > mind, mean that SPDK is committed to supporting CentOS 6 indefinitely. > But > > > certainly that support could continue while it is relatively easy and > > > convenient. > > > > > > > > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > > > > > > paul.e.luse@ > > > <paul.e.luse@%0b>> intel.com> > > > > Sent: Tuesday, May 8, 2018 5:14:10 PM > > > > To: Storage Performance Development Kit > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > Sure, I understand that. SPDK maintains a fork of dpdk in github at > > > > > > https://g > > > > ithub.com/spdk/dpdk for the purpose of temporarily putting things in > > > > > > that we > > > > need that can’t make it upstream yet for whatever reason – this is > the > > > > submodule in the SPDK repo. I’m not saying that it makes sense for > your > > > > > > patch > > > > to land there permanently but it does make sense to talk about it. > > > > > > > > No problem on not being on the call, we’ll get some input from the > > > > > > maintainers > > > > and others I’m sure… > > > > > > > > Thx > > > > Paul > > > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < > spdk-bounces(a)lists.01.org>] > > > > > > On Behalf Of Shahar Salzman > > > > Sent: Tuesday, May 8, 2018 7:04 AM > > > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > Hey Paul, > > > > > > > > The problem is that the patch is in dpdk and not in spdk, and this > is the > > > > reason I offered to support it. > > > > The patch (bellow) is only a few lines long modifying the deprecated > > > > > > attribute > > > > definition, a modification to the config, and an extra compilation > flag. > > > > All of the above are required in order to build dpdk 18.01 on > CentOS6, > > > > > > there > > > > are no problems building spdk on CentOS6. > > > > > > > > I understood that you are already applying minor patches to dpdk, so > > > > > > question > > > > is, whether this patch can be added? > > > > > > > > Unfortunately due to some personal issues, I am unable to > participate in > > > > today's meeting. > > > > > > > > Thanks, > > > > Shahar > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > > > > > > paul.e.luse@ > > > <paul.e.luse@%0b>> intel.com> > > > > Sent: Tuesday, May 8, 2018 4:25:29 PM > > > > To: Storage Performance Development Kit > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > Hi Shahar, > > > > > > > > Couple of quick things: > > > > > > > > * thanks for putting all this time and effort into trying to make > this > > > > > > work! > > > > * in the community model, patches aren’t accepted based on one > > > > > > individual’s > > > > ability/desire to support them so appreciate the offer but in > general any > > > > patch that is accepted becomes the responsibly of the community > (mostly > > > > > > the > > > > maintainers) to support it long term. Otherwise we’d have what I > like > > > > > > to call > > > > “the flea market model” where there’s a bunch of separate individuals > > > > supporting their own things with very little cohesiveness across > > > > > > everything J > > > > > > > > So, wrt next steps, if the patch isn’t a tremendous amount of effort > I > > > > > > would > > > > suggest you go ahead and submit it. That’s the best way to get > everyone > > > > > > on the > > > > same page wrt exactly what we’re talking about. If it is some > > > > > > significant > > > > effort then, as Pawel states, you can call into a community meeting > (see > > > > > > email > > > > I sent out last night) or you can explain more details on this list. > > > > > > > > Thanks again!! > > > > Paul > > > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < > spdk-bounces(a)lists.01.org>] > > > > > > On Behalf Of Wodkowski, PawelX > > > > Sent: Tuesday, May 8, 2018 5:57 AM > > > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > I think someone will answer you on this mailing list but there is > > > > > > community > > > > meeting today too. You can raise this if you wish. > > > > > > > > Pawel > > > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < > spdk-bounces(a)lists.01.org>] > > > > > > On Behalf Of Shahar Salzman > > > > Sent: Tuesday, May 8, 2018 2:23 PM > > > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > Hi, > > > > > > > > I attempted to push the gcc support patch to dpdk, and got rejected > > > > > > since they > > > > do not wish to support CentOS6. > > > > Would it be possible to manually add the patch to the supported dpdk > > > > > > version > > > > (currently 18.01). > > > > I would be happy to support CentOS6 issues. > > > > > > > > Thanks, > > > > Shahar > > > > > > > > From: Shahar Salzman > > > > Sent: Tuesday, April 24, 2018 3:40:51 PM > > > > To: Storage Performance Development Kit > > > > Cc: Ilan Steinberg; Ido Benda; Yael Shavit > > > > Subject: Re: Building spdk on CentOS6 > > > > > > > > OK, got everything to work. > > > > > > > > For DPDK, I had to to do the following: > > > > add a patch to make the new __rte_experimental attribute to work > > > > remove a bunch of modules via config which had some gnarly > compilation > > > > > > errors, > > > > and are not needed for spdk > > > > use -fno-strict-aliasing in the EXTRA_CFLAGS to avoid a ton of strict > > > > > > aliasing > > > > errors. > > > > > > > > I tried spdk 18.01, and ran into some issues with the shared library > > > > > > build. As > > > > I saw that Daniel had already dealt with them, I tried cherry > picking, > > > > > > but > > > > eventually used the master branch. > > > > I got these type of warnings: > > > > > > > > /usr/bin/ld: warning: librte_mbuf.so.3.1, needed by > > > > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- > > > > gcc/lib/librte_bus_pci.so, not found (try using -rpath or > -rpath-link) > > > > /usr/bin/ld: warning: librte_ethdev.so.8.1, needed by > > > > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- > > > > gcc/lib/librte_bus_pci.so, not found (try using -rpath or > -rpath-link) > > > > > > > > I added the following patch bellow to make them go away. > > > > > > > > I'll approach the dpdk community on the mailing list in order to get > the > > > > __rte_experimental stuff to work. > > > > My final configuration is: > > > > CenrOS6.4 > > > > gcc 4.4.7-17 > > > > dpdk v18.02 > > > > spdk master > > > > > > > > What are the next steps? > > > > > > > > Shahar > > > > > > > > P.S. Here is the patch to remove the warnings above. > > > > commit d4ef744de1cc4928972b7042d1c90aff12db7867 > > > > Author: shahar salzman <shahar.salzman(a)kaminario.com&gt; > > > > Date: Tue Apr 24 14:31:52 2018 +0300 > > > > > > > > lib/env_dpdk: add required libraries > > > > > > > > diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk > > > > index 450043c..b46bfed 100644 > > > > --- a/lib/env_dpdk/env.mk > > > > +++ b/lib/env_dpdk/env.mk > > > > @@ -68,6 +68,18 @@ ifneq ($(wildcard > > > > > > $(DPDK_ABS_DIR)/lib/librte_malloc.*),) > > > > DPDK_LIB_LIST += rte_malloc > > > > endif > > > > > > > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_mbuf.*)) > > > > +DPDK_LIB_LIST += rte_mbuf > > > > +endif > > > > + > > > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_net.*)) > > > > +DPDK_LIB_LIST += rte_net > > > > +endif > > > > + > > > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_ethdev.*)) > > > > +DPDK_LIB_LIST += rte_ethdev > > > > +endif > > > > + > > > > # librte_pci and librte_bus_pci were added in DPDK 17.11. Link these > > > > libraries conditionally > > > > # based on their existence to maintain backward compatibility. > > > > ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_pci.*)) > > > > > > > > > > > > > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Shahar Salzman > > > > > > <shahar.sal > > > > zman(a)kaminario.com&gt; > > > > Sent: Tuesday, April 24, 2018 9:57 AM > > > > To: Storage Performance Development Kit > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > Great, and thanks! > > > > > > > > I have been running into issues compiling dpdk 17.11 on CentOS6 > (there > > > > > > seems > > > > to be a gcc issue that does not exist in other versions), I'll focus > my > > > > efforts on dpdk 18.02. > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Verkamp, Daniel > < > > > > > > daniel.ve > > > > rkamp(a)intel.com&gt; > > > > Sent: Monday, April 23, 2018 9:40:36 PM > > > > To: Storage Performance Development Kit > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > Hi Shahar, > > > > > > > > I’ve pushed a patch for the SPDK master branch that should allow it > to > > > > > > compile > > > > with DPDK 16.07: https://review.gerrithub.io/#/c/spdk/spdk/+/408743/ > > > > > > > > Gerrit Code Review > > > > > > > > review.gerrithub.io > > > > > > > > Keep in touch. Copyright © 2017 | GerritForge Ltd. > info(a)gerritforge.com > > > > www.gerritforge.com > > > > > > > > > > > > However, DPDK 16.07 is no longer supported upstream; I highly > recommend > > > > > > that > > > > you update to a newer version. SPDK is currently tested with DPDK > 18.02. > > > > > > > > The VFIO issue you mention has also been fixed on SPDK master, and it > > > > > > will be > > > > part of the SPDK v18.04 release. > > > > > > > > Thanks, > > > > -- Daniel > > > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < > spdk-bounces(a)lists.01.org>] > > > > > > On Behalf Of Shahar Salzman > > > > Sent: Sunday, April 22, 2018 6:41 AM > > > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > > > Subject: [SPDK] Building spdk on CentOS6 > > > > > > > > Hi, > > > > > > > > Finally got to looking at support of spdk build on CentOS6, things > look > > > > > > good, > > > > except for one issue. > > > > spdk is latest 18.01.x version, dpdk is 16.07 (+3 dpdk patches to > allow > > > > compilation) and some minor patches (mainly some memory configuration > > > > > > stuff), > > > > kernel is a patched 4.9.6. > > > > > > > > build succeeded except for the usage of the dpdk function > > > > > > pci_vfio_is_enabled. > > > > I had to apply the patch bellow, removing the usage of this function > and > > > > > > then > > > > compilation completed without any issues. > > > > > > > > It seems that I am missing some sort of dpdk configuration as I see > that > > > > > > the > > > > function is built, but not packaged into the generated archive. > > > > > > > > I went back to square one and ran the instructions in > > > > > > http://www.spdk.io/doc/g > > > > etting_started.html, but I see no mention of dpdk there. Actually the > > > > ./configure requires it. > > > > > > > > My next step is to use a more recent dpdk, but shouldn't this work > with > > > > > > my > > > > version? Am I missing some dpdk configuration? > > > > BTW, as we are not using vhost, on our 17.07 version we simply use > > > > CONFIG_VHOST=n in order to skip this, but I would be happier if we > used a > > > > better solution. > > > > > > > > Shahar > > > > > > > > P.S. Here is the patch to remove use of this function: > > > > diff --git a/lib/env_dpdk/vtophys.c b/lib/env_dpdk/vtophys.c > > > > index 92aa256..f38929f 100644 > > > > --- a/lib/env_dpdk/vtophys.c > > > > +++ b/lib/env_dpdk/vtophys.c > > > > @@ -53,8 +53,10 @@ > > > > #define SPDK_VFIO_ENABLED 1 > > > > #include <linux/vfio.h> > > > > > > > > +#if 0 > > > > /* Internal DPDK function forward declaration */ > > > > int pci_vfio_is_enabled(void); > > > > +#endif > > > > > > > > struct spdk_vfio_dma_map { > > > > struct vfio_iommu_type1_dma_map map; > > > > @@ -341,9 +343,11 @@ spdk_vtophys_iommu_init(void) > > > > DIR *dir; > > > > struct dirent *d; > > > > > > > > +#if 0 > > > > if (!pci_vfio_is_enabled()) { > > > > return; > > > > } > > > > +#endif > > > > > > > > dir = opendir("/proc/self/fd"); > > > > if (!dir) { > > > > > > > > > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > > > _______________________________________________ > > > SPDK mailing list > > > SPDK(a)lists.01.org > > > https://lists.01.org/mailman/listinfo/spdk > > > _______________________________________________ > > > SPDK mailing list > > > SPDK(a)lists.01.org > > > https://lists.01.org/mailman/listinfo/spdk > > > > > > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

Hi Gyan, If you do not need to use your own copy of DPDK, and can just rely on the dpdk submodule in the SPDK repository, then Ben’s suggestions in his e-mail are the best way to go. If you need to use a version of DPDK different from the dpdk submodule – maybe you have some of your own patches, or want to use a newer DPDK version than the dpdk submodule – then my suggestions below will only work if DPDK is first built explicitly. http://doc.dpdk.org/guides/linux_gsg/build_dpdk.html gives more details on how to build DPDK on your own. If you still have any problems, feel free to reach out on the mailing list and someone will help out. Thanks, -Jim On 8/24/18, 4:50 PM, "SPDK on behalf of Gyan Prakash" < spdk-bounces(a)lists.01.org on behalf of gyapra2016(a)gmail.com&gt; wrote: Jim, Okay, sure, thanks for the explanation. BTW, I tried your suggested steps for spdk-18.07 and here is the output: *make CONFIG_DPDK_DIR=./dpdk-18.05/x86_64-native-linuxapp-gcc* CONFIG_RDMA=y CC lib/vhost/vhost.o In file included from vhost_internal.h:39:0, from vhost.c:43: rte_vhost/rte_vhost.h:47:24: fatal error: rte_config.h: No such file or directory #include <rte_config.h> ^ compilation terminated. make[2]: *** [vhost.o] Error 1 make[1]: *** [vhost] Error 2 make: *** [lib] Error 2 For other option, config works fine but make gave me error: usr/bin/ld: /root/spdk-18.07/spdk-18.07/dpdk-18.05/x86_64-native-linuxapp-gcc/lib/librte_eal.a(eal.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC /root/spdk-18.07/spdk-18.07/dpdk-18.05/x86_64-native-linuxapp-gcc/lib/librte_eal.a(eal.o): error adding symbols: Bad value collect2: error: ld returned 1 exit status make[1]: *** [/root/spdk-18.07/spdk-18.07/build/lib/libspdk.so] Error 1 make: *** [shared_lib] Error 2 I will proceed further from the latest version, but not sure why spdk-18.07 is not working. Thanks, Gyan On Fri, Aug 24, 2018 at 4:44 PM Harris, James R < james.r.harris(a)intel.com&gt; wrote: > Hi Gyan, > > Thanks for the update. Those unit test error messages are expected. The > unit tests are specifically testing error cases which generate error > messages like you see here. The run summary at the bottom is the key > output and shows 0 test failures. > > Thanks, > > -Jim > > > On 8/24/18, 4:38 PM, "SPDK on behalf of Gyan Prakash" < > spdk-bounces(a)lists.01.org on behalf of gyapra2016(a)gmail.com&gt; wrote: > > Benjamin, > > Tests passed with your suggested steps, but there are some vHost > errors, > not sure if it is false positive. Please see below: > > lapsed time = 0.000 seconds > ++ uname -s > + '[' Linux = Linux ']' > + /root/spdk-aug24-2018/spdk/test/unit/lib/vhost/vhost.c/vhost_ut > > > CUnit - A unit testing framework for C - Version 2.1-2 > http://cunit.sourceforge.net/ > > > Suite: vhost_suite > Test: desc_to_iov .../root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: > 454:spdk_vhost_vring_desc_to_iov: *ERROR*: SPDK_VHOST_IOVS_MAX(129) > reached > passed > > > > * Test: create_controller > .../root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: > 661:spdk_vhost_dev_register: *ERROR*: Can't register controller with no > name/root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: > 619:spdk_vhost_parse_core_mask: *ERROR*: no cpu is selected among > reactor > mask(=1)/root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: > 673:spdk_vhost_dev_register: *ERROR*: cpumask 0x2 is invalid (app mask > is > 0x1)/root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: > 686:spdk_vhost_dev_register: *ERROR*: Resulting socket path for > controller > > xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: > 679:spdk_vhost_dev_register: *ERROR*: vhost controller vdev_name_0 > already > exists.* > passed > Test: dev_find_by_vid ...passed > Test: remove_controller > .../root/spdk-aug24-2018/spdk/lib/vhost/vhost.c: > 772:spdk_vhost_dev_unregister:* *ERROR**: Controller vdev_name_0 has > still > valid connection. > passed > > Run Summary: Type Total Ran Passed Failed Inactive > suites 1 1 n/a 0 0 > tests 4 4 4 0 0 > asserts 53 53 53 0 n/a > > Elapsed time = 0.000 seconds > + '[' no = yes ']' > + set +x > > > ===================== > *All unit tests passed* > ===================== > WARN: lcov not installed or SPDK built without coverage! > WARN: neither valgrind nor ASAN is enabled! > > > > Thanks for the help, > Gyan > > > On Fri, Aug 24, 2018 at 3:23 PM Walker, Benjamin < > benjamin.walker(a)intel.com&gt; > wrote: > > > On Fri, 2018-08-24 at 15:00 -0700, Gyan Prakash wrote: > > > Hi all, > > > > > > I am trying to build spdk-18.07 with dpdk-18.05 on Centos 7.4 > system with > > > kernel 4.13.9. > > > > > > make DPDK_DIR=./dpdk-18.05/x86_64-native-linuxapp-gcc CONFIG_RDMA=y > > > > > > and seeing following compilation failure. Can someone please tell > me > > what I > > > am missing there? > > > > > > CC lib/vhost/vhost.o > > > In file included from vhost_internal.h:39:0, > > > from vhost.c:43: > > > rte_vhost/rte_vhost.h:47:24: fatal error: rte_config.h: No such > file or > > > directory > > > #include <rte_config.h> > > > ^ > > > compilation terminated. > > > make[2]: *** [vhost.o] Error 1 > > > make[1]: *** [vhost] Error 2 > > > make: *** [lib] Error 2 > > > > > > This means that the build couldn't find your DPDK installation (I > think). > > As a > > quick sanity test, can you try this sequence of commands: > > > > git clone https://github.com/spdk/spdk > > cd spdk > > git submodule update --init > > ./scripts/pkgdep.sh > > ./configure --with-rdma > > make > > ./test/unit/unittest.sh > > > > This sequence is taken from this page: > > http://www.spdk.io/doc/getting_started.html > > > > If you need to do something more advanced, like use a fork of DPDK, > that's > > also > > possible but we should confirm that the simple case is working first. > > > > > > > > > > Thanks, > > > Gyan > > > > > > On Sun, May 13, 2018 at 1:45 AM Shahar Salzman < > > shahar.salzman(a)kaminario.com&gt; > > > wrote: > > > > > > > Hi Daniel, > > > > > > > > > > > > I ran the dpdk build with T=spdk-linuxapp-gcc and build works > fine. > > > > > > > > Isn't config/common_spdk only for configuration options (no > reference > > for > > > > this type of definition) ? I would have thought that the > compilation > > flag > > > > should go in spdk/dpdkbuild/Makefile, or in one of the mk files? > > > > > > > > > > > > Shahar > > > > > > > > > > > > > > > > ------------------------------ > > > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Verkamp, > Daniel > > < > > > > daniel.verkamp(a)intel.com&gt; > > > > *Sent:* Thursday, May 10, 2018 8:31:21 PM > > > > *To:* Storage Performance Development Kit > > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > Hi Shahar, > > > > > > > > > > > > > > > > All of the config options you listed below are already disabled > in the > > > > config that we use to build the DPDK subodule (see > config/common_spdk > > in > > > > our DPDK branch). Can you confirm that this works for you? > > > > > > > > > > > > > > > > I think we can add -fno-strict-aliasing to the DPDK > config/common_spdk > > > > file as well if this is necessary to make it work on RHEL 6 (I am > > surprised > > > > DPDK doesn’t already compile with this enabled). If you could > upload a > > > > second patch for this, that would be great. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > -- Daniel > > > > > > > > > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of > *Shahar > > > > Salzman > > > > *Sent:* Thursday, May 10, 2018 6:39 AM > > > > *To:* Storage Performance Development Kit < spdk(a)lists.01.org&gt; > > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > > > > > Hi Daniel, > > > > > > > > > > > > > > > > I updated the patch as you suggested. > > > > > > > > How do you want to approach the gcc flag and the modules that > spdk does > > > > not need? > > > > > > > > If you are building dpdk from an external makefile (as we are > doing), > > all > > > > of the above can be added to the makefile with CONFIG...=n and > > > > EXTRA_CFLAGS="-fno-strict-aliasing", or I can send you another > dpdk > > patch > > > > fixing this within dpdk, so there is no change in the builder, > and > > maybe > > > > there is another option. > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > Shahar > > > > > > > > > > > > ------------------------------ > > > > > > > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, > Paul E < > > > > paul.e.luse(a)intel.com&gt; > > > > *Sent:* Wednesday, May 9, 2018 8:19:33 PM > > > > *To:* Storage Performance Development Kit > > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > > > > > That’s great, thanks Daniel! > > > > > > > > > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < > > spdk-bounces(a)lists.01.org>] > > > > *On Behalf Of *Verkamp, Daniel > > > > *Sent:* Wednesday, May 9, 2018 9:43 AM > > > > *To:* Storage Performance Development Kit < spdk(a)lists.01.org&gt; > > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > > > > > I agree with Paul’s review; it looks like it shouldn’t be a big > > problem to > > > > carry a patch for this in our DPDK submodule, at least while > CentOS 6 > > is > > > > still officially supported. (I posted a few review comments on > the > > patch, > > > > but the general approach looks fine.) > > > > > > > > > > > > > > > > The Intel build pool had a CentOS 6 test machine in the pool > until > > > > recently, and we should be able to resurrect it to provide test > > coverage > > > > again. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > -- Daniel > > > > > > > > > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < > > spdk-bounces(a)lists.01.org>] > > > > *On Behalf Of *Shahar Salzman > > > > *Sent:* Wednesday, May 9, 2018 6:12 AM > > > > *To:* Storage Performance Development Kit < spdk(a)lists.01.org&gt; > > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > > > > > Hi Paul, > > > > > > > > > > > > > > > > Thanks for the review. > > > > > > > > We are now working on CentOS7 support, but it is not complete > yet, so > > we > > > > will be on CentOS6 about 6-12 months. > > > > > > > > > > > > > > > > Shahar > > > > ------------------------------ > > > > > > > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, > Paul E < > > > > paul.e.luse(a)intel.com&gt; > > > > *Sent:* Wednesday, May 9, 2018 3:51:23 PM > > > > *To:* Storage Performance Development Kit > > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > > > > > Thanks for doing that! I put my 2 cents on the review, one more > data > > point > > > > for the maintainers might help – help long do you believe your > team > > will be > > > > on CentOS6? Don’t need a specific date but are we talking months, > > years, > > > > indefinitely? > > > > > > > > > > > > > > > > I think the other implication of accepting this in the community > would > > be > > > > to spin up a CentOS6 VM for some basic testing for as long as we > hold > > that > > > > patch in our DPDK fork which is not a huge thing but want to > make sure > > > > that’s clear as well. If nothing else we’ll know if something > else > > breaks > > > > it sooner than later and will need to decide, again, at that > time if we > > > > pull it or if it’s a high enough priority for anyone to dig in > and > > find a > > > > potential solution like you did here. Assuming, of course, that > this > > one > > > > is accepted J > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Paul > > > > > > > > > > > > > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < > > spdk-bounces(a)lists.01.org>] > > > > *On Behalf Of *Shahar Salzman > > > > *Sent:* Wednesday, May 9, 2018 2:01 AM > > > > *To:* spdk(a)lists.01.org > > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > > > > > Hi Ben, > > > > > > > > > > > > > > > > Pushed the fix to gerrithub - > > > > https://review.gerrithub.io/#/c/spdk/dpdk/+/410559/ > > > > > > > > > > > > > > > > We are currently using a "frozen" dpdk tree based version 16.07, > but I > > am > > > > pushing to using the spdk fork and performing regular updates so > if > > things > > > > break, I will attempt to fix them, and post on the list. > > > > > > > > > > > > > > > > The above fixes a blunt gcc API issue, but there are more subtle > inner > > > > module issues which I have not investigated, simply removed the > > modules all > > > > together as they are not needed by spdk. > > > > > > > > > > > > > > > > I can set up a howto for users who may need CentOS6 support or > we can > > > > modify the config (as these modules are not really needed for > spdk). > > For > > > > the sake of clarity here are the modules "configed out" (as I > mentioned > > > > they are not used by spdk): > > > > > > > > +CONFIG_RTE_LIBRTE_PMD_BOND=n > > > > > > > > +CONFIG_RTE_LIBRTE_CRYPTODEV=n > > > > > > > > +CONFIG_RTE_LIBRTE_SECURITY=n > > > > > > > > +CONFIG_RTE_LIBRTE_PMD_SW_EVENTDEV=n > > > > > > > > +CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF=n > > > > > > > > +CONFIG_RTE_LIBRTE_LPM=n > > > > > > > > +CONFIG_RTE_LIBRTE_FLOW_CLASSIFY=n > > > > > > > > +CONFIG_RTE_LIBRTE_TABLE=n > > > > > > > > +CONFIG_RTE_LIBRTE_PIPELINE=n > > > > > > > > +CONFIG_RTE_TEST_PMD=n > > > > > > > > +CONFIG_RTE_LIBRTE_PMD_TAP=n > > > > > > > > +CONFIG_RTE_LIBRTE_AVP_PMD=n > > > > > > > > > > > > > > > > Regarding the -fn-strict-aliasing issue, I'll see if I can do > the same > > > > trick I did for the gcc alias in the Makefile, and push the fix > for > > review. > > > > > > > > > > > > > > > > Shahar > > > > > > > > > > > > ------------------------------ > > > > > > > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Walker, > > Benjamin < > > > > benjamin.walker(a)intel.com&gt; > > > > *Sent:* Tuesday, May 8, 2018 7:49:44 PM > > > > *To:* spdk(a)lists.01.org > > > > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > > > > > > > On Tue, 2018-05-08 at 14:34 +0000, Shahar Salzman wrote: > > > > > Got it. > > > > > > > > > > I was tracking the dpdk project, and focused on the branch that > > Daniel > > > > > > > > told me > > > > > to work on. > > > > > So would the correct process be to push the patch to the > spdk/dpdk > > > > > > > > gerrithub > > > > > and discuss the commit, or the other way around? > > > > > > > > Yes - go ahead and push to spdk/dpdk on GerritHub so we can all > review > > it. > > > > The > > > > SPDK project has never written out a policy on which versions of > DPDK > > are > > > > supported for any given SPDK release. The code is always tested > using > > the > > > > latest > > > > released DPDK, and then attempts to support older versions of > DPDK are > > > > done in > > > > an ad hoc manner. Similarly, the project never explicitly states > which > > > > operating > > > > systems are supported. Instead, we indicate which operating > systems the > > > > code is > > > > regularly tested on (Fedora N and N-1, Ubuntu latest, Ubuntu LTS, > > CentOS 7, > > > > FreeBSD 11). > > > > > > > > I see that CentOS 6 is no longer receiving full updates and is in > > > > maintenance > > > > only mode. It will stop receiving even maintenance updates in > 2020. I > > > > think that > > > > is indicative of the level of support SPDK should provide here. > > > > > > > > Given the above, if this patch can be applied to the DPDK fork > and > > doesn't > > > > have > > > > any additional impact, I'm not opposed to applying it. That > wouldn't, > > in my > > > > mind, mean that SPDK is committed to supporting CentOS 6 > indefinitely. > > But > > > > certainly that support could continue while it is relatively > easy and > > > > convenient. > > > > > > > > > > > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, > Paul E < > > > > > > > > paul.e.luse@ > > > > <paul.e.luse@%0b>> intel.com> > > > > > Sent: Tuesday, May 8, 2018 5:14:10 PM > > > > > To: Storage Performance Development Kit > > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > Sure, I understand that. SPDK maintains a fork of dpdk in > github at > > > > > > > > https://g > > > > > ithub.com/spdk/dpdk for the purpose of temporarily putting > things in > > > > > > > > that we > > > > > need that can’t make it upstream yet for whatever reason – > this is > > the > > > > > submodule in the SPDK repo. I’m not saying that it makes > sense for > > your > > > > > > > > patch > > > > > to land there permanently but it does make sense to talk about > it. > > > > > > > > > > No problem on not being on the call, we’ll get some input from > the > > > > > > > > maintainers > > > > > and others I’m sure… > > > > > > > > > > Thx > > > > > Paul > > > > > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < > > spdk-bounces(a)lists.01.org>] > > > > > > > > On Behalf Of Shahar Salzman > > > > > Sent: Tuesday, May 8, 2018 7:04 AM > > > > > To: Storage Performance Development Kit < spdk(a)lists.01.org&gt; > > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > Hey Paul, > > > > > > > > > > The problem is that the patch is in dpdk and not in spdk, and > this > > is the > > > > > reason I offered to support it. > > > > > The patch (bellow) is only a few lines long modifying the > deprecated > > > > > > > > attribute > > > > > definition, a modification to the config, and an extra > compilation > > flag. > > > > > All of the above are required in order to build dpdk 18.01 on > > CentOS6, > > > > > > > > there > > > > > are no problems building spdk on CentOS6. > > > > > > > > > > I understood that you are already applying minor patches to > dpdk, so > > > > > > > > question > > > > > is, whether this patch can be added? > > > > > > > > > > Unfortunately due to some personal issues, I am unable to > > participate in > > > > > today's meeting. > > > > > > > > > > Thanks, > > > > > Shahar > > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, > Paul E < > > > > > > > > paul.e.luse@ > > > > <paul.e.luse@%0b>> intel.com> > > > > > Sent: Tuesday, May 8, 2018 4:25:29 PM > > > > > To: Storage Performance Development Kit > > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > Hi Shahar, > > > > > > > > > > Couple of quick things: > > > > > > > > > > * thanks for putting all this time and effort into trying to > make > > this > > > > > > > > work! > > > > > * in the community model, patches aren’t accepted based on one > > > > > > > > individual’s > > > > > ability/desire to support them so appreciate the offer but in > > general any > > > > > patch that is accepted becomes the responsibly of the community > > (mostly > > > > > > > > the > > > > > maintainers) to support it long term. Otherwise we’d have > what I > > like > > > > > > > > to call > > > > > “the flea market model” where there’s a bunch of separate > individuals > > > > > supporting their own things with very little cohesiveness > across > > > > > > > > everything J > > > > > > > > > > So, wrt next steps, if the patch isn’t a tremendous amount of > effort > > I > > > > > > > > would > > > > > suggest you go ahead and submit it. That’s the best way to get > > everyone > > > > > > > > on the > > > > > same page wrt exactly what we’re talking about. If it is some > > > > > > > > significant > > > > > effort then, as Pawel states, you can call into a community > meeting > > (see > > > > > > > > email > > > > > I sent out last night) or you can explain more details on this > list. > > > > > > > > > > Thanks again!! > > > > > Paul > > > > > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < > > spdk-bounces(a)lists.01.org>] > > > > > > > > On Behalf Of Wodkowski, PawelX > > > > > Sent: Tuesday, May 8, 2018 5:57 AM > > > > > To: Storage Performance Development Kit < spdk(a)lists.01.org&gt; > > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > I think someone will answer you on this mailing list but there > is > > > > > > > > community > > > > > meeting today too. You can raise this if you wish. > > > > > > > > > > Pawel > > > > > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < > > spdk-bounces(a)lists.01.org>] > > > > > > > > On Behalf Of Shahar Salzman > > > > > Sent: Tuesday, May 8, 2018 2:23 PM > > > > > To: Storage Performance Development Kit < spdk(a)lists.01.org&gt; > > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > Hi, > > > > > > > > > > I attempted to push the gcc support patch to dpdk, and got > rejected > > > > > > > > since they > > > > > do not wish to support CentOS6. > > > > > Would it be possible to manually add the patch to the > supported dpdk > > > > > > > > version > > > > > (currently 18.01). > > > > > I would be happy to support CentOS6 issues. > > > > > > > > > > Thanks, > > > > > Shahar > > > > > > > > > > From: Shahar Salzman > > > > > Sent: Tuesday, April 24, 2018 3:40:51 PM > > > > > To: Storage Performance Development Kit > > > > > Cc: Ilan Steinberg; Ido Benda; Yael Shavit > > > > > Subject: Re: Building spdk on CentOS6 > > > > > > > > > > OK, got everything to work. > > > > > > > > > > For DPDK, I had to to do the following: > > > > > add a patch to make the new __rte_experimental attribute to > work > > > > > remove a bunch of modules via config which had some gnarly > > compilation > > > > > > > > errors, > > > > > and are not needed for spdk > > > > > use -fno-strict-aliasing in the EXTRA_CFLAGS to avoid a ton of > strict > > > > > > > > aliasing > > > > > errors. > > > > > > > > > > I tried spdk 18.01, and ran into some issues with the shared > library > > > > > > > > build. As > > > > > I saw that Daniel had already dealt with them, I tried cherry > > picking, > > > > > > > > but > > > > > eventually used the master branch. > > > > > I got these type of warnings: > > > > > > > > > > /usr/bin/ld: warning: librte_mbuf.so.3.1, needed by > > > > > > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- > > > > > gcc/lib/librte_bus_pci.so, not found (try using -rpath or > > -rpath-link) > > > > > /usr/bin/ld: warning: librte_ethdev.so.8.1, needed by > > > > > > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- > > > > > gcc/lib/librte_bus_pci.so, not found (try using -rpath or > > -rpath-link) > > > > > > > > > > I added the following patch bellow to make them go away. > > > > > > > > > > I'll approach the dpdk community on the mailing list in order > to get > > the > > > > > __rte_experimental stuff to work. > > > > > My final configuration is: > > > > > CenrOS6.4 > > > > > gcc 4.4.7-17 > > > > > dpdk v18.02 > > > > > spdk master > > > > > > > > > > What are the next steps? > > > > > > > > > > Shahar > > > > > > > > > > P.S. Here is the patch to remove the warnings above. > > > > > commit d4ef744de1cc4928972b7042d1c90aff12db7867 > > > > > Author: shahar salzman <shahar.salzman(a)kaminario.com&gt; > > > > > Date: Tue Apr 24 14:31:52 2018 +0300 > > > > > > > > > > lib/env_dpdk: add required libraries > > > > > > > > > > diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk > > > > > index 450043c..b46bfed 100644 > > > > > --- a/lib/env_dpdk/env.mk > > > > > +++ b/lib/env_dpdk/env.mk > > > > > @@ -68,6 +68,18 @@ ifneq ($(wildcard > > > > > > > > $(DPDK_ABS_DIR)/lib/librte_malloc.*),) > > > > > DPDK_LIB_LIST += rte_malloc > > > > > endif > > > > > > > > > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_mbuf.*)) > > > > > +DPDK_LIB_LIST += rte_mbuf > > > > > +endif > > > > > + > > > > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_net.*)) > > > > > +DPDK_LIB_LIST += rte_net > > > > > +endif > > > > > + > > > > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_ethdev.*)) > > > > > +DPDK_LIB_LIST += rte_ethdev > > > > > +endif > > > > > + > > > > > # librte_pci and librte_bus_pci were added in DPDK 17.11. > Link these > > > > > libraries conditionally > > > > > # based on their existence to maintain backward compatibility. > > > > > ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_pci.*)) > > > > > > > > > > > > > > > > > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Shahar > Salzman > > > > > > > > <shahar.sal > > > > > zman(a)kaminario.com&gt; > > > > > Sent: Tuesday, April 24, 2018 9:57 AM > > > > > To: Storage Performance Development Kit > > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > Great, and thanks! > > > > > > > > > > I have been running into issues compiling dpdk 17.11 on CentOS6 > > (there > > > > > > > > seems > > > > > to be a gcc issue that does not exist in other versions), I'll > focus > > my > > > > > efforts on dpdk 18.02. > > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Verkamp, > Daniel > > < > > > > > > > > daniel.ve > > > > > rkamp(a)intel.com&gt; > > > > > Sent: Monday, April 23, 2018 9:40:36 PM > > > > > To: Storage Performance Development Kit > > > > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > > > > > > > Hi Shahar, > > > > > > > > > > I’ve pushed a patch for the SPDK master branch that should > allow it > > to > > > > > > > > compile > > > > > with DPDK 16.07: > https://review.gerrithub.io/#/c/spdk/spdk/+/408743/ > > > > > > > > > > Gerrit Code Review > > > > > > > > > > review.gerrithub.io > > > > > > > > > > Keep in touch. Copyright © 2017 | GerritForge Ltd. > > info(a)gerritforge.com > > > > > www.gerritforge.com > > > > > > > > > > > > > > > However, DPDK 16.07 is no longer supported upstream; I highly > > recommend > > > > > > > > that > > > > > you update to a newer version. SPDK is currently tested with > DPDK > > 18.02. > > > > > > > > > > The VFIO issue you mention has also been fixed on SPDK master, > and it > > > > > > > > will be > > > > > part of the SPDK v18.04 release. > > > > > > > > > > Thanks, > > > > > -- Daniel > > > > > > > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < > > spdk-bounces(a)lists.01.org>] > > > > > > > > On Behalf Of Shahar Salzman > > > > > Sent: Sunday, April 22, 2018 6:41 AM > > > > > To: Storage Performance Development Kit < spdk(a)lists.01.org&gt; > > > > > Subject: [SPDK] Building spdk on CentOS6 > > > > > > > > > > Hi, > > > > > > > > > > Finally got to looking at support of spdk build on CentOS6, > things > > look > > > > > > > > good, > > > > > except for one issue. > > > > > spdk is latest 18.01.x version, dpdk is 16.07 (+3 dpdk patches > to > > allow > > > > > compilation) and some minor patches (mainly some memory > configuration > > > > > > > > stuff), > > > > > kernel is a patched 4.9.6. > > > > > > > > > > build succeeded except for the usage of the dpdk function > > > > > > > > pci_vfio_is_enabled. > > > > > I had to apply the patch bellow, removing the usage of this > function > > and > > > > > > > > then > > > > > compilation completed without any issues. > > > > > > > > > > It seems that I am missing some sort of dpdk configuration as > I see > > that > > > > > > > > the > > > > > function is built, but not packaged into the generated archive. > > > > > > > > > > I went back to square one and ran the instructions in > > > > > > > > http://www.spdk.io/doc/g > > > > > etting_started.html, but I see no mention of dpdk there. > Actually the > > > > > ./configure requires it. > > > > > > > > > > My next step is to use a more recent dpdk, but shouldn't this > work > > with > > > > > > > > my > > > > > version? Am I missing some dpdk configuration? > > > > > BTW, as we are not using vhost, on our 17.07 version we simply > use > > > > > CONFIG_VHOST=n in order to skip this, but I would be happier > if we > > used a > > > > > better solution. > > > > > > > > > > Shahar > > > > > > > > > > P.S. Here is the patch to remove use of this function: > > > > > diff --git a/lib/env_dpdk/vtophys.c b/lib/env_dpdk/vtophys.c > > > > > index 92aa256..f38929f 100644 > > > > > --- a/lib/env_dpdk/vtophys.c > > > > > +++ b/lib/env_dpdk/vtophys.c > > > > > @@ -53,8 +53,10 @@ > > > > > #define SPDK_VFIO_ENABLED 1 > > > > > #include <linux/vfio.h> > > > > > > > > > > +#if 0 > > > > > /* Internal DPDK function forward declaration */ > > > > > int pci_vfio_is_enabled(void); > > > > > +#endif > > > > > > > > > > struct spdk_vfio_dma_map { > > > > > struct vfio_iommu_type1_dma_map map; > > > > > @@ -341,9 +343,11 @@ spdk_vtophys_iommu_init(void) > > > > > DIR *dir; > > > > > struct dirent *d; > > > > > > > > > > +#if 0 > > > > > if (!pci_vfio_is_enabled()) { > > > > > return; > > > > > } > > > > > +#endif > > > > > > > > > > dir = opendir("/proc/self/fd"); > > > > > if (!dir) { > > > > > > > > > > > > > > > _______________________________________________ > > > > > SPDK mailing list > > > > > SPDK(a)lists.01.org > > > > > https://lists.01.org/mailman/listinfo/spdk > > > > > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > _______________________________________________ > > > > SPDK mailing list > > > > SPDK(a)lists.01.org > > > > https://lists.01.org/mailman/listinfo/spdk > > > > > > > > > > _______________________________________________ > > > SPDK mailing list > > > SPDK(a)lists.01.org > > > https://lists.01.org/mailman/listinfo/spdk > > > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > > > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

Hi Gyan, Can you try: CONFIG_DPDK_DIR instead of DPDK_DIR? Or: ./configure --with-dpdk=./dpdk-18.05/x86_64-native-linuxapp-gcc --with-rdma make This latter one (using configure) is what is recommended by https://github.com/spdk/spdk/blob/master/README.md. Regards, -Jim On 8/24/18, 3:00 PM, "SPDK on behalf of Gyan Prakash" < spdk-bounces(a)lists.01.org on behalf of gyapra2016(a)gmail.com&gt; wrote: Hi all, I am trying to build spdk-18.07 with dpdk-18.05 on Centos 7.4 system with kernel 4.13.9. make DPDK_DIR=./dpdk-18.05/x86_64-native-linuxapp-gcc CONFIG_RDMA=y and seeing following compilation failure. Can someone please tell me what I am missing there? CC lib/vhost/vhost.o In file included from vhost_internal.h:39:0, from vhost.c:43: rte_vhost/rte_vhost.h:47:24: fatal error: rte_config.h: No such file or directory #include <rte_config.h> ^ compilation terminated. make[2]: *** [vhost.o] Error 1 make[1]: *** [vhost] Error 2 make: *** [lib] Error 2 Thanks, Gyan On Sun, May 13, 2018 at 1:45 AM Shahar Salzman < shahar.salzman(a)kaminario.com&gt; wrote: > Hi Daniel, > > > I ran the dpdk build with T=spdk-linuxapp-gcc and build works fine. > > Isn't config/common_spdk only for configuration options (no reference for > this type of definition) ? I would have thought that the compilation flag > should go in spdk/dpdkbuild/Makefile, or in one of the mk files? > > > Shahar > > > > ------------------------------ > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Verkamp, Daniel < > daniel.verkamp(a)intel.com&gt; > *Sent:* Thursday, May 10, 2018 8:31:21 PM > *To:* Storage Performance Development Kit > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > Hi Shahar, > > > > All of the config options you listed below are already disabled in the > config that we use to build the DPDK subodule (see config/common_spdk in > our DPDK branch). Can you confirm that this works for you? > > > > I think we can add -fno-strict-aliasing to the DPDK config/common_spdk > file as well if this is necessary to make it work on RHEL 6 (I am surprised > DPDK doesn’t already compile with this enabled). If you could upload a > second patch for this, that would be great. > > > > Thanks, > > -- Daniel > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *Shahar > Salzman > *Sent:* Thursday, May 10, 2018 6:39 AM > *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > Hi Daniel, > > > > I updated the patch as you suggested. > > How do you want to approach the gcc flag and the modules that spdk does > not need? > > If you are building dpdk from an external makefile (as we are doing), all > of the above can be added to the makefile with CONFIG...=n and > EXTRA_CFLAGS="-fno-strict-aliasing", or I can send you another dpdk patch > fixing this within dpdk, so there is no change in the builder, and maybe > there is another option. > > > > WDYT? > > > > Shahar > > > ------------------------------ > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > paul.e.luse(a)intel.com&gt; > *Sent:* Wednesday, May 9, 2018 8:19:33 PM > *To:* Storage Performance Development Kit > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > That’s great, thanks Daniel! > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > *On Behalf Of *Verkamp, Daniel > *Sent:* Wednesday, May 9, 2018 9:43 AM > *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > I agree with Paul’s review; it looks like it shouldn’t be a big problem to > carry a patch for this in our DPDK submodule, at least while CentOS 6 is > still officially supported. (I posted a few review comments on the patch, > but the general approach looks fine.) > > > > The Intel build pool had a CentOS 6 test machine in the pool until > recently, and we should be able to resurrect it to provide test coverage > again. > > > > Thanks, > > -- Daniel > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > *On Behalf Of *Shahar Salzman > *Sent:* Wednesday, May 9, 2018 6:12 AM > *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > Hi Paul, > > > > Thanks for the review. > > We are now working on CentOS7 support, but it is not complete yet, so we > will be on CentOS6 about 6-12 months. > > > > Shahar > ------------------------------ > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > paul.e.luse(a)intel.com&gt; > *Sent:* Wednesday, May 9, 2018 3:51:23 PM > *To:* Storage Performance Development Kit > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > Thanks for doing that! I put my 2 cents on the review, one more data point > for the maintainers might help – help long do you believe your team will be > on CentOS6? Don’t need a specific date but are we talking months, years, > indefinitely? > > > > I think the other implication of accepting this in the community would be > to spin up a CentOS6 VM for some basic testing for as long as we hold that > patch in our DPDK fork which is not a huge thing but want to make sure > that’s clear as well. If nothing else we’ll know if something else breaks > it sooner than later and will need to decide, again, at that time if we > pull it or if it’s a high enough priority for anyone to dig in and find a > potential solution like you did here. Assuming, of course, that this one > is accepted J > > > > Thanks, > > Paul > > > > *From:* SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > *On Behalf Of *Shahar Salzman > *Sent:* Wednesday, May 9, 2018 2:01 AM > *To:* spdk(a)lists.01.org > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > Hi Ben, > > > > Pushed the fix to gerrithub - > https://review.gerrithub.io/#/c/spdk/dpdk/+/410559/ > > > > We are currently using a "frozen" dpdk tree based version 16.07, but I am > pushing to using the spdk fork and performing regular updates so if things > break, I will attempt to fix them, and post on the list. > > > > The above fixes a blunt gcc API issue, but there are more subtle inner > module issues which I have not investigated, simply removed the modules all > together as they are not needed by spdk. > > > > I can set up a howto for users who may need CentOS6 support or we can > modify the config (as these modules are not really needed for spdk). For > the sake of clarity here are the modules "configed out" (as I mentioned > they are not used by spdk): > > +CONFIG_RTE_LIBRTE_PMD_BOND=n > > +CONFIG_RTE_LIBRTE_CRYPTODEV=n > > +CONFIG_RTE_LIBRTE_SECURITY=n > > +CONFIG_RTE_LIBRTE_PMD_SW_EVENTDEV=n > > +CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF=n > > +CONFIG_RTE_LIBRTE_LPM=n > > +CONFIG_RTE_LIBRTE_FLOW_CLASSIFY=n > > +CONFIG_RTE_LIBRTE_TABLE=n > > +CONFIG_RTE_LIBRTE_PIPELINE=n > > +CONFIG_RTE_TEST_PMD=n > > +CONFIG_RTE_LIBRTE_PMD_TAP=n > > +CONFIG_RTE_LIBRTE_AVP_PMD=n > > > > Regarding the -fn-strict-aliasing issue, I'll see if I can do the same > trick I did for the gcc alias in the Makefile, and push the fix for review. > > > > Shahar > > > ------------------------------ > > *From:* SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Walker, Benjamin < > benjamin.walker(a)intel.com&gt; > *Sent:* Tuesday, May 8, 2018 7:49:44 PM > *To:* spdk(a)lists.01.org > *Subject:* Re: [SPDK] Building spdk on CentOS6 > > > > On Tue, 2018-05-08 at 14:34 +0000, Shahar Salzman wrote: > > Got it. > > > > I was tracking the dpdk project, and focused on the branch that Daniel > told me > > to work on. > > So would the correct process be to push the patch to the spdk/dpdk > gerrithub > > and discuss the commit, or the other way around? > > Yes - go ahead and push to spdk/dpdk on GerritHub so we can all review it. > The > SPDK project has never written out a policy on which versions of DPDK are > supported for any given SPDK release. The code is always tested using the > latest > released DPDK, and then attempts to support older versions of DPDK are > done in > an ad hoc manner. Similarly, the project never explicitly states which > operating > systems are supported. Instead, we indicate which operating systems the > code is > regularly tested on (Fedora N and N-1, Ubuntu latest, Ubuntu LTS, CentOS 7, > FreeBSD 11). > > I see that CentOS 6 is no longer receiving full updates and is in > maintenance > only mode. It will stop receiving even maintenance updates in 2020. I > think that > is indicative of the level of support SPDK should provide here. > > Given the above, if this patch can be applied to the DPDK fork and doesn't > have > any additional impact, I'm not opposed to applying it. That wouldn't, in my > mind, mean that SPDK is committed to supporting CentOS 6 indefinitely. But > certainly that support could continue while it is relatively easy and > convenient. > > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > paul.e.luse@ > <paul.e.luse@%0b>> intel.com> > > Sent: Tuesday, May 8, 2018 5:14:10 PM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > Sure, I understand that. SPDK maintains a fork of dpdk in github at > https://g > > ithub.com/spdk/dpdk for the purpose of temporarily putting things in > that we > > need that can’t make it upstream yet for whatever reason – this is the > > submodule in the SPDK repo. I’m not saying that it makes sense for your > patch > > to land there permanently but it does make sense to talk about it. > > > > No problem on not being on the call, we’ll get some input from the > maintainers > > and others I’m sure… > > > > Thx > > Paul > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > On Behalf Of Shahar Salzman > > Sent: Tuesday, May 8, 2018 7:04 AM > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > Hey Paul, > > > > The problem is that the patch is in dpdk and not in spdk, and this is the > > reason I offered to support it. > > The patch (bellow) is only a few lines long modifying the deprecated > attribute > > definition, a modification to the config, and an extra compilation flag. > > All of the above are required in order to build dpdk 18.01 on CentOS6, > there > > are no problems building spdk on CentOS6. > > > > I understood that you are already applying minor patches to dpdk, so > question > > is, whether this patch can be added? > > > > Unfortunately due to some personal issues, I am unable to participate in > > today's meeting. > > > > Thanks, > > Shahar > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Luse, Paul E < > paul.e.luse@ > <paul.e.luse@%0b>> intel.com> > > Sent: Tuesday, May 8, 2018 4:25:29 PM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > Hi Shahar, > > > > Couple of quick things: > > > > * thanks for putting all this time and effort into trying to make this > work! > > * in the community model, patches aren’t accepted based on one > individual’s > > ability/desire to support them so appreciate the offer but in general any > > patch that is accepted becomes the responsibly of the community (mostly > the > > maintainers) to support it long term. Otherwise we’d have what I like > to call > > “the flea market model” where there’s a bunch of separate individuals > > supporting their own things with very little cohesiveness across > everything J > > > > So, wrt next steps, if the patch isn’t a tremendous amount of effort I > would > > suggest you go ahead and submit it. That’s the best way to get everyone > on the > > same page wrt exactly what we’re talking about. If it is some > significant > > effort then, as Pawel states, you can call into a community meeting (see > email > > I sent out last night) or you can explain more details on this list. > > > > Thanks again!! > > Paul > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > On Behalf Of Wodkowski, PawelX > > Sent: Tuesday, May 8, 2018 5:57 AM > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > I think someone will answer you on this mailing list but there is > community > > meeting today too. You can raise this if you wish. > > > > Pawel > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > On Behalf Of Shahar Salzman > > Sent: Tuesday, May 8, 2018 2:23 PM > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > Hi, > > > > I attempted to push the gcc support patch to dpdk, and got rejected > since they > > do not wish to support CentOS6. > > Would it be possible to manually add the patch to the supported dpdk > version > > (currently 18.01). > > I would be happy to support CentOS6 issues. > > > > Thanks, > > Shahar > > > > From: Shahar Salzman > > Sent: Tuesday, April 24, 2018 3:40:51 PM > > To: Storage Performance Development Kit > > Cc: Ilan Steinberg; Ido Benda; Yael Shavit > > Subject: Re: Building spdk on CentOS6 > > > > OK, got everything to work. > > > > For DPDK, I had to to do the following: > > add a patch to make the new __rte_experimental attribute to work > > remove a bunch of modules via config which had some gnarly compilation > errors, > > and are not needed for spdk > > use -fno-strict-aliasing in the EXTRA_CFLAGS to avoid a ton of strict > aliasing > > errors. > > > > I tried spdk 18.01, and ran into some issues with the shared library > build. As > > I saw that Daniel had already dealt with them, I tried cherry picking, > but > > eventually used the master branch. > > I got these type of warnings: > > > > /usr/bin/ld: warning: librte_mbuf.so.3.1, needed by > > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- > > gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) > > /usr/bin/ld: warning: librte_ethdev.so.8.1, needed by > > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp- > > gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link) > > > > I added the following patch bellow to make them go away. > > > > I'll approach the dpdk community on the mailing list in order to get the > > __rte_experimental stuff to work. > > My final configuration is: > > CenrOS6.4 > > gcc 4.4.7-17 > > dpdk v18.02 > > spdk master > > > > What are the next steps? > > > > Shahar > > > > P.S. Here is the patch to remove the warnings above. > > commit d4ef744de1cc4928972b7042d1c90aff12db7867 > > Author: shahar salzman <shahar.salzman(a)kaminario.com&gt; > > Date: Tue Apr 24 14:31:52 2018 +0300 > > > > lib/env_dpdk: add required libraries > > > > diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk > > index 450043c..b46bfed 100644 > > --- a/lib/env_dpdk/env.mk > > +++ b/lib/env_dpdk/env.mk > > @@ -68,6 +68,18 @@ ifneq ($(wildcard > $(DPDK_ABS_DIR)/lib/librte_malloc.*),) > > DPDK_LIB_LIST += rte_malloc > > endif > > > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_mbuf.*)) > > +DPDK_LIB_LIST += rte_mbuf > > +endif > > + > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_net.*)) > > +DPDK_LIB_LIST += rte_net > > +endif > > + > > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_ethdev.*)) > > +DPDK_LIB_LIST += rte_ethdev > > +endif > > + > > # librte_pci and librte_bus_pci were added in DPDK 17.11. Link these > > libraries conditionally > > # based on their existence to maintain backward compatibility. > > ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_pci.*)) > > > > > > > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Shahar Salzman > <shahar.sal > > zman(a)kaminario.com&gt; > > Sent: Tuesday, April 24, 2018 9:57 AM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > Great, and thanks! > > > > I have been running into issues compiling dpdk 17.11 on CentOS6 (there > seems > > to be a gcc issue that does not exist in other versions), I'll focus my > > efforts on dpdk 18.02. > > From: SPDK <spdk-bounces(a)lists.01.org&gt; on behalf of Verkamp, Daniel < > daniel.ve > > rkamp(a)intel.com&gt; > > Sent: Monday, April 23, 2018 9:40:36 PM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] Building spdk on CentOS6 > > > > Hi Shahar, > > > > I’ve pushed a patch for the SPDK master branch that should allow it to > compile > > with DPDK 16.07: https://review.gerrithub.io/#/c/spdk/spdk/+/408743/ > > > > Gerrit Code Review > > > > review.gerrithub.io > > > > Keep in touch. Copyright © 2017 | GerritForge Ltd. info(a)gerritforge.com > > www.gerritforge.com > > > > > > However, DPDK 16.07 is no longer supported upstream; I highly recommend > that > > you update to a newer version. SPDK is currently tested with DPDK 18.02. > > > > The VFIO issue you mention has also been fixed on SPDK master, and it > will be > > part of the SPDK v18.04 release. > > > > Thanks, > > -- Daniel > > > > From: SPDK [mailto:spdk-bounces@lists.01.org < spdk-bounces(a)lists.01.org>] > On Behalf Of Shahar Salzman > > Sent: Sunday, April 22, 2018 6:41 AM > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > Subject: [SPDK] Building spdk on CentOS6 > > > > Hi, > > > > Finally got to looking at support of spdk build on CentOS6, things look > good, > > except for one issue. > > spdk is latest 18.01.x version, dpdk is 16.07 (+3 dpdk patches to allow > > compilation) and some minor patches (mainly some memory configuration > stuff), > > kernel is a patched 4.9.6. > > > > build succeeded except for the usage of the dpdk function > pci_vfio_is_enabled. > > I had to apply the patch bellow, removing the usage of this function and > then > > compilation completed without any issues. > > > > It seems that I am missing some sort of dpdk configuration as I see that > the > > function is built, but not packaged into the generated archive. > > > > I went back to square one and ran the instructions in > http://www.spdk.io/doc/g > > etting_started.html, but I see no mention of dpdk there. Actually the > > ./configure requires it. > > > > My next step is to use a more recent dpdk, but shouldn't this work with > my > > version? Am I missing some dpdk configuration? > > BTW, as we are not using vhost, on our 17.07 version we simply use > > CONFIG_VHOST=n in order to skip this, but I would be happier if we used a > > better solution. > > > > Shahar > > > > P.S. Here is the patch to remove use of this function: > > diff --git a/lib/env_dpdk/vtophys.c b/lib/env_dpdk/vtophys.c > > index 92aa256..f38929f 100644 > > --- a/lib/env_dpdk/vtophys.c > > +++ b/lib/env_dpdk/vtophys.c > > @@ -53,8 +53,10 @@ > > #define SPDK_VFIO_ENABLED 1 > > #include <linux/vfio.h> > > > > +#if 0 > > /* Internal DPDK function forward declaration */ > > int pci_vfio_is_enabled(void); > > +#endif > > > > struct spdk_vfio_dma_map { > > struct vfio_iommu_type1_dma_map map; > > @@ -341,9 +343,11 @@ spdk_vtophys_iommu_init(void) > > DIR *dir; > > struct dirent *d; > > > > +#if 0 > > if (!pci_vfio_is_enabled()) { > > return; > > } > > +#endif > > > > dir = opendir("/proc/self/fd"); > > if (!dir) { > > > > > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk