FYI there are other format issues as well, if you look at some of the other test machines you’ll see this, for example, on the patch: Checking coding style... OK Checking comment style... OK Checking blank lines at end of file... Incorrect end-of-file formatting detected py-spdk/README.md: Extra trailing newline Checking for POSIX includes... OK Checking Python style... OK All very easy to catch & fix locally J Thx Paul *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *Zhipeng Huang *Sent:* Wednesday, January 17, 2018 6:33 PM *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; *Subject:* Re: [SPDK] Add py-spdk client for SPDK Hi Paul, Thanks for getting the CI running. Seems most of the problems caused by pep8 missing. Is it possible to add python support in the CI system ? otherwise we will hit many pip pkg missing problems. On Thu, Jan 18, 2018 at 8:17 AM, Luse, Paul E <paul.e.luse(a)intel.com&gt; wrote: Hmm yeah I’m not sure why it ran on patch set 2 but not the last couple… Seth, anyone? We might need one of the maintainers to clear it for CI execution, it’s an extra unfortunate security requirement we have on the test pool, otherwise its fully automated Thx Paul *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *Zhipeng Huang *Sent:* Wednesday, January 17, 2018 4:04 PM *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; *Subject:* Re: [SPDK] Add py-spdk client for SPDK Paul, the problem is that Wewe's later patches doesn't trigger CI at all. Is it how it supposed to work that we should fix the problem before hand ? Since in OpenStack or kubernetes CI should be triggered everytime the patches come in. On Thu, Jan 18, 2018 at 5:39 AM, Luse, Paul E <paul.e.luse(a)intel.com&gt; wrote: Hi WeWe So wrt which repo, I didn’t see that but we don’t really have a precedent for subprojects like that. I’ll let one of the maintainers comment on the options but I believe they would be (a) maintain on your own elsewhere or (b) include in SPDK like any other code. Option (b) is really for “core” SPDK elements that make sense for most use cases and a python binding may or may not fit, it clearly isn’t out of the question in my mind though as it’s not directly tied to OpenStack. On your patch not passing, see the screenshot below where I highlighted the -1 from the CI system. If you click on that link it will take you to the results and you can see the failure I mention. Have you tried running the check_format.sh locally? Thx Paul *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *We We *Sent:* Wednesday, January 17, 2018 10:26 AM *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; *Subject:* Re: [SPDK] Add py-spdk client for SPDK Hi, Paul I am sorry. Is the question on my patch you mentioned the following Q1? Q1: is your intention to propose this as part of the SPDK repo or do you intent to maintain this in your own repo and are simply asking for review comments? If so, I see it and response to you on the patch. Maybe you didn’t see it. 1. About Q1, my idea is to treat the py-spdk as another subrepo in the SPDK community. For example: SPDK repo -->https://github.com/spdk/spdk.git, py-spdk repo -->https://github.com/spdk/py-spdk.git. Am I reasonable？ 2. About a copy of rpc.py. Thank you for your careful review, the py-spdk doesn’t need to modify and copy the files of SPDK, I have deleted the rpc.py and updated the new patch. Please visit it. 3. About my patch didn’t pass CI. My patch didn’t show nothing including the “verified -1”. I didn’t know whether the CI executed, so I can’t track anything down. Is there anyone who met this problem? Regards, Helloway 在 2018年1月17日，下午10:32，Luse, Paul E <paul.e.luse(a)intel.com&gt; 写道： Hi WeWe, Thanks for the trello update, did you see my question on your patch? I went to look specifically at the changes to existing files to start with and it appears that your patch includes a copy of rpc.py basically duplicating it within the repo. Did I miss something or is that what you intended? Wrt question on why your patch didn’t pass CI, it’s pretty easy to track these things down most of the time. In the review, click on the link below where it says verified -1 and you will see a webpage with the status of each of the test machines. Many of the are red and indicate failure. Click on one and drill down to find the builg.log file and you’ll see some formatting issues: ========== Backtrace start: ========== in ./autobuild.sh:23 -> main() ... 18 19 ./configure $config_params 20 21 timing_enter check_format 22 if [ $SPDK_RUN_CHECK_FORMAT -eq 1 ]; then => 23 ./scripts/check_format.sh 24 fi 25 timing_exit check_format 26 27 timing_enter build_kmod 28 if [ $SPDK_BUILD_IOAT_KMOD -eq 1 ]; then ... ========== Backtrace end ========== You can run the script, ./scripts/check_format.sh, locally and correct the errors and resubmit. We always ask that developers run this scripts in advance to make sure things are formatted. Depending on whether the maintainers think the SPDK repo makes sense for this patch you’ll need some test code also – there is clearly a lot of value in this patch though and regardless of whether it lands here or is maintained elsewhere! Use this list or the patch review if you have more questions on getting it to pass. Thanks! Paul *From:* SPDK [mailto:spdk-bounces@lists.01.org <spdk-bounces(a)lists.01.org&gt; ] *On Behalf Of *We We *Sent:* Wednesday, January 17, 2018 5:24 AM *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; *Subject:* Re: [SPDK] Add py-spdk client for SPDK Hi, all I have submitted the introduction of py-spdk on trello https://trello.com/c/Bn3DIG7F/87-add-py-spdk-client-for-spdk, please visit it. I will be grateful to your kindness. Regards, Helloway 在 2018年1月16日，上午11:30，We We <simple_hlw(a)163.com&gt; 写道： Hi, all I have submitted the py-spdk(written in python) for more than six days. Why the code-review couldn’t pass and show nothing? Is there anyone who can help me? Thanks, Helloway 在 2018年1月10日，下午9:46，We We <simple_hlw(a)163.com&gt; 写道： Hi, all I have submitted the py-spdk code onhttps://review.gerrithub.io/ #/c/379741/, please take some time to visit it, I will be very grateful to you. The py-spdk is client which can help the upper-level app to communicate with the SPDK-based app (such as: nvmf_tgt, vhost, iscsi_tgt, etc.). Should I submit it into the other repo I rebuild rather than SPDK repo? Because I think it is a relatively independent kit upon the SPDK. If you have some thoughts about the py-spdk, please share with me. Regards, Helloway _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk -- Zhipeng (Howard) Huang Standard Engineer IT Standard & Patent/IT Product Line Huawei Technologies Co,. Ltd Email: huangzhipeng(a)huawei.com Office: Huawei Industrial Base, Longgang, Shenzhen (Previous) Research Assistant Mobile Ad-Hoc Network Lab, Calit2 University of California, Irvine Email: zhipengh(a)uci.edu Office: Calit2 Building Room 2402 OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk -- Zhipeng (Howard) Huang Standard Engineer IT Standard & Patent/IT Product Line Huawei Technologies Co,. Ltd Email: huangzhipeng(a)huawei.com Office: Huawei Industrial Base, Longgang, Shenzhen (Previous) Research Assistant Mobile Ad-Hoc Network Lab, Calit2 University of California, Irvine Email: zhipengh(a)uci.edu Office: Calit2 Building Room 2402 OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

在 2018年1月18日，下午10:22，Szwed, Maciej <maciej.szwed(a)intel.com <mailto:maciej.szwed@intel.com>> 写道： Hello WeWe, I’ve looked closer at your patch and it looks good (some minor tweaks maybe, some error handling missing), but before I comment it on Gerrit we have make decision where we want to keep it. I would rather see it outside spdk/spdk repo. There is some work ongoing to connect SPDK with Cinder. I think that these two could end up in same repository. Regards, Maciek   <> <>From: SPDK [mailto:spdk-bounces@lists.01.org <mailto:spdk-bounces@lists.01.org>] On Behalf Of Zhipeng Huang Sent: Thursday, January 18, 2018 3:01 AM To: Storage Performance Development Kit <spdk(a)lists.01.org <mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Add py-spdk client for SPDK Thanks and Wewe is working on to fix that in the next patch :) As she stated earlier, we think it is better to have a standalone repo hosting all the different lang bindings for spdk under https://github.com/spdk <https://github.com/spdk> , similar example could be found at openstack sdk, so that we don't blow up the main spdk repo On Thu, Jan 18, 2018 at 9:49 AM, Luse, Paul E <paul.e.luse(a)intel.com <mailto:paul.e.luse@intel.com>> wrote: FYI there are other format issues as well, if you look at some of the other test machines you’ll see this, for example, on the patch: Checking coding style... OK Checking comment style... OK Checking blank lines at end of file... Incorrect end-of-file formatting detected py-spdk/README.md: Extra trailing newline Checking for POSIX includes... OK Checking Python style... OK All very easy to catch & fix locally J Thx Paul   <> <>From: SPDK [mailto:spdk-bounces@lists.01.org <mailto:spdk-bounces@lists.01.org>] On Behalf Of Zhipeng Huang Sent: Wednesday, January 17, 2018 6:33 PM To: Storage Performance Development Kit <spdk(a)lists.01.org <mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Add py-spdk client for SPDK Hi Paul, Thanks for getting the CI running. Seems most of the problems caused by pep8 missing. Is it possible to add python support in the CI system ? otherwise we will hit many pip pkg missing problems. On Thu, Jan 18, 2018 at 8:17 AM, Luse, Paul E <paul.e.luse(a)intel.com <mailto:paul.e.luse@intel.com>> wrote: Hmm yeah I’m not sure why it ran on patch set 2 but not the last couple… Seth, anyone? We might need one of the maintainers to clear it for CI execution, it’s an extra unfortunate security requirement we have on the test pool, otherwise its fully automated Thx Paul From: SPDK [mailto:spdk-bounces@lists.01.org <mailto:spdk-bounces@lists.01.org>] On Behalf Of Zhipeng Huang Sent: Wednesday, January 17, 2018 4:04 PM To: Storage Performance Development Kit <spdk(a)lists.01.org <mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Add py-spdk client for SPDK Paul, the problem is that Wewe's later patches doesn't trigger CI at all. Is it how it supposed to work that we should fix the problem before hand ? Since in OpenStack or kubernetes CI should be triggered everytime the patches come in. On Thu, Jan 18, 2018 at 5:39 AM, Luse, Paul E <paul.e.luse(a)intel.com <mailto:paul.e.luse@intel.com>> wrote: Hi WeWe So wrt which repo, I didn’t see that but we don’t really have a precedent for subprojects like that. I’ll let one of the maintainers comment on the options but I believe they would be (a) maintain on your own elsewhere or (b) include in SPDK like any other code. Option (b) is really for “core” SPDK elements that make sense for most use cases and a python binding may or may not fit, it clearly isn’t out of the question in my mind though as it’s not directly tied to OpenStack. On your patch not passing, see the screenshot below where I highlighted the -1 from the CI system. If you click on that link it will take you to the results and you can see the failure I mention. Have you tried running the check_format.sh locally? Thx Paul <image001.png> <>From: SPDK [mailto:spdk-bounces@lists.01.org <mailto:spdk-bounces@lists.01.org>] On Behalf Of We We Sent: Wednesday, January 17, 2018 10:26 AM To: Storage Performance Development Kit <spdk(a)lists.01.org <mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Add py-spdk client for SPDK Hi, Paul I am sorry. Is the question on my patch you mentioned the following Q1? Q1: is your intention to propose this as part of the SPDK repo or do you intent to maintain this in your own repo and are simply asking for review comments? If so, I see it and response to you on the patch. Maybe you didn’t see it. 1. About Q1, my idea is to treat the py-spdk as another subrepo in the SPDK community. For example: SPDK repo -->https://github.com/spdk/spdk.git <https://github.com/spdk/spdk.git>;, py-spdk repo -->https://github.com/spdk/py-spdk.git <https://github.com/spdk/py-spdk.git>;. Am I reasonable？ 2. About a copy of rpc.py. Thank you for your careful review, the py-spdk doesn’t need to modify and copy the files of SPDK, I have deleted the rpc.py and updated the new patch. Please visit it. 3. About my patch didn’t pass CI. My patch didn’t show nothing including the “verified -1”. I didn’t know whether the CI executed, so I can’t track anything down. Is there anyone who met this problem? Regards, Helloway 在 2018年1月17日，下午10:32，Luse, Paul E <paul.e.luse(a)intel.com <mailto:paul.e.luse@intel.com>> 写道： Hi WeWe, Thanks for the trello update, did you see my question on your patch? I went to look specifically at the changes to existing files to start with and it appears that your patch includes a copy of rpc.py basically duplicating it within the repo. Did I miss something or is that what you intended? Wrt question on why your patch didn’t pass CI, it’s pretty easy to track these things down most of the time. In the review, click on the link below where it says verified -1 and you will see a webpage with the status of each of the test machines. Many of the are red and indicate failure. Click on one and drill down to find the builg.log file and you’ll see some formatting issues: ========== Backtrace start: ========== in ./autobuild.sh:23 -> main() ... 18 19 ./configure $config_params 20 21 timing_enter check_format 22 if [ $SPDK_RUN_CHECK_FORMAT -eq 1 ]; then => 23 ./scripts/check_format.sh 24 fi 25 timing_exit check_format 26 27 timing_enter build_kmod 28 if [ $SPDK_BUILD_IOAT_KMOD -eq 1 ]; then ... ========== Backtrace end ========== You can run the script, ./scripts/check_format.sh, locally and correct the errors and resubmit. We always ask that developers run this scripts in advance to make sure things are formatted. Depending on whether the maintainers think the SPDK repo makes sense for this patch you’ll need some test code also – there is clearly a lot of value in this patch though and regardless of whether it lands here or is maintained elsewhere! Use this list or the patch review if you have more questions on getting it to pass. Thanks! Paul From: SPDK [mailto:spdk-bounces@lists.01.org <mailto:spdk-bounces@lists.01.org>] On Behalf Of We We Sent: Wednesday, January 17, 2018 5:24 AM To: Storage Performance Development Kit <spdk(a)lists.01.org <mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Add py-spdk client for SPDK Hi, all I have submitted the introduction of py-spdk on trello https://trello.com/c/Bn3DIG7F/87-add-py-spdk-client-for-spdk <https://trello.com/c/Bn3DIG7F/87-add-py-spdk-client-for-spdk>;, please visit it. I will be grateful to your kindness. Regards, Helloway 在 2018年1月16日，上午11:30，We We <simple_hlw(a)163.com <mailto:simple_hlw@163.com>> 写道： Hi, all I have submitted the py-spdk(written in python) for more than six days. Why the code-review couldn’t pass and show nothing? Is there anyone who can help me? Thanks, Helloway 在 2018年1月10日，下午9:46，We We <simple_hlw(a)163.com <mailto:simple_hlw@163.com>> 写道： Hi, all I have submitted the py-spdk code onhttps://review.gerrithub.io/#/c/379741/ <https://review.gerrithub.io/#/c/379741/>;, please take some time to visit it, I will be very grateful to you. The py-spdk is client which can help the upper-level app to communicate with the SPDK-based app (such as: nvmf_tgt, vhost, iscsi_tgt, etc.). Should I submit it into the other repo I rebuild rather than SPDK repo? Because I think it is a relatively independent kit upon the SPDK. If you have some thoughts about the py-spdk, please share with me. Regards, Helloway _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk> _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk> _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk> -- Zhipeng (Howard) Huang Standard Engineer IT Standard & Patent/IT Product Line Huawei Technologies Co,. Ltd Email: huangzhipeng(a)huawei.com <mailto:huangzhipeng@huawei.com> Office: Huawei Industrial Base, Longgang, Shenzhen (Previous) Research Assistant Mobile Ad-Hoc Network Lab, Calit2 University of California, Irvine Email: zhipengh(a)uci.edu <mailto:zhipengh@uci.edu> Office: Calit2 Building Room 2402 OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk> -- Zhipeng (Howard) Huang Standard Engineer IT Standard & Patent/IT Product Line Huawei Technologies Co,. Ltd Email: huangzhipeng(a)huawei.com <mailto:huangzhipeng@huawei.com> Office: Huawei Industrial Base, Longgang, Shenzhen (Previous) Research Assistant Mobile Ad-Hoc Network Lab, Calit2 University of California, Irvine Email: zhipengh(a)uci.edu <mailto:zhipengh@uci.edu> Office: Calit2 Building Room 2402 OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk> -- Zhipeng (Howard) Huang Standard Engineer IT Standard & Patent/IT Product Line Huawei Technologies Co,. Ltd Email: huangzhipeng(a)huawei.com <mailto:huangzhipeng@huawei.com> Office: Huawei Industrial Base, Longgang, Shenzhen (Previous) Research Assistant Mobile Ad-Hoc Network Lab, Calit2 University of California, Irvine Email: zhipengh(a)uci.edu <mailto:zhipengh@uci.edu> Office: Calit2 Building Room 2402 OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk>

Hi Ben, You are absolutely right on the first point. On the second point, the main reason for us is that protobuf could auto generate the data model for any interraces. Let's say we want to write a go tool, we could just write the API similar to py-spdk, and the protobuf will auto generate the msg bodies that will be exchanged over the API. Protobuf is purely from an easy to use perspective, if you think it adds too much complexity then we could just use plain JSON RPC :) On Jan 19, 2018 12:58 AM, "Walker, Benjamin" <benjamin.walker(a)intel.com&gt; wrote: > I'm still not at all clear on either point, so please allow me to ask a > few more quesrions. > > These python scripts interact with the SPDK applications through JSON > RPC, not through direct calls, correct? If that is the case, let's call > this a management tool or management library and not use the term binding > from here on out. I think that will go a long way toward helping everyone > understand the role this code is playing. > > As far as using protobuf, most languages have facilities to send json > rpcs natively. This is part of the standard library in go, for example. > Protobufs are great for generating high performance parsers of binary > protocols in multiple languages, but for a management interface I think > JSON RPC is a better choice. It is plain text and much more flexible. Is > there any reason you can't just send the JSON RPCs from your go application > directly? Surely this can't be any harder than sending the protobuf > messages. > > I hope I'm understanding. > > Thanks, > Ben > > > -------- Original message -------- > From: Zhipeng Huang <zhipengh512(a)gmail.com&gt; > Date: 1/18/18 8:06 AM (GMT-08:00) > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > Subject: Re: [SPDK] Add py-spdk client for SPDK > > Hi Ben, > > I could help hellowway answer a few questions. > > First of all, for py-spdk we are shooting for a SDK framework for SPDK so > that if someone develop a userspace storage app based upon SPDK, he could > quickly develop the SDK for his app based upon py-spdk. You are right that > typically when we say binding we mean a direct integration, however for > py-spdk we also want it to be a client that could remotely interact with > the management plane. So in essence py-spdk could be used as a python > binding for SPDK as well as a simple python client. > > Second on the use of protobuf, is that we see possibilities of having > different lang bindings for SPDK. Using protobuf is a great way to define a > common data model and reuse it in these different lang bindings. > > For example we have an open source storage mgmt project called OpenSDS > that might use SPDK for NVMe Over Fabric JBOFs. OpenSDS is written in > golang therefore we could quickly build a go-spdk based upon the protobuf > data model. > > Hope I make things clearer :) > > On Thu, Jan 18, 2018 at 11:49 PM, Walker, Benjamin < > benjamin.walker(a)intel.com&gt; wrote: > >> Hi Helloway, >> >> I'll admit up front that I haven't read through your code yet, but I >> have read your trello document. I have a couple of very basic questions. >> >> First, usually when people say "binding" they mean a way to call a >> library from a different language. A python binding to SPDK would mean a >> wrapper around the C libraries so that they could be called from python >> directly, with no interprocess communication. My understanding of what >> you're writing is that it is not a binding, but rather a management tool >> for the SPDK example target applications. Am I understanding this correctly? >> >> Second, SPDK exposes JSON RPC interfaces for management of the target >> applications. These RPCs can be sent from any language and we already have >> some python scripts that do just that. So why do you need protobufs? What >> new value does your code add above what we have in scripts/rpc.py already? >> Is it just more fully featured? >> >> Thanks in advance for helping me understand. >> >> >> -------- Original message -------- >> From: We We <simple_hlw(a)163.com&gt; >> Date: 1/18/18 8:12 AM (GMT-07:00) >> To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; >> Subject: Re: [SPDK] Add py-spdk client for SPDK >> >> Hi Maciek, >> >> Thanks!! I am very happy to receive your kind feedback. >> In my opinion, I indeed want to put the py-spdk into the spdk/py-spdk >> repo rathan than spdk/spdk repo. Because I think the py-spdk is built upon >> the SPDK which can provide a general python binding for spdk framework. I >> don't know whether I keep consistent with you. Please let me know what you >> think. And is it convenient for you to tell me what you have done with SPDK >> and Cinder? >> Any thoughts are appreciated. >> >> Regards, >> Helloway >> >> >> >> 在 2018年1月18日，下午10:22，Szwed, Maciej <maciej.szwed(a)intel.com&gt; 写道： >> >> Hello WeWe, >> I’ve looked closer at your patch and it looks good (some minor tweaks >> maybe, some error handling missing), but before I comment it on Gerrit we >> have make decision where we want to keep it. I would rather see it outside >> spdk/spdk repo. There is some work ongoing to connect SPDK with Cinder. I >> think that these two could end up in same repository. >> >> Regards, >> Maciek >> >> *From:* SPDK [mailto:spdk-bounces@lists.01.org >> <spdk-bounces(a)lists.01.org>] *On Behalf Of *Zhipeng Huang >> *Sent:* Thursday, January 18, 2018 3:01 AM >> *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; >> *Subject:* Re: [SPDK] Add py-spdk client for SPDK >> >> Thanks and Wewe is working on to fix that in the next patch :) >> >> As she stated earlier, we think it is better to have a standalone repo >> hosting all the different lang bindings for spdk under >> https://github.com/spdk , similar example could be found at openstack >> sdk, so that we don't blow up the main spdk repo >> >> On Thu, Jan 18, 2018 at 9:49 AM, Luse, Paul E <paul.e.luse(a)intel.com&gt; >> wrote: >> >> FYI there are other format issues as well, if you look at some of the >> other test machines you’ll see this, for example, on the patch: >> >> Checking coding style... OK >> Checking comment style... OK >> Checking blank lines at end of file... Incorrect end-of-file formatting >> detected >> py-spdk/README.md: Extra trailing newline >> Checking for POSIX includes... OK >> Checking Python style... OK >> >> >> All very easy to catch & fix locally J >> >> Thx >> Paul >> >> *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *Zhipeng >> Huang >> *Sent:* Wednesday, January 17, 2018 6:33 PM >> >> *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; >> *Subject:* Re: [SPDK] Add py-spdk client for SPDK >> >> Hi Paul, >> >> Thanks for getting the CI running. Seems most of the problems caused by >> pep8 missing. Is it possible to add python support in the CI system ? >> otherwise we will hit many pip pkg missing problems. >> >> On Thu, Jan 18, 2018 at 8:17 AM, Luse, Paul E <paul.e.luse(a)intel.com&gt; >> wrote: >> >> Hmm yeah I’m not sure why it ran on patch set 2 but not the last couple… >> Seth, anyone? >> >> We might need one of the maintainers to clear it for CI execution, it’s >> an extra unfortunate security requirement we have on the test pool, >> otherwise its fully automated >> >> Thx >> Paul >> >> *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *Zhipeng >> Huang >> *Sent:* Wednesday, January 17, 2018 4:04 PM >> >> *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; >> *Subject:* Re: [SPDK] Add py-spdk client for SPDK >> >> Paul, the problem is that Wewe's later patches doesn't trigger CI at >> all. Is it how it supposed to work that we should fix the problem before >> hand ? >> Since in OpenStack or kubernetes CI should be triggered everytime the >> patches come in. >> >> On Thu, Jan 18, 2018 at 5:39 AM, Luse, Paul E <paul.e.luse(a)intel.com&gt; >> wrote: >> >> Hi WeWe >> >> >> So wrt which repo, I didn’t see that but we don’t really have a >> precedent for subprojects like that. I’ll let one of the maintainers >> comment on the options but I believe they would be (a) maintain on your own >> elsewhere or (b) include in SPDK like any other code. Option (b) is really >> for “core” SPDK elements that make sense for most use cases and a python >> binding may or may not fit, it clearly isn’t out of the question in my mind >> though as it’s not directly tied to OpenStack. >> >> On your patch not passing, see the screenshot below where I highlighted >> the -1 from the CI system. If you click on that link it will take you to >> the results and you can see the failure I mention. Have you tried running >> the check_format.sh locally? >> >> Thx >> Paul >> >> >> <image001.png> >> >> *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *We We >> *Sent:* Wednesday, January 17, 2018 10:26 AM >> >> *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; >> *Subject:* Re: [SPDK] Add py-spdk client for SPDK >> >> Hi, Paul >> I am sorry. Is the question on my patch you mentioned the following >> Q1? >> Q1: is your intention to propose this as part of the SPDK repo or do you >> intent to maintain this in your own repo and are simply asking for review >> comments? >> If so, I see it and response to you on the patch. Maybe you didn’t see >> it. >> >> 1. About Q1, my idea is to treat the py-spdk as another subrepo in the >> SPDK community. For example: SPDK repo -->https://github.com/spdk/spd >> k.git, py-spdk repo -->https://github.com/spdk/py-spdk.git. Am I >> reasonable？ >> >> >> 2. About a copy of rpc.py. Thank you for your careful review, the >> py-spdk doesn’t need to modify and copy the files of SPDK, I have deleted >> the rpc.py and updated the new patch. Please visit it. >> >> >> 3. About my patch didn’t pass CI. My patch didn’t show nothing >> including the “verified -1”. I didn’t know whether the CI executed, so >> I can’t track anything down. Is there anyone who met this problem? >> >> Regards, >> Helloway >> >> >> >> >> 在 2018年1月17日，下午10:32，Luse, Paul E <paul.e.luse(a)intel.com&gt; 写道： >> >> Hi WeWe, >> >> Thanks for the trello update, did you see my question on your patch? I >> went to look specifically at the changes to existing files to start with >> and it appears that your patch includes a copy of rpc.py basically >> duplicating it within the repo. Did I miss something or is that what you >> intended? >> >> Wrt question on why your patch didn’t pass CI, it’s pretty easy to >> track these things down most of the time. In the review, click on the link >> below where it says verified -1 and you will see a webpage with the status >> of each of the test machines. Many of the are red and indicate failure. >> Click on one and drill down to find the builg.log file and you’ll see some >> formatting issues: >> >> ========== Backtrace start: ========== >> >> in ./autobuild.sh:23 -> main() >> ... >> 18 >> 19 ./configure $config_params >> 20 >> 21 timing_enter check_format >> 22 if [ $SPDK_RUN_CHECK_FORMAT -eq 1 ]; then >> => 23 ./scripts/check_format.sh >> 24 fi >> 25 timing_exit check_format >> 26 >> 27 timing_enter build_kmod >> 28 if [ $SPDK_BUILD_IOAT_KMOD -eq 1 ]; then >> ... >> >> ========== Backtrace end ========== >> >> You can run the script, ./scripts/check_format.sh, locally and correct >> the errors and resubmit. We always ask that developers run this scripts in >> advance to make sure things are formatted. Depending on whether the >> maintainers think the SPDK repo makes sense for this patch you’ll need some >> test code also – there is clearly a lot of value in this patch though and >> regardless of whether it lands here or is maintained elsewhere! >> >> Use this list or the patch review if you have more questions on getting >> it to pass. >> >> Thanks! >> Paul >> >> >> >> *From:* SPDK [mailto:spdk-bounces@lists.01.org >> <spdk-bounces(a)lists.01.org>] *On Behalf Of *We We >> *Sent:* Wednesday, January 17, 2018 5:24 AM >> *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; >> *Subject:* Re: [SPDK] Add py-spdk client for SPDK >> >> Hi, all >> I have submitted the introduction of py-spdk on trello >> https://trello.com/c/Bn3DIG7F/87-add-py-spdk-client-for-spdk, please >> visit it. I will be grateful to your kindness. >> >> Regards, >> >> Helloway >> >> >> 在 2018年1月16日，上午11:30，We We <simple_hlw(a)163.com&gt; 写道： >> >> Hi, all >> I have submitted the py-spdk(written in python) for more than six days. >> Why the code-review couldn’t pass and show nothing? >> Is there anyone who can help me? >> >> Thanks, >> Helloway >> >> >> 在 2018年1月10日，下午9:46，We We <simple_hlw(a)163.com&gt; 写道： >> >> Hi, all >> I have submitted the py-spdk code onhttps://review.gerrithub.io/ >> #/c/379741/, please take some time to visit it, I will be very grateful >> to you. >> The py-spdk is client which can help the upper-level app to communicate >> with the SPDK-based app (such as: nvmf_tgt, vhost, iscsi_tgt, etc.). Should >> I submit it into the other repo I rebuild rather than SPDK repo? Because I >> think it is a relatively independent kit upon the SPDK. >> >> If you have some thoughts about the py-spdk, please share with me. >> >> Regards, >> Helloway >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> >> >> >> >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> >> >> >> >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> >> >> >> >> -- >> Zhipeng (Howard) Huang >> >> Standard Engineer >> IT Standard & Patent/IT Product Line >> Huawei Technologies Co,. Ltd >> Email: huangzhipeng(a)huawei.com >> Office: Huawei Industrial Base, Longgang, Shenzhen >> >> (Previous) >> Research Assistant >> Mobile Ad-Hoc Network Lab, Calit2 >> University of California, Irvine >> Email: zhipengh(a)uci.edu >> Office: Calit2 Building Room 2402 >> >> OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado >> >> >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> >> >> >> >> -- >> Zhipeng (Howard) Huang >> >> Standard Engineer >> IT Standard & Patent/IT Product Line >> Huawei Technologies Co,. Ltd >> Email: huangzhipeng(a)huawei.com >> Office: Huawei Industrial Base, Longgang, Shenzhen >> >> (Previous) >> Research Assistant >> Mobile Ad-Hoc Network Lab, Calit2 >> University of California, Irvine >> Email: zhipengh(a)uci.edu >> Office: Calit2 Building Room 2402 >> >> OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado >> >> >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> >> >> >> >> -- >> Zhipeng (Howard) Huang >> >> Standard Engineer >> IT Standard & Patent/IT Product Line >> Huawei Technologies Co,. Ltd >> Email: huangzhipeng(a)huawei.com >> Office: Huawei Industrial Base, Longgang, Shenzhen >> >> (Previous) >> Research Assistant >> Mobile Ad-Hoc Network Lab, Calit2 >> University of California, Irvine >> Email: zhipengh(a)uci.edu >> Office: Calit2 Building Room 2402 >> >> OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> >> >> >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> >> > > > -- > Zhipeng (Howard) Huang > > Standard Engineer > IT Standard & Patent/IT Product Line > Huawei Technologies Co,. Ltd > Email: huangzhipeng(a)huawei.com > Office: Huawei Industrial Base, Longgang, Shenzhen > > (Previous) > Research Assistant > Mobile Ad-Hoc Network Lab, Calit2 > University of California, Irvine > Email: zhipengh(a)uci.edu > Office: Calit2 Building Room 2402 > > OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > >

wrote:

On Wed, 2018-01-24 at 07:13 +0800, Zhipeng Huang wrote: > Do we have a conclusion on this issue ? If it is ok to have a spdk/sdk repo, > then wewe will modify the current patch (get rid of protobuff) and resubmit > the patch to the new repo once it is established (meanwhile abandon the > current one to spdk/spdk). If you remove protobuf, can you describe what is left? Recently scripts/rpc.py was refactored to break it up into a set of Python libraries in scripts/rpc, plus the command line tool at scripts/rpc.py. What functionality does this new code provide over and above what is already present there? SPDK is certainly in need of better management tools, so in the most general sense the community is very supportive of your effort here. New management tools can also go directly into the main spdk repository (a separate repository was only suggested when we thought this was a Python binding to the SPDK libraries). I'm wondering if an easier way forward would be to continue refining the Python packages in scripts/rpc to be more general purpose libraries for sending the JSON RPCs. What are your thoughts on that? _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

在 2018年1月28日，上午5:25，Luse, Paul E <paul.e.luse(a)intel.com&gt; 写道： Hi Howard,   <> I think the main question right now is about the new core functionality in the patch versus what Ben mentioned wrt what is there today in scripts/rpc: __init__.py client.py lvol.py nvmf.py app.py iscsi.py nbd.py pmem.py bdev.py log.py net.py vhost.py This is the change he mentioned that happened since the original py-spdk client was added. So for example if I look at what is in the patch in py-spdk/nvmf_client.py I don’t see any capabilities that can’t be done in thescripts/rpc/nvmf.py but I could be missing something. I think the protobuf and repo location discussions are secondary to first establishing the value of the core functionality of the patch to the community. Can you explain how the SDK Framework layer might be a better approach as compared to working on the files listed above? Thanks Paul PS: There is an open community conference call every other week, I think the next one is coming up here soon and its usually announced on IRC a few days in advance. If real time discussion would help we could always look at scheduling a separate open meeting at time more friendly to your time zone. <>From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Zhipeng Huang Sent: Saturday, January 27, 2018 12:11 AM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] Add py-spdk client for SPDK Hi Ben, After discussing with my team, the general feeling is that keeping protobuf would be a preferred choice. The reason, as i stated earlier, is that proto provides a better standardized description of the API than JSON and also we could create data model rather quickly for bindings/tools written in other languages. We have already generated the go-spdk based upon the proto model for OpenSDS's interaction with SPDK drivers. It provides benefits at least judging from our own practice. Frankly given gRPC's wide adoption I don't think this should be a big issue. If you still have doubt about this, I think maybe we could setup a conf call, or I could discuss with Harris when he's in China for SPDK summit. For repo, I think it is entirely up to the community's decision on whether maintain it in the main repo or creating a new one. On Sat, Jan 27, 2018 at 6:28 AM, Walker, Benjamin <benjamin.walker(a)intel.com <mailto:benjamin.walker@intel.com>> wrote: On Wed, 2018-01-24 at 07:13 +0800, Zhipeng Huang wrote: > Do we have a conclusion on this issue ? If it is ok to have a spdk/sdk repo, > then wewe will modify the current patch (get rid of protobuff) and resubmit > the patch to the new repo once it is established (meanwhile abandon the > current one to spdk/spdk). If you remove protobuf, can you describe what is left? Recently scripts/rpc.py was refactored to break it up into a set of Python libraries in scripts/rpc, plus the command line tool at scripts/rpc.py. What functionality does this new code provide over and above what is already present there? SPDK is certainly in need of better management tools, so in the most general sense the community is very supportive of your effort here. New management tools can also go directly into the main spdk repository (a separate repository was only suggested when we thought this was a Python binding to the SPDK libraries). I'm wondering if an easier way forward would be to continue refining the Python packages in scripts/rpc to be more general purpose libraries for sending the JSON RPCs. What are your thoughts on that? _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk> -- Zhipeng (Howard) Huang Standard Engineer IT Standard & Patent/IT Product Line Huawei Technologies Co,. Ltd Email: huangzhipeng(a)huawei.com <mailto:huangzhipeng@huawei.com> Office: Huawei Industrial Base, Longgang, Shenzhen (Previous) Research Assistant Mobile Ad-Hoc Network Lab, Calit2 University of California, Irvine Email: zhipengh(a)uci.edu <mailto:zhipengh@uci.edu> Office: Calit2 Building Room 2402 OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

在 2018年2月6日，上午3:20，Luse, Paul E <paul.e.luse(a)intel.com&gt; 写道： Hi WeWe,   <> Thanks for the additional insight. On the py-spdk client yes I see the value that you are describing however instead of adding on the py-spdk client a better approach might be take the scripts/rpc.py and scripts/rpc components and build those out/refactor them to provide more/better functionality. There’s enough “likeness” between what you are proposing and what’s already there that I’m afraid a lot of the reluctance to adopt this is based on the increased maintenance of adding this new code weighed against the increased value. If you improve on what’s there one piece at a time I think you can get a lot accomplished here. Also, you should be aware that there’s some pending work to consolidate the SPDK target applications themselves, to have one target application with parameters instead of multiple binaries that do very similar work. I can see a good fit for your goals and that activity to work together to have a single python generic interface as well as a single binary underneath it both architected at the same time. Would you be interested in working with some other community folks on that? I’m not sure who it was that was talking about looking into that but hopefully they’ll see this and chime in J I think there might be a community meeting this Thu, if someone knows please reply – we could talk on that call if that would help. Thanks Paul PS: Or, you could work independently to get a generic python layer usable by applications “built into” the existing rpc/scripts code. A good place to start would be to pick one sample app, like nvmf target, and list out the functions that are currently provided via the existing rpc py code and see how you can add the ones that are in your patch by extending/re-organizing/refactor existing code. I would definitely start with just one application though and get buy-in from a maintainer that this is the right design choice. Then, the rest of the work will go quite smoothly I think. <>From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of We We Sent: Monday, February 5, 2018 2:21 AM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] Add py-spdk client for SPDK Hi Paul, I am trying to answer your questions from two points. Firstly, the py-spdk/nvmf_client.py provides the same abilities that can be done in the /scripts/rpc/nvmf.py, the difference, however, is that they are in the upper and lower layer respectively. Why should we do it? Imagine that there is only one management app which needs to invoke SPDK-base apps, we can write the module alone in your own management app and directly communicate with the directly /scripts/rpc/nvmf.py. However, when there are multiple upper management apps that need to invoke SPDK-base apps, we need to provide the unified software named ‘py-spdk’ and laid between the SPDK-based apps and upper management apps. If we don’t do so, then each management app needs to write the same module to interact with /scripts/rpc/nvmf.py. The py-spdk provides a generic interface for upper management apps. In other words, the upper management apps are not care about the functional realization of the backend, they just work through the interfaces provided by the py-spdk. Secondly, why did we use the protobuf as data model? Because the protobuf provides a better standardized description of the API than JSON. From the code point of view, we only need to define a ‘.proto’file that describes the data and compiled it with different languages to the lib file (such as: spdk_pb2.py, spdk.go, etc.). Each ‘message’of the ‘.proto’file will be compiled to a protobuf obj which served as the carrier of the data and returned to the upper management apps. Compared to the JSON, the upper management apps can quickly learn that the returned value which is passed through the lib file rather than the manual document. From the ecological point of view，many mainstream cloud-native apps are using protobuf as their data model (like k8s)，which is also an opportunity for us. I don't know do I make sense. Regards, Helloway 在 2018年1月28日，上午5:25，Luse, Paul E <paul.e.luse(a)intel.com <mailto:paul.e.luse@intel.com>> 写道： Hi Howard, I think the main question right now is about the new core functionality in the patch versus what Ben mentioned wrt what is there today in scripts/rpc: __init__.py client.py lvol.py nvmf.py app.py iscsi.py nbd.py pmem.py bdev.py log.py net.py vhost.py This is the change he mentioned that happened since the original py-spdk client was added. So for example if I look at what is in the patch in py-spdk/nvmf_client.py I don’t see any capabilities that can’t be done in thescripts/rpc/nvmf.py but I could be missing something. I think the protobuf and repo location discussions are secondary to first establishing the value of the core functionality of the patch to the community. Can you explain how the SDK Framework layer might be a better approach as compared to working on the files listed above? Thanks Paul PS: There is an open community conference call every other week, I think the next one is coming up here soon and its usually announced on IRC a few days in advance. If real time discussion would help we could always look at scheduling a separate open meeting at time more friendly to your time zone. From: SPDK [mailto:spdk-bounces@lists.01.org <mailto:spdk-bounces@lists.01.org>] On Behalf Of Zhipeng Huang Sent: Saturday, January 27, 2018 12:11 AM To: Storage Performance Development Kit <spdk(a)lists.01.org <mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Add py-spdk client for SPDK Hi Ben, After discussing with my team, the general feeling is that keeping protobuf would be a preferred choice. The reason, as i stated earlier, is that proto provides a better standardized description of the API than JSON and also we could create data model rather quickly for bindings/tools written in other languages. We have already generated the go-spdk based upon the proto model for OpenSDS's interaction with SPDK drivers. It provides benefits at least judging from our own practice. Frankly given gRPC's wide adoption I don't think this should be a big issue. If you still have doubt about this, I think maybe we could setup a conf call, or I could discuss with Harris when he's in China for SPDK summit. For repo, I think it is entirely up to the community's decision on whether maintain it in the main repo or creating a new one. On Sat, Jan 27, 2018 at 6:28 AM, Walker, Benjamin <benjamin.walker(a)intel.com <mailto:benjamin.walker@intel.com>> wrote: On Wed, 2018-01-24 at 07:13 +0800, Zhipeng Huang wrote: > Do we have a conclusion on this issue ? If it is ok to have a spdk/sdk repo, > then wewe will modify the current patch (get rid of protobuff) and resubmit > the patch to the new repo once it is established (meanwhile abandon the > current one to spdk/spdk). If you remove protobuf, can you describe what is left? Recently scripts/rpc.py was refactored to break it up into a set of Python libraries in scripts/rpc, plus the command line tool at scripts/rpc.py. What functionality does this new code provide over and above what is already present there? SPDK is certainly in need of better management tools, so in the most general sense the community is very supportive of your effort here. New management tools can also go directly into the main spdk repository (a separate repository was only suggested when we thought this was a Python binding to the SPDK libraries). I'm wondering if an easier way forward would be to continue refining the Python packages in scripts/rpc to be more general purpose libraries for sending the JSON RPCs. What are your thoughts on that? _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk> -- Zhipeng (Howard) Huang Standard Engineer IT Standard & Patent/IT Product Line Huawei Technologies Co,. Ltd Email: huangzhipeng(a)huawei.com <mailto:huangzhipeng@huawei.com> Office: Huawei Industrial Base, Longgang, Shenzhen (Previous) Research Assistant Mobile Ad-Hoc Network Lab, Calit2 University of California, Irvine Email: zhipengh(a)uci.edu <mailto:zhipengh@uci.edu> Office: Calit2 Building Room 2402 OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk> _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

在 2018年2月6日，下午12:07，Luse, Paul E <paul.e.luse(a)intel.com&gt; 写道： That’s great Helloway. I would strongly suggest one thing at a time though; first tackle the front end interfaces and then once that’s completed then start seeing where people’s heads are at wrt things like protobuf. That’s pretty common in most open source communities and definitely is here – smaller patches and smaller projects are always better. Easier to understand, easier to review and easier to land! Keep your eyes open on the dist list for the meeting time, look forward to talking to you finally J -Paul   <> <>From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of We We Sent: Monday, February 5, 2018 8:35 PM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] Add py-spdk client for SPDK Hi Paul, Thank you for your reply and advice. First of all I whole-heartly agree with you that it is a better approach to take the scripts/rpc.py and scripts/rpc components and build those out/refactor them to provide more/better functionality. Secondly, I think you really hit on the target and the very essence of the benefit that a consistent data model protobuf could help bring. We are definitely in line on this one. I would like to help to lead the effort which has been discussed thoroughly in this email thread, and work with any contributor in the SPDK community who is interested and willing to participate :) Also look forward to my first community meeting this Thursday ! Thanks, Helloway 在 2018年2月6日，上午3:20，Luse, Paul E <paul.e.luse(a)intel.com <mailto:paul.e.luse@intel.com>> 写道： Hi WeWe, Thanks for the additional insight. On the py-spdk client yes I see the value that you are describing however instead of adding on the py-spdk client a better approach might be take the scripts/rpc.py and scripts/rpc components and build those out/refactor them to provide more/better functionality. There’s enough “likeness” between what you are proposing and what’s already there that I’m afraid a lot of the reluctance to adopt this is based on the increased maintenance of adding this new code weighed against the increased value. If you improve on what’s there one piece at a time I think you can get a lot accomplished here. Also, you should be aware that there’s some pending work to consolidate the SPDK target applications themselves, to have one target application with parameters instead of multiple binaries that do very similar work. I can see a good fit for your goals and that activity to work together to have a single python generic interface as well as a single binary underneath it both architected at the same time. Would you be interested in working with some other community folks on that? I’m not sure who it was that was talking about looking into that but hopefully they’ll see this and chime in J I think there might be a community meeting this Thu, if someone knows please reply – we could talk on that call if that would help. Thanks Paul PS: Or, you could work independently to get a generic python layer usable by applications “built into” the existing rpc/scripts code. A good place to start would be to pick one sample app, like nvmf target, and list out the functions that are currently provided via the existing rpc py code and see how you can add the ones that are in your patch by extending/re-organizing/refactor existing code. I would definitely start with just one application though and get buy-in from a maintainer that this is the right design choice. Then, the rest of the work will go quite smoothly I think. From: SPDK [mailto:spdk-bounces@lists.01.org <mailto:spdk-bounces@lists.01.org>] On Behalf Of We We Sent: Monday, February 5, 2018 2:21 AM To: Storage Performance Development Kit <spdk(a)lists.01.org <mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Add py-spdk client for SPDK Hi Paul, I am trying to answer your questions from two points. Firstly, the py-spdk/nvmf_client.py provides the same abilities that can be done in the /scripts/rpc/nvmf.py, the difference, however, is that they are in the upper and lower layer respectively. Why should we do it? Imagine that there is only one management app which needs to invoke SPDK-base apps, we can write the module alone in your own management app and directly communicate with the directly /scripts/rpc/nvmf.py. However, when there are multiple upper management apps that need to invoke SPDK-base apps, we need to provide the unified software named ‘py-spdk’ and laid between the SPDK-based apps and upper management apps. If we don’t do so, then each management app needs to write the same module to interact with /scripts/rpc/nvmf.py. The py-spdk provides a generic interface for upper management apps. In other words, the upper management apps are not care about the functional realization of the backend, they just work through the interfaces provided by the py-spdk. Secondly, why did we use the protobuf as data model? Because the protobuf provides a better standardized description of the API than JSON. From the code point of view, we only need to define a ‘.proto’file that describes the data and compiled it with different languages to the lib file (such as: spdk_pb2.py, spdk.go, etc.). Each ‘message’of the ‘.proto’file will be compiled to a protobuf obj which served as the carrier of the data and returned to the upper management apps. Compared to the JSON, the upper management apps can quickly learn that the returned value which is passed through the lib file rather than the manual document. From the ecological point of view，many mainstream cloud-native apps are using protobuf as their data model (like k8s)，which is also an opportunity for us. I don't know do I make sense. Regards, Helloway 在 2018年1月28日，上午5:25，Luse, Paul E <paul.e.luse(a)intel.com <mailto:paul.e.luse@intel.com>> 写道： Hi Howard, I think the main question right now is about the new core functionality in the patch versus what Ben mentioned wrt what is there today in scripts/rpc: __init__.py client.py lvol.py nvmf.py app.py iscsi.py nbd.py pmem.py bdev.py log.py net.py vhost.py This is the change he mentioned that happened since the original py-spdk client was added. So for example if I look at what is in the patch in py-spdk/nvmf_client.py I don’t see any capabilities that can’t be done in thescripts/rpc/nvmf.py but I could be missing something. I think the protobuf and repo location discussions are secondary to first establishing the value of the core functionality of the patch to the community. Can you explain how the SDK Framework layer might be a better approach as compared to working on the files listed above? Thanks Paul PS: There is an open community conference call every other week, I think the next one is coming up here soon and its usually announced on IRC a few days in advance. If real time discussion would help we could always look at scheduling a separate open meeting at time more friendly to your time zone. From: SPDK [mailto:spdk-bounces@lists.01.org <mailto:spdk-bounces@lists.01.org>] On Behalf Of Zhipeng Huang Sent: Saturday, January 27, 2018 12:11 AM To: Storage Performance Development Kit <spdk(a)lists.01.org <mailto:spdk@lists.01.org>> Subject: Re: [SPDK] Add py-spdk client for SPDK Hi Ben, After discussing with my team, the general feeling is that keeping protobuf would be a preferred choice. The reason, as i stated earlier, is that proto provides a better standardized description of the API than JSON and also we could create data model rather quickly for bindings/tools written in other languages. We have already generated the go-spdk based upon the proto model for OpenSDS's interaction with SPDK drivers. It provides benefits at least judging from our own practice. Frankly given gRPC's wide adoption I don't think this should be a big issue. If you still have doubt about this, I think maybe we could setup a conf call, or I could discuss with Harris when he's in China for SPDK summit. For repo, I think it is entirely up to the community's decision on whether maintain it in the main repo or creating a new one. On Sat, Jan 27, 2018 at 6:28 AM, Walker, Benjamin <benjamin.walker(a)intel.com <mailto:benjamin.walker@intel.com>> wrote: On Wed, 2018-01-24 at 07:13 +0800, Zhipeng Huang wrote: > Do we have a conclusion on this issue ? If it is ok to have a spdk/sdk repo, > then wewe will modify the current patch (get rid of protobuff) and resubmit > the patch to the new repo once it is established (meanwhile abandon the > current one to spdk/spdk). If you remove protobuf, can you describe what is left? Recently scripts/rpc.py was refactored to break it up into a set of Python libraries in scripts/rpc, plus the command line tool at scripts/rpc.py. What functionality does this new code provide over and above what is already present there? SPDK is certainly in need of better management tools, so in the most general sense the community is very supportive of your effort here. New management tools can also go directly into the main spdk repository (a separate repository was only suggested when we thought this was a Python binding to the SPDK libraries). I'm wondering if an easier way forward would be to continue refining the Python packages in scripts/rpc to be more general purpose libraries for sending the JSON RPCs. What are your thoughts on that? _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk> -- Zhipeng (Howard) Huang Standard Engineer IT Standard & Patent/IT Product Line Huawei Technologies Co,. Ltd Email: huangzhipeng(a)huawei.com <mailto:huangzhipeng@huawei.com> Office: Huawei Industrial Base, Longgang, Shenzhen (Previous) Research Assistant Mobile Ad-Hoc Network Lab, Calit2 University of California, Irvine Email: zhipengh(a)uci.edu <mailto:zhipengh@uci.edu> Office: Calit2 Building Room 2402 OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk> _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk> _______________________________________________ SPDK mailing list SPDK(a)lists.01.org <mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk>

Do we have a conclusion on this issue ? If it is ok to have a spdk/sdk repo, then wewe will modify the current patch (get rid of protobuff) and resubmit the patch to the new repo once it is established (meanwhile abandon the current one to spdk/spdk).