Hi guys,
Do we have a conclusion on this issue ? If it is ok to have a spdk/sdk
repo, then wewe will modify the current patch (get rid of protobuff) and
resubmit the patch to the new repo once it is established (meanwhile
abandon the current one to spdk/spdk).
On Fri, Jan 19, 2018 at 6:57 AM, Zhipeng Huang <zhipengh512(a)gmail.com>
wrote:
Hi Ben,
You are absolutely right on the first point.
On the second point, the main reason for us is that protobuf could auto
generate the data model for any interraces. Let's say we want to write a go
tool, we could just write the API similar to py-spdk, and the protobuf will
auto generate the msg bodies that will be exchanged over the API.
Protobuf is purely from an easy to use perspective, if you think it adds
too much complexity then we could just use plain JSON RPC :)
On Jan 19, 2018 12:58 AM, "Walker, Benjamin" <benjamin.walker(a)intel.com>
wrote:
> I'm still not at all clear on either point, so please allow me to ask a
> few more quesrions.
>
> These python scripts interact with the SPDK applications through JSON
> RPC, not through direct calls, correct? If that is the case, let's call
> this a management tool or management library and not use the term binding
> from here on out. I think that will go a long way toward helping everyone
> understand the role this code is playing.
>
> As far as using protobuf, most languages have facilities to send json
> rpcs natively. This is part of the standard library in go, for example.
> Protobufs are great for generating high performance parsers of binary
> protocols in multiple languages, but for a management interface I think
> JSON RPC is a better choice. It is plain text and much more flexible. Is
> there any reason you can't just send the JSON RPCs from your go application
> directly? Surely this can't be any harder than sending the protobuf
> messages.
>
> I hope I'm understanding.
>
> Thanks,
> Ben
>
>
> -------- Original message --------
> From: Zhipeng Huang <zhipengh512(a)gmail.com>
> Date: 1/18/18 8:06 AM (GMT-08:00)
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Add py-spdk client for SPDK
>
> Hi Ben,
>
> I could help hellowway answer a few questions.
>
> First of all, for py-spdk we are shooting for a SDK framework for SPDK so
> that if someone develop a userspace storage app based upon SPDK, he could
> quickly develop the SDK for his app based upon py-spdk. You are right that
> typically when we say binding we mean a direct integration, however for
> py-spdk we also want it to be a client that could remotely interact with
> the management plane. So in essence py-spdk could be used as a python
> binding for SPDK as well as a simple python client.
>
> Second on the use of protobuf, is that we see possibilities of having
> different lang bindings for SPDK. Using protobuf is a great way to define a
> common data model and reuse it in these different lang bindings.
>
> For example we have an open source storage mgmt project called OpenSDS
> that might use SPDK for NVMe Over Fabric JBOFs. OpenSDS is written in
> golang therefore we could quickly build a go-spdk based upon the protobuf
> data model.
>
> Hope I make things clearer :)
>
> On Thu, Jan 18, 2018 at 11:49 PM, Walker, Benjamin <
> benjamin.walker(a)intel.com> wrote:
>
>> Hi Helloway,
>>
>> I'll admit up front that I haven't read through your code yet, but I
>> have read your trello document. I have a couple of very basic questions.
>>
>> First, usually when people say "binding" they mean a way to call a
>> library from a different language. A python binding to SPDK would mean a
>> wrapper around the C libraries so that they could be called from python
>> directly, with no interprocess communication. My understanding of what
>> you're writing is that it is not a binding, but rather a management tool
>> for the SPDK example target applications. Am I understanding this correctly?
>>
>> Second, SPDK exposes JSON RPC interfaces for management of the target
>> applications. These RPCs can be sent from any language and we already have
>> some python scripts that do just that. So why do you need protobufs? What
>> new value does your code add above what we have in scripts/rpc.py already?
>> Is it just more fully featured?
>>
>> Thanks in advance for helping me understand.
>>
>>
>> -------- Original message --------
>> From: We We <simple_hlw(a)163.com>
>> Date: 1/18/18 8:12 AM (GMT-07:00)
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>
>> Subject: Re: [SPDK] Add py-spdk client for SPDK
>>
>> Hi Maciek,
>>
>> Thanks!! I am very happy to receive your kind feedback.
>> In my opinion, I indeed want to put the py-spdk into the spdk/py-spdk
>> repo rathan than spdk/spdk repo. Because I think the py-spdk is built upon
>> the SPDK which can provide a general python binding for spdk framework. I
>> don't know whether I keep consistent with you. Please let me know what you
>> think. And is it convenient for you to tell me what you have done with SPDK
>> and Cinder?
>> Any thoughts are appreciated.
>>
>> Regards,
>> Helloway
>>
>>
>>
>> 在 2018年1月18日,下午10:22,Szwed, Maciej <maciej.szwed(a)intel.com> 写道:
>>
>> Hello WeWe,
>> I’ve looked closer at your patch and it looks good (some minor tweaks
>> maybe, some error handling missing), but before I comment it on Gerrit we
>> have make decision where we want to keep it. I would rather see it outside
>> spdk/spdk repo. There is some work ongoing to connect SPDK with Cinder. I
>> think that these two could end up in same repository.
>>
>> Regards,
>> Maciek
>>
>> *From:* SPDK [mailto:spdk-bounces@lists.01.org
>> <spdk-bounces(a)lists.01.org>] *On Behalf Of *Zhipeng Huang
>> *Sent:* Thursday, January 18, 2018 3:01 AM
>> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
>> *Subject:* Re: [SPDK] Add py-spdk client for SPDK
>>
>> Thanks and Wewe is working on to fix that in the next patch :)
>>
>> As she stated earlier, we think it is better to have a standalone repo
>> hosting all the different lang bindings for spdk under
>>
https://github.com/spdk , similar example could be found at openstack
>> sdk, so that we don't blow up the main spdk repo
>>
>> On Thu, Jan 18, 2018 at 9:49 AM, Luse, Paul E <paul.e.luse(a)intel.com>
>> wrote:
>>
>> FYI there are other format issues as well, if you look at some of the
>> other test machines you’ll see this, for example, on the patch:
>>
>> Checking coding style... OK
>> Checking comment style... OK
>> Checking blank lines at end of file... Incorrect end-of-file formatting
>> detected
>> py-spdk/README.md: Extra trailing newline
>> Checking for POSIX includes... OK
>> Checking Python style... OK
>>
>>
>> All very easy to catch & fix locally J
>>
>> Thx
>> Paul
>>
>> *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *Zhipeng
>> Huang
>> *Sent:* Wednesday, January 17, 2018 6:33 PM
>>
>> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
>> *Subject:* Re: [SPDK] Add py-spdk client for SPDK
>>
>> Hi Paul,
>>
>> Thanks for getting the CI running. Seems most of the problems caused by
>> pep8 missing. Is it possible to add python support in the CI system ?
>> otherwise we will hit many pip pkg missing problems.
>>
>> On Thu, Jan 18, 2018 at 8:17 AM, Luse, Paul E <paul.e.luse(a)intel.com>
>> wrote:
>>
>> Hmm yeah I’m not sure why it ran on patch set 2 but not the last couple…
>> Seth, anyone?
>>
>> We might need one of the maintainers to clear it for CI execution, it’s
>> an extra unfortunate security requirement we have on the test pool,
>> otherwise its fully automated
>>
>> Thx
>> Paul
>>
>> *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *Zhipeng
>> Huang
>> *Sent:* Wednesday, January 17, 2018 4:04 PM
>>
>> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
>> *Subject:* Re: [SPDK] Add py-spdk client for SPDK
>>
>> Paul, the problem is that Wewe's later patches doesn't trigger CI at
>> all. Is it how it supposed to work that we should fix the problem before
>> hand ?
>> Since in OpenStack or kubernetes CI should be triggered everytime the
>> patches come in.
>>
>> On Thu, Jan 18, 2018 at 5:39 AM, Luse, Paul E <paul.e.luse(a)intel.com>
>> wrote:
>>
>> Hi WeWe
>>
>>
>> So wrt which repo, I didn’t see that but we don’t really have a
>> precedent for subprojects like that. I’ll let one of the maintainers
>> comment on the options but I believe they would be (a) maintain on your own
>> elsewhere or (b) include in SPDK like any other code. Option (b) is really
>> for “core” SPDK elements that make sense for most use cases and a python
>> binding may or may not fit, it clearly isn’t out of the question in my mind
>> though as it’s not directly tied to OpenStack.
>>
>> On your patch not passing, see the screenshot below where I highlighted
>> the -1 from the CI system. If you click on that link it will take you to
>> the results and you can see the failure I mention. Have you tried running
>> the check_format.sh locally?
>>
>> Thx
>> Paul
>>
>>
>> <image001.png>
>>
>> *From:* SPDK [mailto:spdk-bounces@lists.01.org] *On Behalf Of *We We
>> *Sent:* Wednesday, January 17, 2018 10:26 AM
>>
>> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
>> *Subject:* Re: [SPDK] Add py-spdk client for SPDK
>>
>> Hi, Paul
>> I am sorry. Is the question on my patch you mentioned the following
>> Q1?
>> Q1: is your intention to propose this as part of the SPDK repo or do you
>> intent to maintain this in your own repo and are simply asking for review
>> comments?
>> If so, I see it and response to you on the patch. Maybe you didn’t see
>> it.
>>
>> 1. About Q1, my idea is to treat the py-spdk as another subrepo in the
>> SPDK community. For example: SPDK repo -->https://github.com/spdk/spd
>> k.git, py-spdk repo -->https://github.com/spdk/py-spdk.git. Am I
>> reasonable?
>>
>>
>> 2. About a copy of rpc.py. Thank you for your careful review, the
>> py-spdk doesn’t need to modify and copy the files of SPDK, I have deleted
>> the rpc.py and updated the new patch. Please visit it.
>>
>>
>> 3. About my patch didn’t pass CI. My patch didn’t show nothing
>> including the “verified -1”. I didn’t know whether the CI executed, so
>> I can’t track anything down. Is there anyone who met this problem?
>>
>> Regards,
>> Helloway
>>
>>
>>
>>
>> 在 2018年1月17日,下午10:32,Luse, Paul E <paul.e.luse(a)intel.com> 写道:
>>
>> Hi WeWe,
>>
>> Thanks for the trello update, did you see my question on your patch? I
>> went to look specifically at the changes to existing files to start with
>> and it appears that your patch includes a copy of rpc.py basically
>> duplicating it within the repo. Did I miss something or is that what you
>> intended?
>>
>> Wrt question on why your patch didn’t pass CI, it’s pretty easy to
>> track these things down most of the time. In the review, click on the link
>> below where it says verified -1 and you will see a webpage with the status
>> of each of the test machines. Many of the are red and indicate failure.
>> Click on one and drill down to find the builg.log file and you’ll see some
>> formatting issues:
>>
>> ========== Backtrace start: ==========
>>
>> in ./autobuild.sh:23 -> main()
>> ...
>> 18
>> 19 ./configure $config_params
>> 20
>> 21 timing_enter check_format
>> 22 if [ $SPDK_RUN_CHECK_FORMAT -eq 1 ]; then
>> => 23 ./scripts/check_format.sh
>> 24 fi
>> 25 timing_exit check_format
>> 26
>> 27 timing_enter build_kmod
>> 28 if [ $SPDK_BUILD_IOAT_KMOD -eq 1 ]; then
>> ...
>>
>> ========== Backtrace end ==========
>>
>> You can run the script, ./scripts/check_format.sh, locally and correct
>> the errors and resubmit. We always ask that developers run this scripts in
>> advance to make sure things are formatted. Depending on whether the
>> maintainers think the SPDK repo makes sense for this patch you’ll need some
>> test code also – there is clearly a lot of value in this patch though and
>> regardless of whether it lands here or is maintained elsewhere!
>>
>> Use this list or the patch review if you have more questions on getting
>> it to pass.
>>
>> Thanks!
>> Paul
>>
>>
>>
>> *From:* SPDK [mailto:spdk-bounces@lists.01.org
>> <spdk-bounces(a)lists.01.org>] *On Behalf Of *We We
>> *Sent:* Wednesday, January 17, 2018 5:24 AM
>> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
>> *Subject:* Re: [SPDK] Add py-spdk client for SPDK
>>
>> Hi, all
>> I have submitted the introduction of py-spdk on trello
>>
https://trello.com/c/Bn3DIG7F/87-add-py-spdk-client-for-spdk, please
>> visit it. I will be grateful to your kindness.
>>
>> Regards,
>>
>> Helloway
>>
>>
>> 在 2018年1月16日,上午11:30,We We <simple_hlw(a)163.com> 写道:
>>
>> Hi, all
>> I have submitted the py-spdk(written in python) for more than six days.
>> Why the code-review couldn’t pass and show nothing?
>> Is there anyone who can help me?
>>
>> Thanks,
>> Helloway
>>
>>
>> 在 2018年1月10日,下午9:46,We We <simple_hlw(a)163.com> 写道:
>>
>> Hi, all
>> I have submitted the py-spdk code onhttps://review.gerrithub.io/
>> #/c/379741/, please take some time to visit it, I will be very grateful
>> to you.
>> The py-spdk is client which can help the upper-level app to communicate
>> with the SPDK-based app (such as: nvmf_tgt, vhost, iscsi_tgt, etc.). Should
>> I submit it into the other repo I rebuild rather than SPDK repo? Because I
>> think it is a relatively independent kit upon the SPDK.
>>
>> If you have some thoughts about the py-spdk, please share with me.
>>
>> Regards,
>> Helloway
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>>
https://lists.01.org/mailman/listinfo/spdk
>>
>>
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>>
https://lists.01.org/mailman/listinfo/spdk
>>
>>
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>>
https://lists.01.org/mailman/listinfo/spdk
>>
>>
>>
>>
>> --
>> Zhipeng (Howard) Huang
>>
>> Standard Engineer
>> IT Standard & Patent/IT Product Line
>> Huawei Technologies Co,. Ltd
>> Email: huangzhipeng(a)huawei.com
>> Office: Huawei Industrial Base, Longgang, Shenzhen
>>
>> (Previous)
>> Research Assistant
>> Mobile Ad-Hoc Network Lab, Calit2
>> University of California, Irvine
>> Email: zhipengh(a)uci.edu
>> Office: Calit2 Building Room 2402
>>
>> OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>>
https://lists.01.org/mailman/listinfo/spdk
>>
>>
>>
>>
>> --
>> Zhipeng (Howard) Huang
>>
>> Standard Engineer
>> IT Standard & Patent/IT Product Line
>> Huawei Technologies Co,. Ltd
>> Email: huangzhipeng(a)huawei.com
>> Office: Huawei Industrial Base, Longgang, Shenzhen
>>
>> (Previous)
>> Research Assistant
>> Mobile Ad-Hoc Network Lab, Calit2
>> University of California, Irvine
>> Email: zhipengh(a)uci.edu
>> Office: Calit2 Building Room 2402
>>
>> OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>>
https://lists.01.org/mailman/listinfo/spdk
>>
>>
>>
>>
>> --
>> Zhipeng (Howard) Huang
>>
>> Standard Engineer
>> IT Standard & Patent/IT Product Line
>> Huawei Technologies Co,. Ltd
>> Email: huangzhipeng(a)huawei.com
>> Office: Huawei Industrial Base, Longgang, Shenzhen
>>
>> (Previous)
>> Research Assistant
>> Mobile Ad-Hoc Network Lab, Calit2
>> University of California, Irvine
>> Email: zhipengh(a)uci.edu
>> Office: Calit2 Building Room 2402
>>
>> OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>>
https://lists.01.org/mailman/listinfo/spdk
>>
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>>
https://lists.01.org/mailman/listinfo/spdk
>>
>>
>
>
> --
> Zhipeng (Howard) Huang
>
> Standard Engineer
> IT Standard & Patent/IT Product Line
> Huawei Technologies Co,. Ltd
> Email: huangzhipeng(a)huawei.com
> Office: Huawei Industrial Base, Longgang, Shenzhen
>
> (Previous)
> Research Assistant
> Mobile Ad-Hoc Network Lab, Calit2
> University of California, Irvine
> Email: zhipengh(a)uci.edu
> Office: Calit2 Building Room 2402
>
> OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
>
https://lists.01.org/mailman/listinfo/spdk
>
>
--
Zhipeng (Howard) Huang
Standard Engineer
IT Standard & Patent/IT Product Line
Huawei Technologies Co,. Ltd
Email: huangzhipeng(a)huawei.com
Office: Huawei Industrial Base, Longgang, Shenzhen
(Previous)
Research Assistant
Mobile Ad-Hoc Network Lab, Calit2
University of California, Irvine
Email: zhipengh(a)uci.edu
Office: Calit2 Building Room 2402
OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado