6:55 p.m.
Hi all,
There has been a lot of activity lately dealing with abstracting SPDK's dependencies
to allow porting SPDK to additional platforms and frameworks. I think we'll end up
talking about this extensively at the summit this week. The ability to use SPDK in a
variety of environments is very important to us, so I look forward to the discussions. I
wanted to write a basic summary of the current state of SPDK's dependencies for
everyone to have, and so that those who are unable to attend the summit can provide
feedback. I'll try to update this thread with the results of the discussions at the
summit for everyone's benefit.
We've already eliminated all SPDK dependencies except for two:
1) POSIX
2) DPDK
SPDK already has a mechanism for abstracting away the "environment" it is
running in. We do this by moving all calls to dependencies to the "env" library,
whose interface is defined by include/spdk/env.h. The build system allows users to point
to an alternate implementation of the env library if required. SPDK's default
implementation of env is based on DPDK and runs in Linux and FreeBSD user space.
DPDK is a very large toolkit, of course, but until just recently SPDK only used DPDK's
'eal' module (environment abstraction library). The original goal of the SPDK env
library was just to abstract all uses of DPDK's eal, but not all uses of POSIX APIs.
We made some progress in this area, but there are a number of remaining calls as of this
writing. Fortunately, the wider community has really come through. There are at least two
pull requests out from two independent users that mostly finish the job. We'll work
through those pull requests and try to get them merged shortly. The only part of the code
that will continue to explicitly depend on DPDK will be the newly released vhost-scsi
target because that relies on DPDK vhost infrastructure. This infrastructure is outside of
DPDK's eal and is quite extensive. We feel this is acceptable because building vhost
in SPDK is optional - you can just turn it off to eliminate the dependency.
There are a number of legitimate reasons to want to remove the DPDK dependency from SPDK.
For instance, DPDK dictates a particular memory management model, requires a specific
threading model, and performs PCI device management that may conflict with other
libraries. However, DPDK provides tremendous value to SPDK in a number of areas,
especially by implementing memory allocation suitable for DMA in user space. Memory
management is a major challenge with all sorts of hidden traps and challenges, so we see
significant value in using a production-tested library here. Re-implement the env without
DPDK at your own risk!
Abstracting away our POSIX dependency is a similar but larger challenge. While we foresaw
the need for people to replace DPDK, we didn't see any use case for dropping POSIX. We
were definitely wrong though - people are using SPDK in all sorts of environments that we
didn't predict, from embedded firmware to operating system kernels and beyond. So
we'll need to address this as time goes on. I think this will result in further
expansion of the 'env' library. Fortunately, we don't require too much from
POSIX - mostly pthreads, the C standard library, and maybe a couple of other miscellaneous
headers.
There are other less obvious dependencies in SPDK. For instance, it also depends on a
relatively recent version of gcc or clang. This is mostly a practical concern and not so
much by intention. Our continuous integration system (which I'll talk about at the
summit - exciting developments here) currently consists of 4 machines, 3 of which are
running Fedora Linux 25 while the other is running FreeBSD 11.0. These are very much up to
date systems, so we tend not to catch bugs on older systems today. We also use cutting
edge C features and tools, so we'll probably over time need to select a particular
minimum version of the C standard that we're going to develop to.
I'm also concerned with the scope of the env library and I'd like to work as hard
as possible to limit the size of the API. This is for two reasons - first and foremost to
minimize the number of calls to these wrapper functions so that it doesn't impact
performance. Performance really is everything and we can't sacrifice that. This
isn't shaping up to be much of a problem so far because very few calls to our
dependencies happen within the I/O path. Secondarily, I'd like to make it as
straightforward as possible for users to re-implement the env. The best way to make it
easy to do that is to keep the API small.
I look forward to the discussions at the summit this week. We'd love to hear about all
of the different environments, platforms, and frameworks where SPDK is being used and how
we can make it easier to integrate SPDK wherever it is valuable.
Thanks,
Ben
10:09 p.m.
At the summit we did end up discussing this quite a bit. For DPDK it's fairly
clear that we're going to continue to abstract away all of those dependencies
inside of the env library. There are pull requests out to finish this up now and
we'll get those merged relatively quickly. We're also working with the DPDK team
to improve DPDK so that there are hopefully fewer reasons to forego using it.
As far as POSIX, I have a proposal. Assuming POSIX is available makes it
significantly easier to work on SPDK itself, so I'm very hesitant to move all
POSIX calls inside of the env library and force people to use wrapper functions.
I'm concerned that it raises the barrier to contribute too much (can I call
malloc? what about printf?). Further, the vast majority of SPDK users actually
do have POSIX available to them. That said, there are probably parts that we can
abstract out and some other refactoring we can do here.
First, I think pthreads can be refactored into the env library. Locking and
spawning threads is rare enough in SPDK that it isn't an undue burden to use
wrappers, and it also requires some care regarding CPU affinity. So the below
discussion excludes that.
Second, I think we can get away with moving all of our POSIX headers to a single
SPDK header file at spdk/stdinc.h. We can then include that file everywhere
either directly or by using a -include option in the Makefile. This will even
play nicely with precompiling spdk/stdinc.h (thanks Daniel for the suggestion),
although SPDK's build times are not a concern right now.
Third, for the small subset of SPDK users that really don't have POSIX
available, they can carry a patch on top of SPDK that simply replaces
spdk/stdinc.h with their own implementation. Most environments have at least
some part of the POSIX standard, and 90% of what we're using is just the C
standard library, so in reality this header shouldn't be too difficult to
replace.
Thoughts? Opinions?
Thanks,
Ben
On Mon, 2017-04-17 at 18:55 +0000, Walker, Benjamin wrote:
Hi all,
There has been a lot of activity lately dealing with abstracting SPDK's
dependencies to allow porting SPDK to additional platforms and frameworks. I
think we'll end up talking about this extensively at the summit this week. The
ability to use SPDK in a variety of environments is very important to us, so I
look forward to the discussions. I wanted to write a basic summary of the
current state of SPDK's dependencies for everyone to have, and so that those
who are unable to attend the summit can provide feedback. I'll try to update
this thread with the results of the discussions at the summit for everyone's
benefit.
We've already eliminated all SPDK dependencies except for two:
1) POSIX
2) DPDK
SPDK already has a mechanism for abstracting away the "environment" it is
running in. We do this by moving all calls to dependencies to the "env"
library, whose interface is defined by include/spdk/env.h. The build system
allows users to point to an alternate implementation of the env library if
required. SPDK's default implementation of env is based on DPDK and runs in
Linux and FreeBSD user space.
DPDK is a very large toolkit, of course, but until just recently SPDK only
used DPDK's 'eal' module (environment abstraction library). The original
goal
of the SPDK env library was just to abstract all uses of DPDK's eal, but not
all uses of POSIX APIs. We made some progress in this area, but there are a
number of remaining calls as of this writing. Fortunately, the wider community
has really come through. There are at least two pull requests out from two
independent users that mostly finish the job. We'll work through those pull
requests and try to get them merged shortly. The only part of the code that
will continue to explicitly depend on DPDK will be the newly released vhost-
scsi target because that relies on DPDK vhost infrastructure. This
infrastructure is outside of DPDK's eal and is quite extensive. We feel this
is acceptable because building vhost in SPDK is optional - you can just turn
it off to eliminate the dependency.
There are a number of legitimate reasons to want to remove the DPDK dependency
from SPDK. For instance, DPDK dictates a particular memory management model,
requires a specific threading model, and performs PCI device management that
may conflict with other libraries. However, DPDK provides tremendous value to
SPDK in a number of areas, especially by implementing memory allocation
suitable for DMA in user space. Memory management is a major challenge with
all sorts of hidden traps and challenges, so we see significant value in using
a production-tested library here. Re-implement the env without DPDK at your
own risk!
Abstracting away our POSIX dependency is a similar but larger challenge. While
we foresaw the need for people to replace DPDK, we didn't see any use case for
dropping POSIX. We were definitely wrong though - people are using SPDK in all
sorts of environments that we didn't predict, from embedded firmware to
operating system kernels and beyond. So we'll need to address this as time
goes on. I think this will result in further expansion of the 'env' library.
Fortunately, we don't require too much from POSIX - mostly pthreads, the C
standard library, and maybe a couple of other miscellaneous headers.
There are other less obvious dependencies in SPDK. For instance, it also
depends on a relatively recent version of gcc or clang. This is mostly a
practical concern and not so much by intention. Our continuous integration
system (which I'll talk about at the summit - exciting developments here)
currently consists of 4 machines, 3 of which are running Fedora Linux 25 while
the other is running FreeBSD 11.0. These are very much up to date systems, so
we tend not to catch bugs on older systems today. We also use cutting edge C
features and tools, so we'll probably over time need to select a particular
minimum version of the C standard that we're going to develop to.
I'm also concerned with the scope of the env library and I'd like to work as
hard as possible to limit the size of the API. This is for two reasons - first
and foremost to minimize the number of calls to these wrapper functions so
that it doesn't impact performance. Performance really is everything and we
can't sacrifice that. This isn't shaping up to be much of a problem so far
because very few calls to our dependencies happen within the I/O path.
Secondarily, I'd like to make it as straightforward as possible for users to
re-implement the env. The best way to make it easy to do that is to keep the
API small.
I look forward to the discussions at the summit this week. We'd love to hear
about all of the different environments, platforms, and frameworks where SPDK
is being used and how we can make it easier to integrate SPDK wherever it is
valuable.
Thanks,
Ben
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
5:36 a.m.
I think there was also a proposal to consolidate/refactor/reduce the env
API. Where are we with that proposal? Is it an activity that will start
once the current SPDK/DPDK separation completes or do we plan to do it
concurrently. It would be nice to say we are "done" with the API changes
at some point, in a sense freezing the API, unless there is a bug that
needs fixing, or additions that came out of unforeseen requirements.
This allows other environments a stable interface that may be
implemented without too much churn and rebase anxiety.
thanks,
--ranjit
On 4/27/2017 3:39 AM, Walker, Benjamin wrote:
At the summit we did end up discussing this quite a bit. For DPDK
it's fairly
clear that we're going to continue to abstract away all of those dependencies
inside of the env library. There are pull requests out to finish this up now and
we'll get those merged relatively quickly. We're also working with the DPDK team
to improve DPDK so that there are hopefully fewer reasons to forego using it.
As far as POSIX, I have a proposal. Assuming POSIX is available makes it
significantly easier to work on SPDK itself, so I'm very hesitant to move all
POSIX calls inside of the env library and force people to use wrapper functions.
I'm concerned that it raises the barrier to contribute too much (can I call
malloc? what about printf?). Further, the vast majority of SPDK users actually
do have POSIX available to them. That said, there are probably parts that we can
abstract out and some other refactoring we can do here.
First, I think pthreads can be refactored into the env library. Locking and
spawning threads is rare enough in SPDK that it isn't an undue burden to use
wrappers, and it also requires some care regarding CPU affinity. So the below
discussion excludes that.
Second, I think we can get away with moving all of our POSIX headers to a single
SPDK header file at spdk/stdinc.h. We can then include that file everywhere
either directly or by using a -include option in the Makefile. This will even
play nicely with precompiling spdk/stdinc.h (thanks Daniel for the suggestion),
although SPDK's build times are not a concern right now.
Third, for the small subset of SPDK users that really don't have POSIX
available, they can carry a patch on top of SPDK that simply replaces
spdk/stdinc.h with their own implementation. Most environments have at least
some part of the POSIX standard, and 90% of what we're using is just the C
standard library, so in reality this header shouldn't be too difficult to
replace.
Thoughts? Opinions?
Thanks,
Ben
On Mon, 2017-04-17 at 18:55 +0000, Walker, Benjamin wrote:
> Hi all,
>
> There has been a lot of activity lately dealing with abstracting SPDK's
> dependencies to allow porting SPDK to additional platforms and frameworks. I
> think we'll end up talking about this extensively at the summit this week. The
> ability to use SPDK in a variety of environments is very important to us, so I
> look forward to the discussions. I wanted to write a basic summary of the
> current state of SPDK's dependencies for everyone to have, and so that those
> who are unable to attend the summit can provide feedback. I'll try to update
> this thread with the results of the discussions at the summit for everyone's
> benefit.
>
> We've already eliminated all SPDK dependencies except for two:
>
> 1) POSIX
> 2) DPDK
>
> SPDK already has a mechanism for abstracting away the "environment" it is
> running in. We do this by moving all calls to dependencies to the "env"
> library, whose interface is defined by include/spdk/env.h. The build system
> allows users to point to an alternate implementation of the env library if
> required. SPDK's default implementation of env is based on DPDK and runs in
> Linux and FreeBSD user space.
>
> DPDK is a very large toolkit, of course, but until just recently SPDK only
> used DPDK's 'eal' module (environment abstraction library). The original
goal
> of the SPDK env library was just to abstract all uses of DPDK's eal, but not
> all uses of POSIX APIs. We made some progress in this area, but there are a
> number of remaining calls as of this writing. Fortunately, the wider community
> has really come through. There are at least two pull requests out from two
> independent users that mostly finish the job. We'll work through those pull
> requests and try to get them merged shortly. The only part of the code that
> will continue to explicitly depend on DPDK will be the newly released vhost-
> scsi target because that relies on DPDK vhost infrastructure. This
> infrastructure is outside of DPDK's eal and is quite extensive. We feel this
> is acceptable because building vhost in SPDK is optional - you can just turn
> it off to eliminate the dependency.
>
> There are a number of legitimate reasons to want to remove the DPDK dependency
> from SPDK. For instance, DPDK dictates a particular memory management model,
> requires a specific threading model, and performs PCI device management that
> may conflict with other libraries. However, DPDK provides tremendous value to
> SPDK in a number of areas, especially by implementing memory allocation
> suitable for DMA in user space. Memory management is a major challenge with
> all sorts of hidden traps and challenges, so we see significant value in using
> a production-tested library here. Re-implement the env without DPDK at your
> own risk!
>
> Abstracting away our POSIX dependency is a similar but larger challenge. While
> we foresaw the need for people to replace DPDK, we didn't see any use case for
> dropping POSIX. We were definitely wrong though - people are using SPDK in all
> sorts of environments that we didn't predict, from embedded firmware to
> operating system kernels and beyond. So we'll need to address this as time
> goes on. I think this will result in further expansion of the 'env' library.
> Fortunately, we don't require too much from POSIX - mostly pthreads, the C
> standard library, and maybe a couple of other miscellaneous headers.
>
> There are other less obvious dependencies in SPDK. For instance, it also
> depends on a relatively recent version of gcc or clang. This is mostly a
> practical concern and not so much by intention. Our continuous integration
> system (which I'll talk about at the summit - exciting developments here)
> currently consists of 4 machines, 3 of which are running Fedora Linux 25 while
> the other is running FreeBSD 11.0. These are very much up to date systems, so
> we tend not to catch bugs on older systems today. We also use cutting edge C
> features and tools, so we'll probably over time need to select a particular
> minimum version of the C standard that we're going to develop to.
>
> I'm also concerned with the scope of the env library and I'd like to work as
> hard as possible to limit the size of the API. This is for two reasons - first
> and foremost to minimize the number of calls to these wrapper functions so
> that it doesn't impact performance. Performance really is everything and we
> can't sacrifice that. This isn't shaping up to be much of a problem so far
> because very few calls to our dependencies happen within the I/O path.
> Secondarily, I'd like to make it as straightforward as possible for users to
> re-implement the env. The best way to make it easy to do that is to keep the
> API small.
>
> I look forward to the discussions at the summit this week. We'd love to hear
> about all of the different environments, platforms, and frameworks where SPDK
> is being used and how we can make it easier to integrate SPDK wherever it is
> valuable.
>
> Thanks,
> Ben
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
9:10 p.m.
On Thu, 2017-04-27 at 11:06 +0530, Ranjit Noronha wrote:
I think there was also a proposal to consolidate/refactor/reduce the env API.
Where are we with that proposal? Is it an activity that will start once the
current SPDK/DPDK separation completes or do we plan to do it concurrently. It
would be nice to say we are "done" with the API changes at some point, in a
sense freezing the API, unless there is a bug that needs fixing, or additions
that came out of unforeseen requirements.
This allows other environments a stable interface that may be implemented
without too much churn and rebase anxiety.
thanks,
--ranjit
This is an ongoing background activity that will continue for some time. We'll
make sure to keep the env API as stable as possible once DPDK is fully
abstracted and strive to only do wholesale removal of functions once we don't
need them anymore. Removing a function from the API isn't particularly
disruptive in my opinion, whereas changing an API definitely is.
11:38 a.m.
Usage of printf/fprintf ought to be replaced with SPDK logging calls so users of the
library can control the UI. Then there are posix calls in the library to abort/exit, for
which the library should be reworked to eliminate. Nobody’s keen on a library that aborts
the app when parameter validation fails ( I’d also like to see some way to avoid setting
signal handlers in the library, or at least block it out with an ifdef section so that
users can manage signals on their own.
The remaining posix calls are to malloc/free/strdup/calloc, etc which are harder to manage
in non-posix environments. Kernel allocators take multiple parameters besides the size -
they track what system is using memory and flags to manage various attributes like memory
region (dma, pci, etc), should the call block, etc. Even if you get it to compile, then
linking callers of single argument malloc with a kernel malloc that expects 3 arguments is
going to have unpleasant results. So, I’d prefer an spdk env abstraction for the
allocation calls to allow implementers of new environments to better manage memory.
I like the idea of an “spdk/stdinc.h” that collects all the std system headers in one
place. You could even make checking for it part of the format checks so that all library
sources include it first. Or better yet as you point out, use ‘-include’ to include it in
all sources and have a makefile option that will allow users to replace it with their own
version.
Ed R
On 4/26/17, 6:09 PM, "SPDK on behalf of Walker, Benjamin"
<spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com> wrote:
At the summit we did end up discussing this quite a bit. For DPDK it's fairly
clear that we're going to continue to abstract away all of those dependencies
inside of the env library. There are pull requests out to finish this up now and
we'll get those merged relatively quickly. We're also working with the DPDK
team
to improve DPDK so that there are hopefully fewer reasons to forego using it.
As far as POSIX, I have a proposal. Assuming POSIX is available makes it
significantly easier to work on SPDK itself, so I'm very hesitant to move all
POSIX calls inside of the env library and force people to use wrapper functions.
I'm concerned that it raises the barrier to contribute too much (can I call
malloc? what about printf?). Further, the vast majority of SPDK users actually
do have POSIX available to them. That said, there are probably parts that we can
abstract out and some other refactoring we can do here.
First, I think pthreads can be refactored into the env library. Locking and
spawning threads is rare enough in SPDK that it isn't an undue burden to use
wrappers, and it also requires some care regarding CPU affinity. So the below
discussion excludes that.
Second, I think we can get away with moving all of our POSIX headers to a single
SPDK header file at spdk/stdinc.h. We can then include that file everywhere
either directly or by using a -include option in the Makefile. This will even
play nicely with precompiling spdk/stdinc.h (thanks Daniel for the suggestion),
although SPDK's build times are not a concern right now.
Third, for the small subset of SPDK users that really don't have POSIX
available, they can carry a patch on top of SPDK that simply replaces
spdk/stdinc.h with their own implementation. Most environments have at least
some part of the POSIX standard, and 90% of what we're using is just the C
standard library, so in reality this header shouldn't be too difficult to
replace.
Thoughts? Opinions?
Thanks,
Ben
On Mon, 2017-04-17 at 18:55 +0000, Walker, Benjamin wrote:
Hi all,
There has been a lot of activity lately dealing with abstracting SPDK's
dependencies to allow porting SPDK to additional platforms and frameworks. I
think we'll end up talking about this extensively at the summit this week. The
ability to use SPDK in a variety of environments is very important to us, so I
look forward to the discussions. I wanted to write a basic summary of the
current state of SPDK's dependencies for everyone to have, and so that those
who are unable to attend the summit can provide feedback. I'll try to update
this thread with the results of the discussions at the summit for everyone's
benefit.
We've already eliminated all SPDK dependencies except for two:
1) POSIX
2) DPDK
SPDK already has a mechanism for abstracting away the "environment" it is
running in. We do this by moving all calls to dependencies to the "env"
library, whose interface is defined by include/spdk/env.h. The build system
allows users to point to an alternate implementation of the env library if
required. SPDK's default implementation of env is based on DPDK and runs in
Linux and FreeBSD user space.
DPDK is a very large toolkit, of course, but until just recently SPDK only
used DPDK's 'eal' module (environment abstraction library). The original
goal
of the SPDK env library was just to abstract all uses of DPDK's eal, but not
all uses of POSIX APIs. We made some progress in this area, but there are a
number of remaining calls as of this writing. Fortunately, the wider community
has really come through. There are at least two pull requests out from two
independent users that mostly finish the job. We'll work through those pull
requests and try to get them merged shortly. The only part of the code that
will continue to explicitly depend on DPDK will be the newly released vhost-
scsi target because that relies on DPDK vhost infrastructure. This
infrastructure is outside of DPDK's eal and is quite extensive. We feel this
is acceptable because building vhost in SPDK is optional - you can just turn
it off to eliminate the dependency.
There are a number of legitimate reasons to want to remove the DPDK dependency
from SPDK. For instance, DPDK dictates a particular memory management model,
requires a specific threading model, and performs PCI device management that
may conflict with other libraries. However, DPDK provides tremendous value to
SPDK in a number of areas, especially by implementing memory allocation
suitable for DMA in user space. Memory management is a major challenge with
all sorts of hidden traps and challenges, so we see significant value in using
a production-tested library here. Re-implement the env without DPDK at your
own risk!
Abstracting away our POSIX dependency is a similar but larger challenge. While
we foresaw the need for people to replace DPDK, we didn't see any use case for
dropping POSIX. We were definitely wrong though - people are using SPDK in all
sorts of environments that we didn't predict, from embedded firmware to
operating system kernels and beyond. So we'll need to address this as time
goes on. I think this will result in further expansion of the 'env' library.
Fortunately, we don't require too much from POSIX - mostly pthreads, the C
standard library, and maybe a couple of other miscellaneous headers.
There are other less obvious dependencies in SPDK. For instance, it also
depends on a relatively recent version of gcc or clang. This is mostly a
practical concern and not so much by intention. Our continuous integration
system (which I'll talk about at the summit - exciting developments here)
currently consists of 4 machines, 3 of which are running Fedora Linux 25 while
the other is running FreeBSD 11.0. These are very much up to date systems, so
we tend not to catch bugs on older systems today. We also use cutting edge C
features and tools, so we'll probably over time need to select a particular
minimum version of the C standard that we're going to develop to.
I'm also concerned with the scope of the env library and I'd like to work as
hard as possible to limit the size of the API. This is for two reasons - first
and foremost to minimize the number of calls to these wrapper functions so
that it doesn't impact performance. Performance really is everything and we
can't sacrifice that. This isn't shaping up to be much of a problem so far
because very few calls to our dependencies happen within the I/O path.
Secondarily, I'd like to make it as straightforward as possible for users to
re-implement the env. The best way to make it easy to do that is to keep the
API small.
I look forward to the discussions at the summit this week. We'd love to hear
about all of the different environments, platforms, and frameworks where SPDK
is being used and how we can make it easier to integrate SPDK wherever it is
valuable.
Thanks,
Ben
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
11:46 a.m.
Ed, were you able to make any progress yesterday with your testing?
I have these changes ready to push upstream, but I need to know if you are done.
Sent from my iPhone
On Apr 29, 2017, at 7:39 AM, Rodriguez, Edwin
<Ed.Rodriguez(a)netapp.com> wrote:
Usage of printf/fprintf ought to be replaced with SPDK logging calls so users of the
library can control the UI. Then there are posix calls in the library to abort/exit, for
which the library should be reworked to eliminate. Nobody’s keen on a library that aborts
the app when parameter validation fails ( I’d also like to see some way to avoid setting
signal handlers in the library, or at least block it out with an ifdef section so that
users can manage signals on their own.
The remaining posix calls are to malloc/free/strdup/calloc, etc which are harder to
manage in non-posix environments. Kernel allocators take multiple parameters besides the
size - they track what system is using memory and flags to manage various attributes like
memory region (dma, pci, etc), should the call block, etc. Even if you get it to compile,
then linking callers of single argument malloc with a kernel malloc that expects 3
arguments is going to have unpleasant results. So, I’d prefer an spdk env abstraction for
the allocation calls to allow implementers of new environments to better manage memory.
I like the idea of an “spdk/stdinc.h” that collects all the std system headers in one
place. You could even make checking for it part of the format checks so that all library
sources include it first. Or better yet as you point out, use ‘-include’ to include it in
all sources and have a makefile option that will allow users to replace it with their own
version.
Ed R
On 4/26/17, 6:09 PM, "SPDK on behalf of Walker, Benjamin"
<spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com> wrote:
At the summit we did end up discussing this quite a bit. For DPDK it's fairly
clear that we're going to continue to abstract away all of those dependencies
inside of the env library. There are pull requests out to finish this up now and
we'll get those merged relatively quickly. We're also working with the DPDK
team
to improve DPDK so that there are hopefully fewer reasons to forego using it.
As far as POSIX, I have a proposal. Assuming POSIX is available makes it
significantly easier to work on SPDK itself, so I'm very hesitant to move all
POSIX calls inside of the env library and force people to use wrapper functions.
I'm concerned that it raises the barrier to contribute too much (can I call
malloc? what about printf?). Further, the vast majority of SPDK users actually
do have POSIX available to them. That said, there are probably parts that we can
abstract out and some other refactoring we can do here.
First, I think pthreads can be refactored into the env library. Locking and
spawning threads is rare enough in SPDK that it isn't an undue burden to use
wrappers, and it also requires some care regarding CPU affinity. So the below
discussion excludes that.
Second, I think we can get away with moving all of our POSIX headers to a single
SPDK header file at spdk/stdinc.h. We can then include that file everywhere
either directly or by using a -include option in the Makefile. This will even
play nicely with precompiling spdk/stdinc.h (thanks Daniel for the suggestion),
although SPDK's build times are not a concern right now.
Third, for the small subset of SPDK users that really don't have POSIX
available, they can carry a patch on top of SPDK that simply replaces
spdk/stdinc.h with their own implementation. Most environments have at least
some part of the POSIX standard, and 90% of what we're using is just the C
standard library, so in reality this header shouldn't be too difficult to
replace.
Thoughts? Opinions?
Thanks,
Ben
> On Mon, 2017-04-17 at 18:55 +0000, Walker, Benjamin wrote:
> Hi all,
>
> There has been a lot of activity lately dealing with abstracting SPDK's
> dependencies to allow porting SPDK to additional platforms and frameworks. I
> think we'll end up talking about this extensively at the summit this week. The
> ability to use SPDK in a variety of environments is very important to us, so I
> look forward to the discussions. I wanted to write a basic summary of the
> current state of SPDK's dependencies for everyone to have, and so that those
> who are unable to attend the summit can provide feedback. I'll try to update
> this thread with the results of the discussions at the summit for everyone's
> benefit.
>
> We've already eliminated all SPDK dependencies except for two:
>
> 1) POSIX
> 2) DPDK
>
> SPDK already has a mechanism for abstracting away the "environment" it is
> running in. We do this by moving all calls to dependencies to the "env"
> library, whose interface is defined by include/spdk/env.h. The build system
> allows users to point to an alternate implementation of the env library if
> required. SPDK's default implementation of env is based on DPDK and runs in
> Linux and FreeBSD user space.
>
> DPDK is a very large toolkit, of course, but until just recently SPDK only
> used DPDK's 'eal' module (environment abstraction library). The original
goal
> of the SPDK env library was just to abstract all uses of DPDK's eal, but not
> all uses of POSIX APIs. We made some progress in this area, but there are a
> number of remaining calls as of this writing. Fortunately, the wider community
> has really come through. There are at least two pull requests out from two
> independent users that mostly finish the job. We'll work through those pull
> requests and try to get them merged shortly. The only part of the code that
> will continue to explicitly depend on DPDK will be the newly released vhost-
> scsi target because that relies on DPDK vhost infrastructure. This
> infrastructure is outside of DPDK's eal and is quite extensive. We feel this
> is acceptable because building vhost in SPDK is optional - you can just turn
> it off to eliminate the dependency.
>
> There are a number of legitimate reasons to want to remove the DPDK dependency
> from SPDK. For instance, DPDK dictates a particular memory management model,
> requires a specific threading model, and performs PCI device management that
> may conflict with other libraries. However, DPDK provides tremendous value to
> SPDK in a number of areas, especially by implementing memory allocation
> suitable for DMA in user space. Memory management is a major challenge with
> all sorts of hidden traps and challenges, so we see significant value in using
> a production-tested library here. Re-implement the env without DPDK at your
> own risk!
>
> Abstracting away our POSIX dependency is a similar but larger challenge. While
> we foresaw the need for people to replace DPDK, we didn't see any use case for
> dropping POSIX. We were definitely wrong though - people are using SPDK in all
> sorts of environments that we didn't predict, from embedded firmware to
> operating system kernels and beyond. So we'll need to address this as time
> goes on. I think this will result in further expansion of the 'env'
library.
> Fortunately, we don't require too much from POSIX - mostly pthreads, the C
> standard library, and maybe a couple of other miscellaneous headers.
>
> There are other less obvious dependencies in SPDK. For instance, it also
> depends on a relatively recent version of gcc or clang. This is mostly a
> practical concern and not so much by intention. Our continuous integration
> system (which I'll talk about at the summit - exciting developments here)
> currently consists of 4 machines, 3 of which are running Fedora Linux 25 while
> the other is running FreeBSD 11.0. These are very much up to date systems, so
> we tend not to catch bugs on older systems today. We also use cutting edge C
> features and tools, so we'll probably over time need to select a particular
> minimum version of the C standard that we're going to develop to.
>
> I'm also concerned with the scope of the env library and I'd like to work
as
> hard as possible to limit the size of the API. This is for two reasons - first
> and foremost to minimize the number of calls to these wrapper functions so
> that it doesn't impact performance. Performance really is everything and we
> can't sacrifice that. This isn't shaping up to be much of a problem so far
> because very few calls to our dependencies happen within the I/O path.
> Secondarily, I'd like to make it as straightforward as possible for users to
> re-implement the env. The best way to make it easy to do that is to keep the
> API small.
>
> I look forward to the discussions at the summit this week. We'd love to hear
> about all of the different environments, platforms, and frameworks where SPDK
> is being used and how we can make it easier to integrate SPDK wherever it is
> valuable.
>
> Thanks,
> Ben
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
2:23 a.m.
All of the changes that we are proposing to abstract DPDK and POSIX are rolled up in pull
request #152
Master Pull request for DPDK and Posix abstraction changes.
<https://github.com/spdk/spdk/pull/152>
I don’t expect these changes to go in all at once. Instead, I will be cherry picking two
or three patches at a time from this branch and submitting separate, smaller, pull
requests for each testable set of changes.
I encourage everyone to please review these changes and add your comments. If you are
interested in taking a whole copy into your private repo to test and experiment with, I
encourage you to clone my Github fork at https://github.com/johnmeneghini/spdk.git
<https://github.com/johnmeneghini/spdk.git> and checkout the spdk_eal10 branch.
(Note: you can ignore all of the other branches in this repo. I need to clean it up).
Also note that very little testing has been done with this branch yet. I am still trying
to replicate the Intel CIT environment in my test lab so only the simplest of testing has
been done with this branch.
A few of these patches seem pretty invasive because they touch many files. For example:
system: Remove userspace header file dependencies
<https://github.com/spdk/spdk/pull/152/commits/e5fc0bfe53ba618425e3cfe89ab...;.
However, only the following files and libraries have been converted. This is the
complete list of applications and libraries that have been converted to abstract all DPDK
and POSIX dependencies.
[libspdk_nvmf_tgt_app.a]
spdk/app/nvmf_tgt/conf.c
spdk/app/nvmf_tgt/nvmf_tgt.c
[libspdk_bdev.a]
spdk/lib/bdev/bdev.c
[libspdk_conf.a]
spdk/lib/conf/conf.c
[libspdk_copy.a]
spdk/lib/copy/copy_engine.c
[libspdk_event.a]
spdk/lib/event/app.c
spdk/lib/event/app_config.c
spdk/lib/event/reactor.c
spdk/lib/event/subsystem.c
[libspdk_json.a]
spdk/lib/json/json_parse.c
spdk/lib/json/json_util.c
spdk/lib/json/json_write.c
[libspdk_log.a]
spdk/lib/log/log.c
[libspdk_nvme.a]
spdk/lib/nvme/nvme.c
spdk/lib/nvme/nvme_ctrlr.c
spdk/lib/nvme/nvme_ctrlr_cmd.c
spdk/lib/nvme/nvme_ns.c
spdk/lib/nvme/nvme_ns_cmd.c
spdk/lib/nvme/nvme_pcie.c
spdk/lib/nvme/nvme_qpair.c
spdk/lib/nvme/nvme_quirks.c
spdk/lib/nvme/nvme_rdma.c
spdk/lib/nvme/nvme_transport.c
spdk/lib/nvme/nvme_uevent.c
[libspdk_nvmf.a]
spdk/lib/nvmf/direct.c
spdk/lib/nvmf/discovery.c
spdk/lib/nvmf/nvmf.c
spdk/lib/nvmf/rdma.c
spdk/lib/nvmf/request.c
spdk/lib/nvmf/session.c
spdk/lib/nvmf/subsystem.c
spdk/lib/nvmf/transport.c
spdk/lib/nvmf/virtual.c
[libspdk_trace.a]
spdk/lib/trace/trace.c
[libspdk_util.a]
spdk/lib/util/bit_array.c
spdk/lib/util/fd.c
spdk/lib/util/io_channel.c
spdk/lib/util/string.c
John Meneghini
john.a.meneghini(a)gmail.com
On Apr 29, 2017, at 7:38 AM, Rodriguez, Edwin
<Ed.Rodriguez(a)netapp.com> wrote:
Usage of printf/fprintf ought to be replaced with SPDK logging calls so users of the
library can control the UI. Then there are posix calls in the library to abort/exit, for
which the library should be reworked to eliminate. Nobody’s keen on a library that aborts
the app when parameter validation fails ( I’d also like to see some way to avoid setting
signal handlers in the library, or at least block it out with an ifdef section so that
users can manage signals on their own.
The remaining posix calls are to malloc/free/strdup/calloc, etc which are harder to
manage in non-posix environments. Kernel allocators take multiple parameters besides the
size - they track what system is using memory and flags to manage various attributes like
memory region (dma, pci, etc), should the call block, etc. Even if you get it to compile,
then linking callers of single argument malloc with a kernel malloc that expects 3
arguments is going to have unpleasant results. So, I’d prefer an spdk env abstraction for
the allocation calls to allow implementers of new environments to better manage memory.
I like the idea of an “spdk/stdinc.h” that collects all the std system headers in one
place. You could even make checking for it part of the format checks so that all library
sources include it first. Or better yet as you point out, use ‘-include’ to include it in
all sources and have a makefile option that will allow users to replace it with their own
version.
Ed R
On 4/26/17, 6:09 PM, "SPDK on behalf of Walker, Benjamin"
<spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com> wrote:
At the summit we did end up discussing this quite a bit. For DPDK it's fairly
clear that we're going to continue to abstract away all of those dependencies
inside of the env library. There are pull requests out to finish this up now and
we'll get those merged relatively quickly. We're also working with the DPDK
team
to improve DPDK so that there are hopefully fewer reasons to forego using it.
As far as POSIX, I have a proposal. Assuming POSIX is available makes it
significantly easier to work on SPDK itself, so I'm very hesitant to move all
POSIX calls inside of the env library and force people to use wrapper functions.
I'm concerned that it raises the barrier to contribute too much (can I call
malloc? what about printf?). Further, the vast majority of SPDK users actually
do have POSIX available to them. That said, there are probably parts that we can
abstract out and some other refactoring we can do here.
First, I think pthreads can be refactored into the env library. Locking and
spawning threads is rare enough in SPDK that it isn't an undue burden to use
wrappers, and it also requires some care regarding CPU affinity. So the below
discussion excludes that.
Second, I think we can get away with moving all of our POSIX headers to a single
SPDK header file at spdk/stdinc.h. We can then include that file everywhere
either directly or by using a -include option in the Makefile. This will even
play nicely with precompiling spdk/stdinc.h (thanks Daniel for the suggestion),
although SPDK's build times are not a concern right now.
Third, for the small subset of SPDK users that really don't have POSIX
available, they can carry a patch on top of SPDK that simply replaces
spdk/stdinc.h with their own implementation. Most environments have at least
some part of the POSIX standard, and 90% of what we're using is just the C
standard library, so in reality this header shouldn't be too difficult to
replace.
Thoughts? Opinions?
Thanks,
Ben
On Mon, 2017-04-17 at 18:55 +0000, Walker, Benjamin wrote:
> Hi all,
>
> There has been a lot of activity lately dealing with abstracting SPDK's
> dependencies to allow porting SPDK to additional platforms and frameworks. I
> think we'll end up talking about this extensively at the summit this week. The
> ability to use SPDK in a variety of environments is very important to us, so I
> look forward to the discussions. I wanted to write a basic summary of the
> current state of SPDK's dependencies for everyone to have, and so that those
> who are unable to attend the summit can provide feedback. I'll try to update
> this thread with the results of the discussions at the summit for everyone's
> benefit.
>
> We've already eliminated all SPDK dependencies except for two:
>
> 1) POSIX
> 2) DPDK
>
> SPDK already has a mechanism for abstracting away the "environment" it is
> running in. We do this by moving all calls to dependencies to the "env"
> library, whose interface is defined by include/spdk/env.h. The build system
> allows users to point to an alternate implementation of the env library if
> required. SPDK's default implementation of env is based on DPDK and runs in
> Linux and FreeBSD user space.
>
> DPDK is a very large toolkit, of course, but until just recently SPDK only
> used DPDK's 'eal' module (environment abstraction library). The original
goal
> of the SPDK env library was just to abstract all uses of DPDK's eal, but not
> all uses of POSIX APIs. We made some progress in this area, but there are a
> number of remaining calls as of this writing. Fortunately, the wider community
> has really come through. There are at least two pull requests out from two
> independent users that mostly finish the job. We'll work through those pull
> requests and try to get them merged shortly. The only part of the code that
> will continue to explicitly depend on DPDK will be the newly released vhost-
> scsi target because that relies on DPDK vhost infrastructure. This
> infrastructure is outside of DPDK's eal and is quite extensive. We feel this
> is acceptable because building vhost in SPDK is optional - you can just turn
> it off to eliminate the dependency.
>
> There are a number of legitimate reasons to want to remove the DPDK dependency
> from SPDK. For instance, DPDK dictates a particular memory management model,
> requires a specific threading model, and performs PCI device management that
> may conflict with other libraries. However, DPDK provides tremendous value to
> SPDK in a number of areas, especially by implementing memory allocation
> suitable for DMA in user space. Memory management is a major challenge with
> all sorts of hidden traps and challenges, so we see significant value in using
> a production-tested library here. Re-implement the env without DPDK at your
> own risk!
>
> Abstracting away our POSIX dependency is a similar but larger challenge. While
> we foresaw the need for people to replace DPDK, we didn't see any use case for
> dropping POSIX. We were definitely wrong though - people are using SPDK in all
> sorts of environments that we didn't predict, from embedded firmware to
> operating system kernels and beyond. So we'll need to address this as time
> goes on. I think this will result in further expansion of the 'env' library.
> Fortunately, we don't require too much from POSIX - mostly pthreads, the C
> standard library, and maybe a couple of other miscellaneous headers.
>
> There are other less obvious dependencies in SPDK. For instance, it also
> depends on a relatively recent version of gcc or clang. This is mostly a
> practical concern and not so much by intention. Our continuous integration
> system (which I'll talk about at the summit - exciting developments here)
> currently consists of 4 machines, 3 of which are running Fedora Linux 25 while
> the other is running FreeBSD 11.0. These are very much up to date systems, so
> we tend not to catch bugs on older systems today. We also use cutting edge C
> features and tools, so we'll probably over time need to select a particular
> minimum version of the C standard that we're going to develop to.
>
> I'm also concerned with the scope of the env library and I'd like to work as
> hard as possible to limit the size of the API. This is for two reasons - first
> and foremost to minimize the number of calls to these wrapper functions so
> that it doesn't impact performance. Performance really is everything and we
> can't sacrifice that. This isn't shaping up to be much of a problem so far
> because very few calls to our dependencies happen within the I/O path.
> Secondarily, I'd like to make it as straightforward as possible for users to
> re-implement the env. The best way to make it easy to do that is to keep the
> API small.
>
> I look forward to the discussions at the summit this week. We'd love to hear
> about all of the different environments, platforms, and frameworks where SPDK
> is being used and how we can make it easier to integrate SPDK wherever it is
> valuable.
>
> Thanks,
> Ben
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
10:08 a.m.
I’ve closed all of my prior pull requests and the process of integrating patches from
Master Pull request for DPDK and Posix abstraction changes
<https://github.com/spdk/spdk/pull/152> into SPDK has begun.
Ben and Daniel are working on this, and they have agreed to publish a Github pull request
for each patch they rework before they commit it into the master branch.
I would encourage others to watch https://github.com/spdk/spdk/pulls
<https://github.com/spdk/spdk/pulls> and participate in the process of reviewing
these changes.
John Meneghini
john.a.meneghini(a)gmail.com
On May 2, 2017, at 10:23 PM, John Meneghini
<john.a.meneghini(a)gmail.com> wrote:
All of the changes that we are proposing to abstract DPDK and POSIX are rolled up in pull
request #152
Master Pull request for DPDK and Posix abstraction changes.
<https://github.com/spdk/spdk/pull/152>
I don’t expect these changes to go in all at once. Instead, I will be cherry picking two
or three patches at a time from this branch and submitting separate, smaller, pull
requests for each testable set of changes.
I encourage everyone to please review these changes and add your comments. If you are
interested in taking a whole copy into your private repo to test and experiment with, I
encourage you to clone my Github fork at https://github.com/johnmeneghini/spdk.git
<https://github.com/johnmeneghini/spdk.git> and checkout the spdk_eal10 branch.
(Note: you can ignore all of the other branches in this repo. I need to clean it up).
Also note that very little testing has been done with this branch yet. I am still trying
to replicate the Intel CIT environment in my test lab so only the simplest of testing has
been done with this branch.
A few of these patches seem pretty invasive because they touch many files. For example:
system: Remove userspace header file dependencies
<https://github.com/spdk/spdk/pull/152/commits/e5fc0bfe53ba618425e3cfe89ab...;.
However, only the following files and libraries have been converted. This is the
complete list of applications and libraries that have been converted to abstract all DPDK
and POSIX dependencies.
[libspdk_nvmf_tgt_app.a]
spdk/app/nvmf_tgt/conf.c
spdk/app/nvmf_tgt/nvmf_tgt.c
[libspdk_bdev.a]
spdk/lib/bdev/bdev.c
[libspdk_conf.a]
spdk/lib/conf/conf.c
[libspdk_copy.a]
spdk/lib/copy/copy_engine.c
[libspdk_event.a]
spdk/lib/event/app.c
spdk/lib/event/app_config.c
spdk/lib/event/reactor.c
spdk/lib/event/subsystem.c
[libspdk_json.a]
spdk/lib/json/json_parse.c
spdk/lib/json/json_util.c
spdk/lib/json/json_write.c
[libspdk_log.a]
spdk/lib/log/log.c
[libspdk_nvme.a]
spdk/lib/nvme/nvme.c
spdk/lib/nvme/nvme_ctrlr.c
spdk/lib/nvme/nvme_ctrlr_cmd.c
spdk/lib/nvme/nvme_ns.c
spdk/lib/nvme/nvme_ns_cmd.c
spdk/lib/nvme/nvme_pcie.c
spdk/lib/nvme/nvme_qpair.c
spdk/lib/nvme/nvme_quirks.c
spdk/lib/nvme/nvme_rdma.c
spdk/lib/nvme/nvme_transport.c
spdk/lib/nvme/nvme_uevent.c
[libspdk_nvmf.a]
spdk/lib/nvmf/direct.c
spdk/lib/nvmf/discovery.c
spdk/lib/nvmf/nvmf.c
spdk/lib/nvmf/rdma.c
spdk/lib/nvmf/request.c
spdk/lib/nvmf/session.c
spdk/lib/nvmf/subsystem.c
spdk/lib/nvmf/transport.c
spdk/lib/nvmf/virtual.c
[libspdk_trace.a]
spdk/lib/trace/trace.c
[libspdk_util.a]
spdk/lib/util/bit_array.c
spdk/lib/util/fd.c
spdk/lib/util/io_channel.c
spdk/lib/util/string.c
John Meneghini
john.a.meneghini(a)gmail.com <mailto:john.a.meneghini@gmail.com>
> On Apr 29, 2017, at 7:38 AM, Rodriguez, Edwin <Ed.Rodriguez(a)netapp.com
<mailto:Ed.Rodriguez@netapp.com>> wrote:
>
> Usage of printf/fprintf ought to be replaced with SPDK logging calls so users of the
library can control the UI. Then there are posix calls in the library to abort/exit, for
which the library should be reworked to eliminate. Nobody’s keen on a library that aborts
the app when parameter validation fails ( I’d also like to see some way to avoid setting
signal handlers in the library, or at least block it out with an ifdef section so that
users can manage signals on their own.
>
> The remaining posix calls are to malloc/free/strdup/calloc, etc which are harder to
manage in non-posix environments. Kernel allocators take multiple parameters besides the
size - they track what system is using memory and flags to manage various attributes like
memory region (dma, pci, etc), should the call block, etc. Even if you get it to compile,
then linking callers of single argument malloc with a kernel malloc that expects 3
arguments is going to have unpleasant results. So, I’d prefer an spdk env abstraction for
the allocation calls to allow implementers of new environments to better manage memory.
>
> I like the idea of an “spdk/stdinc.h” that collects all the std system headers in one
place. You could even make checking for it part of the format checks so that all library
sources include it first. Or better yet as you point out, use ‘-include’ to include it in
all sources and have a makefile option that will allow users to replace it with their own
version.
>
> Ed R
>
> On 4/26/17, 6:09 PM, "SPDK on behalf of Walker, Benjamin"
<spdk-bounces(a)lists.01.org <mailto:spdk-bounces@lists.01.org> on behalf of
benjamin.walker(a)intel.com <mailto:benjamin.walker@intel.com>> wrote:
>
> At the summit we did end up discussing this quite a bit. For DPDK it's fairly
> clear that we're going to continue to abstract away all of those dependencies
> inside of the env library. There are pull requests out to finish this up now and
> we'll get those merged relatively quickly. We're also working with the
DPDK team
> to improve DPDK so that there are hopefully fewer reasons to forego using it.
>
> As far as POSIX, I have a proposal. Assuming POSIX is available makes it
> significantly easier to work on SPDK itself, so I'm very hesitant to move all
> POSIX calls inside of the env library and force people to use wrapper functions.
> I'm concerned that it raises the barrier to contribute too much (can I call
> malloc? what about printf?). Further, the vast majority of SPDK users actually
> do have POSIX available to them. That said, there are probably parts that we can
> abstract out and some other refactoring we can do here.
>
> First, I think pthreads can be refactored into the env library. Locking and
> spawning threads is rare enough in SPDK that it isn't an undue burden to use
> wrappers, and it also requires some care regarding CPU affinity. So the below
> discussion excludes that.
>
> Second, I think we can get away with moving all of our POSIX headers to a single
> SPDK header file at spdk/stdinc.h. We can then include that file everywhere
> either directly or by using a -include option in the Makefile. This will even
> play nicely with precompiling spdk/stdinc.h (thanks Daniel for the suggestion),
> although SPDK's build times are not a concern right now.
>
> Third, for the small subset of SPDK users that really don't have POSIX
> available, they can carry a patch on top of SPDK that simply replaces
> spdk/stdinc.h with their own implementation. Most environments have at least
> some part of the POSIX standard, and 90% of what we're using is just the C
> standard library, so in reality this header shouldn't be too difficult to
> replace.
>
> Thoughts? Opinions?
>
> Thanks,
> Ben
>
>
> On Mon, 2017-04-17 at 18:55 +0000, Walker, Benjamin wrote:
>> Hi all,
>>
>> There has been a lot of activity lately dealing with abstracting SPDK's
>> dependencies to allow porting SPDK to additional platforms and frameworks. I
>> think we'll end up talking about this extensively at the summit this week.
The
>> ability to use SPDK in a variety of environments is very important to us, so I
>> look forward to the discussions. I wanted to write a basic summary of the
>> current state of SPDK's dependencies for everyone to have, and so that those
>> who are unable to attend the summit can provide feedback. I'll try to update
>> this thread with the results of the discussions at the summit for everyone's
>> benefit.
>>
>> We've already eliminated all SPDK dependencies except for two:
>>
>> 1) POSIX
>> 2) DPDK
>>
>> SPDK already has a mechanism for abstracting away the "environment" it
is
>> running in. We do this by moving all calls to dependencies to the
"env"
>> library, whose interface is defined by include/spdk/env.h. The build system
>> allows users to point to an alternate implementation of the env library if
>> required. SPDK's default implementation of env is based on DPDK and runs in
>> Linux and FreeBSD user space.
>>
>> DPDK is a very large toolkit, of course, but until just recently SPDK only
>> used DPDK's 'eal' module (environment abstraction library). The
original goal
>> of the SPDK env library was just to abstract all uses of DPDK's eal, but not
>> all uses of POSIX APIs. We made some progress in this area, but there are a
>> number of remaining calls as of this writing. Fortunately, the wider community
>> has really come through. There are at least two pull requests out from two
>> independent users that mostly finish the job. We'll work through those pull
>> requests and try to get them merged shortly. The only part of the code that
>> will continue to explicitly depend on DPDK will be the newly released vhost-
>> scsi target because that relies on DPDK vhost infrastructure. This
>> infrastructure is outside of DPDK's eal and is quite extensive. We feel this
>> is acceptable because building vhost in SPDK is optional - you can just turn
>> it off to eliminate the dependency.
>>
>> There are a number of legitimate reasons to want to remove the DPDK dependency
>> from SPDK. For instance, DPDK dictates a particular memory management model,
>> requires a specific threading model, and performs PCI device management that
>> may conflict with other libraries. However, DPDK provides tremendous value to
>> SPDK in a number of areas, especially by implementing memory allocation
>> suitable for DMA in user space. Memory management is a major challenge with
>> all sorts of hidden traps and challenges, so we see significant value in using
>> a production-tested library here. Re-implement the env without DPDK at your
>> own risk!
>>
>> Abstracting away our POSIX dependency is a similar but larger challenge. While
>> we foresaw the need for people to replace DPDK, we didn't see any use case
for
>> dropping POSIX. We were definitely wrong though - people are using SPDK in all
>> sorts of environments that we didn't predict, from embedded firmware to
>> operating system kernels and beyond. So we'll need to address this as time
>> goes on. I think this will result in further expansion of the 'env'
library.
>> Fortunately, we don't require too much from POSIX - mostly pthreads, the C
>> standard library, and maybe a couple of other miscellaneous headers.
>>
>> There are other less obvious dependencies in SPDK. For instance, it also
>> depends on a relatively recent version of gcc or clang. This is mostly a
>> practical concern and not so much by intention. Our continuous integration
>> system (which I'll talk about at the summit - exciting developments here)
>> currently consists of 4 machines, 3 of which are running Fedora Linux 25 while
>> the other is running FreeBSD 11.0. These are very much up to date systems, so
>> we tend not to catch bugs on older systems today. We also use cutting edge C
>> features and tools, so we'll probably over time need to select a particular
>> minimum version of the C standard that we're going to develop to.
>>
>> I'm also concerned with the scope of the env library and I'd like to work
as
>> hard as possible to limit the size of the API. This is for two reasons - first
>> and foremost to minimize the number of calls to these wrapper functions so
>> that it doesn't impact performance. Performance really is everything and we
>> can't sacrifice that. This isn't shaping up to be much of a problem so
far
>> because very few calls to our dependencies happen within the I/O path.
>> Secondarily, I'd like to make it as straightforward as possible for users to
>> re-implement the env. The best way to make it easy to do that is to keep the
>> API small.
>>
>> I look forward to the discussions at the summit this week. We'd love to hear
>> about all of the different environments, platforms, and frameworks where SPDK
>> is being used and how we can make it easier to integrate SPDK wherever it is
>> valuable.
>>
>> Thanks,
>> Ben
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org <mailto:SPDK@lists.01.org>
>> https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org <mailto:SPDK@lists.01.org>
> https://lists.01.org/mailman/listinfo/spdk
1166
days inactive
1184
days old
7 comments
5 participants
participants (5)
-
John Meneghini -
Meneghini, John -
Ranjit Noronha -
Rodriguez, Edwin -
Walker, Benjamin