Thanks Ben, See inline.
On Tue, Sep 10, 2019 at 1:09 AM Walker, Benjamin <benjamin.walker(a)intel.com>
On Thu, 2019-09-05 at 08:40 +0300, Avner Taieb wrote:
> Hi Ben,
> Thanks for replying. I want the NVMe device to be able to DMA directly
> the network memory, also I don't want the network process to handle any
> events related to the nvme device , all nvme handling should be done in
> nvme process without any intervention of the network processes. Also
> few nvme processes and each one handles different device(s).
Does starting the network process as the primary and the NVMe process as
It works only if I made few changes in the spdk code. Because I don't want
the primary to handle any NVME events, I commented out some of the
initialisation of the NVME . In my changes the primary is initializes only
the memory zone for the nvme driver. In the NVME process (as secondary), I
changed the code so beside the memory it will initialise all PCI
I kind of defined new type of process NVME_PRIMARY, to separate the
It did the work, but I don't feel comfortable with these changes, I am sure
I missed something.
In order to DMA directly, the processes must be sharing memory,
so you can't entirely isolate them. Can you outline what the goals of the
separation are, so I can help you find a solution that meets the
You are absolutely right I'll try to explain better our needs:
1. our environment is Kubernetes , each process is running in different POD
2. We need one POD to be the networking processes, more than one POD to
serve as nvme driver, each for different nvme block device, and more than
one POD to serve as nvme target
3. We also want the different PODs to communicate through a very fast IPC,
we chose the spdk rings for this purpose.
4. To meet all the above the different process needs to have a shared
memory over the hugepages
Another option that cross my mind is to have separate hugepages for nvme
and for shared memory, launch all POD as primary. I didn't find an elegant
way of doing it, without massive changes in the spdk code. Is it possible
at all in the in the spkd to use different hugepages ?
Specifically, what does "exposed to the nvme devices" mean
in terms of
What I meant here is that the network process shouldn't access the pci bus,
and shouldn't handle any events that relates to the nvme device.
Events like device unplug and new device plug in.
Non of its functionalities requires so, we need it to handle only
I'll appreciate your input and suggestion to meet our requirements and
still align with the SPDK architecture.
My goal is not to change anything in the SPDK and to control this behaviour
only by configuration/parameter, unless of-course it meets the roadmap of
the spkd , in this case we can contribute.
Another option is to not start the network process and the NVMe
shared memory at all (pass them a different -i value). Then, open a Unix
socket between the two and for the memory you do want to share, pass a file
descriptor over the socket that the other side can mmap. That's a more
complicated set up, but does allow you to share memory selectively.
> On Wed, Sep 4, 2019 at 8:50 PM Walker, Benjamin <
> > On Sun, 2019-08-25 at 20:20 +0300, Avner Taieb wrote:
> > > Hi All,
> > > We are using spdk in a multi processes environment, each process has
> > > different roll, however the processes are communicating using the
> > memory
> > > pool and ring (using the SPDK api).
> > > Because we are using the share memory functionality, we need to
> > designate one
> > > of the processes as primary and all the other as secondary.
> > > Here start our problems: We have two type of processes one type
> > interfaces an
> > > nvme device and the other type interfaces the network.
> > > We need the network process to serve as primary for the shared
> > but we
> > > don't want it to be exposed to the nvme devices.
> > > * How the two type of processes can be initialise to meet our
> > requirements ?
> > > * one solution that cross my mind, was to mount two hugepages
> > directories, one
> > > for shared memory and one for nvme driver, I couldn't find a way
> > it.
> > Are you planning to do a memory copy of all data from memory owned by
> > network process to memory owned by the storage process? How is that
> > hand-off
> > going to work? Or do you need the NVMe device to be able to DMA
> > from
> > the network process memory?
> > Thanks,
> > Ben
> > > Thanks,
> > > Avner Taieb
> > > Reduxio
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > >
> > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> SPDK mailing list
SPDK mailing list