Hi Benjamin thanks for your reply, I also answered inline below.
On 7/1/19 4:51 AM, Walker, Benjamin wrote:
> -----Original Message-----
> From: SPDK [mailto:firstname.lastname@example.org] On Behalf Of Feng Li
> Sent: Thursday, June 27, 2019 10:07 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] memory region requirement for spdk_mem_register
> I was trying to share DMA memory between two processes with SPDK in this way
> but failed:
> * Process A creates a memory region ( 2M aligned pinned memory from
> anonymous mmap with MAP_HUGETLB)
> * Process B then registers the above memory region from A, so that SPDK can
> use it as DMA memory for nvme driver.
> My current approach is (if there are better ways please let me know~):
> 1. Use a kernel module to store the starting physical address of this region
> shared from process A.
> 2. process B then also uses the kernel module to map this range of physical
> region to its virtual space (phys_addr->virt_addr_2) (using
> 3. process B uses spdk_mem_register to register virt_addr_2.
Why not just create files on a hugetlbfs filesystem? That's what DPDK does, and then
any process can mmap them. You don't need your own kernel module then.
That can be a good idea. I did see something similar in systemV IPC
implementation with hugepage support in the kernel tree
), where they
share the file descriptor across the processes to share the hugepage
But in the case of dax device mapped memory( NOT mapped from
hugetlbfs), i am not sure how to share it through hugetlbfs though..
Perhaps i need sth like hugetlbfs_unlinked_fd(), and I will also look
into how libpmem did this.
> My questions are:
> i. The remap_pfn_range in step 2 only gives me 4K mappings(but they are
> actually backed by hugepages allocated in process A). Does this violate "The
> memory region must map to pinned huge pages (2MB or greater"
> requirement in spdk_mem_register? (since the mappings are in 4K). This
SPDK's memory maps only go down to 2MB granularity today, mostly for space reasons.
But if you have 2MB chunks registered in 4K space, just register a 2MB aligned chunk and
it will be fine.
> ii. Currently the registration failed with -14 in process B. I traced into spdk and
> found out that the VFIO_IOMMU_MAP_DMA ioctrol failed in
> vtophys_iommu_map_dma (note that I had vfio enabled). Am i missing some
> steps (like in https://lists.01.org/pipermail/spdk/2018-May/001884.html)?
> failure might just be caused by the 4k mapping i had?
I'm not familiar enough with the ways in which VFIO_IOMMU_MAP_DMA can fail -
you'd have to dig into that driver I think to find what specifically -14 means.
it lead me to the
it seems like it's trying to pin a page already pinned from another
process. I will update once I can get more details.
> iii. if memory from process A is mmaped from a dax device, are there any pitfalls
> i need to be aware of(in the presence of vfio)?
You can use memory allocated on a DAX device in SPDK - that definitely works. I recommend
you use libpmem to manage the persistent memory. That's how we do it in SPDK when we
use persistent memory.
Sure, good suggestion!
> SPDK mailing list
SPDK mailing list