On Dec 13, 2016, at 8:49 AM, yzhu <yzhu(a)cs.fsu.edu> wrote:
Thanks for your quick reply. So I was wondering if spdk_malloc() is the only safe and
correct way to allocate buffer for writing to NVMe device via spdk_nvme_ns_cmd_write.
spdk_malloc() is the best way to allocate the buffer for use with the SPDK NVMe API. One
alternative is to pass your own 2MB-aligned pinned memory buffer, consisting of >= 2MB
huge pages, to the spdk_vtophys_register() function - this will insert the correct vtophys
mappings into the SPDK userspace page table. But spdk_vtophys_register() will not work
with buffers allocated from standard malloc() which will not be 2MB aligned, pinned, nor
guaranteed to be allocated from huge pages.
Previously, I tried to lock the buffer allocated by malloc() in RAM by mlock(), and
immediately write the buffer to NVMe device by spdk_nvme_ns_cmd_write. However, the write
failed because of the failure of converting virtual address to physical address. The error
was "could not find 2MB vfn 0x11 in DPDK mem config". Based on this, I was also
wondering how to correctly write to NVMe device via spdk if we still have to use the
buffer allocated by malloc()?
SPDK is not designed to handle buffers allocated by standard malloc(). Would using glibc
malloc hooks or overriding malloc() such that malloc()/free() calls get forwarded to
spdk_malloc()/spdk_free() help in your situation?
On 2016-12-12 23:46, Harris, James R wrote:
>> On Dec 12, 2016, at 9:13 PM, yzhu <yzhu(a)cs.fsu.edu> wrote:
>> Hi All,
>> Is there any way to directly use the memory allocated by the malloc() function?
The spdk_malloc() allocates a pinned memory buffer in huge page, and the
spdk_nvme_ns_cmd_write also uses the pinned memory for write. However, if we allocate a
memory buffer by malloc(), how could we directly write data to NVMe device by SPDK without
copying data to the buffer allocated by spdk_malloc()?
> Hi Yue,
> There is no safe way to use a malloc() buffer for DMA, since the
> buffer could be swapped out at any time. mlock() can be used to
> ensure the buffer stays present in RAM, but it does not guarantee the
> buffer will not get migrated to a different physical location.
> mlock() also has a lot of software overhead and is not something you
> would want to call for every I/O.
> SPDK mailing list
SPDK mailing list