Hi,

 

> EAL: Can only reserve 259 pages from 1024 requested

> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough

 

I just noticed SPDK has a requirement of at least 5MB physically contiguous memory, which you apparently don't have. I'll try to lift this requirement and send out some patches. For now you can try setting hugepages=N option in host kernel boot options, so that the memory will be reserved before anything else and will likely be physically contiguous. If the memory is still fragmented, you could try switching to 1GB hugepages.

 

Out of curiosity - did you try to run the same app on the host system directly? We haven't tested SPDK in containers officially. Maybe this fragmentation is caused by docker somehow.

 

D.

 

From: SPDK [mailto:[email protected]] On Behalf Of PR PR
Sent: Wednesday, April 25, 2018 2:16 AM
To: Storage Performance Development Kit <[email protected]>
Subject: Re: [SPDK] spdk in docker container

 

Thanks for the quick response. I tried both reducing memory (setting mem_size option to 16 from 256) as well as dropping caches (echo 3 > /proc/sys/vm/drop_caches). It still fails. When I look at /proc/meminfo large amount of memory seems to be free. 

 

Thanks,

PR

 

cat /proc/meminfo

MemTotal:       263597240 kB

MemFree:        241112892 kB

MemAvailable:   241329340 kB

Buffers:               0 kB

Cached:          3244912 kB

SwapCached:        86024 kB

Active:         13700764 kB

Inactive:        3663916 kB

Active(anon):   13193844 kB

Inactive(anon):  3315936 kB

Active(file):     506920 kB

Inactive(file):   347980 kB

Unevictable:           0 kB

Mlocked:               0 kB

SwapTotal:       4194300 kB

SwapFree:        1696376 kB

Dirty:                92 kB

Writeback:             0 kB

AnonPages:      14034892 kB

Mapped:           333672 kB

Shmem:           2389980 kB

Slab:             847508 kB

SReclaimable:     245704 kB

SUnreclaim:       601804 kB

KernelStack:       59520 kB

PageTables:       151484 kB

NFS_Unstable:          0 kB

Bounce:                0 kB

WritebackTmp:          0 kB

CommitLimit:    134944344 kB

Committed_AS:   46069756 kB

VmallocTotal:   34359738367 kB

VmallocUsed:      804760 kB

VmallocChunk:   34224542720 kB

HardwareCorrupted:     0 kB

AnonHugePages:  11982848 kB

HugePages_Total:    1024

HugePages_Free:     1016

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

DirectMap4k:      706604 kB

DirectMap2M:    73385984 kB

DirectMap1G:    196083712 kB

 

On Tue, Apr 24, 2018 at 4:59 PM, Verkamp, Daniel <[email protected]> wrote:

Hi,

 

The significant part of the first error message below is this:

 

> EAL: Can only reserve 259 pages from 1024 requested

> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough

> Please either increase it or request less amount of memory.

> EAL: FATAL: Cannot init memory

 

This is saying that DPDK was unable to allocate hugepages to meet your requested memory size in the maximum number of segments (256) configured at DPDK compile time.  You can try rebooting or dropping caches (echo 3 > /proc/sys/vm/drop_caches) to try to defragment the available memory so DPDK will be able to find larger contiguous memory regions and thus require fewer total memory segments.  You can also try reducing the size of memory requested by your app – by default, DPDK will attempt to allocate all available hugepages.

 

Thanks,

-- Daniel

 

From: SPDK [mailto:[email protected]] On Behalf Of PR PR
Sent: Tuesday, April 24, 2018 4:51 PM
To:
[email protected]
Subject: [SPDK] spdk in docker container

 

Hi, I am trying to run one of the spdk example (reactor_perf) in docker container. Our build system uses bazel so after compiling spdk natively, I have created a bazel target for linking with spdk (I have set the alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The container is run with privileged option on docker cmdline and has /dev bind mounted from host. Before running example app, I do run the scripts/setup.sh which succeeds. The example app (reactor_perf) is copied to new workspace location and compiled as a bazel target. For some reason, it fails to allocate huge pages. Some relevant outputs below. Appreciate any pointers.

 

Thanks,

PR 

 

========

After running scripts/setup.sh

========

 

cat /proc/sys/vm/nr_hugepages

1024

cat /sys/devices/system/node/node*/meminfo | fgrep Huge

Node 0 AnonHugePages:   8429568 kB

Node 0 HugePages_Total:   640

Node 0 HugePages_Free:    512

Node 0 HugePages_Surp:      0

Node 1 AnonHugePages:   3663872 kB

Node 1 HugePages_Total:   384

Node 1 HugePages_Free:    384

Node 1 HugePages_Surp:      0

 

=======

Run example app without mem_size option

=======

/ws/bazel-bin/spdk/spdk_test -t 10

Starting DPDK 17.11.0 initialization...

[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --file-prefix=spdk_pid28875 ]

EAL: Detected 72 lcore(s)

EAL: No free hugepages reported in hugepages-1048576kB

EAL: Probing VFIO support...

EAL: Can only reserve 259 pages from 1024 requested

Current CONFIG_RTE_MAX_MEMSEG=256 is not enough

Please either increase it or request less amount of memory.

EAL: FATAL: Cannot init memory

 

EAL: Cannot init memory

 

Failed to initialize DPDK

app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env

 

=======

with mem_size option

=======

/ws/bazel-bin/spdk/spdk_test -t 10

Starting DPDK 17.11.0 initialization...

[ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci --file-prefix=spdk_pid29246 ]

EAL: Detected 72 lcore(s)

EAL: No free hugepages reported in hugepages-1048576kB

EAL: Probing VFIO support...

app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1

reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1

RING: Cannot reserve memory

reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed on preferred socket 0.

RING: Cannot reserve memory

reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation failed

app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.

 

 

ps: I tried disabling huge pages but spdk seems to be crashing. Looks like this is not an option. 

===========

with —no-huge option

===========

Starting DPDK 17.11.0 initialization...

[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge --file-prefix=spdk_pid23800 ]

EAL: Detected 72 lcore(s)

EAL: Probing VFIO support...

EAL: Started without hugepages support, physical addresses not available

Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info. Error code: 2

invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864

app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1

reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1

Bus error (core dumped)

 


_______________________________________________
SPDK mailing list
[email protected]
https://lists.01.org/mailman/listinfo/spdk