Thanks for the quick response. I tried both reducing memory (setting mem_size option to 16 from 256) as well as dropping caches (echo 3 > /proc/sys/vm/drop_caches). It still fails. When I look at /proc/meminfo large amount of memory seems to be free. 

Thanks,
PR

cat /proc/meminfo
MemTotal:       263597240 kB
MemFree:        241112892 kB
MemAvailable:   241329340 kB
Buffers:               0 kB
Cached:          3244912 kB
SwapCached:        86024 kB
Active:         13700764 kB
Inactive:        3663916 kB
Active(anon):   13193844 kB
Inactive(anon):  3315936 kB
Active(file):     506920 kB
Inactive(file):   347980 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:        1696376 kB
Dirty:                92 kB
Writeback:             0 kB
AnonPages:      14034892 kB
Mapped:           333672 kB
Shmem:           2389980 kB
Slab:             847508 kB
SReclaimable:     245704 kB
SUnreclaim:       601804 kB
KernelStack:       59520 kB
PageTables:       151484 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    134944344 kB
Committed_AS:   46069756 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      804760 kB
VmallocChunk:   34224542720 kB
HardwareCorrupted:     0 kB
AnonHugePages:  11982848 kB
HugePages_Total:    1024
HugePages_Free:     1016
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      706604 kB
DirectMap2M:    73385984 kB
DirectMap1G:    196083712 kB

On Tue, Apr 24, 2018 at 4:59 PM, Verkamp, Daniel <daniel.verkamp@intel.com> wrote:

Hi,

 

The significant part of the first error message below is this:

 

> EAL: Can only reserve 259 pages from 1024 requested

> Current CONFIG_RTE_MAX_MEMSEG=256 is not enough

> Please either increase it or request less amount of memory.

> EAL: FATAL: Cannot init memory

 

This is saying that DPDK was unable to allocate hugepages to meet your requested memory size in the maximum number of segments (256) configured at DPDK compile time.  You can try rebooting or dropping caches (echo 3 > /proc/sys/vm/drop_caches) to try to defragment the available memory so DPDK will be able to find larger contiguous memory regions and thus require fewer total memory segments.  You can also try reducing the size of memory requested by your app – by default, DPDK will attempt to allocate all available hugepages.

 

Thanks,

-- Daniel

 

From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of PR PR
Sent: Tuesday, April 24, 2018 4:51 PM
To: spdk@lists.01.org
Subject: [SPDK] spdk in docker container

 

Hi, I am trying to run one of the spdk example (reactor_perf) in docker container. Our build system uses bazel so after compiling spdk natively, I have created a bazel target for linking with spdk (I have set the alwayslink option to 1 - https://github.com/spdk/spdk/issues/262). The container is run with privileged option on docker cmdline and has /dev bind mounted from host. Before running example app, I do run the scripts/setup.sh which succeeds. The example app (reactor_perf) is copied to new workspace location and compiled as a bazel target. For some reason, it fails to allocate huge pages. Some relevant outputs below. Appreciate any pointers.

 

Thanks,

PR 

 

========

After running scripts/setup.sh

========

 

cat /proc/sys/vm/nr_hugepages

1024

cat /sys/devices/system/node/node*/meminfo | fgrep Huge

Node 0 AnonHugePages:   8429568 kB

Node 0 HugePages_Total:   640

Node 0 HugePages_Free:    512

Node 0 HugePages_Surp:      0

Node 1 AnonHugePages:   3663872 kB

Node 1 HugePages_Total:   384

Node 1 HugePages_Free:    384

Node 1 HugePages_Surp:      0

 

=======

Run example app without mem_size option

=======

/ws/bazel-bin/spdk/spdk_test -t 10

Starting DPDK 17.11.0 initialization...

[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --file-prefix=spdk_pid28875 ]

EAL: Detected 72 lcore(s)

EAL: No free hugepages reported in hugepages-1048576kB

EAL: Probing VFIO support...

EAL: Can only reserve 259 pages from 1024 requested

Current CONFIG_RTE_MAX_MEMSEG=256 is not enough

Please either increase it or request less amount of memory.

EAL: FATAL: Cannot init memory

 

EAL: Cannot init memory

 

Failed to initialize DPDK

app.c: 372:spdk_app_start: *ERROR*: Unable to initialize SPDK env

 

=======

with mem_size option

=======

/ws/bazel-bin/spdk/spdk_test -t 10

Starting DPDK 17.11.0 initialization...

[ DPDK EAL parameters: reactor_perf -c 0x1 -m 256 --no-pci --file-prefix=spdk_pid29246 ]

EAL: Detected 72 lcore(s)

EAL: No free hugepages reported in hugepages-1048576kB

EAL: Probing VFIO support...

app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1

reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1

RING: Cannot reserve memory

reactor.c: 675:spdk_reactors_init: *NOTICE*: Event_mempool creation failed on preferred socket 0.

RING: Cannot reserve memory

reactor.c: 696:spdk_reactors_init: *ERROR*: spdk_event_mempool creation failed

app.c: 385:spdk_app_start: *ERROR*: Invalid reactor mask.

 

 

ps: I tried disabling huge pages but spdk seems to be crashing. Looks like this is not an option. 

===========

with —no-huge option

===========

Starting DPDK 17.11.0 initialization...

[ DPDK EAL parameters: reactor_perf -c 0x1 --no-pci --no-huge --file-prefix=spdk_pid23800 ]

EAL: Detected 72 lcore(s)

EAL: Probing VFIO support...

EAL: Started without hugepages support, physical addresses not available

Unable to unlink shared memory file: /var/run/.spdk_pid23800_hugepage_info. Error code: 2

invalid spdk_mem_register parameters, vaddr=0x7fbf2603b000 len=67108864

app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 1

reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1

Bus error (core dumped)

 


_______________________________________________
SPDK mailing list
SPDK@lists.01.org
https://lists.01.org/mailman/listinfo/spdk