[ext4] b1b4705d54: filebench.sum_bytes_mb/s -20.2% regression
by kernel test robot
Greeting,
FYI, we noticed a -20.2% regression of filebench.sum_bytes_mb/s due to commit:
commit: b1b4705d54abedfd69dcdf42779c521aa1e0fbd3 ("ext4: introduce direct I/O read using iomap infrastructure")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: filebench
on test machine: 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory
with following parameters:
disk: 1HDD
fs: ext4
test: fivestreamreaddirect.f
cpufreq_governor: performance
ucode: 0x27
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen(a)intel.com>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/rootfs/tbox_group/test/testcase/ucode:
gcc-7/performance/1HDD/ext4/x86_64-rhel-7.6/debian-x86_64-2019-11-14.cgz/lkp-hsw-d01/fivestreamreaddirect.f/filebench/0x27
commit:
09edf4d381 ("ext4: introduce new callback for IOMAP_REPORT")
b1b4705d54 ("ext4: introduce direct I/O read using iomap infrastructure")
09edf4d381957b14 b1b4705d54abedfd69dcdf42779
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:4 25% 1:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
:4 25% 1:4 dmesg.WARNING:stack_recursion
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:da
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:db
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:dc
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:dd
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:de
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:df
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ea
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:eb
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ec
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ed
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ee
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ef
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fa
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fb
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fc
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fd
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fe
:4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ff
:4 25% 1:4 kmsg.file:libnss_files-#so_fault:filemap_fault_mmap:generic_file_mmap_readpage:simple_readpage
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a07fffffa0ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a17fffffa1ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a27fffffa2ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a37fffffa3ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a47fffffa4ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a57fffffa5ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a67fffffa6ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a77fffffa7ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a7fffff8aff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a7fffff9aff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a87fffffa8ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a97fffffa9ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#aa7fffffaaff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ab7fffffabff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ac7fffffacff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ad7fffffadff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ae7fffffaeff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#af7fffffafff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b07fffffb0ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b17fffffb1ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b27fffffb2ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b37fffffb3ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b47fffffb4ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b57fffffb5ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b67fffffb6ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b77fffffb7ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b7fffff8bff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b7fffff9bff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b87fffffb8ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b97fffffb9ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ba7fffffbaff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#bb7fffffbbff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#bc7fffffbcff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#bd7fffffbdff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#be7fffffbeff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#bf7fffffbfff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c07fffffc0ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c17fffffc1ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c27fffffc2ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c37fffffc3ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c47fffffc4ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c57fffffc5ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c67fffffc6ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c77fffffc7ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c7fffff8cff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c7fffff9cff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c87fffffc8ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c97fffffc9ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ca7fffffcaff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#cb7fffffcbff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#cc7fffffccff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#cd7fffffcdff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ce7fffffceff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#cf7fffffcfff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d07fffffd0ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d17fffffd1ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d27fffffd2ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d37fffffd3ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d47fffffd4ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d57fffffd5ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d67fffffd6ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d77fffffd7ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d7fffff8dff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d7fffff9dff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d87fffffd8ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d97fffffd9ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#da7fffffdaff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#db7fffffdbff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#dc7fffffdcff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#dd7fffffddff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#de7fffffdeff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#df7fffffdfff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e07fffffe0ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e17fffffe1ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e27fffffe2ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e37fffffe3ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e47fffffe4ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e57fffffe5ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e67fffffe6ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e77fffffe7ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e7fffff8eff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e7fffff9eff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e87fffffe8ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e97fffffe9ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ea7fffffeaff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#eb7fffffebff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ec7fffffecff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ed7fffffedff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ee7fffffeeff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ef7fffffefff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f07ffffff0ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f17ffffff1ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f27ffffff2ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f37ffffff3ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f47ffffff4ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f57ffffff5ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f67ffffff6ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f77ffffff7ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f7fffff8fff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f7fffff9fff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f87ffffff8ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f97ffffff9ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fa7ffffffaff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fb7ffffffbff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fc7ffffffcff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fd7ffffffdff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fe7ffffffeff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ff7fffffffff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff80ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff81ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff82ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff83ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff84ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff85ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff86ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff87ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff88ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff89ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff90ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff91ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff92ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff93ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff94ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff95ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff96ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff97ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff98ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff99ff
:4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffffffffff
%stddev %change %stddev
\ | \
58.30 ± 2% -20.2% 46.53 filebench.sum_bytes_mb/s
3503 ± 2% -20.2% 2796 filebench.sum_operations
58.39 ± 2% -20.2% 46.61 filebench.sum_operations/s
58.33 ± 2% -19.9% 46.75 filebench.sum_reads/s
84.34 ± 4% +27.1% 107.17 filebench.sum_time_ms/op
7174922 ± 2% -20.2% 5727328 filebench.time.file_system_inputs
10963 ± 2% -8.5% 10035 filebench.time.voluntary_context_switches
58.37 ± 3% -9.0 49.38 ± 3% mpstat.cpu.all.idle%
41.08 ± 5% +8.2 49.24 ± 2% mpstat.cpu.all.iowait%
3061 ± 6% +6.1% 3246 ± 4% turbostat.Bzy_MHz
5.48 ± 97% +260.7% 19.78 ± 30% turbostat.Pkg%pc2
27.33 ± 32% -52.0% 13.12 ± 19% sched_debug.cpu.nr_uninterruptible.max
-67.67 -60.8% -26.50 sched_debug.cpu.nr_uninterruptible.min
28.48 ± 30% -57.9% 11.98 ± 23% sched_debug.cpu.nr_uninterruptible.stddev
8210 ± 44% +76.1% 14454 ± 6% softirqs.CPU0.RCU
8200 ± 40% +70.2% 13954 ± 5% softirqs.CPU2.RCU
9772 ± 36% +59.6% 15597 ± 15% softirqs.CPU3.RCU
108.33 ± 27% +113.5% 231.25 ± 26% interrupts.CPU1.RES:Rescheduling_interrupts
1032 -23.9% 786.00 ± 30% interrupts.CPU5.CAL:Function_call_interrupts
229.33 ± 46% +50.1% 344.25 ± 31% interrupts.CPU5.NMI:Non-maskable_interrupts
229.33 ± 46% +50.1% 344.25 ± 31% interrupts.CPU5.PMI:Performance_monitoring_interrupts
457.00 ± 61% -74.2% 117.75 ± 41% interrupts.CPU5.RES:Rescheduling_interrupts
53.33 ± 60% +206.6% 163.50 ± 42% interrupts.CPU7.RES:Rescheduling_interrupts
58.99 ± 3% -14.9% 50.18 ± 3% iostat.cpu.idle
40.36 ± 5% +19.8% 48.35 ± 2% iostat.cpu.iowait
0.18 +386.3% 0.87 ±137% iostat.cpu.user
33012 ± 2% -19.7% 26520 iostat.sda.rkB/s
40.60 ± 6% +16.2% 47.18 ± 4% iostat.sda.w/s
43038 ± 4% +17.1% 50399 ± 2% iostat.sda.wkB/s
58.67 ± 3% -15.6% 49.50 ± 3% vmstat.cpu.id
40.00 ± 5% +19.4% 47.75 ± 2% vmstat.cpu.wa
33010 ± 2% -19.7% 26518 vmstat.io.bi
42837 ± 4% +17.2% 50200 ± 2% vmstat.io.bo
5519500 -15.5% 4664152 vmstat.memory.cache
2183603 +39.2% 3039439 ± 2% vmstat.memory.free
1065500 -19.2% 861375 slabinfo.buffer_head.active_objs
27320 -19.1% 22093 slabinfo.buffer_head.active_slabs
1065521 -19.1% 861671 slabinfo.buffer_head.num_objs
27320 -19.1% 22093 slabinfo.buffer_head.num_slabs
1306 +18.4% 1546 ± 7% slabinfo.kmalloc-512.active_objs
1306 +18.4% 1546 ± 7% slabinfo.kmalloc-512.num_objs
28277 -9.3% 25651 slabinfo.radix_tree_node.active_objs
5388173 -15.3% 4561843 meminfo.Cached
989375 ± 2% -22.6% 765315 ± 3% meminfo.Dirty
4310025 -19.2% 3483318 meminfo.Inactive
4261041 -19.4% 3434223 meminfo.Inactive(file)
162676 -13.7% 140448 meminfo.KReclaimable
2151883 ± 2% +39.5% 3000927 ± 2% meminfo.MemFree
5902816 -14.4% 5053772 meminfo.Memused
162676 -13.7% 140448 meminfo.SReclaimable
197794 -11.3% 175489 meminfo.Slab
80314276 ± 5% +6.5% 85566891 ± 3% perf-stat.i.branch-instructions
4810221 ± 5% +7.5% 5172535 perf-stat.i.cache-references
7.022e+08 ± 3% +6.9% 7.509e+08 ± 2% perf-stat.i.cpu-cycles
3.891e+08 ± 5% +6.3% 4.135e+08 ± 3% perf-stat.i.instructions
195094 ± 2% +10.8% 216142 perf-stat.i.node-loads
79563934 ± 5% +6.5% 84757473 ± 3% perf-stat.ps.branch-instructions
4765201 ± 5% +7.5% 5123552 perf-stat.ps.cache-references
6.956e+08 ± 3% +6.9% 7.438e+08 ± 2% perf-stat.ps.cpu-cycles
3.855e+08 ± 5% +6.3% 4.096e+08 ± 3% perf-stat.ps.instructions
193276 ± 2% +10.8% 214098 perf-stat.ps.node-loads
4.114e+10 ± 5% +5.3% 4.333e+10 ± 3% perf-stat.total.instructions
247900 ± 3% -22.9% 191187 ± 3% proc-vmstat.nr_dirty
1349353 -15.5% 1140545 proc-vmstat.nr_file_pages
535810 ± 2% +40.1% 750444 ± 2% proc-vmstat.nr_free_pages
1067252 -19.6% 858356 proc-vmstat.nr_inactive_file
40723 -13.8% 35112 proc-vmstat.nr_slab_reclaimable
1110681 ± 2% +17.1% 1300126 proc-vmstat.nr_written
1067251 -19.6% 858384 proc-vmstat.nr_zone_inactive_file
249604 ± 3% -22.7% 193058 ± 3% proc-vmstat.nr_zone_write_pending
660225 -3.8% 635276 proc-vmstat.pgalloc_dma32
871298 +2.7% 894930 proc-vmstat.pgalloc_normal
3587465 ± 2% -20.2% 2863670 proc-vmstat.pgpgin
4645372 ± 4% +16.6% 5414779 proc-vmstat.pgpgout
13.16 ± 14% -6.6 6.58 ± 52% perf-profile.calltrace.cycles-pp.unlinkat
13.06 ± 15% -6.6 6.49 ± 53% perf-profile.calltrace.cycles-pp.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat
13.06 ± 15% -6.6 6.49 ± 53% perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat
13.06 ± 15% -6.6 6.49 ± 53% perf-profile.calltrace.cycles-pp.ext4_evict_inode.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
13.06 ± 15% -6.6 6.50 ± 52% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlinkat
13.06 ± 15% -6.6 6.50 ± 52% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat
13.02 ± 14% -6.5 6.48 ± 52% perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.ext4_evict_inode.evict.do_unlinkat.do_syscall_64
10.39 ± 36% -4.6 5.78 ± 8% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel
11.21 ± 33% -4.6 6.64 ± 7% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
11.21 ± 33% -4.6 6.65 ± 7% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64
11.21 ± 33% -4.6 6.65 ± 7% perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64
5.71 ± 17% -2.8 2.95 ± 54% perf-profile.calltrace.cycles-pp.truncate_cleanup_page.truncate_inode_pages_range.ext4_evict_inode.evict.do_unlinkat
3.68 ± 14% -2.0 1.69 ± 50% perf-profile.calltrace.cycles-pp.release_pages.__pagevec_release.truncate_inode_pages_range.ext4_evict_inode.evict
3.70 ± 14% -2.0 1.74 ± 48% perf-profile.calltrace.cycles-pp.__pagevec_release.truncate_inode_pages_range.ext4_evict_inode.evict.do_unlinkat
2.90 ± 23% -1.6 1.32 ± 56% perf-profile.calltrace.cycles-pp.try_to_free_buffers.truncate_cleanup_page.truncate_inode_pages_range.ext4_evict_inode.evict
2.42 ± 20% -1.4 1.01 ± 75% perf-profile.calltrace.cycles-pp.delete_from_page_cache_batch.truncate_inode_pages_range.ext4_evict_inode.evict.do_unlinkat
2.02 ± 17% -1.3 0.68 ±104% perf-profile.calltrace.cycles-pp.free_unref_page_list.release_pages.__pagevec_release.truncate_inode_pages_range.ext4_evict_inode
1.33 ± 28% -0.9 0.45 ±100% perf-profile.calltrace.cycles-pp.free_buffer_head.try_to_free_buffers.truncate_cleanup_page.truncate_inode_pages_range.ext4_evict_inode
1.21 ± 22% -0.8 0.40 ±106% perf-profile.calltrace.cycles-pp.xas_store.delete_from_page_cache_batch.truncate_inode_pages_range.ext4_evict_inode.evict
1.17 ± 3% +0.4 1.57 ± 13% perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
0.35 ±141% +0.7 1.07 ± 37% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
3.71 ± 34% +1.6 5.34 ± 14% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
64.92 ± 5% +9.2 74.17 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
63.80 ± 4% +9.4 73.19 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
69.95 ± 5% +10.4 80.37 ± 5% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
70.02 ± 5% +10.4 80.46 ± 5% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
70.02 ± 5% +10.4 80.46 ± 5% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
13.16 ± 14% -6.6 6.58 ± 52% perf-profile.children.cycles-pp.unlinkat
13.06 ± 15% -6.6 6.49 ± 53% perf-profile.children.cycles-pp.do_unlinkat
13.06 ± 15% -6.6 6.49 ± 53% perf-profile.children.cycles-pp.ext4_evict_inode
13.03 ± 14% -6.5 6.49 ± 53% perf-profile.children.cycles-pp.truncate_inode_pages_range
13.07 ± 15% -6.4 6.64 ± 53% perf-profile.children.cycles-pp.evict
11.21 ± 33% -4.6 6.65 ± 7% perf-profile.children.cycles-pp.start_kernel
5.71 ± 17% -2.7 2.99 ± 54% perf-profile.children.cycles-pp.truncate_cleanup_page
3.74 ± 13% -1.9 1.85 ± 49% perf-profile.children.cycles-pp.__pagevec_release
3.75 ± 12% -1.9 1.88 ± 49% perf-profile.children.cycles-pp.release_pages
2.91 ± 22% -1.5 1.40 ± 53% perf-profile.children.cycles-pp.try_to_free_buffers
2.43 ± 19% -1.3 1.12 ± 57% perf-profile.children.cycles-pp.delete_from_page_cache_batch
2.03 ± 17% -1.1 0.95 ± 58% perf-profile.children.cycles-pp.free_unref_page_list
1.34 ± 27% -0.7 0.65 ± 45% perf-profile.children.cycles-pp.free_buffer_head
1.23 ± 20% -0.6 0.64 ± 50% perf-profile.children.cycles-pp.xas_store
0.78 ± 15% -0.4 0.36 ± 52% perf-profile.children.cycles-pp.jbd2_journal_try_to_free_buffers
0.73 ± 16% -0.4 0.33 ± 52% perf-profile.children.cycles-pp.jbd2_journal_grab_journal_head
0.50 ± 39% -0.4 0.15 ± 53% perf-profile.children.cycles-pp.unaccount_page_cache_page
0.42 ± 24% -0.3 0.14 ± 93% perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
0.76 ± 19% -0.2 0.53 ± 6% perf-profile.children.cycles-pp.lapic_next_deadline
0.32 ± 22% -0.2 0.14 ± 69% perf-profile.children.cycles-pp.free_unref_page_commit
0.25 ± 25% -0.2 0.08 ±102% perf-profile.children.cycles-pp.uncharge_page
0.19 ± 47% -0.1 0.07 ± 68% perf-profile.children.cycles-pp.__might_sleep
0.16 ± 28% -0.1 0.05 ±112% perf-profile.children.cycles-pp.__libc_fork
0.15 ± 22% -0.1 0.04 ±102% perf-profile.children.cycles-pp.xas_find
0.13 ± 12% -0.1 0.08 ± 34% perf-profile.children.cycles-pp.leave_mm
0.04 ± 71% +0.0 0.07 ± 14% perf-profile.children.cycles-pp.irq_work_needs_cpu
0.02 ±141% +0.1 0.09 ± 29% perf-profile.children.cycles-pp.__msecs_to_jiffies
0.04 ± 71% +0.1 0.12 ± 10% perf-profile.children.cycles-pp.cpuidle_not_available
0.09 ± 75% +0.1 0.18 ± 23% perf-profile.children.cycles-pp.__schedule
0.00 +0.1 0.10 ± 40% perf-profile.children.cycles-pp.link_path_walk
0.09 ± 41% +0.1 0.20 ± 19% perf-profile.children.cycles-pp.pm_qos_read_value
0.04 ± 71% +0.1 0.14 ± 52% perf-profile.children.cycles-pp.raise_softirq
0.04 ±141% +0.1 0.15 ± 33% perf-profile.children.cycles-pp.__test_set_page_writeback
0.00 +0.1 0.12 ± 17% perf-profile.children.cycles-pp.__delete_from_page_cache
0.05 ±141% +0.1 0.17 ± 60% perf-profile.children.cycles-pp.exit_mmap
0.19 ± 54% +0.1 0.32 ± 21% perf-profile.children.cycles-pp.hrtimer_get_next_event
0.05 ±141% +0.1 0.18 ± 64% perf-profile.children.cycles-pp.mmput
0.08 ± 30% +0.2 0.24 ± 33% perf-profile.children.cycles-pp.run_local_timers
0.05 ±141% +0.2 0.21 ± 41% perf-profile.children.cycles-pp.ext4_bio_write_page
0.14 ± 32% +0.2 0.30 ± 46% perf-profile.children.cycles-pp.update_rq_clock
0.15 ± 71% +0.2 0.33 ± 17% perf-profile.children.cycles-pp.rcu_idle_exit
0.38 ± 29% +0.2 0.56 ± 16% perf-profile.children.cycles-pp.sched_clock_cpu
0.14 ± 78% +0.2 0.33 ± 41% perf-profile.children.cycles-pp.mpage_map_and_submit_buffers
0.00 +0.2 0.20 ± 52% perf-profile.children.cycles-pp.invalidate_inode_pages2_range
0.07 ± 97% +0.2 0.30 ± 58% perf-profile.children.cycles-pp.__x64_sys_exit_group
0.07 ±141% +0.2 0.31 ± 37% perf-profile.children.cycles-pp.mpage_submit_page
0.38 ± 21% +0.3 0.64 ± 14% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.08 ±101% +0.3 0.34 ± 51% perf-profile.children.cycles-pp.do_group_exit
0.08 ±101% +0.3 0.34 ± 51% perf-profile.children.cycles-pp.do_exit
0.23 ± 29% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.do_writepages
0.23 ± 29% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.ext4_writepages
0.22 ± 35% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.wb_workfn
0.22 ± 35% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.wb_writeback
0.22 ± 35% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.__writeback_inodes_wb
0.22 ± 35% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.writeback_sb_inodes
0.22 ± 35% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.__writeback_single_inode
0.00 +0.3 0.32 ± 49% perf-profile.children.cycles-pp.ext4_file_read_iter
0.00 +0.3 0.32 ± 49% perf-profile.children.cycles-pp.iomap_dio_rw
1.30 ± 4% +0.4 1.72 ± 14% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.74 ± 41% +0.5 1.25 ± 33% perf-profile.children.cycles-pp.scheduler_tick
75.40 +4.7 80.05 ± 4% perf-profile.children.cycles-pp.cpuidle_enter
75.35 +4.7 80.02 ± 4% perf-profile.children.cycles-pp.cpuidle_enter_state
81.28 +5.9 87.18 ± 4% perf-profile.children.cycles-pp.do_idle
70.02 ± 5% +10.4 80.46 ± 5% perf-profile.children.cycles-pp.start_secondary
1.16 ± 22% -0.6 0.56 ± 63% perf-profile.self.cycles-pp.free_pcppages_bulk
0.73 ± 16% -0.4 0.33 ± 52% perf-profile.self.cycles-pp.jbd2_journal_grab_journal_head
0.46 ± 21% -0.3 0.16 ± 36% perf-profile.self.cycles-pp.xas_store
0.70 ± 5% -0.3 0.45 ± 38% perf-profile.self.cycles-pp.release_pages
0.29 ± 17% -0.2 0.07 ± 66% perf-profile.self.cycles-pp.free_unref_page_commit
0.32 ± 26% -0.2 0.13 ± 86% perf-profile.self.cycles-pp.workingset_update_node
0.18 ± 14% -0.1 0.06 ±100% perf-profile.self.cycles-pp.tick_nohz_tick_stopped
0.11 -0.1 0.05 ± 60% perf-profile.self.cycles-pp.leave_mm
0.04 ± 71% +0.0 0.07 ± 14% perf-profile.self.cycles-pp.irq_work_needs_cpu
0.00 +0.1 0.07 ± 14% perf-profile.self.cycles-pp.__msecs_to_jiffies
0.04 ± 71% +0.1 0.11 ± 26% perf-profile.self.cycles-pp.cpuidle_not_available
0.08 ± 81% +0.1 0.18 ± 23% perf-profile.self.cycles-pp.rcu_idle_exit
0.09 ± 41% +0.1 0.20 ± 19% perf-profile.self.cycles-pp.pm_qos_read_value
0.04 ± 71% +0.1 0.14 ± 52% perf-profile.self.cycles-pp.raise_softirq
0.04 ±141% +0.1 0.17 ± 41% perf-profile.self.cycles-pp.calc_global_load_tick
0.16 ± 12% +0.2 0.32 ± 27% perf-profile.self.cycles-pp.run_timer_softirq
0.08 ± 81% +0.2 0.26 ± 46% perf-profile.self.cycles-pp.update_rq_clock
0.17 ± 49% +0.2 0.36 ± 22% perf-profile.self.cycles-pp.load_balance
0.26 ± 8% +0.2 0.47 ± 40% perf-profile.self.cycles-pp.hrtimer_interrupt
filebench.time.file_system_inputs
7.6e+06 +-+---------------------------------------------------------------+
7.4e+06 +-++ .+ +.+ |
|: ++ + +.+.+.+.++. .+ : : +.+.+. +.|
7.2e+06 +-+ : : + + .++ + : : .+ + + + |
7e+06 +-+ : : + + + +: +.+ + :+ + |
6.8e+06 +-+ + + + + +.+ : |
6.6e+06 +-+ +: |
| + |
6.4e+06 +-+ |
6.2e+06 +-+ |
6e+06 +-+ |
5.8e+06 O-+ O O O O |
| O OO O O O O O OO O O O OO O O OO O O O O |
5.6e+06 +-+ O O |
5.4e+06 +-+---------------------------------------------------------------+
filebench.sum_operations
3700 +-+------------------------------------------------------------------+
3600 +-++ .+. +.+ |
|: +. + +.+.+.+.++. .+ : : +.+.+. +.|
3500 +-+ + : + + .+.+ + : : : + + : |
3400 +-+ :: + + :+ : :.+.+ + + + : |
| + + + + + +.+ + |
3300 +-+ :+ |
3200 +-+ + |
3100 +-+ |
| |
3000 +-+ |
2900 +-+ |
O O O O O O O O |
2800 +-O O OO O O O O O O O O O O O OO O O O O |
2700 +-+---------------------------------------------O--------------------+
filebench.sum_operations_s
62 +-+--------------------------------------------------------------------+
| + .+ .+ |
60 +-++ + +.+.+.+.+. .+ + : +. .+. +.|
58 +-+ +.+ : +.+ + .+.+ + : : + + + + : |
|: + : + + :+ : :.+.+ + :+ : |
56 +-+ + + + + + +.+ + |
54 +-+ + + |
| + |
52 +-+ |
50 +-+ |
| |
48 +-+ O |
46 O-O O O O O O O O O O O O O O O OO O O O O O O O O O |
| O O |
44 +-+--------------------------------------------------------------------+
filebench.sum_reads_s
62 +-+--------------------------------------------------------------------+
| + |
60 +-++ .++. .+. +.+ +. .+. .|
58 +-+ +.+ + +.+ +.+.+.+. .+.+ + : : + + + + + |
|: + + + + :+ : : .+ + :+ + |
56 +-+ + + + +.+ + +.+ + |
| : : |
54 +-+ : : |
| + |
52 +-+ |
50 +-+ |
| |
48 +-+ |
O O O O O O O O O O |
46 +-O-O-O-O---OO-O-----O---O-O-O-O-O-OO-O-O-O-O-----O--------------------+
filebench.sum_bytes_mb_s
62 +-+--------------------------------------------------------------------+
| + |
60 +-++ +.++.+.+.+.+. +.+ +. .+ +.|
58 +-+ +.+ : +.+.+. .+. : : + + + : |
|: + : + +. +. : : .+.+ +. +. : |
56 +-+ + + + + + +.+ + |
54 +-+ + + |
| + |
52 +-+ |
50 +-+ |
| |
48 +-+ O |
46 O-O O O O O O O O O O O O O O O O O O O O O O O O |
| O O O O |
44 +-+--------------------------------------------------------------------+
filebench.sum_time_ms_op
110 +-+-O------O----------------------O--------------O--------------------+
| O O OO O O O O O OO O O O O O O O OO O O O O |
105 O-+ O |
100 +-+ |
| |
95 +-+ |
| .+. |
90 +-+ + .+. .+ + .+. .++ + |
|: : : + .+.+.+ + + : +.+. + + + |
85 +-+ +. : +. +.+ :+ +. : +.+. + +.|
80 +-++ + +. .+. + : : + + |
| + + + : : |
75 +-+ : : |
| : |
70 +-+-------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
3 months, 2 weeks
- 5
- 10
[f2fs] fe1897eaa6: aim7.jobs-per-min -60.9% regression
by kernel test robot
Greeting,
FYI, we noticed a -60.9% regression of aim7.jobs-per-min due to commit:
commit: fe1897eaa6646f5a64a4cee0e6473ed9887d324b ("f2fs: fix to update time in lazytime mode")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: aim7
on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
with following parameters:
disk: 4BRD_12G
md: RAID1
fs: f2fs
test: sync_disk_rw
load: 200
cpufreq_governor: performance
ucode: 0x42e
test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen(a)intel.com>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
gcc-7/performance/4BRD_12G/f2fs/x86_64-rhel-7.6/200/RAID1/debian-x86_64-2019-11-14.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e
commit:
b145b0eb20 (" ARM and x86 bugfixes of all kinds. The most visible one is that migrating")
fe1897eaa6 ("f2fs: fix to update time in lazytime mode")
b145b0eb2031a620 fe1897eaa6646f5a64a4cee0e64
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
%stddev %change %stddev
\ | \
1208 -60.9% 473.12 aim7.jobs-per-min
992.70 +155.6% 2537 aim7.time.elapsed_time
992.70 +155.6% 2537 aim7.time.elapsed_time.max
1.078e+08 +60.0% 1.724e+08 aim7.time.file_system_outputs
9646301 -4.2% 9238717 aim7.time.involuntary_context_switches
24107 ± 3% +58.2% 38130 ± 17% aim7.time.minor_page_faults
3295 +202.3% 9963 aim7.time.system_time
26.75 +51.4% 40.49 aim7.time.user_time
5.02e+08 +210.5% 1.559e+09 aim7.time.voluntary_context_switches
9294452 ± 4% -15.7% 7835497 ± 16% numa-numastat.node0.local_node
9298417 ± 4% -15.7% 7840991 ± 16% numa-numastat.node0.numa_hit
3.66 ± 3% +4.2 7.82 ± 18% mpstat.cpu.all.idle%
0.01 ± 32% -0.0 0.00 ± 31% mpstat.cpu.all.soft%
0.06 -0.0 0.04 mpstat.cpu.all.usr%
3.85 ± 3% +105.0% 7.89 ± 17% iostat.cpu.idle
89.80 -4.9% 85.36 iostat.cpu.iowait
6.29 +6.6% 6.70 iostat.cpu.system
23968 -48.4% 12356 iostat.md0.w/s
53780 -37.1% 33824 iostat.md0.wkB/s
89.00 -4.8% 84.75 vmstat.cpu.wa
53779 -37.1% 33833 vmstat.io.bo
29311046 -28.6% 20935254 vmstat.memory.free
1026388 +19.4% 1225080 vmstat.system.cs
103289 +8.5% 112038 vmstat.system.in
24013 ± 29% -58.5% 9976 ±120% numa-meminfo.node0.Dirty
119019 ± 10% +44.6% 172139 ± 13% numa-meminfo.node1.Active
3503 ± 56% +464.8% 19786 ± 38% numa-meminfo.node1.Active(file)
40477 ± 16% -38.5% 24896 ± 47% numa-meminfo.node1.Dirty
19966008 ± 2% -29.5% 14084594 ± 44% numa-meminfo.node1.MemFree
13014643 ± 4% +45.2% 18896052 ± 32% numa-meminfo.node1.MemUsed
12018 ± 41% +184.8% 34228 ± 35% numa-meminfo.node1.Shmem
2.09e+10 +127.4% 4.752e+10 ± 4% cpuidle.C1.time
2.558e+08 +140.1% 6.143e+08 ± 4% cpuidle.C1.usage
3.017e+09 ± 10% +301.2% 1.211e+10 ± 9% cpuidle.C1E.time
18225961 ± 9% +430.1% 96624314 ± 6% cpuidle.C1E.usage
4.627e+08 ± 14% +147.5% 1.145e+09 ± 20% cpuidle.C6.time
802166 ± 17% +186.9% 2301023 ± 26% cpuidle.C6.usage
1.799e+10 +148.9% 4.476e+10 cpuidle.POLL.time
2.616e+08 ± 2% +246.4% 9.063e+08 ± 2% cpuidle.POLL.usage
6017 ± 29% -58.5% 2496 ±120% numa-vmstat.node0.nr_dirty
5742 ± 29% -65.0% 2011 ±120% numa-vmstat.node0.nr_zone_write_pending
874.50 ± 56% +465.6% 4946 ± 38% numa-vmstat.node1.nr_active_file
10139 ± 16% -38.6% 6224 ± 47% numa-vmstat.node1.nr_dirty
4992144 ± 2% -29.5% 3521200 ± 44% numa-vmstat.node1.nr_free_pages
3003 ± 41% +184.9% 8556 ± 35% numa-vmstat.node1.nr_shmem
874.50 ± 56% +465.6% 4946 ± 38% numa-vmstat.node1.nr_zone_active_file
9775 ± 16% -47.9% 5089 ± 47% numa-vmstat.node1.nr_zone_write_pending
2.558e+08 +140.1% 6.143e+08 ± 4% turbostat.C1
43.77 -4.8 38.98 ± 3% turbostat.C1%
18225642 ± 9% +430.2% 96624037 ± 6% turbostat.C1E
6.31 ± 10% +3.6 9.94 ± 9% turbostat.C1E%
795117 ± 17% +188.4% 2292823 ± 26% turbostat.C6
0.38 ± 42% -63.6% 0.14 ± 13% turbostat.CPU%c6
1.028e+08 +176.7% 2.844e+08 turbostat.IRQ
0.23 ± 33% -69.1% 0.07 ± 22% turbostat.Pkg%pc2
275064 +12.3% 308846 ± 4% meminfo.Active
8036 ± 21% +261.7% 29072 ± 52% meminfo.Active(file)
64400 -45.9% 34850 meminfo.Dirty
16481 +10.8% 18264 meminfo.Inactive(anon)
147067 +12.8% 165858 ± 2% meminfo.KReclaimable
29120417 -28.8% 20740075 meminfo.MemAvailable
29335165 -28.6% 20936804 meminfo.MemFree
36511133 +23.0% 44909491 meminfo.Memused
147067 +12.8% 165858 ± 2% meminfo.SReclaimable
36567 ± 6% +37.2% 50173 meminfo.Shmem
51580 -60.8% 20220 meminfo.max_used_kB
4.27 ± 7% -13.9% 3.67 ± 4% perf-stat.i.MPKI
0.71 ± 3% -0.0 0.67 perf-stat.i.branch-miss-rate%
17.93 +3.5 21.41 ± 2% perf-stat.i.cache-miss-rate%
1029337 +19.1% 1226025 perf-stat.i.context-switches
11421 ± 4% +41.1% 16120 ± 6% perf-stat.i.cpu-migrations
0.42 ± 2% +0.0 0.45 ± 6% perf-stat.i.dTLB-load-miss-rate%
0.11 ± 27% -0.0 0.08 ± 3% perf-stat.i.dTLB-store-miss-rate%
84.04 ± 2% -2.3 81.75 ± 2% perf-stat.i.iTLB-load-miss-rate%
7375571 +7.3% 7917282 perf-stat.i.iTLB-load-misses
3.91 ± 7% -9.2% 3.56 ± 4% perf-stat.overall.MPKI
17.90 +4.1 21.96 ± 3% perf-stat.overall.cache-miss-rate%
0.41 ± 2% +0.0 0.45 ± 6% perf-stat.overall.dTLB-load-miss-rate%
0.10 ± 27% -0.0 0.08 ± 3% perf-stat.overall.dTLB-store-miss-rate%
83.99 ± 2% -2.3 81.72 ± 2% perf-stat.overall.iTLB-load-miss-rate%
1028292 +19.2% 1225532 perf-stat.ps.context-switches
11410 ± 4% +41.2% 16114 ± 6% perf-stat.ps.cpu-migrations
7368087 +7.4% 7914119 perf-stat.ps.iTLB-load-misses
2.558e+13 ± 2% +157.0% 6.575e+13 ± 3% perf-stat.total.instructions
9128 ± 2% -25.3% 6821 ± 4% slabinfo.dmaengine-unmap-16.active_objs
9175 ± 2% -24.4% 6938 ± 4% slabinfo.dmaengine-unmap-16.num_objs
28309 +12.5% 31840 slabinfo.f2fs_extent_node.active_objs
28704 +13.0% 32429 slabinfo.f2fs_extent_node.num_objs
964.00 ± 5% -13.3% 836.00 ± 6% slabinfo.kmem_cache_node.active_objs
1008 ± 5% -12.7% 880.00 ± 6% slabinfo.kmem_cache_node.num_objs
13017 -33.1% 8707 slabinfo.nat_entry.active_objs
13036 -33.2% 8707 slabinfo.nat_entry.num_objs
433.75 ± 9% +32.4% 574.50 ± 2% slabinfo.nfs_read_data.active_objs
433.75 ± 9% +32.4% 574.50 ± 2% slabinfo.nfs_read_data.num_objs
697.50 ± 3% +48.6% 1036 ± 33% slabinfo.numa_policy.active_objs
697.50 ± 3% +48.6% 1036 ± 33% slabinfo.numa_policy.num_objs
153098 +22.3% 187296 slabinfo.radix_tree_node.active_objs
2733 +22.3% 3344 slabinfo.radix_tree_node.active_slabs
153098 +22.3% 187296 slabinfo.radix_tree_node.num_objs
2733 +22.3% 3344 slabinfo.radix_tree_node.num_slabs
542.00 ± 10% +29.5% 702.00 ± 6% slabinfo.xfrm_state.active_objs
542.00 ± 10% +29.5% 702.00 ± 6% slabinfo.xfrm_state.num_objs
66757 +4.8% 69942 proc-vmstat.nr_active_anon
2009 ± 21% +261.7% 7267 ± 52% proc-vmstat.nr_active_file
96.00 +3.9% 99.75 proc-vmstat.nr_anon_transparent_hugepages
13466177 +60.0% 21542226 proc-vmstat.nr_dirtied
16154 -46.0% 8719 proc-vmstat.nr_dirty
726643 -28.8% 517541 proc-vmstat.nr_dirty_background_threshold
1455064 -28.8% 1036348 proc-vmstat.nr_dirty_threshold
311402 +2.4% 318957 proc-vmstat.nr_file_pages
7332853 -28.6% 5234441 proc-vmstat.nr_free_pages
4120 +10.8% 4565 proc-vmstat.nr_inactive_anon
13939 -1.2% 13772 proc-vmstat.nr_kernel_stack
2845 -1.5% 2803 proc-vmstat.nr_page_table_pages
9142 ± 6% +37.2% 12542 proc-vmstat.nr_shmem
36767 +12.8% 41463 ± 2% proc-vmstat.nr_slab_reclaimable
13387539 +60.3% 21463781 proc-vmstat.nr_written
66757 +4.8% 69942 proc-vmstat.nr_zone_active_anon
2009 ± 21% +261.7% 7267 ± 52% proc-vmstat.nr_zone_active_file
4120 +10.8% 4565 proc-vmstat.nr_zone_inactive_anon
15494 -54.2% 7089 proc-vmstat.nr_zone_write_pending
582146 ±102% +628.1% 4238850 ± 6% proc-vmstat.numa_foreign
13506 ± 3% +114.5% 28974 ± 31% proc-vmstat.numa_hint_faults
8913 ± 6% +148.6% 22154 ± 35% proc-vmstat.numa_hint_faults_local
582146 ±102% +628.1% 4238850 ± 6% proc-vmstat.numa_miss
597885 ± 99% +611.6% 4254585 ± 6% proc-vmstat.numa_other
1014174 ± 20% -46.1% 547135 ± 57% proc-vmstat.pgalloc_dma32
16157005 +22.4% 19769138 proc-vmstat.pgalloc_normal
2516271 +152.8% 6360808 proc-vmstat.pgfault
4866589 +64.4% 8002519 proc-vmstat.pgfree
53553588 +60.3% 85856803 proc-vmstat.pgpgout
1.55 ± 4% +0.1 1.65 perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.wait_on_page_bit.f2fs_wait_on_page_writeback
1.58 ± 4% +0.1 1.69 perf-profile.calltrace.cycles-pp.schedule.io_schedule.wait_on_page_bit.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback
1.60 ± 4% +0.1 1.71 perf-profile.calltrace.cycles-pp.io_schedule.wait_on_page_bit.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file
0.75 ± 2% +0.2 0.91 ± 2% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.70 ± 4% -0.3 0.36 ± 7% perf-profile.children.cycles-pp.__generic_file_write_iter
0.69 ± 4% -0.3 0.35 ± 7% perf-profile.children.cycles-pp.generic_perform_write
0.57 ± 5% -0.3 0.25 ± 6% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
0.58 ± 5% -0.3 0.26 ± 7% perf-profile.children.cycles-pp.file_write_and_wait_range
0.57 ± 6% -0.3 0.25 ± 6% perf-profile.children.cycles-pp.do_writepages
0.57 ± 5% -0.3 0.25 ± 6% perf-profile.children.cycles-pp.f2fs_write_data_pages
0.64 ± 5% -0.3 0.34 ± 6% perf-profile.children.cycles-pp.f2fs_write_begin
0.57 ± 5% -0.2 0.32 ± 7% perf-profile.children.cycles-pp.f2fs_put_page
0.31 ± 10% -0.2 0.11 ± 7% perf-profile.children.cycles-pp.blk_finish_plug
0.31 ± 10% -0.2 0.11 ± 7% perf-profile.children.cycles-pp.blk_flush_plug_list
0.30 ± 11% -0.2 0.11 ± 6% perf-profile.children.cycles-pp.raid1_unplug
0.26 -0.1 0.14 ± 12% perf-profile.children.cycles-pp.f2fs_write_cache_pages
0.22 ± 20% -0.1 0.11 ± 41% perf-profile.children.cycles-pp.brd_do_bvec
0.14 ± 12% -0.1 0.03 ±100% perf-profile.children.cycles-pp.submit_bio_wait
0.21 ± 10% -0.1 0.11 ± 10% perf-profile.children.cycles-pp.submit_bio
0.23 ± 17% -0.1 0.13 ± 31% perf-profile.children.cycles-pp.worker_thread
0.20 ± 3% -0.1 0.11 ± 14% perf-profile.children.cycles-pp.__write_data_page
0.14 ± 15% -0.1 0.05 ± 9% perf-profile.children.cycles-pp.__submit_flush_wait
0.17 ± 12% -0.1 0.09 ± 12% perf-profile.children.cycles-pp.md_make_request
0.16 ± 11% -0.1 0.08 ± 14% perf-profile.children.cycles-pp.md_handle_request
0.15 ± 9% -0.1 0.07 ± 11% perf-profile.children.cycles-pp.raid1_make_request
0.14 ± 17% -0.1 0.06 ± 11% perf-profile.children.cycles-pp.f2fs_issue_flush
0.14 ± 6% -0.1 0.08 ± 19% perf-profile.children.cycles-pp.f2fs_do_write_data_page
0.10 ± 33% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.reschedule_interrupt
0.09 ± 37% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.scheduler_ipi
0.06 ± 6% +0.0 0.08 ± 10% perf-profile.children.cycles-pp.up_read
0.10 ± 4% +0.0 0.12 ± 13% perf-profile.children.cycles-pp.pick_next_task_idle
0.20 ± 2% +0.0 0.23 ± 5% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.16 ± 6% +0.0 0.19 ± 4% perf-profile.children.cycles-pp.irq_exit
0.01 ±173% +0.1 0.07 ± 7% perf-profile.children.cycles-pp.put_prev_task_fair
0.27 ± 3% +0.1 0.33 ± 4% perf-profile.children.cycles-pp.tick_nohz_next_event
0.35 ± 3% +0.1 0.42 ± 3% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
1.60 ± 4% +0.1 1.72 perf-profile.children.cycles-pp.io_schedule
0.78 ± 2% +0.1 0.92 ± 2% perf-profile.children.cycles-pp.menu_select
0.06 ± 6% +0.0 0.08 ± 10% perf-profile.self.cycles-pp.up_read
0.08 ± 5% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.cpuidle_enter_state
0.35 ± 5% +0.0 0.39 ± 5% perf-profile.self.cycles-pp.f2fs_wait_on_node_pages_writeback
0.35 ± 4% +0.1 0.41 ± 3% perf-profile.self.cycles-pp.menu_select
8096 ± 28% +278.9% 30679 ± 20% sched_debug.cfs_rq:/.MIN_vruntime.avg
158062 ± 17% +299.5% 631442 ± 8% sched_debug.cfs_rq:/.MIN_vruntime.max
32629 ± 22% +289.7% 127148 ± 13% sched_debug.cfs_rq:/.MIN_vruntime.stddev
44053 +192.3% 128774 sched_debug.cfs_rq:/.exec_clock.avg
140530 ± 9% +377.8% 671471 ± 29% sched_debug.cfs_rq:/.exec_clock.max
33108 ± 2% +109.9% 69484 ± 2% sched_debug.cfs_rq:/.exec_clock.min
20291 ± 7% +386.1% 98642 ± 21% sched_debug.cfs_rq:/.exec_clock.stddev
0.25 ± 82% -88.4% 0.03 ±131% sched_debug.cfs_rq:/.load_avg.min
8096 ± 28% +278.9% 30679 ± 20% sched_debug.cfs_rq:/.max_vruntime.avg
158062 ± 17% +299.5% 631442 ± 8% sched_debug.cfs_rq:/.max_vruntime.max
32629 ± 22% +289.7% 127148 ± 13% sched_debug.cfs_rq:/.max_vruntime.stddev
184170 +218.6% 586859 ± 2% sched_debug.cfs_rq:/.min_vruntime.avg
251516 ± 4% +254.5% 891702 ± 11% sched_debug.cfs_rq:/.min_vruntime.max
165614 +131.3% 383025 ± 2% sched_debug.cfs_rq:/.min_vruntime.min
15286 ± 9% +1057.3% 176903 ± 10% sched_debug.cfs_rq:/.min_vruntime.stddev
0.67 ± 6% +45.5% 0.98 ± 9% sched_debug.cfs_rq:/.nr_spread_over.avg
0.74 ± 20% +32.4% 0.98 ± 13% sched_debug.cfs_rq:/.nr_spread_over.stddev
3.43 ± 29% -60.7% 1.35 ±100% sched_debug.cfs_rq:/.removed.load_avg.avg
75.10 ± 34% -84.1% 11.91 ±100% sched_debug.cfs_rq:/.removed.load_avg.max
15.24 ± 23% -75.5% 3.74 ±100% sched_debug.cfs_rq:/.removed.load_avg.stddev
157.58 ± 29% -60.5% 62.19 ±100% sched_debug.cfs_rq:/.removed.runnable_sum.avg
3459 ± 34% -84.0% 552.75 ± 99% sched_debug.cfs_rq:/.removed.runnable_sum.max
700.79 ± 23% -75.4% 172.66 ±100% sched_debug.cfs_rq:/.removed.runnable_sum.stddev
34.81 ± 42% -87.0% 4.53 ±100% sched_debug.cfs_rq:/.removed.util_avg.max
5.89 ± 31% -79.0% 1.23 ±100% sched_debug.cfs_rq:/.removed.util_avg.stddev
59642 ± 58% +506.4% 361657 ± 61% sched_debug.cfs_rq:/.spread0.max
15286 ± 9% +1057.2% 176904 ± 10% sched_debug.cfs_rq:/.spread0.stddev
15.81 ± 15% +28.7% 20.34 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.avg
553.04 ± 15% +42.8% 789.60 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.max
83.37 ± 15% +37.7% 114.78 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.stddev
130721 -17.9% 107295 sched_debug.cpu.avg_idle.avg
32650 ± 7% -16.9% 27145 ± 6% sched_debug.cpu.avg_idle.min
509212 +151.7% 1281639 sched_debug.cpu.clock.avg
509215 +151.7% 1281642 sched_debug.cpu.clock.max
509209 +151.7% 1281636 sched_debug.cpu.clock.min
509212 +151.7% 1281639 sched_debug.cpu.clock_task.avg
509215 +151.7% 1281642 sched_debug.cpu.clock_task.max
509209 +151.7% 1281636 sched_debug.cpu.clock_task.min
480.69 ± 2% +89.2% 909.42 ± 3% sched_debug.cpu.curr->pid.avg
14037 +135.4% 33040 sched_debug.cpu.curr->pid.max
2146 ± 2% +125.6% 4843 sched_debug.cpu.curr->pid.stddev
0.12 ± 5% +10.2% 0.13 ± 3% sched_debug.cpu.nr_running.avg
10174129 +208.2% 31360192 sched_debug.cpu.nr_switches.avg
10849045 +290.1% 42321541 ± 3% sched_debug.cpu.nr_switches.max
564008 ± 15% +1719.4% 10261545 ± 8% sched_debug.cpu.nr_switches.stddev
-87.16 +262.4% -315.83 sched_debug.cpu.nr_uninterruptible.min
66.84 ± 14% +143.1% 162.48 ± 21% sched_debug.cpu.nr_uninterruptible.stddev
10172460 +208.3% 31358619 sched_debug.cpu.sched_count.avg
10847113 +290.1% 42319816 ± 3% sched_debug.cpu.sched_count.max
563945 ± 15% +1719.6% 10261603 ± 8% sched_debug.cpu.sched_count.stddev
4957584 +211.1% 15425344 sched_debug.cpu.sched_goidle.avg
5286410 +291.7% 20709361 ± 3% sched_debug.cpu.sched_goidle.max
272603 ± 14% +1717.5% 4954528 ± 7% sched_debug.cpu.sched_goidle.stddev
5120024 +209.4% 15841794 sched_debug.cpu.ttwu_count.avg
58965381 ± 14% +567.2% 3.934e+08 ± 33% sched_debug.cpu.ttwu_count.max
132694 ± 2% -41.0% 78274 ± 5% sched_debug.cpu.ttwu_count.min
11266172 ± 8% +482.6% 65635267 ± 21% sched_debug.cpu.ttwu_count.stddev
114955 +26.1% 144930 ± 3% sched_debug.cpu.ttwu_local.max
83595 ± 4% -53.0% 39284 ± 30% sched_debug.cpu.ttwu_local.min
5967 ± 16% +460.9% 33468 ± 9% sched_debug.cpu.ttwu_local.stddev
509209 +151.7% 1281636 sched_debug.cpu_clk
506629 +152.5% 1279056 sched_debug.ktime
509627 +151.6% 1282053 sched_debug.sched_clk
18876 +138.7% 45051 ± 4% softirqs.CPU0.RCU
131251 ± 5% +153.2% 332324 ± 8% softirqs.CPU0.SCHED
327888 ± 3% +181.5% 923135 ± 10% softirqs.CPU0.TIMER
19003 ± 4% +121.7% 42132 ± 12% softirqs.CPU1.RCU
125587 ± 4% +130.3% 289267 ± 22% softirqs.CPU1.SCHED
331057 ± 2% +205.9% 1012636 ± 18% softirqs.CPU1.TIMER
19567 ± 5% +137.8% 46538 ± 2% softirqs.CPU10.RCU
131681 +161.3% 344047 softirqs.CPU10.SCHED
318774 +177.5% 884714 ± 8% softirqs.CPU10.TIMER
17876 ± 5% +158.8% 46258 softirqs.CPU11.RCU
128299 ± 4% +167.5% 343196 softirqs.CPU11.SCHED
320847 ± 3% +166.4% 854607 ± 8% softirqs.CPU11.TIMER
19129 ± 4% +121.1% 42292 ± 4% softirqs.CPU12.RCU
135251 +172.0% 367942 softirqs.CPU12.SCHED
310663 +201.8% 937514 ± 24% softirqs.CPU12.TIMER
17993 +143.9% 43882 ± 3% softirqs.CPU13.RCU
137747 ± 4% +157.5% 354715 ± 4% softirqs.CPU13.SCHED
359507 ± 23% +126.2% 813048 ± 13% softirqs.CPU13.TIMER
18173 +141.9% 43961 ± 3% softirqs.CPU14.RCU
134564 +163.8% 355019 ± 4% softirqs.CPU14.SCHED
310599 ± 2% +160.3% 808494 ± 13% softirqs.CPU14.TIMER
18180 +139.4% 43524 ± 3% softirqs.CPU15.RCU
134993 +160.8% 352061 ± 6% softirqs.CPU15.SCHED
308373 +164.7% 816152 ± 12% softirqs.CPU15.TIMER
19918 +144.6% 48716 softirqs.CPU16.RCU
134609 +162.6% 353418 ± 4% softirqs.CPU16.SCHED
307916 +163.0% 809772 ± 13% softirqs.CPU16.TIMER
19672 ± 2% +148.8% 48938 ± 3% softirqs.CPU17.RCU
132926 ± 2% +165.9% 353452 ± 4% softirqs.CPU17.SCHED
314316 +157.0% 807880 ± 13% softirqs.CPU17.TIMER
20021 ± 3% +141.2% 48298 ± 2% softirqs.CPU18.RCU
133347 ± 2% +164.0% 352038 ± 5% softirqs.CPU18.SCHED
312605 +158.6% 808358 ± 13% softirqs.CPU18.TIMER
19458 +139.3% 46570 ± 6% softirqs.CPU19.RCU
134001 +150.1% 335170 ± 14% softirqs.CPU19.SCHED
310521 +171.0% 841639 ± 14% softirqs.CPU19.TIMER
19131 ± 3% +138.3% 45586 ± 3% softirqs.CPU2.RCU
129111 ± 4% +154.4% 328419 ± 9% softirqs.CPU2.SCHED
329064 ± 3% +172.7% 897239 ± 10% softirqs.CPU2.TIMER
19249 ± 2% +145.5% 47257 ± 3% softirqs.CPU20.RCU
132532 +167.8% 354983 ± 3% softirqs.CPU20.SCHED
310659 +163.4% 818268 ± 13% softirqs.CPU20.TIMER
19473 +142.2% 47163 ± 4% softirqs.CPU21.RCU
133856 +166.0% 356005 ± 4% softirqs.CPU21.SCHED
308960 +164.3% 816637 ± 13% softirqs.CPU21.TIMER
19216 ± 2% +146.1% 47292 ± 3% softirqs.CPU22.RCU
133039 +165.4% 353140 ± 4% softirqs.CPU22.SCHED
307621 +166.0% 818260 ± 13% softirqs.CPU22.TIMER
19244 ± 2% +145.9% 47314 ± 2% softirqs.CPU23.RCU
133897 +163.1% 352326 ± 5% softirqs.CPU23.SCHED
310036 ± 2% +161.6% 811061 ± 13% softirqs.CPU23.TIMER
18954 ± 4% +143.9% 46235 ± 4% softirqs.CPU24.RCU
128754 ± 3% +157.8% 331927 ± 7% softirqs.CPU24.SCHED
323118 +169.7% 871609 ± 8% softirqs.CPU24.TIMER
18942 ± 6% +136.0% 44695 ± 3% softirqs.CPU25.RCU
129580 ± 3% +162.2% 339786 ± 3% softirqs.CPU25.SCHED
328907 +168.0% 881566 ± 7% softirqs.CPU25.TIMER
18913 ± 2% +147.2% 46757 ± 2% softirqs.CPU26.RCU
124714 ± 2% +176.4% 344692 softirqs.CPU26.SCHED
328463 ± 4% +170.2% 887463 ± 8% softirqs.CPU26.TIMER
19015 +137.2% 45107 ± 5% softirqs.CPU27.RCU
128551 ± 6% +152.0% 323916 ± 13% softirqs.CPU27.SCHED
325384 ± 2% +193.3% 954266 ± 13% softirqs.CPU27.TIMER
19094 +149.9% 47725 ± 2% softirqs.CPU28.RCU
131563 +161.9% 344579 ± 2% softirqs.CPU28.SCHED
319151 +176.7% 883105 ± 9% softirqs.CPU28.TIMER
19050 ± 4% +145.4% 46756 ± 2% softirqs.CPU29.RCU
124211 ± 6% +176.0% 342763 ± 2% softirqs.CPU29.SCHED
329526 ± 4% +158.9% 853085 ± 8% softirqs.CPU29.TIMER
22721 ± 8% +121.7% 50370 ± 4% softirqs.CPU3.RCU
132516 ± 2% +156.9% 340453 ± 4% softirqs.CPU3.SCHED
323552 ± 2% +174.3% 887612 ± 9% softirqs.CPU3.TIMER
19911 ± 3% +137.4% 47276 softirqs.CPU30.RCU
130778 +161.4% 341879 softirqs.CPU30.SCHED
317264 +181.0% 891407 ± 8% softirqs.CPU30.TIMER
19232 ± 4% +144.4% 47012 softirqs.CPU31.RCU
126109 ± 10% +174.9% 346628 ± 2% softirqs.CPU31.SCHED
329439 ± 4% +169.4% 887674 ± 9% softirqs.CPU31.TIMER
18139 ± 4% +141.4% 43795 ± 4% softirqs.CPU32.RCU
125566 ± 4% +170.9% 340112 ± 4% softirqs.CPU32.SCHED
328833 +184.0% 934027 ± 9% softirqs.CPU32.TIMER
19309 ± 10% +128.3% 44077 ± 3% softirqs.CPU33.RCU
128438 ± 3% +167.7% 343879 ± 2% softirqs.CPU33.SCHED
320949 +172.2% 873631 ± 9% softirqs.CPU33.TIMER
18041 ± 4% +146.1% 44399 ± 2% softirqs.CPU34.RCU
129791 +165.0% 343914 ± 2% softirqs.CPU34.SCHED
319403 ± 2% +177.3% 885711 ± 8% softirqs.CPU34.TIMER
17991 ± 5% +146.2% 44290 ± 2% softirqs.CPU35.RCU
137112 ± 7% +148.6% 340824 ± 3% softirqs.CPU35.SCHED
416552 ± 21% +119.5% 914435 ± 8% softirqs.CPU35.TIMER
20036 ± 4% +138.7% 47823 ± 4% softirqs.CPU36.RCU
138048 ± 3% +165.7% 366854 ± 7% softirqs.CPU36.SCHED
359442 ± 22% +196.1% 1064343 ± 19% softirqs.CPU36.TIMER
19421 ± 5% +153.4% 49203 ± 2% softirqs.CPU37.RCU
135462 +158.7% 350500 ± 6% softirqs.CPU37.SCHED
305531 +166.8% 815287 ± 12% softirqs.CPU37.TIMER
19963 +150.5% 50014 softirqs.CPU38.RCU
132257 +167.5% 353848 ± 5% softirqs.CPU38.SCHED
311979 ± 2% +160.3% 812118 ± 12% softirqs.CPU38.TIMER
20119 ± 2% +143.6% 49011 ± 2% softirqs.CPU39.RCU
135450 +161.9% 354713 ± 4% softirqs.CPU39.SCHED
307797 +164.3% 813377 ± 12% softirqs.CPU39.TIMER
20593 ± 9% +132.1% 47796 ± 5% softirqs.CPU4.RCU
129881 +159.2% 336643 ± 3% softirqs.CPU4.SCHED
321940 +175.0% 885297 ± 8% softirqs.CPU4.TIMER
19874 ± 2% +149.6% 49611 softirqs.CPU40.RCU
134660 +161.9% 352636 ± 5% softirqs.CPU40.SCHED
309870 +160.6% 807374 ± 13% softirqs.CPU40.TIMER
19897 ± 2% +147.2% 49186 ± 2% softirqs.CPU41.RCU
131978 ± 2% +167.4% 352864 ± 5% softirqs.CPU41.SCHED
315198 +157.6% 812013 ± 12% softirqs.CPU41.TIMER
19988 ± 2% +143.9% 48749 ± 2% softirqs.CPU42.RCU
133326 ± 2% +165.6% 354135 ± 4% softirqs.CPU42.SCHED
313042 +159.5% 812298 ± 12% softirqs.CPU42.TIMER
19723 ± 2% +141.9% 47704 ± 4% softirqs.CPU43.RCU
131944 +165.7% 350566 ± 5% softirqs.CPU43.SCHED
314428 +162.9% 826595 ± 13% softirqs.CPU43.TIMER
19824 ± 2% +143.0% 48177 ± 4% softirqs.CPU44.RCU
132047 ± 2% +157.8% 340362 ± 12% softirqs.CPU44.SCHED
312615 ± 2% +165.2% 829094 ± 12% softirqs.CPU44.TIMER
20140 ± 2% +138.6% 48049 ± 7% softirqs.CPU45.RCU
133846 +151.7% 336901 ± 14% softirqs.CPU45.SCHED
309086 ± 2% +170.8% 837094 ± 12% softirqs.CPU45.TIMER
20103 ± 3% +138.8% 48009 ± 3% softirqs.CPU46.RCU
133860 +154.1% 340108 ± 11% softirqs.CPU46.SCHED
311323 +166.1% 828498 ± 13% softirqs.CPU46.TIMER
19628 ± 2% +149.0% 48881 ± 2% softirqs.CPU47.RCU
132835 ± 2% +164.6% 351487 ± 5% softirqs.CPU47.SCHED
309264 +162.8% 812666 ± 13% softirqs.CPU47.TIMER
19773 ± 2% +139.0% 47263 ± 2% softirqs.CPU5.RCU
130891 ± 2% +163.7% 345185 softirqs.CPU5.SCHED
323979 ± 2% +182.2% 914392 ± 8% softirqs.CPU5.TIMER
19904 ± 3% +141.4% 48045 ± 4% softirqs.CPU6.RCU
131836 +160.9% 344009 softirqs.CPU6.SCHED
316150 +181.7% 890614 ± 8% softirqs.CPU6.TIMER
19077 ± 3% +145.1% 46751 ± 3% softirqs.CPU7.RCU
127906 ± 2% +157.6% 329548 ± 5% softirqs.CPU7.SCHED
325087 +181.6% 915494 ± 9% softirqs.CPU7.TIMER
20183 ± 2% +114.9% 43380 ± 10% softirqs.CPU8.RCU
131974 +118.3% 288124 ± 30% softirqs.CPU8.SCHED
319626 +190.1% 927161 ± 13% softirqs.CPU8.TIMER
19292 ± 3% +137.6% 45839 ± 3% softirqs.CPU9.RCU
129715 +158.9% 335788 ± 4% softirqs.CPU9.SCHED
321209 +178.0% 893035 ± 8% softirqs.CPU9.TIMER
931061 +140.9% 2242787 softirqs.RCU
6322319 +160.8% 16487173 softirqs.SCHED
15417545 +169.7% 41581783 ± 8% softirqs.TIMER
853.75 ± 12% +139.9% 2048 ± 7% interrupts.35:PCI-MSI.2621441-edge.eth0-TxRx-0
586.00 ± 14% +130.1% 1348 ± 5% interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
651.00 ± 9% +136.3% 1538 ± 9% interrupts.37:PCI-MSI.2621443-edge.eth0-TxRx-2
565.25 ± 14% +129.7% 1298 interrupts.38:PCI-MSI.2621444-edge.eth0-TxRx-3
591.00 ± 14% +123.8% 1322 ± 2% interrupts.39:PCI-MSI.2621445-edge.eth0-TxRx-4
660.25 ± 13% +111.4% 1396 ± 7% interrupts.40:PCI-MSI.2621446-edge.eth0-TxRx-5
529.50 ± 4% +227.1% 1732 ± 14% interrupts.41:PCI-MSI.2621447-edge.eth0-TxRx-6
585.75 ± 16% +163.2% 1541 ± 10% interrupts.42:PCI-MSI.2621448-edge.eth0-TxRx-7
404968 +145.6% 994665 ± 3% interrupts.CAL:Function_call_interrupts
8396 ± 6% +133.7% 19627 ± 6% interrupts.CPU0.CAL:Function_call_interrupts
1988743 +155.3% 5077644 interrupts.CPU0.LOC:Local_timer_interrupts
8698 ± 5% +121.4% 19255 ± 2% interrupts.CPU1.CAL:Function_call_interrupts
1986989 +155.5% 5077206 interrupts.CPU1.LOC:Local_timer_interrupts
8487 ± 5% +132.3% 19713 ± 8% interrupts.CPU10.CAL:Function_call_interrupts
1988375 +155.4% 5077322 interrupts.CPU10.LOC:Local_timer_interrupts
8770 ± 5% +149.6% 21889 ± 3% interrupts.CPU11.CAL:Function_call_interrupts
1987964 +155.4% 5077484 interrupts.CPU11.LOC:Local_timer_interrupts
78678 ± 8% +541.4% 504605 ±122% interrupts.CPU11.RES:Rescheduling_interrupts
8731 ± 9% +119.5% 19167 ± 6% interrupts.CPU12.CAL:Function_call_interrupts
1988222 +155.4% 5077524 interrupts.CPU12.LOC:Local_timer_interrupts
7960 ± 3% +153.5% 20176 ± 9% interrupts.CPU13.CAL:Function_call_interrupts
1988667 +155.3% 5076359 interrupts.CPU13.LOC:Local_timer_interrupts
8518 ± 6% +155.0% 21723 ± 7% interrupts.CPU14.CAL:Function_call_interrupts
1987350 +155.5% 5077482 interrupts.CPU14.LOC:Local_timer_interrupts
37.00 ± 31% +124.3% 83.00 ± 24% interrupts.CPU14.TLB:TLB_shootdowns
8831 ± 4% +137.4% 20964 ± 10% interrupts.CPU15.CAL:Function_call_interrupts
1987501 +155.5% 5077129 interrupts.CPU15.LOC:Local_timer_interrupts
3708 ± 27% -63.0% 1370 ±103% interrupts.CPU15.NMI:Non-maskable_interrupts
3708 ± 27% -63.0% 1370 ±103% interrupts.CPU15.PMI:Performance_monitoring_interrupts
23.25 ± 24% +253.8% 82.25 ± 28% interrupts.CPU15.TLB:TLB_shootdowns
8708 ± 8% +132.7% 20268 ± 7% interrupts.CPU16.CAL:Function_call_interrupts
1988418 +155.3% 5076923 interrupts.CPU16.LOC:Local_timer_interrupts
8548 ± 7% +148.3% 21226 ± 7% interrupts.CPU17.CAL:Function_call_interrupts
1988416 +155.3% 5077317 interrupts.CPU17.LOC:Local_timer_interrupts
33.00 ± 18% +147.0% 81.50 ± 21% interrupts.CPU17.TLB:TLB_shootdowns
8456 ± 9% +157.0% 21731 ± 6% interrupts.CPU18.CAL:Function_call_interrupts
1988574 +155.3% 5077430 interrupts.CPU18.LOC:Local_timer_interrupts
30.25 ± 15% +181.8% 85.25 ± 25% interrupts.CPU18.TLB:TLB_shootdowns
8884 ± 6% +144.5% 21720 ± 3% interrupts.CPU19.CAL:Function_call_interrupts
1987747 +155.4% 5077260 interrupts.CPU19.LOC:Local_timer_interrupts
37.00 ± 19% +127.0% 84.00 ± 26% interrupts.CPU19.TLB:TLB_shootdowns
8298 ± 11% +145.3% 20352 ± 9% interrupts.CPU2.CAL:Function_call_interrupts
1988446 +155.4% 5077740 interrupts.CPU2.LOC:Local_timer_interrupts
8594 ± 5% +146.1% 21149 ± 5% interrupts.CPU20.CAL:Function_call_interrupts
1987855 +155.4% 5077226 interrupts.CPU20.LOC:Local_timer_interrupts
8376 ± 8% +151.7% 21084 ± 9% interrupts.CPU21.CAL:Function_call_interrupts
1988171 +155.4% 5077212 interrupts.CPU21.LOC:Local_timer_interrupts
41.25 ± 20% +106.1% 85.00 ± 27% interrupts.CPU21.TLB:TLB_shootdowns
9024 ± 2% +141.1% 21757 ± 3% interrupts.CPU22.CAL:Function_call_interrupts
1987613 +155.4% 5077125 interrupts.CPU22.LOC:Local_timer_interrupts
8095 ± 7% +154.9% 20633 ± 6% interrupts.CPU23.CAL:Function_call_interrupts
1989315 +155.2% 5077451 interrupts.CPU23.LOC:Local_timer_interrupts
853.75 ± 12% +139.9% 2048 ± 7% interrupts.CPU24.35:PCI-MSI.2621441-edge.eth0-TxRx-0
8463 ± 5% +147.0% 20901 ± 7% interrupts.CPU24.CAL:Function_call_interrupts
1988251 +155.4% 5077879 interrupts.CPU24.LOC:Local_timer_interrupts
74848 ± 8% +525.2% 467985 ±128% interrupts.CPU24.RES:Rescheduling_interrupts
586.00 ± 14% +130.1% 1348 ± 5% interrupts.CPU25.36:PCI-MSI.2621442-edge.eth0-TxRx-1
8156 ± 7% +170.0% 22019 ± 2% interrupts.CPU25.CAL:Function_call_interrupts
1988295 +155.3% 5077010 interrupts.CPU25.LOC:Local_timer_interrupts
930.75 ±123% +260.9% 3359 ± 15% interrupts.CPU25.NMI:Non-maskable_interrupts
930.75 ±123% +260.9% 3359 ± 15% interrupts.CPU25.PMI:Performance_monitoring_interrupts
72678 ± 7% +575.9% 491267 ±120% interrupts.CPU25.RES:Rescheduling_interrupts
651.00 ± 9% +136.3% 1538 ± 9% interrupts.CPU26.37:PCI-MSI.2621443-edge.eth0-TxRx-2
8538 ± 6% +141.5% 20625 ± 6% interrupts.CPU26.CAL:Function_call_interrupts
1988053 +155.4% 5077342 interrupts.CPU26.LOC:Local_timer_interrupts
75367 ± 8% +522.0% 468794 ±133% interrupts.CPU26.RES:Rescheduling_interrupts
565.25 ± 14% +129.7% 1298 interrupts.CPU27.38:PCI-MSI.2621444-edge.eth0-TxRx-3
8157 ± 5% +132.5% 18969 ± 3% interrupts.CPU27.CAL:Function_call_interrupts
1988437 +155.3% 5077428 interrupts.CPU27.LOC:Local_timer_interrupts
591.00 ± 14% +123.8% 1322 ± 2% interrupts.CPU28.39:PCI-MSI.2621445-edge.eth0-TxRx-4
8181 ± 6% +148.1% 20300 ± 3% interrupts.CPU28.CAL:Function_call_interrupts
1987807 +155.4% 5077738 interrupts.CPU28.LOC:Local_timer_interrupts
71071 ± 7% +581.0% 483987 ±128% interrupts.CPU28.RES:Rescheduling_interrupts
660.25 ± 13% +111.4% 1396 ± 7% interrupts.CPU29.40:PCI-MSI.2621446-edge.eth0-TxRx-5
8178 ± 5% +150.4% 20475 ± 7% interrupts.CPU29.CAL:Function_call_interrupts
1987852 +155.4% 5077382 interrupts.CPU29.LOC:Local_timer_interrupts
70205 ± 8% +585.5% 481286 ±127% interrupts.CPU29.RES:Rescheduling_interrupts
17.75 ± 36% +163.4% 46.75 ± 80% interrupts.CPU29.TLB:TLB_shootdowns
8684 ± 5% +151.7% 21860 ± 4% interrupts.CPU3.CAL:Function_call_interrupts
1988786 +155.3% 5077436 interrupts.CPU3.LOC:Local_timer_interrupts
80407 ± 4% +518.6% 497419 ±122% interrupts.CPU3.RES:Rescheduling_interrupts
529.50 ± 4% +227.1% 1732 ± 14% interrupts.CPU30.41:PCI-MSI.2621447-edge.eth0-TxRx-6
8415 ± 4% +138.0% 20031 ± 3% interrupts.CPU30.CAL:Function_call_interrupts
1988609 +155.3% 5077494 interrupts.CPU30.LOC:Local_timer_interrupts
74875 ± 8% +518.8% 463337 ±134% interrupts.CPU30.RES:Rescheduling_interrupts
585.75 ± 16% +163.2% 1541 ± 10% interrupts.CPU31.42:PCI-MSI.2621448-edge.eth0-TxRx-7
7917 +151.5% 19913 ± 3% interrupts.CPU31.CAL:Function_call_interrupts
1988677 +155.3% 5077502 interrupts.CPU31.LOC:Local_timer_interrupts
67596 ± 3% +596.5% 470844 ±134% interrupts.CPU31.RES:Rescheduling_interrupts
8230 ± 4% +161.9% 21551 ± 8% interrupts.CPU32.CAL:Function_call_interrupts
1988317 +155.4% 5077410 interrupts.CPU32.LOC:Local_timer_interrupts
70449 ± 5% +597.2% 491174 ±127% interrupts.CPU32.RES:Rescheduling_interrupts
8684 ± 6% +137.3% 20610 ± 6% interrupts.CPU33.CAL:Function_call_interrupts
1988440 +155.3% 5077281 interrupts.CPU33.LOC:Local_timer_interrupts
77401 ± 5% +495.8% 461151 ±134% interrupts.CPU33.RES:Rescheduling_interrupts
8412 ± 8% +150.3% 21061 ± 4% interrupts.CPU34.CAL:Function_call_interrupts
1988447 +155.4% 5077546 interrupts.CPU34.LOC:Local_timer_interrupts
75351 ± 8% +537.5% 480376 ±128% interrupts.CPU34.RES:Rescheduling_interrupts
7890 +147.7% 19543 ± 5% interrupts.CPU35.CAL:Function_call_interrupts
1988875 +155.3% 5077717 interrupts.CPU35.LOC:Local_timer_interrupts
70158 ± 2% +577.1% 475011 ±135% interrupts.CPU35.RES:Rescheduling_interrupts
7908 ± 2% +184.9% 22530 ± 3% interrupts.CPU36.CAL:Function_call_interrupts
1988403 +155.4% 5077694 interrupts.CPU36.LOC:Local_timer_interrupts
31.75 ± 20% +178.0% 88.25 ± 28% interrupts.CPU36.TLB:TLB_shootdowns
9078 ± 2% +133.8% 21229 ± 6% interrupts.CPU37.CAL:Function_call_interrupts
1986971 +155.5% 5076871 interrupts.CPU37.LOC:Local_timer_interrupts
34.75 ± 36% +154.7% 88.50 ± 25% interrupts.CPU37.TLB:TLB_shootdowns
8574 ± 7% +131.7% 19864 ± 3% interrupts.CPU38.CAL:Function_call_interrupts
1986780 +155.5% 5076769 interrupts.CPU38.LOC:Local_timer_interrupts
29.25 ± 35% +195.7% 86.50 ± 18% interrupts.CPU38.TLB:TLB_shootdowns
8184 ± 7% +158.0% 21112 ± 4% interrupts.CPU39.CAL:Function_call_interrupts
1986972 +155.5% 5077337 interrupts.CPU39.LOC:Local_timer_interrupts
8368 ± 8% +144.7% 20481 ± 12% interrupts.CPU4.CAL:Function_call_interrupts
1987774 +155.4% 5077506 interrupts.CPU4.LOC:Local_timer_interrupts
8165 ± 3% +163.7% 21528 ± 6% interrupts.CPU40.CAL:Function_call_interrupts
1987659 +155.4% 5077421 interrupts.CPU40.LOC:Local_timer_interrupts
32.00 ± 12% +154.7% 81.50 ± 29% interrupts.CPU40.TLB:TLB_shootdowns
8454 ± 6% +158.4% 21847 ± 8% interrupts.CPU41.CAL:Function_call_interrupts
1987504 +155.5% 5077467 interrupts.CPU41.LOC:Local_timer_interrupts
37.75 ± 45% +122.5% 84.00 ± 17% interrupts.CPU41.TLB:TLB_shootdowns
8529 ± 7% +136.4% 20164 ± 8% interrupts.CPU42.CAL:Function_call_interrupts
1988401 +155.4% 5077407 interrupts.CPU42.LOC:Local_timer_interrupts
32.75 ± 39% +171.8% 89.00 ± 37% interrupts.CPU42.TLB:TLB_shootdowns
8102 ± 5% +150.4% 20283 ± 9% interrupts.CPU43.CAL:Function_call_interrupts
1987414 +155.5% 5077903 interrupts.CPU43.LOC:Local_timer_interrupts
32.50 ± 18% +150.8% 81.50 ± 34% interrupts.CPU43.TLB:TLB_shootdowns
8206 ± 7% +157.3% 21112 ± 8% interrupts.CPU44.CAL:Function_call_interrupts
1987278 +155.5% 5077309 interrupts.CPU44.LOC:Local_timer_interrupts
37.00 ± 25% +119.6% 81.25 ± 24% interrupts.CPU44.TLB:TLB_shootdowns
8467 ± 7% +139.3% 20261 ± 5% interrupts.CPU45.CAL:Function_call_interrupts
1987858 +155.4% 5077234 interrupts.CPU45.LOC:Local_timer_interrupts
38.25 ± 22% +113.1% 81.50 ± 27% interrupts.CPU45.TLB:TLB_shootdowns
7773 ± 4% +162.0% 20367 ± 9% interrupts.CPU46.CAL:Function_call_interrupts
1988584 +155.3% 5077417 interrupts.CPU46.LOC:Local_timer_interrupts
37.50 ± 14% +142.7% 91.00 ± 34% interrupts.CPU46.TLB:TLB_shootdowns
8746 ± 4% +138.7% 20880 ± 8% interrupts.CPU47.CAL:Function_call_interrupts
1988495 +155.3% 5077204 interrupts.CPU47.LOC:Local_timer_interrupts
8722 ± 7% +134.1% 20414 ± 10% interrupts.CPU5.CAL:Function_call_interrupts
1987875 +155.4% 5077418 interrupts.CPU5.LOC:Local_timer_interrupts
8435 ± 8% +143.9% 20570 ± 11% interrupts.CPU6.CAL:Function_call_interrupts
1988389 +155.4% 5077630 interrupts.CPU6.LOC:Local_timer_interrupts
8837 ± 5% +139.1% 21133 ± 9% interrupts.CPU7.CAL:Function_call_interrupts
1987853 +155.4% 5077370 interrupts.CPU7.LOC:Local_timer_interrupts
8852 ± 5% +123.8% 19810 ± 3% interrupts.CPU8.CAL:Function_call_interrupts
1988702 +155.3% 5077569 interrupts.CPU8.LOC:Local_timer_interrupts
8273 ± 6% +151.1% 20771 ± 9% interrupts.CPU9.CAL:Function_call_interrupts
1988033 +155.4% 5077588 interrupts.CPU9.LOC:Local_timer_interrupts
70739 ± 6% +578.9% 480243 ±129% interrupts.CPU9.RES:Rescheduling_interrupts
95428175 +155.4% 2.437e+08 interrupts.LOC:Local_timer_interrupts
144.00 +133.3% 336.00 interrupts.MCP:Machine_check_polls
6773889 ± 8% +482.9% 39483682 ± 7% interrupts.RES:Rescheduling_interrupts
1544 ± 5% +91.5% 2957 ± 11% interrupts.TLB:TLB_shootdowns
aim7.jobs-per-min
1300 +-+------------------------------------------------------------------+
1200 +-+.++.+.+.+.+. +.+.+.+. +.+.+ +.++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.|
| + + + |
1100 +-+ |
1000 +-+ |
| |
900 +-+ |
800 +-+ |
700 +-+ |
| |
600 +-+ |
500 +-+ O |
| O O O O O O O |
400 O-O OO O O O O OO O O O OO O O O O OO O O O O |
300 +-+------------------------------------------------------------------+
aim7.time.user_time
55 +-+--------------------------------------------------------------------+
| |
50 +-+ O |
| O |
| |
45 +-+ |
| O O O O O O O O O O |
40 O-O O OO O O O O O O O O O OO O O O O O |
| |
35 +-+ |
| |
| |
30 +-+ |
|.+.+.++.+.+.+.+.+.+.++.+.+.+.+.+.+.++.+.+.+.+.+.++.+. .+.+.+.+.++.+.+.|
25 +-+--------------------------------------------------------------------+
aim7.time.system_time
11000 +-+-----------------------------------------------------------------+
O O OO O O O OO O O OO O O O OO O O O OO O O |
10000 +-+ O OO O O OO O |
9000 +-+ |
| |
8000 +-+ |
| |
7000 +-+ |
| |
6000 +-+ |
5000 +-+ |
| |
4000 +-+ |
| .++. .+. .+.+. .+.+. .++. .+ .+.+.+ .+. .++.+.|
3000 +-+-----------------------------------------------------------------+
aim7.time.elapsed_time
3500 +-+------------------------------------------------------------------+
O O OO O O O O OO O O OO O O O O OO O O O O |
3000 +-+ O |
| |
| O O O O O O |
2500 +-+ OO |
| |
2000 +-+ |
| |
1500 +-+ |
| |
| |
1000 +-+.++.+.+.+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.|
| |
500 +-+------------------------------------------------------------------+
aim7.time.elapsed_time.max
3500 +-+------------------------------------------------------------------+
O O OO O O O O OO O O OO O O O O OO O O O O |
3000 +-+ O |
| |
| O O O O O O |
2500 +-+ OO |
| |
2000 +-+ |
| |
1500 +-+ |
| |
| |
1000 +-+.++.+.+.+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.|
| |
500 +-+------------------------------------------------------------------+
aim7.time.voluntary_context_switches
1.8e+09 +-+---------------------------------------------------------------+
O O O O OO O OO O O OO O O OO O OO O O |
1.6e+09 +-O O O O O O OO O O O |
| |
1.4e+09 +-+ |
| |
1.2e+09 +-+ |
| |
1e+09 +-+ |
| |
8e+08 +-+ |
| |
6e+08 +-+ |
|.++.+.+.++.+.+.++.+.+.++.+.+.++.+.++.+.+.++.+.+.++.+.+.++.+.+.++.|
4e+08 +-+---------------------------------------------------------------+
aim7.time.file_system_outputs
1.8e+08 +-+---------------------------------------------------------------+
O OO O O OO O O OO O O OO O O OO O OO O O OO O O OO O O O |
1.7e+08 +-+ |
1.6e+08 +-+ |
| |
1.5e+08 +-+ |
| |
1.4e+08 +-+ |
| |
1.3e+08 +-+ |
1.2e+08 +-+ |
| |
1.1e+08 +-+ |
|.++.+.+.++.+.+.++.+.+.++.+.+.++.+.++.+.+.++.+.+.++.+.+.++.+.+.++.|
1e+08 +-+---------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
3 months, 3 weeks
- 3
- 3
[btrfs] 8d47a0d8f7: fio.write_bw_MBps -28.6% regression
by kernel test robot
Greeting,
FYI, we noticed a -28.6% regression of fio.write_bw_MBps due to commit:
commit: 8d47a0d8f7947422dd359ac8e462687f81a7a137 ("btrfs: Do mandatory tree block check before submitting bio")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: fio-basic
on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory
with following parameters:
disk: 2pmem
fs: btrfs
runtime: 200s
nr_task: 50%
time_based: tb
rw: randwrite
bs: 4k
ioengine: libaio
test_size: 100G
cpufreq_governor: performance
ucode: 0x3d
test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
4k/gcc-7/performance/2pmem/btrfs/libaio/x86_64-rhel-7.6/50%/debian-x86_64-2018-04-03.cgz/200s/randwrite/lkp-hsw-ep2/100G/fio-basic/tb/0x3d
commit:
ff2ac107fa ("btrfs: tree-checker: Remove comprehensive root owner check")
8d47a0d8f7 ("btrfs: Do mandatory tree block check before submitting bio")
ff2ac107fae2440b 8d47a0d8f7947422dd359ac8e46
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:4 25% 1:4 dmesg.BUG:unable_to_handle_kernel
:4 25% 1:4 dmesg.Kernel_panic-not_syncing:Fatal_exception_in_interrupt
:4 25% 1:4 dmesg.Oops:#[##]
:4 25% 1:4 dmesg.RIP:cpuidle_enter_state
:4 25% 1:4 dmesg.RIP:native_write_msr
:4 25% 1:4 dmesg.RIP:perf_prepare_sample
1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
%stddev %change %stddev
\ | \
0.01 ± 21% +0.3 0.28 ± 20% fio.latency_1000ms%
4.14 ± 13% +1.8 5.95 ± 9% fio.latency_10ms%
8.73 ± 15% +3.0 11.68 ± 8% fio.latency_20ms%
1.91 ± 5% -0.4 1.52 ± 3% fio.latency_250ms%
66.81 -5.9 60.90 ± 2% fio.latency_4ms%
0.18 ± 15% +0.1 0.29 ± 16% fio.latency_750us%
1.017e+08 -28.5% 72701525 fio.time.file_system_outputs
109.75 -26.8% 80.33 ± 2% fio.time.percent_of_cpu_this_job_got
197.97 -26.4% 145.79 ± 2% fio.time.system_time
38.66 ± 4% -24.0% 29.39 ± 3% fio.time.user_time
12388015 -28.5% 8858878 fio.time.voluntary_context_switches
12711059 -28.5% 9083954 fio.workload
247.91 -28.6% 176.96 fio.write_bw_MBps
309248 ± 2% +71.7% 531114 ± 4% fio.write_clat_99%_us
17574 +40.0% 24609 fio.write_clat_mean_us
57989 ± 4% +82.2% 105661 fio.write_clat_stddev
63463 -28.6% 45302 fio.write_iops
564.61 +40.2% 791.54 fio.write_slat_mean_us
9375 ± 7% +96.3% 18406 fio.write_slat_stddev
21.25 +8.5% 23.06 ± 9% boot-time.dhcp
93.32 +1.4% 94.58 iostat.cpu.idle
6.43 -18.6% 5.23 ± 2% iostat.cpu.system
6.48 -1.2 5.27 ± 2% mpstat.cpu.all.sys%
0.24 ± 5% -0.1 0.18 ± 2% mpstat.cpu.all.usr%
93.00 +1.1% 94.00 vmstat.cpu.id
591200 -30.0% 413921 vmstat.io.bo
31653776 ± 2% -10.9% 28196588 ± 2% vmstat.memory.cache
11927341 ± 7% +29.8% 15486805 ± 4% vmstat.memory.free
5.00 -26.7% 3.67 ± 12% vmstat.procs.r
635910 ± 2% -27.6% 460384 vmstat.system.cs
7822384 ± 2% -25.0% 5870498 numa-numastat.node0.local_node
7839861 ± 2% -24.8% 5893785 numa-numastat.node0.numa_hit
224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node0.numa_miss
242465 ± 44% -84.3% 37964 ± 18% numa-numastat.node0.other_node
7762623 ± 12% -26.8% 5681358 ± 6% numa-numastat.node1.local_node
224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node1.numa_foreign
7768618 ± 12% -26.9% 5681573 ± 6% numa-numastat.node1.numa_hit
9.429e+08 ± 10% -35.4% 6.094e+08 cpuidle.C1.time
53504154 ± 6% -31.8% 36504731 cpuidle.C1.usage
1.093e+09 ± 22% -50.3% 5.435e+08 ± 3% cpuidle.C1E.time
16541925 ± 13% -45.0% 9103775 cpuidle.C1E.usage
5.366e+09 ± 36% +71.7% 9.214e+09 ± 18% cpuidle.C3.time
16163664 ± 33% +57.0% 25379738 ± 5% cpuidle.C3.usage
9697055 ± 16% -35.8% 6225168 ± 19% cpuidle.C6.usage
19313595 ± 17% -16.3% 16166123 cpuidle.POLL.time
10291838 ± 9% -26.8% 7537045 ± 7% meminfo.Active
10034782 ± 9% -27.4% 7280710 ± 7% meminfo.Active(file)
31524905 ± 2% -10.9% 28082249 ± 2% meminfo.Cached
115157 ± 4% +61.1% 185560 ± 3% meminfo.CmaFree
6430720 ± 8% +19.1% 7660885 ± 6% meminfo.DirectMap2M
534230 ± 5% -12.8% 465932 ± 6% meminfo.Dirty
11737322 ± 7% +30.2% 15281189 ± 4% meminfo.MemFree
37604344 ± 2% -9.4% 34060483 meminfo.Memused
806.50 ± 37% +137.4% 1914 ± 8% meminfo.Writeback
370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.active_objs
370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.num_objs
15147 ± 5% -8.3% 13897 ± 2% slabinfo.blkdev_ioc.active_objs
17839 ± 6% -11.7% 15745 ± 2% slabinfo.blkdev_ioc.num_objs
12178 ± 4% -7.4% 11275 ± 2% slabinfo.kmalloc-512.num_objs
7794 ± 3% +25.4% 9774 slabinfo.proc_inode_cache.active_objs
8017 ± 2% +23.4% 9895 slabinfo.proc_inode_cache.num_objs
2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.active_objs
2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.num_objs
263.75 ± 2% -15.1% 224.00 ± 2% turbostat.Avg_MHz
9.46 ± 2% -1.4 8.07 ± 2% turbostat.Busy%
53502248 ± 6% -31.8% 36503898 turbostat.C1
5.86 ± 8% -2.0 3.82 turbostat.C1%
16540396 ± 13% -45.0% 9102733 turbostat.C1E
6.80 ± 22% -3.4 3.40 ± 4% turbostat.C1E%
16163332 ± 33% +57.0% 25379351 ± 5% turbostat.C3
33.55 ± 36% +24.1 57.61 ± 18% turbostat.C3%
9689445 ± 16% -35.8% 6221033 ± 19% turbostat.C6
18.04 ± 26% +49.1% 26.90 ± 9% turbostat.CPU%c3
53.25 ± 3% +10.8% 59.00 ± 4% turbostat.CoreTmp
4.67 ± 14% +62.7% 7.59 ± 26% turbostat.Pkg%pc2
56.50 +9.7% 62.00 ± 2% turbostat.PkgTmp
11.84 -4.8% 11.28 turbostat.RAMWatt
7232478 ± 10% -37.2% 4543037 ± 22% numa-meminfo.node0.Active
7072564 ± 10% -37.6% 4410135 ± 23% numa-meminfo.node0.Active(file)
308167 ± 6% -24.7% 232152 ± 10% numa-meminfo.node0.Dirty
8784028 ± 13% +21.7% 10686327 ± 2% numa-meminfo.node0.Inactive
8669548 ± 12% +20.7% 10465444 ± 2% numa-meminfo.node0.Inactive(file)
572.50 ± 22% +115.9% 1236 ± 25% numa-meminfo.node0.Writeback
15098140 ± 3% -17.7% 12425670 ± 3% numa-meminfo.node1.FilePages
11599549 ± 7% -22.3% 9018051 ± 4% numa-meminfo.node1.Inactive
105067 ± 92% -99.3% 725.33 ± 51% numa-meminfo.node1.Inactive(anon)
11494482 ± 7% -21.6% 9017325 ± 4% numa-meminfo.node1.Inactive(file)
5504 ± 9% -12.5% 4819 ± 2% numa-meminfo.node1.KernelStack
6709938 ± 11% +45.8% 9781880 ± 5% numa-meminfo.node1.MemFree
18029136 ± 4% -17.0% 14957200 ± 3% numa-meminfo.node1.MemUsed
4427 ± 69% -77.5% 996.00 ± 8% numa-meminfo.node1.PageTables
857660 ± 32% -37.2% 538646 ± 8% numa-meminfo.node1.SUnreclaim
109232 ± 88% -96.1% 4274 ± 10% numa-meminfo.node1.Shmem
986740 ± 35% -40.0% 592452 ± 10% numa-meminfo.node1.Slab
575.21 ± 15% +30.1% 748.58 ± 2% sched_debug.cfs_rq:/.runnable_load_avg.max
100.45 ± 15% +28.6% 129.19 ± 5% sched_debug.cfs_rq:/.runnable_load_avg.stddev
830.19 ± 9% +15.7% 960.25 ± 6% sched_debug.cfs_rq:/.util_avg.max
29.97 ± 16% -25.2% 22.43 ± 31% sched_debug.cfs_rq:/.util_est_enqueued.avg
591.17 ± 13% -22.2% 459.83 ± 18% sched_debug.cfs_rq:/.util_est_enqueued.max
107.52 ± 13% -23.0% 82.84 ± 23% sched_debug.cfs_rq:/.util_est_enqueued.stddev
2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock.stddev
2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock_task.stddev
24.51 ± 7% +21.5% 29.77 ± 8% sched_debug.cpu.cpu_load[0].avg
541.29 ± 15% +38.3% 748.58 ± 2% sched_debug.cpu.cpu_load[0].max
93.97 ± 7% +27.8% 120.09 ± 2% sched_debug.cpu.cpu_load[0].stddev
519.08 ± 6% +40.5% 729.42 ± 4% sched_debug.cpu.cpu_load[1].max
87.45 ± 5% +28.5% 112.39 ± 4% sched_debug.cpu.cpu_load[1].stddev
480.83 ± 7% +39.5% 670.92 ± 13% sched_debug.cpu.cpu_load[2].max
80.92 ± 5% +25.7% 101.75 ± 8% sched_debug.cpu.cpu_load[2].stddev
408.83 ± 4% +54.0% 629.58 ± 19% sched_debug.cpu.cpu_load[3].max
71.62 ± 2% +30.7% 93.61 ± 11% sched_debug.cpu.cpu_load[3].stddev
381.50 ± 8% +66.7% 636.00 ± 18% sched_debug.cpu.cpu_load[4].max
64.40 ± 6% +39.7% 89.96 ± 12% sched_debug.cpu.cpu_load[4].stddev
106.75 ± 20% -88.4% 12.33 ± 10% proc-vmstat.kswapd_high_wmark_hit_quickly
2508660 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_active_file
33315360 -30.4% 23192301 proc-vmstat.nr_dirtied
133447 ± 5% -12.6% 116580 ± 6% proc-vmstat.nr_dirty
7881043 ± 2% -10.9% 7019533 ± 2% proc-vmstat.nr_file_pages
28790 ± 4% +61.1% 46392 ± 3% proc-vmstat.nr_free_cma
2934503 ± 7% +30.2% 3821369 ± 4% proc-vmstat.nr_free_pages
5044168 -3.4% 4872965 proc-vmstat.nr_inactive_file
430282 -6.0% 404589 proc-vmstat.nr_slab_unreclaimable
197.75 ± 38% +136.5% 467.67 ± 10% proc-vmstat.nr_writeback
33243480 -30.4% 23143756 proc-vmstat.nr_written
2508665 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_zone_active_file
5044399 -3.4% 4873026 proc-vmstat.nr_zone_inactive_file
134972 ± 5% -12.4% 118262 ± 5% proc-vmstat.nr_zone_write_pending
15632301 ± 5% -25.8% 11598535 ± 3% proc-vmstat.numa_hit
15608826 ± 5% -25.8% 11575029 ± 3% proc-vmstat.numa_local
2300530 ± 10% -31.1% 1585363 ± 10% proc-vmstat.pgactivate
16955745 -26.3% 12499611 ± 2% proc-vmstat.pgalloc_normal
17488000 -25.9% 12959812 ± 2% proc-vmstat.pgfree
1.33e+08 -30.4% 92575440 proc-vmstat.pgpgout
5728177 ± 6% -78.5% 1232433 ± 25% proc-vmstat.pgscan_kswapd
5727967 ± 6% -78.5% 1232244 ± 25% proc-vmstat.pgsteal_kswapd
32760 -7.6% 30285 proc-vmstat.slabs_scanned
1768211 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_active_file
10410188 ± 7% -32.5% 7023939 ± 7% numa-vmstat.node0.nr_dirtied
76995 ± 6% -24.6% 58035 ± 10% numa-vmstat.node0.nr_dirty
2167457 ± 12% +20.7% 2616147 ± 2% numa-vmstat.node0.nr_inactive_file
142.00 ± 24% +120.4% 313.00 ± 24% numa-vmstat.node0.nr_writeback
10308758 ± 7% -32.6% 6949497 ± 7% numa-vmstat.node0.nr_written
1768216 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_zone_active_file
2167563 ± 12% +20.7% 2616189 ± 2% numa-vmstat.node0.nr_zone_inactive_file
77786 ± 7% -24.3% 58899 ± 10% numa-vmstat.node0.nr_zone_write_pending
5889240 ± 3% -14.3% 5046874 ± 3% numa-vmstat.node0.numa_hit
5837851 ± 4% -14.0% 5023120 ± 3% numa-vmstat.node0.numa_local
68376 ± 48% -96.9% 2149 ± 46% numa-vmstat.node0.numa_miss
119766 ± 42% -78.4% 25904 ± 3% numa-vmstat.node0.numa_other
30.50 ± 38% -100.0% 0.00 numa-vmstat.node0.workingset_nodes
6829126 ± 6% -22.8% 5269898 ± 10% numa-vmstat.node1.nr_dirtied
3774645 ± 3% -17.7% 3106081 ± 3% numa-vmstat.node1.nr_file_pages
28796 ± 4% +61.1% 46405 ± 3% numa-vmstat.node1.nr_free_cma
1677332 ± 11% +45.8% 2445778 ± 5% numa-vmstat.node1.nr_free_pages
26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_inactive_anon
2873702 ± 7% -21.6% 2254116 ± 4% numa-vmstat.node1.nr_inactive_file
5506 ± 9% -12.5% 4820 ± 2% numa-vmstat.node1.nr_kernel_stack
29031 ± 85% -88.2% 3436 numa-vmstat.node1.nr_mapped
1106 ± 69% -77.5% 249.00 ± 8% numa-vmstat.node1.nr_page_table_pages
27308 ± 88% -96.1% 1068 ± 10% numa-vmstat.node1.nr_shmem
214418 ± 32% -37.2% 134663 ± 8% numa-vmstat.node1.nr_slab_unreclaimable
75.00 ± 35% +145.3% 184.00 ± 18% numa-vmstat.node1.nr_writeback
6761224 ± 6% -23.1% 5202586 ± 10% numa-vmstat.node1.nr_written
26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_zone_inactive_anon
2873810 ± 7% -21.6% 2254135 ± 4% numa-vmstat.node1.nr_zone_inactive_file
68381 ± 48% -96.9% 2152 ± 46% numa-vmstat.node1.numa_foreign
5035283 ± 15% -26.9% 3679318 ± 7% numa-vmstat.node1.numa_hit
4905249 ± 15% -28.2% 3522229 ± 8% numa-vmstat.node1.numa_local
18.43 ± 6% -28.3% 13.21 ± 3% perf-stat.i.MPKI
2.205e+09 ± 3% +10.5% 2.437e+09 perf-stat.i.branch-instructions
1.83 ± 5% -0.6 1.24 ± 7% perf-stat.i.branch-miss-rate%
37019551 ± 3% -29.4% 26140844 ± 5% perf-stat.i.branch-misses
28932020 ± 6% -31.1% 19933819 ± 3% perf-stat.i.cache-misses
2.03e+08 ± 4% -24.8% 1.526e+08 perf-stat.i.cache-references
645248 ± 2% -27.4% 468301 perf-stat.i.context-switches
1.97 ± 7% -22.1% 1.53 ± 2% perf-stat.i.cpi
1.983e+10 ± 3% -15.1% 1.684e+10 perf-stat.i.cpu-cycles
1.03 ± 5% -0.4 0.64 ± 10% perf-stat.i.dTLB-load-miss-rate%
31879502 ± 6% -37.9% 19807598 ± 9% perf-stat.i.dTLB-load-misses
3.201e+09 ± 3% +18.8% 3.804e+09 perf-stat.i.dTLB-loads
0.13 ± 7% -0.0 0.09 ± 10% perf-stat.i.dTLB-store-miss-rate%
2516181 ± 10% -28.3% 1805272 ± 10% perf-stat.i.dTLB-store-misses
2.143e+09 ± 3% +19.8% 2.568e+09 ± 2% perf-stat.i.dTLB-stores
13781489 ± 3% -28.4% 9866239 ± 3% perf-stat.i.iTLB-loads
1.164e+10 ± 3% +21.6% 1.415e+10 perf-stat.i.instructions
8435 ± 17% +42.5% 12023 ± 14% perf-stat.i.instructions-per-iTLB-miss
0.57 ± 3% +43.6% 0.82 perf-stat.i.ipc
16050361 ± 9% -30.7% 11127721 ± 3% perf-stat.i.node-load-misses
8120610 ± 9% -31.3% 5580996 perf-stat.i.node-loads
2591649 ± 13% -31.1% 1785932 ± 6% perf-stat.i.node-store-misses
1809775 ± 12% -31.2% 1244360 ± 2% perf-stat.i.node-stores
17.45 ± 4% -38.2% 10.79 perf-stat.overall.MPKI
1.68 ± 3% -0.6 1.07 ± 5% perf-stat.overall.branch-miss-rate%
1.70 ± 3% -30.2% 1.19 perf-stat.overall.cpi
689.69 ± 9% +22.6% 845.48 ± 2% perf-stat.overall.cycles-between-cache-misses
0.99 ± 5% -0.5 0.52 ± 10% perf-stat.overall.dTLB-load-miss-rate%
0.12 ± 8% -0.0 0.07 ± 10% perf-stat.overall.dTLB-store-miss-rate%
10.81 ± 14% +2.3 13.09 ± 10% perf-stat.overall.iTLB-load-miss-rate%
7119 ± 14% +35.9% 9679 ± 13% perf-stat.overall.instructions-per-iTLB-miss
0.59 ± 3% +43.1% 0.84 perf-stat.overall.ipc
202912 ± 2% +68.8% 342615 ± 3% perf-stat.overall.path-length
2.195e+09 ± 3% +10.5% 2.425e+09 perf-stat.ps.branch-instructions
36847010 ± 3% -29.4% 26015686 ± 5% perf-stat.ps.branch-misses
28799092 ± 6% -31.1% 19837306 ± 3% perf-stat.ps.cache-misses
2.021e+08 ± 4% -24.8% 1.519e+08 perf-stat.ps.cache-references
642266 ± 2% -27.4% 465968 perf-stat.ps.context-switches
1.974e+10 ± 3% -15.1% 1.676e+10 perf-stat.ps.cpu-cycles
31729472 ± 6% -37.9% 19715594 ± 9% perf-stat.ps.dTLB-load-misses
3.186e+09 ± 3% +18.8% 3.785e+09 perf-stat.ps.dTLB-loads
2504443 ± 10% -28.2% 1797154 ± 10% perf-stat.ps.dTLB-store-misses
2.133e+09 ± 3% +19.8% 2.556e+09 ± 2% perf-stat.ps.dTLB-stores
13717699 ± 3% -28.4% 9816979 ± 3% perf-stat.ps.iTLB-loads
1.159e+10 ± 3% +21.5% 1.408e+10 perf-stat.ps.instructions
15979371 ± 9% -30.7% 11073468 ± 3% perf-stat.ps.node-load-misses
8081205 ± 9% -31.3% 5554114 perf-stat.ps.node-loads
2580198 ± 13% -31.1% 1777169 ± 6% perf-stat.ps.node-store-misses
1801590 ± 12% -31.3% 1238321 ± 2% perf-stat.ps.node-stores
2.579e+12 +20.6% 3.111e+12 ± 2% perf-stat.total.instructions
36149 ± 4% -26.2% 26686 ± 5% softirqs.CPU0.RCU
37255 ± 3% -26.8% 27256 ± 2% softirqs.CPU1.RCU
37834 ± 5% -26.5% 27801 ± 9% softirqs.CPU10.RCU
39075 ± 3% -24.5% 29494 ± 2% softirqs.CPU11.RCU
38137 ± 6% -23.1% 29340 ± 4% softirqs.CPU12.RCU
37480 ± 3% -20.5% 29797 ± 2% softirqs.CPU13.RCU
37690 ± 5% -28.7% 26861 ± 4% softirqs.CPU14.RCU
33863 ± 9% -29.5% 23885 ± 5% softirqs.CPU15.RCU
34306 ± 9% -31.1% 23641 ± 12% softirqs.CPU16.RCU
34374 ± 9% -29.0% 24402 ± 7% softirqs.CPU17.RCU
44013 ± 9% -31.7% 30059 softirqs.CPU18.RCU
43114 ± 8% -29.9% 30219 softirqs.CPU19.RCU
36276 ± 8% -23.3% 27825 ± 5% softirqs.CPU2.RCU
67049 ± 7% +39.1% 93267 ± 18% softirqs.CPU2.TIMER
39721 ± 13% -31.4% 27233 ± 3% softirqs.CPU20.RCU
42340 ± 5% -28.6% 30215 ± 6% softirqs.CPU21.RCU
40896 ± 7% -30.3% 28506 ± 2% softirqs.CPU22.RCU
42422 ± 7% -33.2% 28357 ± 2% softirqs.CPU23.RCU
42330 ± 9% -33.5% 28150 ± 5% softirqs.CPU24.RCU
41406 ± 9% -31.4% 28398 softirqs.CPU25.RCU
39754 ± 5% -29.4% 28063 softirqs.CPU26.RCU
40207 ± 6% -27.9% 28995 ± 12% softirqs.CPU27.RCU
40351 ± 9% -29.4% 28487 ± 3% softirqs.CPU28.RCU
41880 ± 5% -29.6% 29472 softirqs.CPU29.RCU
36293 ± 4% -17.7% 29864 ± 7% softirqs.CPU3.RCU
39341 ± 9% -34.3% 25847 softirqs.CPU30.RCU
39687 ± 9% -33.5% 26400 ± 2% softirqs.CPU31.RCU
38062 ± 14% -39.9% 22878 ± 10% softirqs.CPU32.RCU
39769 ± 13% -33.8% 26314 ± 5% softirqs.CPU33.RCU
39504 ± 10% -37.2% 24824 ± 6% softirqs.CPU34.RCU
39591 ± 11% -38.6% 24300 ± 5% softirqs.CPU35.RCU
40398 ± 8% -27.0% 29488 ± 12% softirqs.CPU36.RCU
39464 ± 12% -25.3% 29478 ± 10% softirqs.CPU37.RCU
41177 ± 9% -34.0% 27190 ± 19% softirqs.CPU38.RCU
38734 ± 13% -24.8% 29145 ± 5% softirqs.CPU39.RCU
36804 ± 5% -23.0% 28354 softirqs.CPU4.RCU
40421 ± 12% -28.8% 28776 ± 6% softirqs.CPU40.RCU
40371 ± 11% -31.1% 27834 ± 8% softirqs.CPU41.RCU
40633 ± 10% -29.9% 28469 ± 10% softirqs.CPU42.RCU
41018 ± 9% -33.6% 27252 ± 6% softirqs.CPU43.RCU
38181 ± 12% -27.0% 27856 ± 10% softirqs.CPU44.RCU
39605 ± 12% -32.3% 26803 ± 12% softirqs.CPU45.RCU
42686 ± 11% -29.7% 30016 ± 9% softirqs.CPU47.RCU
40433 ± 9% -31.8% 27586 ± 11% softirqs.CPU48.RCU
38371 ± 12% -23.0% 29528 ± 7% softirqs.CPU49.RCU
37172 ± 2% -25.9% 27562 ± 3% softirqs.CPU5.RCU
39660 ± 12% -27.5% 28752 ± 11% softirqs.CPU50.RCU
40375 ± 10% -26.7% 29582 ± 3% softirqs.CPU51.RCU
40568 ± 9% -28.8% 28886 ± 6% softirqs.CPU52.RCU
39967 ± 10% -27.3% 29073 ± 11% softirqs.CPU53.RCU
40874 ± 11% -37.1% 25725 ± 6% softirqs.CPU54.RCU
41227 ± 14% -37.9% 25606 ± 6% softirqs.CPU55.RCU
38856 ± 13% -34.7% 25380 ± 8% softirqs.CPU56.RCU
39618 ± 13% -39.1% 24119 ± 4% softirqs.CPU57.RCU
39412 ± 12% -36.6% 24970 ± 3% softirqs.CPU58.RCU
41572 ± 14% -41.3% 24399 ± 10% softirqs.CPU59.RCU
36927 ± 3% -24.4% 27923 ± 22% softirqs.CPU6.RCU
37057 ± 10% -31.8% 25261 ± 2% softirqs.CPU60.RCU
36584 ± 8% -27.8% 26419 ± 2% softirqs.CPU61.RCU
35859 ± 4% -28.2% 25729 ± 5% softirqs.CPU62.RCU
35849 ± 7% -33.2% 23933 ± 5% softirqs.CPU63.RCU
35379 ± 6% -26.4% 26041 ± 4% softirqs.CPU64.RCU
36908 ± 6% -26.5% 27110 ± 2% softirqs.CPU65.RCU
37595 ± 3% -26.1% 27783 ± 4% softirqs.CPU66.RCU
36327 ± 2% -27.0% 26533 ± 2% softirqs.CPU67.RCU
35136 ± 7% -31.7% 23998 ± 9% softirqs.CPU68.RCU
35638 ± 4% -28.6% 25435 ± 4% softirqs.CPU69.RCU
37268 ± 6% -25.4% 27786 ± 4% softirqs.CPU7.RCU
36693 ± 4% -29.2% 25986 ± 5% softirqs.CPU70.RCU
36859 ± 3% -28.8% 26260 ± 4% softirqs.CPU71.RCU
37539 ± 7% -27.7% 27144 ± 4% softirqs.CPU8.RCU
37584 ± 4% -26.0% 27802 ± 15% softirqs.CPU9.RCU
2795313 ± 7% -29.7% 1964363 ± 4% softirqs.RCU
249.50 ± 76% -50.4% 123.67 ± 2% interrupts.41:PCI-MSI.1572870-edge.eth0-TxRx-6
154.50 ± 13% -10.9% 137.67 ± 10% interrupts.42:PCI-MSI.1572871-edge.eth0-TxRx-7
150.50 ± 43% -28.0% 108.33 interrupts.84:PCI-MSI.1572913-edge.eth0-TxRx-49
10393 ± 10% -26.4% 7653 ± 10% interrupts.CPU1.RES:Rescheduling_interrupts
12518 ± 15% -31.0% 8642 ± 21% interrupts.CPU10.RES:Rescheduling_interrupts
945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.NMI:Non-maskable_interrupts
945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.PMI:Performance_monitoring_interrupts
13068 ± 18% -30.6% 9075 ± 23% interrupts.CPU11.RES:Rescheduling_interrupts
11949 ± 11% -49.8% 6001 ± 29% interrupts.CPU12.RES:Rescheduling_interrupts
13711 ± 8% -19.4% 11052 ± 24% interrupts.CPU13.RES:Rescheduling_interrupts
13361 ± 6% -38.8% 8170 ± 25% interrupts.CPU14.RES:Rescheduling_interrupts
7628 ± 12% -29.2% 5401 ± 7% interrupts.CPU18.RES:Rescheduling_interrupts
7909 ± 25% -30.4% 5501 ± 37% interrupts.CPU19.RES:Rescheduling_interrupts
1482 ± 39% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts
1482 ± 39% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts
2116 +23.2% 2606 ± 10% interrupts.CPU20.CAL:Function_call_interrupts
194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.NMI:Non-maskable_interrupts
194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.PMI:Performance_monitoring_interrupts
2096 ± 4% +18.0% 2474 ± 9% interrupts.CPU21.CAL:Function_call_interrupts
11313 ± 20% -56.4% 4935 ± 23% interrupts.CPU21.RES:Rescheduling_interrupts
2111 ± 3% +20.1% 2534 ± 11% interrupts.CPU22.CAL:Function_call_interrupts
2109 ± 4% +22.6% 2585 ± 10% interrupts.CPU23.CAL:Function_call_interrupts
10305 ± 16% -40.7% 6115 ± 17% interrupts.CPU23.RES:Rescheduling_interrupts
2128 ± 2% +17.8% 2507 ± 10% interrupts.CPU24.CAL:Function_call_interrupts
12271 ± 20% -39.0% 7488 ± 29% interrupts.CPU24.RES:Rescheduling_interrupts
2123 +19.7% 2541 ± 7% interrupts.CPU25.CAL:Function_call_interrupts
9780 ± 21% -32.9% 6563 ± 25% interrupts.CPU25.RES:Rescheduling_interrupts
980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.NMI:Non-maskable_interrupts
980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.PMI:Performance_monitoring_interrupts
11163 ± 28% -49.8% 5605 ± 13% interrupts.CPU27.RES:Rescheduling_interrupts
11071 ± 12% -56.3% 4835 ± 26% interrupts.CPU29.RES:Rescheduling_interrupts
728.50 ± 66% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts
728.50 ± 66% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts
10599 ± 10% -38.7% 6501 ± 16% interrupts.CPU31.RES:Rescheduling_interrupts
2213 ± 4% +14.2% 2527 ± 7% interrupts.CPU32.CAL:Function_call_interrupts
8862 ± 16% -39.6% 5354 ± 28% interrupts.CPU32.RES:Rescheduling_interrupts
2156 ± 6% +20.5% 2599 ± 9% interrupts.CPU33.CAL:Function_call_interrupts
2169 ± 5% +19.9% 2601 ± 9% interrupts.CPU34.CAL:Function_call_interrupts
10483 ± 18% -48.8% 5371 ± 33% interrupts.CPU34.RES:Rescheduling_interrupts
2049 +24.4% 2549 ± 8% interrupts.CPU35.CAL:Function_call_interrupts
10929 ± 15% -49.0% 5578 ± 14% interrupts.CPU35.RES:Rescheduling_interrupts
1993 ± 4% +28.5% 2561 ± 12% interrupts.CPU36.CAL:Function_call_interrupts
161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.NMI:Non-maskable_interrupts
161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.PMI:Performance_monitoring_interrupts
1987 ± 4% +30.5% 2594 ± 13% interrupts.CPU37.CAL:Function_call_interrupts
1863 ± 10% +35.7% 2529 ± 14% interrupts.CPU38.CAL:Function_call_interrupts
1984 ± 6% +26.5% 2509 ± 12% interrupts.CPU41.CAL:Function_call_interrupts
12830 ± 30% -26.4% 9445 ± 9% interrupts.CPU43.RES:Rescheduling_interrupts
2067 ± 2% +27.4% 2632 ± 9% interrupts.CPU44.CAL:Function_call_interrupts
2066 ± 3% +27.2% 2629 ± 9% interrupts.CPU45.CAL:Function_call_interrupts
2158 ± 3% +19.6% 2582 ± 9% interrupts.CPU46.CAL:Function_call_interrupts
2036 ± 6% +30.3% 2652 ± 10% interrupts.CPU47.CAL:Function_call_interrupts
150.50 ± 43% -28.0% 108.33 interrupts.CPU49.84:PCI-MSI.1572913-edge.eth0-TxRx-49
11823 ± 12% -36.9% 7460 ± 18% interrupts.CPU49.RES:Rescheduling_interrupts
1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.NMI:Non-maskable_interrupts
1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.PMI:Performance_monitoring_interrupts
11748 ± 22% -40.0% 7048 ± 27% interrupts.CPU5.RES:Rescheduling_interrupts
2102 ± 5% +19.6% 2515 ± 7% interrupts.CPU52.CAL:Function_call_interrupts
2115 ± 6% +19.5% 2527 ± 7% interrupts.CPU53.CAL:Function_call_interrupts
11318 ± 17% -20.4% 9004 ± 26% interrupts.CPU53.RES:Rescheduling_interrupts
2177 ± 5% +17.4% 2556 ± 9% interrupts.CPU54.CAL:Function_call_interrupts
10968 ± 23% -62.3% 4134 ± 21% interrupts.CPU54.RES:Rescheduling_interrupts
2161 ± 5% +20.2% 2598 ± 9% interrupts.CPU55.CAL:Function_call_interrupts
9334 ± 19% -55.4% 4158 ± 34% interrupts.CPU55.RES:Rescheduling_interrupts
1443 ± 22% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts
1443 ± 22% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts
11377 ± 31% -56.9% 4907 ± 43% interrupts.CPU56.RES:Rescheduling_interrupts
601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.NMI:Non-maskable_interrupts
601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.PMI:Performance_monitoring_interrupts
8295 ± 31% -47.7% 4340 ± 24% interrupts.CPU57.RES:Rescheduling_interrupts
9182 ± 37% -59.7% 3701 ± 46% interrupts.CPU58.RES:Rescheduling_interrupts
249.50 ± 76% -50.4% 123.67 ± 2% interrupts.CPU6.41:PCI-MSI.1572870-edge.eth0-TxRx-6
12473 ± 27% -44.4% 6940 ± 13% interrupts.CPU6.RES:Rescheduling_interrupts
7886 ± 12% -42.2% 4555 ± 31% interrupts.CPU60.RES:Rescheduling_interrupts
971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.NMI:Non-maskable_interrupts
971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.PMI:Performance_monitoring_interrupts
9253 ± 26% -56.4% 4033 ± 16% interrupts.CPU61.RES:Rescheduling_interrupts
2103 ± 5% +19.1% 2505 ± 11% interrupts.CPU63.CAL:Function_call_interrupts
2059 ± 2% +20.6% 2483 ± 10% interrupts.CPU64.CAL:Function_call_interrupts
8545 ± 17% -41.2% 5025 ± 8% interrupts.CPU65.RES:Rescheduling_interrupts
7378 ± 23% -51.0% 3614 ± 30% interrupts.CPU68.RES:Rescheduling_interrupts
8573 ± 28% -50.1% 4277 ± 6% interrupts.CPU69.RES:Rescheduling_interrupts
154.50 ± 13% -10.9% 137.67 ± 10% interrupts.CPU7.42:PCI-MSI.1572871-edge.eth0-TxRx-7
9910 ± 28% -51.2% 4839 ± 40% interrupts.CPU70.RES:Rescheduling_interrupts
8836 ± 26% -52.7% 4180 ± 6% interrupts.CPU71.RES:Rescheduling_interrupts
911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.NMI:Non-maskable_interrupts
911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.PMI:Performance_monitoring_interrupts
12326 ± 15% -32.3% 8350 ± 10% interrupts.CPU8.RES:Rescheduling_interrupts
1392 ± 58% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts
1392 ± 58% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts
60185 ± 19% -30.5% 41822 ± 11% interrupts.NMI:Non-maskable_interrupts
60185 ± 19% -30.5% 41822 ± 11% interrupts.PMI:Performance_monitoring_interrupts
749419 -31.9% 510225 ± 3% interrupts.RES:Rescheduling_interrupts
15.05 ± 6% -4.1 10.97 ± 20% perf-profile.calltrace.cycles-pp.normal_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
12.66 ± 5% -4.0 8.68 ± 20% perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread.kthread
5.99 ± 8% -1.6 4.37 ± 6% perf-profile.calltrace.cycles-pp.io_submit
5.83 ± 8% -1.6 4.25 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.io_submit
5.82 ± 8% -1.6 4.24 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit
5.81 ± 8% -1.6 4.23 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit
5.76 ± 8% -1.6 4.19 ± 6% perf-profile.calltrace.cycles-pp.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit
5.65 ± 6% -1.6 4.08 ± 20% perf-profile.calltrace.cycles-pp.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread
4.52 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread
6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback
6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
5.58 ± 8% -1.5 4.07 ± 6% perf-profile.calltrace.cycles-pp.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.50 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work
5.44 ± 8% -1.5 3.96 ± 7% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64
6.63 ± 8% -1.5 5.17 ± 10% perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode
5.34 ± 9% -1.4 3.90 ± 6% perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit
5.55 ± 8% -1.2 4.32 ± 10% perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages
4.98 ± 8% -1.1 3.86 ± 10% perf-profile.calltrace.cycles-pp.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages
5.00 ± 8% -1.1 3.88 ± 10% perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages
2.80 ± 4% -1.0 1.78 ± 22% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work
1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.__btrfs_cow_block.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io
1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper
2.32 ± 3% -0.6 1.72 ± 18% perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper
0.91 ± 8% -0.5 0.42 ± 73% perf-profile.calltrace.cycles-pp.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper
0.88 ± 8% -0.5 0.41 ± 74% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io
0.83 ± 5% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums
1.90 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread
1.89 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
0.82 ± 4% -0.4 0.42 ± 71% perf-profile.calltrace.cycles-pp.run_one_async_done.normal_work_helper.process_one_work.worker_thread.kthread
0.81 ± 4% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.btrfs_map_bio.run_one_async_done.normal_work_helper.process_one_work.worker_thread
0.74 ± 6% -0.4 0.36 ± 70% perf-profile.calltrace.cycles-pp.push_leaf_right.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks
1.19 ± 3% -0.4 0.83 ± 16% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io
1.00 ± 6% -0.4 0.65 ± 7% perf-profile.calltrace.cycles-pp.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one
1.27 ± 5% -0.4 0.92 ± 7% perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one
1.25 ± 6% -0.3 0.90 ± 7% perf-profile.calltrace.cycles-pp.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage
0.93 ± 8% -0.3 0.59 ± 7% perf-profile.calltrace.cycles-pp.__set_extent_bit.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter
0.94 ± 7% -0.3 0.60 ± 6% perf-profile.calltrace.cycles-pp.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write
0.70 ± 8% -0.3 0.37 ± 70% perf-profile.calltrace.cycles-pp.__lookup_extent_mapping.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter
0.79 ± 6% -0.3 0.47 ± 71% perf-profile.calltrace.cycles-pp.setup_items_for_insert.btrfs_duplicate_item.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper
0.71 ± 21% -0.3 0.43 ± 74% perf-profile.calltrace.cycles-pp.__btrfs_free_extent.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space
1.15 ± 11% -0.3 0.88 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc
1.15 ± 11% -0.3 0.88 ± 8% perf-profile.calltrace.cycles-pp.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage
1.43 ± 13% -0.3 1.17 ± 5% perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one
1.06 ± 5% -0.3 0.81 ± 7% perf-profile.calltrace.cycles-pp.btrfs_drop_extent_cache.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc
1.30 ± 15% -0.2 1.06 ± 4% perf-profile.calltrace.cycles-pp.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write
1.03 ± 13% -0.2 0.79 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range
1.20 ± 4% -0.2 0.96 ± 15% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
1.24 ± 3% -0.2 1.01 ± 15% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
1.05 ± 8% -0.2 0.82 ± 12% perf-profile.calltrace.cycles-pp.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages
0.88 ± 13% -0.2 0.71 ± 2% perf-profile.calltrace.cycles-pp.wait_reserve_ticket.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter
1.01 ± 9% -0.2 0.84 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc
1.01 ± 10% -0.2 0.85 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage
0.72 ± 7% -0.2 0.56 ± 6% perf-profile.calltrace.cycles-pp.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write
0.75 ± 8% -0.1 0.60 ± 10% perf-profile.calltrace.cycles-pp.submit_extent_page.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages
0.92 ± 15% +0.2 1.14 ± 3% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
1.01 ± 15% +0.2 1.25 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
1.15 ± 15% +0.3 1.44 ± 6% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
0.27 ±100% +0.4 0.63 ± 4% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
1.74 ± 9% +0.4 2.11 ± 2% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
2.63 ± 10% +0.6 3.26 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
0.00 +0.8 0.75 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook
2.37 ± 19% +0.8 3.19 ± 5% perf-profile.calltrace.cycles-pp.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread.ret_from_fork
2.33 ± 19% +0.8 3.17 ± 5% perf-profile.calltrace.cycles-pp.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread
0.00 +0.9 0.90 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook
4.71 ± 11% +1.0 5.74 perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
5.04 ± 10% +1.2 6.19 perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
0.00 +1.3 1.30 ± 19% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range
0.00 +1.4 1.36 ± 19% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents
0.29 ±101% +1.4 1.67 ± 18% perf-profile.calltrace.cycles-pp.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread
0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction
0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction
0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space
0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space
0.00 +1.5 1.52 ± 18% perf-profile.calltrace.cycles-pp.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio
0.00 +2.2 2.20 ± 14% perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn
7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work.worker_thread
7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work
49.45 ± 2% +4.3 53.79 ± 4% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
61.98 ± 2% +4.4 66.38 ± 3% perf-profile.calltrace.cycles-pp.secondary_startup_64
1.07 ± 18% +5.0 6.06 ± 20% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback
0.93 ± 18% +5.0 5.95 ± 20% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes
61.26 ± 2% +5.1 66.36 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
61.27 ± 2% +5.1 66.37 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
61.21 ± 2% +5.1 66.32 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.33 ±100% +5.3 5.68 ± 20% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode
55.45 ± 2% +5.4 60.88 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.00 +6.4 6.39 ± 13% perf-profile.calltrace.cycles-pp.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page
0.00 +6.5 6.50 ± 13% perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb
0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages
0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages
15.05 ± 6% -4.1 10.97 ± 20% perf-profile.children.cycles-pp.normal_work_helper
12.66 ± 5% -4.0 8.68 ± 20% perf-profile.children.cycles-pp.btrfs_finish_ordered_io
7.84 ± 5% -2.4 5.39 ± 10% perf-profile.children.cycles-pp.btrfs_search_slot
7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_writepages
7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_write_cache_pages
7.03 ± 7% -1.6 5.38 ± 12% perf-profile.children.cycles-pp.__extent_writepage
6.19 ± 8% -1.6 4.56 ± 7% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
6.17 ± 8% -1.6 4.55 ± 7% perf-profile.children.cycles-pp.do_syscall_64
6.01 ± 8% -1.6 4.39 ± 7% perf-profile.children.cycles-pp.io_submit
5.81 ± 8% -1.6 4.23 ± 6% perf-profile.children.cycles-pp.__x64_sys_io_submit
5.76 ± 8% -1.6 4.19 ± 6% perf-profile.children.cycles-pp.io_submit_one
5.66 ± 5% -1.6 4.09 ± 20% perf-profile.children.cycles-pp.btrfs_mark_extent_written
4.52 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.add_pending_csums
5.59 ± 8% -1.5 4.07 ± 6% perf-profile.children.cycles-pp.aio_write
4.51 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.btrfs_csum_file_blocks
5.44 ± 8% -1.5 3.96 ± 7% perf-profile.children.cycles-pp.btrfs_file_write_iter
5.34 ± 9% -1.4 3.90 ± 6% perf-profile.children.cycles-pp.btrfs_buffered_write
5.88 ± 7% -1.4 4.47 ± 11% perf-profile.children.cycles-pp.writepage_delalloc
5.30 ± 8% -1.3 4.01 ± 11% perf-profile.children.cycles-pp.btrfs_run_delalloc_range
5.28 ± 8% -1.3 3.99 ± 11% perf-profile.children.cycles-pp.run_delalloc_nocow
2.55 ± 2% -0.9 1.67 ± 16% perf-profile.children.cycles-pp.btrfs_cow_block
2.54 ± 2% -0.9 1.66 ± 16% perf-profile.children.cycles-pp.__btrfs_cow_block
2.15 ± 13% -0.8 1.31 ± 8% perf-profile.children.cycles-pp.__lookup_extent_mapping
1.87 ± 12% -0.8 1.04 ± 12% perf-profile.children.cycles-pp.__etree_search
0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.printk
0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.vprintk_emit
2.60 ± 2% -0.7 1.92 ± 14% perf-profile.children.cycles-pp.btrfs_insert_empty_items
3.00 ± 5% -0.7 2.33 ± 6% perf-profile.children.cycles-pp._raw_spin_lock
1.89 ± 2% -0.7 1.23 ± 7% perf-profile.children.cycles-pp.__set_extent_bit
2.78 ± 4% -0.6 2.14 ± 14% perf-profile.children.cycles-pp.__schedule
1.57 ± 2% -0.5 1.02 ± 9% perf-profile.children.cycles-pp.lock_extent_bits
1.19 ± 14% -0.5 0.70 ± 11% perf-profile.children.cycles-pp.__clear_extent_bit
1.51 ± 7% -0.5 1.04 ± 9% perf-profile.children.cycles-pp.read_block_for_search
2.08 ± 5% -0.5 1.62 ± 18% perf-profile.children.cycles-pp.setup_items_for_insert
2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.btrfs_run_delayed_refs
2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.__btrfs_run_delayed_refs
1.60 ± 6% -0.4 1.20 ± 12% perf-profile.children.cycles-pp.schedule
1.95 ± 11% -0.4 1.58 ± 12% perf-profile.children.cycles-pp.try_to_wake_up
1.31 ± 6% -0.4 0.94 ± 9% perf-profile.children.cycles-pp.create_io_em
0.99 ± 4% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush
1.00 ± 6% -0.4 0.65 ± 7% perf-profile.children.cycles-pp.lock_and_cleanup_extent_if_need
1.27 ± 9% -0.4 0.92 ± 11% perf-profile.children.cycles-pp.__wake_up_common_lock
0.98 ± 3% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.btrfs_alloc_tree_block
1.27 ± 5% -0.4 0.92 ± 7% perf-profile.children.cycles-pp.btrfs_dirty_pages
1.34 ± 7% -0.3 0.99 ± 10% perf-profile.children.cycles-pp.btrfs_map_bio
1.11 ± 9% -0.3 0.77 ± 11% perf-profile.children.cycles-pp.generic_bin_search
1.16 ± 6% -0.3 0.82 ± 9% perf-profile.children.cycles-pp.find_extent_buffer
0.91 ± 7% -0.3 0.57 ± 21% perf-profile.children.cycles-pp.btrfs_lookup_csum
1.21 ± 11% -0.3 0.90 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_csums_range
1.21 ± 11% -0.3 0.91 ± 9% perf-profile.children.cycles-pp.csum_exist_in_range
1.10 ± 10% -0.3 0.80 ± 9% perf-profile.children.cycles-pp.__wake_up_common
0.37 ± 70% -0.3 0.07 ± 18% perf-profile.children.cycles-pp.test_range_bit
1.61 ± 4% -0.3 1.32 ± 15% perf-profile.children.cycles-pp.push_leaf_right
0.60 ± 4% -0.3 0.31 ± 25% perf-profile.children.cycles-pp.btrfs_extend_item
1.02 ± 9% -0.3 0.73 ± 10% perf-profile.children.cycles-pp.autoremove_wake_function
1.12 ± 4% -0.3 0.84 ± 9% perf-profile.children.cycles-pp.btrfs_drop_extent_cache
0.82 ± 6% -0.3 0.55 ± 4% perf-profile.children.cycles-pp.btrfs_release_path
0.85 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_extent_buffer_full
0.99 ± 4% -0.3 0.73 ± 13% perf-profile.children.cycles-pp.btrfs_set_token_32
1.44 ± 13% -0.3 1.18 ± 5% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata
0.84 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_page
0.87 ± 11% -0.3 0.62 ± 11% perf-profile.children.cycles-pp.pagecache_get_page
0.84 ± 7% -0.2 0.60 ± 11% perf-profile.children.cycles-pp.kmem_cache_alloc
1.12 ± 7% -0.2 0.88 ± 17% perf-profile.children.cycles-pp.__extent_writepage_io
1.30 ± 15% -0.2 1.06 ± 5% perf-profile.children.cycles-pp.reserve_metadata_bytes
0.82 ± 4% -0.2 0.58 ± 13% perf-profile.children.cycles-pp.run_one_async_done
1.25 ± 3% -0.2 1.01 ± 15% perf-profile.children.cycles-pp.schedule_idle
0.34 ± 53% -0.2 0.10 ± 28% perf-profile.children.cycles-pp.delay_tsc
0.70 ± 13% -0.2 0.47 ± 11% perf-profile.children.cycles-pp.prepare_pages
0.50 ± 22% -0.2 0.28 ± 13% perf-profile.children.cycles-pp.clear_state_bit
0.84 ± 5% -0.2 0.63 ± 6% perf-profile.children.cycles-pp.btrfs_get_extent
0.54 ± 11% -0.2 0.34 ± 4% perf-profile.children.cycles-pp.btrfs_free_path
1.16 ± 11% -0.2 0.96 ± 19% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.53 ± 9% -0.2 0.33 ± 18% perf-profile.children.cycles-pp.unpin_extent_cache
1.07 ± 10% -0.2 0.88 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_file_extent
0.50 ± 4% -0.2 0.31 ± 15% perf-profile.children.cycles-pp.clear_extent_bit
0.45 ± 13% -0.2 0.26 ± 11% perf-profile.children.cycles-pp.kmem_cache_free
0.40 ± 14% -0.2 0.21 ± 50% perf-profile.children.cycles-pp.__writeback_inodes_wb
0.64 ± 7% -0.2 0.45 ± 10% perf-profile.children.cycles-pp.dequeue_task_fair
0.50 ± 8% -0.2 0.33 ± 23% perf-profile.children.cycles-pp.btrfs_add_delayed_tree_ref
0.88 ± 13% -0.2 0.71 ± 2% perf-profile.children.cycles-pp.wait_reserve_ticket
0.52 ± 9% -0.2 0.35 ± 7% perf-profile.children.cycles-pp.__radix_tree_lookup
0.52 ± 8% -0.2 0.36 ± 8% perf-profile.children.cycles-pp.xas_load
0.78 ± 19% -0.2 0.62 ± 23% perf-profile.children.cycles-pp.__btrfs_free_extent
0.56 ± 17% -0.1 0.42 ± 15% perf-profile.children.cycles-pp.btrfs_del_items
0.51 ± 5% -0.1 0.36 ± 9% perf-profile.children.cycles-pp.dequeue_entity
0.39 ± 7% -0.1 0.24 ± 6% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
0.82 ± 14% -0.1 0.68 ± 18% perf-profile.children.cycles-pp.ttwu_do_activate
0.55 ± 8% -0.1 0.41 ± 8% perf-profile.children.cycles-pp.btrfs_submit_bio_hook
0.53 ± 5% -0.1 0.40 ± 13% perf-profile.children.cycles-pp.update_load_avg
0.40 ± 8% -0.1 0.27 ± 27% perf-profile.children.cycles-pp.add_delayed_ref_head
0.35 ± 13% -0.1 0.22 ± 3% perf-profile.children.cycles-pp.set_extent_bit
0.52 ± 9% -0.1 0.39 ± 7% perf-profile.children.cycles-pp.btrfs_wq_submit_bio
0.35 ± 3% -0.1 0.23 ± 32% perf-profile.children.cycles-pp.queued_write_lock_slowpath
0.33 ± 6% -0.1 0.21 ± 15% perf-profile.children.cycles-pp.__slab_alloc
0.63 ± 19% -0.1 0.51 ± 13% perf-profile.children.cycles-pp.sched_ttwu_pending
0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode
0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode_fallback
0.57 ± 5% -0.1 0.46 ± 10% perf-profile.children.cycles-pp.find_lock_delalloc_range
0.45 ± 5% -0.1 0.34 ± 9% perf-profile.children.cycles-pp.btrfs_cross_ref_exist
0.29 ± 3% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.free_extent_buffer
0.38 ± 3% -0.1 0.26 ± 8% perf-profile.children.cycles-pp.btrfs_mark_buffer_dirty
0.46 ± 5% -0.1 0.35 ± 13% perf-profile.children.cycles-pp.find_delalloc_range
0.41 ± 9% -0.1 0.30 ± 2% perf-profile.children.cycles-pp.btrfs_copy_from_user
0.40 ± 9% -0.1 0.29 ± 2% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
0.38 ± 10% -0.1 0.27 perf-profile.children.cycles-pp.find_get_entry
0.31 ± 12% -0.1 0.20 ± 31% perf-profile.children.cycles-pp.btrfs_remove_ordered_extent
0.30 ± 6% -0.1 0.20 ± 14% perf-profile.children.cycles-pp.___slab_alloc
0.26 ± 8% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.merge_state
0.38 ± 7% -0.1 0.28 ± 3% perf-profile.children.cycles-pp.copyin
0.26 ± 17% -0.1 0.15 ± 26% perf-profile.children.cycles-pp.btrfs_delayed_update_inode
0.25 ± 4% -0.1 0.15 ± 5% perf-profile.children.cycles-pp.insert_state
0.40 ± 13% -0.1 0.30 ± 13% perf-profile.children.cycles-pp.extent_clear_unlock_delalloc
0.36 ± 9% -0.1 0.26 ± 3% perf-profile.children.cycles-pp.alloc_extent_state
0.38 ± 7% -0.1 0.29 ± 3% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.39 ± 3% -0.1 0.29 ± 14% perf-profile.children.cycles-pp.alloc_extent_buffer
0.31 ± 4% -0.1 0.22 ± 11% perf-profile.children.cycles-pp.set_extent_buffer_dirty
0.26 ± 11% -0.1 0.17 ± 23% perf-profile.children.cycles-pp.unlock_up
0.32 ± 3% -0.1 0.24 ± 13% perf-profile.children.cycles-pp.__btrfs_map_block
0.23 ± 13% -0.1 0.15 ± 28% perf-profile.children.cycles-pp.btrfs_add_delayed_data_ref
0.25 ± 12% -0.1 0.17 ± 2% perf-profile.children.cycles-pp.__slab_free
0.47 ± 6% -0.1 0.39 ± 15% perf-profile.children.cycles-pp.__push_leaf_right
0.26 ± 9% -0.1 0.18 ± 19% perf-profile.children.cycles-pp.__test_set_page_writeback
0.32 ± 11% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.btrfs_add_ordered_extent
0.15 ± 8% -0.1 0.07 ± 6% perf-profile.children.cycles-pp.__xa_set_mark
0.23 ± 7% -0.1 0.15 ± 3% perf-profile.children.cycles-pp.release_extent_buffer
0.18 ± 8% -0.1 0.11 ± 26% perf-profile.children.cycles-pp.alloc_extent_map
0.32 ± 10% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.__btrfs_add_ordered_extent
0.32 ± 17% -0.1 0.24 ± 10% perf-profile.children.cycles-pp.btrfs_tree_read_lock_atomic
0.20 ± 13% -0.1 0.13 ± 27% perf-profile.children.cycles-pp.btrfs_inc_extent_ref
0.14 ± 21% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.find_ref_head
0.13 ± 32% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.rb_erase_cached
0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_submit_bio_start
0.53 ± 12% -0.1 0.46 ± 16% perf-profile.children.cycles-pp.btrfs_tree_read_lock
0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_csum_one_bio
0.17 ± 16% -0.1 0.10 ± 8% perf-profile.children.cycles-pp.rb_erase
0.21 ± 7% -0.1 0.14 ± 16% perf-profile.children.cycles-pp.run_one_async_start
0.22 ± 3% -0.1 0.15 ± 22% perf-profile.children.cycles-pp.btrfs_try_tree_write_lock
0.21 ± 9% -0.1 0.15 ± 21% perf-profile.children.cycles-pp.btrfs_leaf_free_space
0.15 ± 23% -0.1 0.09 ± 35% perf-profile.children.cycles-pp.free_extent_map
0.20 ± 9% -0.1 0.14 ± 20% perf-profile.children.cycles-pp.leaf_space_used
0.19 ± 8% -0.1 0.13 ± 13% perf-profile.children.cycles-pp.check_delayed_ref
0.17 ± 13% -0.1 0.12 ± 31% perf-profile.children.cycles-pp.get_page_from_freelist
0.13 ± 8% -0.1 0.08 ± 26% perf-profile.children.cycles-pp.__update_load_avg_se
0.17 ± 2% -0.1 0.12 ± 18% perf-profile.children.cycles-pp.btrfs_set_range_writeback
0.15 ± 8% -0.1 0.09 ± 20% perf-profile.children.cycles-pp.xas_set_mark
0.18 ± 6% -0.1 0.13 ± 16% perf-profile.children.cycles-pp.___might_sleep
0.20 ± 11% -0.1 0.15 ± 35% perf-profile.children.cycles-pp.__alloc_pages_nodemask
0.10 ± 9% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_unlock
0.14 ± 11% -0.1 0.09 ± 28% perf-profile.children.cycles-pp.btrfs_reserve_extent
0.10 ± 33% -0.1 0.05 ± 72% perf-profile.children.cycles-pp.__btrfs_release_delayed_node
0.22 ± 9% -0.0 0.17 ± 25% perf-profile.children.cycles-pp.mark_extent_buffer_accessed
0.09 ± 13% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking
0.18 ± 8% -0.0 0.14 ± 24% perf-profile.children.cycles-pp.__list_del_entry_valid
0.09 ± 13% -0.0 0.04 ± 73% perf-profile.children.cycles-pp.verify_parent_transid
0.11 ± 19% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.entry_SYSCALL_64
0.22 ± 5% -0.0 0.18 ± 12% perf-profile.children.cycles-pp.check_committed_ref
0.11 ± 19% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.select_idle_sibling
0.14 ± 11% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.do_io_getevents
0.15 ± 7% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.__x64_sys_io_getevents
0.08 ± 10% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_release_extent_buffer_pages
0.17 ± 18% -0.0 0.13 ± 9% perf-profile.children.cycles-pp.clear_page_dirty_for_io
0.14 ± 5% -0.0 0.10 ± 25% perf-profile.children.cycles-pp.block_group_cache_tree_search
0.18 ± 4% -0.0 0.14 ± 11% perf-profile.children.cycles-pp.btrfs_get_chunk_map
0.11 ± 13% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.btrfs_buffer_uptodate
0.15 ± 15% -0.0 0.11 ± 19% perf-profile.children.cycles-pp.xas_find_marked
0.12 ± 15% -0.0 0.08 ± 20% perf-profile.children.cycles-pp.btrfs_unlock_up_safe
0.12 ± 14% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.btrfs_set_path_blocking
0.11 ± 11% -0.0 0.07 ± 28% perf-profile.children.cycles-pp.find_free_extent
0.24 ± 5% -0.0 0.20 ± 6% perf-profile.children.cycles-pp.btrfs_root_node
0.07 ± 6% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.extent_mergeable
0.13 ± 12% -0.0 0.10 ± 9% perf-profile.children.cycles-pp.__switch_to
0.10 ± 18% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.rb_insert_color_cached
0.07 ± 14% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.rb_prev
0.11 ± 11% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.get_io_u
0.10 ± 14% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.available_idle_cpu
0.17 ± 10% -0.0 0.13 ± 3% perf-profile.children.cycles-pp._raw_write_lock
0.10 ± 11% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.__might_sleep
0.17 ± 7% -0.0 0.14 ± 12% perf-profile.children.cycles-pp._raw_read_lock
0.17 ± 10% -0.0 0.14 ± 12% perf-profile.children.cycles-pp.can_overcommit
0.10 ± 11% -0.0 0.07 perf-profile.children.cycles-pp.__list_add_valid
0.09 ± 7% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.aio_read_events
0.11 ± 10% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.___perf_sw_event
0.11 ± 9% -0.0 0.09 ± 18% perf-profile.children.cycles-pp.get_alloc_profile
0.11 ± 12% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.pick_next_task_idle
0.10 ± 13% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.read_events
0.08 ± 6% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.replace_extent_mapping
0.10 ± 5% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.menu_reflect
0.07 ± 17% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.run_posix_cpu_timers
0.05 ± 58% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.rcu_irq_enter
0.19 ± 11% +0.1 0.26 ± 17% perf-profile.children.cycles-pp.native_irq_return_iret
0.20 ± 14% +0.1 0.27 ± 8% perf-profile.children.cycles-pp.btrfs_comp_cpu_keys
0.38 ± 9% +0.1 0.46 perf-profile.children.cycles-pp.load_balance
0.30 ± 8% +0.1 0.38 ± 15% perf-profile.children.cycles-pp.irq_enter
0.21 ± 24% +0.1 0.30 ± 9% perf-profile.children.cycles-pp.rcu_sched_clock_irq
0.22 ± 3% +0.1 0.32 ± 9% perf-profile.children.cycles-pp.lapic_next_deadline
0.52 ± 12% +0.1 0.63 ± 3% perf-profile.children.cycles-pp.rebalance_domains
1.90 ± 10% +0.3 2.23 ± 2% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.23 ± 12% +0.4 0.67 ± 15% perf-profile.children.cycles-pp.read_extent_buffer
2.88 ± 11% +0.5 3.41 perf-profile.children.cycles-pp.hrtimer_interrupt
0.09 ± 12% +0.7 0.83 ± 15% perf-profile.children.cycles-pp.btrfs_get_token_8
2.37 ± 19% +0.8 3.19 ± 5% perf-profile.children.cycles-pp.btrfs_async_reclaim_metadata_space
2.33 ± 19% +0.8 3.17 ± 5% perf-profile.children.cycles-pp.flush_space
0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_and_wait_transaction
0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_marked_extents
0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
0.42 ± 43% +1.3 1.67 ± 18% perf-profile.children.cycles-pp.btrfs_commit_transaction
0.25 ± 5% +1.9 2.17 ± 10% perf-profile.children.cycles-pp.btrfs_get_token_64
1.71 ± 5% +2.1 3.79 ± 5% perf-profile.children.cycles-pp.btrfs_get_token_32
0.43 ± 8% +2.4 2.84 ± 10% perf-profile.children.cycles-pp.map_private_extent_buffer
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_writeback
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.writeback_sb_inodes
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.__writeback_single_inode
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_workfn
50.06 ± 2% +3.8 53.82 ± 4% perf-profile.children.cycles-pp.intel_idle
61.99 ± 2% +4.4 66.39 ± 3% perf-profile.children.cycles-pp.do_idle
61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.secondary_startup_64
61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.cpu_startup_entry
8.51 ± 7% +4.5 13.04 ± 12% perf-profile.children.cycles-pp.do_writepages
56.43 ± 3% +4.9 61.30 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state
61.27 ± 2% +5.1 66.37 ± 3% perf-profile.children.cycles-pp.start_secondary
1.43 ± 7% +6.2 7.64 ± 13% perf-profile.children.cycles-pp.submit_extent_page
1.20 ± 17% +6.2 7.45 ± 13% perf-profile.children.cycles-pp.btree_write_cache_pages
1.05 ± 17% +6.3 7.31 ± 13% perf-profile.children.cycles-pp.write_one_eb
1.11 ± 7% +6.3 7.37 ± 13% perf-profile.children.cycles-pp.submit_one_bio
0.55 ± 16% +6.4 6.95 ± 13% perf-profile.children.cycles-pp.btree_submit_bio_hook
0.00 +6.5 6.48 ± 13% perf-profile.children.cycles-pp.check_leaf
0.00 +6.5 6.53 ± 13% perf-profile.children.cycles-pp.btree_csum_one_bio
2.12 ± 11% -0.8 1.29 ± 7% perf-profile.self.cycles-pp.__lookup_extent_mapping
1.83 ± 12% -0.8 1.03 ± 13% perf-profile.self.cycles-pp.__etree_search
2.06 ± 5% -0.5 1.59 ± 5% perf-profile.self.cycles-pp._raw_spin_lock
0.81 ± 9% -0.3 0.52 ± 10% perf-profile.self.cycles-pp.generic_bin_search
0.84 ± 7% -0.3 0.58 ± 17% perf-profile.self.cycles-pp.copy_page
0.33 ± 54% -0.2 0.10 ± 28% perf-profile.self.cycles-pp.delay_tsc
0.89 ± 9% -0.2 0.69 ± 14% perf-profile.self.cycles-pp.btrfs_set_token_32
0.81 ± 4% -0.2 0.63 ± 14% perf-profile.self.cycles-pp.__schedule
0.51 ± 9% -0.2 0.35 ± 6% perf-profile.self.cycles-pp.__radix_tree_lookup
0.45 ± 8% -0.1 0.32 ± 10% perf-profile.self.cycles-pp.xas_load
0.43 ± 4% -0.1 0.31 ± 6% perf-profile.self.cycles-pp.find_extent_buffer
0.77 ± 5% -0.1 0.66 ± 11% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.31 ± 10% -0.1 0.20 ± 28% perf-profile.self.cycles-pp.add_delayed_ref_head
0.39 ± 16% -0.1 0.29 ± 16% perf-profile.self.cycles-pp.try_to_wake_up
0.31 ± 4% -0.1 0.20 ± 26% perf-profile.self.cycles-pp.queued_write_lock_slowpath
0.38 ± 7% -0.1 0.29 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.20 ± 17% -0.1 0.11 ± 14% perf-profile.self.cycles-pp.kmem_cache_free
0.25 ± 11% -0.1 0.17 ± 2% perf-profile.self.cycles-pp.__slab_free
0.16 ± 9% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.free_extent_buffer
0.28 ± 17% -0.1 0.20 ± 11% perf-profile.self.cycles-pp.btrfs_search_slot
0.31 ± 4% -0.1 0.24 ± 20% perf-profile.self.cycles-pp.kmem_cache_alloc
0.14 ± 21% -0.1 0.06 ± 19% perf-profile.self.cycles-pp.find_ref_head
0.16 ± 10% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.insert_state
0.12 ± 8% -0.1 0.06 ± 72% perf-profile.self.cycles-pp.__update_load_avg_se
0.16 ± 15% -0.1 0.10 ± 8% perf-profile.self.cycles-pp.rb_erase
0.10 ± 9% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_unlock
0.15 ± 13% -0.1 0.10 ± 12% perf-profile.self.cycles-pp.___slab_alloc
0.28 ± 13% -0.1 0.22 ± 5% perf-profile.self.cycles-pp.queued_read_lock_slowpath
0.15 ± 21% -0.1 0.09 ± 35% perf-profile.self.cycles-pp.free_extent_map
0.18 ± 6% -0.1 0.12 ± 11% perf-profile.self.cycles-pp.update_load_avg
0.10 ± 11% -0.1 0.04 ± 70% perf-profile.self.cycles-pp.btrfs_extend_item
0.15 ± 23% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.add_extent_mapping
0.15 ± 8% -0.1 0.09 ± 20% perf-profile.self.cycles-pp.xas_set_mark
0.18 ± 6% -0.1 0.13 ± 16% perf-profile.self.cycles-pp.___might_sleep
0.18 ± 6% -0.0 0.13 ± 22% perf-profile.self.cycles-pp.__list_del_entry_valid
0.18 ± 4% -0.0 0.14 ± 22% perf-profile.self.cycles-pp.pick_next_task_fair
0.09 ± 13% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.verify_parent_transid
0.09 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_read_unlock_blocking
0.16 ± 13% -0.0 0.12 ± 24% perf-profile.self.cycles-pp.mark_page_accessed
0.12 ± 14% -0.0 0.08 ± 20% perf-profile.self.cycles-pp.dequeue_task_fair
0.10 ± 19% -0.0 0.07 ± 18% perf-profile.self.cycles-pp.entry_SYSCALL_64
0.08 ± 14% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.unlock_up
0.15 ± 15% -0.0 0.11 ± 19% perf-profile.self.cycles-pp.xas_find_marked
0.24 ± 6% -0.0 0.20 ± 8% perf-profile.self.cycles-pp.btrfs_root_node
0.17 ± 6% -0.0 0.13 ± 10% perf-profile.self.cycles-pp._raw_read_lock
0.10 ± 13% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.btrfs_csum_one_bio
0.16 ± 7% -0.0 0.13 ± 9% perf-profile.self.cycles-pp.dequeue_entity
0.07 -0.0 0.04 ± 71% perf-profile.self.cycles-pp.io_u_queued_complete
0.08 -0.0 0.05 ± 70% perf-profile.self.cycles-pp.mutex_lock
0.11 ± 11% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.get_io_u
0.16 ± 9% -0.0 0.13 perf-profile.self.cycles-pp._raw_write_lock
0.07 ± 17% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.btrfs_tree_read_lock
0.12 ± 8% -0.0 0.09 ± 13% perf-profile.self.cycles-pp.__switch_to
0.10 ± 11% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.__clear_extent_bit
0.07 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.rb_prev
0.10 ± 15% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.find_get_pages_range_tag
0.10 ± 12% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.available_idle_cpu
0.10 ± 11% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.__list_add_valid
0.08 ± 13% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.__might_sleep
0.12 ± 17% -0.0 0.09 ± 15% perf-profile.self.cycles-pp.release_extent_buffer
0.09 ± 17% -0.0 0.07 ± 14% perf-profile.self.cycles-pp.___perf_sw_event
0.08 ± 5% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.__push_leaf_right
0.11 ± 7% -0.0 0.09 ± 9% perf-profile.self.cycles-pp.set_extent_buffer_dirty
0.11 ± 4% -0.0 0.08 ± 11% perf-profile.self.cycles-pp.__set_extent_bit
0.09 ± 7% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.hrtimer_interrupt
0.07 ± 17% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.run_posix_cpu_timers
0.07 ± 7% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.load_balance
0.01 ±173% +0.1 0.07 ± 17% perf-profile.self.cycles-pp.lapic_next_deadline
0.00 +0.1 0.06 ± 23% perf-profile.self.cycles-pp.find_next_and_bit
0.19 ± 11% +0.1 0.26 ± 17% perf-profile.self.cycles-pp.native_irq_return_iret
0.19 ± 24% +0.1 0.28 ± 7% perf-profile.self.cycles-pp.rcu_sched_clock_irq
0.50 ± 8% +0.2 0.69 ± 17% perf-profile.self.cycles-pp.cpuidle_enter_state
0.07 ± 11% +0.4 0.46 ± 17% perf-profile.self.cycles-pp.btrfs_get_token_8
0.22 ± 13% +0.4 0.66 ± 16% perf-profile.self.cycles-pp.read_extent_buffer
0.00 +0.6 0.65 ± 14% perf-profile.self.cycles-pp.check_leaf
0.15 ± 11% +1.0 1.13 ± 11% perf-profile.self.cycles-pp.btrfs_get_token_64
1.48 ± 7% +1.0 2.51 ± 5% perf-profile.self.cycles-pp.btrfs_get_token_32
0.41 ± 8% +2.2 2.61 ± 9% perf-profile.self.cycles-pp.map_private_extent_buffer
49.99 ± 2% +3.7 53.74 ± 4% perf-profile.self.cycles-pp.intel_idle
fio.write_bw_MBps
300 +-+-------------------------------------------------------------------+
| |
250 +-++.+ +. .+.. .+..+..+.+.. .+..+.. .+ |
| : : +. +..+ + +..+ +. + |
| : : : : : : |
200 +-+ : : : : : : O O |
O O O: O :O O O O O O O O O O: O:O O: O: O O O O O
150 +-+ : : : : : : |
| : : : : : : |
100 +-+ : : : : : : |
| : : : : : : |
| : : : : : : |
50 +-+ :: :: :: |
| : : : |
0 +-+---------------------------------------------O-------------O-------+
fio.write_iops
70000 +-+-----------------------------------------------------------------+
|..+.+ +..+..+.+.. .+.+..+.+..+ +.. +..+.+..+..+ |
60000 +-+ : : +. : : + : |
| : : : : : : |
50000 +-+ : : : : : : |
O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O
40000 +-+ : O: : : : : |
| : : : : : : |
30000 +-+ : : : : : : |
| : : : : : : |
20000 +-+ : : : : : : |
| :: :: :: |
10000 +-+ : : : |
| : : : |
0 +-+--------------------------------------------O------------O-------+
fio.write_clat_mean_us
30000 +-+-----------------------------------------------------------------+
| |
25000 O-+O O O O O O O O O O O O O O O O O O O |
| O O O O O
| |
20000 +-+ |
|..+.+ +..+..+.+..+..+.+..+.+..+ +..+ +..+.+..+..+ |
15000 +-+ : : : : : : |
| : : : : : : |
10000 +-+ : : : : : : |
| : : : : : : |
| : : : : : : |
5000 +-+ : : : : :: |
| : : : |
0 +-+--------------------------------------------O------------O-------+
fio.write_clat_stddev
160000 +-+----------------------------------------------------------------+
| |
140000 O-+O O O O O O O O O O O O O O |
120000 +-+ O O |
| |
100000 +-+ O O O O O O O O
| |
80000 +-+ |
| .+.. |
60000 +-++.+ +..+.+. +.+..+.+..+..+ +.+ +..+..+.+..+ |
40000 +-+ : : : : : : |
| : : : : : : |
20000 +-+ : : : : : : |
| :: :: :: |
0 +-+-------------------------------------------O------------O-------+
fio.write_slat_mean_us
900 +-+-------------------------------------------------------------------+
O O O O O O O O O O O O O O O O O O |
800 +-+O O O O O O O
700 +-+ |
| |
600 +-++.+ +.+..+..+..+.+..+..+.+..+ +..+ +..+..+..+.+ |
500 +-+ : : : : : : |
| : : : : : : |
400 +-+ : : : : : : |
300 +-+ : : : : : : |
| : : : : : : |
200 +-+ : : : : : : |
100 +-+ : : :: :: |
| : : : |
0 +-+---------------------------------------------O-------------O-------+
fio.write_slat_stddev
25000 +-+-----O---------O-------------------------------------------------+
O O O O O O O O O O O O O O O |
| |
20000 +-+ |
| O O O O O O O O
| |
15000 +-+ |
| |
10000 +-++. .+.. .+.. .+..+ .+ .+..+ |
|. + +..+..+ +..+ + : +. : +..+.+. |
| : : : : : : |
5000 +-+ : : : : : : |
| : : : : : : |
| :: : :: |
0 +-+--------------------------------------------O------------O-------+
fio.latency_1000ms_
0.4 +-+------------------------------------------------------------------+
| |
0.35 +-+ O |
0.3 +-+ |
| O O |
0.25 +-+ O O |
| O O O
0.2 O-+ O O O O O O O O O O |
| O O |
0.15 +-+O O O O |
0.1 +-+ |
| |
0.05 +-+ |
| .+. .+..+..+.+..+.. .+. .+.+..+.. |
0 +-+---------------------------------------------O------------O-------+
fio.workload
1.4e+07 +-+---------------------------------------------------------------+
|..+.+ +..+.+..+.+..+.+..+.+..+ +.. +.+..+.+..+ |
1.2e+07 +-+ : : : : + : |
| : : : : : : |
1e+07 +-+ : : : : : : |
O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O
8e+06 +-+ : O: : : : : |
| : : : : : : |
6e+06 +-+ : : : : : : |
| : : : : : : |
4e+06 +-+ : : : : : : |
| :: :: :: |
2e+06 +-+ : : : |
| : : : |
0 +-+-------------------------------------------O-----------O-------+
fio.time.voluntary_context_switches
1.4e+07 +-+---------------------------------------------------------------+
| |
1.2e+07 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ |
| : : : : : : |
1e+07 +-+ : : : : : : |
| O O: : O O O : : :O : O O O O O O O
8e+06 O-+ : O:O O O O O O O: O:O O: : |
| : : : : : : |
6e+06 +-+ : : : : : : |
| : : : : : : |
4e+06 +-+ : : : : : : |
| : : : : :: |
2e+06 +-+ : : :: |
| : : : |
0 +-+-------------------------------------------O-----------O-------+
fio.time.file_system_outputs
1.2e+08 +-+---------------------------------------------------------------+
| |
1e+08 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ |
| : : : : : : |
| : : : : : : |
8e+07 +-+ : : : : : : O O O O O
O O O: O:O O O O O O O O O O: O:O O:O : O O |
6e+07 +-+ : : : : : : |
| : : : : : : |
4e+07 +-+ : : : : : : |
| : : : : : : |
| : : : : :: |
2e+07 +-+ : : :: |
| : : : |
0 +-+-------------------------------------------O-----------O-------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
3 months, 3 weeks
- 4
- 4
[sched/fair] 3c29e651e1: hackbench.throughput -15.2% regression
by kernel test robot
Greeting,
FYI, we noticed a -15.2% regression of hackbench.throughput due to commit:
commit: 3c29e651e16dd3b3179cfb2d055ee9538e37515c ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: hackbench
on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
with following parameters:
nr_threads: 100%
mode: threads
ipc: pipe
cpufreq_governor: performance
ucode: 0xca
test-description: Hackbench is both a benchmark and a stress test for the Linux kernel scheduler.
test-url: https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/sc...
In addition to that, the commit also has significant impact on the following tests:
+------------------+-----------------------------------------------------------------------+
| testcase: change | hackbench: hackbench.throughput -11.3% regression |
| test machine | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory |
| test parameters | cpufreq_governor=performance |
| | ipc=pipe |
| | iterations=18 |
| | mode=threads |
| | nr_threads=1600% |
| | ucode=0x43 |
+------------------+-----------------------------------------------------------------------+
| testcase: change | hackbench: hackbench.throughput -10.6% regression |
| test machine | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory |
| test parameters | cpufreq_governor=performance |
| | ipc=pipe |
| | mode=process |
| | nr_threads=50% |
| | ucode=0xb000038 |
+------------------+-----------------------------------------------------------------------+
| testcase: change | hackbench: hackbench.throughput -6.8% regression |
| test machine | 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory |
| test parameters | cpufreq_governor=performance |
| | ipc=pipe |
| | mode=threads |
| | nr_threads=100% |
| | ucode=0xb8 |
+------------------+-----------------------------------------------------------------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen(a)intel.com>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
gcc-7/performance/pipe/x86_64-rhel-7.6/threads/100%/debian-x86_64-20191114.cgz/lkp-cfl-e1/hackbench/0xca
commit:
43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq")
3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found")
43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055
---------------- ---------------------------
%stddev %change %stddev
\ | \
69492 ± 3% -15.2% 58899 hackbench.throughput
7.727e+08 ± 7% +38.3% 1.069e+09 ± 2% hackbench.time.involuntary_context_switches
131708 ± 4% -15.4% 111382 hackbench.time.minor_page_faults
684.98 ± 2% -2.5% 667.72 hackbench.time.user_time
1.755e+09 ± 2% +11.2% 1.953e+09 hackbench.time.voluntary_context_switches
4.32e+08 ± 4% -15.6% 3.648e+08 hackbench.workload
186.50 ± 12% -16.6% 155.50 ± 11% interrupts.TLB:TLB_shootdowns
4147892 ± 5% +19.6% 4961643 vmstat.system.cs
0.72 ± 10% -0.1 0.58 ± 17% mpstat.cpu.all.idle%
0.00 ±156% +0.0 0.01 ± 62% mpstat.cpu.all.soft%
1038 ± 9% +18.5% 1230 ± 4% slabinfo.avc_xperms_data.active_objs
1038 ± 9% +18.5% 1230 ± 4% slabinfo.avc_xperms_data.num_objs
40647036 ± 10% -34.6% 26589504 proc-vmstat.numa_hit
40647036 ± 10% -34.6% 26589504 proc-vmstat.numa_local
40724029 ± 10% -34.5% 26661546 proc-vmstat.pgalloc_normal
763906 ± 2% -7.5% 706781 proc-vmstat.pgfault
40701242 ± 10% -34.6% 26637947 proc-vmstat.pgfree
1636058 ± 11% -45.0% 899629 ± 11% turbostat.C1
0.10 ± 7% -0.0 0.07 ± 7% turbostat.C1%
15201 ± 25% +113.3% 32422 ± 8% turbostat.C1E
0.01 ± 57% +0.0 0.03 ± 13% turbostat.C1E%
9093 ± 11% +217.2% 28841 ± 11% turbostat.C3
0.00 ±173% +0.0 0.03 ± 13% turbostat.C3%
16977 ± 2% -92.8% 1222 ± 98% turbostat.C8
0.17 ± 4% -0.2 0.01 ±110% turbostat.C8%
0.13 -100.0% 0.00 turbostat.CPU%c7
9623106 ± 5% -35.1% 6241651 ± 8% cpuidle.C1.time
1636668 ± 11% -45.0% 899926 ± 11% cpuidle.C1.usage
899494 ± 33% +272.6% 3351412 ± 5% cpuidle.C1E.time
15327 ± 25% +112.6% 32592 ± 8% cpuidle.C1E.usage
435674 ± 20% +643.5% 3239351 ± 14% cpuidle.C3.time
9179 ± 11% +215.2% 28933 ± 11% cpuidle.C3.usage
16850924 ± 2% -94.2% 979516 ± 86% cpuidle.C8.time
17150 ± 2% -92.0% 1378 ± 88% cpuidle.C8.usage
6022328 ± 18% -57.1% 2584773 ± 7% cpuidle.POLL.time
5339703 ± 19% -60.1% 2130089 ± 7% cpuidle.POLL.usage
14.06 ± 25% +64.0% 23.07 ± 13% sched_debug.cfs_rq:/.runnable_load_avg.min
25.10 ± 5% -15.8% 21.12 ± 7% sched_debug.cfs_rq:/.runnable_load_avg.stddev
13339 ± 24% +45.7% 19433 ± 7% sched_debug.cfs_rq:/.runnable_weight.min
166.51 ± 24% +48.9% 247.86 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.min
253.56 ± 2% -12.3% 222.30 ± 9% sched_debug.cfs_rq:/.util_est_enqueued.stddev
159602 ± 23% +55.6% 248411 ± 6% sched_debug.cpu.avg_idle.max
51567 ± 24% +49.1% 76905 ± 11% sched_debug.cpu.avg_idle.stddev
11576 ± 5% -9.5% 10478 sched_debug.cpu.curr->pid.avg
9976 ± 8% -16.8% 8305 ± 9% sched_debug.cpu.curr->pid.min
4.81 ± 15% +43.3% 6.89 ± 8% sched_debug.cpu.nr_running.min
6.68 ± 4% -17.0% 5.54 ± 8% sched_debug.cpu.nr_running.stddev
76003928 ± 2% +21.0% 91975904 sched_debug.cpu.nr_switches.avg
78115404 ± 2% +23.4% 96397551 sched_debug.cpu.nr_switches.max
74275755 ± 2% +19.0% 88359903 sched_debug.cpu.nr_switches.min
1043054 ± 15% +83.9% 1917752 ± 10% sched_debug.cpu.nr_switches.stddev
-685.26 +64.8% -1129 sched_debug.cpu.nr_uninterruptible.min
15470 ± 9% -31.0% 10674 ± 9% softirqs.CPU0.SCHED
13756 ± 10% -40.8% 8140 ± 4% softirqs.CPU1.SCHED
13645 ± 11% -39.7% 8227 ± 3% softirqs.CPU10.SCHED
14065 ± 12% -38.8% 8609 ± 2% softirqs.CPU11.SCHED
13948 ± 11% -39.9% 8381 ± 6% softirqs.CPU12.SCHED
13823 ± 12% -39.7% 8332 ± 9% softirqs.CPU13.SCHED
73323 +13.4% 83165 ± 11% softirqs.CPU14.RCU
13872 ± 12% -36.5% 8809 ± 7% softirqs.CPU14.SCHED
13726 ± 12% -40.4% 8175 ± 4% softirqs.CPU15.SCHED
13780 ± 11% -34.8% 8987 ± 12% softirqs.CPU2.SCHED
13895 ± 10% -40.5% 8266 ± 6% softirqs.CPU3.SCHED
221879 +13.4% 251589 ± 14% softirqs.CPU3.TIMER
13873 ± 10% -40.4% 8262 ± 6% softirqs.CPU4.SCHED
72625 +12.0% 81355 ± 10% softirqs.CPU5.RCU
13876 ± 13% -41.4% 8137 ± 3% softirqs.CPU5.SCHED
14118 ± 12% -40.8% 8354 ± 3% softirqs.CPU6.SCHED
13503 ± 12% -40.0% 8099 softirqs.CPU7.SCHED
13549 ± 11% -38.2% 8374 ± 8% softirqs.CPU8.SCHED
14273 ± 10% -40.1% 8548 ± 6% softirqs.CPU9.SCHED
223184 ± 11% -38.9% 136383 ± 5% softirqs.SCHED
55.80 +9.9% 61.34 perf-stat.i.MPKI
1.32 +0.1 1.39 perf-stat.i.branch-miss-rate%
1.066e+08 +3.7% 1.105e+08 perf-stat.i.branch-misses
0.15 ± 8% -0.0 0.11 ± 3% perf-stat.i.cache-miss-rate%
2521584 ± 9% -26.5% 1854353 ± 3% perf-stat.i.cache-misses
2.27e+09 +8.0% 2.451e+09 perf-stat.i.cache-references
4162783 ± 4% +19.7% 4981708 perf-stat.i.context-switches
1.52 +2.0% 1.55 perf-stat.i.cpi
227633 ± 14% +75.1% 398694 ± 3% perf-stat.i.cpu-migrations
33931 ± 12% +42.7% 48428 ± 2% perf-stat.i.cycles-between-cache-misses
0.01 -0.0 0.01 ± 4% perf-stat.i.dTLB-load-miss-rate%
1408442 ± 2% -16.4% 1177727 ± 4% perf-stat.i.dTLB-load-misses
1.229e+10 -2.1% 1.203e+10 perf-stat.i.dTLB-loads
0.00 ± 2% -0.0 0.00 ± 4% perf-stat.i.dTLB-store-miss-rate%
35757 ± 2% -16.0% 30032 ± 3% perf-stat.i.dTLB-store-misses
7.449e+09 -2.3% 7.279e+09 perf-stat.i.dTLB-stores
39246970 -5.5% 37079604 perf-stat.i.iTLB-load-misses
76813 ± 8% -26.2% 56724 ± 6% perf-stat.i.iTLB-loads
4.096e+10 -1.9% 4.02e+10 perf-stat.i.instructions
1062 +4.1% 1105 perf-stat.i.instructions-per-iTLB-miss
0.66 -1.9% 0.65 perf-stat.i.ipc
1210 -7.1% 1124 perf-stat.i.minor-faults
105889 ± 10% -24.8% 79596 ± 5% perf-stat.i.node-loads
174508 ± 7% -24.4% 131915 perf-stat.i.node-stores
1210 -7.1% 1124 perf-stat.i.page-faults
55.42 ± 2% +10.0% 60.98 perf-stat.overall.MPKI
1.31 +0.1 1.38 perf-stat.overall.branch-miss-rate%
0.11 ± 11% -0.0 0.08 ± 3% perf-stat.overall.cache-miss-rate%
1.51 +1.9% 1.54 perf-stat.overall.cpi
24740 ± 9% +34.9% 33377 ± 3% perf-stat.overall.cycles-between-cache-misses
0.01 -0.0 0.01 ± 4% perf-stat.overall.dTLB-load-miss-rate%
0.00 ± 2% -0.0 0.00 ± 4% perf-stat.overall.dTLB-store-miss-rate%
1043 +3.9% 1084 perf-stat.overall.instructions-per-iTLB-miss
0.66 -1.9% 0.65 perf-stat.overall.ipc
57826 ± 3% +15.6% 66874 perf-stat.overall.path-length
1.064e+08 +3.7% 1.104e+08 perf-stat.ps.branch-misses
2517524 ± 9% -26.5% 1851335 ± 3% perf-stat.ps.cache-misses
2.266e+09 +8.0% 2.447e+09 perf-stat.ps.cache-references
4155898 ± 4% +19.7% 4973458 perf-stat.ps.context-switches
227258 ± 14% +75.1% 398041 ± 3% perf-stat.ps.cpu-migrations
1406119 ± 2% -16.4% 1175777 ± 4% perf-stat.ps.dTLB-load-misses
1.227e+10 -2.1% 1.201e+10 perf-stat.ps.dTLB-loads
35699 ± 2% -16.0% 29983 ± 3% perf-stat.ps.dTLB-store-misses
7.437e+09 -2.3% 7.267e+09 perf-stat.ps.dTLB-stores
39182154 -5.5% 37018206 perf-stat.ps.iTLB-load-misses
76695 ± 8% -26.2% 56638 ± 6% perf-stat.ps.iTLB-loads
4.09e+10 -1.9% 4.013e+10 perf-stat.ps.instructions
1208 -7.1% 1122 perf-stat.ps.minor-faults
105716 ± 10% -24.8% 79465 ± 5% perf-stat.ps.node-loads
174234 ± 7% -24.4% 131703 perf-stat.ps.node-stores
1208 -7.1% 1122 perf-stat.ps.page-faults
29.39 ±104% -29.4 0.00 perf-profile.calltrace.cycles-pp.start_thread
15.86 ±105% -15.9 0.00 perf-profile.calltrace.cycles-pp.__GI___libc_write.start_thread
14.88 ±105% -14.9 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread
14.75 ±105% -14.8 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread
13.53 ±104% -13.5 0.00 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread
12.51 ±104% -12.5 0.00 perf-profile.calltrace.cycles-pp.__GI___libc_read.start_thread
12.45 ±105% -12.4 0.00 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
11.21 ±105% -11.2 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread
11.08 ±105% -11.1 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread
10.47 ±105% -10.5 0.00 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread
9.73 ±105% -9.7 0.00 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
3.09 ± 12% -0.6 2.50 ± 2% perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read
2.45 ± 14% -0.5 1.95 ± 2% perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write
1.15 ± 13% -0.2 0.95 ± 4% perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.pipe_write.new_sync_write.vfs_write
0.99 ± 13% -0.2 0.83 ± 5% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter.pipe_write.new_sync_write
0.92 ± 15% +0.2 1.10 perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.84 ± 23% +0.3 1.10 ± 2% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait
0.99 ± 19% +0.3 1.26 ± 3% perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
0.97 ± 23% +0.3 1.26 perf-profile.calltrace.cycles-pp.__enqueue_entity.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
0.29 ±100% +0.3 0.60 ± 5% perf-profile.calltrace.cycles-pp.reschedule_interrupt.__lock_text_start.__wake_up_common_lock.pipe_write.new_sync_write
0.55 ± 62% +0.3 0.89 ± 7% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule
0.28 ±100% +0.4 0.63 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop
0.33 ±100% +0.4 0.69 ± 2% perf-profile.calltrace.cycles-pp.prepare_to_wait.pipe_wait.pipe_read.new_sync_read.vfs_read
0.29 ±100% +0.4 0.65 ± 4% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read
1.00 ± 27% +0.4 1.37 ± 2% perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
0.57 ± 63% +0.4 0.95 ± 5% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate
1.46 ± 31% +0.4 1.86 ± 2% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.72 ± 60% +0.4 1.14 ± 2% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.29 ±100% +0.4 0.70 ± 6% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait
0.34 ±100% +0.4 0.77 ± 7% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task
1.26 ± 26% +0.5 1.72 perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
1.13 ± 26% +0.5 1.59 perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
0.35 ±100% +0.5 0.83 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop.do_syscall_64
1.28 ± 28% +0.5 1.77 ± 3% perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait
1.40 ± 26% +0.5 1.89 perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
0.44 ±100% +0.5 0.95 ± 7% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop
2.33 ± 17% +0.5 2.88 ± 2% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_wait.pipe_read
1.33 ± 28% +0.6 1.89 ± 2% perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
1.40 ± 24% +0.6 1.97 ± 4% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
0.83 ± 66% +0.6 1.40 ± 5% perf-profile.calltrace.cycles-pp.native_write_msr
5.01 ± 8% +0.8 5.81 ± 2% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
1.75 ± 39% +0.9 2.61 ± 6% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64
1.33 ± 67% +1.0 2.28 ± 6% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe
6.56 ± 12% +1.2 7.78 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_wait.pipe_read
8.96 ± 9% +1.6 10.56 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
9.23 ± 10% +1.6 10.87 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
9.17 ± 10% +1.6 10.81 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
3.72 ± 31% +1.7 5.44 ± 6% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.37 ± 8% +1.9 5.32 ± 4% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
4.37 ± 10% +2.3 6.65 ± 5% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
15.27 ± 10% +2.3 17.59 perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait.pipe_read.new_sync_read
15.59 ± 10% +2.4 17.96 perf-profile.calltrace.cycles-pp.schedule.pipe_wait.pipe_read.new_sync_read.vfs_read
3.00 ± 71% +2.6 5.60 ± 6% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.10 ± 71% +2.7 5.77 ± 6% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
18.05 ± 10% +2.7 20.73 perf-profile.calltrace.cycles-pp.pipe_wait.pipe_read.new_sync_read.vfs_read.ksys_read
28.21 ± 5% +3.7 31.87 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
24.29 ± 8% +4.7 28.98 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
23.38 ± 8% +4.7 28.08 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
23.72 ± 8% +4.7 28.43 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
31.18 ± 41% +12.7 43.86 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
33.44 ± 40% +13.2 46.61 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
64.81 ± 43% +27.5 92.34 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
65.30 ± 43% +27.7 92.99 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
29.39 ±104% -29.4 0.00 perf-profile.children.cycles-pp.start_thread
16.09 ±105% -16.1 0.00 perf-profile.children.cycles-pp.__GI___libc_write
12.73 ±104% -12.7 0.00 perf-profile.children.cycles-pp.__GI___libc_read
5.80 ± 10% -1.2 4.65 ± 3% perf-profile.children.cycles-pp.security_file_permission
4.10 ± 16% -1.0 3.10 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
3.21 ± 9% -0.7 2.55 ± 3% perf-profile.children.cycles-pp.copy_page_to_iter
1.45 ± 32% -0.6 0.83 perf-profile.children.cycles-pp.entry_SYSCALL_64
3.04 ± 8% -0.6 2.44 ± 3% perf-profile.children.cycles-pp.selinux_file_permission
2.56 ± 11% -0.6 1.99 ± 2% perf-profile.children.cycles-pp.copy_page_from_iter
2.80 ± 9% -0.5 2.25 ± 2% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
3.38 ± 7% -0.5 2.87 ± 2% perf-profile.children.cycles-pp.mutex_lock
1.62 ± 12% -0.4 1.23 ± 3% perf-profile.children.cycles-pp.___might_sleep
1.82 ± 9% -0.4 1.43 ± 4% perf-profile.children.cycles-pp.syscall_return_via_sysret
2.09 ± 12% -0.4 1.72 ± 6% perf-profile.children.cycles-pp.file_has_perm
0.90 ± 20% -0.3 0.57 ± 9% perf-profile.children.cycles-pp.__mutex_lock
1.14 ± 12% -0.3 0.82 ± 3% perf-profile.children.cycles-pp.__inode_security_revalidate
1.26 ± 10% -0.3 0.97 ± 4% perf-profile.children.cycles-pp.copyin
1.70 ± 8% -0.3 1.42 ± 3% perf-profile.children.cycles-pp.copyout
1.25 ± 11% -0.3 0.98 ± 3% perf-profile.children.cycles-pp.fsnotify
1.04 ± 9% -0.3 0.79 ± 3% perf-profile.children.cycles-pp._cond_resched
1.72 ± 7% -0.2 1.48 ± 5% perf-profile.children.cycles-pp.fput_many
1.16 ± 10% -0.2 0.92 ± 4% perf-profile.children.cycles-pp.__might_sleep
1.09 ± 8% -0.2 0.85 ± 3% perf-profile.children.cycles-pp.__fsnotify_parent
0.83 ± 13% -0.2 0.63 ± 3% perf-profile.children.cycles-pp.__might_fault
0.69 ± 11% -0.1 0.56 perf-profile.children.cycles-pp.current_time
0.99 ± 4% -0.1 0.89 ± 4% perf-profile.children.cycles-pp.touch_atime
0.65 ± 6% -0.1 0.55 ± 6% perf-profile.children.cycles-pp.atime_needs_update
0.39 ± 4% -0.1 0.29 perf-profile.children.cycles-pp.rcu_all_qs
0.13 ± 39% -0.1 0.04 ± 59% perf-profile.children.cycles-pp.mutex_spin_on_owner
0.34 ± 11% -0.1 0.26 ± 9% perf-profile.children.cycles-pp.preempt_schedule_common
0.49 ± 5% -0.1 0.41 ± 2% perf-profile.children.cycles-pp.wake_up_q
0.24 ± 2% -0.1 0.17 ± 4% perf-profile.children.cycles-pp.fpregs_assert_state_consistent
0.30 ± 11% -0.1 0.23 ± 3% perf-profile.children.cycles-pp.inode_has_perm
0.23 ± 8% -0.1 0.18 ± 6% perf-profile.children.cycles-pp.__sb_end_write
0.21 ± 9% -0.1 0.16 ± 9% perf-profile.children.cycles-pp.__x64_sys_read
0.29 ± 8% -0.0 0.24 ± 2% perf-profile.children.cycles-pp.timespec64_trunc
0.21 ± 8% -0.0 0.17 ± 8% perf-profile.children.cycles-pp.__x64_sys_write
0.22 ± 7% -0.0 0.19 ± 2% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
0.08 ± 16% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.bpf_fd_pass
0.10 ± 4% +0.0 0.11 ± 3% perf-profile.children.cycles-pp.perf_exclude_event
0.08 ± 24% +0.0 0.13 ± 5% perf-profile.children.cycles-pp.rcu_note_context_switch
0.17 ± 19% +0.1 0.24 ± 8% perf-profile.children.cycles-pp.wakeup_preempt_entity
0.23 ± 19% +0.1 0.32 ± 3% perf-profile.children.cycles-pp.resched_curr
0.61 ± 8% +0.1 0.72 ± 6% perf-profile.children.cycles-pp.sched_clock
0.67 ± 8% +0.1 0.78 ± 6% perf-profile.children.cycles-pp.sched_clock_cpu
0.50 ± 9% +0.1 0.62 ± 3% perf-profile.children.cycles-pp.account_entity_enqueue
0.61 ± 10% +0.2 0.79 perf-profile.children.cycles-pp.account_entity_dequeue
0.43 ± 19% +0.2 0.62 ± 7% perf-profile.children.cycles-pp.clear_buddies
1.42 ± 14% +0.2 1.66 ± 4% perf-profile.children.cycles-pp.native_write_msr
1.17 ± 13% +0.3 1.42 ± 2% perf-profile.children.cycles-pp.check_preempt_wakeup
0.85 ± 23% +0.3 1.12 ± 7% perf-profile.children.cycles-pp.put_prev_entity
0.94 ± 14% +0.3 1.22 ± 5% perf-profile.children.cycles-pp.___perf_sw_event
1.45 ± 11% +0.3 1.73 perf-profile.children.cycles-pp.check_preempt_curr
1.45 ± 12% +0.3 1.76 perf-profile.children.cycles-pp.__enqueue_entity
1.60 ± 10% +0.3 1.91 perf-profile.children.cycles-pp.ttwu_do_wakeup
1.12 ± 9% +0.3 1.43 ± 3% perf-profile.children.cycles-pp.__update_load_avg_se
1.64 ± 8% +0.4 2.00 ± 4% perf-profile.children.cycles-pp.available_idle_cpu
0.96 ± 19% +0.4 1.34 ± 7% perf-profile.children.cycles-pp.pick_next_entity
1.85 ± 16% +0.4 2.28 ± 6% perf-profile.children.cycles-pp.switch_fpu_return
1.90 ± 9% +0.6 2.50 ± 3% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
3.22 ± 13% +0.6 3.87 perf-profile.children.cycles-pp.update_load_avg
4.03 ± 10% +0.6 4.68 ± 2% perf-profile.children.cycles-pp.update_curr
3.01 ± 14% +0.7 3.72 ± 2% perf-profile.children.cycles-pp.reweight_entity
5.10 ± 8% +0.8 5.87 ± 2% perf-profile.children.cycles-pp.enqueue_entity
4.71 ± 15% +1.1 5.77 ± 4% perf-profile.children.cycles-pp.pick_next_task_fair
6.73 ± 11% +1.1 7.87 perf-profile.children.cycles-pp.dequeue_task_fair
9.07 ± 9% +1.5 10.61 perf-profile.children.cycles-pp.enqueue_task_fair
9.32 ± 9% +1.6 10.89 perf-profile.children.cycles-pp.ttwu_do_activate
91.46 +1.6 93.03 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
9.27 ± 9% +1.6 10.84 perf-profile.children.cycles-pp.activate_task
4.16 ± 26% +1.6 5.78 ± 6% perf-profile.children.cycles-pp.exit_to_usermode_loop
90.79 +1.7 92.47 perf-profile.children.cycles-pp.do_syscall_64
3.48 ± 8% +1.9 5.36 ± 4% perf-profile.children.cycles-pp.select_idle_sibling
4.44 ± 10% +2.2 6.67 ± 5% perf-profile.children.cycles-pp.select_task_rq_fair
18.46 ± 9% +2.3 20.81 perf-profile.children.cycles-pp.pipe_wait
19.93 ± 11% +3.5 23.46 ± 2% perf-profile.children.cycles-pp.__schedule
28.82 ± 5% +3.6 32.45 perf-profile.children.cycles-pp.__wake_up_common_lock
19.89 ± 12% +3.7 23.62 ± 2% perf-profile.children.cycles-pp.schedule
23.96 ± 8% +4.5 28.45 perf-profile.children.cycles-pp.try_to_wake_up
24.53 ± 8% +4.5 29.06 perf-profile.children.cycles-pp.__wake_up_common
23.91 ± 8% +4.6 28.46 perf-profile.children.cycles-pp.autoremove_wake_function
2.76 ± 9% -0.5 2.23 ± 2% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
1.46 ± 18% -0.4 1.06 ± 4% perf-profile.self.cycles-pp.pipe_write
1.81 ± 9% -0.4 1.43 ± 4% perf-profile.self.cycles-pp.syscall_return_via_sysret
1.56 ± 12% -0.4 1.19 ± 3% perf-profile.self.cycles-pp.___might_sleep
2.09 ± 7% -0.3 1.81 ± 2% perf-profile.self.cycles-pp.mutex_lock
1.86 ± 8% -0.3 1.59 ± 3% perf-profile.self.cycles-pp.selinux_file_permission
1.21 ± 11% -0.3 0.96 ± 2% perf-profile.self.cycles-pp.fsnotify
1.70 ± 7% -0.2 1.46 ± 5% perf-profile.self.cycles-pp.fput_many
1.04 ± 11% -0.2 0.82 ± 3% perf-profile.self.cycles-pp.__might_sleep
1.01 ± 8% -0.2 0.80 ± 3% perf-profile.self.cycles-pp.__fsnotify_parent
1.04 ± 10% -0.2 0.83 perf-profile.self.cycles-pp.entry_SYSCALL_64
0.67 ± 10% -0.2 0.51 ± 5% perf-profile.self.cycles-pp.new_sync_write
0.64 ± 12% -0.2 0.48 ± 5% perf-profile.self.cycles-pp.vfs_write
1.67 -0.2 1.51 ± 2% perf-profile.self.cycles-pp.pipe_read
0.58 ± 17% -0.1 0.44 ± 7% perf-profile.self.cycles-pp.file_has_perm
0.44 ± 18% -0.1 0.31 ± 7% perf-profile.self.cycles-pp.__mutex_lock
0.49 ± 13% -0.1 0.35 ± 3% perf-profile.self.cycles-pp.copy_page_from_iter
0.64 ± 8% -0.1 0.52 ± 4% perf-profile.self.cycles-pp.copy_page_to_iter
0.67 ± 7% -0.1 0.55 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.30 ± 5% -0.1 0.20 ± 4% perf-profile.self.cycles-pp.ksys_write
0.40 ± 12% -0.1 0.30 ± 7% perf-profile.self.cycles-pp.security_file_permission
0.13 ± 38% -0.1 0.04 ± 59% perf-profile.self.cycles-pp.mutex_spin_on_owner
0.36 ± 11% -0.1 0.28 ± 3% perf-profile.self.cycles-pp.__inode_security_revalidate
0.76 ± 5% -0.1 0.68 ± 5% perf-profile.self.cycles-pp.do_syscall_64
0.21 ± 2% -0.1 0.14 ± 8% perf-profile.self.cycles-pp.fpregs_assert_state_consistent
0.26 ± 12% -0.1 0.20 ± 4% perf-profile.self.cycles-pp.inode_has_perm
0.29 ± 11% -0.1 0.23 ± 6% perf-profile.self.cycles-pp.current_time
0.28 ± 4% -0.1 0.21 ± 2% perf-profile.self.cycles-pp.rcu_all_qs
0.22 ± 7% -0.1 0.17 ± 7% perf-profile.self.cycles-pp.__sb_end_write
0.31 -0.1 0.26 ± 7% perf-profile.self.cycles-pp.ksys_read
0.20 ± 11% -0.0 0.16 ± 9% perf-profile.self.cycles-pp.__x64_sys_read
0.26 ± 8% -0.0 0.22 perf-profile.self.cycles-pp.timespec64_trunc
0.20 ± 5% -0.0 0.16 ± 9% perf-profile.self.cycles-pp.__x64_sys_write
0.24 ± 9% -0.0 0.21 ± 6% perf-profile.self.cycles-pp.__might_fault
0.09 ± 9% -0.0 0.06 perf-profile.self.cycles-pp.copyin
0.14 ± 5% -0.0 0.12 ± 3% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
0.07 ± 10% +0.0 0.09 ± 4% perf-profile.self.cycles-pp.sched_clock
0.16 ± 5% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.ttwu_do_wakeup
0.20 ± 12% +0.0 0.24 ± 5% perf-profile.self.cycles-pp.__list_add_valid
0.10 ± 26% +0.0 0.14 ± 10% perf-profile.self.cycles-pp.update_cfs_rq_h_load
0.15 ± 16% +0.1 0.21 ± 11% perf-profile.self.cycles-pp.wakeup_preempt_entity
0.35 ± 7% +0.1 0.43 ± 4% perf-profile.self.cycles-pp.prepare_to_wait
0.55 ± 2% +0.1 0.63 ± 3% perf-profile.self.cycles-pp.__fget_light
0.22 ± 18% +0.1 0.30 ± 3% perf-profile.self.cycles-pp.resched_curr
0.48 ± 11% +0.1 0.58 ± 3% perf-profile.self.cycles-pp.check_preempt_wakeup
0.39 ± 11% +0.1 0.50 perf-profile.self.cycles-pp.account_entity_enqueue
0.85 ± 7% +0.1 0.97 ± 4% perf-profile.self.cycles-pp.enqueue_entity
0.43 ± 16% +0.1 0.55 ± 7% perf-profile.self.cycles-pp.pick_next_entity
0.86 ± 10% +0.1 0.99 perf-profile.self.cycles-pp.dequeue_task_fair
0.37 ± 21% +0.2 0.55 ± 6% perf-profile.self.cycles-pp.clear_buddies
0.45 ± 14% +0.2 0.65 perf-profile.self.cycles-pp.account_entity_dequeue
1.41 ± 14% +0.2 1.65 ± 4% perf-profile.self.cycles-pp.native_write_msr
0.80 ± 16% +0.3 1.06 ± 4% perf-profile.self.cycles-pp.___perf_sw_event
1.09 ± 9% +0.3 1.39 ± 2% perf-profile.self.cycles-pp.__update_load_avg_se
0.84 ± 16% +0.3 1.14 ± 7% perf-profile.self.cycles-pp.select_task_rq_fair
1.45 ± 12% +0.3 1.75 perf-profile.self.cycles-pp.__enqueue_entity
1.62 ± 8% +0.4 1.99 ± 4% perf-profile.self.cycles-pp.available_idle_cpu
1.83 ± 16% +0.4 2.25 ± 6% perf-profile.self.cycles-pp.switch_fpu_return
2.15 ± 11% +0.5 2.60 ± 5% perf-profile.self.cycles-pp.update_curr
1.88 ± 9% +0.6 2.45 ± 3% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
1.21 ± 9% +0.7 1.92 ± 3% perf-profile.self.cycles-pp._raw_spin_lock
0.80 ± 11% +1.4 2.17 ± 3% perf-profile.self.cycles-pp.select_idle_sibling
hackbench.throughput
74000 +-+-----------------------------------------------------------------+
72000 +-+.+ .+ |
|.+ : .+. .+. + |
70000 +-+ : + + +. : |
68000 +-+ + + : |
| :: |
66000 +-+ + |
64000 +-+ |
62000 +-+ |
| |
60000 +-+ O O OO O O O O O O O OO |
58000 +-+ O O O O O O O O O O O
| O O OO O O |
56000 O-+ O O O O O O |
54000 +-O-----------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
***************************************************************************************************
lkp-hsw-ep4: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory
=========================================================================================
compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
gcc-7/performance/pipe/18/x86_64-rhel-7.6/threads/1600%/debian-x86_64-2019-11-14.cgz/lkp-hsw-ep4/hackbench/0x43
commit:
43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq")
3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found")
43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
%stddev %change %stddev
\ | \
176075 ± 2% -11.3% 156152 hackbench.throughput
891.79 +12.3% 1001 hackbench.time.elapsed_time
891.79 +12.3% 1001 hackbench.time.elapsed_time.max
1.889e+09 ± 9% +33.1% 2.514e+09 ± 3% hackbench.time.involuntary_context_switches
42939 ± 3% +17.0% 50220 hackbench.time.system_time
18978 +3.2% 19592 hackbench.time.user_time
2.558e+09 ± 8% +33.5% 3.415e+09 ± 2% hackbench.time.voluntary_context_switches
43133704 ± 3% +39.2% 60035429 ± 33% cpuidle.C1.time
48661 +17.7% 57274 ± 5% meminfo.Shmem
19043 ± 2% -14.0% 16377 meminfo.max_used_kB
2502 +1.3% 2535 turbostat.Avg_MHz
3.368e+08 ± 5% +30.6% 4.398e+08 turbostat.IRQ
66.25 +4.5% 69.25 vmstat.cpu.sy
29.25 ± 2% -8.5% 26.75 vmstat.cpu.us
5000443 ± 7% +18.6% 5931772 vmstat.system.cs
370173 ± 3% +17.1% 433616 vmstat.system.in
98786923 ± 7% +74.5% 1.724e+08 numa-numastat.node0.local_node
98791606 ± 7% +74.5% 1.724e+08 numa-numastat.node0.numa_hit
4686 ± 99% +297.8% 18642 ± 24% numa-numastat.node0.other_node
1.818e+08 -51.4% 88447082 ± 3% numa-numastat.node1.local_node
1.819e+08 -51.4% 88451833 ± 3% numa-numastat.node1.numa_hit
18694 ± 25% -74.6% 4753 ± 97% numa-numastat.node1.other_node
155316 +1.5% 157632 proc-vmstat.nr_active_anon
99.50 +5.3% 104.75 ± 3% proc-vmstat.nr_dirtied
4440 +4.9% 4656 proc-vmstat.nr_inactive_anon
6283 +1.9% 6399 proc-vmstat.nr_mapped
12248 ± 2% +16.6% 14276 ± 5% proc-vmstat.nr_shmem
98.75 +5.3% 104.00 proc-vmstat.nr_written
155316 +1.5% 157632 proc-vmstat.nr_zone_active_anon
4440 +4.9% 4656 proc-vmstat.nr_zone_inactive_anon
2.807e+08 -7.1% 2.609e+08 proc-vmstat.numa_hit
2.807e+08 -7.1% 2.609e+08 proc-vmstat.numa_local
2.83e+08 -7.0% 2.632e+08 proc-vmstat.pgalloc_normal
3212744 +2.3% 3286078 proc-vmstat.pgfault
2.828e+08 -7.0% 2.631e+08 proc-vmstat.pgfree
39216 ± 35% +153.3% 99342 ± 4% numa-vmstat.node0.nr_active_anon
31864 ± 44% +211.0% 99111 ± 4% numa-vmstat.node0.nr_anon_pages
11183 ± 3% +4212.8% 482321 ± 2% numa-vmstat.node0.nr_kernel_stack
499.50 ± 23% +66.9% 833.75 ± 8% numa-vmstat.node0.nr_page_table_pages
11417 -63.5% 4168 ± 5% numa-vmstat.node0.nr_shmem
15905 ± 4% -12.5% 13919 ± 2% numa-vmstat.node0.nr_slab_reclaimable
17996 ± 3% +587.4% 123708 ± 2% numa-vmstat.node0.nr_slab_unreclaimable
39216 ± 35% +153.3% 99342 ± 4% numa-vmstat.node0.nr_zone_active_anon
52471566 ± 6% +73.3% 90915199 numa-vmstat.node0.numa_hit
52466577 ± 6% +73.2% 90896366 numa-vmstat.node0.numa_local
4990 ± 92% +277.5% 18836 ± 24% numa-vmstat.node0.numa_other
114712 ± 12% -50.1% 57234 ± 7% numa-vmstat.node1.nr_active_anon
114545 ± 12% -58.2% 47874 ± 9% numa-vmstat.node1.nr_anon_pages
462075 -96.6% 15717 ± 67% numa-vmstat.node1.nr_kernel_stack
871.50 ± 13% -37.9% 541.00 ± 13% numa-vmstat.node1.nr_page_table_pages
691.25 ± 22% +1359.2% 10086 ± 8% numa-vmstat.node1.nr_shmem
11357 ± 5% +18.1% 13414 ± 4% numa-vmstat.node1.nr_slab_reclaimable
118594 -85.5% 17243 ± 16% numa-vmstat.node1.nr_slab_unreclaimable
114712 ± 12% -50.1% 57234 ± 7% numa-vmstat.node1.nr_zone_active_anon
93308245 ± 2% -49.7% 46921467 ± 4% numa-vmstat.node1.numa_hit
93128224 ± 2% -49.8% 46755145 ± 4% numa-vmstat.node1.numa_local
157762 ± 35% +154.4% 401333 ± 4% numa-meminfo.node0.Active
157721 ± 35% +154.4% 401249 ± 4% numa-meminfo.node0.Active(anon)
128232 ± 44% +212.2% 400337 ± 4% numa-meminfo.node0.AnonPages
63487 ± 4% -12.0% 55850 ± 2% numa-meminfo.node0.KReclaimable
11691 ± 3% +4087.3% 489553 numa-meminfo.node0.KernelStack
1170596 ± 4% +104.2% 2390602 numa-meminfo.node0.MemUsed
2003 ± 23% +67.3% 3351 ± 8% numa-meminfo.node0.PageTables
63487 ± 4% -12.0% 55850 ± 2% numa-meminfo.node0.SReclaimable
72301 ± 3% +591.9% 500285 numa-meminfo.node0.SUnreclaim
45749 -63.5% 16679 ± 5% numa-meminfo.node0.Shmem
135789 ± 3% +309.6% 556136 numa-meminfo.node0.Slab
460025 ± 11% -50.2% 228882 ± 7% numa-meminfo.node1.Active
459902 ± 11% -50.2% 228802 ± 7% numa-meminfo.node1.Active(anon)
459237 ± 11% -58.3% 191354 ± 9% numa-meminfo.node1.AnonPages
45568 ± 5% +17.8% 53657 ± 4% numa-meminfo.node1.KReclaimable
468955 -96.7% 15461 ± 67% numa-meminfo.node1.KernelStack
2241450 -51.9% 1078199 ± 3% numa-meminfo.node1.MemUsed
3493 ± 13% -38.0% 2167 ± 13% numa-meminfo.node1.PageTables
45568 ± 5% +17.8% 53657 ± 4% numa-meminfo.node1.SReclaimable
479010 -85.6% 68750 ± 16% numa-meminfo.node1.SUnreclaim
2779 ± 23% +1354.1% 40419 ± 8% numa-meminfo.node1.Shmem
524580 -76.7% 122408 ± 8% numa-meminfo.node1.Slab
250988 ± 19% -52.0% 120407 ± 33% sched_debug.cfs_rq:/.MIN_vruntime.avg
1469922 ± 14% -32.1% 998119 ± 33% sched_debug.cfs_rq:/.MIN_vruntime.stddev
20766 ± 9% -25.3% 15519 ± 10% sched_debug.cfs_rq:/.load.avg
43820 ± 19% -40.6% 26044 ± 36% sched_debug.cfs_rq:/.load.stddev
250988 ± 19% -52.0% 120407 ± 33% sched_debug.cfs_rq:/.max_vruntime.avg
1469922 ± 14% -32.1% 998119 ± 33% sched_debug.cfs_rq:/.max_vruntime.stddev
10990567 ± 9% +22.9% 13509735 ± 7% sched_debug.cfs_rq:/.min_vruntime.stddev
102.40 -33.3% 68.27 ± 57% sched_debug.cfs_rq:/.removed.load_avg.max
4711 -33.4% 3136 ± 57% sched_debug.cfs_rq:/.removed.runnable_sum.max
19930 ± 10% -25.1% 14917 ± 10% sched_debug.cfs_rq:/.runnable_weight.avg
-5713498 -349.0% 14226386 ± 8% sched_debug.cfs_rq:/.spread0.avg
8106715 ± 13% +283.1% 31057182 ± 6% sched_debug.cfs_rq:/.spread0.max
-18025645 -99.7% -45393 sched_debug.cfs_rq:/.spread0.min
11027552 ± 9% +21.5% 13398318 ± 7% sched_debug.cfs_rq:/.spread0.stddev
570.46 ± 5% +109.2% 1193 ± 21% sched_debug.cfs_rq:/.util_est_enqueued.avg
2898 ± 11% +90.0% 5507 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.max
563.27 ± 13% +149.3% 1404 ± 18% sched_debug.cfs_rq:/.util_est_enqueued.stddev
155578 ± 22% +84.1% 286457 ± 28% sched_debug.cpu.avg_idle.min
1678 ± 43% +105.9% 3457 ± 31% sched_debug.cpu.clock.stddev
1678 ± 43% +105.9% 3457 ± 31% sched_debug.cpu.clock_task.stddev
50207 -12.3% 44017 ± 7% sched_debug.cpu.curr->pid.avg
82673 -19.5% 66514 ± 5% sched_debug.cpu.curr->pid.max
746.05 ± 11% +554.3% 4881 ± 8% sched_debug.cpu.curr->pid.min
30728 ± 2% -40.8% 18187 ± 10% sched_debug.cpu.curr->pid.stddev
0.00 ± 43% +106.0% 0.00 ± 31% sched_debug.cpu.next_balance.stddev
20.49 ± 6% +149.0% 51.03 ± 23% sched_debug.cpu.nr_running.avg
157.83 ± 9% +77.0% 279.37 ± 8% sched_debug.cpu.nr_running.max
29.95 ± 11% +137.1% 70.99 ± 16% sched_debug.cpu.nr_running.stddev
29803727 ± 9% +29.6% 38623870 ± 3% sched_debug.cpu.nr_switches.avg
51549476 ± 10% +22.5% 63171598 ± 3% sched_debug.cpu.nr_switches.max
12792570 ± 3% +43.5% 18362678 ± 4% sched_debug.cpu.nr_switches.min
2.03 ± 80% +430.9% 10.76 ± 38% sched_debug.cpu.nr_uninterruptible.avg
6.99 ± 4% +48.7% 10.39 ± 5% perf-stat.i.MPKI
7.49 ± 5% -1.2 6.29 ± 5% perf-stat.i.cache-miss-rate%
27202880 ± 2% +46.9% 39966416 ± 2% perf-stat.i.cache-misses
4.662e+08 ± 5% +71.5% 7.998e+08 ± 5% perf-stat.i.cache-references
5989366 ± 4% +12.9% 6760958 ± 5% perf-stat.i.context-switches
197319 ± 12% -19.2% 159504 ± 3% perf-stat.i.cpu-migrations
7415 ± 2% -23.3% 5686 ± 3% perf-stat.i.cycles-between-cache-misses
0.61 ± 35% -0.3 0.35 ± 4% perf-stat.i.dTLB-load-miss-rate%
1.506e+08 ± 38% -44.9% 82967539 ± 5% perf-stat.i.dTLB-load-misses
0.22 ± 4% -0.0 0.20 ± 4% perf-stat.i.dTLB-store-miss-rate%
35974976 ± 5% -13.4% 31146077 ± 6% perf-stat.i.dTLB-store-misses
1.627e+10 -3.7% 1.566e+10 perf-stat.i.dTLB-stores
46665620 ± 3% -14.6% 39838962 ± 5% perf-stat.i.iTLB-load-misses
2191 ± 5% +21.4% 2660 ± 7% perf-stat.i.instructions-per-iTLB-miss
0.47 -2.5% 0.46 perf-stat.i.ipc
6660 ± 8% -19.1% 5391 ± 8% perf-stat.i.minor-faults
62.93 ± 2% +11.1 74.06 perf-stat.i.node-load-miss-rate%
12537290 ± 4% +73.5% 21750457 ± 4% perf-stat.i.node-load-misses
5459308 ± 3% -7.4% 5057404 ± 3% perf-stat.i.node-loads
29.39 ± 2% -4.1 25.29 ± 2% perf-stat.i.node-store-miss-rate%
2863317 +13.6% 3252632 ± 2% perf-stat.i.node-store-misses
6381925 ± 2% +53.5% 9796918 ± 4% perf-stat.i.node-stores
6638 ± 8% -19.4% 5348 ± 8% perf-stat.i.page-faults
4.89 ± 4% +78.4% 8.73 perf-stat.overall.MPKI
1.29 +0.0 1.31 perf-stat.overall.branch-miss-rate%
6.13 ± 4% -1.3 4.85 ± 3% perf-stat.overall.cache-miss-rate%
2.15 +2.0% 2.19 perf-stat.overall.cpi
7187 -27.9% 5182 ± 3% perf-stat.overall.cycles-between-cache-misses
0.63 ± 39% -0.3 0.36 ± 5% perf-stat.overall.dTLB-load-miss-rate%
0.23 ± 5% -0.0 0.21 ± 3% perf-stat.overall.dTLB-store-miss-rate%
75.65 -1.1 74.59 perf-stat.overall.iTLB-load-miss-rate%
1604 ± 3% +15.9% 1859 perf-stat.overall.instructions-per-iTLB-miss
0.47 -2.0% 0.46 perf-stat.overall.ipc
63.14 ± 2% +13.6 76.70 perf-stat.overall.node-load-miss-rate%
30.74 ± 2% -5.1 25.62 ± 2% perf-stat.overall.node-store-miss-rate%
47754 ± 2% +11.7% 53321 perf-stat.overall.path-length
24893507 +40.6% 34993733 ± 2% perf-stat.ps.cache-misses
4.072e+08 ± 5% +77.1% 7.212e+08 perf-stat.ps.cache-references
4968881 ± 7% +19.0% 5911450 perf-stat.ps.context-switches
1.789e+11 +1.3% 1.812e+11 perf-stat.ps.cpu-cycles
1.59e+08 ± 39% -44.1% 88801351 ± 6% perf-stat.ps.dTLB-load-misses
2.502e+10 -2.0% 2.453e+10 perf-stat.ps.dTLB-loads
39138578 ± 5% -12.6% 34190129 ± 3% perf-stat.ps.dTLB-store-misses
1.675e+10 -3.7% 1.613e+10 perf-stat.ps.dTLB-stores
51901139 ± 2% -14.3% 44461197 perf-stat.ps.iTLB-load-misses
16711264 ± 2% -9.3% 15153636 ± 2% perf-stat.ps.iTLB-loads
3573 -9.1% 3247 perf-stat.ps.minor-faults
10427704 ± 3% +73.2% 18055799 ± 3% perf-stat.ps.node-load-misses
6085553 ± 4% -10.0% 5479559 ± 2% perf-stat.ps.node-loads
2566511 +13.4% 2911367 ± 2% perf-stat.ps.node-store-misses
5784179 ± 2% +46.2% 8454210 ± 3% perf-stat.ps.node-stores
3573 -9.1% 3247 perf-stat.ps.page-faults
7.427e+13 ± 2% +11.7% 8.293e+13 perf-stat.total.instructions
127394 ± 5% +19.1% 151665 softirqs.CPU0.RCU
321761 ± 3% +12.8% 363107 ± 3% softirqs.CPU0.TIMER
124526 ± 5% +20.2% 149697 ± 3% softirqs.CPU1.RCU
123768 ± 4% +24.7% 154332 ± 2% softirqs.CPU10.RCU
130315 ± 3% +22.1% 159111 softirqs.CPU11.RCU
126684 ± 4% +26.1% 159755 ± 2% softirqs.CPU12.RCU
124649 ± 9% +27.8% 159298 ± 3% softirqs.CPU13.RCU
124528 ± 4% +23.2% 153372 ± 2% softirqs.CPU14.RCU
326150 ± 2% +9.8% 357991 ± 4% softirqs.CPU14.TIMER
121923 ± 6% +17.4% 143119 softirqs.CPU15.RCU
122114 ± 5% +16.7% 142560 softirqs.CPU16.RCU
121425 ± 6% +14.4% 138861 ± 3% softirqs.CPU17.RCU
330948 ± 2% +22.2% 404570 ± 16% softirqs.CPU17.TIMER
312317 ± 4% +18.2% 369091 ± 2% softirqs.CPU18.TIMER
9626 ± 2% +15.0% 11067 ± 3% softirqs.CPU19.SCHED
322308 ± 3% +21.1% 390454 ± 2% softirqs.CPU19.TIMER
124281 ± 5% +22.8% 152670 ± 2% softirqs.CPU2.RCU
9452 ± 2% +15.6% 10927 ± 3% softirqs.CPU20.SCHED
318575 ± 6% +22.3% 389741 ± 3% softirqs.CPU20.TIMER
9327 ± 3% +16.7% 10886 ± 3% softirqs.CPU21.SCHED
306628 ± 4% +23.3% 377962 ± 3% softirqs.CPU21.TIMER
9424 ± 2% +17.6% 11084 ± 3% softirqs.CPU22.SCHED
302796 ± 4% +24.2% 376126 ± 2% softirqs.CPU22.TIMER
9798 ± 2% +11.4% 10918 ± 4% softirqs.CPU23.SCHED
312009 ± 4% +23.6% 385626 ± 2% softirqs.CPU23.TIMER
9559 ± 3% +13.2% 10822 softirqs.CPU24.SCHED
311772 ± 4% +23.0% 383504 ± 2% softirqs.CPU24.TIMER
9514 ± 4% +13.2% 10766 ± 3% softirqs.CPU25.SCHED
307640 ± 4% +22.4% 376479 ± 2% softirqs.CPU25.TIMER
9573 ± 3% +13.8% 10897 ± 3% softirqs.CPU26.SCHED
314607 ± 9% +21.1% 380918 ± 2% softirqs.CPU26.TIMER
9385 +13.7% 10668 ± 2% softirqs.CPU27.SCHED
301949 ± 4% +25.2% 378140 ± 3% softirqs.CPU27.TIMER
9483 ± 2% +14.7% 10874 ± 2% softirqs.CPU28.SCHED
299462 ± 3% +24.9% 373896 ± 2% softirqs.CPU28.TIMER
9505 ± 3% +13.3% 10769 ± 3% softirqs.CPU29.SCHED
307087 ± 4% +23.8% 380066 ± 3% softirqs.CPU29.TIMER
124992 ± 4% +19.4% 149253 ± 3% softirqs.CPU3.RCU
9630 ± 4% +13.6% 10942 ± 3% softirqs.CPU30.SCHED
306382 ± 4% +25.4% 384355 ± 2% softirqs.CPU30.TIMER
9496 ± 6% +12.6% 10689 ± 2% softirqs.CPU31.SCHED
308689 ± 4% +24.5% 384419 ± 2% softirqs.CPU31.TIMER
9714 +12.6% 10937 ± 4% softirqs.CPU32.SCHED
304137 ± 3% +35.9% 413240 ± 13% softirqs.CPU32.TIMER
9598 ± 2% +13.4% 10882 ± 5% softirqs.CPU33.SCHED
9622 +13.8% 10949 ± 3% softirqs.CPU34.SCHED
305882 ± 4% +27.3% 389315 ± 4% softirqs.CPU34.TIMER
9409 ± 4% +15.3% 10853 ± 4% softirqs.CPU35.SCHED
306215 ± 4% +23.8% 379222 ± 2% softirqs.CPU35.TIMER
321617 ± 4% +12.9% 362966 ± 3% softirqs.CPU36.TIMER
134843 ± 6% +19.4% 161053 ± 2% softirqs.CPU37.RCU
130954 ± 6% +20.4% 157716 softirqs.CPU38.RCU
129884 ± 5% +17.8% 152978 softirqs.CPU39.RCU
124744 ± 4% +20.1% 149790 ± 3% softirqs.CPU4.RCU
130593 ± 5% +17.2% 153092 ± 2% softirqs.CPU40.RCU
134185 ± 5% +19.9% 160871 softirqs.CPU41.RCU
133336 ± 5% +18.1% 157409 softirqs.CPU42.RCU
134486 ± 4% +18.7% 159613 softirqs.CPU43.RCU
129857 ± 6% +18.5% 153887 softirqs.CPU44.RCU
131816 ± 5% +20.0% 158144 softirqs.CPU45.RCU
132691 ± 5% +22.4% 162412 softirqs.CPU46.RCU
138736 ± 3% +18.7% 164668 softirqs.CPU47.RCU
132971 ± 6% +24.2% 165142 softirqs.CPU48.RCU
136142 ± 3% +20.8% 164493 softirqs.CPU49.RCU
125011 ± 4% +23.6% 154462 ± 2% softirqs.CPU5.RCU
133660 ± 4% +19.5% 159753 ± 2% softirqs.CPU50.RCU
140405 ± 4% +19.5% 167772 softirqs.CPU51.RCU
140523 ± 4% +18.5% 166576 softirqs.CPU52.RCU
138849 ± 5% +14.6% 159157 ± 4% softirqs.CPU53.RCU
332171 ± 2% +21.4% 403260 ± 15% softirqs.CPU53.TIMER
133037 ± 4% +14.0% 151688 ± 2% softirqs.CPU54.RCU
9584 +14.3% 10959 ± 2% softirqs.CPU54.SCHED
310603 ± 4% +18.0% 366472 ± 3% softirqs.CPU54.TIMER
9565 ± 2% +13.8% 10889 ± 5% softirqs.CPU55.SCHED
321231 ± 3% +21.3% 389753 ± 2% softirqs.CPU55.TIMER
136633 ± 4% +12.6% 153854 softirqs.CPU56.RCU
9430 ± 2% +15.6% 10898 ± 5% softirqs.CPU56.SCHED
319477 ± 6% +22.1% 390107 ± 2% softirqs.CPU56.TIMER
128672 ± 4% +14.1% 146791 ± 3% softirqs.CPU57.RCU
9493 ± 2% +13.1% 10740 ± 3% softirqs.CPU57.SCHED
305670 ± 4% +23.5% 377597 ± 2% softirqs.CPU57.TIMER
127129 ± 4% +15.6% 146993 ± 2% softirqs.CPU58.RCU
9529 ± 3% +13.6% 10824 ± 4% softirqs.CPU58.SCHED
302317 ± 4% +23.8% 374371 ± 3% softirqs.CPU58.TIMER
9427 ± 2% +14.0% 10743 ± 2% softirqs.CPU59.SCHED
310985 ± 4% +23.5% 384119 ± 3% softirqs.CPU59.TIMER
124937 ± 4% +21.6% 151962 ± 3% softirqs.CPU6.RCU
9435 ± 2% +14.1% 10762 ± 3% softirqs.CPU60.SCHED
311646 ± 4% +22.8% 382731 ± 3% softirqs.CPU60.TIMER
9585 +13.5% 10877 ± 2% softirqs.CPU61.SCHED
306857 ± 4% +22.8% 376944 ± 2% softirqs.CPU61.TIMER
9483 ± 2% +12.5% 10671 ± 5% softirqs.CPU62.SCHED
9620 ± 4% +9.7% 10556 ± 4% softirqs.CPU63.SCHED
301085 ± 3% +24.7% 375397 ± 3% softirqs.CPU63.TIMER
300443 ± 4% +23.7% 371620 ± 2% softirqs.CPU64.TIMER
9667 +11.2% 10751 ± 4% softirqs.CPU65.SCHED
307194 ± 4% +23.4% 379020 ± 3% softirqs.CPU65.TIMER
9493 ± 3% +14.9% 10909 ± 3% softirqs.CPU66.SCHED
306226 ± 4% +25.1% 383140 ± 3% softirqs.CPU66.TIMER
9674 ± 3% +13.8% 11010 ± 3% softirqs.CPU67.SCHED
307870 ± 4% +25.0% 384793 ± 2% softirqs.CPU67.TIMER
9451 ± 2% +14.2% 10792 ± 4% softirqs.CPU68.SCHED
301873 ± 3% +26.1% 380748 softirqs.CPU68.TIMER
9542 +13.8% 10863 ± 3% softirqs.CPU69.SCHED
316206 ± 8% +20.5% 380893 ± 2% softirqs.CPU69.TIMER
128158 ± 4% +20.4% 154294 ± 2% softirqs.CPU7.RCU
9456 ± 3% +13.5% 10734 ± 3% softirqs.CPU70.SCHED
305361 ± 4% +36.4% 416546 ± 15% softirqs.CPU70.TIMER
9520 ± 3% +15.4% 10985 ± 4% softirqs.CPU71.SCHED
305605 ± 4% +24.6% 380653 ± 2% softirqs.CPU71.TIMER
122831 ± 4% +21.6% 149424 ± 2% softirqs.CPU8.RCU
123870 ± 4% +23.0% 152396 ± 2% softirqs.CPU9.RCU
9357957 ± 5% +15.4% 10794953 softirqs.RCU
23209193 ± 3% +15.5% 26806870 ± 3% softirqs.TIMER
4.79 ± 4% -3.1 1.71 ± 16% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.85 ± 4% -2.4 1.41 ± 16% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.39 ± 4% -2.0 1.35 ± 16% perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read
2.93 ± 5% -1.9 0.99 ± 18% perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write
3.25 ± 4% -1.9 1.31 ± 10% perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_write.new_sync_write.vfs_write.ksys_write
2.14 ± 4% -1.2 0.91 ± 14% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64
2.06 ± 4% -1.2 0.85 ± 13% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
1.52 ± 3% -1.0 0.49 ± 59% perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.pipe_read.new_sync_read.vfs_read
1.46 ± 3% -1.0 0.46 ± 59% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.pipe_read.new_sync_read
1.53 ± 8% -0.8 0.68 ± 12% perf-profile.calltrace.cycles-pp.touch_atime.pipe_read.new_sync_read.vfs_read.ksys_read
1.65 ± 4% -0.8 0.81 ± 10% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.45 ± 5% -0.7 0.74 ± 10% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.35 ± 5% -0.7 0.68 ± 10% perf-profile.calltrace.cycles-pp.__fget.__fget_light.__fdget_pos.ksys_write.do_syscall_64
1.28 ± 8% -0.7 0.62 ± 12% perf-profile.calltrace.cycles-pp.file_update_time.pipe_write.new_sync_write.vfs_write.ksys_write
1.29 ± 3% -0.6 0.67 ± 9% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.new_sync_write.vfs_write.ksys_write
0.64 ± 11% +0.3 0.96 ± 7% perf-profile.calltrace.cycles-pp.__lock_text_start.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
0.00 +0.5 0.55 ± 3% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop
0.00 +0.6 0.55 ± 5% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait
0.00 +0.6 0.57 ± 6% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate
0.00 +0.6 0.61 ± 7% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule
0.00 +0.7 0.69 ± 8% perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.pipe_wait.pipe_read
0.00 +0.7 0.70 ± 6% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait
0.00 +0.7 0.71 ± 11% perf-profile.calltrace.cycles-pp.__enqueue_entity.put_prev_entity.pick_next_task_fair.__schedule.schedule
0.00 +0.7 0.72 ± 7% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
0.00 +0.7 0.72 ± 6% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.dequeue_task_fair.__schedule.schedule.pipe_wait
0.00 +0.8 0.76 ± 10% perf-profile.calltrace.cycles-pp.__enqueue_entity.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
0.00 +0.8 0.77 ± 2% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
0.00 +0.8 0.77 ± 5% perf-profile.calltrace.cycles-pp.__switch_to
0.00 +0.8 0.77 ± 6% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read
0.00 +0.8 0.79 ± 5% perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
0.00 +0.8 0.81 ± 4% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_wait.pipe_read.new_sync_read.vfs_read
0.00 +0.8 0.82 ± 6% perf-profile.calltrace.cycles-pp.update_rq_clock.__schedule.schedule.pipe_wait.pipe_read
0.00 +0.8 0.82 ± 6% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
0.00 +0.9 0.90 ± 3% perf-profile.calltrace.cycles-pp.find_next_bit.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up
0.00 +0.9 0.91 ± 4% perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
0.00 +0.9 0.91 ± 6% perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
0.00 +0.9 0.92 ± 2% perf-profile.calltrace.cycles-pp.native_write_msr
0.00 +1.1 1.05 ± 5% perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
0.00 +1.1 1.08 ± 8% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop
0.00 +1.1 1.13 ± 5% perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
0.00 +1.2 1.17 ± 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common
0.00 +1.2 1.18 ± 2% perf-profile.calltrace.cycles-pp.__switch_to_asm
0.00 +1.2 1.20 ± 9% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.pipe_wait.pipe_read
0.00 +1.3 1.30 ± 5% perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
0.00 +1.3 1.34 ± 6% perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait
0.00 +1.6 1.62 ± 4% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +1.9 1.88 ± 5% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_wait.pipe_read
0.00 +1.9 1.93 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
0.13 ±173% +2.0 2.11 ± 5% perf-profile.calltrace.cycles-pp.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
0.00 +2.1 2.06 ± 4% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_wait
0.00 +2.4 2.38 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64
0.00 +2.7 2.65 ± 4% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
0.35 ±103% +4.3 4.67 ± 4% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.22 ±173% +4.6 4.81 ± 4% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.23 ±173% +4.7 4.97 ± 4% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.72 ± 23% +5.0 5.72 ± 5% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_wait.pipe_read
0.69 ± 17% +5.6 6.25 ± 4% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
0.74 ± 17% +5.8 6.50 ± 4% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
0.74 ± 18% +5.8 6.55 ± 5% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
15.99 ± 54% +7.5 23.51 ± 6% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
13.13 +7.7 20.86 ± 3% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
11.49 +8.7 20.19 ± 3% perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
1.31 ± 27% +8.9 10.21 ± 7% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
1.81 ± 22% +11.2 12.98 ± 5% perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait.pipe_read.new_sync_read
1.89 ± 22% +11.4 13.28 ± 5% perf-profile.calltrace.cycles-pp.schedule.pipe_wait.pipe_read.new_sync_read.vfs_read
2.18 ± 21% +13.1 15.28 ± 4% perf-profile.calltrace.cycles-pp.pipe_wait.pipe_read.new_sync_read.vfs_read.ksys_read
2.01 ± 25% +19.1 21.10 ± 7% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
2.38 ± 23% +20.0 22.40 ± 7% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
19.01 ± 3% +23.4 42.38 ± 4% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
21.13 ± 53% +23.8 44.92 ± 5% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
17.85 ± 4% +23.9 41.76 ± 4% perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
19.16 ± 53% +24.6 43.74 ± 5% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.79 ± 22% +30.7 34.45 ± 6% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
3.84 ± 23% +30.9 34.69 ± 6% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
4.13 ± 21% +31.0 35.10 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
5.77 ± 14% +31.0 36.76 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
9.20 ± 6% -6.0 3.16 ± 12% perf-profile.children.cycles-pp.syscall_return_via_sysret
8.88 ± 3% -5.8 3.03 ± 12% perf-profile.children.cycles-pp.entry_SYSCALL_64
8.72 ± 4% -5.3 3.39 ± 11% perf-profile.children.cycles-pp.security_file_permission
4.27 ± 3% -2.4 1.89 ± 9% perf-profile.children.cycles-pp.selinux_file_permission
3.15 ± 5% -2.1 1.07 ± 12% perf-profile.children.cycles-pp.file_has_perm
3.48 ± 3% -2.0 1.46 ± 10% perf-profile.children.cycles-pp.copy_page_to_iter
3.80 ± 3% -2.0 1.79 ± 6% perf-profile.children.cycles-pp.mutex_unlock
3.02 ± 5% -1.9 1.11 ± 11% perf-profile.children.cycles-pp.copy_page_from_iter
3.31 ± 4% -1.6 1.75 ± 6% perf-profile.children.cycles-pp.__fdget_pos
2.55 ± 3% -1.5 1.02 ± 10% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
3.17 ± 4% -1.5 1.70 ± 6% perf-profile.children.cycles-pp.__fget_light
2.10 ± 7% -1.4 0.71 ± 12% perf-profile.children.cycles-pp.fsnotify
2.52 ± 6% -1.3 1.23 ± 8% perf-profile.children.cycles-pp.__fget
1.77 ± 6% -1.1 0.66 ± 9% perf-profile.children.cycles-pp.avc_has_perm
1.89 ± 5% -1.1 0.78 ± 10% perf-profile.children.cycles-pp.___might_sleep
1.44 ± 4% -1.0 0.48 ± 12% perf-profile.children.cycles-pp.__inode_security_revalidate
1.33 ± 10% -0.9 0.40 ± 16% perf-profile.children.cycles-pp.current_time
1.56 ± 3% -0.9 0.66 ± 9% perf-profile.children.cycles-pp.copyout
1.56 ± 8% -0.8 0.73 ± 7% perf-profile.children.cycles-pp.touch_atime
1.33 ± 4% -0.8 0.53 ± 10% perf-profile.children.cycles-pp.__might_sleep
1.15 ± 9% -0.8 0.36 ± 12% perf-profile.children.cycles-pp.atime_needs_update
1.17 ± 4% -0.7 0.43 ± 13% perf-profile.children.cycles-pp.copyin
1.23 ± 5% -0.7 0.54 ± 9% perf-profile.children.cycles-pp.__fsnotify_parent
1.32 ± 7% -0.6 0.68 ± 6% perf-profile.children.cycles-pp.file_update_time
1.02 ± 3% -0.6 0.38 ± 12% perf-profile.children.cycles-pp.__might_fault
2.54 ± 2% -0.5 2.03 perf-profile.children.cycles-pp.mutex_lock
0.78 ± 4% -0.4 0.36 ± 10% perf-profile.children.cycles-pp.fput_many
0.53 ± 4% -0.4 0.15 ± 21% perf-profile.children.cycles-pp.inode_has_perm
0.44 ± 12% -0.3 0.11 ± 22% perf-profile.children.cycles-pp.iov_iter_init
0.76 ± 4% -0.3 0.44 ± 8% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
0.53 ± 3% -0.3 0.21 ± 6% perf-profile.children.cycles-pp.rcu_all_qs
0.41 ± 9% -0.3 0.12 ± 15% perf-profile.children.cycles-pp.timespec64_trunc
0.97 -0.3 0.71 ± 3% perf-profile.children.cycles-pp._cond_resched
0.47 ± 7% -0.3 0.21 ± 11% perf-profile.children.cycles-pp.__x64_sys_read
0.45 ± 7% -0.3 0.20 ± 10% perf-profile.children.cycles-pp.__x64_sys_write
0.37 ± 4% -0.2 0.17 ± 16% perf-profile.children.cycles-pp.generic_pipe_buf_confirm
0.44 ± 12% -0.2 0.24 ± 3% perf-profile.children.cycles-pp.__sb_start_write
0.31 ± 6% -0.2 0.12 ± 13% perf-profile.children.cycles-pp.__sb_end_write
0.36 ± 7% -0.2 0.18 ± 6% perf-profile.children.cycles-pp.rw_verify_area
0.26 ± 14% -0.2 0.09 ± 14% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
0.29 ± 8% -0.2 0.14 ± 16% perf-profile.children.cycles-pp.fpregs_assert_state_consistent
0.22 ± 39% -0.2 0.06 ± 11% perf-profile.children.cycles-pp.__vfs_read
0.21 ± 19% -0.1 0.11 ± 9% perf-profile.children.cycles-pp.__mutex_unlock_slowpath
0.12 ± 9% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__vfs_write
0.12 ± 17% -0.0 0.08 ± 11% perf-profile.children.cycles-pp.wake_up_q
0.08 ± 10% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.kill_fasync
0.11 ± 7% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
0.09 ± 5% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.prepare_exit_to_usermode
0.01 ±173% +0.1 0.06 ± 6% perf-profile.children.cycles-pp.__mnt_want_write
0.07 ± 25% +0.1 0.12 ± 27% perf-profile.children.cycles-pp.__softirqentry_text_start
0.08 ± 30% +0.1 0.14 ± 26% perf-profile.children.cycles-pp.irq_exit
0.06 ± 6% +0.1 0.12 ± 10% perf-profile.children.cycles-pp.__mark_inode_dirty
0.00 +0.1 0.06 perf-profile.children.cycles-pp.preempt_schedule_common
0.00 +0.1 0.07 ± 6% perf-profile.children.cycles-pp.is_cpu_allowed
0.03 ±100% +0.1 0.10 ± 8% perf-profile.children.cycles-pp.native_apic_msr_eoi_write
0.00 +0.1 0.07 ± 5% perf-profile.children.cycles-pp.__x2apic_send_IPI_dest
0.01 ±173% +0.1 0.09 ± 32% perf-profile.children.cycles-pp.migrate_task_rq_fair
0.00 +0.1 0.07 ± 14% perf-profile.children.cycles-pp.update_cfs_rq_h_load
0.05 ± 61% +0.1 0.13 ± 5% perf-profile.children.cycles-pp.native_irq_return_iret
0.00 +0.1 0.08 ± 8% perf-profile.children.cycles-pp.rcu_note_context_switch
0.00 +0.1 0.08 ± 13% perf-profile.children.cycles-pp.smp_reschedule_interrupt
0.00 +0.1 0.09 ± 4% perf-profile.children.cycles-pp.native_load_tls
0.03 ±102% +0.1 0.12 ± 30% perf-profile.children.cycles-pp.set_task_cpu
0.00 +0.1 0.09 ± 11% perf-profile.children.cycles-pp.check_cfs_rq_runtime
0.00 +0.1 0.10 ± 5% perf-profile.children.cycles-pp.rb_next
0.10 ± 10% +0.1 0.20 ± 9% perf-profile.children.cycles-pp.generic_update_time
0.00 +0.1 0.10 ± 29% perf-profile.children.cycles-pp.__x64_sys_exit
0.00 +0.1 0.10 ± 28% perf-profile.children.cycles-pp.do_exit
0.00 +0.1 0.11 ± 8% perf-profile.children.cycles-pp.rb_insert_color
0.00 +0.1 0.11 ± 7% perf-profile.children.cycles-pp.__list_add_valid
0.00 +0.1 0.12 ± 7% perf-profile.children.cycles-pp.resched_curr
0.00 +0.1 0.12 ± 6% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.13 ± 10% +0.1 0.26 ± 4% perf-profile.children.cycles-pp.__list_del_entry_valid
0.00 +0.1 0.15 ± 4% perf-profile.children.cycles-pp.set_next_buddy
0.00 +0.2 0.16 ± 5% perf-profile.children.cycles-pp.deactivate_task
0.05 ± 9% +0.2 0.22 ± 7% perf-profile.children.cycles-pp.rb_erase
0.06 ± 13% +0.2 0.23 ± 9% perf-profile.children.cycles-pp.finish_wait
0.17 ± 12% +0.2 0.35 ± 2% perf-profile.children.cycles-pp.anon_pipe_buf_release
0.01 ±173% +0.2 0.20 ± 8% perf-profile.children.cycles-pp.wakeup_preempt_entity
0.00 +0.2 0.21 ± 4% perf-profile.children.cycles-pp.cpumask_next
0.07 ± 22% +0.3 0.40 ± 2% perf-profile.children.cycles-pp.prepare_to_wait
0.03 ±100% +0.4 0.39 ± 5% perf-profile.children.cycles-pp.update_min_vruntime
0.03 ±100% +0.4 0.39 ± 12% perf-profile.children.cycles-pp.cpus_share_cache
0.06 ± 14% +0.4 0.43 perf-profile.children.cycles-pp.clear_buddies
0.11 ± 15% +0.4 0.50 ± 3% perf-profile.children.cycles-pp.cpuacct_charge
0.10 ± 26% +0.4 0.49 ± 4% perf-profile.children.cycles-pp.native_sched_clock
0.32 ± 31% +0.4 0.75 ± 14% perf-profile.children.cycles-pp.sched_ttwu_pending
0.11 ± 26% +0.4 0.56 ± 3% perf-profile.children.cycles-pp.sched_clock
0.34 ± 32% +0.5 0.80 ± 15% perf-profile.children.cycles-pp.scheduler_ipi
0.08 ± 12% +0.5 0.56 ± 7% perf-profile.children.cycles-pp.account_entity_enqueue
0.13 ± 26% +0.5 0.62 ± 3% perf-profile.children.cycles-pp.sched_clock_cpu
0.14 ± 26% +0.6 0.77 ± 4% perf-profile.children.cycles-pp.__calc_delta
0.08 ± 16% +0.7 0.76 ± 3% perf-profile.children.cycles-pp.account_entity_dequeue
0.84 ± 5% +0.7 1.54 ± 5% perf-profile.children.cycles-pp.__lock_text_start
1.61 ± 16% +0.7 2.31 ± 9% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.42 ± 31% +0.7 1.12 ± 13% perf-profile.children.cycles-pp.reschedule_interrupt
0.24 ± 28% +0.9 1.11 perf-profile.children.cycles-pp.native_write_msr
0.14 ± 22% +0.9 1.01 ± 5% perf-profile.children.cycles-pp.check_preempt_wakeup
0.24 ± 32% +0.9 1.12 ± 7% perf-profile.children.cycles-pp.finish_task_switch
0.22 ± 22% +0.9 1.10 ± 3% perf-profile.children.cycles-pp.find_next_bit
0.22 ± 23% +0.9 1.12 ± 3% perf-profile.children.cycles-pp.set_next_entity
0.21 ± 32% +0.9 1.14 ± 3% perf-profile.children.cycles-pp.update_cfs_group
0.16 ± 23% +1.0 1.15 ± 4% perf-profile.children.cycles-pp.check_preempt_curr
0.21 ± 30% +1.0 1.21 ± 2% perf-profile.children.cycles-pp.__switch_to
0.23 ± 21% +1.0 1.24 ± 2% perf-profile.children.cycles-pp.__switch_to_asm
0.16 ± 38% +1.0 1.20 ± 4% perf-profile.children.cycles-pp.___perf_sw_event
0.15 ± 26% +1.0 1.20 ± 3% perf-profile.children.cycles-pp.pick_next_entity
0.15 ± 20% +1.1 1.21 ± 7% perf-profile.children.cycles-pp.put_prev_entity
0.17 ± 22% +1.1 1.23 ± 4% perf-profile.children.cycles-pp.ttwu_do_wakeup
0.19 ± 10% +1.3 1.52 ± 10% perf-profile.children.cycles-pp.__enqueue_entity
0.24 ± 23% +1.3 1.59 ± 6% perf-profile.children.cycles-pp.__update_load_avg_se
0.28 ± 23% +1.4 1.64 ± 3% perf-profile.children.cycles-pp.switch_fpu_return
0.31 ± 29% +1.6 1.88 ± 5% perf-profile.children.cycles-pp.update_rq_clock
0.44 ± 21% +1.8 2.24 ± 4% perf-profile.children.cycles-pp.cpumask_next_wrap
0.33 ± 20% +1.8 2.15 ± 4% perf-profile.children.cycles-pp.dequeue_entity
23.28 ± 2% +2.0 25.27 ± 2% perf-profile.children.cycles-pp.ksys_read
0.46 ± 25% +2.1 2.57 ± 5% perf-profile.children.cycles-pp.update_load_avg
0.20 ± 21% +2.1 2.34 ± 4% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.91 ± 29% +2.2 3.12 ± 11% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.38 ± 23% +2.4 2.80 ± 4% perf-profile.children.cycles-pp.reweight_entity
0.52 ± 18% +2.4 2.96 ± 3% perf-profile.children.cycles-pp.enqueue_entity
0.49 ± 23% +2.6 3.10 ± 4% perf-profile.children.cycles-pp.update_curr
20.92 ± 2% +3.2 24.12 ± 2% perf-profile.children.cycles-pp.vfs_read
0.21 ± 23% +3.4 3.65 ± 7% perf-profile.children.cycles-pp._raw_spin_lock
0.76 ± 24% +3.9 4.67 ± 3% perf-profile.children.cycles-pp.pick_next_task_fair
0.66 ± 34% +4.4 5.06 ± 2% perf-profile.children.cycles-pp.exit_to_usermode_loop
0.81 ± 22% +5.0 5.83 ± 4% perf-profile.children.cycles-pp.dequeue_task_fair
1.01 ± 21% +5.8 6.86 ± 4% perf-profile.children.cycles-pp.enqueue_task_fair
1.06 ± 21% +6.0 7.10 ± 4% perf-profile.children.cycles-pp.activate_task
1.07 ± 21% +6.1 7.15 ± 4% perf-profile.children.cycles-pp.ttwu_do_activate
13.19 +7.8 20.94 ± 3% perf-profile.children.cycles-pp.new_sync_read
11.61 +8.7 20.29 ± 3% perf-profile.children.cycles-pp.pipe_read
1.45 ± 25% +8.9 10.35 ± 7% perf-profile.children.cycles-pp.available_idle_cpu
78.85 ± 2% +9.8 88.66 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
77.90 ± 2% +10.1 88.00 perf-profile.children.cycles-pp.do_syscall_64
2.42 ± 21% +12.9 15.36 ± 4% perf-profile.children.cycles-pp.pipe_wait
2.71 ± 26% +15.4 18.12 ± 4% perf-profile.children.cycles-pp.__schedule
2.76 ± 25% +15.6 18.37 ± 4% perf-profile.children.cycles-pp.schedule
27.59 ± 2% +18.6 46.17 ± 3% perf-profile.children.cycles-pp.ksys_write
2.26 ± 23% +19.0 21.30 ± 7% perf-profile.children.cycles-pp.select_idle_sibling
24.98 ± 2% +19.9 44.88 ± 3% perf-profile.children.cycles-pp.vfs_write
2.48 ± 23% +20.0 22.46 ± 7% perf-profile.children.cycles-pp.select_task_rq_fair
19.06 ± 3% +23.3 42.41 ± 4% perf-profile.children.cycles-pp.new_sync_write
17.98 ± 4% +23.9 41.93 ± 4% perf-profile.children.cycles-pp.pipe_write
4.03 ± 21% +30.6 34.60 ± 6% perf-profile.children.cycles-pp.try_to_wake_up
3.97 ± 22% +30.8 34.74 ± 6% perf-profile.children.cycles-pp.autoremove_wake_function
4.30 ± 20% +30.9 35.19 ± 6% perf-profile.children.cycles-pp.__wake_up_common
6.06 ± 14% +31.2 37.21 ± 5% perf-profile.children.cycles-pp.__wake_up_common_lock
24.41 ± 6% -15.5 8.96 ± 10% perf-profile.self.cycles-pp.do_syscall_64
9.18 ± 6% -6.0 3.14 ± 12% perf-profile.self.cycles-pp.syscall_return_via_sysret
8.63 ± 5% -5.6 3.00 ± 12% perf-profile.self.cycles-pp.entry_SYSCALL_64
3.71 ± 3% -2.0 1.73 ± 6% perf-profile.self.cycles-pp.mutex_unlock
2.47 ± 3% -1.5 0.96 ± 10% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
2.76 ± 3% -1.4 1.35 ± 8% perf-profile.self.cycles-pp.selinux_file_permission
2.02 ± 7% -1.3 0.68 ± 12% perf-profile.self.cycles-pp.fsnotify
2.46 ± 6% -1.3 1.20 ± 8% perf-profile.self.cycles-pp.__fget
1.75 ± 6% -1.1 0.64 ± 10% perf-profile.self.cycles-pp.avc_has_perm
1.83 ± 5% -1.1 0.75 ± 10% perf-profile.self.cycles-pp.___might_sleep
1.75 -1.0 0.77 ± 7% perf-profile.self.cycles-pp.pipe_write
1.20 ± 4% -0.7 0.46 ± 10% perf-profile.self.cycles-pp.__might_sleep
1.29 ± 11% -0.7 0.58 ± 10% perf-profile.self.cycles-pp.new_sync_read
1.63 ± 4% -0.7 0.97 ± 5% perf-profile.self.cycles-pp.pipe_read
1.15 ± 5% -0.7 0.49 ± 8% perf-profile.self.cycles-pp.__fsnotify_parent
0.89 ± 8% -0.6 0.27 ± 15% perf-profile.self.cycles-pp.copy_page_from_iter
0.89 ± 4% -0.6 0.29 ± 14% perf-profile.self.cycles-pp.security_file_permission
0.83 ± 5% -0.6 0.24 ± 21% perf-profile.self.cycles-pp.file_has_perm
0.90 ± 2% -0.5 0.41 ± 6% perf-profile.self.cycles-pp.new_sync_write
0.67 ± 10% -0.5 0.20 ± 15% perf-profile.self.cycles-pp.current_time
0.83 ± 4% -0.4 0.41 ± 10% perf-profile.self.cycles-pp.copy_page_to_iter
0.72 ± 7% -0.4 0.31 ± 9% perf-profile.self.cycles-pp.vfs_write
0.75 ± 4% -0.4 0.35 ± 9% perf-profile.self.cycles-pp.fput_many
1.01 ± 4% -0.3 0.69 ± 5% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.45 ± 4% -0.3 0.14 ± 19% perf-profile.self.cycles-pp.inode_has_perm
0.54 ± 8% -0.3 0.23 ± 9% perf-profile.self.cycles-pp.file_update_time
0.41 ± 10% -0.3 0.11 ± 20% perf-profile.self.cycles-pp.iov_iter_init
0.49 ± 7% -0.3 0.19 ± 8% perf-profile.self.cycles-pp.atime_needs_update
0.44 ± 7% -0.3 0.15 ± 14% perf-profile.self.cycles-pp.ksys_read
0.44 ± 7% -0.3 0.17 ± 9% perf-profile.self.cycles-pp.__inode_security_revalidate
0.41 ± 2% -0.3 0.15 ± 8% perf-profile.self.cycles-pp.rcu_all_qs
0.43 ± 8% -0.3 0.17 ± 11% perf-profile.self.cycles-pp.ksys_write
0.37 ± 9% -0.3 0.11 ± 17% perf-profile.self.cycles-pp.timespec64_trunc
0.42 ± 14% -0.2 0.18 ± 8% perf-profile.self.cycles-pp.__sb_start_write
0.39 ± 7% -0.2 0.17 ± 11% perf-profile.self.cycles-pp.__x64_sys_write
0.40 ± 8% -0.2 0.17 ± 8% perf-profile.self.cycles-pp.__x64_sys_read
0.41 ± 7% -0.2 0.20 ± 11% perf-profile.self.cycles-pp.__wake_up_common_lock
0.60 ± 4% -0.2 0.39 ± 7% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
1.25 -0.2 1.04 ± 2% perf-profile.self.cycles-pp.mutex_lock
0.30 ± 6% -0.2 0.12 ± 15% perf-profile.self.cycles-pp.__sb_end_write
0.35 ± 5% -0.2 0.18 ± 3% perf-profile.self.cycles-pp.rw_verify_area
0.34 ± 7% -0.2 0.18 ± 8% perf-profile.self.cycles-pp.touch_atime
0.32 ± 4% -0.2 0.15 ± 14% perf-profile.self.cycles-pp.generic_pipe_buf_confirm
0.23 ± 16% -0.2 0.08 ± 11% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
0.74 ± 3% -0.2 0.59 ± 5% perf-profile.self.cycles-pp.vfs_read
0.60 ± 3% -0.2 0.45 ± 4% perf-profile.self.cycles-pp.__fget_light
0.27 ± 8% -0.1 0.12 ± 15% perf-profile.self.cycles-pp.fpregs_assert_state_consistent
0.18 ± 45% -0.1 0.04 ± 58% perf-profile.self.cycles-pp.__vfs_read
0.18 ± 5% -0.1 0.06 ± 17% perf-profile.self.cycles-pp.__fdget_pos
0.25 ± 5% -0.1 0.14 ± 11% perf-profile.self.cycles-pp.__might_fault
0.07 ± 7% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.kill_fasync
0.01 ±173% +0.1 0.06 ± 17% perf-profile.self.cycles-pp.sched_ttwu_pending
0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.scheduler_ipi
0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.rcu_note_context_switch
0.01 ±173% +0.1 0.07 ± 12% perf-profile.self.cycles-pp.generic_update_time
0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.cpumask_next
0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.ttwu_do_activate
0.00 +0.1 0.06 perf-profile.self.cycles-pp.__mnt_want_write
0.06 ± 14% +0.1 0.12 ± 10% perf-profile.self.cycles-pp.__mark_inode_dirty
0.00 +0.1 0.06 ± 13% perf-profile.self.cycles-pp.is_cpu_allowed
0.00 +0.1 0.07 ± 13% perf-profile.self.cycles-pp.sched_clock_cpu
0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.sched_clock
0.03 ±100% +0.1 0.10 ± 11% perf-profile.self.cycles-pp.native_apic_msr_eoi_write
0.00 +0.1 0.07 ± 14% perf-profile.self.cycles-pp.update_cfs_rq_h_load
0.00 +0.1 0.08 perf-profile.self.cycles-pp.native_load_tls
0.05 ± 60% +0.1 0.13 ± 5% perf-profile.self.cycles-pp.native_irq_return_iret
0.00 +0.1 0.09 ± 9% perf-profile.self.cycles-pp.ttwu_do_wakeup
0.00 +0.1 0.09 ± 11% perf-profile.self.cycles-pp.rb_next
0.00 +0.1 0.10 ± 8% perf-profile.self.cycles-pp.__list_add_valid
0.00 +0.1 0.10 ± 10% perf-profile.self.cycles-pp.rb_insert_color
0.00 +0.1 0.11 ± 9% perf-profile.self.cycles-pp.resched_curr
0.00 +0.1 0.11 ± 7% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.32 ± 7% +0.1 0.45 ± 5% perf-profile.self.cycles-pp.__wake_up_common
0.13 ± 10% +0.1 0.25 ± 5% perf-profile.self.cycles-pp.__list_del_entry_valid
0.00 +0.1 0.13 ± 5% perf-profile.self.cycles-pp.autoremove_wake_function
0.00 +0.1 0.13 ± 8% perf-profile.self.cycles-pp.check_preempt_curr
0.04 ± 58% +0.1 0.18 ± 10% perf-profile.self.cycles-pp.finish_wait
0.00 +0.1 0.14 ± 5% perf-profile.self.cycles-pp.set_next_buddy
0.00 +0.1 0.15 ± 3% perf-profile.self.cycles-pp.put_prev_entity
0.00 +0.2 0.15 ± 5% perf-profile.self.cycles-pp.deactivate_task
0.05 ± 9% +0.2 0.21 ± 5% perf-profile.self.cycles-pp.rb_erase
0.00 +0.2 0.15 ± 10% perf-profile.self.cycles-pp.exit_to_usermode_loop
0.00 +0.2 0.18 ± 6% perf-profile.self.cycles-pp.wakeup_preempt_entity
0.15 ± 7% +0.2 0.34 ± 2% perf-profile.self.cycles-pp.anon_pipe_buf_release
0.03 ±100% +0.2 0.21 ± 8% perf-profile.self.cycles-pp.pipe_wait
0.03 ±102% +0.2 0.24 ± 3% perf-profile.self.cycles-pp.activate_task
0.01 ±173% +0.2 0.23 ± 3% perf-profile.self.cycles-pp.prepare_to_wait
0.03 ±100% +0.2 0.25 ± 4% perf-profile.self.cycles-pp.set_next_entity
0.02 ±173% +0.2 0.26 ± 9% perf-profile.self.cycles-pp.dequeue_entity
0.04 ±100% +0.3 0.29 ± 5% perf-profile.self.cycles-pp.enqueue_entity
0.08 ± 31% +0.3 0.34 ± 2% perf-profile.self.cycles-pp.schedule
0.78 ± 3% +0.3 1.09 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.09 ± 24% +0.3 0.42 ± 7% perf-profile.self.cycles-pp.finish_task_switch
0.03 ±100% +0.3 0.37 ± 6% perf-profile.self.cycles-pp.update_min_vruntime
0.03 ±102% +0.4 0.38 perf-profile.self.cycles-pp.clear_buddies
0.03 ±100% +0.4 0.39 ± 12% perf-profile.self.cycles-pp.cpus_share_cache
0.07 ± 22% +0.4 0.44 ± 7% perf-profile.self.cycles-pp.check_preempt_wakeup
0.09 ± 27% +0.4 0.47 ± 4% perf-profile.self.cycles-pp.native_sched_clock
0.07 ± 26% +0.4 0.45 ± 4% perf-profile.self.cycles-pp.pick_next_entity
0.11 ± 15% +0.4 0.50 ± 3% perf-profile.self.cycles-pp.cpuacct_charge
0.07 ± 14% +0.5 0.53 ± 8% perf-profile.self.cycles-pp.account_entity_enqueue
0.10 ± 21% +0.5 0.56 ± 8% perf-profile.self.cycles-pp.try_to_wake_up
0.10 ± 18% +0.5 0.62 ± 6% perf-profile.self.cycles-pp.dequeue_task_fair
0.14 ± 26% +0.6 0.75 ± 4% perf-profile.self.cycles-pp.__calc_delta
0.06 ± 13% +0.6 0.68 ± 4% perf-profile.self.cycles-pp.account_entity_dequeue
0.15 ± 27% +0.6 0.79 ± 6% perf-profile.self.cycles-pp.enqueue_task_fair
0.17 ± 30% +0.7 0.89 ± 4% perf-profile.self.cycles-pp.update_load_avg
0.15 ± 20% +0.7 0.88 ± 4% perf-profile.self.cycles-pp.reweight_entity
0.15 ± 23% +0.8 0.96 ± 3% perf-profile.self.cycles-pp.pick_next_task_fair
0.24 ± 30% +0.9 1.11 perf-profile.self.cycles-pp.native_write_msr
0.20 ± 23% +0.9 1.07 ± 5% perf-profile.self.cycles-pp.select_task_rq_fair
0.20 ± 22% +0.9 1.07 ± 3% perf-profile.self.cycles-pp.find_next_bit
0.20 ± 31% +0.9 1.11 ± 3% perf-profile.self.cycles-pp.update_cfs_group
0.20 ± 28% +0.9 1.14 ± 2% perf-profile.self.cycles-pp.__switch_to
0.14 ± 37% +1.0 1.13 ± 5% perf-profile.self.cycles-pp.___perf_sw_event
0.21 ± 24% +1.0 1.22 ± 2% perf-profile.self.cycles-pp.__switch_to_asm
0.24 ± 23% +1.1 1.31 ± 5% perf-profile.self.cycles-pp.cpumask_next_wrap
0.23 ± 32% +1.3 1.49 ± 5% perf-profile.self.cycles-pp.update_rq_clock
0.19 ± 28% +1.3 1.45 ± 6% perf-profile.self.cycles-pp.update_curr
0.23 ± 23% +1.3 1.55 ± 6% perf-profile.self.cycles-pp.__update_load_avg_se
0.19 ± 10% +1.3 1.51 ± 10% perf-profile.self.cycles-pp.__enqueue_entity
0.28 ± 22% +1.3 1.63 ± 3% perf-profile.self.cycles-pp.switch_fpu_return
0.11 ± 19% +1.6 1.72 ± 5% perf-profile.self.cycles-pp._raw_spin_lock
0.42 ± 26% +1.7 2.12 ± 2% perf-profile.self.cycles-pp.__schedule
0.19 ± 20% +2.0 2.20 ± 4% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.91 ± 29% +2.2 3.11 ± 12% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.32 ± 19% +7.7 8.05 ± 7% perf-profile.self.cycles-pp.select_idle_sibling
1.44 ± 25% +8.8 10.23 ± 7% perf-profile.self.cycles-pp.available_idle_cpu
439.00 +11.7% 490.50 interrupts.100:IR-PCI-MSI.1572923-edge.eth0-TxRx-59
436.25 +13.1% 493.25 interrupts.101:IR-PCI-MSI.1572924-edge.eth0-TxRx-60
436.25 +12.4% 490.50 interrupts.102:IR-PCI-MSI.1572925-edge.eth0-TxRx-61
436.25 +12.4% 490.50 interrupts.103:IR-PCI-MSI.1572926-edge.eth0-TxRx-62
509.75 ± 3% +545.1% 3288 ±120% interrupts.42:IR-PCI-MSI.1572867-edge.eth0-TxRx-3
474.00 ± 3% +49.7% 709.75 ± 24% interrupts.46:IR-PCI-MSI.1572871-edge.eth0-TxRx-7
588.00 ± 35% +196.0% 1740 ± 50% interrupts.51:IR-PCI-MSI.1572876-edge.eth0-TxRx-12
436.75 ± 2% +12.8% 492.50 interrupts.55:IR-PCI-MSI.1572880-edge.eth0-TxRx-16
438.00 +39.5% 611.00 ± 33% interrupts.57:IR-PCI-MSI.1572882-edge.eth0-TxRx-18
436.25 +13.8% 496.25 interrupts.58:IR-PCI-MSI.1572883-edge.eth0-TxRx-19
436.25 +14.3% 498.50 interrupts.59:IR-PCI-MSI.1572884-edge.eth0-TxRx-20
437.00 +12.2% 490.50 interrupts.60:IR-PCI-MSI.1572885-edge.eth0-TxRx-21
436.25 +12.4% 490.50 interrupts.61:IR-PCI-MSI.1572886-edge.eth0-TxRx-22
436.25 +12.4% 490.50 interrupts.62:IR-PCI-MSI.1572887-edge.eth0-TxRx-23
441.00 +11.8% 493.00 interrupts.63:IR-PCI-MSI.1572888-edge.eth0-TxRx-24
437.25 +13.3% 495.25 interrupts.64:IR-PCI-MSI.1572889-edge.eth0-TxRx-25
436.25 +12.4% 490.50 interrupts.65:IR-PCI-MSI.1572890-edge.eth0-TxRx-26
436.25 +12.4% 490.50 interrupts.68:IR-PCI-MSI.1572893-edge.eth0-TxRx-29
436.25 +12.4% 490.50 interrupts.69:IR-PCI-MSI.1572894-edge.eth0-TxRx-30
436.25 +12.4% 490.50 interrupts.71:IR-PCI-MSI.1572896-edge.eth0-TxRx-32
437.25 ± 2% +12.2% 490.50 interrupts.74:IR-PCI-MSI.1572897-edge.eth0-TxRx-33
436.25 +12.6% 491.25 interrupts.75:IR-PCI-MSI.1572898-edge.eth0-TxRx-34
440.75 +12.4% 495.25 ± 2% interrupts.76:IR-PCI-MSI.1572899-edge.eth0-TxRx-35
436.25 +12.6% 491.00 interrupts.77:IR-PCI-MSI.1572900-edge.eth0-TxRx-36
436.75 +13.1% 493.75 interrupts.78:IR-PCI-MSI.1572901-edge.eth0-TxRx-37
436.25 +12.8% 492.25 interrupts.79:IR-PCI-MSI.1572902-edge.eth0-TxRx-38
436.25 +12.8% 492.25 interrupts.80:IR-PCI-MSI.1572903-edge.eth0-TxRx-39
436.25 +12.4% 490.50 interrupts.81:IR-PCI-MSI.1572904-edge.eth0-TxRx-40
436.25 +12.4% 490.50 interrupts.82:IR-PCI-MSI.1572905-edge.eth0-TxRx-41
436.25 +12.4% 490.50 interrupts.83:IR-PCI-MSI.1572906-edge.eth0-TxRx-42
436.25 +14.6% 499.75 ± 2% interrupts.84:IR-PCI-MSI.1572907-edge.eth0-TxRx-43
436.25 +12.4% 490.50 interrupts.85:IR-PCI-MSI.1572908-edge.eth0-TxRx-44
436.50 +12.4% 490.50 interrupts.86:IR-PCI-MSI.1572909-edge.eth0-TxRx-45
436.25 +12.4% 490.50 interrupts.87:IR-PCI-MSI.1572910-edge.eth0-TxRx-46
436.25 +12.4% 490.50 interrupts.88:IR-PCI-MSI.1572911-edge.eth0-TxRx-47
436.25 +12.6% 491.00 interrupts.89:IR-PCI-MSI.1572912-edge.eth0-TxRx-48
441.25 ± 3% +14.1% 503.25 ± 3% interrupts.90:IR-PCI-MSI.1572913-edge.eth0-TxRx-49
436.25 +12.4% 490.50 interrupts.91:IR-PCI-MSI.1572914-edge.eth0-TxRx-50
436.25 +14.4% 499.25 ± 3% interrupts.92:IR-PCI-MSI.1572915-edge.eth0-TxRx-51
439.25 ± 2% +11.7% 490.50 interrupts.94:IR-PCI-MSI.1572917-edge.eth0-TxRx-53
437.75 +12.1% 490.50 interrupts.95:IR-PCI-MSI.1572918-edge.eth0-TxRx-54
436.50 +12.4% 490.50 interrupts.96:IR-PCI-MSI.1572919-edge.eth0-TxRx-55
436.25 +12.4% 490.50 interrupts.97:IR-PCI-MSI.1572920-edge.eth0-TxRx-56
439.00 ± 2% +11.7% 490.50 interrupts.98:IR-PCI-MSI.1572921-edge.eth0-TxRx-57
442.25 ± 3% +10.9% 490.50 interrupts.99:IR-PCI-MSI.1572922-edge.eth0-TxRx-58
1989 +12.1% 2230 interrupts.9:IR-IO-APIC.9-fasteoi.acpi
6596036 ± 10% -21.1% 5206443 ± 6% interrupts.CAL:Function_call_interrupts
86097 ± 13% -18.4% 70229 ± 7% interrupts.CPU0.CAL:Function_call_interrupts
1785369 +12.4% 2006294 interrupts.CPU0.LOC:Local_timer_interrupts
102823 ± 12% -18.3% 84031 ± 5% interrupts.CPU0.TLB:TLB_shootdowns
388425 ± 12% +27.8% 496548 ± 4% interrupts.CPU0.TRM:Thermal_event_interrupts
1989 +12.1% 2230 interrupts.CPU1.9:IR-IO-APIC.9-fasteoi.acpi
90370 ± 8% -21.4% 70995 ± 6% interrupts.CPU1.CAL:Function_call_interrupts
1784834 +12.4% 2006046 interrupts.CPU1.LOC:Local_timer_interrupts
3088391 ± 15% -20.9% 2441882 ± 7% interrupts.CPU1.RES:Rescheduling_interrupts
105391 ± 8% -18.6% 85793 ± 6% interrupts.CPU1.TLB:TLB_shootdowns
388373 ± 12% +27.9% 496558 ± 4% interrupts.CPU1.TRM:Thermal_event_interrupts
88236 ± 11% -19.5% 71001 ± 7% interrupts.CPU10.CAL:Function_call_interrupts
1785471 +12.3% 2004740 interrupts.CPU10.LOC:Local_timer_interrupts
3040774 ± 4% -21.1% 2398396 ± 8% interrupts.CPU10.RES:Rescheduling_interrupts
104483 ± 10% -18.6% 85025 ± 5% interrupts.CPU10.TLB:TLB_shootdowns
388417 ± 12% +27.8% 496560 ± 4% interrupts.CPU10.TRM:Thermal_event_interrupts
1785039 +12.3% 2005274 interrupts.CPU11.LOC:Local_timer_interrupts
925.75 ±173% +461.3% 5196 ± 58% interrupts.CPU11.NMI:Non-maskable_interrupts
925.75 ±173% +461.3% 5196 ± 58% interrupts.CPU11.PMI:Performance_monitoring_interrupts
3094193 ± 14% -21.5% 2430228 ± 8% interrupts.CPU11.RES:Rescheduling_interrupts
102398 ± 10% -16.1% 85884 ± 5% interrupts.CPU11.TLB:TLB_shootdowns
388413 ± 12% +27.8% 496474 ± 4% interrupts.CPU11.TRM:Thermal_event_interrupts
588.00 ± 35% +196.0% 1740 ± 50% interrupts.CPU12.51:IR-PCI-MSI.1572876-edge.eth0-TxRx-12
87438 ± 9% -18.1% 71580 ± 7% interrupts.CPU12.CAL:Function_call_interrupts
1785376 +12.3% 2005068 interrupts.CPU12.LOC:Local_timer_interrupts
103057 ± 10% -16.4% 86186 ± 6% interrupts.CPU12.TLB:TLB_shootdowns
388416 ± 12% +27.8% 496490 ± 4% interrupts.CPU12.TRM:Thermal_event_interrupts
88735 ± 9% -19.4% 71496 ± 9% interrupts.CPU13.CAL:Function_call_interrupts
1785460 +12.3% 2005445 interrupts.CPU13.LOC:Local_timer_interrupts
104230 ± 9% -17.5% 86038 ± 6% interrupts.CPU13.TLB:TLB_shootdowns
388422 ± 12% +27.8% 496563 ± 4% interrupts.CPU13.TRM:Thermal_event_interrupts
88327 ± 12% -19.0% 71507 ± 7% interrupts.CPU14.CAL:Function_call_interrupts
1785094 +12.3% 2005441 interrupts.CPU14.LOC:Local_timer_interrupts
103317 ± 11% -18.1% 84581 ± 6% interrupts.CPU14.TLB:TLB_shootdowns
388412 ± 12% +27.8% 496564 ± 4% interrupts.CPU14.TRM:Thermal_event_interrupts
86972 ± 10% -17.3% 71898 ± 7% interrupts.CPU15.CAL:Function_call_interrupts
1784668 +12.4% 2005375 interrupts.CPU15.LOC:Local_timer_interrupts
3039799 ± 9% -21.1% 2397505 ± 9% interrupts.CPU15.RES:Rescheduling_interrupts
103367 ± 12% -16.9% 85892 ± 6% interrupts.CPU15.TLB:TLB_shootdowns
388342 ± 12% +27.9% 496559 ± 4% interrupts.CPU15.TRM:Thermal_event_interrupts
436.75 ± 2% +12.8% 492.50 interrupts.CPU16.55:IR-PCI-MSI.1572880-edge.eth0-TxRx-16
1784750 +12.4% 2005330 interrupts.CPU16.LOC:Local_timer_interrupts
55.25 ±171% +7696.8% 4307 ± 64% interrupts.CPU16.NMI:Non-maskable_interrupts
55.25 ±171% +7696.8% 4307 ± 64% interrupts.CPU16.PMI:Performance_monitoring_interrupts
2969996 ± 10% -18.7% 2413141 ± 9% interrupts.CPU16.RES:Rescheduling_interrupts
102367 ± 11% -13.7% 88305 ± 3% interrupts.CPU16.TLB:TLB_shootdowns
388423 ± 12% +27.8% 496563 ± 4% interrupts.CPU16.TRM:Thermal_event_interrupts
1785363 +12.3% 2005849 interrupts.CPU17.LOC:Local_timer_interrupts
388418 ± 12% +27.8% 496560 ± 4% interrupts.CPU17.TRM:Thermal_event_interrupts
438.00 +39.5% 611.00 ± 33% interrupts.CPU18.57:IR-PCI-MSI.1572882-edge.eth0-TxRx-18
88992 ± 12% -22.3% 69145 ± 6% interrupts.CPU18.CAL:Function_call_interrupts
1785398 +12.1% 2001316 interrupts.CPU18.LOC:Local_timer_interrupts
2199105 ± 7% +115.2% 4731679 ± 7% interrupts.CPU18.RES:Rescheduling_interrupts
108465 ± 8% -23.0% 83467 ± 5% interrupts.CPU18.TLB:TLB_shootdowns
329294 ± 3% +15.4% 380131 ± 2% interrupts.CPU18.TRM:Thermal_event_interrupts
436.25 +13.8% 496.25 interrupts.CPU19.58:IR-PCI-MSI.1572883-edge.eth0-TxRx-19
92059 ± 6% -24.2% 69776 ± 5% interrupts.CPU19.CAL:Function_call_interrupts
1784665 +12.2% 2001579 interrupts.CPU19.LOC:Local_timer_interrupts
1284622 ± 7% +225.8% 4185208 ± 4% interrupts.CPU19.RES:Rescheduling_interrupts
108492 ± 7% -23.4% 83059 ± 4% interrupts.CPU19.TLB:TLB_shootdowns
329166 ± 3% +15.5% 380141 ± 2% interrupts.CPU19.TRM:Thermal_event_interrupts
1784512 +12.4% 2005572 interrupts.CPU2.LOC:Local_timer_interrupts
3091132 ± 8% -22.3% 2401342 ± 4% interrupts.CPU2.RES:Rescheduling_interrupts
102562 ± 9% -16.5% 85643 ± 5% interrupts.CPU2.TLB:TLB_shootdowns
388380 ± 12% +27.8% 496544 ± 4% interrupts.CPU2.TRM:Thermal_event_interrupts
436.25 +14.3% 498.50 interrupts.CPU20.59:IR-PCI-MSI.1572884-edge.eth0-TxRx-20
91785 ± 9% -23.9% 69853 ± 4% interrupts.CPU20.CAL:Function_call_interrupts
1785284 +12.1% 2001952 interrupts.CPU20.LOC:Local_timer_interrupts
1281621 ± 12% +203.6% 3890516 ± 7% interrupts.CPU20.RES:Rescheduling_interrupts
106297 ± 9% -19.9% 85180 ± 3% interrupts.CPU20.TLB:TLB_shootdowns
329293 ± 3% +15.4% 380131 ± 2% interrupts.CPU20.TRM:Thermal_event_interrupts
437.00 +12.2% 490.50 interrupts.CPU21.60:IR-PCI-MSI.1572885-edge.eth0-TxRx-21
92635 ± 7% -23.9% 70512 ± 6% interrupts.CPU21.CAL:Function_call_interrupts
1785480 +12.1% 2001723 interrupts.CPU21.LOC:Local_timer_interrupts
1595447 ± 6% +187.8% 4591004 interrupts.CPU21.RES:Rescheduling_interrupts
107575 ± 7% -21.6% 84296 ± 5% interrupts.CPU21.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380086 ± 2% interrupts.CPU21.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU22.61:IR-PCI-MSI.1572886-edge.eth0-TxRx-22
92674 ± 10% -24.4% 70047 ± 6% interrupts.CPU22.CAL:Function_call_interrupts
1784881 +12.1% 2001482 interrupts.CPU22.LOC:Local_timer_interrupts
1605143 ± 6% +179.5% 4486945 ± 5% interrupts.CPU22.RES:Rescheduling_interrupts
108749 ± 10% -23.3% 83409 ± 5% interrupts.CPU22.TLB:TLB_shootdowns
329295 ± 3% +15.4% 380136 ± 2% interrupts.CPU22.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU23.62:IR-PCI-MSI.1572887-edge.eth0-TxRx-23
90975 ± 11% -23.6% 69492 ± 5% interrupts.CPU23.CAL:Function_call_interrupts
1785236 +12.1% 2001811 interrupts.CPU23.LOC:Local_timer_interrupts
1600988 ± 2% +180.2% 4486667 ± 5% interrupts.CPU23.RES:Rescheduling_interrupts
106819 ± 10% -22.3% 83007 ± 4% interrupts.CPU23.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380146 ± 2% interrupts.CPU23.TRM:Thermal_event_interrupts
441.00 +11.8% 493.00 interrupts.CPU24.63:IR-PCI-MSI.1572888-edge.eth0-TxRx-24
91919 ± 11% -23.1% 70689 ± 5% interrupts.CPU24.CAL:Function_call_interrupts
1784996 +12.1% 2000922 interrupts.CPU24.LOC:Local_timer_interrupts
1612237 ± 5% +188.4% 4649702 ± 2% interrupts.CPU24.RES:Rescheduling_interrupts
108187 ± 8% -22.5% 83831 ± 3% interrupts.CPU24.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380122 ± 2% interrupts.CPU24.TRM:Thermal_event_interrupts
437.25 +13.3% 495.25 interrupts.CPU25.64:IR-PCI-MSI.1572889-edge.eth0-TxRx-25
88417 ± 12% -20.8% 70029 ± 4% interrupts.CPU25.CAL:Function_call_interrupts
1785404 +12.0% 1999622 interrupts.CPU25.LOC:Local_timer_interrupts
1612074 ± 2% +182.4% 4551699 interrupts.CPU25.RES:Rescheduling_interrupts
105324 ± 10% -20.1% 84201 ± 4% interrupts.CPU25.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380141 ± 2% interrupts.CPU25.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU26.65:IR-PCI-MSI.1572890-edge.eth0-TxRx-26
92632 ± 12% -23.5% 70907 ± 5% interrupts.CPU26.CAL:Function_call_interrupts
1784730 +12.1% 2000264 interrupts.CPU26.LOC:Local_timer_interrupts
1639148 ± 6% +188.0% 4720149 ± 4% interrupts.CPU26.RES:Rescheduling_interrupts
108099 ± 11% -21.6% 84727 ± 4% interrupts.CPU26.TLB:TLB_shootdowns
329271 ± 3% +15.4% 379984 ± 2% interrupts.CPU26.TRM:Thermal_event_interrupts
91312 ± 12% -23.4% 69963 ± 6% interrupts.CPU27.CAL:Function_call_interrupts
1785171 +12.1% 2001743 interrupts.CPU27.LOC:Local_timer_interrupts
1704847 ± 11% +179.4% 4764019 ± 4% interrupts.CPU27.RES:Rescheduling_interrupts
106515 ± 11% -21.6% 83491 ± 5% interrupts.CPU27.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380080 ± 2% interrupts.CPU27.TRM:Thermal_event_interrupts
89178 ± 8% -22.0% 69565 ± 6% interrupts.CPU28.CAL:Function_call_interrupts
1785194 +12.1% 2001836 interrupts.CPU28.LOC:Local_timer_interrupts
1593025 ± 8% +185.7% 4551782 ± 5% interrupts.CPU28.RES:Rescheduling_interrupts
104408 ± 10% -19.2% 84365 ± 6% interrupts.CPU28.TLB:TLB_shootdowns
329294 ± 3% +15.4% 380142 ± 2% interrupts.CPU28.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU29.68:IR-PCI-MSI.1572893-edge.eth0-TxRx-29
90850 ± 12% -21.9% 70993 ± 6% interrupts.CPU29.CAL:Function_call_interrupts
1785398 +12.1% 2000545 interrupts.CPU29.LOC:Local_timer_interrupts
1657508 ± 5% +179.4% 4631109 ± 3% interrupts.CPU29.RES:Rescheduling_interrupts
106383 ± 13% -20.8% 84284 ± 4% interrupts.CPU29.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380133 ± 2% interrupts.CPU29.TRM:Thermal_event_interrupts
509.75 ± 3% +545.1% 3288 ±120% interrupts.CPU3.42:IR-PCI-MSI.1572867-edge.eth0-TxRx-3
87008 ± 11% -19.9% 69730 ± 5% interrupts.CPU3.CAL:Function_call_interrupts
1784446 +12.4% 2005368 interrupts.CPU3.LOC:Local_timer_interrupts
3169674 ± 9% -20.4% 2521963 ± 8% interrupts.CPU3.RES:Rescheduling_interrupts
103703 ± 12% -17.1% 85958 ± 7% interrupts.CPU3.TLB:TLB_shootdowns
388356 ± 12% +27.9% 496551 ± 4% interrupts.CPU3.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU30.69:IR-PCI-MSI.1572894-edge.eth0-TxRx-30
92043 ± 12% -24.2% 69765 ± 5% interrupts.CPU30.CAL:Function_call_interrupts
1785175 +12.1% 2001248 interrupts.CPU30.LOC:Local_timer_interrupts
1637465 ± 6% +191.3% 4769701 ± 3% interrupts.CPU30.RES:Rescheduling_interrupts
106836 ± 11% -22.5% 82810 ± 5% interrupts.CPU30.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380130 ± 2% interrupts.CPU30.TRM:Thermal_event_interrupts
1785301 +12.0% 1999270 interrupts.CPU31.LOC:Local_timer_interrupts
62.75 ±173% +4261.4% 2736 ±113% interrupts.CPU31.NMI:Non-maskable_interrupts
62.75 ±173% +4261.4% 2736 ±113% interrupts.CPU31.PMI:Performance_monitoring_interrupts
1589500 ± 5% +184.1% 4515325 interrupts.CPU31.RES:Rescheduling_interrupts
106576 ± 11% -21.5% 83698 ± 5% interrupts.CPU31.TLB:TLB_shootdowns
329293 ± 3% +15.4% 380131 ± 2% interrupts.CPU31.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU32.71:IR-PCI-MSI.1572896-edge.eth0-TxRx-32
93348 ± 12% -24.4% 70595 ± 8% interrupts.CPU32.CAL:Function_call_interrupts
1785245 +12.1% 2001145 interrupts.CPU32.LOC:Local_timer_interrupts
1626883 ± 6% +176.3% 4494456 ± 5% interrupts.CPU32.RES:Rescheduling_interrupts
107151 ± 12% -21.4% 84236 ± 5% interrupts.CPU32.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380131 ± 2% interrupts.CPU32.TRM:Thermal_event_interrupts
437.25 ± 2% +12.2% 490.50 interrupts.CPU33.74:IR-PCI-MSI.1572897-edge.eth0-TxRx-33
91487 ± 12% -22.7% 70758 ± 3% interrupts.CPU33.CAL:Function_call_interrupts
1785371 +12.0% 1999987 interrupts.CPU33.LOC:Local_timer_interrupts
1578761 ± 6% +201.2% 4754762 ± 6% interrupts.CPU33.RES:Rescheduling_interrupts
106071 ± 12% -21.3% 83430 ± 2% interrupts.CPU33.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380130 ± 2% interrupts.CPU33.TRM:Thermal_event_interrupts
436.25 +12.6% 491.25 interrupts.CPU34.75:IR-PCI-MSI.1572898-edge.eth0-TxRx-34
91116 ± 11% -24.3% 68943 ± 7% interrupts.CPU34.CAL:Function_call_interrupts
1785353 +12.1% 2001525 interrupts.CPU34.LOC:Local_timer_interrupts
1553828 ± 4% +186.0% 4444035 ± 3% interrupts.CPU34.RES:Rescheduling_interrupts
105575 ± 10% -20.7% 83762 ± 4% interrupts.CPU34.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380139 ± 2% interrupts.CPU34.TRM:Thermal_event_interrupts
440.75 +12.4% 495.25 ± 2% interrupts.CPU35.76:IR-PCI-MSI.1572899-edge.eth0-TxRx-35
91396 ± 11% -25.6% 67986 ± 11% interrupts.CPU35.CAL:Function_call_interrupts
1785237 +12.0% 2000288 interrupts.CPU35.LOC:Local_timer_interrupts
1602616 ± 10% +189.7% 4642115 ± 3% interrupts.CPU35.RES:Rescheduling_interrupts
107669 ± 12% -21.3% 84736 ± 4% interrupts.CPU35.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380145 ± 2% interrupts.CPU35.TRM:Thermal_event_interrupts
436.25 +12.6% 491.00 interrupts.CPU36.77:IR-PCI-MSI.1572900-edge.eth0-TxRx-36
93553 ± 8% -19.8% 75044 ± 6% interrupts.CPU36.CAL:Function_call_interrupts
1784636 +12.4% 2005389 interrupts.CPU36.LOC:Local_timer_interrupts
2588984 ± 6% +25.8% 3256348 ± 6% interrupts.CPU36.RES:Rescheduling_interrupts
107053 ± 7% -19.0% 86754 ± 5% interrupts.CPU36.TLB:TLB_shootdowns
388423 ± 12% +27.8% 496560 ± 4% interrupts.CPU36.TRM:Thermal_event_interrupts
436.75 +13.1% 493.75 interrupts.CPU37.78:IR-PCI-MSI.1572901-edge.eth0-TxRx-37
89518 ± 10% -18.9% 72619 ± 5% interrupts.CPU37.CAL:Function_call_interrupts
1784648 +12.4% 2005651 interrupts.CPU37.LOC:Local_timer_interrupts
3112476 ± 10% -20.8% 2464103 ± 5% interrupts.CPU37.RES:Rescheduling_interrupts
102576 ± 10% -14.4% 87794 ± 6% interrupts.CPU37.TLB:TLB_shootdowns
388412 ± 12% +27.8% 496563 ± 4% interrupts.CPU37.TRM:Thermal_event_interrupts
436.25 +12.8% 492.25 interrupts.CPU38.79:IR-PCI-MSI.1572902-edge.eth0-TxRx-38
92566 ± 10% -20.6% 73466 ± 6% interrupts.CPU38.CAL:Function_call_interrupts
1785132 +12.4% 2005883 interrupts.CPU38.LOC:Local_timer_interrupts
3163520 ± 10% -26.0% 2340820 ± 11% interrupts.CPU38.RES:Rescheduling_interrupts
105823 ± 10% -18.0% 86785 ± 4% interrupts.CPU38.TLB:TLB_shootdowns
388401 ± 12% +27.8% 496554 ± 4% interrupts.CPU38.TRM:Thermal_event_interrupts
436.25 +12.8% 492.25 interrupts.CPU39.80:IR-PCI-MSI.1572903-edge.eth0-TxRx-39
93197 ± 11% -21.8% 72854 ± 7% interrupts.CPU39.CAL:Function_call_interrupts
1784624 +12.4% 2005934 interrupts.CPU39.LOC:Local_timer_interrupts
105953 ± 11% -20.8% 83925 ± 6% interrupts.CPU39.TLB:TLB_shootdowns
388418 ± 12% +27.8% 496548 ± 4% interrupts.CPU39.TRM:Thermal_event_interrupts
86883 ± 13% -17.8% 71449 ± 7% interrupts.CPU4.CAL:Function_call_interrupts
1784433 +12.4% 2005462 interrupts.CPU4.LOC:Local_timer_interrupts
3119739 ± 9% -21.1% 2461192 ± 4% interrupts.CPU4.RES:Rescheduling_interrupts
104829 ± 11% -19.1% 84802 ± 5% interrupts.CPU4.TLB:TLB_shootdowns
388417 ± 12% +27.8% 496563 ± 4% interrupts.CPU4.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU40.81:IR-PCI-MSI.1572904-edge.eth0-TxRx-40
91881 ± 12% -21.4% 72188 ± 8% interrupts.CPU40.CAL:Function_call_interrupts
1784831 +12.4% 2005991 interrupts.CPU40.LOC:Local_timer_interrupts
3101288 ± 10% -17.2% 2569221 ± 7% interrupts.CPU40.RES:Rescheduling_interrupts
105884 ± 10% -19.0% 85772 ± 5% interrupts.CPU40.TLB:TLB_shootdowns
388415 ± 12% +27.8% 496564 ± 4% interrupts.CPU40.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU41.82:IR-PCI-MSI.1572905-edge.eth0-TxRx-41
91739 ± 12% -19.9% 73475 ± 7% interrupts.CPU41.CAL:Function_call_interrupts
1785231 +12.3% 2005398 interrupts.CPU41.LOC:Local_timer_interrupts
3223405 ± 11% -27.9% 2325498 ± 5% interrupts.CPU41.RES:Rescheduling_interrupts
104410 ± 11% -18.7% 84846 ± 6% interrupts.CPU41.TLB:TLB_shootdowns
388408 ± 12% +27.8% 496557 ± 4% interrupts.CPU41.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU42.83:IR-PCI-MSI.1572906-edge.eth0-TxRx-42
1785354 +12.4% 2006034 interrupts.CPU42.LOC:Local_timer_interrupts
3068304 ± 9% -22.1% 2390347 ± 6% interrupts.CPU42.RES:Rescheduling_interrupts
107102 ± 9% -20.7% 84890 ± 4% interrupts.CPU42.TLB:TLB_shootdowns
388423 ± 12% +27.8% 496564 ± 4% interrupts.CPU42.TRM:Thermal_event_interrupts
436.25 +14.6% 499.75 ± 2% interrupts.CPU43.84:IR-PCI-MSI.1572907-edge.eth0-TxRx-43
92178 ± 9% -18.3% 75306 ± 7% interrupts.CPU43.CAL:Function_call_interrupts
1784759 +12.4% 2005986 interrupts.CPU43.LOC:Local_timer_interrupts
6434 ± 7% -64.3% 2297 ±106% interrupts.CPU43.NMI:Non-maskable_interrupts
6434 ± 7% -64.3% 2297 ±106% interrupts.CPU43.PMI:Performance_monitoring_interrupts
104773 ± 9% -17.6% 86369 ± 6% interrupts.CPU43.TLB:TLB_shootdowns
388425 ± 12% +27.8% 496556 ± 4% interrupts.CPU43.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU44.85:IR-PCI-MSI.1572908-edge.eth0-TxRx-44
92387 ± 10% -21.9% 72174 ± 9% interrupts.CPU44.CAL:Function_call_interrupts
1784603 +12.4% 2006064 interrupts.CPU44.LOC:Local_timer_interrupts
106179 ± 9% -18.6% 86461 ± 6% interrupts.CPU44.TLB:TLB_shootdowns
388426 ± 12% +27.8% 496562 ± 4% interrupts.CPU44.TRM:Thermal_event_interrupts
436.50 +12.4% 490.50 interrupts.CPU45.86:IR-PCI-MSI.1572909-edge.eth0-TxRx-45
93641 ± 7% -21.9% 73155 ± 6% interrupts.CPU45.CAL:Function_call_interrupts
1784811 +12.4% 2005654 interrupts.CPU45.LOC:Local_timer_interrupts
3093195 ± 11% -15.6% 2609354 ± 4% interrupts.CPU45.RES:Rescheduling_interrupts
106463 ± 7% -19.6% 85593 ± 5% interrupts.CPU45.TLB:TLB_shootdowns
388408 ± 12% +27.8% 496564 ± 4% interrupts.CPU45.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU46.87:IR-PCI-MSI.1572910-edge.eth0-TxRx-46
91739 ± 10% -19.0% 74279 ± 8% interrupts.CPU46.CAL:Function_call_interrupts
1784609 +12.4% 2005522 interrupts.CPU46.LOC:Local_timer_interrupts
2970471 ± 9% -21.0% 2345439 ± 8% interrupts.CPU46.RES:Rescheduling_interrupts
108158 ± 10% -19.8% 86743 ± 4% interrupts.CPU46.TLB:TLB_shootdowns
388368 ± 12% +27.8% 496469 ± 4% interrupts.CPU46.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU47.88:IR-PCI-MSI.1572911-edge.eth0-TxRx-47
92856 ± 9% -20.0% 74327 ± 8% interrupts.CPU47.CAL:Function_call_interrupts
1784835 +12.4% 2005700 interrupts.CPU47.LOC:Local_timer_interrupts
3116242 ± 10% -20.4% 2480398 ± 5% interrupts.CPU47.RES:Rescheduling_interrupts
104927 ± 9% -18.6% 85393 ± 6% interrupts.CPU47.TLB:TLB_shootdowns
388403 ± 12% +27.8% 496554 ± 4% interrupts.CPU47.TRM:Thermal_event_interrupts
436.25 +12.6% 491.00 interrupts.CPU48.89:IR-PCI-MSI.1572912-edge.eth0-TxRx-48
92889 ± 11% -21.5% 72890 ± 8% interrupts.CPU48.CAL:Function_call_interrupts
1784772 +12.4% 2006205 interrupts.CPU48.LOC:Local_timer_interrupts
3175210 ± 12% -19.7% 2548339 ± 13% interrupts.CPU48.RES:Rescheduling_interrupts
105183 ± 11% -18.6% 85618 ± 6% interrupts.CPU48.TLB:TLB_shootdowns
388426 ± 12% +27.8% 496563 ± 4% interrupts.CPU48.TRM:Thermal_event_interrupts
441.25 ± 3% +14.1% 503.25 ± 3% interrupts.CPU49.90:IR-PCI-MSI.1572913-edge.eth0-TxRx-49
90134 ± 10% -18.3% 73603 ± 6% interrupts.CPU49.CAL:Function_call_interrupts
1785145 +12.4% 2005932 interrupts.CPU49.LOC:Local_timer_interrupts
3185609 ± 11% -17.6% 2626502 ± 4% interrupts.CPU49.RES:Rescheduling_interrupts
102737 ± 10% -17.3% 84932 ± 5% interrupts.CPU49.TLB:TLB_shootdowns
388425 ± 12% +27.8% 496560 ± 4% interrupts.CPU49.TRM:Thermal_event_interrupts
92114 ± 9% -23.2% 70721 ± 6% interrupts.CPU5.CAL:Function_call_interrupts
1785135 +12.3% 2005424 interrupts.CPU5.LOC:Local_timer_interrupts
3210646 ± 12% -24.8% 2415935 ± 4% interrupts.CPU5.RES:Rescheduling_interrupts
107349 ± 9% -18.9% 87034 ± 6% interrupts.CPU5.TLB:TLB_shootdowns
388418 ± 12% +27.8% 496564 ± 4% interrupts.CPU5.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU50.91:IR-PCI-MSI.1572914-edge.eth0-TxRx-50
92406 ± 10% -20.2% 73771 ± 8% interrupts.CPU50.CAL:Function_call_interrupts
1785200 +12.4% 2006488 interrupts.CPU50.LOC:Local_timer_interrupts
106635 ± 9% -19.9% 85431 ± 6% interrupts.CPU50.TLB:TLB_shootdowns
388419 ± 12% +27.8% 496564 ± 4% interrupts.CPU50.TRM:Thermal_event_interrupts
436.25 +14.4% 499.25 ± 3% interrupts.CPU51.92:IR-PCI-MSI.1572915-edge.eth0-TxRx-51
91491 ± 9% -20.8% 72473 ± 6% interrupts.CPU51.CAL:Function_call_interrupts
1784683 +12.4% 2005527 interrupts.CPU51.LOC:Local_timer_interrupts
3051494 ± 8% -24.6% 2300032 ± 10% interrupts.CPU51.RES:Rescheduling_interrupts
104954 ± 10% -20.2% 83775 ± 5% interrupts.CPU51.TLB:TLB_shootdowns
388264 ± 12% +27.9% 496462 ± 4% interrupts.CPU51.TRM:Thermal_event_interrupts
92938 ± 11% -19.7% 74594 ± 7% interrupts.CPU52.CAL:Function_call_interrupts
1785451 +12.4% 2006171 interrupts.CPU52.LOC:Local_timer_interrupts
3025681 ± 11% -17.1% 2506800 ± 10% interrupts.CPU52.RES:Rescheduling_interrupts
107100 ± 11% -19.6% 86156 ± 6% interrupts.CPU52.TLB:TLB_shootdowns
388422 ± 12% +27.8% 496563 ± 4% interrupts.CPU52.TRM:Thermal_event_interrupts
439.25 ± 2% +11.7% 490.50 interrupts.CPU53.94:IR-PCI-MSI.1572917-edge.eth0-TxRx-53
93000 ± 12% -20.9% 73549 ± 7% interrupts.CPU53.CAL:Function_call_interrupts
1785290 +12.3% 2005441 interrupts.CPU53.LOC:Local_timer_interrupts
3212623 ± 7% -18.2% 2629478 ± 10% interrupts.CPU53.RES:Rescheduling_interrupts
107303 ± 11% -17.5% 88567 ± 10% interrupts.CPU53.TLB:TLB_shootdowns
388411 ± 12% +27.8% 496557 ± 4% interrupts.CPU53.TRM:Thermal_event_interrupts
437.75 +12.1% 490.50 interrupts.CPU54.95:IR-PCI-MSI.1572918-edge.eth0-TxRx-54
1785170 +12.0% 2000090 interrupts.CPU54.LOC:Local_timer_interrupts
5351 ± 26% -64.5% 1900 ± 99% interrupts.CPU54.NMI:Non-maskable_interrupts
5351 ± 26% -64.5% 1900 ± 99% interrupts.CPU54.PMI:Performance_monitoring_interrupts
2066106 ± 9% +92.7% 3982033 ± 6% interrupts.CPU54.RES:Rescheduling_interrupts
108150 ± 12% -21.2% 85173 ± 5% interrupts.CPU54.TLB:TLB_shootdowns
329294 ± 3% +15.4% 380146 ± 2% interrupts.CPU54.TRM:Thermal_event_interrupts
436.50 +12.4% 490.50 interrupts.CPU55.96:IR-PCI-MSI.1572919-edge.eth0-TxRx-55
91164 ± 12% -19.9% 73014 ± 6% interrupts.CPU55.CAL:Function_call_interrupts
1785472 +12.0% 1999115 interrupts.CPU55.LOC:Local_timer_interrupts
1286654 ± 6% +228.2% 4223034 ± 5% interrupts.CPU55.RES:Rescheduling_interrupts
104943 ± 11% -19.5% 84466 ± 5% interrupts.CPU55.TLB:TLB_shootdowns
329292 ± 3% +15.4% 379918 ± 2% interrupts.CPU55.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU56.97:IR-PCI-MSI.1572920-edge.eth0-TxRx-56
96934 ± 13% -23.4% 74215 ± 6% interrupts.CPU56.CAL:Function_call_interrupts
1785010 +12.1% 2000169 interrupts.CPU56.LOC:Local_timer_interrupts
1312758 ± 17% +194.5% 3866522 ± 3% interrupts.CPU56.RES:Rescheduling_interrupts
109307 ± 12% -22.1% 85139 ± 4% interrupts.CPU56.TLB:TLB_shootdowns
329273 ± 3% +15.4% 380131 ± 2% interrupts.CPU56.TRM:Thermal_event_interrupts
439.00 ± 2% +11.7% 490.50 interrupts.CPU57.98:IR-PCI-MSI.1572921-edge.eth0-TxRx-57
94960 ± 11% -22.9% 73246 ± 5% interrupts.CPU57.CAL:Function_call_interrupts
1785297 +12.0% 2000019 interrupts.CPU57.LOC:Local_timer_interrupts
1666475 ± 10% +169.6% 4492937 interrupts.CPU57.RES:Rescheduling_interrupts
107534 ± 10% -22.0% 83871 ± 5% interrupts.CPU57.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380136 ± 2% interrupts.CPU57.TRM:Thermal_event_interrupts
442.25 ± 3% +10.9% 490.50 interrupts.CPU58.99:IR-PCI-MSI.1572922-edge.eth0-TxRx-58
92818 ± 12% -21.4% 72925 ± 5% interrupts.CPU58.CAL:Function_call_interrupts
1785429 +12.0% 1999528 interrupts.CPU58.LOC:Local_timer_interrupts
1606064 ± 5% +176.8% 4446179 ± 6% interrupts.CPU58.RES:Rescheduling_interrupts
103696 ± 11% -19.5% 83458 ± 4% interrupts.CPU58.TLB:TLB_shootdowns
329293 ± 3% +15.4% 380022 ± 2% interrupts.CPU58.TRM:Thermal_event_interrupts
439.00 +11.7% 490.50 interrupts.CPU59.100:IR-PCI-MSI.1572923-edge.eth0-TxRx-59
93876 ± 13% -20.5% 74669 ± 6% interrupts.CPU59.CAL:Function_call_interrupts
1785085 +12.0% 1999992 interrupts.CPU59.LOC:Local_timer_interrupts
1634812 ± 5% +174.5% 4487501 ± 4% interrupts.CPU59.RES:Rescheduling_interrupts
106929 ± 13% -21.0% 84522 ± 5% interrupts.CPU59.TLB:TLB_shootdowns
329291 ± 3% +15.4% 380111 ± 2% interrupts.CPU59.TRM:Thermal_event_interrupts
89930 ± 9% -20.5% 71520 ± 7% interrupts.CPU6.CAL:Function_call_interrupts
1785220 +12.3% 2005468 interrupts.CPU6.LOC:Local_timer_interrupts
106307 ± 7% -19.9% 85162 ± 6% interrupts.CPU6.TLB:TLB_shootdowns
388421 ± 12% +27.8% 496564 ± 4% interrupts.CPU6.TRM:Thermal_event_interrupts
436.25 +13.1% 493.25 interrupts.CPU60.101:IR-PCI-MSI.1572924-edge.eth0-TxRx-60
98595 ± 11% -25.6% 73379 ± 5% interrupts.CPU60.CAL:Function_call_interrupts
1785116 +12.1% 2000324 interrupts.CPU60.LOC:Local_timer_interrupts
1604314 ± 8% +186.8% 4601614 ± 5% interrupts.CPU60.RES:Rescheduling_interrupts
110442 ± 10% -24.0% 83932 ± 5% interrupts.CPU60.TLB:TLB_shootdowns
329256 ± 3% +15.5% 380140 ± 2% interrupts.CPU60.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU61.102:IR-PCI-MSI.1572925-edge.eth0-TxRx-61
93035 ± 11% -21.4% 73124 ± 5% interrupts.CPU61.CAL:Function_call_interrupts
1785176 +12.1% 2000960 interrupts.CPU61.LOC:Local_timer_interrupts
4567 ± 67% -79.2% 952.00 ±171% interrupts.CPU61.NMI:Non-maskable_interrupts
4567 ± 67% -79.2% 952.00 ±171% interrupts.CPU61.PMI:Performance_monitoring_interrupts
1575186 ± 6% +194.9% 4644760 ± 4% interrupts.CPU61.RES:Rescheduling_interrupts
103665 ± 11% -18.7% 84259 ± 5% interrupts.CPU61.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380127 ± 2% interrupts.CPU61.TRM:Thermal_event_interrupts
436.25 +12.4% 490.50 interrupts.CPU62.103:IR-PCI-MSI.1572926-edge.eth0-TxRx-62
95188 ± 10% -21.9% 74376 ± 6% interrupts.CPU62.CAL:Function_call_interrupts
1785028 +12.0% 1999262 interrupts.CPU62.LOC:Local_timer_interrupts
1613834 ± 8% +184.7% 4594499 ± 3% interrupts.CPU62.RES:Rescheduling_interrupts
106856 ± 10% -21.4% 83987 ± 5% interrupts.CPU62.TLB:TLB_shootdowns
329270 ± 3% +15.4% 380120 ± 2% interrupts.CPU62.TRM:Thermal_event_interrupts
94411 ± 11% -21.3% 74269 ± 5% interrupts.CPU63.CAL:Function_call_interrupts
1785322 +12.0% 1999237 interrupts.CPU63.LOC:Local_timer_interrupts
1669218 ± 6% +187.2% 4794684 ± 3% interrupts.CPU63.RES:Rescheduling_interrupts
105402 ± 10% -18.3% 86076 ± 8% interrupts.CPU63.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380141 ± 2% interrupts.CPU63.TRM:Thermal_event_interrupts
96060 ± 11% -22.6% 74364 ± 6% interrupts.CPU64.CAL:Function_call_interrupts
1785304 +12.0% 1999571 interrupts.CPU64.LOC:Local_timer_interrupts
1681501 ± 7% +164.0% 4438647 ± 3% interrupts.CPU64.RES:Rescheduling_interrupts
108534 ± 10% -22.8% 83813 ± 5% interrupts.CPU64.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380142 ± 2% interrupts.CPU64.TRM:Thermal_event_interrupts
94178 ± 9% -22.7% 72758 ± 8% interrupts.CPU65.CAL:Function_call_interrupts
1785338 +12.0% 1999408 interrupts.CPU65.LOC:Local_timer_interrupts
1674108 ± 4% +182.2% 4724262 ± 3% interrupts.CPU65.RES:Rescheduling_interrupts
107519 ± 10% -22.6% 83259 ± 5% interrupts.CPU65.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380128 ± 2% interrupts.CPU65.TRM:Thermal_event_interrupts
95135 ± 8% -22.4% 73795 ± 5% interrupts.CPU66.CAL:Function_call_interrupts
1785413 +12.0% 2000352 interrupts.CPU66.LOC:Local_timer_interrupts
616.75 ±171% +902.4% 6182 ± 28% interrupts.CPU66.NMI:Non-maskable_interrupts
616.75 ±171% +902.4% 6182 ± 28% interrupts.CPU66.PMI:Performance_monitoring_interrupts
1646166 ± 6% +169.3% 4433267 ± 6% interrupts.CPU66.RES:Rescheduling_interrupts
107190 ± 7% -22.5% 83034 ± 4% interrupts.CPU66.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380118 ± 2% interrupts.CPU66.TRM:Thermal_event_interrupts
95627 ± 10% -22.4% 74249 ± 5% interrupts.CPU67.CAL:Function_call_interrupts
1785314 +12.1% 2001128 interrupts.CPU67.LOC:Local_timer_interrupts
1611146 ± 4% +183.8% 4572204 ± 7% interrupts.CPU67.RES:Rescheduling_interrupts
109283 ± 11% -22.5% 84676 ± 4% interrupts.CPU67.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380142 ± 2% interrupts.CPU67.TRM:Thermal_event_interrupts
95479 ± 11% -20.6% 75791 ± 5% interrupts.CPU68.CAL:Function_call_interrupts
1785192 +12.0% 1999411 interrupts.CPU68.LOC:Local_timer_interrupts
1575394 ± 7% +189.9% 4567277 ± 4% interrupts.CPU68.RES:Rescheduling_interrupts
106972 ± 12% -18.6% 87039 ± 5% interrupts.CPU68.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380129 ± 2% interrupts.CPU68.TRM:Thermal_event_interrupts
93067 ± 14% -19.9% 74538 ± 4% interrupts.CPU69.CAL:Function_call_interrupts
1785257 +12.1% 2000636 interrupts.CPU69.LOC:Local_timer_interrupts
1578663 ± 9% +192.9% 4624518 ± 3% interrupts.CPU69.RES:Rescheduling_interrupts
104271 ± 12% -18.5% 84989 ± 2% interrupts.CPU69.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380066 ± 2% interrupts.CPU69.TRM:Thermal_event_interrupts
474.00 ± 3% +49.7% 709.75 ± 24% interrupts.CPU7.46:IR-PCI-MSI.1572871-edge.eth0-TxRx-7
92062 ± 9% -19.7% 73884 ± 5% interrupts.CPU7.CAL:Function_call_interrupts
1784266 +12.4% 2004980 interrupts.CPU7.LOC:Local_timer_interrupts
2505703 ± 11% -19.0% 2029187 ± 5% interrupts.CPU7.RES:Rescheduling_interrupts
107698 ± 8% -18.4% 87903 ± 6% interrupts.CPU7.TLB:TLB_shootdowns
388281 ± 12% +27.8% 496347 ± 4% interrupts.CPU7.TRM:Thermal_event_interrupts
92337 ± 12% -17.5% 76148 ± 2% interrupts.CPU70.CAL:Function_call_interrupts
1785161 +12.0% 1999757 interrupts.CPU70.LOC:Local_timer_interrupts
1562360 ± 4% +170.8% 4231208 ± 9% interrupts.CPU70.RES:Rescheduling_interrupts
103862 ± 10% -18.8% 84371 ± 3% interrupts.CPU70.TLB:TLB_shootdowns
329296 ± 3% +15.4% 380141 ± 2% interrupts.CPU70.TRM:Thermal_event_interrupts
95829 ± 12% -21.2% 75509 ± 5% interrupts.CPU71.CAL:Function_call_interrupts
1784894 +12.1% 2001287 interrupts.CPU71.LOC:Local_timer_interrupts
1533567 ± 7% +196.9% 4553440 interrupts.CPU71.RES:Rescheduling_interrupts
105883 ± 11% -20.4% 84248 ± 6% interrupts.CPU71.TLB:TLB_shootdowns
329247 ± 3% +15.5% 380146 ± 2% interrupts.CPU71.TRM:Thermal_event_interrupts
88372 ± 11% -20.3% 70389 ± 7% interrupts.CPU8.CAL:Function_call_interrupts
1784452 +12.4% 2005305 interrupts.CPU8.LOC:Local_timer_interrupts
103992 ± 10% -17.4% 85887 ± 5% interrupts.CPU8.TLB:TLB_shootdowns
388422 ± 12% +27.8% 496557 ± 4% interrupts.CPU8.TRM:Thermal_event_interrupts
89290 ± 10% -19.1% 72221 ± 6% interrupts.CPU9.CAL:Function_call_interrupts
1784716 +12.4% 2005710 interrupts.CPU9.LOC:Local_timer_interrupts
104340 ± 10% -16.0% 87647 ± 3% interrupts.CPU9.TLB:TLB_shootdowns
388408 ± 12% +27.8% 496557 ± 4% interrupts.CPU9.TRM:Thermal_event_interrupts
1.285e+08 +12.2% 1.442e+08 interrupts.LOC:Local_timer_interrupts
144.00 +50.0% 216.00 interrupts.MCP:Machine_check_polls
1.677e+08 ± 8% +50.4% 2.522e+08 ± 2% interrupts.RES:Rescheduling_interrupts
7624145 ± 10% -19.7% 6125835 ± 5% interrupts.TLB:TLB_shootdowns
25836833 ± 8% +22.2% 31559779 ± 2% interrupts.TRM:Thermal_event_interrupts
***************************************************************************************************
lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
gcc-7/performance/pipe/x86_64-rhel-7.6/process/50%/debian-x86_64-2019-11-14.cgz/lkp-bdw-ep6/hackbench/0xb000038
commit:
43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq")
3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found")
43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
%stddev %change %stddev
\ | \
134091 -10.6% 119937 hackbench.throughput
3.352e+09 ± 2% +16.1% 3.892e+09 ± 5% hackbench.time.involuntary_context_switches
535571 ± 2% +10.6% 592420 ± 4% numa-meminfo.node0.FilePages
12614 ± 18% -12.2% 11078 ± 18% numa-meminfo.node1.Mapped
3472 ± 6% +18.0% 4096 slabinfo.skbuff_head_cache.active_objs
3568 ± 6% +15.7% 4128 slabinfo.skbuff_head_cache.num_objs
730128 ± 5% -52.2% 349073 ± 25% turbostat.C3
14.77 +5.9% 15.64 turbostat.RAMWatt
24613783 ± 5% -32.6% 16595193 ± 23% numa-numastat.node0.local_node
24632941 ± 5% -32.6% 16605181 ± 23% numa-numastat.node0.numa_hit
25827961 -29.9% 18097150 ± 19% numa-numastat.node1.local_node
25836948 -29.9% 18115676 ± 19% numa-numastat.node1.numa_hit
50467170 ± 2% -31.2% 34733456 ± 10% proc-vmstat.numa_hit
50439016 ± 2% -31.2% 34704933 ± 10% proc-vmstat.numa_local
50622650 ± 2% -31.1% 34896854 ± 10% proc-vmstat.pgalloc_normal
50548381 ± 2% -31.1% 34834391 ± 10% proc-vmstat.pgfree
79.00 +2.2% 80.75 vmstat.cpu.sy
19.00 -7.9% 17.50 ± 2% vmstat.cpu.us
13450755 +4.6% 14066265 vmstat.system.cs
1122772 -12.1% 986681 vmstat.system.in
133891 ± 2% +10.6% 148112 ± 4% numa-vmstat.node0.nr_file_pages
12301995 ± 4% -30.2% 8590153 ± 22% numa-vmstat.node0.numa_hit
12282910 ± 4% -30.1% 8580139 ± 22% numa-vmstat.node0.numa_local
12964354 ± 5% -31.2% 8914408 ± 18% numa-vmstat.node1.numa_hit
12806275 ± 5% -31.7% 8746928 ± 19% numa-vmstat.node1.numa_local
33816695 ± 10% +17.8% 39824104 ± 7% cpuidle.C1.time
21015951 ± 3% +19.6% 25127666 ± 7% cpuidle.C1E.time
730991 ± 5% -52.1% 350128 ± 25% cpuidle.C3.usage
1.497e+08 ± 28% +41.6% 2.119e+08 ± 15% cpuidle.C6.time
6771631 ± 4% -45.2% 3713804 ± 15% cpuidle.POLL.time
5209842 ± 4% -44.1% 2913245 ± 15% cpuidle.POLL.usage
26965506 +5.5% 28444381 ± 4% sched_debug.cfs_rq:/.min_vruntime.avg
28257967 +6.6% 30126596 ± 4% sched_debug.cfs_rq:/.min_vruntime.max
582203 ± 9% +37.0% 797353 ± 8% sched_debug.cfs_rq:/.min_vruntime.stddev
4.65 ± 8% -10.7% 4.16 ± 6% sched_debug.cfs_rq:/.runnable_load_avg.stddev
4620 ± 6% -10.7% 4125 ± 4% sched_debug.cfs_rq:/.runnable_weight.stddev
582831 ± 9% +36.6% 796226 ± 7% sched_debug.cfs_rq:/.spread0.stddev
154.73 ± 2% -14.7% 132.00 ± 3% sched_debug.cfs_rq:/.util_avg.stddev
120.93 ± 25% +43.5% 173.50 ± 19% sched_debug.cfs_rq:/.util_est_enqueued.min
184.93 ± 2% -11.0% 164.54 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev
182135 ± 18% +57.1% 286098 ± 6% sched_debug.cpu.avg_idle.avg
185926 ± 19% +49.0% 276991 ± 9% sched_debug.cpu.avg_idle.stddev
3.32 ± 4% -10.8% 2.96 ± 3% sched_debug.cpu.nr_running.stddev
46710201 +9.0% 50934866 ± 5% sched_debug.cpu.nr_switches.avg
50053735 +16.9% 58491253 ± 7% sched_debug.cpu.nr_switches.max
43751407 +6.7% 46698664 ± 4% sched_debug.cpu.nr_switches.min
1344975 ± 10% +78.9% 2406504 ± 16% sched_debug.cpu.nr_switches.stddev
1.63 ± 3% -32.8% 1.09 ± 11% sched_debug.cpu.nr_uninterruptible.avg
1017 ± 27% -59.5% 411.75 ± 16% interrupts.40:IR-PCI-MSI.1572871-edge.eth0-TxRx-6
6547829 -14.4% 5603910 ± 5% interrupts.CPU16.RES:Rescheduling_interrupts
1017 ± 27% -59.5% 411.75 ± 16% interrupts.CPU19.40:IR-PCI-MSI.1572871-edge.eth0-TxRx-6
6801598 -14.7% 5801847 ± 5% interrupts.CPU19.RES:Rescheduling_interrupts
6609756 -13.7% 5703084 ± 8% interrupts.CPU20.RES:Rescheduling_interrupts
6446878 ± 2% -11.8% 5689145 ± 7% interrupts.CPU21.RES:Rescheduling_interrupts
6634825 -9.4% 6008203 ± 6% interrupts.CPU24.RES:Rescheduling_interrupts
6801182 -9.8% 6137528 ± 3% interrupts.CPU25.RES:Rescheduling_interrupts
6687252 -12.9% 5821967 ± 6% interrupts.CPU28.RES:Rescheduling_interrupts
6583387 -10.9% 5868659 ± 6% interrupts.CPU29.RES:Rescheduling_interrupts
6585631 -16.6% 5490726 ± 9% interrupts.CPU37.RES:Rescheduling_interrupts
6759973 -13.2% 5867800 ± 7% interrupts.CPU57.RES:Rescheduling_interrupts
6459616 ± 2% -10.5% 5783047 ± 6% interrupts.CPU58.RES:Rescheduling_interrupts
6580818 ± 2% -16.9% 5468945 ± 3% interrupts.CPU60.RES:Rescheduling_interrupts
6723178 ± 2% -12.4% 5887778 ± 3% interrupts.CPU63.RES:Rescheduling_interrupts
6480129 -16.2% 5427168 ± 7% interrupts.CPU64.RES:Rescheduling_interrupts
6439558 ± 2% -11.6% 5691160 ± 5% interrupts.CPU65.RES:Rescheduling_interrupts
6662822 -12.0% 5866390 ± 7% interrupts.CPU68.RES:Rescheduling_interrupts
6602178 -12.0% 5809760 ± 7% interrupts.CPU7.RES:Rescheduling_interrupts
6616172 -14.1% 5685889 ± 7% interrupts.CPU73.RES:Rescheduling_interrupts
6591477 -13.8% 5682655 ± 8% interrupts.CPU81.RES:Rescheduling_interrupts
13.97 +15.8% 16.18 perf-stat.i.MPKI
1.70 -0.1 1.65 perf-stat.i.branch-miss-rate%
3.607e+08 -1.6% 3.549e+08 perf-stat.i.branch-misses
1.22 ± 6% +0.8 2.01 ± 18% perf-stat.i.cache-miss-rate%
17650711 ± 6% +98.0% 34953649 ± 18% perf-stat.i.cache-misses
1.496e+09 +17.3% 1.755e+09 perf-stat.i.cache-references
13503062 +4.6% 14122231 perf-stat.i.context-switches
2946654 ± 2% +18.5% 3491269 ± 3% perf-stat.i.cpu-migrations
29365 ± 8% -30.6% 20387 ± 31% perf-stat.i.cycles-between-cache-misses
0.35 ± 2% +0.0 0.36 ± 2% perf-stat.i.dTLB-store-miss-rate%
2.048e+10 -2.1% 2.006e+10 perf-stat.i.dTLB-stores
55.17 -1.4 53.74 perf-stat.i.iTLB-load-miss-rate%
1.019e+08 +5.5% 1.074e+08 perf-stat.i.iTLB-loads
3068 -3.0% 2975 perf-stat.i.minor-faults
9195204 ± 6% +113.0% 19589023 ± 18% perf-stat.i.node-load-misses
28132 ± 6% +125.3% 63396 ± 13% perf-stat.i.node-loads
58.92 -9.8 49.16 ± 4% perf-stat.i.node-store-miss-rate%
3588196 ± 6% +60.5% 5760265 ± 19% perf-stat.i.node-store-misses
2415953 ± 5% +145.2% 5924042 ± 18% perf-stat.i.node-stores
3068 -3.0% 2975 perf-stat.i.page-faults
13.96 +16.0% 16.19 perf-stat.overall.MPKI
1.70 -0.1 1.65 perf-stat.overall.branch-miss-rate%
1.18 ± 6% +0.8 1.99 ± 18% perf-stat.overall.cache-miss-rate%
13823 ± 6% -47.8% 7221 ± 20% perf-stat.overall.cycles-between-cache-misses
0.35 ± 2% +0.0 0.36 ± 2% perf-stat.overall.dTLB-store-miss-rate%
55.11 -1.4 53.70 perf-stat.overall.iTLB-load-miss-rate%
59.75 -10.5 49.23 perf-stat.overall.node-store-miss-rate%
78192 +13.0% 88370 perf-stat.overall.path-length
3.601e+08 -1.6% 3.543e+08 perf-stat.ps.branch-misses
17622620 ± 6% +98.0% 34899005 ± 18% perf-stat.ps.cache-misses
1.494e+09 +17.3% 1.752e+09 perf-stat.ps.cache-references
13480657 +4.6% 14099389 perf-stat.ps.context-switches
2941749 ± 2% +18.5% 3485631 ± 3% perf-stat.ps.cpu-migrations
2.045e+10 -2.1% 2.003e+10 perf-stat.ps.dTLB-stores
1.017e+08 +5.5% 1.072e+08 perf-stat.ps.iTLB-loads
3064 -3.0% 2972 perf-stat.ps.minor-faults
9180406 ± 6% +113.0% 19558101 ± 18% perf-stat.ps.node-load-misses
28101 ± 6% +125.3% 63314 ± 13% perf-stat.ps.node-loads
3582467 ± 6% +60.5% 5751138 ± 19% perf-stat.ps.node-store-misses
2412086 ± 5% +145.2% 5914636 ± 18% perf-stat.ps.node-stores
3064 -3.0% 2972 perf-stat.ps.page-faults
223485 +9.1% 243918 ± 5% softirqs.CPU0.TIMER
11887 ± 11% -18.2% 9723 ± 6% softirqs.CPU1.SCHED
215411 +9.2% 235267 ± 6% softirqs.CPU1.TIMER
11293 ± 3% -14.3% 9678 ± 10% softirqs.CPU10.SCHED
213916 +9.8% 234908 ± 5% softirqs.CPU10.TIMER
212758 +7.7% 229141 ± 5% softirqs.CPU11.TIMER
11098 ± 2% -15.9% 9337 ± 10% softirqs.CPU12.SCHED
213397 +9.0% 232706 ± 6% softirqs.CPU12.TIMER
212419 +8.3% 230115 ± 6% softirqs.CPU13.TIMER
10777 ± 4% -12.5% 9426 ± 6% softirqs.CPU14.SCHED
11241 ± 3% -11.5% 9950 ± 6% softirqs.CPU15.SCHED
214604 +10.0% 236116 ± 8% softirqs.CPU15.TIMER
11147 -14.9% 9482 ± 8% softirqs.CPU16.SCHED
216343 +8.4% 234558 ± 6% softirqs.CPU16.TIMER
214680 +7.0% 229748 ± 6% softirqs.CPU17.TIMER
212791 +8.6% 230988 ± 6% softirqs.CPU19.TIMER
11211 ± 4% -10.4% 10047 ± 6% softirqs.CPU2.SCHED
219918 +9.8% 241513 ± 6% softirqs.CPU2.TIMER
10825 ± 4% -14.3% 9273 ± 7% softirqs.CPU20.SCHED
215209 +8.8% 234149 ± 6% softirqs.CPU20.TIMER
10869 ± 4% -11.4% 9626 ± 4% softirqs.CPU21.SCHED
216225 +8.7% 235105 ± 5% softirqs.CPU21.TIMER
13013 ± 7% -14.0% 11192 ± 3% softirqs.CPU22.SCHED
220406 +8.9% 239975 ± 4% softirqs.CPU22.TIMER
11925 -14.9% 10145 ± 3% softirqs.CPU23.SCHED
220231 +7.6% 237056 ± 5% softirqs.CPU23.TIMER
11140 ± 2% -11.5% 9861 ± 7% softirqs.CPU24.SCHED
216091 +8.9% 235338 ± 5% softirqs.CPU24.TIMER
11438 ± 3% -16.3% 9578 ± 2% softirqs.CPU25.SCHED
214880 +9.4% 235143 ± 5% softirqs.CPU25.TIMER
214888 +8.8% 233874 ± 5% softirqs.CPU26.TIMER
11410 ± 2% -13.7% 9844 ± 10% softirqs.CPU27.SCHED
215012 +8.7% 233790 ± 4% softirqs.CPU27.TIMER
220618 ± 2% +7.8% 237788 ± 5% softirqs.CPU28.TIMER
11096 ± 5% -14.3% 9513 ± 6% softirqs.CPU29.SCHED
217194 +8.2% 235108 ± 5% softirqs.CPU29.TIMER
11445 ± 2% -16.7% 9536 ± 10% softirqs.CPU3.SCHED
214692 +9.1% 234221 ± 5% softirqs.CPU3.TIMER
11127 ± 5% -10.5% 9954 ± 6% softirqs.CPU30.SCHED
214861 +8.2% 232532 ± 5% softirqs.CPU30.TIMER
217582 +7.8% 234493 ± 4% softirqs.CPU31.TIMER
214606 +8.8% 233473 ± 4% softirqs.CPU32.TIMER
11299 ± 2% -11.7% 9975 ± 6% softirqs.CPU33.SCHED
11503 ± 2% -16.0% 9658 ± 10% softirqs.CPU34.SCHED
213233 +10.9% 236580 ± 6% softirqs.CPU34.TIMER
11162 ± 2% -11.6% 9863 ± 7% softirqs.CPU35.SCHED
11241 -13.9% 9678 ± 6% softirqs.CPU36.SCHED
215681 +9.2% 235446 ± 4% softirqs.CPU36.TIMER
11358 -15.2% 9632 ± 6% softirqs.CPU37.SCHED
219691 ± 2% +18.5% 260407 ± 13% softirqs.CPU37.TIMER
11570 ± 2% -13.1% 10049 ± 6% softirqs.CPU38.SCHED
215559 +8.2% 233189 ± 4% softirqs.CPU38.TIMER
11295 ± 4% -13.1% 9819 ± 10% softirqs.CPU39.SCHED
215781 ± 2% +7.6% 232108 ± 5% softirqs.CPU39.TIMER
11506 ± 7% -13.0% 10006 ± 6% softirqs.CPU4.SCHED
214973 +8.9% 234140 ± 6% softirqs.CPU4.TIMER
11304 ± 2% -13.7% 9759 ± 10% softirqs.CPU40.SCHED
213827 +10.1% 235341 ± 4% softirqs.CPU40.TIMER
11462 ± 4% -13.5% 9913 ± 8% softirqs.CPU41.SCHED
215879 ± 2% +7.2% 231499 ± 5% softirqs.CPU41.TIMER
11418 ± 3% -14.0% 9819 ± 8% softirqs.CPU43.SCHED
214864 +8.3% 232698 ± 4% softirqs.CPU43.TIMER
10706 ± 5% -13.9% 9223 ± 11% softirqs.CPU44.SCHED
217386 +9.4% 237916 ± 5% softirqs.CPU44.TIMER
215191 +9.7% 236017 ± 5% softirqs.CPU45.TIMER
214214 +9.9% 235466 ± 6% softirqs.CPU46.TIMER
214471 +9.9% 235675 ± 5% softirqs.CPU47.TIMER
11251 ± 2% -13.0% 9785 ± 8% softirqs.CPU48.SCHED
212500 +9.9% 233451 ± 6% softirqs.CPU48.TIMER
213609 +9.9% 234657 ± 5% softirqs.CPU49.TIMER
216836 +8.5% 235225 ± 5% softirqs.CPU5.TIMER
215854 +9.0% 235383 ± 5% softirqs.CPU50.TIMER
11468 -14.4% 9820 ± 9% softirqs.CPU51.SCHED
215402 +8.8% 234360 ± 5% softirqs.CPU51.TIMER
214152 +8.8% 233038 ± 5% softirqs.CPU52.TIMER
212843 +8.4% 230716 ± 5% softirqs.CPU53.TIMER
11373 ± 2% -14.9% 9679 ± 7% softirqs.CPU54.SCHED
214008 +9.5% 234380 ± 5% softirqs.CPU54.TIMER
212619 +7.5% 228539 ± 5% softirqs.CPU55.TIMER
11136 ± 3% -16.4% 9312 ± 10% softirqs.CPU56.SCHED
213404 +9.0% 232542 ± 6% softirqs.CPU56.TIMER
11171 ± 4% -11.9% 9837 ± 7% softirqs.CPU57.SCHED
212283 +8.6% 230474 ± 6% softirqs.CPU57.TIMER
11137 ± 4% -14.0% 9578 ± 6% softirqs.CPU58.SCHED
11226 ± 4% -17.2% 9289 ± 4% softirqs.CPU59.SCHED
214193 +23.1% 263668 ± 22% softirqs.CPU59.TIMER
217195 +8.2% 234975 ± 5% softirqs.CPU6.TIMER
11051 ± 4% -14.4% 9460 ± 9% softirqs.CPU60.SCHED
217131 +7.9% 234363 ± 5% softirqs.CPU60.TIMER
11206 ± 3% -12.6% 9793 ± 9% softirqs.CPU61.SCHED
211791 +9.2% 231302 ± 5% softirqs.CPU61.TIMER
214956 ± 2% +8.8% 233770 ± 5% softirqs.CPU62.TIMER
11180 ± 3% -14.8% 9526 ± 8% softirqs.CPU63.SCHED
213413 +8.7% 232008 ± 5% softirqs.CPU63.TIMER
215674 +8.1% 233086 ± 6% softirqs.CPU64.TIMER
11032 ± 3% -15.7% 9302 ± 7% softirqs.CPU65.SCHED
215870 +9.0% 235386 ± 5% softirqs.CPU65.TIMER
11403 ± 3% -14.2% 9786 ± 6% softirqs.CPU66.SCHED
215968 +9.4% 236175 ± 5% softirqs.CPU66.TIMER
11098 ± 3% -7.4% 10281 ± 4% softirqs.CPU67.SCHED
217846 +8.3% 236023 ± 4% softirqs.CPU67.TIMER
11094 -12.2% 9740 ± 7% softirqs.CPU68.SCHED
215512 +9.4% 235796 ± 5% softirqs.CPU68.TIMER
11321 ± 3% -14.1% 9724 ± 3% softirqs.CPU69.SCHED
213564 +9.0% 232867 ± 5% softirqs.CPU69.TIMER
218822 +8.8% 238080 ± 6% softirqs.CPU7.TIMER
11507 ± 2% -12.9% 10026 ± 4% softirqs.CPU70.SCHED
213928 +9.4% 233986 ± 4% softirqs.CPU70.TIMER
213528 +8.5% 231730 ± 4% softirqs.CPU71.TIMER
11262 ± 2% -11.1% 10013 ± 5% softirqs.CPU72.SCHED
11145 ± 3% -14.5% 9533 ± 8% softirqs.CPU73.SCHED
216467 +9.2% 236409 ± 5% softirqs.CPU73.TIMER
11447 ± 4% -13.2% 9935 ± 4% softirqs.CPU74.SCHED
212578 +9.1% 232022 ± 5% softirqs.CPU74.TIMER
215256 +7.8% 232072 ± 5% softirqs.CPU75.TIMER
11324 ± 4% -11.8% 9985 ± 3% softirqs.CPU76.SCHED
214170 +8.6% 232516 ± 5% softirqs.CPU76.TIMER
11646 ± 5% -15.7% 9824 ± 7% softirqs.CPU77.SCHED
213706 +8.0% 230880 ± 5% softirqs.CPU77.TIMER
11397 ± 6% -13.9% 9810 ± 10% softirqs.CPU78.SCHED
11179 ± 3% -11.5% 9894 ± 5% softirqs.CPU79.SCHED
213215 +8.4% 231081 ± 5% softirqs.CPU79.TIMER
11165 ± 4% -16.2% 9352 ± 9% softirqs.CPU8.SCHED
216508 ± 2% +7.9% 233692 ± 6% softirqs.CPU8.TIMER
11203 ± 2% -13.9% 9643 ± 4% softirqs.CPU80.SCHED
216446 +8.3% 234469 ± 5% softirqs.CPU80.TIMER
11151 ± 2% -12.8% 9726 ± 6% softirqs.CPU81.SCHED
11373 ± 3% -13.0% 9891 ± 6% softirqs.CPU82.SCHED
213597 +8.9% 232608 ± 5% softirqs.CPU82.TIMER
214308 +7.9% 231148 ± 5% softirqs.CPU83.TIMER
214384 +20.9% 259149 ± 15% softirqs.CPU84.TIMER
11458 -13.7% 9887 ± 7% softirqs.CPU86.SCHED
215459 +8.2% 233058 ± 5% softirqs.CPU86.TIMER
11303 ± 4% -12.4% 9904 ± 8% softirqs.CPU87.SCHED
214185 +7.9% 231116 ± 5% softirqs.CPU87.TIMER
11030 ± 2% -13.0% 9593 ± 8% softirqs.CPU9.SCHED
213229 +8.7% 231788 ± 6% softirqs.CPU9.TIMER
995635 -12.9% 867688 ± 5% softirqs.SCHED
19052052 +8.6% 20690944 ± 5% softirqs.TIMER
31.32 -1.6 29.74 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
30.24 -1.5 28.76 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
26.06 -0.9 25.17 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.35 ± 5% -0.9 1.48 ± 12% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common
25.36 -0.8 24.54 perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
2.75 ± 4% -0.7 2.08 ± 7% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
4.24 ± 2% -0.6 3.60 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
3.84 ± 2% -0.6 3.23 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
7.83 -0.5 7.30 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
1.68 ± 6% -0.5 1.17 ± 10% perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
8.02 -0.5 7.53 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
8.08 -0.5 7.60 ± 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
1.63 ± 4% -0.4 1.19 ± 6% perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_write.new_sync_write.vfs_write.ksys_write
1.71 ± 6% -0.4 1.28 ± 8% perf-profile.calltrace.cycles-pp.update_cfs_group.dequeue_task_fair.__schedule.schedule.pipe_wait
2.60 -0.4 2.19 ± 4% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.03 ± 2% -0.4 1.64 ± 4% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.47 -0.4 3.08 ± 4% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
1.95 ± 2% -0.3 1.61 ± 3% perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write
2.67 -0.3 2.35 ± 3% perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read
0.56 -0.3 0.26 ±100% perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.22 ± 4% -0.3 0.94 ± 2% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.new_sync_write.vfs_write.ksys_write
1.25 ± 2% -0.3 1.00 ± 5% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
2.64 -0.2 2.41 ± 2% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_wait
1.53 -0.2 1.34 ± 7% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64
0.59 ± 4% -0.2 0.40 ± 57% perf-profile.calltrace.cycles-pp.__enqueue_entity.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
1.34 -0.2 1.16 ± 6% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
1.07 ± 2% -0.2 0.90 ± 4% perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.pipe_write.new_sync_write.vfs_write
1.05 -0.2 0.88 ± 6% perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
1.56 ± 2% -0.2 1.38 ± 2% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.pipe_read.new_sync_read
1.17 ± 4% -0.2 1.01 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
0.98 ± 2% -0.2 0.82 ± 4% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.66 ± 2% -0.2 1.50 ± 5% perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.pipe_read.new_sync_read.vfs_read
0.73 ± 3% -0.2 0.57 ± 2% perf-profile.calltrace.cycles-pp.file_has_perm.security_file_permission.vfs_read.ksys_read.do_syscall_64
0.96 ± 2% -0.1 0.82 ± 3% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter.pipe_write.new_sync_write
0.89 ± 2% -0.1 0.77 ± 3% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.66 ± 6% -0.1 0.54 ± 2% perf-profile.calltrace.cycles-pp.file_has_perm.security_file_permission.vfs_write.ksys_write.do_syscall_64
2.42 -0.1 2.32 perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.pipe_wait.pipe_read
0.89 ± 4% -0.1 0.79 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write
0.74 ± 2% -0.1 0.66 ± 3% perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
0.84 -0.1 0.79 ± 3% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait
0.56 -0.0 0.54 ± 2% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate
0.77 ± 2% +0.1 0.82 perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
0.90 ± 3% +0.1 0.97 ± 2% perf-profile.calltrace.cycles-pp.native_write_msr
0.69 ± 3% +0.1 0.77 ± 3% perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
0.80 ± 4% +0.1 0.89 ± 2% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read
1.10 +0.1 1.20 perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
0.73 ± 3% +0.1 0.83 ± 2% perf-profile.calltrace.cycles-pp.__switch_to
1.05 +0.1 1.16 perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait
0.63 ± 4% +0.1 0.75 ± 2% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop
0.99 ± 5% +0.1 1.12 ± 5% perf-profile.calltrace.cycles-pp.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
1.07 ± 3% +0.1 1.21 ± 2% perf-profile.calltrace.cycles-pp.__switch_to_asm
0.64 ± 6% +0.2 0.80 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.pipe_wait.pipe_read
1.41 ± 3% +0.2 1.59 ± 2% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.77 ± 5% +0.2 1.00 ± 5% perf-profile.calltrace.cycles-pp.load_new_mm_cr3.switch_mm_irqs_off.__schedule.schedule.exit_to_usermode_loop
1.73 ± 3% +0.3 2.06 ± 3% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64
0.14 ±173% +0.5 0.63 ± 5% perf-profile.calltrace.cycles-pp.update_rq_clock.__schedule.schedule.pipe_wait.pipe_read
0.13 ±173% +0.5 0.64 ± 5% perf-profile.calltrace.cycles-pp.migrate_task_rq_fair.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common
1.85 ± 5% +0.5 2.38 ± 3% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.exit_to_usermode_loop.do_syscall_64
0.00 +0.6 0.59 ± 4% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_wait.pipe_read.new_sync_read.vfs_read
0.00 +0.6 0.60 ± 5% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.dequeue_task_fair.__schedule.schedule.pipe_wait
0.00 +0.7 0.67 ± 5% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
0.15 ±173% +1.0 1.14 ± 12% perf-profile.calltrace.cycles-pp.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
87.66 +1.1 88.78 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
86.87 +1.1 88.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.59 ± 4% +1.4 6.96 ± 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.72 ± 4% +1.4 7.12 ± 3% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.87 ± 4% +1.4 7.28 ± 3% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.70 ± 30% +3.3 4.98 ± 13% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
35.54 +3.3 38.87 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
34.28 +3.5 37.82 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
31.12 +4.1 35.22 ± 2% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
30.49 +4.1 34.63 ± 2% perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
22.54 ± 2% +6.0 28.53 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
20.58 ± 2% +6.2 26.76 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
19.97 ± 2% +6.2 26.16 ± 3% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
19.73 ± 3% +6.2 25.92 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
3.39 ± 20% +7.0 10.39 ± 12% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
4.89 ± 15% +7.1 11.98 ± 10% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
31.34 -1.6 29.75 perf-profile.children.cycles-pp.ksys_read
30.28 -1.5 28.79 perf-profile.children.cycles-pp.vfs_read
4.14 ± 5% -1.3 2.85 ± 7% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
3.51 ± 6% -1.0 2.54 ± 8% perf-profile.children.cycles-pp.update_cfs_group
26.10 -0.9 25.20 perf-profile.children.cycles-pp.new_sync_read
25.42 -0.8 24.59 perf-profile.children.cycles-pp.pipe_read
4.67 -0.8 3.86 ± 4% perf-profile.children.cycles-pp.security_file_permission
4.33 ± 2% -0.7 3.65 ± 2% perf-profile.children.cycles-pp.syscall_return_via_sysret
4.24 ± 2% -0.6 3.60 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64
2.18 ± 4% -0.5 1.64 ± 6% perf-profile.children.cycles-pp.mutex_unlock
7.92 -0.5 7.40 ± 3% perf-profile.children.cycles-pp.enqueue_task_fair
8.10 -0.5 7.61 ± 3% perf-profile.children.cycles-pp.activate_task
8.16 -0.5 7.67 ± 3% perf-profile.children.cycles-pp.ttwu_do_activate
4.12 ± 4% -0.5 3.64 ± 5% perf-profile.children.cycles-pp._raw_spin_lock
2.83 -0.4 2.38 ± 6% perf-profile.children.cycles-pp.selinux_file_permission
3.53 -0.4 3.13 ± 4% perf-profile.children.cycles-pp.enqueue_entity
1.99 ± 2% -0.4 1.64 ± 3% perf-profile.children.cycles-pp.copy_page_from_iter
3.34 -0.4 2.99 ± 4% perf-profile.children.cycles-pp.update_load_avg
2.70 -0.3 2.39 ± 3% perf-profile.children.cycles-pp.copy_page_to_iter
2.66 -0.3 2.35 ± 3% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.93 ± 3% -0.3 0.64 ± 7% perf-profile.children.cycles-pp.__mutex_unlock_slowpath
1.41 ± 4% -0.3 1.13 perf-profile.children.cycles-pp.file_has_perm
2.75 -0.3 2.49 ± 2% perf-profile.children.cycles-pp.dequeue_entity
2.29 ± 2% -0.3 2.04 perf-profile.children.cycles-pp.mutex_lock
0.85 ± 3% -0.2 0.61 ± 4% perf-profile.children.cycles-pp.__inode_security_revalidate
0.59 ± 3% -0.2 0.39 ± 6% perf-profile.children.cycles-pp.__mutex_lock
1.79 ± 2% -0.2 1.59 ± 4% perf-profile.children.cycles-pp.__fdget_pos
1.70 ± 2% -0.2 1.52 ± 3% perf-profile.children.cycles-pp.__fget_light
0.93 ± 3% -0.2 0.75 perf-profile.children.cycles-pp.avc_has_perm
1.09 ± 2% -0.2 0.92 ± 4% perf-profile.children.cycles-pp.copyin
2.33 ± 2% -0.2 2.16 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.96 ± 2% -0.2 0.81 ± 2% perf-profile.children.cycles-pp.___might_sleep
1.67 ± 2% -0.2 1.52 ± 5% perf-profile.children.cycles-pp.copyout
0.83 ± 4% -0.1 0.68 ± 2% perf-profile.children.cycles-pp._cond_resched
0.33 ± 5% -0.1 0.19 ± 10% perf-profile.children.cycles-pp.preempt_schedule_common
0.42 ± 2% -0.1 0.28 ± 8% perf-profile.children.cycles-pp.wake_up_q
0.70 ± 3% -0.1 0.59 ± 3% perf-profile.children.cycles-pp.__might_sleep
0.65 ± 5% -0.1 0.54 ± 7% perf-profile.children.cycles-pp.__fsnotify_parent
0.52 ± 2% -0.1 0.43 perf-profile.children.cycles-pp.__might_fault
0.53 ± 5% -0.1 0.45 perf-profile.children.cycles-pp.current_time
0.84 ± 3% -0.1 0.77 perf-profile.children.cycles-pp.fsnotify
0.16 ± 7% -0.1 0.09 ± 13% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
0.15 ± 8% -0.1 0.08 ± 15% perf-profile.children.cycles-pp.prepare_exit_to_usermode
0.22 ± 3% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.wake_q_add
0.49 -0.1 0.44 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
0.26 ± 4% -0.1 0.21 ± 2% perf-profile.children.cycles-pp.rcu_all_qs
0.08 ± 14% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__vfs_read
0.25 ± 4% -0.0 0.20 ± 5% perf-profile.children.cycles-pp.finish_wait
0.38 -0.0 0.34 ± 2% perf-profile.children.cycles-pp.__x64_sys_read
0.43 ± 3% -0.0 0.39 ± 2% perf-profile.children.cycles-pp.prepare_to_wait
0.14 ± 20% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.task_tick_fair
0.16 ± 19% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.scheduler_tick
0.10 ± 8% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.mutex_spin_on_owner
0.18 ± 8% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.inode_has_perm
0.21 ± 2% -0.0 0.18 ± 6% perf-profile.children.cycles-pp.deactivate_task
0.20 ± 2% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.generic_pipe_buf_confirm
0.13 ± 3% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.__sb_end_write
0.14 ± 9% -0.0 0.12 perf-profile.children.cycles-pp.timespec64_trunc
0.14 ± 3% -0.0 0.11 ± 3% perf-profile.children.cycles-pp.native_irq_return_iret
0.11 ± 4% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.smp_reschedule_interrupt
0.17 ± 4% -0.0 0.15 perf-profile.children.cycles-pp.iov_iter_init
0.11 ± 4% -0.0 0.09 ± 10% perf-profile.children.cycles-pp.native_apic_msr_eoi_write
0.09 ± 4% -0.0 0.07 perf-profile.children.cycles-pp.__x2apic_send_IPI_dest
0.15 -0.0 0.13 ± 3% perf-profile.children.cycles-pp.rb_insert_color
0.18 ± 2% -0.0 0.17 ± 2% perf-profile.children.cycles-pp.set_next_buddy
0.08 ± 5% -0.0 0.07 perf-profile.children.cycles-pp.interrupt_entry
0.23 ± 2% +0.0 0.24 perf-profile.children.cycles-pp.cpumask_next
0.67 +0.0 0.69 perf-profile.children.cycles-pp.__calc_delta
0.15 ± 3% +0.0 0.18 ± 6% perf-profile.children.cycles-pp.wakeup_preempt_entity
0.17 ± 6% +0.1 0.23 ± 3% perf-profile.children.cycles-pp.remove_entity_load_avg
0.14 ± 6% +0.1 0.19 ± 7% perf-profile.children.cycles-pp.attach_entity_load_avg
0.80 ± 3% +0.1 0.86 perf-profile.children.cycles-pp.check_preempt_wakeup
0.37 ± 3% +0.1 0.45 ± 4% perf-profile.children.cycles-pp.account_entity_enqueue
0.79 ± 4% +0.1 0.88 ± 5% perf-profile.children.cycles-pp.finish_task_switch
0.80 ± 2% +0.1 0.89 ± 2% perf-profile.children.cycles-pp.put_prev_entity
0.54 ± 2% +0.1 0.65 perf-profile.children.cycles-pp.account_entity_dequeue
1.01 ± 5% +0.1 1.13 ± 5% perf-profile.children.cycles-pp.set_task_cpu
0.49 ± 7% +0.2 0.65 ± 5% perf-profile.children.cycles-pp.migrate_task_rq_fair
1.23 ± 4% +0.2 1.40 ± 2% perf-profile.children.cycles-pp.__switch_to_asm
1.15 ± 2% +0.2 1.33 ± 2% perf-profile.children.cycles-pp.__switch_to
1.42 ± 3% +0.2 1.59 ± 2% perf-profile.children.cycles-pp.switch_fpu_return
0.76 ± 3% +0.2 0.96 ± 3% perf-profile.children.cycles-pp.pick_next_entity
2.21 +0.2 2.42 perf-profile.children.cycles-pp.reweight_entity
2.20 ± 2% +0.2 2.43 perf-profile.children.cycles-pp.load_new_mm_cr3
1.24 ± 5% +0.2 1.48 ± 4% perf-profile.children.cycles-pp.update_rq_clock
1.43 ± 5% +0.3 1.73 ± 2% perf-profile.children.cycles-pp.___perf_sw_event
0.38 ± 15% +0.3 0.68 ± 9% perf-profile.children.cycles-pp.find_next_bit
3.96 +0.3 4.28 perf-profile.children.cycles-pp.pick_next_task_fair
4.43 ± 2% +0.4 4.82 perf-profile.children.cycles-pp.switch_mm_irqs_off
1.11 +0.7 1.83 ± 4% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.46 ± 26% +0.7 1.20 ± 12% perf-profile.children.cycles-pp.cpumask_next_wrap
87.76 +1.1 88.86 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
86.98 +1.1 88.10 perf-profile.children.cycles-pp.do_syscall_64
21.71 +1.3 23.02 perf-profile.children.cycles-pp.__schedule
5.98 ± 4% +1.4 7.34 ± 3% perf-profile.children.cycles-pp.exit_to_usermode_loop
21.59 +1.4 22.99 perf-profile.children.cycles-pp.schedule
1.73 ± 29% +3.3 5.03 ± 13% perf-profile.children.cycles-pp.available_idle_cpu
35.55 +3.3 38.88 perf-profile.children.cycles-pp.ksys_write
34.31 +3.5 37.84 perf-profile.children.cycles-pp.vfs_write
31.14 +4.1 35.23 ± 2% perf-profile.children.cycles-pp.new_sync_write
30.56 +4.2 34.74 ± 2% perf-profile.children.cycles-pp.pipe_write
22.96 ± 2% +6.0 28.93 ± 3% perf-profile.children.cycles-pp.__wake_up_common_lock
20.11 ± 2% +6.1 26.19 ± 3% perf-profile.children.cycles-pp.try_to_wake_up
20.66 ± 2% +6.2 26.82 ± 3% perf-profile.children.cycles-pp.__wake_up_common
20.01 ± 2% +6.2 26.19 ± 3% perf-profile.children.cycles-pp.autoremove_wake_function
3.46 ± 20% +7.0 10.48 ± 12% perf-profile.children.cycles-pp.select_idle_sibling
4.92 ± 15% +7.1 12.01 ± 10% perf-profile.children.cycles-pp.select_task_rq_fair
11.85 ± 3% -2.1 9.72 ± 2% perf-profile.self.cycles-pp.do_syscall_64
4.13 ± 5% -1.3 2.85 ± 8% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
3.49 ± 6% -1.0 2.52 ± 9% perf-profile.self.cycles-pp.update_cfs_group
4.33 ± 2% -0.7 3.64 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret
4.24 ± 2% -0.6 3.60 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64
2.12 ± 4% -0.5 1.59 ± 6% perf-profile.self.cycles-pp.mutex_unlock
2.61 -0.3 2.31 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
1.60 ± 2% -0.3 1.30 ± 5% perf-profile.self.cycles-pp.update_load_avg
1.48 ± 3% -0.2 1.25 ± 2% perf-profile.self.cycles-pp.mutex_lock
1.95 -0.2 1.75 ± 7% perf-profile.self.cycles-pp.selinux_file_permission
1.06 ± 5% -0.2 0.86 ± 5% perf-profile.self.cycles-pp.pipe_write
1.46 ± 2% -0.2 1.28 ± 3% perf-profile.self.cycles-pp.pipe_read
1.65 ± 2% -0.2 1.49 ± 4% perf-profile.self.cycles-pp.__fget_light
0.91 ± 3% -0.2 0.74 perf-profile.self.cycles-pp.avc_has_perm
0.94 ± 2% -0.2 0.79 ± 2% perf-profile.self.cycles-pp.___might_sleep
1.84 -0.1 1.72 ± 2% perf-profile.self.cycles-pp.update_curr
1.05 -0.1 0.93 perf-profile.self.cycles-pp.enqueue_task_fair
0.63 ± 3% -0.1 0.52 ± 2% perf-profile.self.cycles-pp.__might_sleep
0.60 ± 6% -0.1 0.51 ± 7% perf-profile.self.cycles-pp.__fsnotify_parent
0.29 ± 3% -0.1 0.19 ± 5% perf-profile.self.cycles-pp.__mutex_lock
0.38 ± 4% -0.1 0.29 ± 4% perf-profile.self.cycles-pp.vfs_write
0.51 -0.1 0.43 ± 4% perf-profile.self.cycles-pp.copy_page_to_iter
0.49 -0.1 0.41 ± 2% perf-profile.self.cycles-pp.new_sync_write
0.35 ± 4% -0.1 0.28 ± 2% perf-profile.self.cycles-pp.copy_page_from_iter
0.81 ± 3% -0.1 0.75 perf-profile.self.cycles-pp.fsnotify
0.26 -0.1 0.19 ± 3% perf-profile.self.cycles-pp.__inode_security_revalidate
0.20 ± 4% -0.1 0.13 ± 6% perf-profile.self.cycles-pp.__mutex_unlock_slowpath
0.29 ± 5% -0.1 0.23 ± 3% perf-profile.self.cycles-pp.file_has_perm
0.22 -0.1 0.16 ± 8% perf-profile.self.cycles-pp.wake_q_add
0.57 ± 2% -0.1 0.52 perf-profile.self.cycles-pp.new_sync_read
0.34 ± 2% -0.1 0.29 ± 3% perf-profile.self.cycles-pp.dequeue_entity
0.26 ± 3% -0.1 0.21 ± 3% perf-profile.self.cycles-pp.ksys_read
0.28 ± 2% -0.0 0.23 perf-profile.self.cycles-pp.security_file_permission
0.22 ± 3% -0.0 0.17 ± 4% perf-profile.self.cycles-pp.finish_wait
0.25 -0.0 0.21 ± 4% perf-profile.self.cycles-pp.ksys_write
0.88 -0.0 0.83 perf-profile.self.cycles-pp.__lock_text_start
0.66 -0.0 0.61 perf-profile.self.cycles-pp.vfs_read
1.37 -0.0 1.32 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.29 ± 4% -0.0 0.25 perf-profile.self.cycles-pp.current_time
0.20 ± 3% -0.0 0.16 ± 4% perf-profile.self.cycles-pp.rcu_all_qs
0.10 ± 8% -0.0 0.06 ± 6% perf-profile.self.cycles-pp.mutex_spin_on_owner
0.42 -0.0 0.39 ± 2% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
0.34 -0.0 0.31 ± 2% perf-profile.self.cycles-pp.__x64_sys_read
0.09 ± 5% -0.0 0.05 ± 9% perf-profile.self.cycles-pp.wake_up_q
0.15 ± 8% -0.0 0.12 ± 4% perf-profile.self.cycles-pp.inode_has_perm
0.14 ± 5% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.rb_next
0.20 ± 2% -0.0 0.17 ± 4% perf-profile.self.cycles-pp.deactivate_task
0.11 ± 4% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.__fdget_pos
0.24 ± 2% -0.0 0.22 ± 3% perf-profile.self.cycles-pp.prepare_to_wait
0.14 ± 3% -0.0 0.11 ± 3% perf-profile.self.cycles-pp.native_irq_return_iret
0.13 ± 7% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.timespec64_trunc
0.06 -0.0 0.04 ± 57% perf-profile.self.cycles-pp.copyout
0.11 ± 4% -0.0 0.09 ± 10% perf-profile.self.cycles-pp.native_apic_msr_eoi_write
0.17 ± 4% -0.0 0.15 ± 2% perf-profile.self.cycles-pp.generic_pipe_buf_confirm
0.18 ± 2% -0.0 0.16 ± 4% perf-profile.self.cycles-pp.__might_fault
0.17 ± 2% -0.0 0.16 ± 2% perf-profile.self.cycles-pp.set_next_buddy
0.15 ± 2% -0.0 0.13 ± 3% perf-profile.self.cycles-pp.rb_insert_color
0.12 ± 3% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.__sb_end_write
0.10 ± 8% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
0.06 -0.0 0.05 perf-profile.self.cycles-pp.kill_fasync
0.14 ± 3% +0.0 0.16 ± 5% perf-profile.self.cycles-pp.exit_to_usermode_loop
0.66 +0.0 0.68 perf-profile.self.cycles-pp.__calc_delta
0.16 ± 5% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.native_load_tls
0.14 ± 3% +0.0 0.17 ± 4% perf-profile.self.cycles-pp.wakeup_preempt_entity
0.67 +0.0 0.70 perf-profile.self.cycles-pp.dequeue_task_fair
0.23 ± 3% +0.0 0.27 ± 3% perf-profile.self.cycles-pp._cond_resched
0.17 ± 4% +0.0 0.21 ± 6% perf-profile.self.cycles-pp.activate_task
0.35 ± 3% +0.0 0.40 perf-profile.self.cycles-pp.pick_next_entity
1.20 +0.1 1.25 perf-profile.self.cycles-pp.select_task_rq_fair
0.13 ± 6% +0.1 0.18 ± 6% perf-profile.self.cycles-pp.attach_entity_load_avg
0.87 ± 3% +0.1 0.96 ± 3% perf-profile.self.cycles-pp.pick_next_task_fair
0.27 ± 6% +0.1 0.36 ± 7% perf-profile.self.cycles-pp.migrate_task_rq_fair
0.32 ± 2% +0.1 0.41 ± 4% perf-profile.self.cycles-pp.account_entity_enqueue
0.42 +0.1 0.56 ± 2% perf-profile.self.cycles-pp.account_entity_dequeue
2.24 ± 3% +0.1 2.39 ± 2% perf-profile.self.cycles-pp.switch_mm_irqs_off
1.09 ± 2% +0.2 1.26 ± 2% perf-profile.self.cycles-pp.__switch_to
1.20 ± 4% +0.2 1.37 ± 2% perf-profile.self.cycles-pp.__switch_to_asm
1.40 ± 3% +0.2 1.58 ± 2% perf-profile.self.cycles-pp.switch_fpu_return
0.89 ± 6% +0.2 1.10 ± 5% perf-profile.self.cycles-pp.update_rq_clock
2.19 ± 2% +0.2 2.42 perf-profile.self.cycles-pp.load_new_mm_cr3
1.33 ± 4% +0.3 1.60 ± 3% perf-profile.self.cycles-pp.___perf_sw_event
0.34 ± 14% +0.3 0.66 ± 9% perf-profile.self.cycles-pp.find_next_bit
0.28 ± 23% +0.4 0.72 ± 12% perf-profile.self.cycles-pp.cpumask_next_wrap
0.89 +0.7 1.58 ± 4% perf-profile.self.cycles-pp._raw_spin_lock
1.04 +0.7 1.76 ± 4% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.65 ± 12% +3.0 3.61 ± 13% perf-profile.self.cycles-pp.select_idle_sibling
1.71 ± 29% +3.3 4.97 ± 13% perf-profile.self.cycles-pp.available_idle_cpu
***************************************************************************************************
lkp-cfl-e1: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
=========================================================================================
compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
gcc-7/performance/pipe/x86_64-rhel-7.6/threads/100%/debian-x86_64-2018-04-03.cgz/lkp-cfl-e1/hackbench/0xb8
commit:
43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq")
3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found")
43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
1:2 -50% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
1:2 -50% :4 dmesg.WARNING:stack_recursion
%stddev %change %stddev
\ | \
49845 -6.8% 46466 hackbench.throughput
8.776e+08 +13.0% 9.915e+08 hackbench.time.involuntary_context_switches
93623 -6.3% 87707 hackbench.time.minor_page_faults
740.08 -1.4% 729.54 hackbench.time.user_time
1.569e+09 +3.5% 1.623e+09 hackbench.time.voluntary_context_switches
3.072e+08 -6.2% 2.88e+08 hackbench.workload
21.45 -1.7% 21.09 boot-time.boot
1747212 ± 3% -31.0% 1205072 ± 14% cpuidle.POLL.time
1289000 ± 4% -36.6% 817407 ± 13% cpuidle.POLL.usage
4038486 +6.3% 4293331 vmstat.system.cs
394711 -10.2% 354353 ± 2% vmstat.system.in
10059 -15.4% 8506 ± 5% softirqs.CPU0.SCHED
309881 -27.3% 225159 ± 4% softirqs.CPU2.TIMER
220105 +20.8% 265807 ± 16% softirqs.CPU6.TIMER
120937 ± 3% -17.6% 99667 ± 2% softirqs.SCHED
18883 -8.9% 17200 ± 3% slabinfo.kmalloc-32.active_objs
18883 -8.9% 17200 ± 3% slabinfo.kmalloc-32.num_objs
8573 ± 2% -7.6% 7924 slabinfo.lsm_file_cache.active_objs
8573 ± 2% -7.6% 7924 slabinfo.lsm_file_cache.num_objs
5280 -8.8% 4814 slabinfo.task_delay_info.active_objs
5280 -8.8% 4814 slabinfo.task_delay_info.num_objs
65295 -1.9% 64034 proc-vmstat.nr_active_anon
60605 -1.5% 59671 proc-vmstat.nr_anon_pages
12935 -0.9% 12820 proc-vmstat.nr_slab_unreclaimable
65295 -1.9% 64034 proc-vmstat.nr_zone_active_anon
24774010 -16.6% 20673098 proc-vmstat.numa_hit
24774010 -16.6% 20673098 proc-vmstat.numa_local
24839243 -16.5% 20735391 proc-vmstat.pgalloc_normal
673024 -2.4% 657142 proc-vmstat.pgfault
24822112 -16.5% 20717243 proc-vmstat.pgfree
247.95 ± 14% -19.1% 200.59 ± 4% sched_debug.cfs_rq:/.load_avg.max
43.55 ± 3% -19.9% 34.86 ± 6% sched_debug.cfs_rq:/.load_avg.min
55.51 ± 6% -13.2% 48.19 ± 7% sched_debug.cfs_rq:/.load_avg.stddev
20116 ± 8% -12.8% 17531 ± 5% sched_debug.cfs_rq:/.runnable_weight.min
1288 +7.5% 1384 ± 3% sched_debug.cfs_rq:/.util_avg.max
54763 ± 9% +130.8% 126390 ± 25% sched_debug.cpu.avg_idle.avg
301502 ± 18% +84.9% 557406 ± 14% sched_debug.cpu.avg_idle.max
82233 ± 22% +122.7% 183153 ± 17% sched_debug.cpu.avg_idle.stddev
0.00 ± 12% -12.0% 0.00 ± 8% sched_debug.cpu.next_balance.stddev
16.92 ± 4% -7.5% 15.66 ± 3% sched_debug.cpu.nr_running.avg
2.92 ± 10% -20.9% 2.31 ± 16% sched_debug.cpu.nr_uninterruptible.avg
2977 ± 50% -73.0% 803.75 ± 43% interrupts.133:IR-PCI-MSI.2097154-edge.eth1-TxRx-1
546.00 ± 23% +119.5% 1198 ± 16% interrupts.135:IR-PCI-MSI.2097156-edge.eth1-TxRx-3
13831842 -11.5% 12245495 ± 2% interrupts.CPU0.RES:Rescheduling_interrupts
45259 ± 5% +68.9% 76440 ± 15% interrupts.CPU0.TRM:Thermal_event_interrupts
2977 ± 50% -73.0% 803.75 ± 43% interrupts.CPU1.133:IR-PCI-MSI.2097154-edge.eth1-TxRx-1
45283 ± 5% +68.9% 76474 ± 15% interrupts.CPU1.TRM:Thermal_event_interrupts
13947887 -11.0% 12413074 interrupts.CPU10.RES:Rescheduling_interrupts
45275 ± 5% +68.9% 76468 ± 15% interrupts.CPU10.TRM:Thermal_event_interrupts
45286 ± 5% +68.9% 76468 ± 15% interrupts.CPU11.TRM:Thermal_event_interrupts
13579741 -11.8% 11974820 ± 5% interrupts.CPU12.RES:Rescheduling_interrupts
45286 ± 5% +68.9% 76471 ± 15% interrupts.CPU12.TRM:Thermal_event_interrupts
13668073 -11.3% 12119346 interrupts.CPU13.RES:Rescheduling_interrupts
45286 ± 5% +68.9% 76474 ± 15% interrupts.CPU13.TRM:Thermal_event_interrupts
13578574 -7.8% 12518854 interrupts.CPU14.RES:Rescheduling_interrupts
45286 ± 5% +68.9% 76467 ± 15% interrupts.CPU14.TRM:Thermal_event_interrupts
13689117 -10.1% 12310290 ± 2% interrupts.CPU15.RES:Rescheduling_interrupts
45286 ± 5% +68.9% 76473 ± 15% interrupts.CPU15.TRM:Thermal_event_interrupts
13924470 -10.5% 12461613 interrupts.CPU2.RES:Rescheduling_interrupts
45264 ± 5% +68.9% 76467 ± 15% interrupts.CPU2.TRM:Thermal_event_interrupts
546.00 ± 23% +119.5% 1198 ± 16% interrupts.CPU3.135:IR-PCI-MSI.2097156-edge.eth1-TxRx-3
13951884 -10.5% 12480739 interrupts.CPU3.RES:Rescheduling_interrupts
45286 ± 5% +68.9% 76469 ± 15% interrupts.CPU3.TRM:Thermal_event_interrupts
45285 ± 5% +68.9% 76471 ± 15% interrupts.CPU4.TRM:Thermal_event_interrupts
14030690 -10.7% 12530992 ± 3% interrupts.CPU5.RES:Rescheduling_interrupts
45286 ± 5% +68.9% 76471 ± 15% interrupts.CPU5.TRM:Thermal_event_interrupts
45285 ± 5% +68.9% 76474 ± 15% interrupts.CPU6.TRM:Thermal_event_interrupts
45285 ± 5% +68.9% 76472 ± 15% interrupts.CPU7.TRM:Thermal_event_interrupts
13319036 -10.2% 11966842 ± 2% interrupts.CPU8.RES:Rescheduling_interrupts
45280 ± 5% +68.7% 76405 ± 15% interrupts.CPU8.TRM:Thermal_event_interrupts
13401328 -11.8% 11816711 ± 3% interrupts.CPU9.RES:Rescheduling_interrupts
45279 ± 5% +68.9% 76467 ± 15% interrupts.CPU9.TRM:Thermal_event_interrupts
2.19e+08 -9.8% 1.974e+08 ± 2% interrupts.RES:Rescheduling_interrupts
724498 ± 5% +68.9% 1223435 ± 15% interrupts.TRM:Thermal_event_interrupts
53.44 +4.2% 55.68 perf-stat.i.MPKI
0.12 -0.0 0.10 ± 4% perf-stat.i.cache-miss-rate%
1764490 -13.1% 1533362 ± 3% perf-stat.i.cache-misses
1.767e+09 +3.6% 1.832e+09 perf-stat.i.cache-references
4055785 +6.2% 4306694 perf-stat.i.context-switches
336154 +21.5% 408363 perf-stat.i.cpu-migrations
34528 ± 5% +10.3% 38089 ± 3% perf-stat.i.cycles-between-cache-misses
0.01 ± 6% -0.0 0.01 ± 6% perf-stat.i.dTLB-load-miss-rate%
899577 ± 6% -15.2% 762553 ± 5% perf-stat.i.dTLB-load-misses
0.00 ± 7% -0.0 0.00 ± 4% perf-stat.i.dTLB-store-miss-rate%
22816 ± 7% -13.2% 19813 ± 4% perf-stat.i.dTLB-store-misses
6.044e+09 -1.2% 5.969e+09 perf-stat.i.dTLB-stores
31331199 -3.4% 30272777 perf-stat.i.iTLB-load-misses
1061 +3.1% 1094 perf-stat.i.instructions-per-iTLB-miss
1077 -2.9% 1046 perf-stat.i.minor-faults
73018 -10.9% 65028 ± 2% perf-stat.i.node-loads
122632 ± 2% -9.3% 111181 ± 4% perf-stat.i.node-stores
1077 -2.9% 1046 perf-stat.i.page-faults
53.29 +4.2% 55.53 perf-stat.overall.MPKI
0.10 -0.0 0.08 ± 3% perf-stat.overall.cache-miss-rate%
25264 +15.3% 29120 ± 3% perf-stat.overall.cycles-between-cache-misses
0.01 ± 6% -0.0 0.01 ± 6% perf-stat.overall.dTLB-load-miss-rate%
0.00 ± 7% -0.0 0.00 ± 4% perf-stat.overall.dTLB-store-miss-rate%
1058 +2.9% 1089 perf-stat.overall.instructions-per-iTLB-miss
65158 +6.6% 69487 perf-stat.overall.path-length
1761814 -13.1% 1530883 ± 3% perf-stat.ps.cache-misses
1.765e+09 +3.7% 1.829e+09 perf-stat.ps.cache-references
4048770 +6.2% 4299578 perf-stat.ps.context-switches
335565 +21.5% 407697 perf-stat.ps.cpu-migrations
898100 ± 6% -15.2% 761293 ± 5% perf-stat.ps.dTLB-load-misses
22777 ± 7% -13.2% 19781 ± 4% perf-stat.ps.dTLB-store-misses
6.034e+09 -1.2% 5.96e+09 perf-stat.ps.dTLB-stores
31277565 -3.4% 30222833 perf-stat.ps.iTLB-load-misses
1075 -2.9% 1044 perf-stat.ps.minor-faults
72896 -10.9% 64922 ± 2% perf-stat.ps.node-loads
122424 ± 2% -9.3% 111005 ± 4% perf-stat.ps.node-stores
1075 -2.9% 1044 perf-stat.ps.page-faults
39.37 ±100% -36.4 2.98 ±173% perf-profile.calltrace.cycles-pp.start_thread
21.68 ±100% -20.0 1.64 ±173% perf-profile.calltrace.cycles-pp.__GI___libc_write.start_thread
20.14 ±100% -18.6 1.52 ±173% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread
20.00 ±100% -18.5 1.51 ±173% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread
18.05 ±100% -16.7 1.39 ±173% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread
16.75 ±100% -15.5 1.22 ±173% perf-profile.calltrace.cycles-pp.__GI___libc_read.start_thread
16.70 ±100% -15.4 1.26 ±173% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
14.98 ±100% -13.9 1.09 ±173% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread
14.82 ±100% -13.7 1.08 ±173% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread
13.95 ±100% -12.9 1.03 ±173% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread
12.97 ±100% -12.0 0.95 ±173% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
3.69 ± 6% -1.1 2.58 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common
4.25 ± 6% -1.0 3.29 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
1.54 ± 10% -0.5 1.05 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
1.07 ± 16% -0.5 0.60 ± 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write
1.14 ± 2% -0.3 0.80 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__schedule.schedule.pipe_wait
0.55 ± 4% -0.3 0.26 ±100% perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_read.new_sync_read.vfs_read.ksys_read
1.90 ± 9% -0.2 1.67 ± 7% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.71 ± 7% -0.2 1.48 ± 7% perf-profile.calltrace.cycles-pp.__fget.__fget_light.__fdget_pos.ksys_write.do_syscall_64
0.68 ± 19% +0.2 0.86 perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
0.67 ± 16% +0.2 0.88 ± 4% perf-profile.calltrace.cycles-pp.update_cfs_group.dequeue_task_fair.__schedule.schedule.pipe_wait
0.29 ±100% +0.4 0.64 ± 4% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read
0.31 ±100% +0.4 0.67 ± 4% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_wait.pipe_read.new_sync_read.vfs_read
0.32 ±100% +0.4 0.68 ± 2% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait
0.29 ±100% +0.4 0.68 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop.do_syscall_64
0.35 ±100% +0.4 0.78 perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule
1.23 ± 19% +0.4 1.68 ± 2% perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait
1.29 ± 18% +0.5 1.76 ± 5% perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
0.00 +0.5 0.53 ± 2% perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__schedule.schedule
0.00 +0.6 0.59 ± 7% perf-profile.calltrace.cycles-pp.__enqueue_entity.put_prev_entity.pick_next_task_fair.__schedule.schedule
8.82 +0.6 9.46 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
0.47 ±100% +0.7 1.14 ± 6% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop
2.21 ± 10% +0.7 2.88 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64
9.10 +0.7 9.79 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
9.04 +0.7 9.74 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
0.71 ±100% +0.9 1.61 ± 3% perf-profile.calltrace.cycles-pp.__switch_to
0.68 ± 99% +0.9 1.60 ± 3% perf-profile.calltrace.cycles-pp.native_write_msr
0.86 ±100% +0.9 1.79 ± 2% perf-profile.calltrace.cycles-pp.__switch_to_asm
6.54 +1.0 7.53 ± 2% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_wait.pipe_read
26.96 +1.0 27.95 perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
1.42 ± 63% +1.0 2.47 ± 3% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.54 ± 8% +1.2 5.79 ± 4% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
14.99 +1.3 16.27 perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait.pipe_read.new_sync_read
15.35 +1.3 16.66 perf-profile.calltrace.cycles-pp.schedule.pipe_wait.pipe_read.new_sync_read.vfs_read
27.37 +1.3 28.70 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
17.81 +1.6 19.38 perf-profile.calltrace.cycles-pp.pipe_wait.pipe_read.new_sync_read.vfs_read.ksys_read
3.95 ± 11% +1.8 5.70 ± 6% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
23.38 +2.0 25.33 ± 2% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
24.30 +2.0 26.27 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
23.73 +2.0 25.71 ± 2% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
5.24 ± 10% +2.0 7.25 ± 6% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
3.23 ± 57% +2.7 5.98 ± 4% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.34 ± 57% +2.8 6.18 ± 4% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
39.37 ±100% -36.4 2.98 ±173% perf-profile.children.cycles-pp.start_thread
21.68 ±100% -20.0 1.67 ±173% perf-profile.children.cycles-pp.__GI___libc_write
16.99 ±100% -15.7 1.24 ±173% perf-profile.children.cycles-pp.__GI___libc_read
6.84 ± 10% -2.1 4.71 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
45.35 -1.0 44.33 perf-profile.children.cycles-pp.ksys_write
6.70 ± 5% -1.0 5.70 ± 2% perf-profile.children.cycles-pp._raw_spin_lock
5.90 ± 4% -0.6 5.34 ± 6% perf-profile.children.cycles-pp.security_file_permission
3.35 ± 9% -0.6 2.80 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
42.14 -0.5 41.63 perf-profile.children.cycles-pp.vfs_write
3.41 -0.4 3.01 perf-profile.children.cycles-pp.__fdget_pos
3.30 -0.4 2.91 perf-profile.children.cycles-pp.__fget_light
2.71 ± 2% -0.4 2.32 perf-profile.children.cycles-pp.__fget
3.23 -0.3 2.93 ± 3% perf-profile.children.cycles-pp.mutex_lock
2.76 -0.3 2.47 ± 5% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
2.21 ± 3% -0.3 1.93 ± 10% perf-profile.children.cycles-pp.file_has_perm
3.14 -0.2 2.90 ± 4% perf-profile.children.cycles-pp.copy_page_to_iter
3.03 ± 5% -0.2 2.79 ± 5% perf-profile.children.cycles-pp.selinux_file_permission
2.10 ± 5% -0.2 1.86 ± 2% perf-profile.children.cycles-pp.mutex_unlock
1.50 ± 5% -0.2 1.27 ± 2% perf-profile.children.cycles-pp.fput_many
0.68 ± 17% -0.2 0.48 ± 4% perf-profile.children.cycles-pp.__mutex_lock
1.66 ± 2% -0.2 1.47 ± 4% perf-profile.children.cycles-pp.copyout
1.73 -0.2 1.57 ± 2% perf-profile.children.cycles-pp.update_rq_clock
1.30 -0.1 1.17 ± 6% perf-profile.children.cycles-pp.copyin
0.27 ± 11% -0.1 0.15 ± 9% perf-profile.children.cycles-pp.preempt_schedule_common
1.25 ± 3% -0.1 1.15 ± 3% perf-profile.children.cycles-pp.fsnotify
0.79 -0.1 0.69 ± 7% perf-profile.children.cycles-pp.reschedule_interrupt
0.91 -0.1 0.82 ± 5% perf-profile.children.cycles-pp._cond_resched
0.80 ± 2% -0.1 0.71 ± 3% perf-profile.children.cycles-pp.file_update_time
0.29 ± 10% -0.1 0.24 ± 5% perf-profile.children.cycles-pp.rb_insert_color
0.26 -0.0 0.21 ± 2% perf-profile.children.cycles-pp.native_irq_return_iret
0.07 ± 14% -0.0 0.03 ±100% perf-profile.children.cycles-pp.scheduler_ipi
0.22 ± 6% -0.0 0.18 ± 4% perf-profile.children.cycles-pp.__x64_sys_read
0.25 ± 4% -0.0 0.21 ± 2% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
0.27 -0.0 0.23 ± 9% perf-profile.children.cycles-pp.timespec64_trunc
0.23 ± 2% -0.0 0.20 ± 5% perf-profile.children.cycles-pp.smp_reschedule_interrupt
0.22 -0.0 0.20 ± 5% perf-profile.children.cycles-pp.__x64_sys_write
0.24 -0.0 0.22 ± 3% perf-profile.children.cycles-pp.__list_add_valid
0.14 -0.0 0.12 ± 10% perf-profile.children.cycles-pp.kill_fasync
0.11 ± 4% +0.0 0.12 ± 3% perf-profile.children.cycles-pp.perf_exclude_event
0.10 +0.0 0.12 ± 8% perf-profile.children.cycles-pp.is_cpu_allowed
0.09 ± 11% +0.0 0.12 ± 7% perf-profile.children.cycles-pp.rcu_note_context_switch
0.07 +0.0 0.10 ± 10% perf-profile.children.cycles-pp.pkg_thermal_notify
0.17 ± 3% +0.0 0.20 ± 9% perf-profile.children.cycles-pp.deactivate_task
0.16 +0.0 0.21 ± 10% perf-profile.children.cycles-pp.update_cfs_rq_h_load
0.08 ± 6% +0.0 0.12 ± 10% perf-profile.children.cycles-pp.intel_thermal_interrupt
0.08 ± 6% +0.0 0.12 ± 6% perf-profile.children.cycles-pp.smp_thermal_interrupt
0.08 +0.1 0.13 ± 9% perf-profile.children.cycles-pp.thermal_interrupt
0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.default_wake_function
0.19 ± 5% +0.1 0.25 ± 5% perf-profile.children.cycles-pp.wakeup_preempt_entity
0.19 ± 5% +0.1 0.27 ± 4% perf-profile.children.cycles-pp.finish_wait
0.89 ± 8% +0.1 0.98 perf-profile.children.cycles-pp.__calc_delta
0.63 ± 4% +0.1 0.74 ± 4% perf-profile.children.cycles-pp.native_sched_clock
0.33 ± 4% +0.1 0.44 ± 5% perf-profile.children.cycles-pp.set_next_buddy
1.23 ± 3% +0.1 1.34 ± 3% perf-profile.children.cycles-pp.check_preempt_wakeup
0.69 ± 4% +0.1 0.81 ± 3% perf-profile.children.cycles-pp.sched_clock
1.52 ± 3% +0.1 1.65 ± 2% perf-profile.children.cycles-pp.set_next_entity
0.73 ± 4% +0.1 0.88 ± 3% perf-profile.children.cycles-pp.sched_clock_cpu
0.57 ± 4% +0.2 0.74 ± 4% perf-profile.children.cycles-pp.account_entity_dequeue
1.21 ± 4% +0.2 1.39 ± 2% perf-profile.children.cycles-pp.__update_load_avg_se
1.99 ± 3% +0.2 2.20 ± 3% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
1.64 ± 6% +0.2 1.85 perf-profile.children.cycles-pp.__switch_to_asm
1.60 ± 4% +0.2 1.85 ± 3% perf-profile.children.cycles-pp.update_cfs_group
3.91 +0.2 4.15 perf-profile.children.cycles-pp.update_curr
1.04 ± 5% +0.3 1.30 ± 4% perf-profile.children.cycles-pp.___perf_sw_event
1.05 ± 9% +0.3 1.33 ± 2% perf-profile.children.cycles-pp.pick_next_entity
1.57 ± 3% +0.3 1.89 perf-profile.children.cycles-pp.native_write_msr
1.01 ± 9% +0.3 1.33 ± 5% perf-profile.children.cycles-pp.put_prev_entity
2.17 ± 7% +0.3 2.51 ± 2% perf-profile.children.cycles-pp.switch_fpu_return
1.93 ± 7% +0.4 2.36 perf-profile.children.cycles-pp.__switch_to
2.99 ± 4% +0.6 3.57 ± 2% perf-profile.children.cycles-pp.reweight_entity
8.89 +0.6 9.50 ± 2% perf-profile.children.cycles-pp.enqueue_task_fair
9.16 +0.7 9.82 perf-profile.children.cycles-pp.ttwu_do_activate
9.09 +0.7 9.76 ± 2% perf-profile.children.cycles-pp.activate_task
5.18 ± 6% +0.9 6.04 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair
6.71 +0.9 7.63 ± 2% perf-profile.children.cycles-pp.dequeue_task_fair
28.01 +1.4 29.36 ± 2% perf-profile.children.cycles-pp.__wake_up_common_lock
4.91 ± 7% +1.4 6.28 ± 4% perf-profile.children.cycles-pp.exit_to_usermode_loop
18.19 +1.4 19.56 perf-profile.children.cycles-pp.pipe_wait
4.03 ± 11% +1.7 5.75 ± 6% perf-profile.children.cycles-pp.select_idle_sibling
23.86 ± 2% +1.8 25.71 ± 2% perf-profile.children.cycles-pp.try_to_wake_up
24.44 +1.9 26.36 ± 2% perf-profile.children.cycles-pp.__wake_up_common
23.84 +1.9 25.77 ± 2% perf-profile.children.cycles-pp.autoremove_wake_function
5.29 ± 10% +2.0 7.29 ± 6% perf-profile.children.cycles-pp.select_task_rq_fair
20.27 ± 2% +2.3 22.53 ± 2% perf-profile.children.cycles-pp.__schedule
20.36 ± 2% +2.5 22.82 ± 2% perf-profile.children.cycles-pp.schedule
6.82 ± 10% -2.1 4.69 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
2.67 ± 2% -0.4 2.29 perf-profile.self.cycles-pp.__fget
2.72 -0.3 2.44 ± 5% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
1.22 -0.2 0.98 ± 2% perf-profile.self.cycles-pp.update_rq_clock
1.36 ± 3% -0.2 1.13 ± 6% perf-profile.self.cycles-pp.pipe_write
1.48 ± 5% -0.2 1.24 ± 3% perf-profile.self.cycles-pp.fput_many
2.04 ± 5% -0.2 1.81 ± 2% perf-profile.self.cycles-pp.mutex_unlock
1.96 -0.2 1.73 ± 2% perf-profile.self.cycles-pp.mutex_lock
0.66 ± 12% -0.1 0.53 ± 10% perf-profile.self.cycles-pp.file_has_perm
0.38 ± 10% -0.1 0.28 ± 9% perf-profile.self.cycles-pp.ksys_read
0.34 ± 7% -0.1 0.25 ± 7% perf-profile.self.cycles-pp.__mutex_lock
1.21 ± 3% -0.1 1.11 ± 2% perf-profile.self.cycles-pp.fsnotify
1.00 -0.1 0.94 ± 5% perf-profile.self.cycles-pp.__might_sleep
0.60 ± 4% -0.1 0.55 ± 8% perf-profile.self.cycles-pp.vfs_write
0.29 ± 10% -0.1 0.23 ± 3% perf-profile.self.cycles-pp.rb_insert_color
0.32 ± 6% -0.0 0.28 ± 6% perf-profile.self.cycles-pp.ksys_write
0.26 -0.0 0.21 ± 2% perf-profile.self.cycles-pp.native_irq_return_iret
0.07 ± 14% -0.0 0.03 ±100% perf-profile.self.cycles-pp.scheduler_ipi
0.36 ± 5% -0.0 0.32 ± 8% perf-profile.self.cycles-pp.__sb_start_write
0.29 -0.0 0.24 ± 4% perf-profile.self.cycles-pp.check_preempt_curr
0.22 ± 6% -0.0 0.17 ± 6% perf-profile.self.cycles-pp.__x64_sys_read
0.25 -0.0 0.21 ± 8% perf-profile.self.cycles-pp.timespec64_trunc
0.22 ± 4% -0.0 0.19 ± 11% perf-profile.self.cycles-pp.wake_q_add
0.21 ± 2% -0.0 0.18 ± 6% perf-profile.self.cycles-pp.__x64_sys_write
0.21 ± 2% -0.0 0.20 perf-profile.self.cycles-pp.__list_add_valid
0.11 ± 4% -0.0 0.10 ± 7% perf-profile.self.cycles-pp.kill_fasync
0.06 ± 9% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.rcu_note_context_switch
0.15 +0.0 0.18 ± 6% perf-profile.self.cycles-pp.autoremove_wake_function
0.16 +0.0 0.19 ± 9% perf-profile.self.cycles-pp.deactivate_task
0.26 +0.0 0.29 ± 5% perf-profile.self.cycles-pp.rcu_all_qs
0.33 ± 3% +0.0 0.37 ± 4% perf-profile.self.cycles-pp.set_next_entity
0.18 ± 5% +0.0 0.22 ± 6% perf-profile.self.cycles-pp.wakeup_preempt_entity
0.03 ±100% +0.0 0.07 perf-profile.self.cycles-pp.check_cfs_rq_runtime
0.16 ± 6% +0.0 0.20 ± 6% perf-profile.self.cycles-pp.exit_to_usermode_loop
0.16 +0.0 0.20 ± 12% perf-profile.self.cycles-pp.update_cfs_rq_h_load
0.20 ± 2% +0.1 0.25 ± 2% perf-profile.self.cycles-pp.activate_task
0.41 ± 6% +0.1 0.46 perf-profile.self.cycles-pp.schedule
1.96 +0.1 2.02 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.15 ± 6% +0.1 0.22 perf-profile.self.cycles-pp.finish_wait
0.41 ± 4% +0.1 0.48 ± 6% perf-profile.self.cycles-pp.account_entity_enqueue
0.61 ± 4% +0.1 0.71 ± 3% perf-profile.self.cycles-pp.native_sched_clock
0.83 ± 2% +0.1 0.93 perf-profile.self.cycles-pp.dequeue_task_fair
0.29 ± 3% +0.1 0.40 ± 5% perf-profile.self.cycles-pp.set_next_buddy
1.02 +0.1 1.15 ± 7% perf-profile.self.cycles-pp.enqueue_task_fair
0.46 ± 12% +0.2 0.61 ± 3% perf-profile.self.cycles-pp.pick_next_entity
1.17 ± 4% +0.2 1.35 ± 2% perf-profile.self.cycles-pp.__update_load_avg_se
0.42 ± 8% +0.2 0.61 ± 5% perf-profile.self.cycles-pp.account_entity_dequeue
1.64 ± 6% +0.2 1.85 perf-profile.self.cycles-pp.__switch_to_asm
1.23 ± 4% +0.2 1.46 ± 3% perf-profile.self.cycles-pp.reweight_entity
1.07 ± 5% +0.2 1.31 ± 4% perf-profile.self.cycles-pp.select_task_rq_fair
0.90 ± 5% +0.2 1.14 ± 5% perf-profile.self.cycles-pp.___perf_sw_event
1.57 ± 4% +0.2 1.82 ± 2% perf-profile.self.cycles-pp.update_cfs_group
1.17 ± 3% +0.3 1.42 ± 7% perf-profile.self.cycles-pp.update_load_avg
1.56 ± 4% +0.3 1.88 perf-profile.self.cycles-pp.native_write_msr
2.16 ± 7% +0.3 2.50 ± 2% perf-profile.self.cycles-pp.switch_fpu_return
1.84 ± 8% +0.4 2.25 ± 2% perf-profile.self.cycles-pp.__switch_to
1.22 +0.5 1.73 ± 4% perf-profile.self.cycles-pp._raw_spin_lock
0.92 ± 11% +1.3 2.27 ± 6% perf-profile.self.cycles-pp.select_idle_sibling
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
4 months
- 2
- 1
[sched/numa] f6183ef98b: phoronix-test-suite.aom-av1.0.frames_per_second -25.0% regression
by kernel test robot
Greeting,
FYI, we noticed a -25.0% regression of phoronix-test-suite.aom-av1.0.frames_per_second due to commit:
commit: f6183ef98bba39a5de563d4ab4bc889ff2b600d7 ("sched/numa: replace runnable_load_avg by load_avg")
https://git.kernel.org/cgit/linux/kernel/git/mel/linux.git sched-lbnuma-rewrite-v2r6
in testcase: phoronix-test-suite
on test machine: 16 threads Intel(R) Xeon(R) CPU X5570 @ 2.93GHz with 48G memory
with following parameters:
test: aom-av1-1.2.0
cpufreq_governor: performance
ucode: 0x1d
test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added.
test-url: http://www.phoronix-test-suite.com/
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen(a)intel.com>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode:
gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-phoronix/lkp-nhm-2ep1/aom-av1-1.2.0/phoronix-test-suite/0x1d
commit:
91d7637912 ("sched/fair: reorder enqueue/dequeue_task_fair path")
f6183ef98b ("sched/numa: replace runnable_load_avg by load_avg")
91d76379120e4640 f6183ef98bba39a5de563d4ab4b
---------------- ---------------------------
%stddev %change %stddev
\ | \
0.04 -25.0% 0.03 phoronix-test-suite.aom-av1.0.frames_per_second
1648 +24.6% 2055 phoronix-test-suite.time.elapsed_time
1648 +24.6% 2055 phoronix-test-suite.time.elapsed_time.max
1935 -19.6% 1556 meminfo.max_used_kB
1.518e+10 ± 29% +39.9% 2.124e+10 cpuidle.C1E.time
33131701 ± 34% +35.9% 45011103 ± 2% cpuidle.C1E.usage
672.00 -100.0% 0.00 slabinfo.dmaengine-unmap-16.active_objs
672.00 -100.0% 0.00 slabinfo.dmaengine-unmap-16.num_objs
67.50 -2.2% 66.00 vmstat.cpu.id
31.00 +6.5% 33.00 vmstat.cpu.us
467.75 +5.0% 491.00 proc-vmstat.nr_mlock
484.75 +4.8% 508.00 proc-vmstat.nr_unevictable
484.75 +4.8% 508.00 proc-vmstat.nr_zone_unevictable
3492646 ± 7% +67.4% 5846384 ± 3% proc-vmstat.numa_hint_faults
2478506 ± 6% +127.8% 5646434 ± 4% proc-vmstat.numa_hint_faults_local
3650509 +13.5% 4141674 proc-vmstat.numa_hit
76734 +98.1% 151980 ± 2% proc-vmstat.numa_huge_pte_updates
3647139 +13.5% 4138249 proc-vmstat.numa_local
3408776 ± 7% -80.4% 667824 ± 8% proc-vmstat.numa_pages_migrated
42828088 +95.3% 83628988 ± 2% proc-vmstat.numa_pte_updates
7766914 ± 3% -25.6% 5776215 proc-vmstat.pgalloc_normal
7771023 ± 3% +43.5% 11147904 ± 2% proc-vmstat.pgfault
7716195 ± 3% -25.8% 5726088 proc-vmstat.pgfree
1152 ± 19% -88.9% 128.00 ±173% proc-vmstat.pgmigrate_fail
3408776 ± 7% -80.4% 667824 ± 8% proc-vmstat.pgmigrate_success
0.13 ±173% +0.5 0.63 ± 7% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
0.14 ±173% +0.5 0.67 ± 8% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
4.40 ± 47% +3.5 7.87 ± 16% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel
4.40 ± 47% +3.5 7.87 ± 16% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
4.41 ± 47% +3.5 7.94 ± 16% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64
4.41 ± 47% +3.5 7.94 ± 16% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
4.41 ± 47% +3.5 7.94 ± 16% perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64
0.05 ± 67% +0.1 0.12 ± 17% perf-profile.children.cycles-pp.ktime_get
0.01 ±173% +0.1 0.10 ± 27% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.00 +0.1 0.10 ± 36% perf-profile.children.cycles-pp.rcu_core
0.32 ± 86% +0.5 0.85 ± 54% perf-profile.children.cycles-pp.__libc_start_main
0.60 ± 46% +0.6 1.19 ± 41% perf-profile.children.cycles-pp.do_syscall_64
0.60 ± 46% +0.6 1.19 ± 41% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
0.34 ± 82% +0.6 0.94 ± 51% perf-profile.children.cycles-pp.vfs_read
0.34 ± 82% +0.6 0.94 ± 51% perf-profile.children.cycles-pp.ksys_read
0.29 ± 96% +0.6 0.90 ± 53% perf-profile.children.cycles-pp.perf_event_read
0.29 ± 96% +0.6 0.90 ± 53% perf-profile.children.cycles-pp.smp_call_function_single
0.30 ± 96% +0.6 0.91 ± 53% perf-profile.children.cycles-pp.__libc_read
0.29 ± 96% +0.6 0.91 ± 52% perf-profile.children.cycles-pp.perf_read
4.41 ± 47% +3.5 7.94 ± 16% perf-profile.children.cycles-pp.start_kernel
0.01 ±173% +0.1 0.07 ± 19% perf-profile.self.cycles-pp.cpuidle_enter_state
0.29 ± 98% +0.6 0.90 ± 53% perf-profile.self.cycles-pp.smp_call_function_single
161465 ± 7% +66.5% 268796 ± 16% softirqs.CPU0.SCHED
154149 ± 4% +57.3% 242412 ± 5% softirqs.CPU1.SCHED
217783 ± 10% +17.4% 255655 ± 3% softirqs.CPU10.RCU
685538 ± 6% +16.3% 797133 ± 3% softirqs.CPU10.TIMER
647482 +29.6% 839256 ± 14% softirqs.CPU11.TIMER
215207 ± 9% +19.8% 257893 ± 5% softirqs.CPU12.RCU
676241 ± 4% +20.5% 814890 softirqs.CPU12.TIMER
218513 ± 8% +17.8% 257505 ± 4% softirqs.CPU14.RCU
665815 ± 5% +20.2% 800244 ± 3% softirqs.CPU14.TIMER
217764 ± 10% +16.0% 252579 ± 2% softirqs.CPU15.RCU
671996 ± 2% +18.0% 792743 ± 10% softirqs.CPU15.TIMER
216810 ± 10% +16.2% 251864 ± 3% softirqs.CPU2.RCU
168050 ± 8% +53.2% 257377 ± 4% softirqs.CPU2.SCHED
687608 ± 7% +13.9% 783359 ± 5% softirqs.CPU2.TIMER
214659 ± 10% +17.4% 252030 ± 2% softirqs.CPU3.RCU
201381 ± 8% +26.8% 255401 ± 10% softirqs.CPU3.SCHED
218567 ± 9% +15.1% 251588 ± 4% softirqs.CPU4.RCU
162228 ± 14% +39.7% 226690 ± 21% softirqs.CPU4.SCHED
686728 ± 7% +15.4% 792699 ± 4% softirqs.CPU4.TIMER
598763 ± 4% +33.1% 796983 ± 15% softirqs.CPU5.TIMER
219495 ± 10% +15.8% 254260 ± 4% softirqs.CPU6.RCU
685621 ± 6% +15.4% 790913 ± 4% softirqs.CPU6.TIMER
213675 ± 10% +18.1% 252314 ± 2% softirqs.CPU7.RCU
607580 ± 4% +27.7% 776035 ± 7% softirqs.CPU7.TIMER
217328 ± 10% +17.9% 256261 ± 4% softirqs.CPU8.RCU
672277 ± 8% +18.4% 795798 ± 3% softirqs.CPU8.TIMER
662696 ± 15% +22.2% 809936 ± 6% softirqs.CPU9.TIMER
3471815 ± 10% +16.7% 4051741 ± 2% softirqs.RCU
2788089 +21.0% 3372673 ± 2% softirqs.SCHED
10668573 ± 2% +17.5% 12539415 ± 5% softirqs.TIMER
250541 +31.8% 330137 ± 3% sched_debug.cfs_rq:/.exec_clock.avg
446877 ± 7% +68.2% 751493 ± 18% sched_debug.cfs_rq:/.exec_clock.max
112331 ± 17% +122.7% 250151 ± 35% sched_debug.cfs_rq:/.exec_clock.stddev
1517645 +35.0% 2049553 ± 3% sched_debug.cfs_rq:/.min_vruntime.avg
2633991 ± 6% +65.2% 4352247 ± 18% sched_debug.cfs_rq:/.min_vruntime.max
684251 ± 18% +118.2% 1492888 ± 37% sched_debug.cfs_rq:/.min_vruntime.stddev
0.46 ± 2% +8.1% 0.50 ± 4% sched_debug.cfs_rq:/.nr_running.avg
-292907 -321.0% 647293 ± 49% sched_debug.cfs_rq:/.spread0.avg
823435 ± 35% +258.3% 2949987 ± 32% sched_debug.cfs_rq:/.spread0.max
684251 ± 18% +118.2% 1492888 ± 37% sched_debug.cfs_rq:/.spread0.stddev
389.08 +10.3% 429.08 ± 3% sched_debug.cfs_rq:/.util_avg.avg
282.99 ± 2% +13.1% 320.08 ± 6% sched_debug.cfs_rq:/.util_est_enqueued.avg
383.98 ± 2% +8.6% 416.84 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev
842328 +25.8% 1059503 sched_debug.cpu.clock.avg
842329 +25.8% 1059503 sched_debug.cpu.clock.max
842327 +25.8% 1059502 sched_debug.cpu.clock.min
1.02 ± 69% -49.5% 0.51 sched_debug.cpu.clock.stddev
842328 +25.8% 1059503 sched_debug.cpu.clock_task.avg
842329 +25.8% 1059503 sched_debug.cpu.clock_task.max
842327 +25.8% 1059502 sched_debug.cpu.clock_task.min
1.02 ± 69% -49.5% 0.51 sched_debug.cpu.clock_task.stddev
6159 ± 3% +29.3% 7966 ± 6% sched_debug.cpu.curr->pid.avg
21434 +25.0% 26795 sched_debug.cpu.curr->pid.max
7691 ± 2% +26.8% 9755 sched_debug.cpu.curr->pid.stddev
57507 ± 2% +27.2% 73171 sched_debug.cpu.nr_switches.avg
98634 ± 8% +51.9% 149817 ± 22% sched_debug.cpu.nr_switches.max
21105 ± 18% +85.3% 39113 ± 41% sched_debug.cpu.nr_switches.stddev
52692 ± 2% +29.9% 68458 sched_debug.cpu.sched_count.avg
94966 ± 8% +50.3% 142769 ± 25% sched_debug.cpu.sched_count.max
25787 ± 2% +27.6% 32917 sched_debug.cpu.sched_goidle.avg
47100 ± 8% +48.8% 70064 ± 26% sched_debug.cpu.sched_goidle.max
25109 ± 2% +28.2% 32200 sched_debug.cpu.ttwu_count.avg
47736 ± 4% +55.9% 74430 ± 9% sched_debug.cpu.ttwu_count.max
10778 ± 13% +80.8% 19490 ± 29% sched_debug.cpu.ttwu_count.stddev
11888 ± 4% +30.1% 15462 sched_debug.cpu.ttwu_local.avg
842327 +25.8% 1059502 sched_debug.cpu_clk
839741 +25.9% 1056916 sched_debug.ktime
843284 +25.8% 1060461 sched_debug.sched_clk
1307 +27.3% 1663 ± 10% interrupts.35:PCI-MSI.524289-edge.eth0-rx-0
884.50 ± 2% +45.9% 1290 ± 26% interrupts.42:PCI-MSI.524296-edge.eth0-tx-3
1810554 ± 9% -24.9% 1359040 interrupts.CAL:Function_call_interrupts
128994 ± 13% -54.5% 58688 ± 11% interrupts.CPU0.CAL:Function_call_interrupts
3136833 ± 9% +30.6% 4097106 interrupts.CPU0.LOC:Local_timer_interrupts
147624 ± 12% -63.5% 53903 ± 7% interrupts.CPU0.TLB:TLB_shootdowns
131316 ± 12% -58.5% 54545 ± 3% interrupts.CPU1.CAL:Function_call_interrupts
3196967 ± 6% +28.8% 4117877 interrupts.CPU1.LOC:Local_timer_interrupts
1316 ± 44% +134.5% 3086 ± 54% interrupts.CPU1.RES:Rescheduling_interrupts
150414 ± 17% -65.7% 51585 ± 14% interrupts.CPU1.TLB:TLB_shootdowns
3214233 ± 5% +28.6% 4132526 interrupts.CPU10.LOC:Local_timer_interrupts
9879 ± 80% -85.4% 1445 ± 45% interrupts.CPU10.RES:Rescheduling_interrupts
3177700 ± 6% +29.8% 4126192 interrupts.CPU11.LOC:Local_timer_interrupts
559.50 ± 46% +253.3% 1976 ± 51% interrupts.CPU11.RES:Rescheduling_interrupts
1307 +27.3% 1663 ± 10% interrupts.CPU12.35:PCI-MSI.524289-edge.eth0-rx-0
3234073 ± 4% +27.7% 4130544 interrupts.CPU12.LOC:Local_timer_interrupts
9543 ± 47% -86.8% 1263 ± 55% interrupts.CPU12.RES:Rescheduling_interrupts
3178037 ± 7% +29.8% 4124112 interrupts.CPU13.LOC:Local_timer_interrupts
3234950 ± 4% +27.7% 4130372 interrupts.CPU14.LOC:Local_timer_interrupts
7845 ± 61% -86.3% 1074 ± 77% interrupts.CPU14.RES:Rescheduling_interrupts
3177518 ± 7% +29.8% 4123846 interrupts.CPU15.LOC:Local_timer_interrupts
115381 ± 14% -46.8% 61375 ± 38% interrupts.CPU2.CAL:Function_call_interrupts
3211923 ± 4% +28.3% 4122113 interrupts.CPU2.LOC:Local_timer_interrupts
128739 ± 17% -57.7% 54470 ± 48% interrupts.CPU2.TLB:TLB_shootdowns
884.50 ± 2% +45.9% 1290 ± 26% interrupts.CPU3.42:PCI-MSI.524296-edge.eth0-tx-3
3189990 ± 6% +29.1% 4119429 interrupts.CPU3.LOC:Local_timer_interrupts
1680 ± 89% +222.4% 5418 ± 50% interrupts.CPU3.RES:Rescheduling_interrupts
3207702 ± 5% +28.7% 4126732 interrupts.CPU4.LOC:Local_timer_interrupts
132646 ± 21% -46.4% 71034 ± 60% interrupts.CPU4.TLB:TLB_shootdowns
3185259 ± 7% +29.4% 4121092 interrupts.CPU5.LOC:Local_timer_interrupts
1237 ± 56% +203.8% 3757 ± 37% interrupts.CPU5.RES:Rescheduling_interrupts
3200005 ± 5% +29.0% 4128870 interrupts.CPU6.LOC:Local_timer_interrupts
3187267 ± 7% +29.4% 4125551 interrupts.CPU7.LOC:Local_timer_interrupts
802.00 ± 53% +280.5% 3051 ± 26% interrupts.CPU7.RES:Rescheduling_interrupts
3266308 ± 2% +26.5% 4132734 interrupts.CPU8.LOC:Local_timer_interrupts
23706 ± 90% -82.0% 4262 ± 66% interrupts.CPU8.RES:Rescheduling_interrupts
3163000 ± 8% +30.5% 4128371 interrupts.CPU9.LOC:Local_timer_interrupts
51161770 ± 6% +29.0% 65987472 interrupts.LOC:Local_timer_interrupts
1944653 ± 10% -31.9% 1325214 interrupts.TLB:TLB_shootdowns
4.79 +19.6% 5.73 perf-stat.i.MPKI
1.596e+09 -19.9% 1.278e+09 perf-stat.i.branch-instructions
1.66 +0.1 1.76 perf-stat.i.branch-miss-rate%
26901389 -14.4% 23036625 perf-stat.i.branch-misses
1.39e+08 -2.8% 1.351e+08 perf-stat.i.cache-references
0.61 ± 6% +17.9% 0.72 perf-stat.i.cpi
16.50 ± 5% +145.5% 40.52 ± 2% perf-stat.i.cpu-migrations
5919 -3.6% 5706 ± 2% perf-stat.i.cycles-between-cache-misses
0.05 ± 2% +0.0 0.07 perf-stat.i.dTLB-load-miss-rate%
3285384 ± 3% +15.2% 3784496 perf-stat.i.dTLB-load-misses
7.201e+09 -19.5% 5.793e+09 perf-stat.i.dTLB-loads
0.20 ± 2% +0.0 0.22 ± 4% perf-stat.i.dTLB-store-miss-rate%
3641778 ± 2% -7.6% 3366197 ± 5% perf-stat.i.dTLB-store-misses
1.925e+09 -19.5% 1.549e+09 perf-stat.i.dTLB-stores
0.01 ± 5% +0.0 0.02 ± 2% perf-stat.i.iTLB-load-miss-rate%
2872660 ± 2% +25.9% 3617061 ± 2% perf-stat.i.iTLB-load-misses
2.816e+10 -19.7% 2.261e+10 perf-stat.i.iTLB-loads
2.819e+10 -19.8% 2.261e+10 perf-stat.i.instructions
10654 ± 3% -27.1% 7764 ± 2% perf-stat.i.instructions-per-iTLB-miss
1.70 ± 4% -14.9% 1.45 perf-stat.i.ipc
0.08 ± 2% -20.0% 0.07 ± 2% perf-stat.i.major-faults
4638 ± 3% +15.4% 5354 ± 2% perf-stat.i.minor-faults
4638 ± 3% +15.4% 5354 ± 2% perf-stat.i.page-faults
4.93 +21.2% 5.97 perf-stat.overall.MPKI
1.69 +0.1 1.80 perf-stat.overall.branch-miss-rate%
2.48 ± 4% +0.2 2.66 ± 2% perf-stat.overall.cache-miss-rate%
0.58 ± 2% +25.2% 0.73 perf-stat.overall.cpi
0.05 ± 3% +0.0 0.07 perf-stat.overall.dTLB-load-miss-rate%
0.19 ± 2% +0.0 0.22 ± 5% perf-stat.overall.dTLB-store-miss-rate%
0.01 +0.0 0.02 ± 2% perf-stat.overall.iTLB-load-miss-rate%
9815 ± 2% -36.3% 6253 ± 2% perf-stat.overall.instructions-per-iTLB-miss
1.72 ± 2% -20.2% 1.37 perf-stat.overall.ipc
1.595e+09 -19.9% 1.278e+09 perf-stat.ps.branch-instructions
26891393 -14.3% 23033727 perf-stat.ps.branch-misses
1.389e+08 -2.8% 1.35e+08 perf-stat.ps.cache-references
16.50 ± 5% +145.4% 40.50 ± 2% perf-stat.ps.cpu-migrations
3283187 ± 3% +15.2% 3782278 perf-stat.ps.dTLB-load-misses
7.196e+09 -19.5% 5.79e+09 perf-stat.ps.dTLB-loads
3639422 ± 2% -7.6% 3364334 ± 5% perf-stat.ps.dTLB-store-misses
1.924e+09 -19.5% 1.548e+09 perf-stat.ps.dTLB-stores
2870820 ± 2% +25.9% 3614797 ± 2% perf-stat.ps.iTLB-load-misses
2.814e+10 -19.7% 2.26e+10 perf-stat.ps.iTLB-loads
2.817e+10 -19.8% 2.259e+10 perf-stat.ps.instructions
0.08 ± 2% -20.0% 0.07 ± 2% perf-stat.ps.major-faults
4636 ± 3% +15.5% 5353 ± 2% perf-stat.ps.minor-faults
4636 ± 3% +15.5% 5353 ± 2% perf-stat.ps.page-faults
phoronix-test-suite.aom-av1.0.frames_per_second
0.04 +-------------------------------------------------------------------+
| |
0.035 |-+ |
0.03 |-O O O O O O O O O O O O O O O O O O O O O O O O O O O |
| |
0.025 |-+ |
| |
0.02 |-+ |
| |
0.015 |-+ |
0.01 |-+ |
| |
0.005 |-+ |
| |
0 +-------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
4 months, 2 weeks
- 5
- 11
[drm/i915/gt] 36511637bd: WARNING:at_drivers/gpu/drm/i915/gt/intel_workarounds.c:#intel_gt_init_workarounds[i915]
by kernel test robot
FYI, we noticed the following commit (built with gcc-7):
commit: 36511637bd8526b63943cc06f40bf612de73080c ("drm/i915/gt: Apply Enable Boot Fetch to MBC control register")
https://github.com/0day-ci/linux/commits/UPDATE-20200209-084341/Chris-Wil...
in testcase: suspend_stress
with following parameters:
mode: mem
iterations: 10
on test machine: 4 threads BroadWell with 8G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <lkp(a)intel.com>
kern :warn : [ 201.333927] WARNING: CPU: 1 PID: 281 at drivers/gpu/drm/i915/gt/intel_workarounds.c:979 intel_gt_init_workarounds+0xc4/0x350 [i915]
kern :warn : [ 201.334108] Modules linked in: i915(+) crct10dif_pclmul(+) crc32_pclmul crc32c_intel ghash_clmulni_intel iwlwifi btusb btrtl btbcm btintel snd_hda_codec_realtek(+) snd_hda_codec_generic drm_kms_helper ledtrig_audio ahci aesni_intel syscopyarea libahci sysfillrect crypto_simd snd_hda_intel snd_intel_dspcfg cryptd snd_hda_codec glue_helper bluetooth sysimgblt snd_hda_core fb_sys_fops pcspkr snd_hwdep libata ir_rc6_decoder cfg80211 snd_pcm ecdh_generic i2c_i801 lpc_ich joydev rc_rc6_mce ecc drm nuvoton_cir rfkill snd_timer snd mei_me mei soundcore rc_core video acpi_pad intel_pmc_core ip_tables
kern :warn : [ 201.334837] CPU: 1 PID: 281 Comm: systemd-udevd Not tainted 5.5.0-rc7-02125-g36511637bd8526 #2
kern :warn : [ 201.334957] Hardware name: /NUC5i3RYB, BIOS RYBDWi35.86A.0363.2017.0316.1028 03/16/2017
kern :warn : [ 201.335176] RIP: 0010:intel_gt_init_workarounds+0xc4/0x350 [i915]
kern :warn : [ 201.335266] Code: 80 fa 07 0f 84 55 02 00 00 80 fa 06 0f 84 4c 02 00 00 80 fa 05 76 60 48 c7 c6 49 4f 9d c0 48 c7 c7 62 4f 9d c0 e8 4c 9c 80 c0 <0f> 0b eb 49 48 89 ee e8 d0 f1 ff ff f6 83 8b 06 00 00 40 49 89 e4
kern :warn : [ 201.335524] RSP: 0018:ffffc90000247a08 EFLAGS: 00010282
kern :warn : [ 201.335599] RAX: 0000000000000000 RBX: ffff8881f9800000 RCX: 0000000000000007
kern :warn : [ 201.335700] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff888256899b30
kern :warn : [ 201.335801] RBP: ffff8881f9806fa0 R08: 0000000000000a26 R09: 000000000000003b
kern :debug : [ 201.335841] initcall crct10dif_intel_mod_init+0x0/0x1000 [crct10dif_pclmul] returned 0 after 7181 usecs
kern :warn : [ 201.335901] R10: 0000000000000010 R11: ffffc900002478a8 R12: ffff888107c67000
kern :warn : [ 201.335904] R13: ffff8881f9807758 R14: ffff8881f98007b0 R15: ffff8881f9807948
kern :warn : [ 201.336233] FS: 00007f175bad78c0(0000) GS:ffff888256880000(0000) knlGS:0000000000000000
kern :warn : [ 201.336346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kern :warn : [ 201.336428] CR2: 00007f9ecc0e7520 CR3: 0000000202216003 CR4: 00000000003606e0
kern :warn : [ 201.336527] Call Trace:
kern :warn : [ 201.336662] i915_driver_probe+0xdb1/0x15a0 [i915]
kern :warn : [ 201.336739] ? __kernfs_new_node+0x174/0x1e0
kern :warn : [ 201.336804] ? _cond_resched+0x19/0x30
kern :warn : [ 201.336927] i915_pci_probe+0x54/0x170 [i915]
kern :warn : [ 201.336996] local_pci_probe+0x42/0x90
kern :warn : [ 201.337053] ? _cond_resched+0x19/0x30
kern :warn : [ 201.337110] pci_device_probe+0x10b/0x1c0
kern :warn : [ 201.337171] really_probe+0xef/0x420
kern :warn : [ 201.337226] driver_probe_device+0x61/0x120
kern :warn : [ 201.337289] device_driver_attach+0x4f/0x60
kern :warn : [ 201.337353] __driver_attach+0x9a/0x140
kern :warn : [ 201.337411] ? device_driver_attach+0x60/0x60
kern :warn : [ 201.337475] bus_for_each_dev+0x76/0xc0
kern :warn : [ 201.337533] ? klist_add_tail+0x3b/0x70
kern :warn : [ 201.337591] bus_add_driver+0x144/0x220
kern :warn : [ 201.337649] driver_register+0x5b/0xf0
kern :warn : [ 201.337777] i915_init+0x5b/0x6b [i915]
kern :warn : [ 201.337835] ? 0xffffffffc064b000
kern :warn : [ 201.337887] do_one_initcall+0x46/0x220
kern :warn : [ 201.337947] ? _cond_resched+0x19/0x30
kern :warn : [ 201.338005] ? kmem_cache_alloc_trace+0x3b/0x230
kern :warn : [ 201.338074] do_init_module+0x5b/0x21d
kern :warn : [ 201.338132] load_module+0x1b8f/0x2010
kern :warn : [ 201.338191] ? ima_post_read_file+0xe2/0x120
kern :warn : [ 201.338256] ? __do_sys_finit_module+0xe9/0x110
kern :warn : [ 201.338323] __do_sys_finit_module+0xe9/0x110
kern :warn : [ 201.338391] do_syscall_64+0x5b/0x1f0
kern :warn : [ 201.338448] entry_SYSCALL_64_after_hwframe+0x44/0xa9
kern :warn : [ 201.338522] RIP: 0033:0x7f175a954229
kern :warn : [ 201.338575] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 3f 4c 2b 00 f7 d8 64 89 01 48
kern :warn : [ 201.338829] RSP: 002b:00007ffd11f22ba8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
kern :warn : [ 201.338938] RAX: ffffffffffffffda RBX: 00005632e1713910 RCX: 00007f175a954229
kern :warn : [ 201.339039] RDX: 0000000000000000 RSI: 00007f175b26d265 RDI: 0000000000000013
kern :warn : [ 201.339140] RBP: 00007f175b26d265 R08: 0000000000000000 R09: 00007ffd11f23120
kern :warn : [ 201.339243] R10: 0000000000000013 R11: 0000000000000246 R12: 0000000000000000
kern :warn : [ 201.339345] R13: 00005632e170a0a0 R14: 0000000000020000 R15: 00005632df731cbc
kern :warn : [ 201.339448] ---[ end trace 197ca55b67255237 ]---
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
Thanks,
lkp
5 months, 1 week
- 1
- 0
2c7abad740 ("kunit: test: create a single centralized executor .."): [ 409.679449] BUG: kernel NULL pointer dereference, address: 00000000
by kernel test robot
Greetings,
0day kernel testing robot got the below dmesg and the first bad commit is
https://github.com/0day-ci/linux/commits/Brendan-Higgins/kunit-create-a-c...
commit 2c7abad74010ee4f27b0b1f3c68353171e19967b
Author: Alan Maguire <alan.maguire(a)oracle.com>
AuthorDate: Thu Jan 30 15:08:08 2020 -0800
Commit: 0day robot <lkp(a)intel.com>
CommitDate: Mon Feb 3 03:50:54 2020 +0800
kunit: test: create a single centralized executor for all tests
Add a centralized executor to dispatch tests rather than relying on
late_initcall to schedule each test suite separately. Centralized
execution is for built-in tests only; modules will execute tests
when loaded.
Signed-off-by: Alan Maguire <alan.maguire(a)oracle.com>
Co-developed-by: Iurii Zaikin <yzaikin(a)google.com>
Signed-off-by: Iurii Zaikin <yzaikin(a)google.com>
Co-developed-by: Brendan Higgins <brendanhiggins(a)google.com>
Signed-off-by: Brendan Higgins <brendanhiggins(a)google.com>
Reviewed-by: Stephen Boyd <sboyd(a)kernel.org>
bf1af76d02 arch: um: add linker section for KUnit test suites
2c7abad740 kunit: test: create a single centralized executor for all tests
11385b32f9 Documentation: Add kunit_shutdown to kernel-parameters.txt
+-----------------------------------------------------------+------------+------------+------------+
| | bf1af76d02 | 2c7abad740 | 11385b32f9 |
+-----------------------------------------------------------+------------+------------+------------+
| boot_successes | 28 | 1 | 0 |
| boot_failures | 7 | 10 | 7 |
| WARNING:at_kernel/rcu/rcutorture.c:#rcutorture_oom_notify | 1 | | |
| EIP:rcutorture_oom_notify | 1 | | |
| Mem-Info | 1 | | |
| BUG:kernel_NULL_pointer_dereference,address | 1 | 7 | 7 |
| Oops:#[##] | 1 | 7 | 7 |
| EIP:rcu_torture_fwd_cb_hist | 1 | | |
| Kernel_panic-not_syncing:Fatal_exception | 3 | 7 | 7 |
| kernel_BUG_at_mm/vmalloc.c | 2 | | |
| invalid_opcode:#[##] | 2 | | |
| EIP:alloc_vmap_area | 2 | | |
| BUG:kernel_timeout_in_torture_test_stage | 4 | 3 | |
| EIP:kunit_run_all_tests | 0 | 7 | 7 |
+-----------------------------------------------------------+------------+------------+------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <lkp(a)intel.com>
[ 409.675220] ok 7 - sysctl_test_dointvec_write_happy_single_positive
[ 409.675796] ok 8 - sysctl_test_dointvec_write_happy_single_negative
[ 409.676601] ok 9 - sysctl_test_api_dointvec_write_single_less_int_min
[ 409.678166] ok 10 - sysctl_test_api_dointvec_write_single_greater_int_max
[ 409.679009] ok 1 - sysctl_test
[ 409.679449] BUG: kernel NULL pointer dereference, address: 00000000
[ 409.680224] #PF: supervisor read access in kernel mode
[ 409.680846] #PF: error_code(0x0000) - not-present page
[ 409.682123] *pdpt = 0000000000000000 *pde = f000ff53f000ff53
[ 409.682801] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC PTI
[ 409.683475] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G T 5.5.0-08693-g2c7abad74010e #1
[ 409.684540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
[ 409.685584] EIP: kunit_run_all_tests+0x22/0x43
[ 409.686112] Code: 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 57 56 31 f6 53 bb b0 44 3a 5d eb 1c 8b 3b eb 0f e8 86 f7 ff ff 85 c0 b0 01 0f 45 f0 83 c7 04 <8b> 07 85 c0 75 eb 83 c3 04 81 fb bc 44 3a 5d 72 dc 89 f0 3c 01 19
[ 409.688277] EAX: 00000000 EBX: 5d3a44b4 ECX: 00000000 EDX: 00000001
[ 409.689019] ESI: 00000000 EDI: 00000000 EBP: 40073f28 ESP: 40073f1c
[ 409.689841] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010293
[ 409.690633] CR0: 80050033 CR2: 00000000 CR3: 1d516000 CR4: 003406b0
[ 409.691386] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 409.692226] DR6: fffe0ff0 DR7: 00000400
[ 409.692690] Call Trace:
[ 409.693002] ? kunit_try_catch_run+0xd1/0xd1
[ 409.693550] ? do_one_initcall+0xa4/0x1cd
[ 409.694053] ? trace_initcall_level+0x7e/0x82
[ 409.694559] ? kernel_init_freeable+0x100/0x16f
[ 409.695092] ? kernel_init_freeable+0x118/0x16f
[ 409.695628] ? rest_init+0xdc/0xdc
[ 409.696032] ? kernel_init+0x8/0xd5
[ 409.696452] ? ret_from_fork+0x2e/0x38
[ 409.696882] Modules linked in:
[ 409.697290] CR2: 0000000000000000
[ 409.697628] ---[ end trace 045ceeef56b43c7f ]---
[ 409.697984] EIP: kunit_run_all_tests+0x22/0x43
# HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD
git bisect start 7f777d6b2b6ba2646d7facc3c6201073ff3d91ed d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 --
git bisect good 213420b8b9d31ed07f465bfeecc4ac136b12eb2f # 20:37 G 10 0 1 4 Merge 'hwmon/hwmon' into devel-catchup-202002030406
git bisect bad 71e7078961cd57921eccfcea97a5ac01652a3372 # 21:38 B 0 1 17 0 Merge 'linux-review/Brendan-Higgins/kunit-create-a-centralized-executor-to-dispatch-all-KUnit-tests/20200203-035040' into devel-catchup-202002030406
git bisect good a2c8e1e21fa488760d9854643c459b9d13fa1d0e # 22:18 G 10 0 2 2 Merge 'linux-review/Yulia-Kartseva/runqslower-fix-Makefile/20200203-035351' into devel-catchup-202002030406
git bisect good 8f9cc5912605f2f915ba317d01d0076068f0a7e2 # 23:55 G 10 0 2 2 Merge 'linux-review/Vinicius-Costa-Gomes/igc-Add-initial-TSN-qdiscs-offloading/20200203-025747' into devel-catchup-202002030406
git bisect bad 0401f14bd224ed7f2d47b0b9f27bdfbb1552c399 # 00:29 B 0 1 17 0 init: main: add KUnit to kernel init
git bisect good bf1af76d02c3998127b3164b1b5e8638f175a602 # 01:39 G 10 0 3 3 arch: um: add linker section for KUnit test suites
git bisect bad 2c7abad74010ee4f27b0b1f3c68353171e19967b # 02:24 B 0 1 17 0 kunit: test: create a single centralized executor for all tests
# first bad commit: [2c7abad74010ee4f27b0b1f3c68353171e19967b] kunit: test: create a single centralized executor for all tests
git bisect good bf1af76d02c3998127b3164b1b5e8638f175a602 # 03:04 G 30 0 2 6 arch: um: add linker section for KUnit test suites
# extra tests with debug options
git bisect good 2c7abad74010ee4f27b0b1f3c68353171e19967b # 03:42 G 10 0 10 10 kunit: test: create a single centralized executor for all tests
# extra tests on head commit of linux-review/Brendan-Higgins/kunit-create-a-centralized-executor-to-dispatch-all-KUnit-tests/20200203-035040
git bisect bad 11385b32f9e52707959f8566f2c719e66159d2e0 # 04:29 B 0 1 17 0 Documentation: Add kunit_shutdown to kernel-parameters.txt
# bad: [11385b32f9e52707959f8566f2c719e66159d2e0] Documentation: Add kunit_shutdown to kernel-parameters.txt
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation
5 months, 1 week
- 3
- 3
[rcu] 76550daa4d: will-it-scale.per_thread_ops -10.5% regression
by kernel test robot
Greeting,
FYI, we noticed a -10.5% regression of will-it-scale.per_thread_ops due to commit:
commit: 76550daa4d1edaa8251460bd1d4a11b5df23c1c0 ("rcu: Support kfree_bulk() interface in kfree_rcu()")
https://git.kernel.org/cgit/linux/kernel/git/paulmck/linux-rcu.git dev.2020.02.13c
in testcase: will-it-scale
on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
with following parameters:
nr_task: 100%
mode: thread
test: open2
cpufreq_governor: performance
ucode: 0x11
test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen(a)intel.com>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/open2/will-it-scale/0x11
commit:
e8afd73e56 ("rcu: Don't flag non-starting GPs before GP kthread is running")
76550daa4d ("rcu: Support kfree_bulk() interface in kfree_rcu()")
e8afd73e56e02b4d 76550daa4d1edaa8251460bd1d4
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
2:4 -50% :4 dmesg.WARNING:at_ip__fsnotify_parent/0x
:4 25% 1:4 dmesg.WARNING:at_ip__slab_free/0x
%stddev %change %stddev
\ | \
1021 -10.5% 914.50 will-it-scale.per_thread_ops
807081 -33.7% 534989 ± 2% will-it-scale.time.involuntary_context_switches
489.19 ± 9% -40.3% 292.19 ± 3% will-it-scale.time.user_time
294257 -10.5% 263455 will-it-scale.workload
1.68 ± 2% +30.8% 2.20 turbostat.RAMWatt
6340 -28.8% 4512 ± 2% vmstat.system.cs
0.09 ± 12% -0.0 0.06 ± 3% mpstat.cpu.all.soft%
0.71 ± 7% -0.2 0.49 ± 3% mpstat.cpu.all.usr%
999.25 ± 6% -17.3% 826.50 ± 13% slabinfo.skbuff_fclone_cache.active_objs
999.25 ± 6% -17.3% 826.50 ± 13% slabinfo.skbuff_fclone_cache.num_objs
19880 ± 5% -9.1% 18079 softirqs.CPU100.RCU
21159 ± 2% -6.7% 19750 ± 2% softirqs.CPU13.RCU
122401 ± 10% +23.7% 151373 ± 8% softirqs.CPU144.TIMER
21180 -8.5% 19379 softirqs.CPU15.RCU
21456 ± 7% -11.7% 18945 softirqs.CPU24.RCU
124140 ± 7% +12.3% 139469 ± 7% softirqs.CPU250.TIMER
20964 ± 11% -17.5% 17289 softirqs.CPU261.RCU
20945 -8.8% 19102 softirqs.CPU30.RCU
21274 -9.8% 19194 softirqs.CPU31.RCU
21024 ± 2% -10.3% 18853 softirqs.CPU33.RCU
20571 -8.9% 18746 softirqs.CPU35.RCU
21289 ± 2% -8.0% 19588 softirqs.CPU5.RCU
1199 ± 3% +11.2% 1332 ± 2% sched_debug.cfs_rq:/.exec_clock.stddev
969782 ± 8% -26.1% 716472 ± 22% sched_debug.cfs_rq:/.load.max
226.86 ± 6% -70.5% 66.95 ± 18% sched_debug.cfs_rq:/.nr_spread_over.avg
321.05 ± 5% -47.1% 169.95 ± 6% sched_debug.cfs_rq:/.nr_spread_over.max
107.45 ± 6% -68.8% 33.55 ± 26% sched_debug.cfs_rq:/.nr_spread_over.min
39.24 ± 17% -61.2% 15.24 ± 9% sched_debug.cfs_rq:/.nr_spread_over.stddev
969780 ± 8% -26.1% 716278 ± 22% sched_debug.cfs_rq:/.runnable_weight.max
1145 ± 21% -25.2% 856.65 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.max
169.65 ± 46% -80.8% 32.65 ± 79% sched_debug.cfs_rq:/.util_est_enqueued.min
871.94 ± 5% +23.4% 1075 ± 3% sched_debug.cpu.clock.stddev
871.94 ± 5% +23.4% 1075 ± 3% sched_debug.cpu.clock_task.stddev
5190 ± 11% +22.1% 6338 ± 6% sched_debug.cpu.curr->pid.max
271.88 ± 6% +30.5% 354.73 ± 9% sched_debug.cpu.curr->pid.stddev
0.00 ± 5% +23.8% 0.00 ± 2% sched_debug.cpu.next_balance.stddev
5624 -13.1% 4890 sched_debug.cpu.nr_switches.avg
2344 ± 3% -31.0% 1617 ± 3% sched_debug.cpu.nr_switches.min
2803 -25.7% 2082 ± 2% sched_debug.cpu.sched_count.avg
1906 -30.9% 1316 sched_debug.cpu.sched_count.min
1322 -27.8% 954.60 ± 2% sched_debug.cpu.ttwu_count.avg
919.25 -32.6% 619.90 sched_debug.cpu.ttwu_count.min
1255 -29.3% 888.47 ± 2% sched_debug.cpu.ttwu_local.avg
887.45 -32.9% 595.50 sched_debug.cpu.ttwu_local.min
8.476e+09 +1.3% 8.589e+09 perf-stat.i.branch-instructions
48929091 -3.8% 47071099 perf-stat.i.cache-misses
6396 -30.2% 4467 ± 2% perf-stat.i.context-switches
12.78 -1.4% 12.61 perf-stat.i.cpi
9065 +3.7% 9404 perf-stat.i.cycles-between-cache-misses
71922406 ± 2% -7.7% 66350699 ± 5% perf-stat.i.iTLB-load-misses
3.475e+10 +1.3% 3.519e+10 perf-stat.i.iTLB-loads
3.47e+10 +1.2% 3.512e+10 perf-stat.i.instructions
482.82 ± 2% +10.0% 531.13 ± 6% perf-stat.i.instructions-per-iTLB-miss
0.08 +1.4% 0.08 perf-stat.i.ipc
12.81 -1.3% 12.64 perf-stat.overall.cpi
9077 +3.8% 9423 perf-stat.overall.cycles-between-cache-misses
0.21 ± 2% -0.0 0.19 ± 5% perf-stat.overall.iTLB-load-miss-rate%
483.60 ± 2% +9.9% 531.72 ± 6% perf-stat.overall.instructions-per-iTLB-miss
0.08 +1.3% 0.08 perf-stat.overall.ipc
36101370 +13.0% 40812598 perf-stat.overall.path-length
8.46e+09 +1.2% 8.565e+09 perf-stat.ps.branch-instructions
48864485 -3.9% 46979669 perf-stat.ps.cache-misses
6244 -29.3% 4417 ± 2% perf-stat.ps.context-switches
71662456 ± 2% -7.8% 66078101 ± 5% perf-stat.ps.iTLB-load-misses
3.466e+10 +1.2% 3.507e+10 perf-stat.ps.iTLB-loads
3.463e+10 +1.1% 3.502e+10 perf-stat.ps.instructions
1.062e+13 +1.2% 1.075e+13 perf-stat.total.instructions
1.31 ± 2% -0.2 1.16 ± 4% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.26 ± 2% -0.1 1.13 ± 3% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
48.05 +0.1 48.16 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__alloc_fd.do_sys_openat2.do_sys_open
48.09 +0.1 48.21 perf-profile.calltrace.cycles-pp._raw_spin_lock.__alloc_fd.do_sys_openat2.do_sys_open.do_syscall_64
48.16 +0.2 48.31 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__close_fd.__x64_sys_close.do_syscall_64
48.20 +0.2 48.37 perf-profile.calltrace.cycles-pp._raw_spin_lock.__close_fd.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe
48.32 +0.2 48.49 perf-profile.calltrace.cycles-pp.__close_fd.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_close
48.24 +0.2 48.41 perf-profile.calltrace.cycles-pp.__alloc_fd.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
48.46 +0.2 48.64 perf-profile.calltrace.cycles-pp.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_close
1.32 ± 2% -0.2 1.16 ± 3% perf-profile.children.cycles-pp.do_filp_open
0.35 ± 25% -0.1 0.21 ± 14% perf-profile.children.cycles-pp.update_curr
1.27 ± 2% -0.1 1.14 ± 4% perf-profile.children.cycles-pp.path_openat
0.44 ± 2% -0.1 0.37 ± 2% perf-profile.children.cycles-pp.alloc_empty_file
0.19 ± 5% -0.1 0.12 ± 4% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.33 ± 3% -0.1 0.27 ± 5% perf-profile.children.cycles-pp.__fput
0.50 ± 3% -0.1 0.44 ± 5% perf-profile.children.cycles-pp.exit_to_usermode_loop
0.15 ± 4% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.security_file_alloc
0.45 ± 3% -0.1 0.40 ± 4% perf-profile.children.cycles-pp.task_work_run
0.39 ± 2% -0.0 0.35 ± 2% perf-profile.children.cycles-pp.__alloc_file
0.32 -0.0 0.29 ± 3% perf-profile.children.cycles-pp.link_path_walk
0.30 ± 5% -0.0 0.27 ± 4% perf-profile.children.cycles-pp.rcu_do_batch
0.30 ± 4% -0.0 0.28 ± 4% perf-profile.children.cycles-pp.rcu_core
0.22 -0.0 0.19 ± 2% perf-profile.children.cycles-pp.kmem_cache_free
0.12 ± 5% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.inode_permission
0.06 ± 7% +0.0 0.07 ± 5% perf-profile.children.cycles-pp.___might_sleep
0.06 ± 11% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.run_timer_softirq
0.26 ± 2% +0.0 0.29 ± 3% perf-profile.children.cycles-pp.kmem_cache_alloc
0.10 ± 9% +0.0 0.14 ± 8% perf-profile.children.cycles-pp.enqueue_hrtimer
0.08 ± 10% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.timerqueue_add
0.04 ± 57% +0.1 0.09 ± 11% perf-profile.children.cycles-pp.expand_files
0.05 ± 8% +0.1 0.11 ± 7% perf-profile.children.cycles-pp.perf_event_task_tick
0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.__might_sleep
0.00 +0.1 0.06 ± 20% perf-profile.children.cycles-pp.locks_remove_posix
0.11 ± 7% +0.1 0.17 ± 17% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.08 ± 15% +0.1 0.15 ± 19% perf-profile.children.cycles-pp.rcu_irq_enter
99.41 +0.1 99.49 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
99.37 +0.1 99.46 perf-profile.children.cycles-pp.do_syscall_64
48.32 +0.2 48.49 perf-profile.children.cycles-pp.__close_fd
48.27 +0.2 48.44 perf-profile.children.cycles-pp.__alloc_fd
48.46 +0.2 48.64 perf-profile.children.cycles-pp.__x64_sys_close
96.38 +0.3 96.64 perf-profile.children.cycles-pp._raw_spin_lock
96.31 +0.3 96.59 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.19 ± 5% -0.1 0.12 ± 4% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.11 ± 4% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.__alloc_file
0.21 ± 6% -0.0 0.18 ± 4% perf-profile.self.cycles-pp.file_free_rcu
0.08 ± 5% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.inode_permission
0.15 ± 3% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.kmem_cache_free
0.07 +0.0 0.09 perf-profile.self.cycles-pp.do_syscall_64
0.03 ±100% +0.0 0.06 ± 6% perf-profile.self.cycles-pp.run_timer_softirq
0.06 ± 6% +0.0 0.11 ± 8% perf-profile.self.cycles-pp.timerqueue_add
0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.__might_sleep
0.05 ± 8% +0.1 0.11 ± 7% perf-profile.self.cycles-pp.perf_event_task_tick
0.00 +0.1 0.06 ± 20% perf-profile.self.cycles-pp.locks_remove_posix
0.11 ± 4% +0.1 0.17 ± 17% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.00 +0.1 0.06 ± 17% perf-profile.self.cycles-pp.expand_files
0.08 ± 15% +0.1 0.15 ± 19% perf-profile.self.cycles-pp.rcu_irq_enter
396.75 ± 31% +53.1% 607.25 ± 12% interrupts.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2
6719 ± 24% -42.5% 3865 interrupts.CPU0.NMI:Non-maskable_interrupts
6719 ± 24% -42.5% 3865 interrupts.CPU0.PMI:Performance_monitoring_interrupts
1541 ± 14% -38.1% 954.75 ± 32% interrupts.CPU0.RES:Rescheduling_interrupts
85.00 ± 34% -74.4% 21.75 ± 54% interrupts.CPU104.RES:Rescheduling_interrupts
67.75 ±109% -80.8% 13.00 ± 14% interrupts.CPU105.RES:Rescheduling_interrupts
5717 ± 32% -33.3% 3814 interrupts.CPU106.NMI:Non-maskable_interrupts
5717 ± 32% -33.3% 3814 interrupts.CPU106.PMI:Performance_monitoring_interrupts
3828 +73.2% 6630 ± 24% interrupts.CPU109.NMI:Non-maskable_interrupts
3828 +73.2% 6630 ± 24% interrupts.CPU109.PMI:Performance_monitoring_interrupts
396.75 ± 31% +53.1% 607.25 ± 12% interrupts.CPU12.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2
5694 ± 32% -33.7% 3777 interrupts.CPU131.NMI:Non-maskable_interrupts
5694 ± 32% -33.7% 3777 interrupts.CPU131.PMI:Performance_monitoring_interrupts
4725 ± 34% +58.8% 7506 interrupts.CPU173.NMI:Non-maskable_interrupts
4725 ± 34% +58.8% 7506 interrupts.CPU173.PMI:Performance_monitoring_interrupts
6578 ± 24% -42.9% 3756 interrupts.CPU179.NMI:Non-maskable_interrupts
6578 ± 24% -42.9% 3756 interrupts.CPU179.PMI:Performance_monitoring_interrupts
5716 ± 33% -17.2% 4735 ± 35% interrupts.CPU183.NMI:Non-maskable_interrupts
5716 ± 33% -17.2% 4735 ± 35% interrupts.CPU183.PMI:Performance_monitoring_interrupts
4686 ± 34% +41.7% 6639 ± 24% interrupts.CPU187.NMI:Non-maskable_interrupts
4686 ± 34% +41.7% 6639 ± 24% interrupts.CPU187.PMI:Performance_monitoring_interrupts
63.00 ± 81% -66.3% 21.25 ± 91% interrupts.CPU187.RES:Rescheduling_interrupts
5799 ± 32% -17.4% 4787 ± 33% interrupts.CPU20.NMI:Non-maskable_interrupts
5799 ± 32% -17.4% 4787 ± 33% interrupts.CPU20.PMI:Performance_monitoring_interrupts
3802 +48.6% 5649 ± 31% interrupts.CPU207.NMI:Non-maskable_interrupts
3802 +48.6% 5649 ± 31% interrupts.CPU207.PMI:Performance_monitoring_interrupts
51.25 ± 96% -82.4% 9.00 ± 7% interrupts.CPU211.RES:Rescheduling_interrupts
5692 ± 30% -17.6% 4690 ± 32% interrupts.CPU215.NMI:Non-maskable_interrupts
5692 ± 30% -17.6% 4690 ± 32% interrupts.CPU215.PMI:Performance_monitoring_interrupts
70.00 ±116% -80.7% 13.50 ± 49% interrupts.CPU217.RES:Rescheduling_interrupts
7.25 ± 35% +903.4% 72.75 ± 85% interrupts.CPU222.RES:Rescheduling_interrupts
3786 +49.4% 5655 ± 32% interrupts.CPU224.NMI:Non-maskable_interrupts
3786 +49.4% 5655 ± 32% interrupts.CPU224.PMI:Performance_monitoring_interrupts
3778 +48.2% 5599 ± 31% interrupts.CPU227.NMI:Non-maskable_interrupts
3778 +48.2% 5599 ± 31% interrupts.CPU227.PMI:Performance_monitoring_interrupts
66.75 ± 97% -86.1% 9.25 ± 20% interrupts.CPU228.RES:Rescheduling_interrupts
4710 ± 33% +60.1% 7539 interrupts.CPU240.NMI:Non-maskable_interrupts
4710 ± 33% +60.1% 7539 interrupts.CPU240.PMI:Performance_monitoring_interrupts
18.75 ± 40% +618.7% 134.75 ± 69% interrupts.CPU240.RES:Rescheduling_interrupts
12.50 ± 30% +582.0% 85.25 ± 81% interrupts.CPU25.RES:Rescheduling_interrupts
4672 ± 33% +39.4% 6515 ± 23% interrupts.CPU284.NMI:Non-maskable_interrupts
4672 ± 33% +39.4% 6515 ± 23% interrupts.CPU284.PMI:Performance_monitoring_interrupts
48.75 ±123% -78.5% 10.50 ± 24% interrupts.CPU284.RES:Rescheduling_interrupts
6705 ± 24% -28.7% 4781 ± 34% interrupts.CPU5.NMI:Non-maskable_interrupts
6705 ± 24% -28.7% 4781 ± 34% interrupts.CPU5.PMI:Performance_monitoring_interrupts
3801 +73.9% 6612 ± 24% interrupts.CPU52.NMI:Non-maskable_interrupts
3801 +73.9% 6612 ± 24% interrupts.CPU52.PMI:Performance_monitoring_interrupts
4731 ± 34% +40.3% 6638 ± 24% interrupts.CPU53.NMI:Non-maskable_interrupts
4731 ± 34% +40.3% 6638 ± 24% interrupts.CPU53.PMI:Performance_monitoring_interrupts
3808 +49.4% 5689 ± 32% interrupts.CPU54.NMI:Non-maskable_interrupts
3808 +49.4% 5689 ± 32% interrupts.CPU54.PMI:Performance_monitoring_interrupts
5676 ± 32% -16.8% 4720 ± 34% interrupts.CPU59.NMI:Non-maskable_interrupts
5676 ± 32% -16.8% 4720 ± 34% interrupts.CPU59.PMI:Performance_monitoring_interrupts
3778 +50.0% 5666 ± 32% interrupts.CPU63.NMI:Non-maskable_interrupts
3778 +50.0% 5666 ± 32% interrupts.CPU63.PMI:Performance_monitoring_interrupts
5739 ± 32% -18.0% 4706 ± 33% interrupts.CPU72.NMI:Non-maskable_interrupts
5739 ± 32% -18.0% 4706 ± 33% interrupts.CPU72.PMI:Performance_monitoring_interrupts
4750 ± 33% +59.4% 7571 interrupts.CPU87.NMI:Non-maskable_interrupts
4750 ± 33% +59.4% 7571 interrupts.CPU87.PMI:Performance_monitoring_interrupts
5720 ± 33% -34.0% 3778 interrupts.CPU98.NMI:Non-maskable_interrupts
5720 ± 33% -34.0% 3778 interrupts.CPU98.PMI:Performance_monitoring_interrupts
will-it-scale.per_thread_ops
1040 +--------------------------------------------------------------------+
| +.. .+.|
1020 |-+ .+ + +.+ |
1000 |-+ .+. .+ + .+.. +.+ .+ .+..+.+. + |
|.+.+.+. + + + + .+.+..+ + .+.+.+ +. + |
980 |-+ + + + + |
960 |-+ |
| |
940 |-+ |
920 |-+ O O O O O O |
| O O O O O O O |
900 |-+ O O O O O O O O |
880 |-+ O O O O O |
| |
860 +--------------------------------------------------------------------+
will-it-scale.workload
300000 +------------------------------------------------------------------+
295000 |-+ .|
| + .+.+.+.+ |
290000 |-+ + : .+.+ + |
285000 |-+ .+.+.+..+ : .+. .+.+. .+.+. .+.+ + .. |
|.+.+ + + +.+.+. +.+.+ +.+ |
280000 |-+ |
275000 |-+ |
270000 |-+ |
| O O |
265000 |-+ O O O O O O |
260000 |-O O O O O O O O |
| O O O O O O O O |
255000 |-+ O O |
250000 +------------------------------------------------------------------+
will-it-scale.time.involuntary_context_switches
850000 +------------------------------------------------------------------+
| .+.. +.+ +. + +.+.|
800000 |-+. .+.+ +. + : : + +. .+.+ +. .+.+.+.+ + + + |
|+ + + : : + + +. + + + + + + |
750000 |-+ + + + +. .. |
| + |
700000 |-+ |
| |
650000 |-+ |
| |
600000 |-+ |
| |
550000 |-+ O O O O O O O |
| O O O O O O O O O O O O O O O O O O O |
500000 +------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
5 months, 1 week
- 2
- 1
[mm/workingset] 323c95f095: fio.read_bw_MBps 19.5% improvement
by kernel test robot
Greeting,
FYI, we noticed a 19.5% improvement of fio.read_bw_MBps due to commit:
commit: 323c95f0957c8a7d59cd9d05f43f19dfc9bbc584 ("[PATCH 3/9] mm/workingset: extend the workingset detection for anon LRU")
url: https://github.com/0day-ci/linux/commits/js1304-gmail-com/workingset-prot...
in testcase: fio-basic
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:
disk: 2pmem
fs: ext4
runtime: 200s
nr_task: 50%
time_based: tb
rw: randrw
bs: 2M
ioengine: mmap
test_size: 200G
cpufreq_governor: performance
ucode: 0x500002c
test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
2M/gcc-7/performance/2pmem/ext4/mmap/x86_64-rhel-7.6/50%/debian-x86_64-20191114.cgz/200s/randrw/lkp-csl-2sp6/200G/fio-basic/tb/0x500002c
commit:
fffae40d0e ("mm/vmscan: protect the workingset on anonymous LRU")
323c95f095 ("mm/workingset: extend the workingset detection for anon LRU")
fffae40d0ee3fd1e 323c95f0957c8a7d59cd9d05f43
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
1:4 -25% :4 dmesg.WARNING:at_ip___perf_sw_event/0x
:4 25% 1:4 dmesg.WARNING:at_ip_generic_make_request/0x
2:4 45% 4:4 perf-profile.calltrace.cycles-pp.sync_regs.error_entry
2:4 50% 4:4 perf-profile.calltrace.cycles-pp.error_entry
2:4 53% 4:4 perf-profile.children.cycles-pp.error_entry
0:4 6% 0:4 perf-profile.self.cycles-pp.error_entry
%stddev %change %stddev
\ | \
0.28 ± 11% -0.1 0.16 ± 28% fio.latency_100ms%
8.36 ± 8% -5.9 2.50 ± 5% fio.latency_10ms%
6.10 ± 17% -5.3 0.82 ± 6% fio.latency_20ms%
0.00 ±173% +0.0 0.01 ± 10% fio.latency_250ms%
0.71 ± 16% +2.8 3.56 ± 4% fio.latency_250us%
13.45 ± 5% +9.4 22.85 ± 2% fio.latency_2ms%
29.90 ± 2% -7.3 22.58 ± 2% fio.latency_4ms%
8.43 ± 7% +9.4 17.84 ± 4% fio.latency_500us%
7.43 ± 3% -3.3 4.13 ± 12% fio.latency_750us%
4336 ± 2% +19.5% 5182 ± 2% fio.read_bw_MBps
7430144 ± 16% -64.3% 2654208 fio.read_clat_90%_us
11501568 ± 9% -70.8% 3358720 fio.read_clat_95%_us
17170432 ± 8% -57.7% 7258112 ± 4% fio.read_clat_99%_us
2960419 ± 7% -53.3% 1381107 fio.read_clat_mean_us
4105907 ± 14% -45.6% 2232352 ± 11% fio.read_clat_stddev
2168 ± 2% +19.5% 2591 ± 2% fio.read_iops
1.942e+09 ± 2% -22.0% 1.514e+09 ± 2% fio.time.file_system_inputs
1.783e+09 ± 2% +19.5% 2.131e+09 ± 2% fio.time.file_system_outputs
216950 ± 12% -48.8% 111051 ± 16% fio.time.involuntary_context_switches
2.427e+08 ± 2% -22.0% 1.892e+08 ± 2% fio.time.major_page_faults
1.271e+08 ± 5% +66.8% 2.12e+08 ± 2% fio.time.minor_page_faults
1619 ± 9% -33.6% 1074 ± 2% fio.time.percent_of_cpu_this_job_got
2851 ± 9% -40.1% 1708 ± 2% fio.time.system_time
404.54 ± 6% +11.4% 450.81 ± 2% fio.time.user_time
869238 ± 2% +19.5% 1038826 ± 2% fio.workload
4350 ± 2% +19.5% 5199 ± 2% fio.write_bw_MBps
38404096 -12.9% 33456128 fio.write_clat_90%_us
39452672 -14.0% 33947648 fio.write_clat_95%_us
45744128 -19.5% 36831232 ± 3% fio.write_clat_99%_us
17993869 ± 4% -9.2% 16330675 ± 2% fio.write_clat_mean_us
16656042 ± 5% -6.2% 15615331 fio.write_clat_stddev
2175 ± 2% +19.5% 2599 ± 2% fio.write_iops
31.58 ± 4% +5.8 37.39 mpstat.cpu.all.iowait%
18.54 ± 7% -6.5 12.04 ± 2% mpstat.cpu.all.sys%
1.008e+08 ± 10% -37.9% 62633861 ± 18% numa-numastat.node0.local_node
1.008e+08 ± 9% -37.9% 62658517 ± 18% numa-numastat.node0.numa_hit
48.29 +1.1% 48.80 iostat.cpu.idle
31.27 ± 4% +18.4% 37.03 iostat.cpu.iowait
18.38 ± 7% -35.0% 11.94 ± 2% iostat.cpu.system
618.00 ± 6% -27.0% 451.25 turbostat.Avg_MHz
22.14 ± 6% -6.0 16.17 turbostat.Busy%
1.361e+08 ± 7% -26.6% 99961074 ± 3% turbostat.IRQ
224.76 -4.7% 214.16 turbostat.PkgWatt
4580495 +85.7% 8505839 slabinfo.buffer_head.active_objs
119518 +82.7% 218335 slabinfo.buffer_head.active_slabs
4661221 +82.7% 8515085 slabinfo.buffer_head.num_objs
119518 +82.7% 218335 slabinfo.buffer_head.num_slabs
19148 ± 3% +9.2% 20917 ± 3% slabinfo.kmalloc-512.num_objs
30.75 ± 4% +19.5% 36.75 vmstat.cpu.wa
4753706 ± 2% -21.9% 3710468 ± 2% vmstat.io.bi
4361808 ± 2% +19.7% 5220006 ± 2% vmstat.io.bo
619.75 ± 8% +86.0% 1153 ± 5% vmstat.memory.buff
31.25 ± 4% +15.2% 36.00 vmstat.procs.b
19.00 ± 9% -28.9% 13.50 ± 3% vmstat.procs.r
409038 ± 5% -18.7% 332463 ± 2% vmstat.system.in
9785202 +221.1% 31421653 meminfo.Active
60190 ± 8% -34.3% 39569 ± 3% meminfo.Active(anon)
9725011 +222.7% 31382083 meminfo.Active(file)
619.25 ± 7% +86.7% 1156 ± 5% meminfo.Buffers
32198569 -68.1% 10269890 meminfo.Inactive
28407610 -77.2% 6464247 meminfo.Inactive(file)
650015 +60.3% 1041779 meminfo.KReclaimable
650015 +60.3% 1041779 meminfo.SReclaimable
829409 +47.3% 1221631 meminfo.Slab
2564 ± 4% +42.5% 3654 ± 4% meminfo.Writeback
4673366 ± 9% +204.3% 14223125 numa-meminfo.node0.Active
4670584 ± 9% +204.4% 14218202 numa-meminfo.node0.Active(file)
3328794 ± 5% -10.7% 2972288 numa-meminfo.node0.Dirty
16103425 ± 3% -60.3% 6392546 ± 2% numa-meminfo.node0.Inactive
13636450 ± 5% -77.2% 3114750 ± 3% numa-meminfo.node0.Inactive(file)
361437 ± 2% +64.5% 594674 ± 2% numa-meminfo.node0.KReclaimable
361437 ± 2% +64.5% 594674 ± 2% numa-meminfo.node0.SReclaimable
454217 +51.0% 686089 ± 2% numa-meminfo.node0.Slab
5117666 ± 8% +236.1% 17201387 numa-meminfo.node1.Active
57417 ± 8% -39.6% 34652 ± 7% numa-meminfo.node1.Active(anon)
5060248 ± 8% +239.2% 17166734 numa-meminfo.node1.Active(file)
16086677 ± 3% -76.0% 3867738 ± 5% numa-meminfo.node1.Inactive
14767296 ± 5% -77.4% 3343591 ± 2% numa-meminfo.node1.Inactive(file)
288589 ± 3% +54.9% 446992 ± 3% numa-meminfo.node1.KReclaimable
288589 ± 3% +54.9% 446992 ± 3% numa-meminfo.node1.SReclaimable
375202 ± 2% +42.7% 535430 ± 2% numa-meminfo.node1.Slab
1310 ± 9% +57.4% 2062 ± 5% numa-meminfo.node1.Writeback
0.02 ± 38% +151.1% 0.04 ± 4% sched_debug.cfs_rq:/.nr_spread_over.avg
0.14 ± 30% +62.3% 0.22 ± 11% sched_debug.cfs_rq:/.nr_spread_over.stddev
702.00 ± 3% +20.3% 844.48 ± 11% sched_debug.cfs_rq:/.runnable_load_avg.max
8658 ± 41% -88.5% 994.62 ±366% sched_debug.cfs_rq:/.spread0.avg
-5693 +147.0% -14059 sched_debug.cfs_rq:/.spread0.min
280.56 ± 11% -20.9% 221.88 ± 5% sched_debug.cfs_rq:/.util_avg.avg
16889 ± 8% +31.2% 22166 ± 13% sched_debug.cpu.nr_switches.avg
0.21 ± 23% +44.0% 0.30 ± 7% sched_debug.cpu.nr_uninterruptible.avg
15025 ± 9% +35.5% 20353 ± 14% sched_debug.cpu.sched_count.avg
16991 ± 21% +31.0% 22263 ± 16% sched_debug.cpu.sched_count.stddev
6755 ± 11% +39.0% 9388 ± 14% sched_debug.cpu.sched_goidle.avg
478.75 ± 16% +37.8% 659.79 ± 19% sched_debug.cpu.sched_goidle.min
8464 ± 21% +31.2% 11108 ± 16% sched_debug.cpu.sched_goidle.stddev
7465 ± 9% +35.5% 10114 ± 14% sched_debug.cpu.ttwu_count.avg
3184 ± 15% +79.5% 5715 ± 14% sched_debug.cpu.ttwu_local.max
349.50 ± 15% +32.8% 464.02 ± 16% sched_debug.cpu.ttwu_local.min
516.10 ± 9% +58.8% 819.50 ± 16% sched_debug.cpu.ttwu_local.stddev
135179 +16.5% 157474 ± 8% sched_debug.ktime
20801 ± 9% +15.2% 23973 ± 3% softirqs.CPU13.SCHED
21834 ± 5% +11.4% 24317 ± 4% softirqs.CPU15.SCHED
20068 ± 13% +26.6% 25416 ± 2% softirqs.CPU18.SCHED
21227 ± 8% +18.3% 25107 ± 2% softirqs.CPU19.SCHED
21493 ± 6% +14.2% 24542 ± 4% softirqs.CPU21.SCHED
21659 ± 5% +17.5% 25448 ± 3% softirqs.CPU22.SCHED
21989 ± 4% +17.2% 25773 ± 2% softirqs.CPU23.SCHED
20132 ± 14% +23.2% 24812 ± 2% softirqs.CPU25.SCHED
20294 ± 16% +23.7% 25112 ± 5% softirqs.CPU26.SCHED
21996 ± 2% +14.2% 25115 ± 2% softirqs.CPU3.SCHED
21985 ± 8% +17.2% 25765 ± 3% softirqs.CPU31.SCHED
21974 ± 6% +14.6% 25184 softirqs.CPU37.SCHED
21943 +12.4% 24665 ± 3% softirqs.CPU40.SCHED
21177 ± 9% +17.8% 24954 ± 2% softirqs.CPU41.SCHED
21555 ± 4% +16.1% 25018 softirqs.CPU43.SCHED
21565 ± 2% +15.0% 24804 softirqs.CPU45.SCHED
22158 ± 6% +14.9% 25466 ± 3% softirqs.CPU50.SCHED
22571 ± 7% +13.4% 25597 softirqs.CPU51.SCHED
21477 ± 7% +17.8% 25310 softirqs.CPU53.SCHED
21648 ± 4% +15.7% 25037 ± 2% softirqs.CPU54.SCHED
22303 ± 4% +10.0% 24524 ± 5% softirqs.CPU57.SCHED
22250 ± 8% +13.7% 25302 ± 2% softirqs.CPU6.SCHED
21945 ± 6% +15.1% 25253 softirqs.CPU62.SCHED
20860 ± 9% +23.0% 25655 ± 3% softirqs.CPU64.SCHED
22491 ± 4% +7.5% 24180 ± 5% softirqs.CPU81.SCHED
19570 ± 21% +28.5% 25157 ± 2% softirqs.CPU83.SCHED
1167199 ± 9% +204.5% 3554443 numa-vmstat.node0.nr_active_file
833733 ± 5% -11.3% 739589 numa-vmstat.node0.nr_dirty
3409188 ± 5% -77.1% 779009 ± 3% numa-vmstat.node0.nr_inactive_file
83.50 ± 22% -60.8% 32.75 ± 16% numa-vmstat.node0.nr_isolated_file
90376 ± 2% +64.5% 148639 ± 2% numa-vmstat.node0.nr_slab_reclaimable
1394052 ± 7% +282.6% 5333857 ± 17% numa-vmstat.node0.nr_vmscan_immediate_reclaim
315.25 ± 3% +22.5% 386.25 ± 13% numa-vmstat.node0.nr_writeback
1167965 ± 9% +204.3% 3554646 numa-vmstat.node0.nr_zone_active_file
3409621 ± 5% -77.1% 780149 ± 3% numa-vmstat.node0.nr_zone_inactive_file
835951 ± 5% -11.3% 741848 numa-vmstat.node0.nr_zone_write_pending
62116218 ± 6% -31.7% 42426214 ± 23% numa-vmstat.node0.numa_hit
62089335 ± 6% -31.8% 42365222 ± 23% numa-vmstat.node0.numa_local
18221559 ± 10% -100.0% 0.00 numa-vmstat.node0.workingset_activate
13281 ± 7% -22.3% 10326 ± 8% numa-vmstat.node0.workingset_nodes
33316100 ± 6% -100.0% 0.00 numa-vmstat.node0.workingset_refault
10372861 ± 12% -100.0% 1.25 ± 66% numa-vmstat.node0.workingset_restore
14346 ± 8% -39.6% 8669 ± 7% numa-vmstat.node1.nr_active_anon
1263972 ± 8% +239.6% 4292221 numa-vmstat.node1.nr_active_file
61372170 ± 9% +31.3% 80554020 ± 8% numa-vmstat.node1.nr_dirtied
3692694 ± 5% -77.4% 835792 numa-vmstat.node1.nr_inactive_file
72150 ± 3% +54.8% 111693 ± 3% numa-vmstat.node1.nr_slab_reclaimable
1341628 ± 12% +750.8% 11415031 ± 6% numa-vmstat.node1.nr_vmscan_immediate_reclaim
331.25 ± 10% +66.0% 549.75 ± 9% numa-vmstat.node1.nr_writeback
60469655 ± 9% +31.7% 79613669 ± 8% numa-vmstat.node1.nr_written
14347 ± 8% -39.6% 8670 ± 7% numa-vmstat.node1.nr_zone_active_anon
1264780 ± 8% +239.4% 4292348 numa-vmstat.node1.nr_zone_active_file
3693150 ± 5% -77.3% 837009 numa-vmstat.node1.nr_zone_inactive_file
19519796 ± 8% -100.0% 0.00 numa-vmstat.node1.workingset_activate
12632 ± 5% -20.7% 10012 ± 4% numa-vmstat.node1.workingset_nodes
36195648 ± 4% -100.0% 0.00 numa-vmstat.node1.workingset_refault
11279550 ± 9% -100.0% 0.25 ±173% numa-vmstat.node1.workingset_restore
15.72 +11.1% 17.46 perf-stat.i.MPKI
7.221e+09 ± 3% -14.9% 6.144e+09 ± 2% perf-stat.i.branch-instructions
29447507 ± 3% -13.8% 25382627 ± 2% perf-stat.i.branch-misses
67.78 +2.0 69.77 perf-stat.i.cache-miss-rate%
1.55 ± 4% -14.7% 1.33 perf-stat.i.cpi
5.857e+10 ± 7% -27.4% 4.252e+10 perf-stat.i.cpu-cycles
148.26 ± 2% -12.2% 130.23 perf-stat.i.cpu-migrations
154.03 ± 3% -24.7% 115.95 perf-stat.i.cycles-between-cache-misses
0.20 ± 2% -0.0 0.15 ± 3% perf-stat.i.dTLB-load-miss-rate%
20033658 ± 4% -34.6% 13108913 perf-stat.i.dTLB-load-misses
9.948e+09 ± 3% -13.4% 8.617e+09 ± 3% perf-stat.i.dTLB-loads
0.37 ± 3% -0.1 0.28 ± 3% perf-stat.i.dTLB-store-miss-rate%
20362529 ± 4% -30.4% 14162666 ± 5% perf-stat.i.dTLB-store-misses
5.532e+09 ± 2% -10.6% 4.945e+09 ± 2% perf-stat.i.dTLB-stores
73.19 -1.4 71.76 perf-stat.i.iTLB-load-miss-rate%
12785996 ± 2% -10.5% 11440556 ± 3% perf-stat.i.iTLB-load-misses
4812663 ± 2% -6.0% 4526122 perf-stat.i.iTLB-loads
3.712e+10 ± 3% -14.2% 3.185e+10 ± 2% perf-stat.i.instructions
2895 -4.1% 2777 perf-stat.i.instructions-per-iTLB-miss
0.66 ± 4% +16.5% 0.77 perf-stat.i.ipc
1203000 ± 2% -22.0% 938451 ± 2% perf-stat.i.major-faults
632441 ± 5% +66.7% 1054315 ± 2% perf-stat.i.minor-faults
1835441 ± 3% +8.6% 1992767 ± 2% perf-stat.i.page-faults
15.80 +10.3% 17.43 perf-stat.overall.MPKI
67.75 +2.1 69.87 perf-stat.overall.cache-miss-rate%
1.58 ± 4% -15.3% 1.34 perf-stat.overall.cpi
147.21 ± 3% -25.5% 109.71 perf-stat.overall.cycles-between-cache-misses
0.20 ± 2% -0.0 0.15 ± 3% perf-stat.overall.dTLB-load-miss-rate%
0.37 ± 3% -0.1 0.29 ± 3% perf-stat.overall.dTLB-store-miss-rate%
72.65 -1.0 71.64 perf-stat.overall.iTLB-load-miss-rate%
2902 -4.1% 2784 perf-stat.overall.instructions-per-iTLB-miss
0.64 ± 4% +17.8% 0.75 perf-stat.overall.ipc
8598809 -28.3% 6163102 perf-stat.overall.path-length
7.185e+09 ± 3% -14.9% 6.114e+09 ± 2% perf-stat.ps.branch-instructions
29306041 ± 3% -13.8% 25261462 ± 2% perf-stat.ps.branch-misses
5.829e+10 ± 7% -27.4% 4.232e+10 perf-stat.ps.cpu-cycles
147.55 ± 2% -12.2% 129.61 perf-stat.ps.cpu-migrations
19935636 ± 4% -34.6% 13047562 perf-stat.ps.dTLB-load-misses
9.899e+09 ± 3% -13.4% 8.575e+09 ± 2% perf-stat.ps.dTLB-loads
20263384 ± 4% -30.4% 14096250 ± 5% perf-stat.ps.dTLB-store-misses
5.505e+09 ± 2% -10.6% 4.92e+09 ± 2% perf-stat.ps.dTLB-stores
12723152 ± 2% -10.5% 11384126 ± 3% perf-stat.ps.iTLB-load-misses
4789226 ± 2% -6.0% 4503341 perf-stat.ps.iTLB-loads
3.694e+10 ± 3% -14.2% 3.169e+10 ± 2% perf-stat.ps.instructions
1197099 ± 2% -22.0% 933922 ± 2% perf-stat.ps.major-faults
629469 ± 5% +66.7% 1049045 ± 2% perf-stat.ps.minor-faults
1826569 ± 3% +8.6% 1982969 ± 2% perf-stat.ps.page-faults
7.476e+12 ± 3% -14.4% 6.403e+12 ± 2% perf-stat.total.instructions
104876 ± 15% -55.2% 46978 ± 8% proc-vmstat.allocstall_movable
6670 ± 13% +90.1% 12681 ± 23% proc-vmstat.allocstall_normal
270.00 ± 34% -62.4% 101.50 ± 33% proc-vmstat.compact_fail
277.50 ± 33% -61.0% 108.25 ± 36% proc-vmstat.compact_stall
629.25 ± 35% +596.1% 4380 ± 8% proc-vmstat.kswapd_low_wmark_hit_quickly
15034 ± 8% -34.2% 9894 ± 3% proc-vmstat.nr_active_anon
2432763 +222.5% 7846521 proc-vmstat.nr_active_file
2.229e+08 ± 2% +19.5% 2.664e+08 ± 2% proc-vmstat.nr_dirtied
1732719 -3.1% 1679210 proc-vmstat.nr_dirty
280680 -4.7% 267623 proc-vmstat.nr_free_pages
7100824 -77.3% 1613637 proc-vmstat.nr_inactive_file
164.75 ± 9% -54.9% 74.25 ± 8% proc-vmstat.nr_isolated_file
9582341 -2.0% 9389351 proc-vmstat.nr_mapped
121263 -4.0% 116373 proc-vmstat.nr_shmem
162508 +60.3% 260428 proc-vmstat.nr_slab_reclaimable
5464481 ± 4% +578.9% 37099437 ± 2% proc-vmstat.nr_vmscan_immediate_reclaim
639.25 ± 3% +43.5% 917.50 ± 4% proc-vmstat.nr_writeback
2.217e+08 ± 2% +19.4% 2.646e+08 ± 2% proc-vmstat.nr_written
15036 ± 8% -34.2% 9895 ± 3% proc-vmstat.nr_zone_active_anon
2434338 +222.3% 7846813 proc-vmstat.nr_zone_active_file
7101719 -77.2% 1615989 proc-vmstat.nr_zone_inactive_file
1737241 -3.1% 1683939 proc-vmstat.nr_zone_write_pending
40016131 ± 6% -16.6% 33361886 ± 6% proc-vmstat.numa_foreign
25745506 ± 14% +56.4% 40263559 ± 5% proc-vmstat.numa_hint_faults
19974893 ± 14% +49.8% 29926746 ± 8% proc-vmstat.numa_hint_faults_local
2.039e+08 ± 2% -22.5% 1.581e+08 ± 2% proc-vmstat.numa_hit
1808 ± 2% -3.3% 1749 proc-vmstat.numa_huge_pte_updates
2.039e+08 ± 2% -22.5% 1.58e+08 ± 2% proc-vmstat.numa_local
40016131 ± 6% -16.6% 33361886 ± 6% proc-vmstat.numa_miss
40047410 ± 6% -16.6% 33393063 ± 6% proc-vmstat.numa_other
251563 ± 29% +420.2% 1308599 ± 10% proc-vmstat.numa_pages_migrated
635.00 ± 35% +596.9% 4425 ± 8% proc-vmstat.pageoutrun
1.127e+08 ± 2% -55.0% 50732767 proc-vmstat.pgactivate
9619324 ± 4% -37.5% 6013199 ± 7% proc-vmstat.pgalloc_dma32
2.354e+08 ± 2% -20.7% 1.868e+08 ± 2% proc-vmstat.pgalloc_normal
1.829e+08 ± 2% -97.1% 5298999 ± 4% proc-vmstat.pgdeactivate
7.145e+08 ± 3% +6.8% 7.627e+08 ± 2% proc-vmstat.pgfault
2.353e+08 ± 2% -22.4% 1.827e+08 ± 3% proc-vmstat.pgfree
2.427e+08 ± 2% -22.0% 1.892e+08 ± 2% proc-vmstat.pgmajfault
427089 ± 38% +278.2% 1615323 ± 11% proc-vmstat.pgmigrate_success
9.71e+08 ± 2% -22.0% 7.569e+08 ± 2% proc-vmstat.pgpgin
8.867e+08 ± 2% +19.4% 1.059e+09 ± 2% proc-vmstat.pgpgout
1.829e+08 ± 2% -97.1% 5298820 ± 4% proc-vmstat.pgrefill
5485949 ± 3% +574.1% 36983260 ± 2% proc-vmstat.pgrotated
1.984e+08 ± 12% -78.9% 41928600 ± 14% proc-vmstat.pgscan_direct
3.395e+08 ± 3% +11.0% 3.768e+08 proc-vmstat.pgscan_kswapd
89080449 ± 12% -87.6% 11029910 ± 8% proc-vmstat.pgsteal_direct
1.439e+08 ± 3% +17.0% 1.684e+08 ± 2% proc-vmstat.pgsteal_kswapd
37738291 ± 5% -100.0% 0.00 proc-vmstat.workingset_activate
25988 ± 2% -22.6% 20107 ± 2% proc-vmstat.workingset_nodes
69506838 ± 2% -100.0% 0.00 proc-vmstat.workingset_refault
21650634 ± 5% -100.0% 1.50 ± 33% proc-vmstat.workingset_restore
87.50 ±168% -100.0% 0.00 interrupts.114:PCI-MSI.31981647-edge.i40e-eth0-TxRx-78
2.75 ±110% +5090.9% 142.75 ±103% interrupts.61:PCI-MSI.31981594-edge.i40e-eth0-TxRx-25
50.25 ±153% -98.5% 0.75 ±110% interrupts.83:PCI-MSI.31981616-edge.i40e-eth0-TxRx-47
0.25 ±173% +28300.0% 71.00 ±159% interrupts.86:PCI-MSI.31981619-edge.i40e-eth0-TxRx-50
43979343 ± 10% -35.3% 28436814 ± 6% interrupts.CAL:Function_call_interrupts
309869 ± 12% -48.7% 158885 ± 43% interrupts.CPU0.CAL:Function_call_interrupts
1496 ± 20% -59.7% 603.25 ± 30% interrupts.CPU0.NMI:Non-maskable_interrupts
1496 ± 20% -59.7% 603.25 ± 30% interrupts.CPU0.PMI:Performance_monitoring_interrupts
711.25 ± 39% +120.0% 1564 ± 46% interrupts.CPU0.RES:Rescheduling_interrupts
378389 ± 11% -40.7% 224400 ± 36% interrupts.CPU0.TLB:TLB_shootdowns
564422 ± 6% -51.1% 276198 ± 18% interrupts.CPU11.CAL:Function_call_interrupts
675725 ± 13% -47.0% 358389 ± 15% interrupts.CPU11.TLB:TLB_shootdowns
1834 ± 31% -58.9% 754.25 ± 16% interrupts.CPU12.NMI:Non-maskable_interrupts
1834 ± 31% -58.9% 754.25 ± 16% interrupts.CPU12.PMI:Performance_monitoring_interrupts
483343 ± 18% -43.3% 273844 ± 35% interrupts.CPU13.CAL:Function_call_interrupts
4095 ± 34% -72.2% 1137 ± 55% interrupts.CPU13.NMI:Non-maskable_interrupts
4095 ± 34% -72.2% 1137 ± 55% interrupts.CPU13.PMI:Performance_monitoring_interrupts
662959 ± 25% -52.6% 313982 ± 30% interrupts.CPU13.TLB:TLB_shootdowns
560563 ± 24% -49.1% 285449 ± 9% interrupts.CPU14.CAL:Function_call_interrupts
693290 ± 34% -47.1% 366618 ± 5% interrupts.CPU14.TLB:TLB_shootdowns
558990 ± 21% -61.4% 215573 ± 24% interrupts.CPU15.CAL:Function_call_interrupts
2204 ± 31% -67.0% 727.50 ± 35% interrupts.CPU15.NMI:Non-maskable_interrupts
2204 ± 31% -67.0% 727.50 ± 35% interrupts.CPU15.PMI:Performance_monitoring_interrupts
655709 ± 30% -64.3% 233774 ± 18% interrupts.CPU15.TLB:TLB_shootdowns
531963 ± 30% -51.7% 257180 ± 61% interrupts.CPU16.CAL:Function_call_interrupts
3104 ± 49% -66.2% 1050 ± 71% interrupts.CPU16.NMI:Non-maskable_interrupts
3104 ± 49% -66.2% 1050 ± 71% interrupts.CPU16.PMI:Performance_monitoring_interrupts
758.50 ± 52% +58.1% 1199 ± 18% interrupts.CPU16.RES:Rescheduling_interrupts
624333 ± 30% -53.8% 288559 ± 54% interrupts.CPU16.TLB:TLB_shootdowns
414942 ± 20% -51.1% 203078 ± 32% interrupts.CPU18.CAL:Function_call_interrupts
480458 ± 21% -52.4% 228645 ± 25% interrupts.CPU18.TLB:TLB_shootdowns
1109 ± 44% +205.5% 3390 ± 41% interrupts.CPU2.RES:Rescheduling_interrupts
461027 ± 25% -57.0% 198359 ± 30% interrupts.CPU20.CAL:Function_call_interrupts
3798 ± 63% -81.0% 720.75 ± 86% interrupts.CPU20.NMI:Non-maskable_interrupts
3798 ± 63% -81.0% 720.75 ± 86% interrupts.CPU20.PMI:Performance_monitoring_interrupts
564227 ± 37% -62.5% 211819 ± 31% interrupts.CPU20.TLB:TLB_shootdowns
513743 ± 29% -48.3% 265392 ± 32% interrupts.CPU21.CAL:Function_call_interrupts
644566 ± 43% -57.1% 276789 ± 32% interrupts.CPU21.TLB:TLB_shootdowns
503293 ± 25% -63.1% 185627 ± 52% interrupts.CPU22.CAL:Function_call_interrupts
645000 ± 13% -67.5% 209945 ± 56% interrupts.CPU22.TLB:TLB_shootdowns
553939 ± 27% -61.8% 211825 ± 28% interrupts.CPU23.CAL:Function_call_interrupts
687580 ± 25% -67.1% 225909 ± 29% interrupts.CPU23.TLB:TLB_shootdowns
474.00 ± 52% +258.5% 1699 ± 57% interrupts.CPU24.RES:Rescheduling_interrupts
2.25 ±123% +6233.3% 142.50 ±103% interrupts.CPU25.61:PCI-MSI.31981594-edge.i40e-eth0-TxRx-25
4043 ± 63% -71.3% 1160 ± 31% interrupts.CPU25.NMI:Non-maskable_interrupts
4043 ± 63% -71.3% 1160 ± 31% interrupts.CPU25.PMI:Performance_monitoring_interrupts
451692 ± 11% -39.6% 272701 ± 25% interrupts.CPU26.CAL:Function_call_interrupts
815.25 ± 39% +110.4% 1715 ± 29% interrupts.CPU26.RES:Rescheduling_interrupts
584283 ± 11% -41.2% 343585 ± 33% interrupts.CPU26.TLB:TLB_shootdowns
517137 ± 11% -41.0% 305137 ± 22% interrupts.CPU27.CAL:Function_call_interrupts
2414 ± 26% -45.2% 1323 ± 26% interrupts.CPU27.NMI:Non-maskable_interrupts
2414 ± 26% -45.2% 1323 ± 26% interrupts.CPU27.PMI:Performance_monitoring_interrupts
614104 ± 7% -39.7% 370314 ± 19% interrupts.CPU27.TLB:TLB_shootdowns
441205 ± 12% -32.1% 299453 ± 20% interrupts.CPU28.CAL:Function_call_interrupts
2273 ± 24% -39.7% 1369 ± 19% interrupts.CPU28.NMI:Non-maskable_interrupts
2273 ± 24% -39.7% 1369 ± 19% interrupts.CPU28.PMI:Performance_monitoring_interrupts
544289 ± 4% -31.6% 372065 ± 21% interrupts.CPU28.TLB:TLB_shootdowns
520217 ± 28% -31.2% 357653 ± 14% interrupts.CPU29.CAL:Function_call_interrupts
3744 ± 63% -55.3% 1674 ± 27% interrupts.CPU29.NMI:Non-maskable_interrupts
3744 ± 63% -55.3% 1674 ± 27% interrupts.CPU29.PMI:Performance_monitoring_interrupts
814.25 ± 53% +128.8% 1863 ± 39% interrupts.CPU29.RES:Rescheduling_interrupts
613340 ± 29% -31.8% 418352 ± 11% interrupts.CPU29.TLB:TLB_shootdowns
516303 ± 28% -48.8% 264227 ± 29% interrupts.CPU3.CAL:Function_call_interrupts
1951 ± 32% -71.4% 558.25 ± 28% interrupts.CPU3.NMI:Non-maskable_interrupts
1951 ± 32% -71.4% 558.25 ± 28% interrupts.CPU3.PMI:Performance_monitoring_interrupts
649962 ± 17% -54.0% 298822 ± 40% interrupts.CPU3.TLB:TLB_shootdowns
518481 ± 11% -22.2% 403431 ± 14% interrupts.CPU30.CAL:Function_call_interrupts
678.50 ± 47% +154.1% 1724 ± 16% interrupts.CPU30.RES:Rescheduling_interrupts
740472 ± 16% -40.8% 438014 ± 16% interrupts.CPU30.TLB:TLB_shootdowns
497599 ± 13% -26.9% 363979 ± 2% interrupts.CPU31.CAL:Function_call_interrupts
560143 ± 11% -21.8% 438031 ± 10% interrupts.CPU31.TLB:TLB_shootdowns
919.75 ± 37% +68.1% 1546 ± 16% interrupts.CPU32.RES:Rescheduling_interrupts
526266 ± 22% -40.5% 312935 ± 27% interrupts.CPU33.CAL:Function_call_interrupts
2250 ± 15% -38.3% 1387 ± 3% interrupts.CPU33.NMI:Non-maskable_interrupts
2250 ± 15% -38.3% 1387 ± 3% interrupts.CPU33.PMI:Performance_monitoring_interrupts
959.25 ± 31% +197.4% 2852 ± 59% interrupts.CPU33.RES:Rescheduling_interrupts
589843 ± 17% -40.4% 351483 ± 9% interrupts.CPU34.CAL:Function_call_interrupts
724580 ± 19% -43.0% 412731 ± 9% interrupts.CPU34.TLB:TLB_shootdowns
3061 ± 51% -66.5% 1026 ± 54% interrupts.CPU35.NMI:Non-maskable_interrupts
3061 ± 51% -66.5% 1026 ± 54% interrupts.CPU35.PMI:Performance_monitoring_interrupts
527518 ± 26% -41.3% 309832 ± 20% interrupts.CPU36.CAL:Function_call_interrupts
1935 ± 29% -41.1% 1139 ± 25% interrupts.CPU36.NMI:Non-maskable_interrupts
1935 ± 29% -41.1% 1139 ± 25% interrupts.CPU36.PMI:Performance_monitoring_interrupts
605919 ± 27% -33.5% 402973 ± 36% interrupts.CPU36.TLB:TLB_shootdowns
1968 ± 19% -43.1% 1120 ± 33% interrupts.CPU37.NMI:Non-maskable_interrupts
1968 ± 19% -43.1% 1120 ± 33% interrupts.CPU37.PMI:Performance_monitoring_interrupts
552091 ± 17% -37.9% 343088 ± 20% interrupts.CPU38.CAL:Function_call_interrupts
741448 ± 25% -42.9% 423500 ± 21% interrupts.CPU38.TLB:TLB_shootdowns
893.75 ± 22% +66.8% 1490 ± 34% interrupts.CPU39.RES:Rescheduling_interrupts
651520 ± 10% -32.9% 437091 ± 33% interrupts.CPU39.TLB:TLB_shootdowns
443156 ± 27% -74.1% 114892 ± 21% interrupts.CPU4.CAL:Function_call_interrupts
536804 ± 32% -73.9% 139975 ± 23% interrupts.CPU4.TLB:TLB_shootdowns
2376 ± 9% -31.2% 1635 ± 25% interrupts.CPU40.NMI:Non-maskable_interrupts
2376 ± 9% -31.2% 1635 ± 25% interrupts.CPU40.PMI:Performance_monitoring_interrupts
634311 ± 13% -30.5% 441067 ± 19% interrupts.CPU42.TLB:TLB_shootdowns
605811 ± 24% -44.1% 338572 ± 5% interrupts.CPU45.CAL:Function_call_interrupts
824.00 ± 43% +169.8% 2223 ± 39% interrupts.CPU45.RES:Rescheduling_interrupts
789721 ± 28% -45.9% 427130 ± 12% interrupts.CPU45.TLB:TLB_shootdowns
527252 ± 26% -41.4% 309202 ± 18% interrupts.CPU46.CAL:Function_call_interrupts
747063 ± 34% -55.4% 333517 ± 18% interrupts.CPU46.TLB:TLB_shootdowns
50.00 ±154% -99.5% 0.25 ±173% interrupts.CPU47.83:PCI-MSI.31981616-edge.i40e-eth0-TxRx-47
552975 ± 11% -35.1% 358953 ± 23% interrupts.CPU47.CAL:Function_call_interrupts
698382 ± 5% -34.2% 459438 ± 31% interrupts.CPU47.TLB:TLB_shootdowns
668725 ± 25% -71.0% 193871 ± 59% interrupts.CPU48.CAL:Function_call_interrupts
2351 ± 23% -63.0% 869.00 ± 74% interrupts.CPU48.NMI:Non-maskable_interrupts
2351 ± 23% -63.0% 869.00 ± 74% interrupts.CPU48.PMI:Performance_monitoring_interrupts
1036 ± 39% +143.7% 2524 ± 59% interrupts.CPU48.RES:Rescheduling_interrupts
723105 ± 28% -72.2% 200997 ± 57% interrupts.CPU48.TLB:TLB_shootdowns
681246 ± 37% -69.7% 206357 ± 66% interrupts.CPU49.CAL:Function_call_interrupts
734750 ± 36% -70.8% 214575 ± 68% interrupts.CPU49.TLB:TLB_shootdowns
532908 ± 33% -54.9% 240375 ± 53% interrupts.CPU50.CAL:Function_call_interrupts
622804 ± 27% -59.6% 251480 ± 56% interrupts.CPU50.TLB:TLB_shootdowns
2612 ± 56% -64.4% 929.00 ± 37% interrupts.CPU54.RES:Rescheduling_interrupts
1525 ± 19% -45.9% 825.25 ± 38% interrupts.CPU6.NMI:Non-maskable_interrupts
1525 ± 19% -45.9% 825.25 ± 38% interrupts.CPU6.PMI:Performance_monitoring_interrupts
658.25 ± 61% +183.8% 1868 ± 51% interrupts.CPU60.RES:Rescheduling_interrupts
1826 ± 29% -66.0% 620.75 ± 53% interrupts.CPU61.NMI:Non-maskable_interrupts
1826 ± 29% -66.0% 620.75 ± 53% interrupts.CPU61.PMI:Performance_monitoring_interrupts
2182 ± 35% -53.4% 1017 ± 20% interrupts.CPU62.NMI:Non-maskable_interrupts
2182 ± 35% -53.4% 1017 ± 20% interrupts.CPU62.PMI:Performance_monitoring_interrupts
479189 ± 47% -50.5% 237071 ± 37% interrupts.CPU64.TLB:TLB_shootdowns
417282 ± 22% -36.7% 263997 ± 22% interrupts.CPU7.CAL:Function_call_interrupts
826.50 ± 48% +73.7% 1435 ± 27% interrupts.CPU7.RES:Rescheduling_interrupts
533501 ± 13% -45.4% 291545 ± 28% interrupts.CPU7.TLB:TLB_shootdowns
618059 ± 18% -53.0% 290711 ± 27% interrupts.CPU72.CAL:Function_call_interrupts
680178 ± 18% -53.6% 315576 ± 36% interrupts.CPU72.TLB:TLB_shootdowns
2597 ± 16% -43.7% 1463 ± 24% interrupts.CPU73.NMI:Non-maskable_interrupts
2597 ± 16% -43.7% 1463 ± 24% interrupts.CPU73.PMI:Performance_monitoring_interrupts
459.50 ± 41% +431.0% 2440 ± 26% interrupts.CPU73.RES:Rescheduling_interrupts
521.75 ± 55% +281.9% 1992 ± 21% interrupts.CPU74.RES:Rescheduling_interrupts
2656 ± 21% -45.7% 1441 ± 33% interrupts.CPU75.NMI:Non-maskable_interrupts
2656 ± 21% -45.7% 1441 ± 33% interrupts.CPU75.PMI:Performance_monitoring_interrupts
524.75 ± 54% +141.9% 1269 ± 32% interrupts.CPU75.RES:Rescheduling_interrupts
87.00 ±169% -100.0% 0.00 interrupts.CPU78.114:PCI-MSI.31981647-edge.i40e-eth0-TxRx-78
650.00 ± 35% +102.7% 1317 ± 31% interrupts.CPU78.RES:Rescheduling_interrupts
903.25 ± 25% +76.1% 1590 ± 14% interrupts.CPU84.NMI:Non-maskable_interrupts
903.25 ± 25% +76.1% 1590 ± 14% interrupts.CPU84.PMI:Performance_monitoring_interrupts
421.00 ± 38% +147.6% 1042 ± 39% interrupts.CPU85.RES:Rescheduling_interrupts
807.25 ± 29% +56.5% 1263 ± 24% interrupts.CPU89.RES:Rescheduling_interrupts
521370 ± 3% -46.6% 278562 ± 38% interrupts.CPU9.CAL:Function_call_interrupts
613194 ± 20% -46.4% 328688 ± 32% interrupts.CPU9.TLB:TLB_shootdowns
440.00 ± 33% +163.4% 1158 ± 26% interrupts.CPU93.RES:Rescheduling_interrupts
208.75 ± 9% -40.4% 124.50 ± 3% interrupts.IWI:IRQ_work_interrupts
204364 ± 7% -31.0% 140939 ± 2% interrupts.NMI:Non-maskable_interrupts
204364 ± 7% -31.0% 140939 ± 2% interrupts.PMI:Performance_monitoring_interrupts
111435 ± 10% +35.2% 150649 ± 6% interrupts.RES:Rescheduling_interrupts
52728394 ± 10% -38.7% 32297958 ± 4% interrupts.TLB:TLB_shootdowns
22.74 ± 16% -18.0 4.75 ± 6% perf-profile.calltrace.cycles-pp.pagecache_get_page.filemap_fault.ext4_filemap_fault.__do_fault.handle_pte_fault
29.75 ± 13% -16.9 12.83 ± 2% perf-profile.calltrace.cycles-pp.filemap_fault.ext4_filemap_fault.__do_fault.handle_pte_fault.__handle_mm_fault
30.06 ± 13% -16.9 13.17 ± 2% perf-profile.calltrace.cycles-pp.__do_fault.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault
30.04 ± 13% -16.9 13.15 ± 2% perf-profile.calltrace.cycles-pp.ext4_filemap_fault.__do_fault.handle_pte_fault.__handle_mm_fault.handle_mm_fault
17.51 ± 18% -16.4 1.14 ± 28% perf-profile.calltrace.cycles-pp.__alloc_pages_slowpath.__alloc_pages_nodemask.pagecache_get_page.filemap_fault.ext4_filemap_fault
18.25 ± 17% -16.1 2.16 ± 12% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.filemap_fault.ext4_filemap_fault.__do_fault
15.88 ± 19% -15.2 0.66 ± 61% perf-profile.calltrace.cycles-pp.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.pagecache_get_page.filemap_fault
15.86 ± 19% -15.2 0.66 ± 61% perf-profile.calltrace.cycles-pp.shrink_node.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask
15.86 ± 19% -15.2 0.66 ± 61% perf-profile.calltrace.cycles-pp.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.pagecache_get_page
15.78 ± 19% -15.1 0.64 ± 61% perf-profile.calltrace.cycles-pp.shrink_lruvec.shrink_node.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath
37.45 ± 11% -14.3 23.12 ± 2% perf-profile.calltrace.cycles-pp.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault
38.00 ± 11% -14.0 24.01 ± 2% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault
38.59 ± 11% -13.7 24.93 ± 2% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault
39.53 ± 11% -13.0 26.49 ± 2% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
39.80 ± 11% -12.8 26.95 ± 2% perf-profile.calltrace.cycles-pp.page_fault
12.67 ± 19% -12.0 0.63 ± 61% perf-profile.calltrace.cycles-pp.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages.try_to_free_pages
6.30 ± 14% -6.2 0.13 ±173% perf-profile.calltrace.cycles-pp.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages
6.03 ± 27% -6.0 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node
5.53 ± 25% -5.5 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages
4.11 ± 12% -2.1 2.05 ± 3% perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.filemap_fault.ext4_filemap_fault.__do_fault
2.75 ± 28% -2.0 0.72 ± 12% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.pagevec_lru_move_fn.deactivate_file_page.invalidate_mapping_pages
2.85 ± 27% -2.0 0.83 ± 10% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.pagevec_lru_move_fn.deactivate_file_page.invalidate_mapping_pages.generic_fadvise
3.32 ± 9% -2.0 1.35 ± 14% perf-profile.calltrace.cycles-pp.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
3.19 ± 23% -1.8 1.35 ± 7% perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.deactivate_file_page.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64
3.36 ± 22% -1.7 1.70 ± 5% perf-profile.calltrace.cycles-pp.deactivate_file_page.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
2.33 ± 17% -1.3 0.98 ± 2% perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.filemap_fault
2.39 ± 16% -1.3 1.07 ± 3% perf-profile.calltrace.cycles-pp.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.filemap_fault.ext4_filemap_fault
2.16 ± 8% -0.7 1.50 ± 11% perf-profile.calltrace.cycles-pp.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
1.90 ± 9% -0.6 1.29 ± 12% perf-profile.calltrace.cycles-pp.rmap_walk_file.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec
1.69 ± 8% -0.4 1.33 ± 4% perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.flush_tlb_mm_range.ptep_clear_flush.page_mkclean_one.rmap_walk_file
1.75 ± 7% -0.3 1.44 ± 3% perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.ptep_clear_flush.page_mkclean_one.rmap_walk_file.page_mkclean
1.49 ± 8% -0.3 1.19 ± 6% perf-profile.calltrace.cycles-pp.smp_call_function_single.on_each_cpu_cond_mask.flush_tlb_mm_range.ptep_clear_flush.page_mkclean_one
1.82 ± 7% -0.3 1.54 ± 4% perf-profile.calltrace.cycles-pp.ptep_clear_flush.page_mkclean_one.rmap_walk_file.page_mkclean.clear_page_dirty_for_io
0.61 ± 8% +0.3 0.89 ± 8% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.pagecache_get_page.filemap_fault.ext4_filemap_fault
2.60 ± 8% +0.4 3.01 ± 3% perf-profile.calltrace.cycles-pp.filemap_map_pages.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault
2.08 ± 6% +0.4 2.51 ± 4% perf-profile.calltrace.cycles-pp.page_mkclean_one.rmap_walk_file.page_mkclean.clear_page_dirty_for_io.mpage_submit_page
0.73 ± 11% +0.4 1.16 ± 16% perf-profile.calltrace.cycles-pp.isolate_lru_pages.shrink_inactive_list.shrink_lruvec.shrink_node.balance_pgdat
0.93 ± 9% +0.5 1.38 ± 4% perf-profile.calltrace.cycles-pp.pagevec_lookup_entries.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
0.92 ± 9% +0.5 1.38 ± 3% perf-profile.calltrace.cycles-pp.find_get_entries.pagevec_lookup_entries.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64
0.77 ± 7% +0.5 1.24 ± 5% perf-profile.calltrace.cycles-pp.get_io_u
0.00 +0.6 0.61 ± 5% perf-profile.calltrace.cycles-pp.fault_dirty_shared_page.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault
2.46 ± 10% +0.6 3.08 ± 4% perf-profile.calltrace.cycles-pp.page_mkclean.clear_page_dirty_for_io.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map
2.23 ± 6% +0.6 2.85 ± 4% perf-profile.calltrace.cycles-pp.rmap_walk_file.page_mkclean.clear_page_dirty_for_io.mpage_submit_page.mpage_process_page_bufs
0.00 +0.6 0.63 ± 9% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
1.22 ± 10% +0.7 1.87 ± 4% perf-profile.calltrace.cycles-pp.block_page_mkwrite.ext4_page_mkwrite.do_page_mkwrite.do_wp_page.handle_pte_fault
4.32 ± 6% +0.7 4.98 ± 4% perf-profile.calltrace.cycles-pp.__memcpy_mcsafe.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio
4.41 ± 6% +0.7 5.08 ± 4% perf-profile.calltrace.cycles-pp.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio.ext4_mpage_readpages
3.50 ± 5% +0.7 4.20 ± 3% perf-profile.calltrace.cycles-pp.clear_page_dirty_for_io.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages
0.00 +0.8 0.77 ± 8% perf-profile.calltrace.cycles-pp.__set_page_dirty.mark_buffer_dirty.__block_commit_write.block_page_mkwrite.ext4_page_mkwrite
5.00 ± 6% +0.8 5.78 ± 4% perf-profile.calltrace.cycles-pp.pmem_make_request.generic_make_request.submit_bio.ext4_mpage_readpages.filemap_fault
0.39 ± 57% +0.8 1.20 ± 5% perf-profile.calltrace.cycles-pp.__block_commit_write.block_page_mkwrite.ext4_page_mkwrite.do_page_mkwrite.do_wp_page
0.00 +0.9 0.90 ± 4% perf-profile.calltrace.cycles-pp.__test_set_page_writeback.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map
5.42 ± 6% +0.9 6.33 ± 4% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.ext4_mpage_readpages.filemap_fault.ext4_filemap_fault
5.46 ± 6% +0.9 6.38 ± 4% perf-profile.calltrace.cycles-pp.submit_bio.ext4_mpage_readpages.filemap_fault.ext4_filemap_fault.__do_fault
0.00 +0.9 0.94 ± 23% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.75 ± 9% +1.0 1.78 ± 19% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
1.42 ± 10% +1.0 2.46 ± 3% perf-profile.calltrace.cycles-pp.ext4_page_mkwrite.do_page_mkwrite.do_wp_page.handle_pte_fault.__handle_mm_fault
1.44 ± 10% +1.1 2.50 ± 3% perf-profile.calltrace.cycles-pp.do_page_mkwrite.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault
6.32 ± 5% +1.1 7.40 ± 4% perf-profile.calltrace.cycles-pp.ext4_mpage_readpages.filemap_fault.ext4_filemap_fault.__do_fault.handle_pte_fault
0.42 ± 57% +1.1 1.53 ± 4% perf-profile.calltrace.cycles-pp.mark_buffer_dirty.__block_commit_write.block_page_mkwrite.ext4_page_mkwrite.do_page_mkwrite
0.83 ± 8% +1.1 1.95 ± 19% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
0.65 ± 4% +1.2 1.87 ± 11% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.ext4_finish_bio.ext4_end_bio.pmem_make_request
0.71 ± 5% +1.7 2.40 ± 3% perf-profile.calltrace.cycles-pp.end_page_writeback.ext4_finish_bio.ext4_end_bio.pmem_make_request.generic_make_request
1.92 ± 5% +1.7 3.61 perf-profile.calltrace.cycles-pp.ext4_finish_bio.ext4_end_bio.pmem_make_request.generic_make_request.submit_bio
1.93 ± 5% +1.7 3.63 perf-profile.calltrace.cycles-pp.ext4_end_bio.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit
1.93 ± 8% +1.8 3.68 ± 3% perf-profile.calltrace.cycles-pp.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault
8.26 ± 9% +1.8 10.04 ± 5% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.posix_fadvise
8.26 ± 9% +1.8 10.04 ± 5% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
8.24 ± 9% +1.8 10.02 ± 5% perf-profile.calltrace.cycles-pp.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
8.24 ± 9% +1.8 10.02 ± 5% perf-profile.calltrace.cycles-pp.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise
8.24 ± 9% +1.8 10.02 ± 5% perf-profile.calltrace.cycles-pp.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe
8.28 ± 9% +1.8 10.07 ± 5% perf-profile.calltrace.cycles-pp.posix_fadvise
3.07 ± 9% +2.1 5.21 ± 8% perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__filemap_fdatawrite_range
3.29 ± 9% +2.3 5.56 ± 8% perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__filemap_fdatawrite_range.generic_fadvise
3.30 ± 9% +2.3 5.59 ± 8% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64
3.30 ± 9% +2.3 5.59 ± 8% perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64
3.30 ± 9% +2.3 5.59 ± 8% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64
4.45 ± 11% +2.4 6.80 ± 13% perf-profile.calltrace.cycles-pp.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node.balance_pgdat
3.45 ± 16% +3.3 6.74 ± 3% perf-profile.calltrace.cycles-pp.__memcpy_flushcache.write_pmem.pmem_do_bvec.pmem_make_request.generic_make_request
3.49 ± 16% +3.3 6.81 ± 3% perf-profile.calltrace.cycles-pp.write_pmem.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio
3.51 ± 16% +3.3 6.86 ± 3% perf-profile.calltrace.cycles-pp.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit
7.42 ± 7% +4.7 12.16 ± 2% perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__writeback_single_inode
7.77 ± 6% +5.0 12.77 perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
7.94 ± 7% +5.0 12.96 perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
7.96 ± 7% +5.0 13.00 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback
7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb
7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work.worker_thread
7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work
7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn
5.48 ± 12% +5.1 10.57 ± 2% perf-profile.calltrace.cycles-pp.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit.ext4_bio_write_page
5.49 ± 12% +5.1 10.58 ± 2% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.ext4_io_submit.ext4_bio_write_page.mpage_submit_page
5.50 ± 12% +5.1 10.58 ± 2% perf-profile.calltrace.cycles-pp.submit_bio.ext4_io_submit.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs
5.50 ± 12% +5.1 10.58 ± 2% perf-profile.calltrace.cycles-pp.ext4_io_submit.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map
6.60 ± 11% +6.0 12.63 ± 2% perf-profile.calltrace.cycles-pp.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages
15.73 ± 8% +6.1 21.88 ± 5% perf-profile.calltrace.cycles-pp.ret_from_fork
15.73 ± 8% +6.1 21.88 ± 5% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
10.14 ± 7% +6.8 16.91 ± 2% perf-profile.calltrace.cycles-pp.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages
22.84 ± 16% -18.1 4.78 ± 6% perf-profile.children.cycles-pp.pagecache_get_page
29.77 ± 13% -16.9 12.85 ± 2% perf-profile.children.cycles-pp.filemap_fault
30.04 ± 13% -16.9 13.15 ± 2% perf-profile.children.cycles-pp.ext4_filemap_fault
30.07 ± 13% -16.9 13.18 ± 2% perf-profile.children.cycles-pp.__do_fault
18.08 ± 18% -16.8 1.31 ± 26% perf-profile.children.cycles-pp.__alloc_pages_slowpath
18.83 ± 17% -16.5 2.35 ± 14% perf-profile.children.cycles-pp.__alloc_pages_nodemask
16.42 ± 19% -15.5 0.89 ± 37% perf-profile.children.cycles-pp.try_to_free_pages
16.40 ± 19% -15.5 0.89 ± 37% perf-profile.children.cycles-pp.do_try_to_free_pages
24.16 ± 14% -14.4 9.73 ± 11% perf-profile.children.cycles-pp.shrink_node
24.07 ± 14% -14.4 9.67 ± 11% perf-profile.children.cycles-pp.shrink_lruvec
37.48 ± 11% -14.3 23.17 ± 2% perf-profile.children.cycles-pp.handle_pte_fault
38.05 ± 11% -14.0 24.07 ± 2% perf-profile.children.cycles-pp.__handle_mm_fault
38.64 ± 11% -13.6 25.00 ± 2% perf-profile.children.cycles-pp.handle_mm_fault
39.58 ± 11% -13.0 26.58 ± 2% perf-profile.children.cycles-pp.do_page_fault
39.84 ± 11% -12.8 27.01 ± 2% perf-profile.children.cycles-pp.page_fault
15.30 ± 22% -12.8 2.48 ± 12% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
19.34 ± 14% -9.8 9.52 ± 10% perf-profile.children.cycles-pp.shrink_inactive_list
8.85 ± 23% -8.1 0.72 ± 23% perf-profile.children.cycles-pp._raw_spin_lock_irq
4.72 ± 13% -4.6 0.11 ± 68% perf-profile.children.cycles-pp.shrink_active_list
6.92 ± 18% -4.0 2.88 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
11.04 ± 10% -3.8 7.26 ± 11% perf-profile.children.cycles-pp.shrink_page_list
5.65 ± 20% -3.0 2.67 ± 3% perf-profile.children.cycles-pp.pagevec_lru_move_fn
4.12 ± 12% -2.1 2.06 ± 3% perf-profile.children.cycles-pp.add_to_page_cache_lru
3.42 ± 10% -1.9 1.51 ± 13% perf-profile.children.cycles-pp.__remove_mapping
3.37 ± 22% -1.7 1.70 ± 5% perf-profile.children.cycles-pp.deactivate_file_page
3.11 ± 15% -1.5 1.57 ± 5% perf-profile.children.cycles-pp.on_each_cpu_cond_mask
2.41 ± 16% -1.3 1.08 ± 2% perf-profile.children.cycles-pp.__lru_cache_add
2.96 ± 10% -1.3 1.66 ± 9% perf-profile.children.cycles-pp.page_referenced
1.43 ± 25% -1.2 0.23 ± 11% perf-profile.children.cycles-pp.try_to_unmap_flush
1.43 ± 25% -1.2 0.23 ± 11% perf-profile.children.cycles-pp.arch_tlbbatch_flush
1.29 ± 24% -1.0 0.31 ± 15% perf-profile.children.cycles-pp.smp_call_function_many_cond
1.64 ± 10% -0.7 0.92 ± 8% perf-profile.children.cycles-pp.page_referenced_one
0.80 ± 9% -0.7 0.14 ± 23% perf-profile.children.cycles-pp.workingset_refault
0.80 ± 13% -0.6 0.18 ± 15% perf-profile.children.cycles-pp.wake_all_kswapds
0.78 ± 14% -0.6 0.17 ± 16% perf-profile.children.cycles-pp.wakeup_kswapd
1.80 ± 10% -0.6 1.23 ± 5% perf-profile.children.cycles-pp.smp_call_function_single
1.73 ± 11% -0.5 1.22 ± 15% perf-profile.children.cycles-pp.isolate_lru_pages
1.19 ± 20% -0.5 0.68 ± 29% perf-profile.children.cycles-pp.create_empty_buffers
1.21 ± 19% -0.5 0.71 ± 28% perf-profile.children.cycles-pp.create_page_buffers
1.74 ± 10% -0.5 1.25 ± 5% perf-profile.children.cycles-pp.page_vma_mapped_walk
0.85 ± 26% -0.4 0.44 ± 46% perf-profile.children.cycles-pp.alloc_page_buffers
1.61 ± 10% -0.4 1.22 ± 9% perf-profile.children.cycles-pp.__list_del_entry_valid
0.80 ± 28% -0.4 0.40 ± 50% perf-profile.children.cycles-pp.alloc_buffer_head
0.64 ± 32% -0.4 0.27 ± 71% perf-profile.children.cycles-pp.__slab_alloc
1.54 ± 17% -0.4 1.17 ± 16% perf-profile.children.cycles-pp.__block_write_begin_int
0.86 ± 26% -0.4 0.49 ± 40% perf-profile.children.cycles-pp.kmem_cache_alloc
0.63 ± 32% -0.4 0.27 ± 73% perf-profile.children.cycles-pp.___slab_alloc
0.81 ± 10% -0.4 0.46 ± 16% perf-profile.children.cycles-pp.move_pages_to_lru
0.48 ± 43% -0.3 0.15 ±138% perf-profile.children.cycles-pp.new_slab
0.35 ± 10% -0.3 0.09 ± 15% perf-profile.children.cycles-pp.advance_inactive_age
0.40 ± 8% -0.2 0.19 ± 9% perf-profile.children.cycles-pp.workingset_eviction
0.49 ± 8% -0.2 0.30 ± 13% perf-profile.children.cycles-pp.try_to_free_buffers
0.35 ± 10% -0.2 0.19 ± 21% perf-profile.children.cycles-pp.jbd2_journal_try_to_free_buffers
0.33 ± 8% -0.2 0.18 ± 22% perf-profile.children.cycles-pp.jbd2_journal_grab_journal_head
0.37 ± 9% -0.1 0.26 ± 4% perf-profile.children.cycles-pp.page_add_file_rmap
0.32 ± 13% -0.1 0.20 ± 18% perf-profile.children.cycles-pp.page_evictable
0.28 ± 12% -0.1 0.19 ± 19% perf-profile.children.cycles-pp.page_remove_rmap
0.14 ± 6% -0.1 0.07 ± 12% perf-profile.children.cycles-pp.vma_interval_tree_iter_first
0.09 ± 11% -0.1 0.03 ±100% perf-profile.children.cycles-pp.get_partial_node
0.17 ± 10% -0.1 0.11 ± 15% perf-profile.children.cycles-pp.drop_buffers
0.19 ± 13% -0.1 0.12 ± 12% perf-profile.children.cycles-pp.free_buffer_head
0.24 ± 10% -0.0 0.19 ± 3% perf-profile.children.cycles-pp.ext4_da_get_block_prep
0.15 ± 10% -0.0 0.10 ± 12% perf-profile.children.cycles-pp.mem_cgroup_page_lruvec
0.12 ± 14% -0.0 0.08 ± 15% perf-profile.children.cycles-pp.__slab_free
0.10 ± 11% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.ptep_clear_flush_young
0.06 ± 9% +0.0 0.07 ± 5% perf-profile.children.cycles-pp.__indirect_thunk_start
0.06 ± 11% +0.0 0.09 ± 12% perf-profile.children.cycles-pp.generic_update_time
0.08 ± 11% +0.0 0.11 ± 8% perf-profile.children.cycles-pp.fput_many
0.14 ± 7% +0.0 0.17 ± 7% perf-profile.children.cycles-pp.mempool_alloc
0.13 ± 6% +0.0 0.17 ± 6% perf-profile.children.cycles-pp._find_next_bit
0.09 ± 24% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.try_to_wake_up
0.08 ± 5% +0.0 0.11 ± 14% perf-profile.children.cycles-pp.task_tick_fair
0.07 ± 19% +0.0 0.10 ± 10% perf-profile.children.cycles-pp.io_serial_in
0.04 ± 58% +0.0 0.08 ± 10% perf-profile.children.cycles-pp.ext4_dirty_inode
0.06 ± 9% +0.0 0.09 ± 8% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
0.09 ± 19% +0.0 0.13 ± 18% perf-profile.children.cycles-pp.wait_for_xmitr
0.09 ± 19% +0.0 0.13 ± 18% perf-profile.children.cycles-pp.serial8250_console_putchar
0.06 ± 7% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.__unlock_page_memcg
0.07 ± 6% +0.0 0.11 ± 6% perf-profile.children.cycles-pp.pmd_devmap_trans_unstable
0.09 ± 5% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.bio_add_page
0.11 ± 7% +0.0 0.15 ± 10% perf-profile.children.cycles-pp.PageHuge
0.10 ± 21% +0.0 0.14 ± 16% perf-profile.children.cycles-pp.serial8250_console_write
0.07 ± 7% +0.0 0.11 ± 13% perf-profile.children.cycles-pp.schedule
0.07 ± 5% +0.0 0.12 ± 11% perf-profile.children.cycles-pp.ext4_convert_inline_data
0.13 ± 14% +0.0 0.18 ± 10% perf-profile.children.cycles-pp.blk_throtl_bio
0.06 ± 20% +0.0 0.11 ± 7% perf-profile.children.cycles-pp.native_write_msr
0.17 ± 6% +0.0 0.22 ± 8% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
0.10 ± 12% +0.0 0.15 ± 8% perf-profile.children.cycles-pp.read_tsc
0.01 ±173% +0.1 0.06 ± 13% perf-profile.children.cycles-pp.update_load_avg
0.04 ± 57% +0.1 0.09 ± 13% perf-profile.children.cycles-pp.__next_timer_interrupt
0.11 ± 17% +0.1 0.16 ± 6% perf-profile.children.cycles-pp.irq_work_run_list
0.01 ±173% +0.1 0.07 ± 23% perf-profile.children.cycles-pp.pick_next_task_fair
0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.__bio_try_merge_page
0.10 ± 17% +0.1 0.15 ± 7% perf-profile.children.cycles-pp.console_unlock
0.06 ± 9% +0.1 0.11 ± 11% perf-profile.children.cycles-pp.page_mapped
0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.sched_clock_cpu
0.22 ± 9% +0.1 0.28 perf-profile.children.cycles-pp.bio_alloc_bioset
0.43 ± 9% +0.1 0.49 ± 3% perf-profile.children.cycles-pp.__mod_memcg_state
0.10 ± 7% +0.1 0.16 ± 5% perf-profile.children.cycles-pp.vmacache_find
0.08 ± 16% +0.1 0.14 ± 6% perf-profile.children.cycles-pp.lru_deactivate_file_fn
0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp._raw_spin_trylock
0.01 ±173% +0.1 0.07 ± 10% perf-profile.children.cycles-pp.mem_cgroup_wb_domain
0.12 ± 13% +0.1 0.18 ± 6% perf-profile.children.cycles-pp.xas_clear_mark
0.08 ± 10% +0.1 0.14 ± 5% perf-profile.children.cycles-pp.down_read_trylock
0.03 ±100% +0.1 0.09 ± 9% perf-profile.children.cycles-pp.lapic_next_deadline
0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.note_gp_changes
0.06 ± 13% +0.1 0.12 ± 11% perf-profile.children.cycles-pp.__sb_start_write
0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.ext4_inode_journal_mode
0.32 ± 11% +0.1 0.39 ± 3% perf-profile.children.cycles-pp.find_get_entry
0.00 +0.1 0.07 ± 17% perf-profile.children.cycles-pp.migrate_pages
0.04 ± 57% +0.1 0.11 ± 15% perf-profile.children.cycles-pp.load_balance
0.00 +0.1 0.07 ± 6% perf-profile.children.cycles-pp.current_time
0.00 +0.1 0.07 ± 6% perf-profile.children.cycles-pp.run_timer_softirq
0.08 +0.1 0.15 ± 10% perf-profile.children.cycles-pp.balance_dirty_pages
0.07 ± 13% +0.1 0.14 ± 6% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.04 ± 58% +0.1 0.11 ± 11% perf-profile.children.cycles-pp.inc_zone_page_state
0.01 ±173% +0.1 0.09 ± 10% perf-profile.children.cycles-pp.ktime_get_update_offsets_now
0.11 ± 3% +0.1 0.19 ± 4% perf-profile.children.cycles-pp.find_vma
0.01 ±173% +0.1 0.09 ± 9% perf-profile.children.cycles-pp.xas_set_mark
0.23 ± 7% +0.1 0.31 ± 5% perf-profile.children.cycles-pp.xas_find
0.07 ± 12% +0.1 0.15 ± 7% perf-profile.children.cycles-pp.__fprop_inc_percpu_max
0.21 ± 7% +0.1 0.29 ± 5% perf-profile.children.cycles-pp.__might_sleep
0.29 ± 11% +0.1 0.36 ± 9% perf-profile.children.cycles-pp._cond_resched
0.00 +0.1 0.08 ± 19% perf-profile.children.cycles-pp.find_busiest_group
0.18 ± 6% +0.1 0.26 ± 4% perf-profile.children.cycles-pp.xas_start
0.13 ± 6% +0.1 0.22 ± 14% perf-profile.children.cycles-pp.scheduler_tick
0.12 ± 15% +0.1 0.20 ± 7% perf-profile.children.cycles-pp.__mark_inode_dirty
0.13 ± 12% +0.1 0.21 ± 6% perf-profile.children.cycles-pp.file_update_time
0.00 +0.1 0.09 ± 18% perf-profile.children.cycles-pp.migrate_misplaced_page
0.04 ± 57% +0.1 0.13 ± 19% perf-profile.children.cycles-pp.rebalance_domains
0.08 ± 5% +0.1 0.18 ± 8% perf-profile.children.cycles-pp.invalidate_inode_page
0.04 ± 58% +0.1 0.14 ± 3% perf-profile.children.cycles-pp.finish_mkwrite_fault
0.35 ± 8% +0.1 0.45 ± 4% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.12 ± 16% +0.1 0.23 ± 22% perf-profile.children.cycles-pp.tick_nohz_next_event
0.26 ± 7% +0.1 0.38 ± 8% perf-profile.children.cycles-pp.flush_tlb_func_common
0.12 ± 6% +0.1 0.24 ± 3% perf-profile.children.cycles-pp.set_page_dirty
0.35 ± 6% +0.1 0.47 ± 5% perf-profile.children.cycles-pp.generic_make_request_checks
0.20 ± 7% +0.1 0.31 ± 6% perf-profile.children.cycles-pp.native_flush_tlb_one_user
0.14 ± 13% +0.1 0.26 ± 8% perf-profile.children.cycles-pp.dec_zone_page_state
0.45 ± 6% +0.1 0.57 ± 7% perf-profile.children.cycles-pp.___might_sleep
0.14 ± 17% +0.1 0.27 ± 22% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.21 ± 2% +0.1 0.33 ± 9% perf-profile.children.cycles-pp.___perf_sw_event
0.14 ± 10% +0.1 0.27 ± 5% perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
0.11 ± 7% +0.1 0.24 ± 6% perf-profile.children.cycles-pp.__xa_set_mark
0.14 ± 23% +0.1 0.27 ± 11% perf-profile.children.cycles-pp.rcu_core
0.11 ± 6% +0.1 0.25 ± 6% perf-profile.children.cycles-pp.__xa_clear_mark
0.03 ±100% +0.1 0.17 ± 42% perf-profile.children.cycles-pp.tick_irq_enter
0.04 ± 58% +0.2 0.20 ± 35% perf-profile.children.cycles-pp.irq_enter
0.00 +0.2 0.17 ± 6% perf-profile.children.cycles-pp.pagevec_move_tail_fn
0.18 ± 15% +0.2 0.36 ± 5% perf-profile.children.cycles-pp.__pagevec_release
0.31 ± 2% +0.2 0.51 ± 7% perf-profile.children.cycles-pp.__perf_sw_event
0.24 ± 11% +0.2 0.43 ± 15% perf-profile.children.cycles-pp.update_process_times
0.52 ± 7% +0.2 0.72 ± 6% perf-profile.children.cycles-pp.account_page_dirtied
0.24 ± 12% +0.2 0.45 ± 16% perf-profile.children.cycles-pp.tick_sched_handle
0.33 ± 2% +0.2 0.55 ± 7% perf-profile.children.cycles-pp.find_get_pages_range_tag
0.33 ± 2% +0.2 0.55 ± 6% perf-profile.children.cycles-pp.pagevec_lookup_range_tag
0.94 ± 7% +0.2 1.17 ± 2% perf-profile.children.cycles-pp.unlock_page
0.05 ± 59% +0.2 0.29 ± 8% perf-profile.children.cycles-pp.pagevec_move_tail
0.53 ± 9% +0.2 0.76 ± 5% perf-profile.children.cycles-pp.page_mapping
0.29 ± 7% +0.3 0.56 ± 18% perf-profile.children.cycles-pp.tick_sched_timer
0.22 ± 13% +0.3 0.49 ± 22% perf-profile.children.cycles-pp.clockevents_program_event
0.05 ± 59% +0.3 0.33 ± 5% perf-profile.children.cycles-pp.rotate_reclaimable_page
0.29 ± 12% +0.3 0.57 ± 14% perf-profile.children.cycles-pp.__softirqentry_text_start
0.42 ± 13% +0.3 0.72 ± 4% perf-profile.children.cycles-pp.release_pages
2.77 ± 5% +0.3 3.08 ± 3% perf-profile.children.cycles-pp.page_mkclean
0.43 ± 6% +0.3 0.78 ± 4% perf-profile.children.cycles-pp.percpu_counter_add_batch
0.39 ± 6% +0.3 0.74 ± 15% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.39 ± 8% +0.4 0.75 ± 18% perf-profile.children.cycles-pp.irq_exit
0.28 ± 15% +0.4 0.64 ± 8% perf-profile.children.cycles-pp.menu_select
0.75 ± 7% +0.4 1.16 ± 5% perf-profile.children.cycles-pp.__set_page_dirty
2.63 ± 8% +0.4 3.04 ± 3% perf-profile.children.cycles-pp.filemap_map_pages
0.72 ± 8% +0.4 1.14 ± 3% perf-profile.children.cycles-pp.xas_load
0.42 ± 5% +0.4 0.86 ± 5% perf-profile.children.cycles-pp.fault_dirty_shared_page
0.39 ± 7% +0.4 0.84 ± 15% perf-profile.children.cycles-pp.ktime_get
0.93 ± 9% +0.5 1.38 ± 4% perf-profile.children.cycles-pp.find_get_entries
0.61 ± 7% +0.5 1.07 ± 4% perf-profile.children.cycles-pp.sync_regs
0.93 ± 9% +0.5 1.38 ± 3% perf-profile.children.cycles-pp.pagevec_lookup_entries
0.78 ± 7% +0.5 1.26 ± 5% perf-profile.children.cycles-pp.get_io_u
0.46 ± 7% +0.5 0.97 ± 4% perf-profile.children.cycles-pp.__set_page_dirty_buffers
0.96 ± 8% +0.6 1.54 ± 3% perf-profile.children.cycles-pp.mark_buffer_dirty
2.93 ± 12% +0.6 3.55 ± 4% perf-profile.children.cycles-pp.block_page_mkwrite
0.70 ± 4% +0.6 1.33 ± 2% perf-profile.children.cycles-pp.__test_set_page_writeback
4.37 ± 6% +0.7 5.02 ± 4% perf-profile.children.cycles-pp.__memcpy_mcsafe
1.09 ± 9% +0.7 1.76 ± 5% perf-profile.children.cycles-pp.native_irq_return_iret
0.69 ± 8% +0.7 1.39 ± 15% perf-profile.children.cycles-pp.hrtimer_interrupt
1.11 ± 8% +0.7 1.81 ± 3% perf-profile.children.cycles-pp.__block_commit_write
3.53 ± 5% +0.7 4.27 ± 3% perf-profile.children.cycles-pp.clear_page_dirty_for_io
3.40 ± 12% +1.0 4.42 ± 3% perf-profile.children.cycles-pp.ext4_page_mkwrite
3.42 ± 12% +1.0 4.46 ± 3% perf-profile.children.cycles-pp.do_page_mkwrite
1.00 ± 6% +1.1 2.06 ± 3% perf-profile.children.cycles-pp.test_clear_page_writeback
6.32 ± 5% +1.1 7.41 ± 4% perf-profile.children.cycles-pp.ext4_mpage_readpages
1.15 ± 8% +1.2 2.36 ± 14% perf-profile.children.cycles-pp.smp_apic_timer_interrupt
1.26 ± 7% +1.3 2.59 ± 13% perf-profile.children.cycles-pp.apic_timer_interrupt
1.07 ± 7% +1.3 2.42 ± 3% perf-profile.children.cycles-pp.end_page_writeback
1.94 ± 5% +1.7 3.66 perf-profile.children.cycles-pp.ext4_end_bio
1.94 ± 5% +1.7 3.66 perf-profile.children.cycles-pp.ext4_finish_bio
8.51 ± 9% +1.7 10.24 ± 5% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
8.51 ± 9% +1.7 10.24 ± 5% perf-profile.children.cycles-pp.do_syscall_64
1.94 ± 8% +1.8 3.69 ± 4% perf-profile.children.cycles-pp.do_wp_page
8.24 ± 9% +1.8 10.02 ± 5% perf-profile.children.cycles-pp.__x64_sys_fadvise64
8.24 ± 9% +1.8 10.02 ± 5% perf-profile.children.cycles-pp.ksys_fadvise64_64
8.24 ± 9% +1.8 10.02 ± 5% perf-profile.children.cycles-pp.generic_fadvise
8.28 ± 9% +1.8 10.07 ± 5% perf-profile.children.cycles-pp.posix_fadvise
3.30 ± 9% +2.3 5.59 ± 8% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
3.47 ± 16% +3.3 6.81 ± 3% perf-profile.children.cycles-pp.__memcpy_flushcache
3.50 ± 16% +3.4 6.88 ± 3% perf-profile.children.cycles-pp.write_pmem
7.95 ± 7% +4.1 12.00 perf-profile.children.cycles-pp.pmem_do_bvec
7.94 ± 7% +5.0 12.96 perf-profile.children.cycles-pp.process_one_work
7.96 ± 7% +5.0 13.00 perf-profile.children.cycles-pp.worker_thread
7.79 ± 6% +5.1 12.85 perf-profile.children.cycles-pp.wb_workfn
7.79 ± 6% +5.1 12.85 perf-profile.children.cycles-pp.wb_writeback
7.79 ± 6% +5.1 12.85 perf-profile.children.cycles-pp.__writeback_inodes_wb
7.79 ± 6% +5.1 12.85 perf-profile.children.cycles-pp.writeback_sb_inodes
7.79 ± 6% +5.1 12.85 perf-profile.children.cycles-pp.__writeback_single_inode
5.52 ± 12% +5.2 10.67 ± 2% perf-profile.children.cycles-pp.ext4_io_submit
10.52 ± 7% +5.9 16.44 perf-profile.children.cycles-pp.pmem_make_request
6.60 ± 11% +6.0 12.63 ± 2% perf-profile.children.cycles-pp.ext4_bio_write_page
10.95 ± 7% +6.1 17.01 perf-profile.children.cycles-pp.generic_make_request
10.98 ± 6% +6.1 17.05 perf-profile.children.cycles-pp.submit_bio
15.73 ± 8% +6.1 21.88 ± 5% perf-profile.children.cycles-pp.ret_from_fork
15.73 ± 8% +6.1 21.88 ± 5% perf-profile.children.cycles-pp.kthread
10.14 ± 7% +6.8 16.92 ± 2% perf-profile.children.cycles-pp.mpage_submit_page
10.50 ± 7% +6.9 17.36 ± 2% perf-profile.children.cycles-pp.mpage_process_page_bufs
11.06 ± 7% +7.3 18.36 ± 2% perf-profile.children.cycles-pp.mpage_prepare_extent_to_map
11.09 ± 7% +7.4 18.45 ± 2% perf-profile.children.cycles-pp.do_writepages
11.09 ± 7% +7.4 18.45 ± 2% perf-profile.children.cycles-pp.ext4_writepages
15.29 ± 22% -12.8 2.47 ± 12% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
1.16 ± 26% -1.0 0.20 ± 21% perf-profile.self.cycles-pp.smp_call_function_many_cond
0.76 ± 13% -0.6 0.17 ± 17% perf-profile.self.cycles-pp.wakeup_kswapd
1.66 ± 10% -0.6 1.09 ± 5% perf-profile.self.cycles-pp.smp_call_function_single
1.60 ± 10% -0.4 1.21 ± 9% perf-profile.self.cycles-pp.__list_del_entry_valid
0.42 ± 8% -0.3 0.10 ± 23% perf-profile.self.cycles-pp.workingset_refault
0.34 ± 10% -0.2 0.09 ± 15% perf-profile.self.cycles-pp.advance_inactive_age
0.99 ± 9% -0.2 0.79 ± 7% perf-profile.self.cycles-pp.page_vma_mapped_walk
0.74 ± 10% -0.2 0.55 ± 5% perf-profile.self.cycles-pp.get_page_from_freelist
0.43 ± 9% -0.2 0.25 ± 16% perf-profile.self.cycles-pp.move_pages_to_lru
0.62 ± 9% -0.2 0.47 ± 7% perf-profile.self.cycles-pp.__mod_lruvec_state
0.32 ± 9% -0.1 0.17 ± 25% perf-profile.self.cycles-pp.jbd2_journal_grab_journal_head
0.37 ± 9% -0.1 0.24 ± 18% perf-profile.self.cycles-pp.__remove_mapping
0.37 ± 10% -0.1 0.26 ± 10% perf-profile.self.cycles-pp.page_referenced_one
0.23 ± 12% -0.1 0.13 ± 17% perf-profile.self.cycles-pp.page_evictable
0.17 ± 6% -0.1 0.10 ± 11% perf-profile.self.cycles-pp.workingset_eviction
0.16 ± 9% -0.1 0.10 ± 11% perf-profile.self.cycles-pp.page_add_file_rmap
0.14 ± 15% -0.1 0.08 ± 6% perf-profile.self.cycles-pp.pagecache_get_page
0.17 ± 12% -0.1 0.11 ± 15% perf-profile.self.cycles-pp.drop_buffers
0.12 ± 5% -0.1 0.06 ± 14% perf-profile.self.cycles-pp.vma_interval_tree_iter_first
0.26 ± 11% -0.1 0.20 ± 9% perf-profile.self.cycles-pp.isolate_lru_pages
0.12 ± 11% -0.1 0.06 ± 13% perf-profile.self.cycles-pp.page_referenced
0.14 ± 8% -0.1 0.09 ± 13% perf-profile.self.cycles-pp.mem_cgroup_page_lruvec
0.12 ± 11% -0.1 0.07 ± 12% perf-profile.self.cycles-pp.page_remove_rmap
0.12 ± 14% -0.0 0.08 ± 15% perf-profile.self.cycles-pp.__slab_free
0.10 ± 11% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.ptep_clear_flush_young
0.08 ± 10% -0.0 0.06 perf-profile.self.cycles-pp.ext4_da_get_block_prep
0.10 ± 7% +0.0 0.12 ± 3% perf-profile.self.cycles-pp.rcu_all_qs
0.06 ± 6% +0.0 0.08 ± 5% perf-profile.self.cycles-pp.__lru_cache_add
0.07 ± 17% +0.0 0.10 ± 8% perf-profile.self.cycles-pp.xas_find
0.07 ± 10% +0.0 0.10 ± 8% perf-profile.self.cycles-pp.fput_many
0.12 ± 8% +0.0 0.16 ± 7% perf-profile.self.cycles-pp._find_next_bit
0.15 ± 8% +0.0 0.18 ± 7% perf-profile.self.cycles-pp.alloc_set_pte
0.08 ± 10% +0.0 0.12 ± 7% perf-profile.self.cycles-pp.generic_make_request_checks
0.08 ± 14% +0.0 0.11 ± 6% perf-profile.self.cycles-pp.pmem_do_bvec
0.06 ± 14% +0.0 0.09 ± 4% perf-profile.self.cycles-pp.lru_deactivate_file_fn
0.07 ± 19% +0.0 0.10 ± 10% perf-profile.self.cycles-pp.io_serial_in
0.08 ± 19% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.free_unref_page_list
0.09 ± 11% +0.0 0.13 ± 11% perf-profile.self.cycles-pp.PageHuge
0.06 ± 6% +0.0 0.10 ± 7% perf-profile.self.cycles-pp.page_mkclean_one
0.07 ± 6% +0.0 0.11 ± 8% perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
0.41 ± 6% +0.0 0.45 ± 8% perf-profile.self.cycles-pp.filemap_fault
0.12 ± 13% +0.0 0.15 ± 7% perf-profile.self.cycles-pp.xas_store
0.10 ± 4% +0.0 0.14 ± 10% perf-profile.self.cycles-pp.ptep_clear_flush
0.06 ± 7% +0.0 0.10 ± 11% perf-profile.self.cycles-pp.pmd_devmap_trans_unstable
0.04 ± 57% +0.0 0.08 ± 5% perf-profile.self.cycles-pp.__unlock_page_memcg
0.10 ± 12% +0.0 0.14 ± 9% perf-profile.self.cycles-pp.read_tsc
0.08 ± 14% +0.0 0.12 ± 4% perf-profile.self.cycles-pp.down_read_trylock
0.07 +0.0 0.12 ± 9% perf-profile.self.cycles-pp.ext4_convert_inline_data
0.07 ± 11% +0.0 0.12 ± 5% perf-profile.self.cycles-pp.flush_tlb_mm_range
0.06 ± 6% +0.0 0.11 ± 6% perf-profile.self.cycles-pp.set_page_dirty
0.07 ± 17% +0.0 0.11 ± 9% perf-profile.self.cycles-pp.pagevec_lru_move_fn
0.06 ± 13% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.__mark_inode_dirty
0.06 ± 20% +0.0 0.11 ± 7% perf-profile.self.cycles-pp.native_write_msr
0.42 ± 8% +0.1 0.47 ± 3% perf-profile.self.cycles-pp.__mod_memcg_state
0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.__indirect_thunk_start
0.12 ± 13% +0.1 0.18 ± 14% perf-profile.self.cycles-pp._cond_resched
0.10 ± 9% +0.1 0.15 ± 4% perf-profile.self.cycles-pp.vmacache_find
0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.note_gp_changes
0.11 ± 11% +0.1 0.17 ± 10% perf-profile.self.cycles-pp.xas_clear_mark
0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.write_pmem
0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp._raw_spin_trylock
0.01 ±173% +0.1 0.07 ± 10% perf-profile.self.cycles-pp.mem_cgroup_wb_domain
0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.pagevec_move_tail_fn
0.00 +0.1 0.06 ± 13% perf-profile.self.cycles-pp.__sb_start_write
0.00 +0.1 0.06 ± 17% perf-profile.self.cycles-pp.balance_dirty_pages
0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp.ext4_inode_journal_mode
0.18 ± 8% +0.1 0.25 ± 5% perf-profile.self.cycles-pp.__might_sleep
0.04 ± 59% +0.1 0.11 ± 9% perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited
0.13 ± 8% +0.1 0.20 ± 10% perf-profile.self.cycles-pp.pmem_make_request
0.01 ±173% +0.1 0.08 ± 13% perf-profile.self.cycles-pp.xas_set_mark
0.03 ±100% +0.1 0.10 ± 11% perf-profile.self.cycles-pp.inc_zone_page_state
0.03 ±100% +0.1 0.10 ± 11% perf-profile.self.cycles-pp.block_page_mkwrite
0.01 ±173% +0.1 0.09 ± 10% perf-profile.self.cycles-pp.page_mapped
0.00 +0.1 0.07 ± 5% perf-profile.self.cycles-pp.invalidate_inode_page
0.10 ± 10% +0.1 0.17 ± 6% perf-profile.self.cycles-pp.__perf_sw_event
0.10 ± 10% +0.1 0.17 ± 10% perf-profile.self.cycles-pp.page_fault
0.00 +0.1 0.07 ± 22% perf-profile.self.cycles-pp.fault_dirty_shared_page
0.08 ± 8% +0.1 0.16 ± 7% perf-profile.self.cycles-pp.__block_commit_write
0.00 +0.1 0.08 ± 11% perf-profile.self.cycles-pp.ktime_get_update_offsets_now
0.09 ± 13% +0.1 0.16 ± 11% perf-profile.self.cycles-pp.mark_buffer_dirty
0.16 ± 5% +0.1 0.24 ± 4% perf-profile.self.cycles-pp.xas_start
0.01 ±173% +0.1 0.09 ± 8% perf-profile.self.cycles-pp.ext4_page_mkwrite
0.35 ± 6% +0.1 0.44 ± 7% perf-profile.self.cycles-pp.mpage_process_page_bufs
0.17 ± 2% +0.1 0.27 ± 11% perf-profile.self.cycles-pp.___perf_sw_event
0.11 ± 9% +0.1 0.22 ± 5% perf-profile.self.cycles-pp.mpage_prepare_extent_to_map
0.12 ± 13% +0.1 0.23 ± 8% perf-profile.self.cycles-pp.dec_zone_page_state
0.44 ± 7% +0.1 0.55 ± 6% perf-profile.self.cycles-pp.___might_sleep
0.20 ± 7% +0.1 0.31 ± 6% perf-profile.self.cycles-pp.native_flush_tlb_one_user
0.28 ± 7% +0.1 0.41 ± 4% perf-profile.self.cycles-pp.clear_page_dirty_for_io
0.23 ± 6% +0.1 0.36 ± 4% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.22 ± 4% +0.1 0.36 ± 8% perf-profile.self.cycles-pp.do_wp_page
0.20 ± 6% +0.1 0.35 ± 4% perf-profile.self.cycles-pp.do_page_fault
0.10 ± 12% +0.1 0.25 ± 5% perf-profile.self.cycles-pp.cpuidle_enter_state
0.17 ± 13% +0.2 0.32 ± 8% perf-profile.self.cycles-pp.deactivate_file_page
0.26 ± 4% +0.2 0.43 ± 7% perf-profile.self.cycles-pp.handle_pte_fault
0.22 ± 14% +0.2 0.39 ± 7% perf-profile.self.cycles-pp.invalidate_mapping_pages
0.09 ± 9% +0.2 0.27 ± 3% perf-profile.self.cycles-pp.__block_write_begin_int
0.29 ± 3% +0.2 0.48 ± 6% perf-profile.self.cycles-pp.find_get_pages_range_tag
0.21 ± 6% +0.2 0.41 ± 3% perf-profile.self.cycles-pp.__test_set_page_writeback
0.49 ± 8% +0.2 0.71 ± 5% perf-profile.self.cycles-pp.page_mapping
0.12 ± 27% +0.2 0.35 ± 3% perf-profile.self.cycles-pp.menu_select
0.90 ± 7% +0.2 1.14 ± 2% perf-profile.self.cycles-pp.unlock_page
0.25 ± 5% +0.2 0.49 ± 4% perf-profile.self.cycles-pp.ext4_bio_write_page
0.33 ± 8% +0.2 0.58 ± 6% perf-profile.self.cycles-pp.percpu_counter_add_batch
0.17 ± 11% +0.3 0.43 ± 8% perf-profile.self.cycles-pp.__set_page_dirty_buffers
0.36 ± 12% +0.3 0.63 ± 4% perf-profile.self.cycles-pp.release_pages
0.35 ± 9% +0.3 0.64 ± 4% perf-profile.self.cycles-pp.handle_mm_fault
0.51 ± 6% +0.3 0.81 ± 8% perf-profile.self.cycles-pp.__handle_mm_fault
0.22 ± 6% +0.3 0.53 ± 7% perf-profile.self.cycles-pp.test_clear_page_writeback
1.61 ± 7% +0.3 1.94 ± 3% perf-profile.self.cycles-pp.filemap_map_pages
0.54 ± 9% +0.3 0.87 ± 5% perf-profile.self.cycles-pp.xas_load
0.86 ± 5% +0.4 1.23 ± 4% perf-profile.self.cycles-pp.ext4_finish_bio
0.69 ± 9% +0.4 1.06 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.86 ± 9% +0.4 1.25 ± 4% perf-profile.self.cycles-pp.find_get_entries
0.30 ± 8% +0.4 0.70 ± 17% perf-profile.self.cycles-pp.ktime_get
0.61 ± 6% +0.4 1.06 ± 4% perf-profile.self.cycles-pp.sync_regs
0.76 ± 7% +0.5 1.23 ± 5% perf-profile.self.cycles-pp.get_io_u
4.28 ± 6% +0.6 4.92 ± 4% perf-profile.self.cycles-pp.__memcpy_mcsafe
1.09 ± 9% +0.7 1.76 ± 5% perf-profile.self.cycles-pp.native_irq_return_iret
3.43 ± 16% +3.3 6.72 ± 3% perf-profile.self.cycles-pp.__memcpy_flushcache
fio.write_bw_MBps
6000 +--------------------------------------------------------------------+
| O O |
5000 |-+ O O O O O O O O O O O O O O O O O O O O O O O |
| O .+. .+. |
|.+.+.+.+.+.+.+ + +.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.|
4000 |-+ |
| |
3000 |-+ |
| |
2000 |-+ |
| |
| |
1000 |-+ |
| |
0 +--------------------------------------------------------------------+
fio.read_clat_mean_us
3.5e+06 +-----------------------------------------------------------------+
|.+ .+ .+ .+. +. |
3e+06 |-++ .+.+ .+.+.+.+.+ + .+ + .++.+ +.+. + +.|
| +.+.+.+.+ +.+ +.+.+ +.+.+ + |
2.5e+06 |-+ |
| |
2e+06 |-+ |
| |
1.5e+06 |-+ O O O O O O O O OO O O |
| O O O O OO O O O O O O O O |
1e+06 |-+ |
| |
500000 |-+ |
| |
0 +-----------------------------------------------------------------+
fio.read_clat_90__us
9e+06 +-------------------------------------------------------------------+
| + +. : + |
8e+06 |.+ .+. + : + +.+.+ .+.+ +. .+.+ : :|
7e+06 |-+: +.+ +.+ : .+.+.+ : +.+ : +.+.+. + + + : :|
| : + + : + : + + + |
6e+06 |-+ + + + |
5e+06 |-+ |
| |
4e+06 |-+ |
3e+06 |-+ |
| O O O O O O O O O O O O O O O O O O O O O O O O O O |
2e+06 |-+ |
1e+06 |-+ |
| |
0 +-------------------------------------------------------------------+
fio.read_clat_95__us
1.4e+07 +-----------------------------------------------------------------+
|.+ .+ .+ +.+. +. |
1.2e+07 |-++ +. .+.+.+.+.+ + .+ + .++. + +. + + |
| +.+.+.+. + ++.+ +.+.+ +.+.+ + + +|
1e+07 |-+ + |
| |
8e+06 |-+ |
| |
6e+06 |-+ |
| |
4e+06 |-+ O |
| O O O O O OO O O O O O O O O O O O O O O OO O O |
2e+06 |-+ |
| |
0 +-----------------------------------------------------------------+
fio.read_clat_99__us
2.5e+07 +-----------------------------------------------------------------+
| |
| +. |
2e+07 |.+. + +. .+. .+. ++. + +.+ +. |
| +. : + + +.+.+.+ +. .+ +. + + + + +.|
| +.+.+. : ++.+ +.+ +.+ + |
1.5e+07 |-+ + |
| |
1e+07 |-+ |
| O |
| O O O O O O O O O O O OO O O |
5e+06 |-+ O O O O O O O O O O |
| |
| |
0 +-----------------------------------------------------------------+
fio.latency_500us_
25 +----------------------------------------------------------------------+
| |
| |
20 |-+ O O O O O O O O O O O O |
| O O O O O O O O O O |
| O O O O |
15 |-+ |
| |
10 |-+ |
| .+.+..+. .+.+.+. .+. .+.+. .+. .+. .|
|.+.+.+ + +.+.+.+.+.+..+.+ +.+.+.+ + +..+.+ +.+ |
5 |-+ |
| |
| |
0 +----------------------------------------------------------------------+
fio.latency_2ms_
25 +----------------------------------------------------------------------+
| O O O O O O O O O O |
| O O O O O O O O O O O O O |
20 |-+ O |
| |
| |
15 |.+. .+.+.+..+.+.+.+.+.+.+.+.+.+.+.. .+.+.+.+. .+.+.+. .+.+..+.+. .+.+.|
| + + + + + |
10 |-+ |
| |
| |
5 |-+ |
| |
| |
0 +----------------------------------------------------------------------+
fio.latency_4ms_
35 +----------------------------------------------------------------------+
| |
30 |.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+. .+.+. .+.+.+.+.|
| + +..+ |
25 |-+ O O O O O O O O O |
| O O O O O O O O O O O O O O O O O |
20 |-+ |
| |
15 |-+ |
| |
10 |-+ |
| |
5 |-+ |
| |
0 +----------------------------------------------------------------------+
fio.latency_10ms_
10 +----------------------------------------------------------------------+
9 |-+ .+ + +.+ |
| .+ + .+. + + .+.+.. + :|
8 |.+. .+.+.+..+ +.+ + +.+.+.. .+.+.+.+. .+.+.+.+ +.+.+ :|
7 |-+ + + + |
| |
6 |-+ |
5 |-+ |
4 |-+ |
| |
3 |-+ O O O O O O O O |
2 |-+ O O O O O O O O O O O O O O O O O O |
| |
1 |-+ |
0 +----------------------------------------------------------------------+
fio.latency_20ms_
8 +-----------------------------------------------------------------------+
| +. |
7 |.+ .+ +. .+ .+ : + |
6 |-+: + : +. .. +.+ : .+ + .+. .+.+.+..+. : :|
| : .+..+ : : + + : .+ + .+ + + :|
5 |-+ +.+ + : +.+ +.+ +.+ |
| + |
4 |-+ |
| |
3 |-+ |
2 |-+ |
| |
1 |-+ O |
| O O O O O O O O O O O O O O O O O O O O O O O O O |
0 +-----------------------------------------------------------------------+
fio.time.system_time
3500 +--------------------------------------------------------------------+
| + +. |
3000 |-+ + + .+. .+. .+ : + |
|.+. .+.+.+. .+ +.+.+.+ +.+.+.. .+.+.+.+. .+.+.+.+.+ + + : +|
2500 |-+ + + + + + |
| |
2000 |-+ |
| O O O O O O O O O O O O O O O O O O O O O O O O O |
1500 |-+ O |
| |
1000 |-+ |
| |
500 |-+ |
| |
0 +--------------------------------------------------------------------+
fio.time.percent_of_cpu_this_job_got
2000 +--------------------------------------------------------------------+
1800 |-+ +. |
| .+. .+. .+. .+ : + |
1600 |.+. .+.+.+.+.+ +.+.+.+ +.+.+.. .+.+.+.+. .+.+.+.+.+ + + : +|
1400 |-+ + + + + |
| |
1200 |-+ O O O O O O O O O O O O O |
1000 |-+ O O O O O O O O O O O O O |
800 |-+ |
| |
600 |-+ |
400 |-+ |
| |
200 |-+ |
0 +--------------------------------------------------------------------+
fio.time.major_page_faults
3e+08 +-----------------------------------------------------------------+
| |
2.5e+08 |-+ .+ .+. .+.+.|
|.+.+.+.+.+.+.+ +.+.+.+ +.+.+.+.+.+.+.+.+.+.+.++.+. .+.+.+ |
| + |
2e+08 |-+ O O O O OO O O O O O O O O O O O O O O O OO O O |
| O |
1.5e+08 |-+ |
| |
1e+08 |-+ |
| |
| |
5e+07 |-+ |
| |
0 +-----------------------------------------------------------------+
fio.time.minor_page_faults
2.5e+08 +-----------------------------------------------------------------+
| O O O O O |
| O O OO O O O O O O O O O O O O OO O O |
2e+08 |-+ O |
| |
| |
1.5e+08 |-+ .+ .+. .+.+. .+. |
|.+.+.+.+.+.+.+ + +.+.+.+.+.+.+.+.+.+.+.+.+.+.++ +.+.+ +.|
1e+08 |-+ |
| |
| |
5e+07 |-+ |
| |
| |
0 +-----------------------------------------------------------------+
fio.time.file_system_inputs
2.5e+09 +-----------------------------------------------------------------+
| |
| + |
2e+09 |.+ .+.+.+.+. + +.+. .+.+.+. .+.+.+. .+.+. .+. .+. .+.+.|
| +.+ + +.+ +.+ +.+ ++ +.+ + |
| O O O O OO O O O O O O O O O O O |
1.5e+09 |-+ O O O O O O O O O |
| |
1e+09 |-+ |
| |
| |
5e+08 |-+ |
| |
| |
0 +-----------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
5 months, 1 week
- 2
- 1
[KVM] a06230b62b: kvm-unit-tests.vmx.fail
by kernel test robot
FYI, we noticed the following commit (built with gcc-7):
commit: a06230b62b898e51cfb1de256b2042a09a691f58 ("KVM: x86: Deliver exception payload on KVM_GET_VCPU_EVENTS")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: kvm-unit-tests
with following parameters:
ucode: 0x27
on test machine: 8 threads Intel(R) Core(TM) i7-4790 v3 @ 3.60GHz with 6G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen(a)intel.com>
2020-02-28 02:00:44 ./run_tests.sh
[32mPASS[0m apic-split (53 tests)
[32mPASS[0m ioapic-split (19 tests)
[32mPASS[0m apic (53 tests)
[32mPASS[0m ioapic (26 tests)
[32mPASS[0m smptest (1 tests)
[32mPASS[0m smptest3 (1 tests)
[32mPASS[0m vmexit_cpuid
[32mPASS[0m vmexit_vmcall
[32mPASS[0m vmexit_mov_from_cr8
[32mPASS[0m vmexit_mov_to_cr8
[32mPASS[0m vmexit_inl_pmtimer
[32mPASS[0m vmexit_ipi
[32mPASS[0m vmexit_ipi_halt
[32mPASS[0m vmexit_ple_round_robin
[32mPASS[0m vmexit_tscdeadline
[32mPASS[0m vmexit_tscdeadline_immed
[32mPASS[0m access
[33mSKIP[0m smap (0 tests)
[33mSKIP[0m pku (0 tests)
[32mPASS[0m asyncpf (1 tests)
[32mPASS[0m emulator (125 tests, 2 skipped)
[32mPASS[0m eventinj (13 tests)
[32mPASS[0m hypercall (2 tests)
[32mPASS[0m idt_test (4 tests)
[32mPASS[0m memory (8 tests)
[32mPASS[0m msr (12 tests)
[32mPASS[0m pmu (67 tests)
[32mPASS[0m vmware_backdoors (11 tests)
[32mPASS[0m port80
[32mPASS[0m realmode
[32mPASS[0m s3
[32mPASS[0m sieve
[32mPASS[0m syscall (2 tests)
[32mPASS[0m tsc (3 tests)
[32mPASS[0m tsc_adjust (5 tests)
[32mPASS[0m xsave (17 tests)
[32mPASS[0m rmap_chain
[33mSKIP[0m svm (0 tests)
[33mSKIP[0m taskswitch (i386 only)
[33mSKIP[0m taskswitch2 (i386 only)
[32mPASS[0m kvmclock_test
[32mPASS[0m pcid (3 tests)
[32mPASS[0m rdpru (1 tests)
[32mPASS[0m umip (21 tests)
[31mFAIL[0m vmx (408415 tests, 3 unexpected failures, 2 expected failures, 5 skipped)
[33mSKIP[0m ept (0 tests)
[33mSKIP[0m vmx_eoi_bitmap_ioapic_scan (1 tests, 1 skipped)
[33mSKIP[0m vmx_hlt_with_rvi_test (1 tests, 1 skipped)
[33mSKIP[0m vmx_apicv_test (2 tests, 2 skipped)
[32mPASS[0m vmx_apic_passthrough_thread (8 tests)
[32mPASS[0m vmx_init_signal_test (9 tests)
[32mPASS[0m vmx_apic_passthrough_tpr_threshold_test (6 tests)
[32mPASS[0m vmx_vmcs_shadow_test (142218 tests)
[32mPASS[0m debug (11 tests)
[32mPASS[0m hyperv_synic (1 tests)
[33mSKIP[0m hyperv_connections (1 tests, 1 skipped)
[32mPASS[0m hyperv_stimer (12 tests)
[32mPASS[0m hyperv_clock
[32mPASS[0m intel_iommu (11 tests)
[32mPASS[0m tsx-ctrl
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
Thanks,
Rong Chen
5 months, 1 week
- 3
- 2