February 2020 - LKP - Ml01.01.Org

[sched/fair] 3c29e651e1: hackbench.throughput -15.2% regression

by kernel test robot

Greeting, FYI, we noticed a -15.2% regression of hackbench.throughput due to commit: commit: 3c29e651e16dd3b3179cfb2d055ee9538e37515c ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: hackbench on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory with following parameters: nr_threads: 100% mode: threads ipc: pipe cpufreq_governor: performance ucode: 0xca test-description: Hackbench is both a benchmark and a stress test for the Linux kernel scheduler. test-url: https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/sc... In addition to that, the commit also has significant impact on the following tests: +------------------+-----------------------------------------------------------------------+ | testcase: change | hackbench: hackbench.throughput -11.3% regression | | test machine | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory | | test parameters | cpufreq_governor=performance | | | ipc=pipe | | | iterations=18 | | | mode=threads | | | nr_threads=1600% | | | ucode=0x43 | +------------------+-----------------------------------------------------------------------+ | testcase: change | hackbench: hackbench.throughput -10.6% regression | | test machine | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory | | test parameters | cpufreq_governor=performance | | | ipc=pipe | | | mode=process | | | nr_threads=50% | | | ucode=0xb000038 | +------------------+-----------------------------------------------------------------------+ | testcase: change | hackbench: hackbench.throughput -6.8% regression | | test machine | 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory | | test parameters | cpufreq_governor=performance | | | ipc=pipe | | | mode=threads | | | nr_threads=100% | | | ucode=0xb8 | +------------------+-----------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode: gcc-7/performance/pipe/x86_64-rhel-7.6/threads/100%/debian-x86_64-20191114.cgz/lkp-cfl-e1/hackbench/0xca commit: 43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq") 3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found") 43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055 ---------------- --------------------------- %stddev %change %stddev \ | \ 69492 ± 3% -15.2% 58899 hackbench.throughput 7.727e+08 ± 7% +38.3% 1.069e+09 ± 2% hackbench.time.involuntary_context_switches 131708 ± 4% -15.4% 111382 hackbench.time.minor_page_faults 684.98 ± 2% -2.5% 667.72 hackbench.time.user_time 1.755e+09 ± 2% +11.2% 1.953e+09 hackbench.time.voluntary_context_switches 4.32e+08 ± 4% -15.6% 3.648e+08 hackbench.workload 186.50 ± 12% -16.6% 155.50 ± 11% interrupts.TLB:TLB_shootdowns 4147892 ± 5% +19.6% 4961643 vmstat.system.cs 0.72 ± 10% -0.1 0.58 ± 17% mpstat.cpu.all.idle% 0.00 ±156% +0.0 0.01 ± 62% mpstat.cpu.all.soft% 1038 ± 9% +18.5% 1230 ± 4% slabinfo.avc_xperms_data.active_objs 1038 ± 9% +18.5% 1230 ± 4% slabinfo.avc_xperms_data.num_objs 40647036 ± 10% -34.6% 26589504 proc-vmstat.numa_hit 40647036 ± 10% -34.6% 26589504 proc-vmstat.numa_local 40724029 ± 10% -34.5% 26661546 proc-vmstat.pgalloc_normal 763906 ± 2% -7.5% 706781 proc-vmstat.pgfault 40701242 ± 10% -34.6% 26637947 proc-vmstat.pgfree 1636058 ± 11% -45.0% 899629 ± 11% turbostat.C1 0.10 ± 7% -0.0 0.07 ± 7% turbostat.C1% 15201 ± 25% +113.3% 32422 ± 8% turbostat.C1E 0.01 ± 57% +0.0 0.03 ± 13% turbostat.C1E% 9093 ± 11% +217.2% 28841 ± 11% turbostat.C3 0.00 ±173% +0.0 0.03 ± 13% turbostat.C3% 16977 ± 2% -92.8% 1222 ± 98% turbostat.C8 0.17 ± 4% -0.2 0.01 ±110% turbostat.C8% 0.13 -100.0% 0.00 turbostat.CPU%c7 9623106 ± 5% -35.1% 6241651 ± 8% cpuidle.C1.time 1636668 ± 11% -45.0% 899926 ± 11% cpuidle.C1.usage 899494 ± 33% +272.6% 3351412 ± 5% cpuidle.C1E.time 15327 ± 25% +112.6% 32592 ± 8% cpuidle.C1E.usage 435674 ± 20% +643.5% 3239351 ± 14% cpuidle.C3.time 9179 ± 11% +215.2% 28933 ± 11% cpuidle.C3.usage 16850924 ± 2% -94.2% 979516 ± 86% cpuidle.C8.time 17150 ± 2% -92.0% 1378 ± 88% cpuidle.C8.usage 6022328 ± 18% -57.1% 2584773 ± 7% cpuidle.POLL.time 5339703 ± 19% -60.1% 2130089 ± 7% cpuidle.POLL.usage 14.06 ± 25% +64.0% 23.07 ± 13% sched_debug.cfs_rq:/.runnable_load_avg.min 25.10 ± 5% -15.8% 21.12 ± 7% sched_debug.cfs_rq:/.runnable_load_avg.stddev 13339 ± 24% +45.7% 19433 ± 7% sched_debug.cfs_rq:/.runnable_weight.min 166.51 ± 24% +48.9% 247.86 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.min 253.56 ± 2% -12.3% 222.30 ± 9% sched_debug.cfs_rq:/.util_est_enqueued.stddev 159602 ± 23% +55.6% 248411 ± 6% sched_debug.cpu.avg_idle.max 51567 ± 24% +49.1% 76905 ± 11% sched_debug.cpu.avg_idle.stddev 11576 ± 5% -9.5% 10478 sched_debug.cpu.curr->pid.avg 9976 ± 8% -16.8% 8305 ± 9% sched_debug.cpu.curr->pid.min 4.81 ± 15% +43.3% 6.89 ± 8% sched_debug.cpu.nr_running.min 6.68 ± 4% -17.0% 5.54 ± 8% sched_debug.cpu.nr_running.stddev 76003928 ± 2% +21.0% 91975904 sched_debug.cpu.nr_switches.avg 78115404 ± 2% +23.4% 96397551 sched_debug.cpu.nr_switches.max 74275755 ± 2% +19.0% 88359903 sched_debug.cpu.nr_switches.min 1043054 ± 15% +83.9% 1917752 ± 10% sched_debug.cpu.nr_switches.stddev -685.26 +64.8% -1129 sched_debug.cpu.nr_uninterruptible.min 15470 ± 9% -31.0% 10674 ± 9% softirqs.CPU0.SCHED 13756 ± 10% -40.8% 8140 ± 4% softirqs.CPU1.SCHED 13645 ± 11% -39.7% 8227 ± 3% softirqs.CPU10.SCHED 14065 ± 12% -38.8% 8609 ± 2% softirqs.CPU11.SCHED 13948 ± 11% -39.9% 8381 ± 6% softirqs.CPU12.SCHED 13823 ± 12% -39.7% 8332 ± 9% softirqs.CPU13.SCHED 73323 +13.4% 83165 ± 11% softirqs.CPU14.RCU 13872 ± 12% -36.5% 8809 ± 7% softirqs.CPU14.SCHED 13726 ± 12% -40.4% 8175 ± 4% softirqs.CPU15.SCHED 13780 ± 11% -34.8% 8987 ± 12% softirqs.CPU2.SCHED 13895 ± 10% -40.5% 8266 ± 6% softirqs.CPU3.SCHED 221879 +13.4% 251589 ± 14% softirqs.CPU3.TIMER 13873 ± 10% -40.4% 8262 ± 6% softirqs.CPU4.SCHED 72625 +12.0% 81355 ± 10% softirqs.CPU5.RCU 13876 ± 13% -41.4% 8137 ± 3% softirqs.CPU5.SCHED 14118 ± 12% -40.8% 8354 ± 3% softirqs.CPU6.SCHED 13503 ± 12% -40.0% 8099 softirqs.CPU7.SCHED 13549 ± 11% -38.2% 8374 ± 8% softirqs.CPU8.SCHED 14273 ± 10% -40.1% 8548 ± 6% softirqs.CPU9.SCHED 223184 ± 11% -38.9% 136383 ± 5% softirqs.SCHED 55.80 +9.9% 61.34 perf-stat.i.MPKI 1.32 +0.1 1.39 perf-stat.i.branch-miss-rate% 1.066e+08 +3.7% 1.105e+08 perf-stat.i.branch-misses 0.15 ± 8% -0.0 0.11 ± 3% perf-stat.i.cache-miss-rate% 2521584 ± 9% -26.5% 1854353 ± 3% perf-stat.i.cache-misses 2.27e+09 +8.0% 2.451e+09 perf-stat.i.cache-references 4162783 ± 4% +19.7% 4981708 perf-stat.i.context-switches 1.52 +2.0% 1.55 perf-stat.i.cpi 227633 ± 14% +75.1% 398694 ± 3% perf-stat.i.cpu-migrations 33931 ± 12% +42.7% 48428 ± 2% perf-stat.i.cycles-between-cache-misses 0.01 -0.0 0.01 ± 4% perf-stat.i.dTLB-load-miss-rate% 1408442 ± 2% -16.4% 1177727 ± 4% perf-stat.i.dTLB-load-misses 1.229e+10 -2.1% 1.203e+10 perf-stat.i.dTLB-loads 0.00 ± 2% -0.0 0.00 ± 4% perf-stat.i.dTLB-store-miss-rate% 35757 ± 2% -16.0% 30032 ± 3% perf-stat.i.dTLB-store-misses 7.449e+09 -2.3% 7.279e+09 perf-stat.i.dTLB-stores 39246970 -5.5% 37079604 perf-stat.i.iTLB-load-misses 76813 ± 8% -26.2% 56724 ± 6% perf-stat.i.iTLB-loads 4.096e+10 -1.9% 4.02e+10 perf-stat.i.instructions 1062 +4.1% 1105 perf-stat.i.instructions-per-iTLB-miss 0.66 -1.9% 0.65 perf-stat.i.ipc 1210 -7.1% 1124 perf-stat.i.minor-faults 105889 ± 10% -24.8% 79596 ± 5% perf-stat.i.node-loads 174508 ± 7% -24.4% 131915 perf-stat.i.node-stores 1210 -7.1% 1124 perf-stat.i.page-faults 55.42 ± 2% +10.0% 60.98 perf-stat.overall.MPKI 1.31 +0.1 1.38 perf-stat.overall.branch-miss-rate% 0.11 ± 11% -0.0 0.08 ± 3% perf-stat.overall.cache-miss-rate% 1.51 +1.9% 1.54 perf-stat.overall.cpi 24740 ± 9% +34.9% 33377 ± 3% perf-stat.overall.cycles-between-cache-misses 0.01 -0.0 0.01 ± 4% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 2% -0.0 0.00 ± 4% perf-stat.overall.dTLB-store-miss-rate% 1043 +3.9% 1084 perf-stat.overall.instructions-per-iTLB-miss 0.66 -1.9% 0.65 perf-stat.overall.ipc 57826 ± 3% +15.6% 66874 perf-stat.overall.path-length 1.064e+08 +3.7% 1.104e+08 perf-stat.ps.branch-misses 2517524 ± 9% -26.5% 1851335 ± 3% perf-stat.ps.cache-misses 2.266e+09 +8.0% 2.447e+09 perf-stat.ps.cache-references 4155898 ± 4% +19.7% 4973458 perf-stat.ps.context-switches 227258 ± 14% +75.1% 398041 ± 3% perf-stat.ps.cpu-migrations 1406119 ± 2% -16.4% 1175777 ± 4% perf-stat.ps.dTLB-load-misses 1.227e+10 -2.1% 1.201e+10 perf-stat.ps.dTLB-loads 35699 ± 2% -16.0% 29983 ± 3% perf-stat.ps.dTLB-store-misses 7.437e+09 -2.3% 7.267e+09 perf-stat.ps.dTLB-stores 39182154 -5.5% 37018206 perf-stat.ps.iTLB-load-misses 76695 ± 8% -26.2% 56638 ± 6% perf-stat.ps.iTLB-loads 4.09e+10 -1.9% 4.013e+10 perf-stat.ps.instructions 1208 -7.1% 1122 perf-stat.ps.minor-faults 105716 ± 10% -24.8% 79465 ± 5% perf-stat.ps.node-loads 174234 ± 7% -24.4% 131703 perf-stat.ps.node-stores 1208 -7.1% 1122 perf-stat.ps.page-faults 29.39 ±104% -29.4 0.00 perf-profile.calltrace.cycles-pp.start_thread 15.86 ±105% -15.9 0.00 perf-profile.calltrace.cycles-pp.__GI___libc_write.start_thread 14.88 ±105% -14.9 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 14.75 ±105% -14.8 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 13.53 ±104% -13.5 0.00 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 12.51 ±104% -12.5 0.00 perf-profile.calltrace.cycles-pp.__GI___libc_read.start_thread 12.45 ±105% -12.4 0.00 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 11.21 ±105% -11.2 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 11.08 ±105% -11.1 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 10.47 ±105% -10.5 0.00 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 9.73 ±105% -9.7 0.00 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read 3.09 ± 12% -0.6 2.50 ± 2% perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read 2.45 ± 14% -0.5 1.95 ± 2% perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write 1.15 ± 13% -0.2 0.95 ± 4% perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.pipe_write.new_sync_write.vfs_write 0.99 ± 13% -0.2 0.83 ± 5% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter.pipe_write.new_sync_write 0.92 ± 15% +0.2 1.10 perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.84 ± 23% +0.3 1.10 ± 2% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.99 ± 19% +0.3 1.26 ± 3% perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.97 ± 23% +0.3 1.26 perf-profile.calltrace.cycles-pp.__enqueue_entity.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.29 ±100% +0.3 0.60 ± 5% perf-profile.calltrace.cycles-pp.reschedule_interrupt.__lock_text_start.__wake_up_common_lock.pipe_write.new_sync_write 0.55 ± 62% +0.3 0.89 ± 7% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule 0.28 ±100% +0.4 0.63 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop 0.33 ±100% +0.4 0.69 ± 2% perf-profile.calltrace.cycles-pp.prepare_to_wait.pipe_wait.pipe_read.new_sync_read.vfs_read 0.29 ±100% +0.4 0.65 ± 4% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read 1.00 ± 27% +0.4 1.37 ± 2% perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function 0.57 ± 63% +0.4 0.95 ± 5% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate 1.46 ± 31% +0.4 1.86 ± 2% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.72 ± 60% +0.4 1.14 ± 2% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.29 ±100% +0.4 0.70 ± 6% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.34 ±100% +0.4 0.77 ± 7% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task 1.26 ± 26% +0.5 1.72 perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common 1.13 ± 26% +0.5 1.59 perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.35 ±100% +0.5 0.83 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 1.28 ± 28% +0.5 1.77 ± 3% perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 1.40 ± 26% +0.5 1.89 perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.44 ±100% +0.5 0.95 ± 7% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop 2.33 ± 17% +0.5 2.88 ± 2% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_wait.pipe_read 1.33 ± 28% +0.6 1.89 ± 2% perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 1.40 ± 24% +0.6 1.97 ± 4% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function 0.83 ± 66% +0.6 1.40 ± 5% perf-profile.calltrace.cycles-pp.native_write_msr 5.01 ± 8% +0.8 5.81 ± 2% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 1.75 ± 39% +0.9 2.61 ± 6% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 1.33 ± 67% +1.0 2.28 ± 6% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.56 ± 12% +1.2 7.78 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_wait.pipe_read 8.96 ± 9% +1.6 10.56 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 9.23 ± 10% +1.6 10.87 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 9.17 ± 10% +1.6 10.81 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 3.72 ± 31% +1.7 5.44 ± 6% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.37 ± 8% +1.9 5.32 ± 4% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common 4.37 ± 10% +2.3 6.65 ± 5% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 15.27 ± 10% +2.3 17.59 perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait.pipe_read.new_sync_read 15.59 ± 10% +2.4 17.96 perf-profile.calltrace.cycles-pp.schedule.pipe_wait.pipe_read.new_sync_read.vfs_read 3.00 ± 71% +2.6 5.60 ± 6% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.10 ± 71% +2.7 5.77 ± 6% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 18.05 ± 10% +2.7 20.73 perf-profile.calltrace.cycles-pp.pipe_wait.pipe_read.new_sync_read.vfs_read.ksys_read 28.21 ± 5% +3.7 31.87 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write 24.29 ± 8% +4.7 28.98 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 23.38 ± 8% +4.7 28.08 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write 23.72 ± 8% +4.7 28.43 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write 31.18 ± 41% +12.7 43.86 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 33.44 ± 40% +13.2 46.61 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 64.81 ± 43% +27.5 92.34 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 65.30 ± 43% +27.7 92.99 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 29.39 ±104% -29.4 0.00 perf-profile.children.cycles-pp.start_thread 16.09 ±105% -16.1 0.00 perf-profile.children.cycles-pp.__GI___libc_write 12.73 ±104% -12.7 0.00 perf-profile.children.cycles-pp.__GI___libc_read 5.80 ± 10% -1.2 4.65 ± 3% perf-profile.children.cycles-pp.security_file_permission 4.10 ± 16% -1.0 3.10 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 3.21 ± 9% -0.7 2.55 ± 3% perf-profile.children.cycles-pp.copy_page_to_iter 1.45 ± 32% -0.6 0.83 perf-profile.children.cycles-pp.entry_SYSCALL_64 3.04 ± 8% -0.6 2.44 ± 3% perf-profile.children.cycles-pp.selinux_file_permission 2.56 ± 11% -0.6 1.99 ± 2% perf-profile.children.cycles-pp.copy_page_from_iter 2.80 ± 9% -0.5 2.25 ± 2% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 3.38 ± 7% -0.5 2.87 ± 2% perf-profile.children.cycles-pp.mutex_lock 1.62 ± 12% -0.4 1.23 ± 3% perf-profile.children.cycles-pp.___might_sleep 1.82 ± 9% -0.4 1.43 ± 4% perf-profile.children.cycles-pp.syscall_return_via_sysret 2.09 ± 12% -0.4 1.72 ± 6% perf-profile.children.cycles-pp.file_has_perm 0.90 ± 20% -0.3 0.57 ± 9% perf-profile.children.cycles-pp.__mutex_lock 1.14 ± 12% -0.3 0.82 ± 3% perf-profile.children.cycles-pp.__inode_security_revalidate 1.26 ± 10% -0.3 0.97 ± 4% perf-profile.children.cycles-pp.copyin 1.70 ± 8% -0.3 1.42 ± 3% perf-profile.children.cycles-pp.copyout 1.25 ± 11% -0.3 0.98 ± 3% perf-profile.children.cycles-pp.fsnotify 1.04 ± 9% -0.3 0.79 ± 3% perf-profile.children.cycles-pp._cond_resched 1.72 ± 7% -0.2 1.48 ± 5% perf-profile.children.cycles-pp.fput_many 1.16 ± 10% -0.2 0.92 ± 4% perf-profile.children.cycles-pp.__might_sleep 1.09 ± 8% -0.2 0.85 ± 3% perf-profile.children.cycles-pp.__fsnotify_parent 0.83 ± 13% -0.2 0.63 ± 3% perf-profile.children.cycles-pp.__might_fault 0.69 ± 11% -0.1 0.56 perf-profile.children.cycles-pp.current_time 0.99 ± 4% -0.1 0.89 ± 4% perf-profile.children.cycles-pp.touch_atime 0.65 ± 6% -0.1 0.55 ± 6% perf-profile.children.cycles-pp.atime_needs_update 0.39 ± 4% -0.1 0.29 perf-profile.children.cycles-pp.rcu_all_qs 0.13 ± 39% -0.1 0.04 ± 59% perf-profile.children.cycles-pp.mutex_spin_on_owner 0.34 ± 11% -0.1 0.26 ± 9% perf-profile.children.cycles-pp.preempt_schedule_common 0.49 ± 5% -0.1 0.41 ± 2% perf-profile.children.cycles-pp.wake_up_q 0.24 ± 2% -0.1 0.17 ± 4% perf-profile.children.cycles-pp.fpregs_assert_state_consistent 0.30 ± 11% -0.1 0.23 ± 3% perf-profile.children.cycles-pp.inode_has_perm 0.23 ± 8% -0.1 0.18 ± 6% perf-profile.children.cycles-pp.__sb_end_write 0.21 ± 9% -0.1 0.16 ± 9% perf-profile.children.cycles-pp.__x64_sys_read 0.29 ± 8% -0.0 0.24 ± 2% perf-profile.children.cycles-pp.timespec64_trunc 0.21 ± 8% -0.0 0.17 ± 8% perf-profile.children.cycles-pp.__x64_sys_write 0.22 ± 7% -0.0 0.19 ± 2% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.08 ± 16% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.bpf_fd_pass 0.10 ± 4% +0.0 0.11 ± 3% perf-profile.children.cycles-pp.perf_exclude_event 0.08 ± 24% +0.0 0.13 ± 5% perf-profile.children.cycles-pp.rcu_note_context_switch 0.17 ± 19% +0.1 0.24 ± 8% perf-profile.children.cycles-pp.wakeup_preempt_entity 0.23 ± 19% +0.1 0.32 ± 3% perf-profile.children.cycles-pp.resched_curr 0.61 ± 8% +0.1 0.72 ± 6% perf-profile.children.cycles-pp.sched_clock 0.67 ± 8% +0.1 0.78 ± 6% perf-profile.children.cycles-pp.sched_clock_cpu 0.50 ± 9% +0.1 0.62 ± 3% perf-profile.children.cycles-pp.account_entity_enqueue 0.61 ± 10% +0.2 0.79 perf-profile.children.cycles-pp.account_entity_dequeue 0.43 ± 19% +0.2 0.62 ± 7% perf-profile.children.cycles-pp.clear_buddies 1.42 ± 14% +0.2 1.66 ± 4% perf-profile.children.cycles-pp.native_write_msr 1.17 ± 13% +0.3 1.42 ± 2% perf-profile.children.cycles-pp.check_preempt_wakeup 0.85 ± 23% +0.3 1.12 ± 7% perf-profile.children.cycles-pp.put_prev_entity 0.94 ± 14% +0.3 1.22 ± 5% perf-profile.children.cycles-pp.___perf_sw_event 1.45 ± 11% +0.3 1.73 perf-profile.children.cycles-pp.check_preempt_curr 1.45 ± 12% +0.3 1.76 perf-profile.children.cycles-pp.__enqueue_entity 1.60 ± 10% +0.3 1.91 perf-profile.children.cycles-pp.ttwu_do_wakeup 1.12 ± 9% +0.3 1.43 ± 3% perf-profile.children.cycles-pp.__update_load_avg_se 1.64 ± 8% +0.4 2.00 ± 4% perf-profile.children.cycles-pp.available_idle_cpu 0.96 ± 19% +0.4 1.34 ± 7% perf-profile.children.cycles-pp.pick_next_entity 1.85 ± 16% +0.4 2.28 ± 6% perf-profile.children.cycles-pp.switch_fpu_return 1.90 ± 9% +0.6 2.50 ± 3% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 3.22 ± 13% +0.6 3.87 perf-profile.children.cycles-pp.update_load_avg 4.03 ± 10% +0.6 4.68 ± 2% perf-profile.children.cycles-pp.update_curr 3.01 ± 14% +0.7 3.72 ± 2% perf-profile.children.cycles-pp.reweight_entity 5.10 ± 8% +0.8 5.87 ± 2% perf-profile.children.cycles-pp.enqueue_entity 4.71 ± 15% +1.1 5.77 ± 4% perf-profile.children.cycles-pp.pick_next_task_fair 6.73 ± 11% +1.1 7.87 perf-profile.children.cycles-pp.dequeue_task_fair 9.07 ± 9% +1.5 10.61 perf-profile.children.cycles-pp.enqueue_task_fair 9.32 ± 9% +1.6 10.89 perf-profile.children.cycles-pp.ttwu_do_activate 91.46 +1.6 93.03 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 9.27 ± 9% +1.6 10.84 perf-profile.children.cycles-pp.activate_task 4.16 ± 26% +1.6 5.78 ± 6% perf-profile.children.cycles-pp.exit_to_usermode_loop 90.79 +1.7 92.47 perf-profile.children.cycles-pp.do_syscall_64 3.48 ± 8% +1.9 5.36 ± 4% perf-profile.children.cycles-pp.select_idle_sibling 4.44 ± 10% +2.2 6.67 ± 5% perf-profile.children.cycles-pp.select_task_rq_fair 18.46 ± 9% +2.3 20.81 perf-profile.children.cycles-pp.pipe_wait 19.93 ± 11% +3.5 23.46 ± 2% perf-profile.children.cycles-pp.__schedule 28.82 ± 5% +3.6 32.45 perf-profile.children.cycles-pp.__wake_up_common_lock 19.89 ± 12% +3.7 23.62 ± 2% perf-profile.children.cycles-pp.schedule 23.96 ± 8% +4.5 28.45 perf-profile.children.cycles-pp.try_to_wake_up 24.53 ± 8% +4.5 29.06 perf-profile.children.cycles-pp.__wake_up_common 23.91 ± 8% +4.6 28.46 perf-profile.children.cycles-pp.autoremove_wake_function 2.76 ± 9% -0.5 2.23 ± 2% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 1.46 ± 18% -0.4 1.06 ± 4% perf-profile.self.cycles-pp.pipe_write 1.81 ± 9% -0.4 1.43 ± 4% perf-profile.self.cycles-pp.syscall_return_via_sysret 1.56 ± 12% -0.4 1.19 ± 3% perf-profile.self.cycles-pp.___might_sleep 2.09 ± 7% -0.3 1.81 ± 2% perf-profile.self.cycles-pp.mutex_lock 1.86 ± 8% -0.3 1.59 ± 3% perf-profile.self.cycles-pp.selinux_file_permission 1.21 ± 11% -0.3 0.96 ± 2% perf-profile.self.cycles-pp.fsnotify 1.70 ± 7% -0.2 1.46 ± 5% perf-profile.self.cycles-pp.fput_many 1.04 ± 11% -0.2 0.82 ± 3% perf-profile.self.cycles-pp.__might_sleep 1.01 ± 8% -0.2 0.80 ± 3% perf-profile.self.cycles-pp.__fsnotify_parent 1.04 ± 10% -0.2 0.83 perf-profile.self.cycles-pp.entry_SYSCALL_64 0.67 ± 10% -0.2 0.51 ± 5% perf-profile.self.cycles-pp.new_sync_write 0.64 ± 12% -0.2 0.48 ± 5% perf-profile.self.cycles-pp.vfs_write 1.67 -0.2 1.51 ± 2% perf-profile.self.cycles-pp.pipe_read 0.58 ± 17% -0.1 0.44 ± 7% perf-profile.self.cycles-pp.file_has_perm 0.44 ± 18% -0.1 0.31 ± 7% perf-profile.self.cycles-pp.__mutex_lock 0.49 ± 13% -0.1 0.35 ± 3% perf-profile.self.cycles-pp.copy_page_from_iter 0.64 ± 8% -0.1 0.52 ± 4% perf-profile.self.cycles-pp.copy_page_to_iter 0.67 ± 7% -0.1 0.55 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.30 ± 5% -0.1 0.20 ± 4% perf-profile.self.cycles-pp.ksys_write 0.40 ± 12% -0.1 0.30 ± 7% perf-profile.self.cycles-pp.security_file_permission 0.13 ± 38% -0.1 0.04 ± 59% perf-profile.self.cycles-pp.mutex_spin_on_owner 0.36 ± 11% -0.1 0.28 ± 3% perf-profile.self.cycles-pp.__inode_security_revalidate 0.76 ± 5% -0.1 0.68 ± 5% perf-profile.self.cycles-pp.do_syscall_64 0.21 ± 2% -0.1 0.14 ± 8% perf-profile.self.cycles-pp.fpregs_assert_state_consistent 0.26 ± 12% -0.1 0.20 ± 4% perf-profile.self.cycles-pp.inode_has_perm 0.29 ± 11% -0.1 0.23 ± 6% perf-profile.self.cycles-pp.current_time 0.28 ± 4% -0.1 0.21 ± 2% perf-profile.self.cycles-pp.rcu_all_qs 0.22 ± 7% -0.1 0.17 ± 7% perf-profile.self.cycles-pp.__sb_end_write 0.31 -0.1 0.26 ± 7% perf-profile.self.cycles-pp.ksys_read 0.20 ± 11% -0.0 0.16 ± 9% perf-profile.self.cycles-pp.__x64_sys_read 0.26 ± 8% -0.0 0.22 perf-profile.self.cycles-pp.timespec64_trunc 0.20 ± 5% -0.0 0.16 ± 9% perf-profile.self.cycles-pp.__x64_sys_write 0.24 ± 9% -0.0 0.21 ± 6% perf-profile.self.cycles-pp.__might_fault 0.09 ± 9% -0.0 0.06 perf-profile.self.cycles-pp.copyin 0.14 ± 5% -0.0 0.12 ± 3% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.07 ± 10% +0.0 0.09 ± 4% perf-profile.self.cycles-pp.sched_clock 0.16 ± 5% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.ttwu_do_wakeup 0.20 ± 12% +0.0 0.24 ± 5% perf-profile.self.cycles-pp.__list_add_valid 0.10 ± 26% +0.0 0.14 ± 10% perf-profile.self.cycles-pp.update_cfs_rq_h_load 0.15 ± 16% +0.1 0.21 ± 11% perf-profile.self.cycles-pp.wakeup_preempt_entity 0.35 ± 7% +0.1 0.43 ± 4% perf-profile.self.cycles-pp.prepare_to_wait 0.55 ± 2% +0.1 0.63 ± 3% perf-profile.self.cycles-pp.__fget_light 0.22 ± 18% +0.1 0.30 ± 3% perf-profile.self.cycles-pp.resched_curr 0.48 ± 11% +0.1 0.58 ± 3% perf-profile.self.cycles-pp.check_preempt_wakeup 0.39 ± 11% +0.1 0.50 perf-profile.self.cycles-pp.account_entity_enqueue 0.85 ± 7% +0.1 0.97 ± 4% perf-profile.self.cycles-pp.enqueue_entity 0.43 ± 16% +0.1 0.55 ± 7% perf-profile.self.cycles-pp.pick_next_entity 0.86 ± 10% +0.1 0.99 perf-profile.self.cycles-pp.dequeue_task_fair 0.37 ± 21% +0.2 0.55 ± 6% perf-profile.self.cycles-pp.clear_buddies 0.45 ± 14% +0.2 0.65 perf-profile.self.cycles-pp.account_entity_dequeue 1.41 ± 14% +0.2 1.65 ± 4% perf-profile.self.cycles-pp.native_write_msr 0.80 ± 16% +0.3 1.06 ± 4% perf-profile.self.cycles-pp.___perf_sw_event 1.09 ± 9% +0.3 1.39 ± 2% perf-profile.self.cycles-pp.__update_load_avg_se 0.84 ± 16% +0.3 1.14 ± 7% perf-profile.self.cycles-pp.select_task_rq_fair 1.45 ± 12% +0.3 1.75 perf-profile.self.cycles-pp.__enqueue_entity 1.62 ± 8% +0.4 1.99 ± 4% perf-profile.self.cycles-pp.available_idle_cpu 1.83 ± 16% +0.4 2.25 ± 6% perf-profile.self.cycles-pp.switch_fpu_return 2.15 ± 11% +0.5 2.60 ± 5% perf-profile.self.cycles-pp.update_curr 1.88 ± 9% +0.6 2.45 ± 3% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 1.21 ± 9% +0.7 1.92 ± 3% perf-profile.self.cycles-pp._raw_spin_lock 0.80 ± 11% +1.4 2.17 ± 3% perf-profile.self.cycles-pp.select_idle_sibling hackbench.throughput 74000 +-+-----------------------------------------------------------------+ 72000 +-+.+ .+ | |.+ : .+. .+. + | 70000 +-+ : + + +. : | 68000 +-+ + + : | | :: | 66000 +-+ + | 64000 +-+ | 62000 +-+ | | | 60000 +-+ O O OO O O O O O O O OO | 58000 +-+ O O O O O O O O O O O | O O OO O O | 56000 O-+ O O O O O O | 54000 +-O-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-hsw-ep4: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory ========================================================================================= compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode: gcc-7/performance/pipe/18/x86_64-rhel-7.6/threads/1600%/debian-x86_64-2019-11-14.cgz/lkp-hsw-ep4/hackbench/0x43 commit: 43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq") 3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found") 43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 176075 ± 2% -11.3% 156152 hackbench.throughput 891.79 +12.3% 1001 hackbench.time.elapsed_time 891.79 +12.3% 1001 hackbench.time.elapsed_time.max 1.889e+09 ± 9% +33.1% 2.514e+09 ± 3% hackbench.time.involuntary_context_switches 42939 ± 3% +17.0% 50220 hackbench.time.system_time 18978 +3.2% 19592 hackbench.time.user_time 2.558e+09 ± 8% +33.5% 3.415e+09 ± 2% hackbench.time.voluntary_context_switches 43133704 ± 3% +39.2% 60035429 ± 33% cpuidle.C1.time 48661 +17.7% 57274 ± 5% meminfo.Shmem 19043 ± 2% -14.0% 16377 meminfo.max_used_kB 2502 +1.3% 2535 turbostat.Avg_MHz 3.368e+08 ± 5% +30.6% 4.398e+08 turbostat.IRQ 66.25 +4.5% 69.25 vmstat.cpu.sy 29.25 ± 2% -8.5% 26.75 vmstat.cpu.us 5000443 ± 7% +18.6% 5931772 vmstat.system.cs 370173 ± 3% +17.1% 433616 vmstat.system.in 98786923 ± 7% +74.5% 1.724e+08 numa-numastat.node0.local_node 98791606 ± 7% +74.5% 1.724e+08 numa-numastat.node0.numa_hit 4686 ± 99% +297.8% 18642 ± 24% numa-numastat.node0.other_node 1.818e+08 -51.4% 88447082 ± 3% numa-numastat.node1.local_node 1.819e+08 -51.4% 88451833 ± 3% numa-numastat.node1.numa_hit 18694 ± 25% -74.6% 4753 ± 97% numa-numastat.node1.other_node 155316 +1.5% 157632 proc-vmstat.nr_active_anon 99.50 +5.3% 104.75 ± 3% proc-vmstat.nr_dirtied 4440 +4.9% 4656 proc-vmstat.nr_inactive_anon 6283 +1.9% 6399 proc-vmstat.nr_mapped 12248 ± 2% +16.6% 14276 ± 5% proc-vmstat.nr_shmem 98.75 +5.3% 104.00 proc-vmstat.nr_written 155316 +1.5% 157632 proc-vmstat.nr_zone_active_anon 4440 +4.9% 4656 proc-vmstat.nr_zone_inactive_anon 2.807e+08 -7.1% 2.609e+08 proc-vmstat.numa_hit 2.807e+08 -7.1% 2.609e+08 proc-vmstat.numa_local 2.83e+08 -7.0% 2.632e+08 proc-vmstat.pgalloc_normal 3212744 +2.3% 3286078 proc-vmstat.pgfault 2.828e+08 -7.0% 2.631e+08 proc-vmstat.pgfree 39216 ± 35% +153.3% 99342 ± 4% numa-vmstat.node0.nr_active_anon 31864 ± 44% +211.0% 99111 ± 4% numa-vmstat.node0.nr_anon_pages 11183 ± 3% +4212.8% 482321 ± 2% numa-vmstat.node0.nr_kernel_stack 499.50 ± 23% +66.9% 833.75 ± 8% numa-vmstat.node0.nr_page_table_pages 11417 -63.5% 4168 ± 5% numa-vmstat.node0.nr_shmem 15905 ± 4% -12.5% 13919 ± 2% numa-vmstat.node0.nr_slab_reclaimable 17996 ± 3% +587.4% 123708 ± 2% numa-vmstat.node0.nr_slab_unreclaimable 39216 ± 35% +153.3% 99342 ± 4% numa-vmstat.node0.nr_zone_active_anon 52471566 ± 6% +73.3% 90915199 numa-vmstat.node0.numa_hit 52466577 ± 6% +73.2% 90896366 numa-vmstat.node0.numa_local 4990 ± 92% +277.5% 18836 ± 24% numa-vmstat.node0.numa_other 114712 ± 12% -50.1% 57234 ± 7% numa-vmstat.node1.nr_active_anon 114545 ± 12% -58.2% 47874 ± 9% numa-vmstat.node1.nr_anon_pages 462075 -96.6% 15717 ± 67% numa-vmstat.node1.nr_kernel_stack 871.50 ± 13% -37.9% 541.00 ± 13% numa-vmstat.node1.nr_page_table_pages 691.25 ± 22% +1359.2% 10086 ± 8% numa-vmstat.node1.nr_shmem 11357 ± 5% +18.1% 13414 ± 4% numa-vmstat.node1.nr_slab_reclaimable 118594 -85.5% 17243 ± 16% numa-vmstat.node1.nr_slab_unreclaimable 114712 ± 12% -50.1% 57234 ± 7% numa-vmstat.node1.nr_zone_active_anon 93308245 ± 2% -49.7% 46921467 ± 4% numa-vmstat.node1.numa_hit 93128224 ± 2% -49.8% 46755145 ± 4% numa-vmstat.node1.numa_local 157762 ± 35% +154.4% 401333 ± 4% numa-meminfo.node0.Active 157721 ± 35% +154.4% 401249 ± 4% numa-meminfo.node0.Active(anon) 128232 ± 44% +212.2% 400337 ± 4% numa-meminfo.node0.AnonPages 63487 ± 4% -12.0% 55850 ± 2% numa-meminfo.node0.KReclaimable 11691 ± 3% +4087.3% 489553 numa-meminfo.node0.KernelStack 1170596 ± 4% +104.2% 2390602 numa-meminfo.node0.MemUsed 2003 ± 23% +67.3% 3351 ± 8% numa-meminfo.node0.PageTables 63487 ± 4% -12.0% 55850 ± 2% numa-meminfo.node0.SReclaimable 72301 ± 3% +591.9% 500285 numa-meminfo.node0.SUnreclaim 45749 -63.5% 16679 ± 5% numa-meminfo.node0.Shmem 135789 ± 3% +309.6% 556136 numa-meminfo.node0.Slab 460025 ± 11% -50.2% 228882 ± 7% numa-meminfo.node1.Active 459902 ± 11% -50.2% 228802 ± 7% numa-meminfo.node1.Active(anon) 459237 ± 11% -58.3% 191354 ± 9% numa-meminfo.node1.AnonPages 45568 ± 5% +17.8% 53657 ± 4% numa-meminfo.node1.KReclaimable 468955 -96.7% 15461 ± 67% numa-meminfo.node1.KernelStack 2241450 -51.9% 1078199 ± 3% numa-meminfo.node1.MemUsed 3493 ± 13% -38.0% 2167 ± 13% numa-meminfo.node1.PageTables 45568 ± 5% +17.8% 53657 ± 4% numa-meminfo.node1.SReclaimable 479010 -85.6% 68750 ± 16% numa-meminfo.node1.SUnreclaim 2779 ± 23% +1354.1% 40419 ± 8% numa-meminfo.node1.Shmem 524580 -76.7% 122408 ± 8% numa-meminfo.node1.Slab 250988 ± 19% -52.0% 120407 ± 33% sched_debug.cfs_rq:/.MIN_vruntime.avg 1469922 ± 14% -32.1% 998119 ± 33% sched_debug.cfs_rq:/.MIN_vruntime.stddev 20766 ± 9% -25.3% 15519 ± 10% sched_debug.cfs_rq:/.load.avg 43820 ± 19% -40.6% 26044 ± 36% sched_debug.cfs_rq:/.load.stddev 250988 ± 19% -52.0% 120407 ± 33% sched_debug.cfs_rq:/.max_vruntime.avg 1469922 ± 14% -32.1% 998119 ± 33% sched_debug.cfs_rq:/.max_vruntime.stddev 10990567 ± 9% +22.9% 13509735 ± 7% sched_debug.cfs_rq:/.min_vruntime.stddev 102.40 -33.3% 68.27 ± 57% sched_debug.cfs_rq:/.removed.load_avg.max 4711 -33.4% 3136 ± 57% sched_debug.cfs_rq:/.removed.runnable_sum.max 19930 ± 10% -25.1% 14917 ± 10% sched_debug.cfs_rq:/.runnable_weight.avg -5713498 -349.0% 14226386 ± 8% sched_debug.cfs_rq:/.spread0.avg 8106715 ± 13% +283.1% 31057182 ± 6% sched_debug.cfs_rq:/.spread0.max -18025645 -99.7% -45393 sched_debug.cfs_rq:/.spread0.min 11027552 ± 9% +21.5% 13398318 ± 7% sched_debug.cfs_rq:/.spread0.stddev 570.46 ± 5% +109.2% 1193 ± 21% sched_debug.cfs_rq:/.util_est_enqueued.avg 2898 ± 11% +90.0% 5507 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.max 563.27 ± 13% +149.3% 1404 ± 18% sched_debug.cfs_rq:/.util_est_enqueued.stddev 155578 ± 22% +84.1% 286457 ± 28% sched_debug.cpu.avg_idle.min 1678 ± 43% +105.9% 3457 ± 31% sched_debug.cpu.clock.stddev 1678 ± 43% +105.9% 3457 ± 31% sched_debug.cpu.clock_task.stddev 50207 -12.3% 44017 ± 7% sched_debug.cpu.curr->pid.avg 82673 -19.5% 66514 ± 5% sched_debug.cpu.curr->pid.max 746.05 ± 11% +554.3% 4881 ± 8% sched_debug.cpu.curr->pid.min 30728 ± 2% -40.8% 18187 ± 10% sched_debug.cpu.curr->pid.stddev 0.00 ± 43% +106.0% 0.00 ± 31% sched_debug.cpu.next_balance.stddev 20.49 ± 6% +149.0% 51.03 ± 23% sched_debug.cpu.nr_running.avg 157.83 ± 9% +77.0% 279.37 ± 8% sched_debug.cpu.nr_running.max 29.95 ± 11% +137.1% 70.99 ± 16% sched_debug.cpu.nr_running.stddev 29803727 ± 9% +29.6% 38623870 ± 3% sched_debug.cpu.nr_switches.avg 51549476 ± 10% +22.5% 63171598 ± 3% sched_debug.cpu.nr_switches.max 12792570 ± 3% +43.5% 18362678 ± 4% sched_debug.cpu.nr_switches.min 2.03 ± 80% +430.9% 10.76 ± 38% sched_debug.cpu.nr_uninterruptible.avg 6.99 ± 4% +48.7% 10.39 ± 5% perf-stat.i.MPKI 7.49 ± 5% -1.2 6.29 ± 5% perf-stat.i.cache-miss-rate% 27202880 ± 2% +46.9% 39966416 ± 2% perf-stat.i.cache-misses 4.662e+08 ± 5% +71.5% 7.998e+08 ± 5% perf-stat.i.cache-references 5989366 ± 4% +12.9% 6760958 ± 5% perf-stat.i.context-switches 197319 ± 12% -19.2% 159504 ± 3% perf-stat.i.cpu-migrations 7415 ± 2% -23.3% 5686 ± 3% perf-stat.i.cycles-between-cache-misses 0.61 ± 35% -0.3 0.35 ± 4% perf-stat.i.dTLB-load-miss-rate% 1.506e+08 ± 38% -44.9% 82967539 ± 5% perf-stat.i.dTLB-load-misses 0.22 ± 4% -0.0 0.20 ± 4% perf-stat.i.dTLB-store-miss-rate% 35974976 ± 5% -13.4% 31146077 ± 6% perf-stat.i.dTLB-store-misses 1.627e+10 -3.7% 1.566e+10 perf-stat.i.dTLB-stores 46665620 ± 3% -14.6% 39838962 ± 5% perf-stat.i.iTLB-load-misses 2191 ± 5% +21.4% 2660 ± 7% perf-stat.i.instructions-per-iTLB-miss 0.47 -2.5% 0.46 perf-stat.i.ipc 6660 ± 8% -19.1% 5391 ± 8% perf-stat.i.minor-faults 62.93 ± 2% +11.1 74.06 perf-stat.i.node-load-miss-rate% 12537290 ± 4% +73.5% 21750457 ± 4% perf-stat.i.node-load-misses 5459308 ± 3% -7.4% 5057404 ± 3% perf-stat.i.node-loads 29.39 ± 2% -4.1 25.29 ± 2% perf-stat.i.node-store-miss-rate% 2863317 +13.6% 3252632 ± 2% perf-stat.i.node-store-misses 6381925 ± 2% +53.5% 9796918 ± 4% perf-stat.i.node-stores 6638 ± 8% -19.4% 5348 ± 8% perf-stat.i.page-faults 4.89 ± 4% +78.4% 8.73 perf-stat.overall.MPKI 1.29 +0.0 1.31 perf-stat.overall.branch-miss-rate% 6.13 ± 4% -1.3 4.85 ± 3% perf-stat.overall.cache-miss-rate% 2.15 +2.0% 2.19 perf-stat.overall.cpi 7187 -27.9% 5182 ± 3% perf-stat.overall.cycles-between-cache-misses 0.63 ± 39% -0.3 0.36 ± 5% perf-stat.overall.dTLB-load-miss-rate% 0.23 ± 5% -0.0 0.21 ± 3% perf-stat.overall.dTLB-store-miss-rate% 75.65 -1.1 74.59 perf-stat.overall.iTLB-load-miss-rate% 1604 ± 3% +15.9% 1859 perf-stat.overall.instructions-per-iTLB-miss 0.47 -2.0% 0.46 perf-stat.overall.ipc 63.14 ± 2% +13.6 76.70 perf-stat.overall.node-load-miss-rate% 30.74 ± 2% -5.1 25.62 ± 2% perf-stat.overall.node-store-miss-rate% 47754 ± 2% +11.7% 53321 perf-stat.overall.path-length 24893507 +40.6% 34993733 ± 2% perf-stat.ps.cache-misses 4.072e+08 ± 5% +77.1% 7.212e+08 perf-stat.ps.cache-references 4968881 ± 7% +19.0% 5911450 perf-stat.ps.context-switches 1.789e+11 +1.3% 1.812e+11 perf-stat.ps.cpu-cycles 1.59e+08 ± 39% -44.1% 88801351 ± 6% perf-stat.ps.dTLB-load-misses 2.502e+10 -2.0% 2.453e+10 perf-stat.ps.dTLB-loads 39138578 ± 5% -12.6% 34190129 ± 3% perf-stat.ps.dTLB-store-misses 1.675e+10 -3.7% 1.613e+10 perf-stat.ps.dTLB-stores 51901139 ± 2% -14.3% 44461197 perf-stat.ps.iTLB-load-misses 16711264 ± 2% -9.3% 15153636 ± 2% perf-stat.ps.iTLB-loads 3573 -9.1% 3247 perf-stat.ps.minor-faults 10427704 ± 3% +73.2% 18055799 ± 3% perf-stat.ps.node-load-misses 6085553 ± 4% -10.0% 5479559 ± 2% perf-stat.ps.node-loads 2566511 +13.4% 2911367 ± 2% perf-stat.ps.node-store-misses 5784179 ± 2% +46.2% 8454210 ± 3% perf-stat.ps.node-stores 3573 -9.1% 3247 perf-stat.ps.page-faults 7.427e+13 ± 2% +11.7% 8.293e+13 perf-stat.total.instructions 127394 ± 5% +19.1% 151665 softirqs.CPU0.RCU 321761 ± 3% +12.8% 363107 ± 3% softirqs.CPU0.TIMER 124526 ± 5% +20.2% 149697 ± 3% softirqs.CPU1.RCU 123768 ± 4% +24.7% 154332 ± 2% softirqs.CPU10.RCU 130315 ± 3% +22.1% 159111 softirqs.CPU11.RCU 126684 ± 4% +26.1% 159755 ± 2% softirqs.CPU12.RCU 124649 ± 9% +27.8% 159298 ± 3% softirqs.CPU13.RCU 124528 ± 4% +23.2% 153372 ± 2% softirqs.CPU14.RCU 326150 ± 2% +9.8% 357991 ± 4% softirqs.CPU14.TIMER 121923 ± 6% +17.4% 143119 softirqs.CPU15.RCU 122114 ± 5% +16.7% 142560 softirqs.CPU16.RCU 121425 ± 6% +14.4% 138861 ± 3% softirqs.CPU17.RCU 330948 ± 2% +22.2% 404570 ± 16% softirqs.CPU17.TIMER 312317 ± 4% +18.2% 369091 ± 2% softirqs.CPU18.TIMER 9626 ± 2% +15.0% 11067 ± 3% softirqs.CPU19.SCHED 322308 ± 3% +21.1% 390454 ± 2% softirqs.CPU19.TIMER 124281 ± 5% +22.8% 152670 ± 2% softirqs.CPU2.RCU 9452 ± 2% +15.6% 10927 ± 3% softirqs.CPU20.SCHED 318575 ± 6% +22.3% 389741 ± 3% softirqs.CPU20.TIMER 9327 ± 3% +16.7% 10886 ± 3% softirqs.CPU21.SCHED 306628 ± 4% +23.3% 377962 ± 3% softirqs.CPU21.TIMER 9424 ± 2% +17.6% 11084 ± 3% softirqs.CPU22.SCHED 302796 ± 4% +24.2% 376126 ± 2% softirqs.CPU22.TIMER 9798 ± 2% +11.4% 10918 ± 4% softirqs.CPU23.SCHED 312009 ± 4% +23.6% 385626 ± 2% softirqs.CPU23.TIMER 9559 ± 3% +13.2% 10822 softirqs.CPU24.SCHED 311772 ± 4% +23.0% 383504 ± 2% softirqs.CPU24.TIMER 9514 ± 4% +13.2% 10766 ± 3% softirqs.CPU25.SCHED 307640 ± 4% +22.4% 376479 ± 2% softirqs.CPU25.TIMER 9573 ± 3% +13.8% 10897 ± 3% softirqs.CPU26.SCHED 314607 ± 9% +21.1% 380918 ± 2% softirqs.CPU26.TIMER 9385 +13.7% 10668 ± 2% softirqs.CPU27.SCHED 301949 ± 4% +25.2% 378140 ± 3% softirqs.CPU27.TIMER 9483 ± 2% +14.7% 10874 ± 2% softirqs.CPU28.SCHED 299462 ± 3% +24.9% 373896 ± 2% softirqs.CPU28.TIMER 9505 ± 3% +13.3% 10769 ± 3% softirqs.CPU29.SCHED 307087 ± 4% +23.8% 380066 ± 3% softirqs.CPU29.TIMER 124992 ± 4% +19.4% 149253 ± 3% softirqs.CPU3.RCU 9630 ± 4% +13.6% 10942 ± 3% softirqs.CPU30.SCHED 306382 ± 4% +25.4% 384355 ± 2% softirqs.CPU30.TIMER 9496 ± 6% +12.6% 10689 ± 2% softirqs.CPU31.SCHED 308689 ± 4% +24.5% 384419 ± 2% softirqs.CPU31.TIMER 9714 +12.6% 10937 ± 4% softirqs.CPU32.SCHED 304137 ± 3% +35.9% 413240 ± 13% softirqs.CPU32.TIMER 9598 ± 2% +13.4% 10882 ± 5% softirqs.CPU33.SCHED 9622 +13.8% 10949 ± 3% softirqs.CPU34.SCHED 305882 ± 4% +27.3% 389315 ± 4% softirqs.CPU34.TIMER 9409 ± 4% +15.3% 10853 ± 4% softirqs.CPU35.SCHED 306215 ± 4% +23.8% 379222 ± 2% softirqs.CPU35.TIMER 321617 ± 4% +12.9% 362966 ± 3% softirqs.CPU36.TIMER 134843 ± 6% +19.4% 161053 ± 2% softirqs.CPU37.RCU 130954 ± 6% +20.4% 157716 softirqs.CPU38.RCU 129884 ± 5% +17.8% 152978 softirqs.CPU39.RCU 124744 ± 4% +20.1% 149790 ± 3% softirqs.CPU4.RCU 130593 ± 5% +17.2% 153092 ± 2% softirqs.CPU40.RCU 134185 ± 5% +19.9% 160871 softirqs.CPU41.RCU 133336 ± 5% +18.1% 157409 softirqs.CPU42.RCU 134486 ± 4% +18.7% 159613 softirqs.CPU43.RCU 129857 ± 6% +18.5% 153887 softirqs.CPU44.RCU 131816 ± 5% +20.0% 158144 softirqs.CPU45.RCU 132691 ± 5% +22.4% 162412 softirqs.CPU46.RCU 138736 ± 3% +18.7% 164668 softirqs.CPU47.RCU 132971 ± 6% +24.2% 165142 softirqs.CPU48.RCU 136142 ± 3% +20.8% 164493 softirqs.CPU49.RCU 125011 ± 4% +23.6% 154462 ± 2% softirqs.CPU5.RCU 133660 ± 4% +19.5% 159753 ± 2% softirqs.CPU50.RCU 140405 ± 4% +19.5% 167772 softirqs.CPU51.RCU 140523 ± 4% +18.5% 166576 softirqs.CPU52.RCU 138849 ± 5% +14.6% 159157 ± 4% softirqs.CPU53.RCU 332171 ± 2% +21.4% 403260 ± 15% softirqs.CPU53.TIMER 133037 ± 4% +14.0% 151688 ± 2% softirqs.CPU54.RCU 9584 +14.3% 10959 ± 2% softirqs.CPU54.SCHED 310603 ± 4% +18.0% 366472 ± 3% softirqs.CPU54.TIMER 9565 ± 2% +13.8% 10889 ± 5% softirqs.CPU55.SCHED 321231 ± 3% +21.3% 389753 ± 2% softirqs.CPU55.TIMER 136633 ± 4% +12.6% 153854 softirqs.CPU56.RCU 9430 ± 2% +15.6% 10898 ± 5% softirqs.CPU56.SCHED 319477 ± 6% +22.1% 390107 ± 2% softirqs.CPU56.TIMER 128672 ± 4% +14.1% 146791 ± 3% softirqs.CPU57.RCU 9493 ± 2% +13.1% 10740 ± 3% softirqs.CPU57.SCHED 305670 ± 4% +23.5% 377597 ± 2% softirqs.CPU57.TIMER 127129 ± 4% +15.6% 146993 ± 2% softirqs.CPU58.RCU 9529 ± 3% +13.6% 10824 ± 4% softirqs.CPU58.SCHED 302317 ± 4% +23.8% 374371 ± 3% softirqs.CPU58.TIMER 9427 ± 2% +14.0% 10743 ± 2% softirqs.CPU59.SCHED 310985 ± 4% +23.5% 384119 ± 3% softirqs.CPU59.TIMER 124937 ± 4% +21.6% 151962 ± 3% softirqs.CPU6.RCU 9435 ± 2% +14.1% 10762 ± 3% softirqs.CPU60.SCHED 311646 ± 4% +22.8% 382731 ± 3% softirqs.CPU60.TIMER 9585 +13.5% 10877 ± 2% softirqs.CPU61.SCHED 306857 ± 4% +22.8% 376944 ± 2% softirqs.CPU61.TIMER 9483 ± 2% +12.5% 10671 ± 5% softirqs.CPU62.SCHED 9620 ± 4% +9.7% 10556 ± 4% softirqs.CPU63.SCHED 301085 ± 3% +24.7% 375397 ± 3% softirqs.CPU63.TIMER 300443 ± 4% +23.7% 371620 ± 2% softirqs.CPU64.TIMER 9667 +11.2% 10751 ± 4% softirqs.CPU65.SCHED 307194 ± 4% +23.4% 379020 ± 3% softirqs.CPU65.TIMER 9493 ± 3% +14.9% 10909 ± 3% softirqs.CPU66.SCHED 306226 ± 4% +25.1% 383140 ± 3% softirqs.CPU66.TIMER 9674 ± 3% +13.8% 11010 ± 3% softirqs.CPU67.SCHED 307870 ± 4% +25.0% 384793 ± 2% softirqs.CPU67.TIMER 9451 ± 2% +14.2% 10792 ± 4% softirqs.CPU68.SCHED 301873 ± 3% +26.1% 380748 softirqs.CPU68.TIMER 9542 +13.8% 10863 ± 3% softirqs.CPU69.SCHED 316206 ± 8% +20.5% 380893 ± 2% softirqs.CPU69.TIMER 128158 ± 4% +20.4% 154294 ± 2% softirqs.CPU7.RCU 9456 ± 3% +13.5% 10734 ± 3% softirqs.CPU70.SCHED 305361 ± 4% +36.4% 416546 ± 15% softirqs.CPU70.TIMER 9520 ± 3% +15.4% 10985 ± 4% softirqs.CPU71.SCHED 305605 ± 4% +24.6% 380653 ± 2% softirqs.CPU71.TIMER 122831 ± 4% +21.6% 149424 ± 2% softirqs.CPU8.RCU 123870 ± 4% +23.0% 152396 ± 2% softirqs.CPU9.RCU 9357957 ± 5% +15.4% 10794953 softirqs.RCU 23209193 ± 3% +15.5% 26806870 ± 3% softirqs.TIMER 4.79 ± 4% -3.1 1.71 ± 16% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.85 ± 4% -2.4 1.41 ± 16% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.39 ± 4% -2.0 1.35 ± 16% perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read 2.93 ± 5% -1.9 0.99 ± 18% perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write 3.25 ± 4% -1.9 1.31 ± 10% perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_write.new_sync_write.vfs_write.ksys_write 2.14 ± 4% -1.2 0.91 ± 14% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64 2.06 ± 4% -1.2 0.85 ± 13% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64 1.52 ± 3% -1.0 0.49 ± 59% perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.pipe_read.new_sync_read.vfs_read 1.46 ± 3% -1.0 0.46 ± 59% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.pipe_read.new_sync_read 1.53 ± 8% -0.8 0.68 ± 12% perf-profile.calltrace.cycles-pp.touch_atime.pipe_read.new_sync_read.vfs_read.ksys_read 1.65 ± 4% -0.8 0.81 ± 10% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.45 ± 5% -0.7 0.74 ± 10% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.35 ± 5% -0.7 0.68 ± 10% perf-profile.calltrace.cycles-pp.__fget.__fget_light.__fdget_pos.ksys_write.do_syscall_64 1.28 ± 8% -0.7 0.62 ± 12% perf-profile.calltrace.cycles-pp.file_update_time.pipe_write.new_sync_write.vfs_write.ksys_write 1.29 ± 3% -0.6 0.67 ± 9% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.new_sync_write.vfs_write.ksys_write 0.64 ± 11% +0.3 0.96 ± 7% perf-profile.calltrace.cycles-pp.__lock_text_start.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 0.00 +0.5 0.55 ± 3% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop 0.00 +0.6 0.55 ± 5% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.00 +0.6 0.57 ± 6% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.00 +0.6 0.61 ± 7% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule 0.00 +0.7 0.69 ± 8% perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.pipe_wait.pipe_read 0.00 +0.7 0.70 ± 6% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.00 +0.7 0.71 ± 11% perf-profile.calltrace.cycles-pp.__enqueue_entity.put_prev_entity.pick_next_task_fair.__schedule.schedule 0.00 +0.7 0.72 ± 7% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.00 +0.7 0.72 ± 6% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.dequeue_task_fair.__schedule.schedule.pipe_wait 0.00 +0.8 0.76 ± 10% perf-profile.calltrace.cycles-pp.__enqueue_entity.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.00 +0.8 0.77 ± 2% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule 0.00 +0.8 0.77 ± 5% perf-profile.calltrace.cycles-pp.__switch_to 0.00 +0.8 0.77 ± 6% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read 0.00 +0.8 0.79 ± 5% perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule 0.00 +0.8 0.81 ± 4% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_wait.pipe_read.new_sync_read.vfs_read 0.00 +0.8 0.82 ± 6% perf-profile.calltrace.cycles-pp.update_rq_clock.__schedule.schedule.pipe_wait.pipe_read 0.00 +0.8 0.82 ± 6% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.00 +0.9 0.90 ± 3% perf-profile.calltrace.cycles-pp.find_next_bit.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up 0.00 +0.9 0.91 ± 4% perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.00 +0.9 0.91 ± 6% perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function 0.00 +0.9 0.92 ± 2% perf-profile.calltrace.cycles-pp.native_write_msr 0.00 +1.1 1.05 ± 5% perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.00 +1.1 1.08 ± 8% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop 0.00 +1.1 1.13 ± 5% perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.00 +1.2 1.17 ± 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.00 +1.2 1.18 ± 2% perf-profile.calltrace.cycles-pp.__switch_to_asm 0.00 +1.2 1.20 ± 9% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.pipe_wait.pipe_read 0.00 +1.3 1.30 ± 5% perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.00 +1.3 1.34 ± 6% perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 0.00 +1.6 1.62 ± 4% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +1.9 1.88 ± 5% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_wait.pipe_read 0.00 +1.9 1.93 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.13 ±173% +2.0 2.11 ± 5% perf-profile.calltrace.cycles-pp.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function 0.00 +2.1 2.06 ± 4% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 0.00 +2.4 2.38 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 0.00 +2.7 2.65 ± 4% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.35 ±103% +4.3 4.67 ± 4% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.22 ±173% +4.6 4.81 ± 4% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.23 ±173% +4.7 4.97 ± 4% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.72 ± 23% +5.0 5.72 ± 5% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_wait.pipe_read 0.69 ± 17% +5.6 6.25 ± 4% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 0.74 ± 17% +5.8 6.50 ± 4% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.74 ± 18% +5.8 6.55 ± 5% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 15.99 ± 54% +7.5 23.51 ± 6% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.13 +7.7 20.86 ± 3% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.49 +8.7 20.19 ± 3% perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64 1.31 ± 27% +8.9 10.21 ± 7% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function 1.81 ± 22% +11.2 12.98 ± 5% perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait.pipe_read.new_sync_read 1.89 ± 22% +11.4 13.28 ± 5% perf-profile.calltrace.cycles-pp.schedule.pipe_wait.pipe_read.new_sync_read.vfs_read 2.18 ± 21% +13.1 15.28 ± 4% perf-profile.calltrace.cycles-pp.pipe_wait.pipe_read.new_sync_read.vfs_read.ksys_read 2.01 ± 25% +19.1 21.10 ± 7% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common 2.38 ± 23% +20.0 22.40 ± 7% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 19.01 ± 3% +23.4 42.38 ± 4% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 21.13 ± 53% +23.8 44.92 ± 5% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 17.85 ± 4% +23.9 41.76 ± 4% perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 19.16 ± 53% +24.6 43.74 ± 5% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.79 ± 22% +30.7 34.45 ± 6% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write 3.84 ± 23% +30.9 34.69 ± 6% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write 4.13 ± 21% +31.0 35.10 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 5.77 ± 14% +31.0 36.76 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write 9.20 ± 6% -6.0 3.16 ± 12% perf-profile.children.cycles-pp.syscall_return_via_sysret 8.88 ± 3% -5.8 3.03 ± 12% perf-profile.children.cycles-pp.entry_SYSCALL_64 8.72 ± 4% -5.3 3.39 ± 11% perf-profile.children.cycles-pp.security_file_permission 4.27 ± 3% -2.4 1.89 ± 9% perf-profile.children.cycles-pp.selinux_file_permission 3.15 ± 5% -2.1 1.07 ± 12% perf-profile.children.cycles-pp.file_has_perm 3.48 ± 3% -2.0 1.46 ± 10% perf-profile.children.cycles-pp.copy_page_to_iter 3.80 ± 3% -2.0 1.79 ± 6% perf-profile.children.cycles-pp.mutex_unlock 3.02 ± 5% -1.9 1.11 ± 11% perf-profile.children.cycles-pp.copy_page_from_iter 3.31 ± 4% -1.6 1.75 ± 6% perf-profile.children.cycles-pp.__fdget_pos 2.55 ± 3% -1.5 1.02 ± 10% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 3.17 ± 4% -1.5 1.70 ± 6% perf-profile.children.cycles-pp.__fget_light 2.10 ± 7% -1.4 0.71 ± 12% perf-profile.children.cycles-pp.fsnotify 2.52 ± 6% -1.3 1.23 ± 8% perf-profile.children.cycles-pp.__fget 1.77 ± 6% -1.1 0.66 ± 9% perf-profile.children.cycles-pp.avc_has_perm 1.89 ± 5% -1.1 0.78 ± 10% perf-profile.children.cycles-pp.___might_sleep 1.44 ± 4% -1.0 0.48 ± 12% perf-profile.children.cycles-pp.__inode_security_revalidate 1.33 ± 10% -0.9 0.40 ± 16% perf-profile.children.cycles-pp.current_time 1.56 ± 3% -0.9 0.66 ± 9% perf-profile.children.cycles-pp.copyout 1.56 ± 8% -0.8 0.73 ± 7% perf-profile.children.cycles-pp.touch_atime 1.33 ± 4% -0.8 0.53 ± 10% perf-profile.children.cycles-pp.__might_sleep 1.15 ± 9% -0.8 0.36 ± 12% perf-profile.children.cycles-pp.atime_needs_update 1.17 ± 4% -0.7 0.43 ± 13% perf-profile.children.cycles-pp.copyin 1.23 ± 5% -0.7 0.54 ± 9% perf-profile.children.cycles-pp.__fsnotify_parent 1.32 ± 7% -0.6 0.68 ± 6% perf-profile.children.cycles-pp.file_update_time 1.02 ± 3% -0.6 0.38 ± 12% perf-profile.children.cycles-pp.__might_fault 2.54 ± 2% -0.5 2.03 perf-profile.children.cycles-pp.mutex_lock 0.78 ± 4% -0.4 0.36 ± 10% perf-profile.children.cycles-pp.fput_many 0.53 ± 4% -0.4 0.15 ± 21% perf-profile.children.cycles-pp.inode_has_perm 0.44 ± 12% -0.3 0.11 ± 22% perf-profile.children.cycles-pp.iov_iter_init 0.76 ± 4% -0.3 0.44 ± 8% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.53 ± 3% -0.3 0.21 ± 6% perf-profile.children.cycles-pp.rcu_all_qs 0.41 ± 9% -0.3 0.12 ± 15% perf-profile.children.cycles-pp.timespec64_trunc 0.97 -0.3 0.71 ± 3% perf-profile.children.cycles-pp._cond_resched 0.47 ± 7% -0.3 0.21 ± 11% perf-profile.children.cycles-pp.__x64_sys_read 0.45 ± 7% -0.3 0.20 ± 10% perf-profile.children.cycles-pp.__x64_sys_write 0.37 ± 4% -0.2 0.17 ± 16% perf-profile.children.cycles-pp.generic_pipe_buf_confirm 0.44 ± 12% -0.2 0.24 ± 3% perf-profile.children.cycles-pp.__sb_start_write 0.31 ± 6% -0.2 0.12 ± 13% perf-profile.children.cycles-pp.__sb_end_write 0.36 ± 7% -0.2 0.18 ± 6% perf-profile.children.cycles-pp.rw_verify_area 0.26 ± 14% -0.2 0.09 ± 14% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64 0.29 ± 8% -0.2 0.14 ± 16% perf-profile.children.cycles-pp.fpregs_assert_state_consistent 0.22 ± 39% -0.2 0.06 ± 11% perf-profile.children.cycles-pp.__vfs_read 0.21 ± 19% -0.1 0.11 ± 9% perf-profile.children.cycles-pp.__mutex_unlock_slowpath 0.12 ± 9% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__vfs_write 0.12 ± 17% -0.0 0.08 ± 11% perf-profile.children.cycles-pp.wake_up_q 0.08 ± 10% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.kill_fasync 0.11 ± 7% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.09 ± 5% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.prepare_exit_to_usermode 0.01 ±173% +0.1 0.06 ± 6% perf-profile.children.cycles-pp.__mnt_want_write 0.07 ± 25% +0.1 0.12 ± 27% perf-profile.children.cycles-pp.__softirqentry_text_start 0.08 ± 30% +0.1 0.14 ± 26% perf-profile.children.cycles-pp.irq_exit 0.06 ± 6% +0.1 0.12 ± 10% perf-profile.children.cycles-pp.__mark_inode_dirty 0.00 +0.1 0.06 perf-profile.children.cycles-pp.preempt_schedule_common 0.00 +0.1 0.07 ± 6% perf-profile.children.cycles-pp.is_cpu_allowed 0.03 ±100% +0.1 0.10 ± 8% perf-profile.children.cycles-pp.native_apic_msr_eoi_write 0.00 +0.1 0.07 ± 5% perf-profile.children.cycles-pp.__x2apic_send_IPI_dest 0.01 ±173% +0.1 0.09 ± 32% perf-profile.children.cycles-pp.migrate_task_rq_fair 0.00 +0.1 0.07 ± 14% perf-profile.children.cycles-pp.update_cfs_rq_h_load 0.05 ± 61% +0.1 0.13 ± 5% perf-profile.children.cycles-pp.native_irq_return_iret 0.00 +0.1 0.08 ± 8% perf-profile.children.cycles-pp.rcu_note_context_switch 0.00 +0.1 0.08 ± 13% perf-profile.children.cycles-pp.smp_reschedule_interrupt 0.00 +0.1 0.09 ± 4% perf-profile.children.cycles-pp.native_load_tls 0.03 ±102% +0.1 0.12 ± 30% perf-profile.children.cycles-pp.set_task_cpu 0.00 +0.1 0.09 ± 11% perf-profile.children.cycles-pp.check_cfs_rq_runtime 0.00 +0.1 0.10 ± 5% perf-profile.children.cycles-pp.rb_next 0.10 ± 10% +0.1 0.20 ± 9% perf-profile.children.cycles-pp.generic_update_time 0.00 +0.1 0.10 ± 29% perf-profile.children.cycles-pp.__x64_sys_exit 0.00 +0.1 0.10 ± 28% perf-profile.children.cycles-pp.do_exit 0.00 +0.1 0.11 ± 8% perf-profile.children.cycles-pp.rb_insert_color 0.00 +0.1 0.11 ± 7% perf-profile.children.cycles-pp.__list_add_valid 0.00 +0.1 0.12 ± 7% perf-profile.children.cycles-pp.resched_curr 0.00 +0.1 0.12 ± 6% perf-profile.children.cycles-pp.switch_mm_irqs_off 0.13 ± 10% +0.1 0.26 ± 4% perf-profile.children.cycles-pp.__list_del_entry_valid 0.00 +0.1 0.15 ± 4% perf-profile.children.cycles-pp.set_next_buddy 0.00 +0.2 0.16 ± 5% perf-profile.children.cycles-pp.deactivate_task 0.05 ± 9% +0.2 0.22 ± 7% perf-profile.children.cycles-pp.rb_erase 0.06 ± 13% +0.2 0.23 ± 9% perf-profile.children.cycles-pp.finish_wait 0.17 ± 12% +0.2 0.35 ± 2% perf-profile.children.cycles-pp.anon_pipe_buf_release 0.01 ±173% +0.2 0.20 ± 8% perf-profile.children.cycles-pp.wakeup_preempt_entity 0.00 +0.2 0.21 ± 4% perf-profile.children.cycles-pp.cpumask_next 0.07 ± 22% +0.3 0.40 ± 2% perf-profile.children.cycles-pp.prepare_to_wait 0.03 ±100% +0.4 0.39 ± 5% perf-profile.children.cycles-pp.update_min_vruntime 0.03 ±100% +0.4 0.39 ± 12% perf-profile.children.cycles-pp.cpus_share_cache 0.06 ± 14% +0.4 0.43 perf-profile.children.cycles-pp.clear_buddies 0.11 ± 15% +0.4 0.50 ± 3% perf-profile.children.cycles-pp.cpuacct_charge 0.10 ± 26% +0.4 0.49 ± 4% perf-profile.children.cycles-pp.native_sched_clock 0.32 ± 31% +0.4 0.75 ± 14% perf-profile.children.cycles-pp.sched_ttwu_pending 0.11 ± 26% +0.4 0.56 ± 3% perf-profile.children.cycles-pp.sched_clock 0.34 ± 32% +0.5 0.80 ± 15% perf-profile.children.cycles-pp.scheduler_ipi 0.08 ± 12% +0.5 0.56 ± 7% perf-profile.children.cycles-pp.account_entity_enqueue 0.13 ± 26% +0.5 0.62 ± 3% perf-profile.children.cycles-pp.sched_clock_cpu 0.14 ± 26% +0.6 0.77 ± 4% perf-profile.children.cycles-pp.__calc_delta 0.08 ± 16% +0.7 0.76 ± 3% perf-profile.children.cycles-pp.account_entity_dequeue 0.84 ± 5% +0.7 1.54 ± 5% perf-profile.children.cycles-pp.__lock_text_start 1.61 ± 16% +0.7 2.31 ± 9% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.42 ± 31% +0.7 1.12 ± 13% perf-profile.children.cycles-pp.reschedule_interrupt 0.24 ± 28% +0.9 1.11 perf-profile.children.cycles-pp.native_write_msr 0.14 ± 22% +0.9 1.01 ± 5% perf-profile.children.cycles-pp.check_preempt_wakeup 0.24 ± 32% +0.9 1.12 ± 7% perf-profile.children.cycles-pp.finish_task_switch 0.22 ± 22% +0.9 1.10 ± 3% perf-profile.children.cycles-pp.find_next_bit 0.22 ± 23% +0.9 1.12 ± 3% perf-profile.children.cycles-pp.set_next_entity 0.21 ± 32% +0.9 1.14 ± 3% perf-profile.children.cycles-pp.update_cfs_group 0.16 ± 23% +1.0 1.15 ± 4% perf-profile.children.cycles-pp.check_preempt_curr 0.21 ± 30% +1.0 1.21 ± 2% perf-profile.children.cycles-pp.__switch_to 0.23 ± 21% +1.0 1.24 ± 2% perf-profile.children.cycles-pp.__switch_to_asm 0.16 ± 38% +1.0 1.20 ± 4% perf-profile.children.cycles-pp.___perf_sw_event 0.15 ± 26% +1.0 1.20 ± 3% perf-profile.children.cycles-pp.pick_next_entity 0.15 ± 20% +1.1 1.21 ± 7% perf-profile.children.cycles-pp.put_prev_entity 0.17 ± 22% +1.1 1.23 ± 4% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.19 ± 10% +1.3 1.52 ± 10% perf-profile.children.cycles-pp.__enqueue_entity 0.24 ± 23% +1.3 1.59 ± 6% perf-profile.children.cycles-pp.__update_load_avg_se 0.28 ± 23% +1.4 1.64 ± 3% perf-profile.children.cycles-pp.switch_fpu_return 0.31 ± 29% +1.6 1.88 ± 5% perf-profile.children.cycles-pp.update_rq_clock 0.44 ± 21% +1.8 2.24 ± 4% perf-profile.children.cycles-pp.cpumask_next_wrap 0.33 ± 20% +1.8 2.15 ± 4% perf-profile.children.cycles-pp.dequeue_entity 23.28 ± 2% +2.0 25.27 ± 2% perf-profile.children.cycles-pp.ksys_read 0.46 ± 25% +2.1 2.57 ± 5% perf-profile.children.cycles-pp.update_load_avg 0.20 ± 21% +2.1 2.34 ± 4% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.91 ± 29% +2.2 3.12 ± 11% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.38 ± 23% +2.4 2.80 ± 4% perf-profile.children.cycles-pp.reweight_entity 0.52 ± 18% +2.4 2.96 ± 3% perf-profile.children.cycles-pp.enqueue_entity 0.49 ± 23% +2.6 3.10 ± 4% perf-profile.children.cycles-pp.update_curr 20.92 ± 2% +3.2 24.12 ± 2% perf-profile.children.cycles-pp.vfs_read 0.21 ± 23% +3.4 3.65 ± 7% perf-profile.children.cycles-pp._raw_spin_lock 0.76 ± 24% +3.9 4.67 ± 3% perf-profile.children.cycles-pp.pick_next_task_fair 0.66 ± 34% +4.4 5.06 ± 2% perf-profile.children.cycles-pp.exit_to_usermode_loop 0.81 ± 22% +5.0 5.83 ± 4% perf-profile.children.cycles-pp.dequeue_task_fair 1.01 ± 21% +5.8 6.86 ± 4% perf-profile.children.cycles-pp.enqueue_task_fair 1.06 ± 21% +6.0 7.10 ± 4% perf-profile.children.cycles-pp.activate_task 1.07 ± 21% +6.1 7.15 ± 4% perf-profile.children.cycles-pp.ttwu_do_activate 13.19 +7.8 20.94 ± 3% perf-profile.children.cycles-pp.new_sync_read 11.61 +8.7 20.29 ± 3% perf-profile.children.cycles-pp.pipe_read 1.45 ± 25% +8.9 10.35 ± 7% perf-profile.children.cycles-pp.available_idle_cpu 78.85 ± 2% +9.8 88.66 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 77.90 ± 2% +10.1 88.00 perf-profile.children.cycles-pp.do_syscall_64 2.42 ± 21% +12.9 15.36 ± 4% perf-profile.children.cycles-pp.pipe_wait 2.71 ± 26% +15.4 18.12 ± 4% perf-profile.children.cycles-pp.__schedule 2.76 ± 25% +15.6 18.37 ± 4% perf-profile.children.cycles-pp.schedule 27.59 ± 2% +18.6 46.17 ± 3% perf-profile.children.cycles-pp.ksys_write 2.26 ± 23% +19.0 21.30 ± 7% perf-profile.children.cycles-pp.select_idle_sibling 24.98 ± 2% +19.9 44.88 ± 3% perf-profile.children.cycles-pp.vfs_write 2.48 ± 23% +20.0 22.46 ± 7% perf-profile.children.cycles-pp.select_task_rq_fair 19.06 ± 3% +23.3 42.41 ± 4% perf-profile.children.cycles-pp.new_sync_write 17.98 ± 4% +23.9 41.93 ± 4% perf-profile.children.cycles-pp.pipe_write 4.03 ± 21% +30.6 34.60 ± 6% perf-profile.children.cycles-pp.try_to_wake_up 3.97 ± 22% +30.8 34.74 ± 6% perf-profile.children.cycles-pp.autoremove_wake_function 4.30 ± 20% +30.9 35.19 ± 6% perf-profile.children.cycles-pp.__wake_up_common 6.06 ± 14% +31.2 37.21 ± 5% perf-profile.children.cycles-pp.__wake_up_common_lock 24.41 ± 6% -15.5 8.96 ± 10% perf-profile.self.cycles-pp.do_syscall_64 9.18 ± 6% -6.0 3.14 ± 12% perf-profile.self.cycles-pp.syscall_return_via_sysret 8.63 ± 5% -5.6 3.00 ± 12% perf-profile.self.cycles-pp.entry_SYSCALL_64 3.71 ± 3% -2.0 1.73 ± 6% perf-profile.self.cycles-pp.mutex_unlock 2.47 ± 3% -1.5 0.96 ± 10% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 2.76 ± 3% -1.4 1.35 ± 8% perf-profile.self.cycles-pp.selinux_file_permission 2.02 ± 7% -1.3 0.68 ± 12% perf-profile.self.cycles-pp.fsnotify 2.46 ± 6% -1.3 1.20 ± 8% perf-profile.self.cycles-pp.__fget 1.75 ± 6% -1.1 0.64 ± 10% perf-profile.self.cycles-pp.avc_has_perm 1.83 ± 5% -1.1 0.75 ± 10% perf-profile.self.cycles-pp.___might_sleep 1.75 -1.0 0.77 ± 7% perf-profile.self.cycles-pp.pipe_write 1.20 ± 4% -0.7 0.46 ± 10% perf-profile.self.cycles-pp.__might_sleep 1.29 ± 11% -0.7 0.58 ± 10% perf-profile.self.cycles-pp.new_sync_read 1.63 ± 4% -0.7 0.97 ± 5% perf-profile.self.cycles-pp.pipe_read 1.15 ± 5% -0.7 0.49 ± 8% perf-profile.self.cycles-pp.__fsnotify_parent 0.89 ± 8% -0.6 0.27 ± 15% perf-profile.self.cycles-pp.copy_page_from_iter 0.89 ± 4% -0.6 0.29 ± 14% perf-profile.self.cycles-pp.security_file_permission 0.83 ± 5% -0.6 0.24 ± 21% perf-profile.self.cycles-pp.file_has_perm 0.90 ± 2% -0.5 0.41 ± 6% perf-profile.self.cycles-pp.new_sync_write 0.67 ± 10% -0.5 0.20 ± 15% perf-profile.self.cycles-pp.current_time 0.83 ± 4% -0.4 0.41 ± 10% perf-profile.self.cycles-pp.copy_page_to_iter 0.72 ± 7% -0.4 0.31 ± 9% perf-profile.self.cycles-pp.vfs_write 0.75 ± 4% -0.4 0.35 ± 9% perf-profile.self.cycles-pp.fput_many 1.01 ± 4% -0.3 0.69 ± 5% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.45 ± 4% -0.3 0.14 ± 19% perf-profile.self.cycles-pp.inode_has_perm 0.54 ± 8% -0.3 0.23 ± 9% perf-profile.self.cycles-pp.file_update_time 0.41 ± 10% -0.3 0.11 ± 20% perf-profile.self.cycles-pp.iov_iter_init 0.49 ± 7% -0.3 0.19 ± 8% perf-profile.self.cycles-pp.atime_needs_update 0.44 ± 7% -0.3 0.15 ± 14% perf-profile.self.cycles-pp.ksys_read 0.44 ± 7% -0.3 0.17 ± 9% perf-profile.self.cycles-pp.__inode_security_revalidate 0.41 ± 2% -0.3 0.15 ± 8% perf-profile.self.cycles-pp.rcu_all_qs 0.43 ± 8% -0.3 0.17 ± 11% perf-profile.self.cycles-pp.ksys_write 0.37 ± 9% -0.3 0.11 ± 17% perf-profile.self.cycles-pp.timespec64_trunc 0.42 ± 14% -0.2 0.18 ± 8% perf-profile.self.cycles-pp.__sb_start_write 0.39 ± 7% -0.2 0.17 ± 11% perf-profile.self.cycles-pp.__x64_sys_write 0.40 ± 8% -0.2 0.17 ± 8% perf-profile.self.cycles-pp.__x64_sys_read 0.41 ± 7% -0.2 0.20 ± 11% perf-profile.self.cycles-pp.__wake_up_common_lock 0.60 ± 4% -0.2 0.39 ± 7% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 1.25 -0.2 1.04 ± 2% perf-profile.self.cycles-pp.mutex_lock 0.30 ± 6% -0.2 0.12 ± 15% perf-profile.self.cycles-pp.__sb_end_write 0.35 ± 5% -0.2 0.18 ± 3% perf-profile.self.cycles-pp.rw_verify_area 0.34 ± 7% -0.2 0.18 ± 8% perf-profile.self.cycles-pp.touch_atime 0.32 ± 4% -0.2 0.15 ± 14% perf-profile.self.cycles-pp.generic_pipe_buf_confirm 0.23 ± 16% -0.2 0.08 ± 11% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64 0.74 ± 3% -0.2 0.59 ± 5% perf-profile.self.cycles-pp.vfs_read 0.60 ± 3% -0.2 0.45 ± 4% perf-profile.self.cycles-pp.__fget_light 0.27 ± 8% -0.1 0.12 ± 15% perf-profile.self.cycles-pp.fpregs_assert_state_consistent 0.18 ± 45% -0.1 0.04 ± 58% perf-profile.self.cycles-pp.__vfs_read 0.18 ± 5% -0.1 0.06 ± 17% perf-profile.self.cycles-pp.__fdget_pos 0.25 ± 5% -0.1 0.14 ± 11% perf-profile.self.cycles-pp.__might_fault 0.07 ± 7% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.kill_fasync 0.01 ±173% +0.1 0.06 ± 17% perf-profile.self.cycles-pp.sched_ttwu_pending 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.scheduler_ipi 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.rcu_note_context_switch 0.01 ±173% +0.1 0.07 ± 12% perf-profile.self.cycles-pp.generic_update_time 0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.cpumask_next 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.ttwu_do_activate 0.00 +0.1 0.06 perf-profile.self.cycles-pp.__mnt_want_write 0.06 ± 14% +0.1 0.12 ± 10% perf-profile.self.cycles-pp.__mark_inode_dirty 0.00 +0.1 0.06 ± 13% perf-profile.self.cycles-pp.is_cpu_allowed 0.00 +0.1 0.07 ± 13% perf-profile.self.cycles-pp.sched_clock_cpu 0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.sched_clock 0.03 ±100% +0.1 0.10 ± 11% perf-profile.self.cycles-pp.native_apic_msr_eoi_write 0.00 +0.1 0.07 ± 14% perf-profile.self.cycles-pp.update_cfs_rq_h_load 0.00 +0.1 0.08 perf-profile.self.cycles-pp.native_load_tls 0.05 ± 60% +0.1 0.13 ± 5% perf-profile.self.cycles-pp.native_irq_return_iret 0.00 +0.1 0.09 ± 9% perf-profile.self.cycles-pp.ttwu_do_wakeup 0.00 +0.1 0.09 ± 11% perf-profile.self.cycles-pp.rb_next 0.00 +0.1 0.10 ± 8% perf-profile.self.cycles-pp.__list_add_valid 0.00 +0.1 0.10 ± 10% perf-profile.self.cycles-pp.rb_insert_color 0.00 +0.1 0.11 ± 9% perf-profile.self.cycles-pp.resched_curr 0.00 +0.1 0.11 ± 7% perf-profile.self.cycles-pp.switch_mm_irqs_off 0.32 ± 7% +0.1 0.45 ± 5% perf-profile.self.cycles-pp.__wake_up_common 0.13 ± 10% +0.1 0.25 ± 5% perf-profile.self.cycles-pp.__list_del_entry_valid 0.00 +0.1 0.13 ± 5% perf-profile.self.cycles-pp.autoremove_wake_function 0.00 +0.1 0.13 ± 8% perf-profile.self.cycles-pp.check_preempt_curr 0.04 ± 58% +0.1 0.18 ± 10% perf-profile.self.cycles-pp.finish_wait 0.00 +0.1 0.14 ± 5% perf-profile.self.cycles-pp.set_next_buddy 0.00 +0.1 0.15 ± 3% perf-profile.self.cycles-pp.put_prev_entity 0.00 +0.2 0.15 ± 5% perf-profile.self.cycles-pp.deactivate_task 0.05 ± 9% +0.2 0.21 ± 5% perf-profile.self.cycles-pp.rb_erase 0.00 +0.2 0.15 ± 10% perf-profile.self.cycles-pp.exit_to_usermode_loop 0.00 +0.2 0.18 ± 6% perf-profile.self.cycles-pp.wakeup_preempt_entity 0.15 ± 7% +0.2 0.34 ± 2% perf-profile.self.cycles-pp.anon_pipe_buf_release 0.03 ±100% +0.2 0.21 ± 8% perf-profile.self.cycles-pp.pipe_wait 0.03 ±102% +0.2 0.24 ± 3% perf-profile.self.cycles-pp.activate_task 0.01 ±173% +0.2 0.23 ± 3% perf-profile.self.cycles-pp.prepare_to_wait 0.03 ±100% +0.2 0.25 ± 4% perf-profile.self.cycles-pp.set_next_entity 0.02 ±173% +0.2 0.26 ± 9% perf-profile.self.cycles-pp.dequeue_entity 0.04 ±100% +0.3 0.29 ± 5% perf-profile.self.cycles-pp.enqueue_entity 0.08 ± 31% +0.3 0.34 ± 2% perf-profile.self.cycles-pp.schedule 0.78 ± 3% +0.3 1.09 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.09 ± 24% +0.3 0.42 ± 7% perf-profile.self.cycles-pp.finish_task_switch 0.03 ±100% +0.3 0.37 ± 6% perf-profile.self.cycles-pp.update_min_vruntime 0.03 ±102% +0.4 0.38 perf-profile.self.cycles-pp.clear_buddies 0.03 ±100% +0.4 0.39 ± 12% perf-profile.self.cycles-pp.cpus_share_cache 0.07 ± 22% +0.4 0.44 ± 7% perf-profile.self.cycles-pp.check_preempt_wakeup 0.09 ± 27% +0.4 0.47 ± 4% perf-profile.self.cycles-pp.native_sched_clock 0.07 ± 26% +0.4 0.45 ± 4% perf-profile.self.cycles-pp.pick_next_entity 0.11 ± 15% +0.4 0.50 ± 3% perf-profile.self.cycles-pp.cpuacct_charge 0.07 ± 14% +0.5 0.53 ± 8% perf-profile.self.cycles-pp.account_entity_enqueue 0.10 ± 21% +0.5 0.56 ± 8% perf-profile.self.cycles-pp.try_to_wake_up 0.10 ± 18% +0.5 0.62 ± 6% perf-profile.self.cycles-pp.dequeue_task_fair 0.14 ± 26% +0.6 0.75 ± 4% perf-profile.self.cycles-pp.__calc_delta 0.06 ± 13% +0.6 0.68 ± 4% perf-profile.self.cycles-pp.account_entity_dequeue 0.15 ± 27% +0.6 0.79 ± 6% perf-profile.self.cycles-pp.enqueue_task_fair 0.17 ± 30% +0.7 0.89 ± 4% perf-profile.self.cycles-pp.update_load_avg 0.15 ± 20% +0.7 0.88 ± 4% perf-profile.self.cycles-pp.reweight_entity 0.15 ± 23% +0.8 0.96 ± 3% perf-profile.self.cycles-pp.pick_next_task_fair 0.24 ± 30% +0.9 1.11 perf-profile.self.cycles-pp.native_write_msr 0.20 ± 23% +0.9 1.07 ± 5% perf-profile.self.cycles-pp.select_task_rq_fair 0.20 ± 22% +0.9 1.07 ± 3% perf-profile.self.cycles-pp.find_next_bit 0.20 ± 31% +0.9 1.11 ± 3% perf-profile.self.cycles-pp.update_cfs_group 0.20 ± 28% +0.9 1.14 ± 2% perf-profile.self.cycles-pp.__switch_to 0.14 ± 37% +1.0 1.13 ± 5% perf-profile.self.cycles-pp.___perf_sw_event 0.21 ± 24% +1.0 1.22 ± 2% perf-profile.self.cycles-pp.__switch_to_asm 0.24 ± 23% +1.1 1.31 ± 5% perf-profile.self.cycles-pp.cpumask_next_wrap 0.23 ± 32% +1.3 1.49 ± 5% perf-profile.self.cycles-pp.update_rq_clock 0.19 ± 28% +1.3 1.45 ± 6% perf-profile.self.cycles-pp.update_curr 0.23 ± 23% +1.3 1.55 ± 6% perf-profile.self.cycles-pp.__update_load_avg_se 0.19 ± 10% +1.3 1.51 ± 10% perf-profile.self.cycles-pp.__enqueue_entity 0.28 ± 22% +1.3 1.63 ± 3% perf-profile.self.cycles-pp.switch_fpu_return 0.11 ± 19% +1.6 1.72 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 0.42 ± 26% +1.7 2.12 ± 2% perf-profile.self.cycles-pp.__schedule 0.19 ± 20% +2.0 2.20 ± 4% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.91 ± 29% +2.2 3.11 ± 12% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.32 ± 19% +7.7 8.05 ± 7% perf-profile.self.cycles-pp.select_idle_sibling 1.44 ± 25% +8.8 10.23 ± 7% perf-profile.self.cycles-pp.available_idle_cpu 439.00 +11.7% 490.50 interrupts.100:IR-PCI-MSI.1572923-edge.eth0-TxRx-59 436.25 +13.1% 493.25 interrupts.101:IR-PCI-MSI.1572924-edge.eth0-TxRx-60 436.25 +12.4% 490.50 interrupts.102:IR-PCI-MSI.1572925-edge.eth0-TxRx-61 436.25 +12.4% 490.50 interrupts.103:IR-PCI-MSI.1572926-edge.eth0-TxRx-62 509.75 ± 3% +545.1% 3288 ±120% interrupts.42:IR-PCI-MSI.1572867-edge.eth0-TxRx-3 474.00 ± 3% +49.7% 709.75 ± 24% interrupts.46:IR-PCI-MSI.1572871-edge.eth0-TxRx-7 588.00 ± 35% +196.0% 1740 ± 50% interrupts.51:IR-PCI-MSI.1572876-edge.eth0-TxRx-12 436.75 ± 2% +12.8% 492.50 interrupts.55:IR-PCI-MSI.1572880-edge.eth0-TxRx-16 438.00 +39.5% 611.00 ± 33% interrupts.57:IR-PCI-MSI.1572882-edge.eth0-TxRx-18 436.25 +13.8% 496.25 interrupts.58:IR-PCI-MSI.1572883-edge.eth0-TxRx-19 436.25 +14.3% 498.50 interrupts.59:IR-PCI-MSI.1572884-edge.eth0-TxRx-20 437.00 +12.2% 490.50 interrupts.60:IR-PCI-MSI.1572885-edge.eth0-TxRx-21 436.25 +12.4% 490.50 interrupts.61:IR-PCI-MSI.1572886-edge.eth0-TxRx-22 436.25 +12.4% 490.50 interrupts.62:IR-PCI-MSI.1572887-edge.eth0-TxRx-23 441.00 +11.8% 493.00 interrupts.63:IR-PCI-MSI.1572888-edge.eth0-TxRx-24 437.25 +13.3% 495.25 interrupts.64:IR-PCI-MSI.1572889-edge.eth0-TxRx-25 436.25 +12.4% 490.50 interrupts.65:IR-PCI-MSI.1572890-edge.eth0-TxRx-26 436.25 +12.4% 490.50 interrupts.68:IR-PCI-MSI.1572893-edge.eth0-TxRx-29 436.25 +12.4% 490.50 interrupts.69:IR-PCI-MSI.1572894-edge.eth0-TxRx-30 436.25 +12.4% 490.50 interrupts.71:IR-PCI-MSI.1572896-edge.eth0-TxRx-32 437.25 ± 2% +12.2% 490.50 interrupts.74:IR-PCI-MSI.1572897-edge.eth0-TxRx-33 436.25 +12.6% 491.25 interrupts.75:IR-PCI-MSI.1572898-edge.eth0-TxRx-34 440.75 +12.4% 495.25 ± 2% interrupts.76:IR-PCI-MSI.1572899-edge.eth0-TxRx-35 436.25 +12.6% 491.00 interrupts.77:IR-PCI-MSI.1572900-edge.eth0-TxRx-36 436.75 +13.1% 493.75 interrupts.78:IR-PCI-MSI.1572901-edge.eth0-TxRx-37 436.25 +12.8% 492.25 interrupts.79:IR-PCI-MSI.1572902-edge.eth0-TxRx-38 436.25 +12.8% 492.25 interrupts.80:IR-PCI-MSI.1572903-edge.eth0-TxRx-39 436.25 +12.4% 490.50 interrupts.81:IR-PCI-MSI.1572904-edge.eth0-TxRx-40 436.25 +12.4% 490.50 interrupts.82:IR-PCI-MSI.1572905-edge.eth0-TxRx-41 436.25 +12.4% 490.50 interrupts.83:IR-PCI-MSI.1572906-edge.eth0-TxRx-42 436.25 +14.6% 499.75 ± 2% interrupts.84:IR-PCI-MSI.1572907-edge.eth0-TxRx-43 436.25 +12.4% 490.50 interrupts.85:IR-PCI-MSI.1572908-edge.eth0-TxRx-44 436.50 +12.4% 490.50 interrupts.86:IR-PCI-MSI.1572909-edge.eth0-TxRx-45 436.25 +12.4% 490.50 interrupts.87:IR-PCI-MSI.1572910-edge.eth0-TxRx-46 436.25 +12.4% 490.50 interrupts.88:IR-PCI-MSI.1572911-edge.eth0-TxRx-47 436.25 +12.6% 491.00 interrupts.89:IR-PCI-MSI.1572912-edge.eth0-TxRx-48 441.25 ± 3% +14.1% 503.25 ± 3% interrupts.90:IR-PCI-MSI.1572913-edge.eth0-TxRx-49 436.25 +12.4% 490.50 interrupts.91:IR-PCI-MSI.1572914-edge.eth0-TxRx-50 436.25 +14.4% 499.25 ± 3% interrupts.92:IR-PCI-MSI.1572915-edge.eth0-TxRx-51 439.25 ± 2% +11.7% 490.50 interrupts.94:IR-PCI-MSI.1572917-edge.eth0-TxRx-53 437.75 +12.1% 490.50 interrupts.95:IR-PCI-MSI.1572918-edge.eth0-TxRx-54 436.50 +12.4% 490.50 interrupts.96:IR-PCI-MSI.1572919-edge.eth0-TxRx-55 436.25 +12.4% 490.50 interrupts.97:IR-PCI-MSI.1572920-edge.eth0-TxRx-56 439.00 ± 2% +11.7% 490.50 interrupts.98:IR-PCI-MSI.1572921-edge.eth0-TxRx-57 442.25 ± 3% +10.9% 490.50 interrupts.99:IR-PCI-MSI.1572922-edge.eth0-TxRx-58 1989 +12.1% 2230 interrupts.9:IR-IO-APIC.9-fasteoi.acpi 6596036 ± 10% -21.1% 5206443 ± 6% interrupts.CAL:Function_call_interrupts 86097 ± 13% -18.4% 70229 ± 7% interrupts.CPU0.CAL:Function_call_interrupts 1785369 +12.4% 2006294 interrupts.CPU0.LOC:Local_timer_interrupts 102823 ± 12% -18.3% 84031 ± 5% interrupts.CPU0.TLB:TLB_shootdowns 388425 ± 12% +27.8% 496548 ± 4% interrupts.CPU0.TRM:Thermal_event_interrupts 1989 +12.1% 2230 interrupts.CPU1.9:IR-IO-APIC.9-fasteoi.acpi 90370 ± 8% -21.4% 70995 ± 6% interrupts.CPU1.CAL:Function_call_interrupts 1784834 +12.4% 2006046 interrupts.CPU1.LOC:Local_timer_interrupts 3088391 ± 15% -20.9% 2441882 ± 7% interrupts.CPU1.RES:Rescheduling_interrupts 105391 ± 8% -18.6% 85793 ± 6% interrupts.CPU1.TLB:TLB_shootdowns 388373 ± 12% +27.9% 496558 ± 4% interrupts.CPU1.TRM:Thermal_event_interrupts 88236 ± 11% -19.5% 71001 ± 7% interrupts.CPU10.CAL:Function_call_interrupts 1785471 +12.3% 2004740 interrupts.CPU10.LOC:Local_timer_interrupts 3040774 ± 4% -21.1% 2398396 ± 8% interrupts.CPU10.RES:Rescheduling_interrupts 104483 ± 10% -18.6% 85025 ± 5% interrupts.CPU10.TLB:TLB_shootdowns 388417 ± 12% +27.8% 496560 ± 4% interrupts.CPU10.TRM:Thermal_event_interrupts 1785039 +12.3% 2005274 interrupts.CPU11.LOC:Local_timer_interrupts 925.75 ±173% +461.3% 5196 ± 58% interrupts.CPU11.NMI:Non-maskable_interrupts 925.75 ±173% +461.3% 5196 ± 58% interrupts.CPU11.PMI:Performance_monitoring_interrupts 3094193 ± 14% -21.5% 2430228 ± 8% interrupts.CPU11.RES:Rescheduling_interrupts 102398 ± 10% -16.1% 85884 ± 5% interrupts.CPU11.TLB:TLB_shootdowns 388413 ± 12% +27.8% 496474 ± 4% interrupts.CPU11.TRM:Thermal_event_interrupts 588.00 ± 35% +196.0% 1740 ± 50% interrupts.CPU12.51:IR-PCI-MSI.1572876-edge.eth0-TxRx-12 87438 ± 9% -18.1% 71580 ± 7% interrupts.CPU12.CAL:Function_call_interrupts 1785376 +12.3% 2005068 interrupts.CPU12.LOC:Local_timer_interrupts 103057 ± 10% -16.4% 86186 ± 6% interrupts.CPU12.TLB:TLB_shootdowns 388416 ± 12% +27.8% 496490 ± 4% interrupts.CPU12.TRM:Thermal_event_interrupts 88735 ± 9% -19.4% 71496 ± 9% interrupts.CPU13.CAL:Function_call_interrupts 1785460 +12.3% 2005445 interrupts.CPU13.LOC:Local_timer_interrupts 104230 ± 9% -17.5% 86038 ± 6% interrupts.CPU13.TLB:TLB_shootdowns 388422 ± 12% +27.8% 496563 ± 4% interrupts.CPU13.TRM:Thermal_event_interrupts 88327 ± 12% -19.0% 71507 ± 7% interrupts.CPU14.CAL:Function_call_interrupts 1785094 +12.3% 2005441 interrupts.CPU14.LOC:Local_timer_interrupts 103317 ± 11% -18.1% 84581 ± 6% interrupts.CPU14.TLB:TLB_shootdowns 388412 ± 12% +27.8% 496564 ± 4% interrupts.CPU14.TRM:Thermal_event_interrupts 86972 ± 10% -17.3% 71898 ± 7% interrupts.CPU15.CAL:Function_call_interrupts 1784668 +12.4% 2005375 interrupts.CPU15.LOC:Local_timer_interrupts 3039799 ± 9% -21.1% 2397505 ± 9% interrupts.CPU15.RES:Rescheduling_interrupts 103367 ± 12% -16.9% 85892 ± 6% interrupts.CPU15.TLB:TLB_shootdowns 388342 ± 12% +27.9% 496559 ± 4% interrupts.CPU15.TRM:Thermal_event_interrupts 436.75 ± 2% +12.8% 492.50 interrupts.CPU16.55:IR-PCI-MSI.1572880-edge.eth0-TxRx-16 1784750 +12.4% 2005330 interrupts.CPU16.LOC:Local_timer_interrupts 55.25 ±171% +7696.8% 4307 ± 64% interrupts.CPU16.NMI:Non-maskable_interrupts 55.25 ±171% +7696.8% 4307 ± 64% interrupts.CPU16.PMI:Performance_monitoring_interrupts 2969996 ± 10% -18.7% 2413141 ± 9% interrupts.CPU16.RES:Rescheduling_interrupts 102367 ± 11% -13.7% 88305 ± 3% interrupts.CPU16.TLB:TLB_shootdowns 388423 ± 12% +27.8% 496563 ± 4% interrupts.CPU16.TRM:Thermal_event_interrupts 1785363 +12.3% 2005849 interrupts.CPU17.LOC:Local_timer_interrupts 388418 ± 12% +27.8% 496560 ± 4% interrupts.CPU17.TRM:Thermal_event_interrupts 438.00 +39.5% 611.00 ± 33% interrupts.CPU18.57:IR-PCI-MSI.1572882-edge.eth0-TxRx-18 88992 ± 12% -22.3% 69145 ± 6% interrupts.CPU18.CAL:Function_call_interrupts 1785398 +12.1% 2001316 interrupts.CPU18.LOC:Local_timer_interrupts 2199105 ± 7% +115.2% 4731679 ± 7% interrupts.CPU18.RES:Rescheduling_interrupts 108465 ± 8% -23.0% 83467 ± 5% interrupts.CPU18.TLB:TLB_shootdowns 329294 ± 3% +15.4% 380131 ± 2% interrupts.CPU18.TRM:Thermal_event_interrupts 436.25 +13.8% 496.25 interrupts.CPU19.58:IR-PCI-MSI.1572883-edge.eth0-TxRx-19 92059 ± 6% -24.2% 69776 ± 5% interrupts.CPU19.CAL:Function_call_interrupts 1784665 +12.2% 2001579 interrupts.CPU19.LOC:Local_timer_interrupts 1284622 ± 7% +225.8% 4185208 ± 4% interrupts.CPU19.RES:Rescheduling_interrupts 108492 ± 7% -23.4% 83059 ± 4% interrupts.CPU19.TLB:TLB_shootdowns 329166 ± 3% +15.5% 380141 ± 2% interrupts.CPU19.TRM:Thermal_event_interrupts 1784512 +12.4% 2005572 interrupts.CPU2.LOC:Local_timer_interrupts 3091132 ± 8% -22.3% 2401342 ± 4% interrupts.CPU2.RES:Rescheduling_interrupts 102562 ± 9% -16.5% 85643 ± 5% interrupts.CPU2.TLB:TLB_shootdowns 388380 ± 12% +27.8% 496544 ± 4% interrupts.CPU2.TRM:Thermal_event_interrupts 436.25 +14.3% 498.50 interrupts.CPU20.59:IR-PCI-MSI.1572884-edge.eth0-TxRx-20 91785 ± 9% -23.9% 69853 ± 4% interrupts.CPU20.CAL:Function_call_interrupts 1785284 +12.1% 2001952 interrupts.CPU20.LOC:Local_timer_interrupts 1281621 ± 12% +203.6% 3890516 ± 7% interrupts.CPU20.RES:Rescheduling_interrupts 106297 ± 9% -19.9% 85180 ± 3% interrupts.CPU20.TLB:TLB_shootdowns 329293 ± 3% +15.4% 380131 ± 2% interrupts.CPU20.TRM:Thermal_event_interrupts 437.00 +12.2% 490.50 interrupts.CPU21.60:IR-PCI-MSI.1572885-edge.eth0-TxRx-21 92635 ± 7% -23.9% 70512 ± 6% interrupts.CPU21.CAL:Function_call_interrupts 1785480 +12.1% 2001723 interrupts.CPU21.LOC:Local_timer_interrupts 1595447 ± 6% +187.8% 4591004 interrupts.CPU21.RES:Rescheduling_interrupts 107575 ± 7% -21.6% 84296 ± 5% interrupts.CPU21.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380086 ± 2% interrupts.CPU21.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU22.61:IR-PCI-MSI.1572886-edge.eth0-TxRx-22 92674 ± 10% -24.4% 70047 ± 6% interrupts.CPU22.CAL:Function_call_interrupts 1784881 +12.1% 2001482 interrupts.CPU22.LOC:Local_timer_interrupts 1605143 ± 6% +179.5% 4486945 ± 5% interrupts.CPU22.RES:Rescheduling_interrupts 108749 ± 10% -23.3% 83409 ± 5% interrupts.CPU22.TLB:TLB_shootdowns 329295 ± 3% +15.4% 380136 ± 2% interrupts.CPU22.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU23.62:IR-PCI-MSI.1572887-edge.eth0-TxRx-23 90975 ± 11% -23.6% 69492 ± 5% interrupts.CPU23.CAL:Function_call_interrupts 1785236 +12.1% 2001811 interrupts.CPU23.LOC:Local_timer_interrupts 1600988 ± 2% +180.2% 4486667 ± 5% interrupts.CPU23.RES:Rescheduling_interrupts 106819 ± 10% -22.3% 83007 ± 4% interrupts.CPU23.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380146 ± 2% interrupts.CPU23.TRM:Thermal_event_interrupts 441.00 +11.8% 493.00 interrupts.CPU24.63:IR-PCI-MSI.1572888-edge.eth0-TxRx-24 91919 ± 11% -23.1% 70689 ± 5% interrupts.CPU24.CAL:Function_call_interrupts 1784996 +12.1% 2000922 interrupts.CPU24.LOC:Local_timer_interrupts 1612237 ± 5% +188.4% 4649702 ± 2% interrupts.CPU24.RES:Rescheduling_interrupts 108187 ± 8% -22.5% 83831 ± 3% interrupts.CPU24.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380122 ± 2% interrupts.CPU24.TRM:Thermal_event_interrupts 437.25 +13.3% 495.25 interrupts.CPU25.64:IR-PCI-MSI.1572889-edge.eth0-TxRx-25 88417 ± 12% -20.8% 70029 ± 4% interrupts.CPU25.CAL:Function_call_interrupts 1785404 +12.0% 1999622 interrupts.CPU25.LOC:Local_timer_interrupts 1612074 ± 2% +182.4% 4551699 interrupts.CPU25.RES:Rescheduling_interrupts 105324 ± 10% -20.1% 84201 ± 4% interrupts.CPU25.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380141 ± 2% interrupts.CPU25.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU26.65:IR-PCI-MSI.1572890-edge.eth0-TxRx-26 92632 ± 12% -23.5% 70907 ± 5% interrupts.CPU26.CAL:Function_call_interrupts 1784730 +12.1% 2000264 interrupts.CPU26.LOC:Local_timer_interrupts 1639148 ± 6% +188.0% 4720149 ± 4% interrupts.CPU26.RES:Rescheduling_interrupts 108099 ± 11% -21.6% 84727 ± 4% interrupts.CPU26.TLB:TLB_shootdowns 329271 ± 3% +15.4% 379984 ± 2% interrupts.CPU26.TRM:Thermal_event_interrupts 91312 ± 12% -23.4% 69963 ± 6% interrupts.CPU27.CAL:Function_call_interrupts 1785171 +12.1% 2001743 interrupts.CPU27.LOC:Local_timer_interrupts 1704847 ± 11% +179.4% 4764019 ± 4% interrupts.CPU27.RES:Rescheduling_interrupts 106515 ± 11% -21.6% 83491 ± 5% interrupts.CPU27.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380080 ± 2% interrupts.CPU27.TRM:Thermal_event_interrupts 89178 ± 8% -22.0% 69565 ± 6% interrupts.CPU28.CAL:Function_call_interrupts 1785194 +12.1% 2001836 interrupts.CPU28.LOC:Local_timer_interrupts 1593025 ± 8% +185.7% 4551782 ± 5% interrupts.CPU28.RES:Rescheduling_interrupts 104408 ± 10% -19.2% 84365 ± 6% interrupts.CPU28.TLB:TLB_shootdowns 329294 ± 3% +15.4% 380142 ± 2% interrupts.CPU28.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU29.68:IR-PCI-MSI.1572893-edge.eth0-TxRx-29 90850 ± 12% -21.9% 70993 ± 6% interrupts.CPU29.CAL:Function_call_interrupts 1785398 +12.1% 2000545 interrupts.CPU29.LOC:Local_timer_interrupts 1657508 ± 5% +179.4% 4631109 ± 3% interrupts.CPU29.RES:Rescheduling_interrupts 106383 ± 13% -20.8% 84284 ± 4% interrupts.CPU29.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380133 ± 2% interrupts.CPU29.TRM:Thermal_event_interrupts 509.75 ± 3% +545.1% 3288 ±120% interrupts.CPU3.42:IR-PCI-MSI.1572867-edge.eth0-TxRx-3 87008 ± 11% -19.9% 69730 ± 5% interrupts.CPU3.CAL:Function_call_interrupts 1784446 +12.4% 2005368 interrupts.CPU3.LOC:Local_timer_interrupts 3169674 ± 9% -20.4% 2521963 ± 8% interrupts.CPU3.RES:Rescheduling_interrupts 103703 ± 12% -17.1% 85958 ± 7% interrupts.CPU3.TLB:TLB_shootdowns 388356 ± 12% +27.9% 496551 ± 4% interrupts.CPU3.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU30.69:IR-PCI-MSI.1572894-edge.eth0-TxRx-30 92043 ± 12% -24.2% 69765 ± 5% interrupts.CPU30.CAL:Function_call_interrupts 1785175 +12.1% 2001248 interrupts.CPU30.LOC:Local_timer_interrupts 1637465 ± 6% +191.3% 4769701 ± 3% interrupts.CPU30.RES:Rescheduling_interrupts 106836 ± 11% -22.5% 82810 ± 5% interrupts.CPU30.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380130 ± 2% interrupts.CPU30.TRM:Thermal_event_interrupts 1785301 +12.0% 1999270 interrupts.CPU31.LOC:Local_timer_interrupts 62.75 ±173% +4261.4% 2736 ±113% interrupts.CPU31.NMI:Non-maskable_interrupts 62.75 ±173% +4261.4% 2736 ±113% interrupts.CPU31.PMI:Performance_monitoring_interrupts 1589500 ± 5% +184.1% 4515325 interrupts.CPU31.RES:Rescheduling_interrupts 106576 ± 11% -21.5% 83698 ± 5% interrupts.CPU31.TLB:TLB_shootdowns 329293 ± 3% +15.4% 380131 ± 2% interrupts.CPU31.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU32.71:IR-PCI-MSI.1572896-edge.eth0-TxRx-32 93348 ± 12% -24.4% 70595 ± 8% interrupts.CPU32.CAL:Function_call_interrupts 1785245 +12.1% 2001145 interrupts.CPU32.LOC:Local_timer_interrupts 1626883 ± 6% +176.3% 4494456 ± 5% interrupts.CPU32.RES:Rescheduling_interrupts 107151 ± 12% -21.4% 84236 ± 5% interrupts.CPU32.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380131 ± 2% interrupts.CPU32.TRM:Thermal_event_interrupts 437.25 ± 2% +12.2% 490.50 interrupts.CPU33.74:IR-PCI-MSI.1572897-edge.eth0-TxRx-33 91487 ± 12% -22.7% 70758 ± 3% interrupts.CPU33.CAL:Function_call_interrupts 1785371 +12.0% 1999987 interrupts.CPU33.LOC:Local_timer_interrupts 1578761 ± 6% +201.2% 4754762 ± 6% interrupts.CPU33.RES:Rescheduling_interrupts 106071 ± 12% -21.3% 83430 ± 2% interrupts.CPU33.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380130 ± 2% interrupts.CPU33.TRM:Thermal_event_interrupts 436.25 +12.6% 491.25 interrupts.CPU34.75:IR-PCI-MSI.1572898-edge.eth0-TxRx-34 91116 ± 11% -24.3% 68943 ± 7% interrupts.CPU34.CAL:Function_call_interrupts 1785353 +12.1% 2001525 interrupts.CPU34.LOC:Local_timer_interrupts 1553828 ± 4% +186.0% 4444035 ± 3% interrupts.CPU34.RES:Rescheduling_interrupts 105575 ± 10% -20.7% 83762 ± 4% interrupts.CPU34.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380139 ± 2% interrupts.CPU34.TRM:Thermal_event_interrupts 440.75 +12.4% 495.25 ± 2% interrupts.CPU35.76:IR-PCI-MSI.1572899-edge.eth0-TxRx-35 91396 ± 11% -25.6% 67986 ± 11% interrupts.CPU35.CAL:Function_call_interrupts 1785237 +12.0% 2000288 interrupts.CPU35.LOC:Local_timer_interrupts 1602616 ± 10% +189.7% 4642115 ± 3% interrupts.CPU35.RES:Rescheduling_interrupts 107669 ± 12% -21.3% 84736 ± 4% interrupts.CPU35.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380145 ± 2% interrupts.CPU35.TRM:Thermal_event_interrupts 436.25 +12.6% 491.00 interrupts.CPU36.77:IR-PCI-MSI.1572900-edge.eth0-TxRx-36 93553 ± 8% -19.8% 75044 ± 6% interrupts.CPU36.CAL:Function_call_interrupts 1784636 +12.4% 2005389 interrupts.CPU36.LOC:Local_timer_interrupts 2588984 ± 6% +25.8% 3256348 ± 6% interrupts.CPU36.RES:Rescheduling_interrupts 107053 ± 7% -19.0% 86754 ± 5% interrupts.CPU36.TLB:TLB_shootdowns 388423 ± 12% +27.8% 496560 ± 4% interrupts.CPU36.TRM:Thermal_event_interrupts 436.75 +13.1% 493.75 interrupts.CPU37.78:IR-PCI-MSI.1572901-edge.eth0-TxRx-37 89518 ± 10% -18.9% 72619 ± 5% interrupts.CPU37.CAL:Function_call_interrupts 1784648 +12.4% 2005651 interrupts.CPU37.LOC:Local_timer_interrupts 3112476 ± 10% -20.8% 2464103 ± 5% interrupts.CPU37.RES:Rescheduling_interrupts 102576 ± 10% -14.4% 87794 ± 6% interrupts.CPU37.TLB:TLB_shootdowns 388412 ± 12% +27.8% 496563 ± 4% interrupts.CPU37.TRM:Thermal_event_interrupts 436.25 +12.8% 492.25 interrupts.CPU38.79:IR-PCI-MSI.1572902-edge.eth0-TxRx-38 92566 ± 10% -20.6% 73466 ± 6% interrupts.CPU38.CAL:Function_call_interrupts 1785132 +12.4% 2005883 interrupts.CPU38.LOC:Local_timer_interrupts 3163520 ± 10% -26.0% 2340820 ± 11% interrupts.CPU38.RES:Rescheduling_interrupts 105823 ± 10% -18.0% 86785 ± 4% interrupts.CPU38.TLB:TLB_shootdowns 388401 ± 12% +27.8% 496554 ± 4% interrupts.CPU38.TRM:Thermal_event_interrupts 436.25 +12.8% 492.25 interrupts.CPU39.80:IR-PCI-MSI.1572903-edge.eth0-TxRx-39 93197 ± 11% -21.8% 72854 ± 7% interrupts.CPU39.CAL:Function_call_interrupts 1784624 +12.4% 2005934 interrupts.CPU39.LOC:Local_timer_interrupts 105953 ± 11% -20.8% 83925 ± 6% interrupts.CPU39.TLB:TLB_shootdowns 388418 ± 12% +27.8% 496548 ± 4% interrupts.CPU39.TRM:Thermal_event_interrupts 86883 ± 13% -17.8% 71449 ± 7% interrupts.CPU4.CAL:Function_call_interrupts 1784433 +12.4% 2005462 interrupts.CPU4.LOC:Local_timer_interrupts 3119739 ± 9% -21.1% 2461192 ± 4% interrupts.CPU4.RES:Rescheduling_interrupts 104829 ± 11% -19.1% 84802 ± 5% interrupts.CPU4.TLB:TLB_shootdowns 388417 ± 12% +27.8% 496563 ± 4% interrupts.CPU4.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU40.81:IR-PCI-MSI.1572904-edge.eth0-TxRx-40 91881 ± 12% -21.4% 72188 ± 8% interrupts.CPU40.CAL:Function_call_interrupts 1784831 +12.4% 2005991 interrupts.CPU40.LOC:Local_timer_interrupts 3101288 ± 10% -17.2% 2569221 ± 7% interrupts.CPU40.RES:Rescheduling_interrupts 105884 ± 10% -19.0% 85772 ± 5% interrupts.CPU40.TLB:TLB_shootdowns 388415 ± 12% +27.8% 496564 ± 4% interrupts.CPU40.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU41.82:IR-PCI-MSI.1572905-edge.eth0-TxRx-41 91739 ± 12% -19.9% 73475 ± 7% interrupts.CPU41.CAL:Function_call_interrupts 1785231 +12.3% 2005398 interrupts.CPU41.LOC:Local_timer_interrupts 3223405 ± 11% -27.9% 2325498 ± 5% interrupts.CPU41.RES:Rescheduling_interrupts 104410 ± 11% -18.7% 84846 ± 6% interrupts.CPU41.TLB:TLB_shootdowns 388408 ± 12% +27.8% 496557 ± 4% interrupts.CPU41.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU42.83:IR-PCI-MSI.1572906-edge.eth0-TxRx-42 1785354 +12.4% 2006034 interrupts.CPU42.LOC:Local_timer_interrupts 3068304 ± 9% -22.1% 2390347 ± 6% interrupts.CPU42.RES:Rescheduling_interrupts 107102 ± 9% -20.7% 84890 ± 4% interrupts.CPU42.TLB:TLB_shootdowns 388423 ± 12% +27.8% 496564 ± 4% interrupts.CPU42.TRM:Thermal_event_interrupts 436.25 +14.6% 499.75 ± 2% interrupts.CPU43.84:IR-PCI-MSI.1572907-edge.eth0-TxRx-43 92178 ± 9% -18.3% 75306 ± 7% interrupts.CPU43.CAL:Function_call_interrupts 1784759 +12.4% 2005986 interrupts.CPU43.LOC:Local_timer_interrupts 6434 ± 7% -64.3% 2297 ±106% interrupts.CPU43.NMI:Non-maskable_interrupts 6434 ± 7% -64.3% 2297 ±106% interrupts.CPU43.PMI:Performance_monitoring_interrupts 104773 ± 9% -17.6% 86369 ± 6% interrupts.CPU43.TLB:TLB_shootdowns 388425 ± 12% +27.8% 496556 ± 4% interrupts.CPU43.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU44.85:IR-PCI-MSI.1572908-edge.eth0-TxRx-44 92387 ± 10% -21.9% 72174 ± 9% interrupts.CPU44.CAL:Function_call_interrupts 1784603 +12.4% 2006064 interrupts.CPU44.LOC:Local_timer_interrupts 106179 ± 9% -18.6% 86461 ± 6% interrupts.CPU44.TLB:TLB_shootdowns 388426 ± 12% +27.8% 496562 ± 4% interrupts.CPU44.TRM:Thermal_event_interrupts 436.50 +12.4% 490.50 interrupts.CPU45.86:IR-PCI-MSI.1572909-edge.eth0-TxRx-45 93641 ± 7% -21.9% 73155 ± 6% interrupts.CPU45.CAL:Function_call_interrupts 1784811 +12.4% 2005654 interrupts.CPU45.LOC:Local_timer_interrupts 3093195 ± 11% -15.6% 2609354 ± 4% interrupts.CPU45.RES:Rescheduling_interrupts 106463 ± 7% -19.6% 85593 ± 5% interrupts.CPU45.TLB:TLB_shootdowns 388408 ± 12% +27.8% 496564 ± 4% interrupts.CPU45.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU46.87:IR-PCI-MSI.1572910-edge.eth0-TxRx-46 91739 ± 10% -19.0% 74279 ± 8% interrupts.CPU46.CAL:Function_call_interrupts 1784609 +12.4% 2005522 interrupts.CPU46.LOC:Local_timer_interrupts 2970471 ± 9% -21.0% 2345439 ± 8% interrupts.CPU46.RES:Rescheduling_interrupts 108158 ± 10% -19.8% 86743 ± 4% interrupts.CPU46.TLB:TLB_shootdowns 388368 ± 12% +27.8% 496469 ± 4% interrupts.CPU46.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU47.88:IR-PCI-MSI.1572911-edge.eth0-TxRx-47 92856 ± 9% -20.0% 74327 ± 8% interrupts.CPU47.CAL:Function_call_interrupts 1784835 +12.4% 2005700 interrupts.CPU47.LOC:Local_timer_interrupts 3116242 ± 10% -20.4% 2480398 ± 5% interrupts.CPU47.RES:Rescheduling_interrupts 104927 ± 9% -18.6% 85393 ± 6% interrupts.CPU47.TLB:TLB_shootdowns 388403 ± 12% +27.8% 496554 ± 4% interrupts.CPU47.TRM:Thermal_event_interrupts 436.25 +12.6% 491.00 interrupts.CPU48.89:IR-PCI-MSI.1572912-edge.eth0-TxRx-48 92889 ± 11% -21.5% 72890 ± 8% interrupts.CPU48.CAL:Function_call_interrupts 1784772 +12.4% 2006205 interrupts.CPU48.LOC:Local_timer_interrupts 3175210 ± 12% -19.7% 2548339 ± 13% interrupts.CPU48.RES:Rescheduling_interrupts 105183 ± 11% -18.6% 85618 ± 6% interrupts.CPU48.TLB:TLB_shootdowns 388426 ± 12% +27.8% 496563 ± 4% interrupts.CPU48.TRM:Thermal_event_interrupts 441.25 ± 3% +14.1% 503.25 ± 3% interrupts.CPU49.90:IR-PCI-MSI.1572913-edge.eth0-TxRx-49 90134 ± 10% -18.3% 73603 ± 6% interrupts.CPU49.CAL:Function_call_interrupts 1785145 +12.4% 2005932 interrupts.CPU49.LOC:Local_timer_interrupts 3185609 ± 11% -17.6% 2626502 ± 4% interrupts.CPU49.RES:Rescheduling_interrupts 102737 ± 10% -17.3% 84932 ± 5% interrupts.CPU49.TLB:TLB_shootdowns 388425 ± 12% +27.8% 496560 ± 4% interrupts.CPU49.TRM:Thermal_event_interrupts 92114 ± 9% -23.2% 70721 ± 6% interrupts.CPU5.CAL:Function_call_interrupts 1785135 +12.3% 2005424 interrupts.CPU5.LOC:Local_timer_interrupts 3210646 ± 12% -24.8% 2415935 ± 4% interrupts.CPU5.RES:Rescheduling_interrupts 107349 ± 9% -18.9% 87034 ± 6% interrupts.CPU5.TLB:TLB_shootdowns 388418 ± 12% +27.8% 496564 ± 4% interrupts.CPU5.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU50.91:IR-PCI-MSI.1572914-edge.eth0-TxRx-50 92406 ± 10% -20.2% 73771 ± 8% interrupts.CPU50.CAL:Function_call_interrupts 1785200 +12.4% 2006488 interrupts.CPU50.LOC:Local_timer_interrupts 106635 ± 9% -19.9% 85431 ± 6% interrupts.CPU50.TLB:TLB_shootdowns 388419 ± 12% +27.8% 496564 ± 4% interrupts.CPU50.TRM:Thermal_event_interrupts 436.25 +14.4% 499.25 ± 3% interrupts.CPU51.92:IR-PCI-MSI.1572915-edge.eth0-TxRx-51 91491 ± 9% -20.8% 72473 ± 6% interrupts.CPU51.CAL:Function_call_interrupts 1784683 +12.4% 2005527 interrupts.CPU51.LOC:Local_timer_interrupts 3051494 ± 8% -24.6% 2300032 ± 10% interrupts.CPU51.RES:Rescheduling_interrupts 104954 ± 10% -20.2% 83775 ± 5% interrupts.CPU51.TLB:TLB_shootdowns 388264 ± 12% +27.9% 496462 ± 4% interrupts.CPU51.TRM:Thermal_event_interrupts 92938 ± 11% -19.7% 74594 ± 7% interrupts.CPU52.CAL:Function_call_interrupts 1785451 +12.4% 2006171 interrupts.CPU52.LOC:Local_timer_interrupts 3025681 ± 11% -17.1% 2506800 ± 10% interrupts.CPU52.RES:Rescheduling_interrupts 107100 ± 11% -19.6% 86156 ± 6% interrupts.CPU52.TLB:TLB_shootdowns 388422 ± 12% +27.8% 496563 ± 4% interrupts.CPU52.TRM:Thermal_event_interrupts 439.25 ± 2% +11.7% 490.50 interrupts.CPU53.94:IR-PCI-MSI.1572917-edge.eth0-TxRx-53 93000 ± 12% -20.9% 73549 ± 7% interrupts.CPU53.CAL:Function_call_interrupts 1785290 +12.3% 2005441 interrupts.CPU53.LOC:Local_timer_interrupts 3212623 ± 7% -18.2% 2629478 ± 10% interrupts.CPU53.RES:Rescheduling_interrupts 107303 ± 11% -17.5% 88567 ± 10% interrupts.CPU53.TLB:TLB_shootdowns 388411 ± 12% +27.8% 496557 ± 4% interrupts.CPU53.TRM:Thermal_event_interrupts 437.75 +12.1% 490.50 interrupts.CPU54.95:IR-PCI-MSI.1572918-edge.eth0-TxRx-54 1785170 +12.0% 2000090 interrupts.CPU54.LOC:Local_timer_interrupts 5351 ± 26% -64.5% 1900 ± 99% interrupts.CPU54.NMI:Non-maskable_interrupts 5351 ± 26% -64.5% 1900 ± 99% interrupts.CPU54.PMI:Performance_monitoring_interrupts 2066106 ± 9% +92.7% 3982033 ± 6% interrupts.CPU54.RES:Rescheduling_interrupts 108150 ± 12% -21.2% 85173 ± 5% interrupts.CPU54.TLB:TLB_shootdowns 329294 ± 3% +15.4% 380146 ± 2% interrupts.CPU54.TRM:Thermal_event_interrupts 436.50 +12.4% 490.50 interrupts.CPU55.96:IR-PCI-MSI.1572919-edge.eth0-TxRx-55 91164 ± 12% -19.9% 73014 ± 6% interrupts.CPU55.CAL:Function_call_interrupts 1785472 +12.0% 1999115 interrupts.CPU55.LOC:Local_timer_interrupts 1286654 ± 6% +228.2% 4223034 ± 5% interrupts.CPU55.RES:Rescheduling_interrupts 104943 ± 11% -19.5% 84466 ± 5% interrupts.CPU55.TLB:TLB_shootdowns 329292 ± 3% +15.4% 379918 ± 2% interrupts.CPU55.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU56.97:IR-PCI-MSI.1572920-edge.eth0-TxRx-56 96934 ± 13% -23.4% 74215 ± 6% interrupts.CPU56.CAL:Function_call_interrupts 1785010 +12.1% 2000169 interrupts.CPU56.LOC:Local_timer_interrupts 1312758 ± 17% +194.5% 3866522 ± 3% interrupts.CPU56.RES:Rescheduling_interrupts 109307 ± 12% -22.1% 85139 ± 4% interrupts.CPU56.TLB:TLB_shootdowns 329273 ± 3% +15.4% 380131 ± 2% interrupts.CPU56.TRM:Thermal_event_interrupts 439.00 ± 2% +11.7% 490.50 interrupts.CPU57.98:IR-PCI-MSI.1572921-edge.eth0-TxRx-57 94960 ± 11% -22.9% 73246 ± 5% interrupts.CPU57.CAL:Function_call_interrupts 1785297 +12.0% 2000019 interrupts.CPU57.LOC:Local_timer_interrupts 1666475 ± 10% +169.6% 4492937 interrupts.CPU57.RES:Rescheduling_interrupts 107534 ± 10% -22.0% 83871 ± 5% interrupts.CPU57.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380136 ± 2% interrupts.CPU57.TRM:Thermal_event_interrupts 442.25 ± 3% +10.9% 490.50 interrupts.CPU58.99:IR-PCI-MSI.1572922-edge.eth0-TxRx-58 92818 ± 12% -21.4% 72925 ± 5% interrupts.CPU58.CAL:Function_call_interrupts 1785429 +12.0% 1999528 interrupts.CPU58.LOC:Local_timer_interrupts 1606064 ± 5% +176.8% 4446179 ± 6% interrupts.CPU58.RES:Rescheduling_interrupts 103696 ± 11% -19.5% 83458 ± 4% interrupts.CPU58.TLB:TLB_shootdowns 329293 ± 3% +15.4% 380022 ± 2% interrupts.CPU58.TRM:Thermal_event_interrupts 439.00 +11.7% 490.50 interrupts.CPU59.100:IR-PCI-MSI.1572923-edge.eth0-TxRx-59 93876 ± 13% -20.5% 74669 ± 6% interrupts.CPU59.CAL:Function_call_interrupts 1785085 +12.0% 1999992 interrupts.CPU59.LOC:Local_timer_interrupts 1634812 ± 5% +174.5% 4487501 ± 4% interrupts.CPU59.RES:Rescheduling_interrupts 106929 ± 13% -21.0% 84522 ± 5% interrupts.CPU59.TLB:TLB_shootdowns 329291 ± 3% +15.4% 380111 ± 2% interrupts.CPU59.TRM:Thermal_event_interrupts 89930 ± 9% -20.5% 71520 ± 7% interrupts.CPU6.CAL:Function_call_interrupts 1785220 +12.3% 2005468 interrupts.CPU6.LOC:Local_timer_interrupts 106307 ± 7% -19.9% 85162 ± 6% interrupts.CPU6.TLB:TLB_shootdowns 388421 ± 12% +27.8% 496564 ± 4% interrupts.CPU6.TRM:Thermal_event_interrupts 436.25 +13.1% 493.25 interrupts.CPU60.101:IR-PCI-MSI.1572924-edge.eth0-TxRx-60 98595 ± 11% -25.6% 73379 ± 5% interrupts.CPU60.CAL:Function_call_interrupts 1785116 +12.1% 2000324 interrupts.CPU60.LOC:Local_timer_interrupts 1604314 ± 8% +186.8% 4601614 ± 5% interrupts.CPU60.RES:Rescheduling_interrupts 110442 ± 10% -24.0% 83932 ± 5% interrupts.CPU60.TLB:TLB_shootdowns 329256 ± 3% +15.5% 380140 ± 2% interrupts.CPU60.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU61.102:IR-PCI-MSI.1572925-edge.eth0-TxRx-61 93035 ± 11% -21.4% 73124 ± 5% interrupts.CPU61.CAL:Function_call_interrupts 1785176 +12.1% 2000960 interrupts.CPU61.LOC:Local_timer_interrupts 4567 ± 67% -79.2% 952.00 ±171% interrupts.CPU61.NMI:Non-maskable_interrupts 4567 ± 67% -79.2% 952.00 ±171% interrupts.CPU61.PMI:Performance_monitoring_interrupts 1575186 ± 6% +194.9% 4644760 ± 4% interrupts.CPU61.RES:Rescheduling_interrupts 103665 ± 11% -18.7% 84259 ± 5% interrupts.CPU61.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380127 ± 2% interrupts.CPU61.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU62.103:IR-PCI-MSI.1572926-edge.eth0-TxRx-62 95188 ± 10% -21.9% 74376 ± 6% interrupts.CPU62.CAL:Function_call_interrupts 1785028 +12.0% 1999262 interrupts.CPU62.LOC:Local_timer_interrupts 1613834 ± 8% +184.7% 4594499 ± 3% interrupts.CPU62.RES:Rescheduling_interrupts 106856 ± 10% -21.4% 83987 ± 5% interrupts.CPU62.TLB:TLB_shootdowns 329270 ± 3% +15.4% 380120 ± 2% interrupts.CPU62.TRM:Thermal_event_interrupts 94411 ± 11% -21.3% 74269 ± 5% interrupts.CPU63.CAL:Function_call_interrupts 1785322 +12.0% 1999237 interrupts.CPU63.LOC:Local_timer_interrupts 1669218 ± 6% +187.2% 4794684 ± 3% interrupts.CPU63.RES:Rescheduling_interrupts 105402 ± 10% -18.3% 86076 ± 8% interrupts.CPU63.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380141 ± 2% interrupts.CPU63.TRM:Thermal_event_interrupts 96060 ± 11% -22.6% 74364 ± 6% interrupts.CPU64.CAL:Function_call_interrupts 1785304 +12.0% 1999571 interrupts.CPU64.LOC:Local_timer_interrupts 1681501 ± 7% +164.0% 4438647 ± 3% interrupts.CPU64.RES:Rescheduling_interrupts 108534 ± 10% -22.8% 83813 ± 5% interrupts.CPU64.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380142 ± 2% interrupts.CPU64.TRM:Thermal_event_interrupts 94178 ± 9% -22.7% 72758 ± 8% interrupts.CPU65.CAL:Function_call_interrupts 1785338 +12.0% 1999408 interrupts.CPU65.LOC:Local_timer_interrupts 1674108 ± 4% +182.2% 4724262 ± 3% interrupts.CPU65.RES:Rescheduling_interrupts 107519 ± 10% -22.6% 83259 ± 5% interrupts.CPU65.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380128 ± 2% interrupts.CPU65.TRM:Thermal_event_interrupts 95135 ± 8% -22.4% 73795 ± 5% interrupts.CPU66.CAL:Function_call_interrupts 1785413 +12.0% 2000352 interrupts.CPU66.LOC:Local_timer_interrupts 616.75 ±171% +902.4% 6182 ± 28% interrupts.CPU66.NMI:Non-maskable_interrupts 616.75 ±171% +902.4% 6182 ± 28% interrupts.CPU66.PMI:Performance_monitoring_interrupts 1646166 ± 6% +169.3% 4433267 ± 6% interrupts.CPU66.RES:Rescheduling_interrupts 107190 ± 7% -22.5% 83034 ± 4% interrupts.CPU66.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380118 ± 2% interrupts.CPU66.TRM:Thermal_event_interrupts 95627 ± 10% -22.4% 74249 ± 5% interrupts.CPU67.CAL:Function_call_interrupts 1785314 +12.1% 2001128 interrupts.CPU67.LOC:Local_timer_interrupts 1611146 ± 4% +183.8% 4572204 ± 7% interrupts.CPU67.RES:Rescheduling_interrupts 109283 ± 11% -22.5% 84676 ± 4% interrupts.CPU67.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380142 ± 2% interrupts.CPU67.TRM:Thermal_event_interrupts 95479 ± 11% -20.6% 75791 ± 5% interrupts.CPU68.CAL:Function_call_interrupts 1785192 +12.0% 1999411 interrupts.CPU68.LOC:Local_timer_interrupts 1575394 ± 7% +189.9% 4567277 ± 4% interrupts.CPU68.RES:Rescheduling_interrupts 106972 ± 12% -18.6% 87039 ± 5% interrupts.CPU68.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380129 ± 2% interrupts.CPU68.TRM:Thermal_event_interrupts 93067 ± 14% -19.9% 74538 ± 4% interrupts.CPU69.CAL:Function_call_interrupts 1785257 +12.1% 2000636 interrupts.CPU69.LOC:Local_timer_interrupts 1578663 ± 9% +192.9% 4624518 ± 3% interrupts.CPU69.RES:Rescheduling_interrupts 104271 ± 12% -18.5% 84989 ± 2% interrupts.CPU69.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380066 ± 2% interrupts.CPU69.TRM:Thermal_event_interrupts 474.00 ± 3% +49.7% 709.75 ± 24% interrupts.CPU7.46:IR-PCI-MSI.1572871-edge.eth0-TxRx-7 92062 ± 9% -19.7% 73884 ± 5% interrupts.CPU7.CAL:Function_call_interrupts 1784266 +12.4% 2004980 interrupts.CPU7.LOC:Local_timer_interrupts 2505703 ± 11% -19.0% 2029187 ± 5% interrupts.CPU7.RES:Rescheduling_interrupts 107698 ± 8% -18.4% 87903 ± 6% interrupts.CPU7.TLB:TLB_shootdowns 388281 ± 12% +27.8% 496347 ± 4% interrupts.CPU7.TRM:Thermal_event_interrupts 92337 ± 12% -17.5% 76148 ± 2% interrupts.CPU70.CAL:Function_call_interrupts 1785161 +12.0% 1999757 interrupts.CPU70.LOC:Local_timer_interrupts 1562360 ± 4% +170.8% 4231208 ± 9% interrupts.CPU70.RES:Rescheduling_interrupts 103862 ± 10% -18.8% 84371 ± 3% interrupts.CPU70.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380141 ± 2% interrupts.CPU70.TRM:Thermal_event_interrupts 95829 ± 12% -21.2% 75509 ± 5% interrupts.CPU71.CAL:Function_call_interrupts 1784894 +12.1% 2001287 interrupts.CPU71.LOC:Local_timer_interrupts 1533567 ± 7% +196.9% 4553440 interrupts.CPU71.RES:Rescheduling_interrupts 105883 ± 11% -20.4% 84248 ± 6% interrupts.CPU71.TLB:TLB_shootdowns 329247 ± 3% +15.5% 380146 ± 2% interrupts.CPU71.TRM:Thermal_event_interrupts 88372 ± 11% -20.3% 70389 ± 7% interrupts.CPU8.CAL:Function_call_interrupts 1784452 +12.4% 2005305 interrupts.CPU8.LOC:Local_timer_interrupts 103992 ± 10% -17.4% 85887 ± 5% interrupts.CPU8.TLB:TLB_shootdowns 388422 ± 12% +27.8% 496557 ± 4% interrupts.CPU8.TRM:Thermal_event_interrupts 89290 ± 10% -19.1% 72221 ± 6% interrupts.CPU9.CAL:Function_call_interrupts 1784716 +12.4% 2005710 interrupts.CPU9.LOC:Local_timer_interrupts 104340 ± 10% -16.0% 87647 ± 3% interrupts.CPU9.TLB:TLB_shootdowns 388408 ± 12% +27.8% 496557 ± 4% interrupts.CPU9.TRM:Thermal_event_interrupts 1.285e+08 +12.2% 1.442e+08 interrupts.LOC:Local_timer_interrupts 144.00 +50.0% 216.00 interrupts.MCP:Machine_check_polls 1.677e+08 ± 8% +50.4% 2.522e+08 ± 2% interrupts.RES:Rescheduling_interrupts 7624145 ± 10% -19.7% 6125835 ± 5% interrupts.TLB:TLB_shootdowns 25836833 ± 8% +22.2% 31559779 ± 2% interrupts.TRM:Thermal_event_interrupts *************************************************************************************************** lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory ========================================================================================= compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode: gcc-7/performance/pipe/x86_64-rhel-7.6/process/50%/debian-x86_64-2019-11-14.cgz/lkp-bdw-ep6/hackbench/0xb000038 commit: 43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq") 3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found") 43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 134091 -10.6% 119937 hackbench.throughput 3.352e+09 ± 2% +16.1% 3.892e+09 ± 5% hackbench.time.involuntary_context_switches 535571 ± 2% +10.6% 592420 ± 4% numa-meminfo.node0.FilePages 12614 ± 18% -12.2% 11078 ± 18% numa-meminfo.node1.Mapped 3472 ± 6% +18.0% 4096 slabinfo.skbuff_head_cache.active_objs 3568 ± 6% +15.7% 4128 slabinfo.skbuff_head_cache.num_objs 730128 ± 5% -52.2% 349073 ± 25% turbostat.C3 14.77 +5.9% 15.64 turbostat.RAMWatt 24613783 ± 5% -32.6% 16595193 ± 23% numa-numastat.node0.local_node 24632941 ± 5% -32.6% 16605181 ± 23% numa-numastat.node0.numa_hit 25827961 -29.9% 18097150 ± 19% numa-numastat.node1.local_node 25836948 -29.9% 18115676 ± 19% numa-numastat.node1.numa_hit 50467170 ± 2% -31.2% 34733456 ± 10% proc-vmstat.numa_hit 50439016 ± 2% -31.2% 34704933 ± 10% proc-vmstat.numa_local 50622650 ± 2% -31.1% 34896854 ± 10% proc-vmstat.pgalloc_normal 50548381 ± 2% -31.1% 34834391 ± 10% proc-vmstat.pgfree 79.00 +2.2% 80.75 vmstat.cpu.sy 19.00 -7.9% 17.50 ± 2% vmstat.cpu.us 13450755 +4.6% 14066265 vmstat.system.cs 1122772 -12.1% 986681 vmstat.system.in 133891 ± 2% +10.6% 148112 ± 4% numa-vmstat.node0.nr_file_pages 12301995 ± 4% -30.2% 8590153 ± 22% numa-vmstat.node0.numa_hit 12282910 ± 4% -30.1% 8580139 ± 22% numa-vmstat.node0.numa_local 12964354 ± 5% -31.2% 8914408 ± 18% numa-vmstat.node1.numa_hit 12806275 ± 5% -31.7% 8746928 ± 19% numa-vmstat.node1.numa_local 33816695 ± 10% +17.8% 39824104 ± 7% cpuidle.C1.time 21015951 ± 3% +19.6% 25127666 ± 7% cpuidle.C1E.time 730991 ± 5% -52.1% 350128 ± 25% cpuidle.C3.usage 1.497e+08 ± 28% +41.6% 2.119e+08 ± 15% cpuidle.C6.time 6771631 ± 4% -45.2% 3713804 ± 15% cpuidle.POLL.time 5209842 ± 4% -44.1% 2913245 ± 15% cpuidle.POLL.usage 26965506 +5.5% 28444381 ± 4% sched_debug.cfs_rq:/.min_vruntime.avg 28257967 +6.6% 30126596 ± 4% sched_debug.cfs_rq:/.min_vruntime.max 582203 ± 9% +37.0% 797353 ± 8% sched_debug.cfs_rq:/.min_vruntime.stddev 4.65 ± 8% -10.7% 4.16 ± 6% sched_debug.cfs_rq:/.runnable_load_avg.stddev 4620 ± 6% -10.7% 4125 ± 4% sched_debug.cfs_rq:/.runnable_weight.stddev 582831 ± 9% +36.6% 796226 ± 7% sched_debug.cfs_rq:/.spread0.stddev 154.73 ± 2% -14.7% 132.00 ± 3% sched_debug.cfs_rq:/.util_avg.stddev 120.93 ± 25% +43.5% 173.50 ± 19% sched_debug.cfs_rq:/.util_est_enqueued.min 184.93 ± 2% -11.0% 164.54 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev 182135 ± 18% +57.1% 286098 ± 6% sched_debug.cpu.avg_idle.avg 185926 ± 19% +49.0% 276991 ± 9% sched_debug.cpu.avg_idle.stddev 3.32 ± 4% -10.8% 2.96 ± 3% sched_debug.cpu.nr_running.stddev 46710201 +9.0% 50934866 ± 5% sched_debug.cpu.nr_switches.avg 50053735 +16.9% 58491253 ± 7% sched_debug.cpu.nr_switches.max 43751407 +6.7% 46698664 ± 4% sched_debug.cpu.nr_switches.min 1344975 ± 10% +78.9% 2406504 ± 16% sched_debug.cpu.nr_switches.stddev 1.63 ± 3% -32.8% 1.09 ± 11% sched_debug.cpu.nr_uninterruptible.avg 1017 ± 27% -59.5% 411.75 ± 16% interrupts.40:IR-PCI-MSI.1572871-edge.eth0-TxRx-6 6547829 -14.4% 5603910 ± 5% interrupts.CPU16.RES:Rescheduling_interrupts 1017 ± 27% -59.5% 411.75 ± 16% interrupts.CPU19.40:IR-PCI-MSI.1572871-edge.eth0-TxRx-6 6801598 -14.7% 5801847 ± 5% interrupts.CPU19.RES:Rescheduling_interrupts 6609756 -13.7% 5703084 ± 8% interrupts.CPU20.RES:Rescheduling_interrupts 6446878 ± 2% -11.8% 5689145 ± 7% interrupts.CPU21.RES:Rescheduling_interrupts 6634825 -9.4% 6008203 ± 6% interrupts.CPU24.RES:Rescheduling_interrupts 6801182 -9.8% 6137528 ± 3% interrupts.CPU25.RES:Rescheduling_interrupts 6687252 -12.9% 5821967 ± 6% interrupts.CPU28.RES:Rescheduling_interrupts 6583387 -10.9% 5868659 ± 6% interrupts.CPU29.RES:Rescheduling_interrupts 6585631 -16.6% 5490726 ± 9% interrupts.CPU37.RES:Rescheduling_interrupts 6759973 -13.2% 5867800 ± 7% interrupts.CPU57.RES:Rescheduling_interrupts 6459616 ± 2% -10.5% 5783047 ± 6% interrupts.CPU58.RES:Rescheduling_interrupts 6580818 ± 2% -16.9% 5468945 ± 3% interrupts.CPU60.RES:Rescheduling_interrupts 6723178 ± 2% -12.4% 5887778 ± 3% interrupts.CPU63.RES:Rescheduling_interrupts 6480129 -16.2% 5427168 ± 7% interrupts.CPU64.RES:Rescheduling_interrupts 6439558 ± 2% -11.6% 5691160 ± 5% interrupts.CPU65.RES:Rescheduling_interrupts 6662822 -12.0% 5866390 ± 7% interrupts.CPU68.RES:Rescheduling_interrupts 6602178 -12.0% 5809760 ± 7% interrupts.CPU7.RES:Rescheduling_interrupts 6616172 -14.1% 5685889 ± 7% interrupts.CPU73.RES:Rescheduling_interrupts 6591477 -13.8% 5682655 ± 8% interrupts.CPU81.RES:Rescheduling_interrupts 13.97 +15.8% 16.18 perf-stat.i.MPKI 1.70 -0.1 1.65 perf-stat.i.branch-miss-rate% 3.607e+08 -1.6% 3.549e+08 perf-stat.i.branch-misses 1.22 ± 6% +0.8 2.01 ± 18% perf-stat.i.cache-miss-rate% 17650711 ± 6% +98.0% 34953649 ± 18% perf-stat.i.cache-misses 1.496e+09 +17.3% 1.755e+09 perf-stat.i.cache-references 13503062 +4.6% 14122231 perf-stat.i.context-switches 2946654 ± 2% +18.5% 3491269 ± 3% perf-stat.i.cpu-migrations 29365 ± 8% -30.6% 20387 ± 31% perf-stat.i.cycles-between-cache-misses 0.35 ± 2% +0.0 0.36 ± 2% perf-stat.i.dTLB-store-miss-rate% 2.048e+10 -2.1% 2.006e+10 perf-stat.i.dTLB-stores 55.17 -1.4 53.74 perf-stat.i.iTLB-load-miss-rate% 1.019e+08 +5.5% 1.074e+08 perf-stat.i.iTLB-loads 3068 -3.0% 2975 perf-stat.i.minor-faults 9195204 ± 6% +113.0% 19589023 ± 18% perf-stat.i.node-load-misses 28132 ± 6% +125.3% 63396 ± 13% perf-stat.i.node-loads 58.92 -9.8 49.16 ± 4% perf-stat.i.node-store-miss-rate% 3588196 ± 6% +60.5% 5760265 ± 19% perf-stat.i.node-store-misses 2415953 ± 5% +145.2% 5924042 ± 18% perf-stat.i.node-stores 3068 -3.0% 2975 perf-stat.i.page-faults 13.96 +16.0% 16.19 perf-stat.overall.MPKI 1.70 -0.1 1.65 perf-stat.overall.branch-miss-rate% 1.18 ± 6% +0.8 1.99 ± 18% perf-stat.overall.cache-miss-rate% 13823 ± 6% -47.8% 7221 ± 20% perf-stat.overall.cycles-between-cache-misses 0.35 ± 2% +0.0 0.36 ± 2% perf-stat.overall.dTLB-store-miss-rate% 55.11 -1.4 53.70 perf-stat.overall.iTLB-load-miss-rate% 59.75 -10.5 49.23 perf-stat.overall.node-store-miss-rate% 78192 +13.0% 88370 perf-stat.overall.path-length 3.601e+08 -1.6% 3.543e+08 perf-stat.ps.branch-misses 17622620 ± 6% +98.0% 34899005 ± 18% perf-stat.ps.cache-misses 1.494e+09 +17.3% 1.752e+09 perf-stat.ps.cache-references 13480657 +4.6% 14099389 perf-stat.ps.context-switches 2941749 ± 2% +18.5% 3485631 ± 3% perf-stat.ps.cpu-migrations 2.045e+10 -2.1% 2.003e+10 perf-stat.ps.dTLB-stores 1.017e+08 +5.5% 1.072e+08 perf-stat.ps.iTLB-loads 3064 -3.0% 2972 perf-stat.ps.minor-faults 9180406 ± 6% +113.0% 19558101 ± 18% perf-stat.ps.node-load-misses 28101 ± 6% +125.3% 63314 ± 13% perf-stat.ps.node-loads 3582467 ± 6% +60.5% 5751138 ± 19% perf-stat.ps.node-store-misses 2412086 ± 5% +145.2% 5914636 ± 18% perf-stat.ps.node-stores 3064 -3.0% 2972 perf-stat.ps.page-faults 223485 +9.1% 243918 ± 5% softirqs.CPU0.TIMER 11887 ± 11% -18.2% 9723 ± 6% softirqs.CPU1.SCHED 215411 +9.2% 235267 ± 6% softirqs.CPU1.TIMER 11293 ± 3% -14.3% 9678 ± 10% softirqs.CPU10.SCHED 213916 +9.8% 234908 ± 5% softirqs.CPU10.TIMER 212758 +7.7% 229141 ± 5% softirqs.CPU11.TIMER 11098 ± 2% -15.9% 9337 ± 10% softirqs.CPU12.SCHED 213397 +9.0% 232706 ± 6% softirqs.CPU12.TIMER 212419 +8.3% 230115 ± 6% softirqs.CPU13.TIMER 10777 ± 4% -12.5% 9426 ± 6% softirqs.CPU14.SCHED 11241 ± 3% -11.5% 9950 ± 6% softirqs.CPU15.SCHED 214604 +10.0% 236116 ± 8% softirqs.CPU15.TIMER 11147 -14.9% 9482 ± 8% softirqs.CPU16.SCHED 216343 +8.4% 234558 ± 6% softirqs.CPU16.TIMER 214680 +7.0% 229748 ± 6% softirqs.CPU17.TIMER 212791 +8.6% 230988 ± 6% softirqs.CPU19.TIMER 11211 ± 4% -10.4% 10047 ± 6% softirqs.CPU2.SCHED 219918 +9.8% 241513 ± 6% softirqs.CPU2.TIMER 10825 ± 4% -14.3% 9273 ± 7% softirqs.CPU20.SCHED 215209 +8.8% 234149 ± 6% softirqs.CPU20.TIMER 10869 ± 4% -11.4% 9626 ± 4% softirqs.CPU21.SCHED 216225 +8.7% 235105 ± 5% softirqs.CPU21.TIMER 13013 ± 7% -14.0% 11192 ± 3% softirqs.CPU22.SCHED 220406 +8.9% 239975 ± 4% softirqs.CPU22.TIMER 11925 -14.9% 10145 ± 3% softirqs.CPU23.SCHED 220231 +7.6% 237056 ± 5% softirqs.CPU23.TIMER 11140 ± 2% -11.5% 9861 ± 7% softirqs.CPU24.SCHED 216091 +8.9% 235338 ± 5% softirqs.CPU24.TIMER 11438 ± 3% -16.3% 9578 ± 2% softirqs.CPU25.SCHED 214880 +9.4% 235143 ± 5% softirqs.CPU25.TIMER 214888 +8.8% 233874 ± 5% softirqs.CPU26.TIMER 11410 ± 2% -13.7% 9844 ± 10% softirqs.CPU27.SCHED 215012 +8.7% 233790 ± 4% softirqs.CPU27.TIMER 220618 ± 2% +7.8% 237788 ± 5% softirqs.CPU28.TIMER 11096 ± 5% -14.3% 9513 ± 6% softirqs.CPU29.SCHED 217194 +8.2% 235108 ± 5% softirqs.CPU29.TIMER 11445 ± 2% -16.7% 9536 ± 10% softirqs.CPU3.SCHED 214692 +9.1% 234221 ± 5% softirqs.CPU3.TIMER 11127 ± 5% -10.5% 9954 ± 6% softirqs.CPU30.SCHED 214861 +8.2% 232532 ± 5% softirqs.CPU30.TIMER 217582 +7.8% 234493 ± 4% softirqs.CPU31.TIMER 214606 +8.8% 233473 ± 4% softirqs.CPU32.TIMER 11299 ± 2% -11.7% 9975 ± 6% softirqs.CPU33.SCHED 11503 ± 2% -16.0% 9658 ± 10% softirqs.CPU34.SCHED 213233 +10.9% 236580 ± 6% softirqs.CPU34.TIMER 11162 ± 2% -11.6% 9863 ± 7% softirqs.CPU35.SCHED 11241 -13.9% 9678 ± 6% softirqs.CPU36.SCHED 215681 +9.2% 235446 ± 4% softirqs.CPU36.TIMER 11358 -15.2% 9632 ± 6% softirqs.CPU37.SCHED 219691 ± 2% +18.5% 260407 ± 13% softirqs.CPU37.TIMER 11570 ± 2% -13.1% 10049 ± 6% softirqs.CPU38.SCHED 215559 +8.2% 233189 ± 4% softirqs.CPU38.TIMER 11295 ± 4% -13.1% 9819 ± 10% softirqs.CPU39.SCHED 215781 ± 2% +7.6% 232108 ± 5% softirqs.CPU39.TIMER 11506 ± 7% -13.0% 10006 ± 6% softirqs.CPU4.SCHED 214973 +8.9% 234140 ± 6% softirqs.CPU4.TIMER 11304 ± 2% -13.7% 9759 ± 10% softirqs.CPU40.SCHED 213827 +10.1% 235341 ± 4% softirqs.CPU40.TIMER 11462 ± 4% -13.5% 9913 ± 8% softirqs.CPU41.SCHED 215879 ± 2% +7.2% 231499 ± 5% softirqs.CPU41.TIMER 11418 ± 3% -14.0% 9819 ± 8% softirqs.CPU43.SCHED 214864 +8.3% 232698 ± 4% softirqs.CPU43.TIMER 10706 ± 5% -13.9% 9223 ± 11% softirqs.CPU44.SCHED 217386 +9.4% 237916 ± 5% softirqs.CPU44.TIMER 215191 +9.7% 236017 ± 5% softirqs.CPU45.TIMER 214214 +9.9% 235466 ± 6% softirqs.CPU46.TIMER 214471 +9.9% 235675 ± 5% softirqs.CPU47.TIMER 11251 ± 2% -13.0% 9785 ± 8% softirqs.CPU48.SCHED 212500 +9.9% 233451 ± 6% softirqs.CPU48.TIMER 213609 +9.9% 234657 ± 5% softirqs.CPU49.TIMER 216836 +8.5% 235225 ± 5% softirqs.CPU5.TIMER 215854 +9.0% 235383 ± 5% softirqs.CPU50.TIMER 11468 -14.4% 9820 ± 9% softirqs.CPU51.SCHED 215402 +8.8% 234360 ± 5% softirqs.CPU51.TIMER 214152 +8.8% 233038 ± 5% softirqs.CPU52.TIMER 212843 +8.4% 230716 ± 5% softirqs.CPU53.TIMER 11373 ± 2% -14.9% 9679 ± 7% softirqs.CPU54.SCHED 214008 +9.5% 234380 ± 5% softirqs.CPU54.TIMER 212619 +7.5% 228539 ± 5% softirqs.CPU55.TIMER 11136 ± 3% -16.4% 9312 ± 10% softirqs.CPU56.SCHED 213404 +9.0% 232542 ± 6% softirqs.CPU56.TIMER 11171 ± 4% -11.9% 9837 ± 7% softirqs.CPU57.SCHED 212283 +8.6% 230474 ± 6% softirqs.CPU57.TIMER 11137 ± 4% -14.0% 9578 ± 6% softirqs.CPU58.SCHED 11226 ± 4% -17.2% 9289 ± 4% softirqs.CPU59.SCHED 214193 +23.1% 263668 ± 22% softirqs.CPU59.TIMER 217195 +8.2% 234975 ± 5% softirqs.CPU6.TIMER 11051 ± 4% -14.4% 9460 ± 9% softirqs.CPU60.SCHED 217131 +7.9% 234363 ± 5% softirqs.CPU60.TIMER 11206 ± 3% -12.6% 9793 ± 9% softirqs.CPU61.SCHED 211791 +9.2% 231302 ± 5% softirqs.CPU61.TIMER 214956 ± 2% +8.8% 233770 ± 5% softirqs.CPU62.TIMER 11180 ± 3% -14.8% 9526 ± 8% softirqs.CPU63.SCHED 213413 +8.7% 232008 ± 5% softirqs.CPU63.TIMER 215674 +8.1% 233086 ± 6% softirqs.CPU64.TIMER 11032 ± 3% -15.7% 9302 ± 7% softirqs.CPU65.SCHED 215870 +9.0% 235386 ± 5% softirqs.CPU65.TIMER 11403 ± 3% -14.2% 9786 ± 6% softirqs.CPU66.SCHED 215968 +9.4% 236175 ± 5% softirqs.CPU66.TIMER 11098 ± 3% -7.4% 10281 ± 4% softirqs.CPU67.SCHED 217846 +8.3% 236023 ± 4% softirqs.CPU67.TIMER 11094 -12.2% 9740 ± 7% softirqs.CPU68.SCHED 215512 +9.4% 235796 ± 5% softirqs.CPU68.TIMER 11321 ± 3% -14.1% 9724 ± 3% softirqs.CPU69.SCHED 213564 +9.0% 232867 ± 5% softirqs.CPU69.TIMER 218822 +8.8% 238080 ± 6% softirqs.CPU7.TIMER 11507 ± 2% -12.9% 10026 ± 4% softirqs.CPU70.SCHED 213928 +9.4% 233986 ± 4% softirqs.CPU70.TIMER 213528 +8.5% 231730 ± 4% softirqs.CPU71.TIMER 11262 ± 2% -11.1% 10013 ± 5% softirqs.CPU72.SCHED 11145 ± 3% -14.5% 9533 ± 8% softirqs.CPU73.SCHED 216467 +9.2% 236409 ± 5% softirqs.CPU73.TIMER 11447 ± 4% -13.2% 9935 ± 4% softirqs.CPU74.SCHED 212578 +9.1% 232022 ± 5% softirqs.CPU74.TIMER 215256 +7.8% 232072 ± 5% softirqs.CPU75.TIMER 11324 ± 4% -11.8% 9985 ± 3% softirqs.CPU76.SCHED 214170 +8.6% 232516 ± 5% softirqs.CPU76.TIMER 11646 ± 5% -15.7% 9824 ± 7% softirqs.CPU77.SCHED 213706 +8.0% 230880 ± 5% softirqs.CPU77.TIMER 11397 ± 6% -13.9% 9810 ± 10% softirqs.CPU78.SCHED 11179 ± 3% -11.5% 9894 ± 5% softirqs.CPU79.SCHED 213215 +8.4% 231081 ± 5% softirqs.CPU79.TIMER 11165 ± 4% -16.2% 9352 ± 9% softirqs.CPU8.SCHED 216508 ± 2% +7.9% 233692 ± 6% softirqs.CPU8.TIMER 11203 ± 2% -13.9% 9643 ± 4% softirqs.CPU80.SCHED 216446 +8.3% 234469 ± 5% softirqs.CPU80.TIMER 11151 ± 2% -12.8% 9726 ± 6% softirqs.CPU81.SCHED 11373 ± 3% -13.0% 9891 ± 6% softirqs.CPU82.SCHED 213597 +8.9% 232608 ± 5% softirqs.CPU82.TIMER 214308 +7.9% 231148 ± 5% softirqs.CPU83.TIMER 214384 +20.9% 259149 ± 15% softirqs.CPU84.TIMER 11458 -13.7% 9887 ± 7% softirqs.CPU86.SCHED 215459 +8.2% 233058 ± 5% softirqs.CPU86.TIMER 11303 ± 4% -12.4% 9904 ± 8% softirqs.CPU87.SCHED 214185 +7.9% 231116 ± 5% softirqs.CPU87.TIMER 11030 ± 2% -13.0% 9593 ± 8% softirqs.CPU9.SCHED 213229 +8.7% 231788 ± 6% softirqs.CPU9.TIMER 995635 -12.9% 867688 ± 5% softirqs.SCHED 19052052 +8.6% 20690944 ± 5% softirqs.TIMER 31.32 -1.6 29.74 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 30.24 -1.5 28.76 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 26.06 -0.9 25.17 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.35 ± 5% -0.9 1.48 ± 12% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common 25.36 -0.8 24.54 perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64 2.75 ± 4% -0.7 2.08 ± 7% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 4.24 ± 2% -0.6 3.60 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64 3.84 ± 2% -0.6 3.23 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 7.83 -0.5 7.30 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 1.68 ± 6% -0.5 1.17 ± 10% perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 8.02 -0.5 7.53 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 8.08 -0.5 7.60 ± 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 1.63 ± 4% -0.4 1.19 ± 6% perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_write.new_sync_write.vfs_write.ksys_write 1.71 ± 6% -0.4 1.28 ± 8% perf-profile.calltrace.cycles-pp.update_cfs_group.dequeue_task_fair.__schedule.schedule.pipe_wait 2.60 -0.4 2.19 ± 4% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.03 ± 2% -0.4 1.64 ± 4% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.47 -0.4 3.08 ± 4% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 1.95 ± 2% -0.3 1.61 ± 3% perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write 2.67 -0.3 2.35 ± 3% perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read 0.56 -0.3 0.26 ±100% perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.22 ± 4% -0.3 0.94 ± 2% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.new_sync_write.vfs_write.ksys_write 1.25 ± 2% -0.3 1.00 ± 5% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64 2.64 -0.2 2.41 ± 2% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 1.53 -0.2 1.34 ± 7% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64 0.59 ± 4% -0.2 0.40 ± 57% perf-profile.calltrace.cycles-pp.__enqueue_entity.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 1.34 -0.2 1.16 ± 6% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 1.07 ± 2% -0.2 0.90 ± 4% perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.pipe_write.new_sync_write.vfs_write 1.05 -0.2 0.88 ± 6% perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule 1.56 ± 2% -0.2 1.38 ± 2% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.pipe_read.new_sync_read 1.17 ± 4% -0.2 1.01 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 0.98 ± 2% -0.2 0.82 ± 4% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.66 ± 2% -0.2 1.50 ± 5% perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.pipe_read.new_sync_read.vfs_read 0.73 ± 3% -0.2 0.57 ± 2% perf-profile.calltrace.cycles-pp.file_has_perm.security_file_permission.vfs_read.ksys_read.do_syscall_64 0.96 ± 2% -0.1 0.82 ± 3% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter.pipe_write.new_sync_write 0.89 ± 2% -0.1 0.77 ± 3% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.66 ± 6% -0.1 0.54 ± 2% perf-profile.calltrace.cycles-pp.file_has_perm.security_file_permission.vfs_write.ksys_write.do_syscall_64 2.42 -0.1 2.32 perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.pipe_wait.pipe_read 0.89 ± 4% -0.1 0.79 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write 0.74 ± 2% -0.1 0.66 ± 3% perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.84 -0.1 0.79 ± 3% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.56 -0.0 0.54 ± 2% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.77 ± 2% +0.1 0.82 perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function 0.90 ± 3% +0.1 0.97 ± 2% perf-profile.calltrace.cycles-pp.native_write_msr 0.69 ± 3% +0.1 0.77 ± 3% perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.80 ± 4% +0.1 0.89 ± 2% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read 1.10 +0.1 1.20 perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.73 ± 3% +0.1 0.83 ± 2% perf-profile.calltrace.cycles-pp.__switch_to 1.05 +0.1 1.16 perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 0.63 ± 4% +0.1 0.75 ± 2% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop 0.99 ± 5% +0.1 1.12 ± 5% perf-profile.calltrace.cycles-pp.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 1.07 ± 3% +0.1 1.21 ± 2% perf-profile.calltrace.cycles-pp.__switch_to_asm 0.64 ± 6% +0.2 0.80 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.pipe_wait.pipe_read 1.41 ± 3% +0.2 1.59 ± 2% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.77 ± 5% +0.2 1.00 ± 5% perf-profile.calltrace.cycles-pp.load_new_mm_cr3.switch_mm_irqs_off.__schedule.schedule.exit_to_usermode_loop 1.73 ± 3% +0.3 2.06 ± 3% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 0.14 ±173% +0.5 0.63 ± 5% perf-profile.calltrace.cycles-pp.update_rq_clock.__schedule.schedule.pipe_wait.pipe_read 0.13 ±173% +0.5 0.64 ± 5% perf-profile.calltrace.cycles-pp.migrate_task_rq_fair.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common 1.85 ± 5% +0.5 2.38 ± 3% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 0.00 +0.6 0.59 ± 4% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_wait.pipe_read.new_sync_read.vfs_read 0.00 +0.6 0.60 ± 5% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.dequeue_task_fair.__schedule.schedule.pipe_wait 0.00 +0.7 0.67 ± 5% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.15 ±173% +1.0 1.14 ± 12% perf-profile.calltrace.cycles-pp.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function 87.66 +1.1 88.78 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 86.87 +1.1 88.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.59 ± 4% +1.4 6.96 ± 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.72 ± 4% +1.4 7.12 ± 3% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.87 ± 4% +1.4 7.28 ± 3% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.70 ± 30% +3.3 4.98 ± 13% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function 35.54 +3.3 38.87 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 34.28 +3.5 37.82 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 31.12 +4.1 35.22 ± 2% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 30.49 +4.1 34.63 ± 2% perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 22.54 ± 2% +6.0 28.53 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write 20.58 ± 2% +6.2 26.76 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 19.97 ± 2% +6.2 26.16 ± 3% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write 19.73 ± 3% +6.2 25.92 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write 3.39 ± 20% +7.0 10.39 ± 12% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common 4.89 ± 15% +7.1 11.98 ± 10% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 31.34 -1.6 29.75 perf-profile.children.cycles-pp.ksys_read 30.28 -1.5 28.79 perf-profile.children.cycles-pp.vfs_read 4.14 ± 5% -1.3 2.85 ± 7% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 3.51 ± 6% -1.0 2.54 ± 8% perf-profile.children.cycles-pp.update_cfs_group 26.10 -0.9 25.20 perf-profile.children.cycles-pp.new_sync_read 25.42 -0.8 24.59 perf-profile.children.cycles-pp.pipe_read 4.67 -0.8 3.86 ± 4% perf-profile.children.cycles-pp.security_file_permission 4.33 ± 2% -0.7 3.65 ± 2% perf-profile.children.cycles-pp.syscall_return_via_sysret 4.24 ± 2% -0.6 3.60 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64 2.18 ± 4% -0.5 1.64 ± 6% perf-profile.children.cycles-pp.mutex_unlock 7.92 -0.5 7.40 ± 3% perf-profile.children.cycles-pp.enqueue_task_fair 8.10 -0.5 7.61 ± 3% perf-profile.children.cycles-pp.activate_task 8.16 -0.5 7.67 ± 3% perf-profile.children.cycles-pp.ttwu_do_activate 4.12 ± 4% -0.5 3.64 ± 5% perf-profile.children.cycles-pp._raw_spin_lock 2.83 -0.4 2.38 ± 6% perf-profile.children.cycles-pp.selinux_file_permission 3.53 -0.4 3.13 ± 4% perf-profile.children.cycles-pp.enqueue_entity 1.99 ± 2% -0.4 1.64 ± 3% perf-profile.children.cycles-pp.copy_page_from_iter 3.34 -0.4 2.99 ± 4% perf-profile.children.cycles-pp.update_load_avg 2.70 -0.3 2.39 ± 3% perf-profile.children.cycles-pp.copy_page_to_iter 2.66 -0.3 2.35 ± 3% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.93 ± 3% -0.3 0.64 ± 7% perf-profile.children.cycles-pp.__mutex_unlock_slowpath 1.41 ± 4% -0.3 1.13 perf-profile.children.cycles-pp.file_has_perm 2.75 -0.3 2.49 ± 2% perf-profile.children.cycles-pp.dequeue_entity 2.29 ± 2% -0.3 2.04 perf-profile.children.cycles-pp.mutex_lock 0.85 ± 3% -0.2 0.61 ± 4% perf-profile.children.cycles-pp.__inode_security_revalidate 0.59 ± 3% -0.2 0.39 ± 6% perf-profile.children.cycles-pp.__mutex_lock 1.79 ± 2% -0.2 1.59 ± 4% perf-profile.children.cycles-pp.__fdget_pos 1.70 ± 2% -0.2 1.52 ± 3% perf-profile.children.cycles-pp.__fget_light 0.93 ± 3% -0.2 0.75 perf-profile.children.cycles-pp.avc_has_perm 1.09 ± 2% -0.2 0.92 ± 4% perf-profile.children.cycles-pp.copyin 2.33 ± 2% -0.2 2.16 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.96 ± 2% -0.2 0.81 ± 2% perf-profile.children.cycles-pp.___might_sleep 1.67 ± 2% -0.2 1.52 ± 5% perf-profile.children.cycles-pp.copyout 0.83 ± 4% -0.1 0.68 ± 2% perf-profile.children.cycles-pp._cond_resched 0.33 ± 5% -0.1 0.19 ± 10% perf-profile.children.cycles-pp.preempt_schedule_common 0.42 ± 2% -0.1 0.28 ± 8% perf-profile.children.cycles-pp.wake_up_q 0.70 ± 3% -0.1 0.59 ± 3% perf-profile.children.cycles-pp.__might_sleep 0.65 ± 5% -0.1 0.54 ± 7% perf-profile.children.cycles-pp.__fsnotify_parent 0.52 ± 2% -0.1 0.43 perf-profile.children.cycles-pp.__might_fault 0.53 ± 5% -0.1 0.45 perf-profile.children.cycles-pp.current_time 0.84 ± 3% -0.1 0.77 perf-profile.children.cycles-pp.fsnotify 0.16 ± 7% -0.1 0.09 ± 13% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.15 ± 8% -0.1 0.08 ± 15% perf-profile.children.cycles-pp.prepare_exit_to_usermode 0.22 ± 3% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.wake_q_add 0.49 -0.1 0.44 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.26 ± 4% -0.1 0.21 ± 2% perf-profile.children.cycles-pp.rcu_all_qs 0.08 ± 14% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__vfs_read 0.25 ± 4% -0.0 0.20 ± 5% perf-profile.children.cycles-pp.finish_wait 0.38 -0.0 0.34 ± 2% perf-profile.children.cycles-pp.__x64_sys_read 0.43 ± 3% -0.0 0.39 ± 2% perf-profile.children.cycles-pp.prepare_to_wait 0.14 ± 20% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.task_tick_fair 0.16 ± 19% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.scheduler_tick 0.10 ± 8% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.mutex_spin_on_owner 0.18 ± 8% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.inode_has_perm 0.21 ± 2% -0.0 0.18 ± 6% perf-profile.children.cycles-pp.deactivate_task 0.20 ± 2% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.generic_pipe_buf_confirm 0.13 ± 3% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.__sb_end_write 0.14 ± 9% -0.0 0.12 perf-profile.children.cycles-pp.timespec64_trunc 0.14 ± 3% -0.0 0.11 ± 3% perf-profile.children.cycles-pp.native_irq_return_iret 0.11 ± 4% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.smp_reschedule_interrupt 0.17 ± 4% -0.0 0.15 perf-profile.children.cycles-pp.iov_iter_init 0.11 ± 4% -0.0 0.09 ± 10% perf-profile.children.cycles-pp.native_apic_msr_eoi_write 0.09 ± 4% -0.0 0.07 perf-profile.children.cycles-pp.__x2apic_send_IPI_dest 0.15 -0.0 0.13 ± 3% perf-profile.children.cycles-pp.rb_insert_color 0.18 ± 2% -0.0 0.17 ± 2% perf-profile.children.cycles-pp.set_next_buddy 0.08 ± 5% -0.0 0.07 perf-profile.children.cycles-pp.interrupt_entry 0.23 ± 2% +0.0 0.24 perf-profile.children.cycles-pp.cpumask_next 0.67 +0.0 0.69 perf-profile.children.cycles-pp.__calc_delta 0.15 ± 3% +0.0 0.18 ± 6% perf-profile.children.cycles-pp.wakeup_preempt_entity 0.17 ± 6% +0.1 0.23 ± 3% perf-profile.children.cycles-pp.remove_entity_load_avg 0.14 ± 6% +0.1 0.19 ± 7% perf-profile.children.cycles-pp.attach_entity_load_avg 0.80 ± 3% +0.1 0.86 perf-profile.children.cycles-pp.check_preempt_wakeup 0.37 ± 3% +0.1 0.45 ± 4% perf-profile.children.cycles-pp.account_entity_enqueue 0.79 ± 4% +0.1 0.88 ± 5% perf-profile.children.cycles-pp.finish_task_switch 0.80 ± 2% +0.1 0.89 ± 2% perf-profile.children.cycles-pp.put_prev_entity 0.54 ± 2% +0.1 0.65 perf-profile.children.cycles-pp.account_entity_dequeue 1.01 ± 5% +0.1 1.13 ± 5% perf-profile.children.cycles-pp.set_task_cpu 0.49 ± 7% +0.2 0.65 ± 5% perf-profile.children.cycles-pp.migrate_task_rq_fair 1.23 ± 4% +0.2 1.40 ± 2% perf-profile.children.cycles-pp.__switch_to_asm 1.15 ± 2% +0.2 1.33 ± 2% perf-profile.children.cycles-pp.__switch_to 1.42 ± 3% +0.2 1.59 ± 2% perf-profile.children.cycles-pp.switch_fpu_return 0.76 ± 3% +0.2 0.96 ± 3% perf-profile.children.cycles-pp.pick_next_entity 2.21 +0.2 2.42 perf-profile.children.cycles-pp.reweight_entity 2.20 ± 2% +0.2 2.43 perf-profile.children.cycles-pp.load_new_mm_cr3 1.24 ± 5% +0.2 1.48 ± 4% perf-profile.children.cycles-pp.update_rq_clock 1.43 ± 5% +0.3 1.73 ± 2% perf-profile.children.cycles-pp.___perf_sw_event 0.38 ± 15% +0.3 0.68 ± 9% perf-profile.children.cycles-pp.find_next_bit 3.96 +0.3 4.28 perf-profile.children.cycles-pp.pick_next_task_fair 4.43 ± 2% +0.4 4.82 perf-profile.children.cycles-pp.switch_mm_irqs_off 1.11 +0.7 1.83 ± 4% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.46 ± 26% +0.7 1.20 ± 12% perf-profile.children.cycles-pp.cpumask_next_wrap 87.76 +1.1 88.86 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 86.98 +1.1 88.10 perf-profile.children.cycles-pp.do_syscall_64 21.71 +1.3 23.02 perf-profile.children.cycles-pp.__schedule 5.98 ± 4% +1.4 7.34 ± 3% perf-profile.children.cycles-pp.exit_to_usermode_loop 21.59 +1.4 22.99 perf-profile.children.cycles-pp.schedule 1.73 ± 29% +3.3 5.03 ± 13% perf-profile.children.cycles-pp.available_idle_cpu 35.55 +3.3 38.88 perf-profile.children.cycles-pp.ksys_write 34.31 +3.5 37.84 perf-profile.children.cycles-pp.vfs_write 31.14 +4.1 35.23 ± 2% perf-profile.children.cycles-pp.new_sync_write 30.56 +4.2 34.74 ± 2% perf-profile.children.cycles-pp.pipe_write 22.96 ± 2% +6.0 28.93 ± 3% perf-profile.children.cycles-pp.__wake_up_common_lock 20.11 ± 2% +6.1 26.19 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 20.66 ± 2% +6.2 26.82 ± 3% perf-profile.children.cycles-pp.__wake_up_common 20.01 ± 2% +6.2 26.19 ± 3% perf-profile.children.cycles-pp.autoremove_wake_function 3.46 ± 20% +7.0 10.48 ± 12% perf-profile.children.cycles-pp.select_idle_sibling 4.92 ± 15% +7.1 12.01 ± 10% perf-profile.children.cycles-pp.select_task_rq_fair 11.85 ± 3% -2.1 9.72 ± 2% perf-profile.self.cycles-pp.do_syscall_64 4.13 ± 5% -1.3 2.85 ± 8% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 3.49 ± 6% -1.0 2.52 ± 9% perf-profile.self.cycles-pp.update_cfs_group 4.33 ± 2% -0.7 3.64 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret 4.24 ± 2% -0.6 3.60 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64 2.12 ± 4% -0.5 1.59 ± 6% perf-profile.self.cycles-pp.mutex_unlock 2.61 -0.3 2.31 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 1.60 ± 2% -0.3 1.30 ± 5% perf-profile.self.cycles-pp.update_load_avg 1.48 ± 3% -0.2 1.25 ± 2% perf-profile.self.cycles-pp.mutex_lock 1.95 -0.2 1.75 ± 7% perf-profile.self.cycles-pp.selinux_file_permission 1.06 ± 5% -0.2 0.86 ± 5% perf-profile.self.cycles-pp.pipe_write 1.46 ± 2% -0.2 1.28 ± 3% perf-profile.self.cycles-pp.pipe_read 1.65 ± 2% -0.2 1.49 ± 4% perf-profile.self.cycles-pp.__fget_light 0.91 ± 3% -0.2 0.74 perf-profile.self.cycles-pp.avc_has_perm 0.94 ± 2% -0.2 0.79 ± 2% perf-profile.self.cycles-pp.___might_sleep 1.84 -0.1 1.72 ± 2% perf-profile.self.cycles-pp.update_curr 1.05 -0.1 0.93 perf-profile.self.cycles-pp.enqueue_task_fair 0.63 ± 3% -0.1 0.52 ± 2% perf-profile.self.cycles-pp.__might_sleep 0.60 ± 6% -0.1 0.51 ± 7% perf-profile.self.cycles-pp.__fsnotify_parent 0.29 ± 3% -0.1 0.19 ± 5% perf-profile.self.cycles-pp.__mutex_lock 0.38 ± 4% -0.1 0.29 ± 4% perf-profile.self.cycles-pp.vfs_write 0.51 -0.1 0.43 ± 4% perf-profile.self.cycles-pp.copy_page_to_iter 0.49 -0.1 0.41 ± 2% perf-profile.self.cycles-pp.new_sync_write 0.35 ± 4% -0.1 0.28 ± 2% perf-profile.self.cycles-pp.copy_page_from_iter 0.81 ± 3% -0.1 0.75 perf-profile.self.cycles-pp.fsnotify 0.26 -0.1 0.19 ± 3% perf-profile.self.cycles-pp.__inode_security_revalidate 0.20 ± 4% -0.1 0.13 ± 6% perf-profile.self.cycles-pp.__mutex_unlock_slowpath 0.29 ± 5% -0.1 0.23 ± 3% perf-profile.self.cycles-pp.file_has_perm 0.22 -0.1 0.16 ± 8% perf-profile.self.cycles-pp.wake_q_add 0.57 ± 2% -0.1 0.52 perf-profile.self.cycles-pp.new_sync_read 0.34 ± 2% -0.1 0.29 ± 3% perf-profile.self.cycles-pp.dequeue_entity 0.26 ± 3% -0.1 0.21 ± 3% perf-profile.self.cycles-pp.ksys_read 0.28 ± 2% -0.0 0.23 perf-profile.self.cycles-pp.security_file_permission 0.22 ± 3% -0.0 0.17 ± 4% perf-profile.self.cycles-pp.finish_wait 0.25 -0.0 0.21 ± 4% perf-profile.self.cycles-pp.ksys_write 0.88 -0.0 0.83 perf-profile.self.cycles-pp.__lock_text_start 0.66 -0.0 0.61 perf-profile.self.cycles-pp.vfs_read 1.37 -0.0 1.32 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.29 ± 4% -0.0 0.25 perf-profile.self.cycles-pp.current_time 0.20 ± 3% -0.0 0.16 ± 4% perf-profile.self.cycles-pp.rcu_all_qs 0.10 ± 8% -0.0 0.06 ± 6% perf-profile.self.cycles-pp.mutex_spin_on_owner 0.42 -0.0 0.39 ± 2% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.34 -0.0 0.31 ± 2% perf-profile.self.cycles-pp.__x64_sys_read 0.09 ± 5% -0.0 0.05 ± 9% perf-profile.self.cycles-pp.wake_up_q 0.15 ± 8% -0.0 0.12 ± 4% perf-profile.self.cycles-pp.inode_has_perm 0.14 ± 5% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.rb_next 0.20 ± 2% -0.0 0.17 ± 4% perf-profile.self.cycles-pp.deactivate_task 0.11 ± 4% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.__fdget_pos 0.24 ± 2% -0.0 0.22 ± 3% perf-profile.self.cycles-pp.prepare_to_wait 0.14 ± 3% -0.0 0.11 ± 3% perf-profile.self.cycles-pp.native_irq_return_iret 0.13 ± 7% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.timespec64_trunc 0.06 -0.0 0.04 ± 57% perf-profile.self.cycles-pp.copyout 0.11 ± 4% -0.0 0.09 ± 10% perf-profile.self.cycles-pp.native_apic_msr_eoi_write 0.17 ± 4% -0.0 0.15 ± 2% perf-profile.self.cycles-pp.generic_pipe_buf_confirm 0.18 ± 2% -0.0 0.16 ± 4% perf-profile.self.cycles-pp.__might_fault 0.17 ± 2% -0.0 0.16 ± 2% perf-profile.self.cycles-pp.set_next_buddy 0.15 ± 2% -0.0 0.13 ± 3% perf-profile.self.cycles-pp.rb_insert_color 0.12 ± 3% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.__sb_end_write 0.10 ± 8% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64 0.06 -0.0 0.05 perf-profile.self.cycles-pp.kill_fasync 0.14 ± 3% +0.0 0.16 ± 5% perf-profile.self.cycles-pp.exit_to_usermode_loop 0.66 +0.0 0.68 perf-profile.self.cycles-pp.__calc_delta 0.16 ± 5% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.native_load_tls 0.14 ± 3% +0.0 0.17 ± 4% perf-profile.self.cycles-pp.wakeup_preempt_entity 0.67 +0.0 0.70 perf-profile.self.cycles-pp.dequeue_task_fair 0.23 ± 3% +0.0 0.27 ± 3% perf-profile.self.cycles-pp._cond_resched 0.17 ± 4% +0.0 0.21 ± 6% perf-profile.self.cycles-pp.activate_task 0.35 ± 3% +0.0 0.40 perf-profile.self.cycles-pp.pick_next_entity 1.20 +0.1 1.25 perf-profile.self.cycles-pp.select_task_rq_fair 0.13 ± 6% +0.1 0.18 ± 6% perf-profile.self.cycles-pp.attach_entity_load_avg 0.87 ± 3% +0.1 0.96 ± 3% perf-profile.self.cycles-pp.pick_next_task_fair 0.27 ± 6% +0.1 0.36 ± 7% perf-profile.self.cycles-pp.migrate_task_rq_fair 0.32 ± 2% +0.1 0.41 ± 4% perf-profile.self.cycles-pp.account_entity_enqueue 0.42 +0.1 0.56 ± 2% perf-profile.self.cycles-pp.account_entity_dequeue 2.24 ± 3% +0.1 2.39 ± 2% perf-profile.self.cycles-pp.switch_mm_irqs_off 1.09 ± 2% +0.2 1.26 ± 2% perf-profile.self.cycles-pp.__switch_to 1.20 ± 4% +0.2 1.37 ± 2% perf-profile.self.cycles-pp.__switch_to_asm 1.40 ± 3% +0.2 1.58 ± 2% perf-profile.self.cycles-pp.switch_fpu_return 0.89 ± 6% +0.2 1.10 ± 5% perf-profile.self.cycles-pp.update_rq_clock 2.19 ± 2% +0.2 2.42 perf-profile.self.cycles-pp.load_new_mm_cr3 1.33 ± 4% +0.3 1.60 ± 3% perf-profile.self.cycles-pp.___perf_sw_event 0.34 ± 14% +0.3 0.66 ± 9% perf-profile.self.cycles-pp.find_next_bit 0.28 ± 23% +0.4 0.72 ± 12% perf-profile.self.cycles-pp.cpumask_next_wrap 0.89 +0.7 1.58 ± 4% perf-profile.self.cycles-pp._raw_spin_lock 1.04 +0.7 1.76 ± 4% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.65 ± 12% +3.0 3.61 ± 13% perf-profile.self.cycles-pp.select_idle_sibling 1.71 ± 29% +3.3 4.97 ± 13% perf-profile.self.cycles-pp.available_idle_cpu *************************************************************************************************** lkp-cfl-e1: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory ========================================================================================= compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode: gcc-7/performance/pipe/x86_64-rhel-7.6/threads/100%/debian-x86_64-2018-04-03.cgz/lkp-cfl-e1/hackbench/0xb8 commit: 43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq") 3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found") 43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:2 -50% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x 1:2 -50% :4 dmesg.WARNING:stack_recursion %stddev %change %stddev \ | \ 49845 -6.8% 46466 hackbench.throughput 8.776e+08 +13.0% 9.915e+08 hackbench.time.involuntary_context_switches 93623 -6.3% 87707 hackbench.time.minor_page_faults 740.08 -1.4% 729.54 hackbench.time.user_time 1.569e+09 +3.5% 1.623e+09 hackbench.time.voluntary_context_switches 3.072e+08 -6.2% 2.88e+08 hackbench.workload 21.45 -1.7% 21.09 boot-time.boot 1747212 ± 3% -31.0% 1205072 ± 14% cpuidle.POLL.time 1289000 ± 4% -36.6% 817407 ± 13% cpuidle.POLL.usage 4038486 +6.3% 4293331 vmstat.system.cs 394711 -10.2% 354353 ± 2% vmstat.system.in 10059 -15.4% 8506 ± 5% softirqs.CPU0.SCHED 309881 -27.3% 225159 ± 4% softirqs.CPU2.TIMER 220105 +20.8% 265807 ± 16% softirqs.CPU6.TIMER 120937 ± 3% -17.6% 99667 ± 2% softirqs.SCHED 18883 -8.9% 17200 ± 3% slabinfo.kmalloc-32.active_objs 18883 -8.9% 17200 ± 3% slabinfo.kmalloc-32.num_objs 8573 ± 2% -7.6% 7924 slabinfo.lsm_file_cache.active_objs 8573 ± 2% -7.6% 7924 slabinfo.lsm_file_cache.num_objs 5280 -8.8% 4814 slabinfo.task_delay_info.active_objs 5280 -8.8% 4814 slabinfo.task_delay_info.num_objs 65295 -1.9% 64034 proc-vmstat.nr_active_anon 60605 -1.5% 59671 proc-vmstat.nr_anon_pages 12935 -0.9% 12820 proc-vmstat.nr_slab_unreclaimable 65295 -1.9% 64034 proc-vmstat.nr_zone_active_anon 24774010 -16.6% 20673098 proc-vmstat.numa_hit 24774010 -16.6% 20673098 proc-vmstat.numa_local 24839243 -16.5% 20735391 proc-vmstat.pgalloc_normal 673024 -2.4% 657142 proc-vmstat.pgfault 24822112 -16.5% 20717243 proc-vmstat.pgfree 247.95 ± 14% -19.1% 200.59 ± 4% sched_debug.cfs_rq:/.load_avg.max 43.55 ± 3% -19.9% 34.86 ± 6% sched_debug.cfs_rq:/.load_avg.min 55.51 ± 6% -13.2% 48.19 ± 7% sched_debug.cfs_rq:/.load_avg.stddev 20116 ± 8% -12.8% 17531 ± 5% sched_debug.cfs_rq:/.runnable_weight.min 1288 +7.5% 1384 ± 3% sched_debug.cfs_rq:/.util_avg.max 54763 ± 9% +130.8% 126390 ± 25% sched_debug.cpu.avg_idle.avg 301502 ± 18% +84.9% 557406 ± 14% sched_debug.cpu.avg_idle.max 82233 ± 22% +122.7% 183153 ± 17% sched_debug.cpu.avg_idle.stddev 0.00 ± 12% -12.0% 0.00 ± 8% sched_debug.cpu.next_balance.stddev 16.92 ± 4% -7.5% 15.66 ± 3% sched_debug.cpu.nr_running.avg 2.92 ± 10% -20.9% 2.31 ± 16% sched_debug.cpu.nr_uninterruptible.avg 2977 ± 50% -73.0% 803.75 ± 43% interrupts.133:IR-PCI-MSI.2097154-edge.eth1-TxRx-1 546.00 ± 23% +119.5% 1198 ± 16% interrupts.135:IR-PCI-MSI.2097156-edge.eth1-TxRx-3 13831842 -11.5% 12245495 ± 2% interrupts.CPU0.RES:Rescheduling_interrupts 45259 ± 5% +68.9% 76440 ± 15% interrupts.CPU0.TRM:Thermal_event_interrupts 2977 ± 50% -73.0% 803.75 ± 43% interrupts.CPU1.133:IR-PCI-MSI.2097154-edge.eth1-TxRx-1 45283 ± 5% +68.9% 76474 ± 15% interrupts.CPU1.TRM:Thermal_event_interrupts 13947887 -11.0% 12413074 interrupts.CPU10.RES:Rescheduling_interrupts 45275 ± 5% +68.9% 76468 ± 15% interrupts.CPU10.TRM:Thermal_event_interrupts 45286 ± 5% +68.9% 76468 ± 15% interrupts.CPU11.TRM:Thermal_event_interrupts 13579741 -11.8% 11974820 ± 5% interrupts.CPU12.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76471 ± 15% interrupts.CPU12.TRM:Thermal_event_interrupts 13668073 -11.3% 12119346 interrupts.CPU13.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76474 ± 15% interrupts.CPU13.TRM:Thermal_event_interrupts 13578574 -7.8% 12518854 interrupts.CPU14.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76467 ± 15% interrupts.CPU14.TRM:Thermal_event_interrupts 13689117 -10.1% 12310290 ± 2% interrupts.CPU15.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76473 ± 15% interrupts.CPU15.TRM:Thermal_event_interrupts 13924470 -10.5% 12461613 interrupts.CPU2.RES:Rescheduling_interrupts 45264 ± 5% +68.9% 76467 ± 15% interrupts.CPU2.TRM:Thermal_event_interrupts 546.00 ± 23% +119.5% 1198 ± 16% interrupts.CPU3.135:IR-PCI-MSI.2097156-edge.eth1-TxRx-3 13951884 -10.5% 12480739 interrupts.CPU3.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76469 ± 15% interrupts.CPU3.TRM:Thermal_event_interrupts 45285 ± 5% +68.9% 76471 ± 15% interrupts.CPU4.TRM:Thermal_event_interrupts 14030690 -10.7% 12530992 ± 3% interrupts.CPU5.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76471 ± 15% interrupts.CPU5.TRM:Thermal_event_interrupts 45285 ± 5% +68.9% 76474 ± 15% interrupts.CPU6.TRM:Thermal_event_interrupts 45285 ± 5% +68.9% 76472 ± 15% interrupts.CPU7.TRM:Thermal_event_interrupts 13319036 -10.2% 11966842 ± 2% interrupts.CPU8.RES:Rescheduling_interrupts 45280 ± 5% +68.7% 76405 ± 15% interrupts.CPU8.TRM:Thermal_event_interrupts 13401328 -11.8% 11816711 ± 3% interrupts.CPU9.RES:Rescheduling_interrupts 45279 ± 5% +68.9% 76467 ± 15% interrupts.CPU9.TRM:Thermal_event_interrupts 2.19e+08 -9.8% 1.974e+08 ± 2% interrupts.RES:Rescheduling_interrupts 724498 ± 5% +68.9% 1223435 ± 15% interrupts.TRM:Thermal_event_interrupts 53.44 +4.2% 55.68 perf-stat.i.MPKI 0.12 -0.0 0.10 ± 4% perf-stat.i.cache-miss-rate% 1764490 -13.1% 1533362 ± 3% perf-stat.i.cache-misses 1.767e+09 +3.6% 1.832e+09 perf-stat.i.cache-references 4055785 +6.2% 4306694 perf-stat.i.context-switches 336154 +21.5% 408363 perf-stat.i.cpu-migrations 34528 ± 5% +10.3% 38089 ± 3% perf-stat.i.cycles-between-cache-misses 0.01 ± 6% -0.0 0.01 ± 6% perf-stat.i.dTLB-load-miss-rate% 899577 ± 6% -15.2% 762553 ± 5% perf-stat.i.dTLB-load-misses 0.00 ± 7% -0.0 0.00 ± 4% perf-stat.i.dTLB-store-miss-rate% 22816 ± 7% -13.2% 19813 ± 4% perf-stat.i.dTLB-store-misses 6.044e+09 -1.2% 5.969e+09 perf-stat.i.dTLB-stores 31331199 -3.4% 30272777 perf-stat.i.iTLB-load-misses 1061 +3.1% 1094 perf-stat.i.instructions-per-iTLB-miss 1077 -2.9% 1046 perf-stat.i.minor-faults 73018 -10.9% 65028 ± 2% perf-stat.i.node-loads 122632 ± 2% -9.3% 111181 ± 4% perf-stat.i.node-stores 1077 -2.9% 1046 perf-stat.i.page-faults 53.29 +4.2% 55.53 perf-stat.overall.MPKI 0.10 -0.0 0.08 ± 3% perf-stat.overall.cache-miss-rate% 25264 +15.3% 29120 ± 3% perf-stat.overall.cycles-between-cache-misses 0.01 ± 6% -0.0 0.01 ± 6% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 7% -0.0 0.00 ± 4% perf-stat.overall.dTLB-store-miss-rate% 1058 +2.9% 1089 perf-stat.overall.instructions-per-iTLB-miss 65158 +6.6% 69487 perf-stat.overall.path-length 1761814 -13.1% 1530883 ± 3% perf-stat.ps.cache-misses 1.765e+09 +3.7% 1.829e+09 perf-stat.ps.cache-references 4048770 +6.2% 4299578 perf-stat.ps.context-switches 335565 +21.5% 407697 perf-stat.ps.cpu-migrations 898100 ± 6% -15.2% 761293 ± 5% perf-stat.ps.dTLB-load-misses 22777 ± 7% -13.2% 19781 ± 4% perf-stat.ps.dTLB-store-misses 6.034e+09 -1.2% 5.96e+09 perf-stat.ps.dTLB-stores 31277565 -3.4% 30222833 perf-stat.ps.iTLB-load-misses 1075 -2.9% 1044 perf-stat.ps.minor-faults 72896 -10.9% 64922 ± 2% perf-stat.ps.node-loads 122424 ± 2% -9.3% 111005 ± 4% perf-stat.ps.node-stores 1075 -2.9% 1044 perf-stat.ps.page-faults 39.37 ±100% -36.4 2.98 ±173% perf-profile.calltrace.cycles-pp.start_thread 21.68 ±100% -20.0 1.64 ±173% perf-profile.calltrace.cycles-pp.__GI___libc_write.start_thread 20.14 ±100% -18.6 1.52 ±173% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 20.00 ±100% -18.5 1.51 ±173% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 18.05 ±100% -16.7 1.39 ±173% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 16.75 ±100% -15.5 1.22 ±173% perf-profile.calltrace.cycles-pp.__GI___libc_read.start_thread 16.70 ±100% -15.4 1.26 ±173% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 14.98 ±100% -13.9 1.09 ±173% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 14.82 ±100% -13.7 1.08 ±173% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 13.95 ±100% -12.9 1.03 ±173% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 12.97 ±100% -12.0 0.95 ±173% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read 3.69 ± 6% -1.1 2.58 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common 4.25 ± 6% -1.0 3.29 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 1.54 ± 10% -0.5 1.05 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 1.07 ± 16% -0.5 0.60 ± 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write 1.14 ± 2% -0.3 0.80 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__schedule.schedule.pipe_wait 0.55 ± 4% -0.3 0.26 ±100% perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_read.new_sync_read.vfs_read.ksys_read 1.90 ± 9% -0.2 1.67 ± 7% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.71 ± 7% -0.2 1.48 ± 7% perf-profile.calltrace.cycles-pp.__fget.__fget_light.__fdget_pos.ksys_write.do_syscall_64 0.68 ± 19% +0.2 0.86 perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.67 ± 16% +0.2 0.88 ± 4% perf-profile.calltrace.cycles-pp.update_cfs_group.dequeue_task_fair.__schedule.schedule.pipe_wait 0.29 ±100% +0.4 0.64 ± 4% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read 0.31 ±100% +0.4 0.67 ± 4% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_wait.pipe_read.new_sync_read.vfs_read 0.32 ±100% +0.4 0.68 ± 2% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.29 ±100% +0.4 0.68 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 0.35 ±100% +0.4 0.78 perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule 1.23 ± 19% +0.4 1.68 ± 2% perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 1.29 ± 18% +0.5 1.76 ± 5% perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.00 +0.5 0.53 ± 2% perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__schedule.schedule 0.00 +0.6 0.59 ± 7% perf-profile.calltrace.cycles-pp.__enqueue_entity.put_prev_entity.pick_next_task_fair.__schedule.schedule 8.82 +0.6 9.46 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 0.47 ±100% +0.7 1.14 ± 6% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop 2.21 ± 10% +0.7 2.88 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 9.10 +0.7 9.79 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 9.04 +0.7 9.74 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.71 ±100% +0.9 1.61 ± 3% perf-profile.calltrace.cycles-pp.__switch_to 0.68 ± 99% +0.9 1.60 ± 3% perf-profile.calltrace.cycles-pp.native_write_msr 0.86 ±100% +0.9 1.79 ± 2% perf-profile.calltrace.cycles-pp.__switch_to_asm 6.54 +1.0 7.53 ± 2% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_wait.pipe_read 26.96 +1.0 27.95 perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64 1.42 ± 63% +1.0 2.47 ± 3% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.54 ± 8% +1.2 5.79 ± 4% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 14.99 +1.3 16.27 perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait.pipe_read.new_sync_read 15.35 +1.3 16.66 perf-profile.calltrace.cycles-pp.schedule.pipe_wait.pipe_read.new_sync_read.vfs_read 27.37 +1.3 28.70 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write 17.81 +1.6 19.38 perf-profile.calltrace.cycles-pp.pipe_wait.pipe_read.new_sync_read.vfs_read.ksys_read 3.95 ± 11% +1.8 5.70 ± 6% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common 23.38 +2.0 25.33 ± 2% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write 24.30 +2.0 26.27 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 23.73 +2.0 25.71 ± 2% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write 5.24 ± 10% +2.0 7.25 ± 6% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 3.23 ± 57% +2.7 5.98 ± 4% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.34 ± 57% +2.8 6.18 ± 4% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 39.37 ±100% -36.4 2.98 ±173% perf-profile.children.cycles-pp.start_thread 21.68 ±100% -20.0 1.67 ±173% perf-profile.children.cycles-pp.__GI___libc_write 16.99 ±100% -15.7 1.24 ±173% perf-profile.children.cycles-pp.__GI___libc_read 6.84 ± 10% -2.1 4.71 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 45.35 -1.0 44.33 perf-profile.children.cycles-pp.ksys_write 6.70 ± 5% -1.0 5.70 ± 2% perf-profile.children.cycles-pp._raw_spin_lock 5.90 ± 4% -0.6 5.34 ± 6% perf-profile.children.cycles-pp.security_file_permission 3.35 ± 9% -0.6 2.80 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 42.14 -0.5 41.63 perf-profile.children.cycles-pp.vfs_write 3.41 -0.4 3.01 perf-profile.children.cycles-pp.__fdget_pos 3.30 -0.4 2.91 perf-profile.children.cycles-pp.__fget_light 2.71 ± 2% -0.4 2.32 perf-profile.children.cycles-pp.__fget 3.23 -0.3 2.93 ± 3% perf-profile.children.cycles-pp.mutex_lock 2.76 -0.3 2.47 ± 5% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 2.21 ± 3% -0.3 1.93 ± 10% perf-profile.children.cycles-pp.file_has_perm 3.14 -0.2 2.90 ± 4% perf-profile.children.cycles-pp.copy_page_to_iter 3.03 ± 5% -0.2 2.79 ± 5% perf-profile.children.cycles-pp.selinux_file_permission 2.10 ± 5% -0.2 1.86 ± 2% perf-profile.children.cycles-pp.mutex_unlock 1.50 ± 5% -0.2 1.27 ± 2% perf-profile.children.cycles-pp.fput_many 0.68 ± 17% -0.2 0.48 ± 4% perf-profile.children.cycles-pp.__mutex_lock 1.66 ± 2% -0.2 1.47 ± 4% perf-profile.children.cycles-pp.copyout 1.73 -0.2 1.57 ± 2% perf-profile.children.cycles-pp.update_rq_clock 1.30 -0.1 1.17 ± 6% perf-profile.children.cycles-pp.copyin 0.27 ± 11% -0.1 0.15 ± 9% perf-profile.children.cycles-pp.preempt_schedule_common 1.25 ± 3% -0.1 1.15 ± 3% perf-profile.children.cycles-pp.fsnotify 0.79 -0.1 0.69 ± 7% perf-profile.children.cycles-pp.reschedule_interrupt 0.91 -0.1 0.82 ± 5% perf-profile.children.cycles-pp._cond_resched 0.80 ± 2% -0.1 0.71 ± 3% perf-profile.children.cycles-pp.file_update_time 0.29 ± 10% -0.1 0.24 ± 5% perf-profile.children.cycles-pp.rb_insert_color 0.26 -0.0 0.21 ± 2% perf-profile.children.cycles-pp.native_irq_return_iret 0.07 ± 14% -0.0 0.03 ±100% perf-profile.children.cycles-pp.scheduler_ipi 0.22 ± 6% -0.0 0.18 ± 4% perf-profile.children.cycles-pp.__x64_sys_read 0.25 ± 4% -0.0 0.21 ± 2% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.27 -0.0 0.23 ± 9% perf-profile.children.cycles-pp.timespec64_trunc 0.23 ± 2% -0.0 0.20 ± 5% perf-profile.children.cycles-pp.smp_reschedule_interrupt 0.22 -0.0 0.20 ± 5% perf-profile.children.cycles-pp.__x64_sys_write 0.24 -0.0 0.22 ± 3% perf-profile.children.cycles-pp.__list_add_valid 0.14 -0.0 0.12 ± 10% perf-profile.children.cycles-pp.kill_fasync 0.11 ± 4% +0.0 0.12 ± 3% perf-profile.children.cycles-pp.perf_exclude_event 0.10 +0.0 0.12 ± 8% perf-profile.children.cycles-pp.is_cpu_allowed 0.09 ± 11% +0.0 0.12 ± 7% perf-profile.children.cycles-pp.rcu_note_context_switch 0.07 +0.0 0.10 ± 10% perf-profile.children.cycles-pp.pkg_thermal_notify 0.17 ± 3% +0.0 0.20 ± 9% perf-profile.children.cycles-pp.deactivate_task 0.16 +0.0 0.21 ± 10% perf-profile.children.cycles-pp.update_cfs_rq_h_load 0.08 ± 6% +0.0 0.12 ± 10% perf-profile.children.cycles-pp.intel_thermal_interrupt 0.08 ± 6% +0.0 0.12 ± 6% perf-profile.children.cycles-pp.smp_thermal_interrupt 0.08 +0.1 0.13 ± 9% perf-profile.children.cycles-pp.thermal_interrupt 0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.default_wake_function 0.19 ± 5% +0.1 0.25 ± 5% perf-profile.children.cycles-pp.wakeup_preempt_entity 0.19 ± 5% +0.1 0.27 ± 4% perf-profile.children.cycles-pp.finish_wait 0.89 ± 8% +0.1 0.98 perf-profile.children.cycles-pp.__calc_delta 0.63 ± 4% +0.1 0.74 ± 4% perf-profile.children.cycles-pp.native_sched_clock 0.33 ± 4% +0.1 0.44 ± 5% perf-profile.children.cycles-pp.set_next_buddy 1.23 ± 3% +0.1 1.34 ± 3% perf-profile.children.cycles-pp.check_preempt_wakeup 0.69 ± 4% +0.1 0.81 ± 3% perf-profile.children.cycles-pp.sched_clock 1.52 ± 3% +0.1 1.65 ± 2% perf-profile.children.cycles-pp.set_next_entity 0.73 ± 4% +0.1 0.88 ± 3% perf-profile.children.cycles-pp.sched_clock_cpu 0.57 ± 4% +0.2 0.74 ± 4% perf-profile.children.cycles-pp.account_entity_dequeue 1.21 ± 4% +0.2 1.39 ± 2% perf-profile.children.cycles-pp.__update_load_avg_se 1.99 ± 3% +0.2 2.20 ± 3% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 1.64 ± 6% +0.2 1.85 perf-profile.children.cycles-pp.__switch_to_asm 1.60 ± 4% +0.2 1.85 ± 3% perf-profile.children.cycles-pp.update_cfs_group 3.91 +0.2 4.15 perf-profile.children.cycles-pp.update_curr 1.04 ± 5% +0.3 1.30 ± 4% perf-profile.children.cycles-pp.___perf_sw_event 1.05 ± 9% +0.3 1.33 ± 2% perf-profile.children.cycles-pp.pick_next_entity 1.57 ± 3% +0.3 1.89 perf-profile.children.cycles-pp.native_write_msr 1.01 ± 9% +0.3 1.33 ± 5% perf-profile.children.cycles-pp.put_prev_entity 2.17 ± 7% +0.3 2.51 ± 2% perf-profile.children.cycles-pp.switch_fpu_return 1.93 ± 7% +0.4 2.36 perf-profile.children.cycles-pp.__switch_to 2.99 ± 4% +0.6 3.57 ± 2% perf-profile.children.cycles-pp.reweight_entity 8.89 +0.6 9.50 ± 2% perf-profile.children.cycles-pp.enqueue_task_fair 9.16 +0.7 9.82 perf-profile.children.cycles-pp.ttwu_do_activate 9.09 +0.7 9.76 ± 2% perf-profile.children.cycles-pp.activate_task 5.18 ± 6% +0.9 6.04 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair 6.71 +0.9 7.63 ± 2% perf-profile.children.cycles-pp.dequeue_task_fair 28.01 +1.4 29.36 ± 2% perf-profile.children.cycles-pp.__wake_up_common_lock 4.91 ± 7% +1.4 6.28 ± 4% perf-profile.children.cycles-pp.exit_to_usermode_loop 18.19 +1.4 19.56 perf-profile.children.cycles-pp.pipe_wait 4.03 ± 11% +1.7 5.75 ± 6% perf-profile.children.cycles-pp.select_idle_sibling 23.86 ± 2% +1.8 25.71 ± 2% perf-profile.children.cycles-pp.try_to_wake_up 24.44 +1.9 26.36 ± 2% perf-profile.children.cycles-pp.__wake_up_common 23.84 +1.9 25.77 ± 2% perf-profile.children.cycles-pp.autoremove_wake_function 5.29 ± 10% +2.0 7.29 ± 6% perf-profile.children.cycles-pp.select_task_rq_fair 20.27 ± 2% +2.3 22.53 ± 2% perf-profile.children.cycles-pp.__schedule 20.36 ± 2% +2.5 22.82 ± 2% perf-profile.children.cycles-pp.schedule 6.82 ± 10% -2.1 4.69 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 2.67 ± 2% -0.4 2.29 perf-profile.self.cycles-pp.__fget 2.72 -0.3 2.44 ± 5% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 1.22 -0.2 0.98 ± 2% perf-profile.self.cycles-pp.update_rq_clock 1.36 ± 3% -0.2 1.13 ± 6% perf-profile.self.cycles-pp.pipe_write 1.48 ± 5% -0.2 1.24 ± 3% perf-profile.self.cycles-pp.fput_many 2.04 ± 5% -0.2 1.81 ± 2% perf-profile.self.cycles-pp.mutex_unlock 1.96 -0.2 1.73 ± 2% perf-profile.self.cycles-pp.mutex_lock 0.66 ± 12% -0.1 0.53 ± 10% perf-profile.self.cycles-pp.file_has_perm 0.38 ± 10% -0.1 0.28 ± 9% perf-profile.self.cycles-pp.ksys_read 0.34 ± 7% -0.1 0.25 ± 7% perf-profile.self.cycles-pp.__mutex_lock 1.21 ± 3% -0.1 1.11 ± 2% perf-profile.self.cycles-pp.fsnotify 1.00 -0.1 0.94 ± 5% perf-profile.self.cycles-pp.__might_sleep 0.60 ± 4% -0.1 0.55 ± 8% perf-profile.self.cycles-pp.vfs_write 0.29 ± 10% -0.1 0.23 ± 3% perf-profile.self.cycles-pp.rb_insert_color 0.32 ± 6% -0.0 0.28 ± 6% perf-profile.self.cycles-pp.ksys_write 0.26 -0.0 0.21 ± 2% perf-profile.self.cycles-pp.native_irq_return_iret 0.07 ± 14% -0.0 0.03 ±100% perf-profile.self.cycles-pp.scheduler_ipi 0.36 ± 5% -0.0 0.32 ± 8% perf-profile.self.cycles-pp.__sb_start_write 0.29 -0.0 0.24 ± 4% perf-profile.self.cycles-pp.check_preempt_curr 0.22 ± 6% -0.0 0.17 ± 6% perf-profile.self.cycles-pp.__x64_sys_read 0.25 -0.0 0.21 ± 8% perf-profile.self.cycles-pp.timespec64_trunc 0.22 ± 4% -0.0 0.19 ± 11% perf-profile.self.cycles-pp.wake_q_add 0.21 ± 2% -0.0 0.18 ± 6% perf-profile.self.cycles-pp.__x64_sys_write 0.21 ± 2% -0.0 0.20 perf-profile.self.cycles-pp.__list_add_valid 0.11 ± 4% -0.0 0.10 ± 7% perf-profile.self.cycles-pp.kill_fasync 0.06 ± 9% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.rcu_note_context_switch 0.15 +0.0 0.18 ± 6% perf-profile.self.cycles-pp.autoremove_wake_function 0.16 +0.0 0.19 ± 9% perf-profile.self.cycles-pp.deactivate_task 0.26 +0.0 0.29 ± 5% perf-profile.self.cycles-pp.rcu_all_qs 0.33 ± 3% +0.0 0.37 ± 4% perf-profile.self.cycles-pp.set_next_entity 0.18 ± 5% +0.0 0.22 ± 6% perf-profile.self.cycles-pp.wakeup_preempt_entity 0.03 ±100% +0.0 0.07 perf-profile.self.cycles-pp.check_cfs_rq_runtime 0.16 ± 6% +0.0 0.20 ± 6% perf-profile.self.cycles-pp.exit_to_usermode_loop 0.16 +0.0 0.20 ± 12% perf-profile.self.cycles-pp.update_cfs_rq_h_load 0.20 ± 2% +0.1 0.25 ± 2% perf-profile.self.cycles-pp.activate_task 0.41 ± 6% +0.1 0.46 perf-profile.self.cycles-pp.schedule 1.96 +0.1 2.02 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.15 ± 6% +0.1 0.22 perf-profile.self.cycles-pp.finish_wait 0.41 ± 4% +0.1 0.48 ± 6% perf-profile.self.cycles-pp.account_entity_enqueue 0.61 ± 4% +0.1 0.71 ± 3% perf-profile.self.cycles-pp.native_sched_clock 0.83 ± 2% +0.1 0.93 perf-profile.self.cycles-pp.dequeue_task_fair 0.29 ± 3% +0.1 0.40 ± 5% perf-profile.self.cycles-pp.set_next_buddy 1.02 +0.1 1.15 ± 7% perf-profile.self.cycles-pp.enqueue_task_fair 0.46 ± 12% +0.2 0.61 ± 3% perf-profile.self.cycles-pp.pick_next_entity 1.17 ± 4% +0.2 1.35 ± 2% perf-profile.self.cycles-pp.__update_load_avg_se 0.42 ± 8% +0.2 0.61 ± 5% perf-profile.self.cycles-pp.account_entity_dequeue 1.64 ± 6% +0.2 1.85 perf-profile.self.cycles-pp.__switch_to_asm 1.23 ± 4% +0.2 1.46 ± 3% perf-profile.self.cycles-pp.reweight_entity 1.07 ± 5% +0.2 1.31 ± 4% perf-profile.self.cycles-pp.select_task_rq_fair 0.90 ± 5% +0.2 1.14 ± 5% perf-profile.self.cycles-pp.___perf_sw_event 1.57 ± 4% +0.2 1.82 ± 2% perf-profile.self.cycles-pp.update_cfs_group 1.17 ± 3% +0.3 1.42 ± 7% perf-profile.self.cycles-pp.update_load_avg 1.56 ± 4% +0.3 1.88 perf-profile.self.cycles-pp.native_write_msr 2.16 ± 7% +0.3 2.50 ± 2% perf-profile.self.cycles-pp.switch_fpu_return 1.84 ± 8% +0.4 2.25 ± 2% perf-profile.self.cycles-pp.__switch_to 1.22 +0.5 1.73 ± 4% perf-profile.self.cycles-pp._raw_spin_lock 0.92 ± 11% +1.3 2.27 ± 6% perf-profile.self.cycles-pp.select_idle_sibling Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

1 year, 7 months

4
6
0 / 0

[ptrace] 201766a20e: kernel_selftests.seccomp.make_fail

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: 201766a20e30f982ccfe36bebfad9602c3ff574a ("ptrace: add PTRACE_GET_SYSCALL_INFO request") https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux.git master in testcase: kernel_selftests with following parameters: group: kselftests-02 test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel. test-url: https://www.kernel.org/doc/Documentation/kselftest.txt on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> 2019-07-26 16:52:03 make run_tests -C seccomp make: Entering directory '/usr/src/perf_selftests-x86_64-rhel-7.6-201766a20e30f982ccfe36bebfad9602c3ff574a/tools/testing/selftests/seccomp' gcc -Wl,-no-as-needed -Wall seccomp_bpf.c -lpthread -o seccomp_bpf In file included from seccomp_bpf.c:51:0: seccomp_bpf.c: In function ‘tracer_ptrace’: seccomp_bpf.c:1787:20: error: ‘PTRACE_EVENTMSG_SYSCALL_ENTRY’ undeclared (first use in this function) EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^ ../kselftest_harness.h:608:13: note: in definition of macro ‘__EXPECT’ __typeof__(_expected) __exp = (_expected); \ ^~~~~~~~~ seccomp_bpf.c:1787:2: note: in expansion of macro ‘EXPECT_EQ’ EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^~~~~~~~~ seccomp_bpf.c:1787:20: note: each undeclared identifier is reported only once for each function it appears in EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^ ../kselftest_harness.h:608:13: note: in definition of macro ‘__EXPECT’ __typeof__(_expected) __exp = (_expected); \ ^~~~~~~~~ seccomp_bpf.c:1787:2: note: in expansion of macro ‘EXPECT_EQ’ EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^~~~~~~~~ seccomp_bpf.c:1788:6: error: ‘PTRACE_EVENTMSG_SYSCALL_EXIT’ undeclared (first use in this function) : PTRACE_EVENTMSG_SYSCALL_EXIT, msg); ^ ../kselftest_harness.h:608:13: note: in definition of macro ‘__EXPECT’ __typeof__(_expected) __exp = (_expected); \ ^~~~~~~~~ seccomp_bpf.c:1787:2: note: in expansion of macro ‘EXPECT_EQ’ EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^~~~~~~~~ Makefile:12: recipe for target 'seccomp_bpf' failed make: *** [seccomp_bpf] Error 1 make: Leaving directory '/usr/src/perf_selftests-x86_64-rhel-7.6-201766a20e30f982ccfe36bebfad9602c3ff574a/tools/testing/selftests/seccomp' To reproduce: # build kernel cd linux cp config-5.2.0-10889-g201766a20e30f9 .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

1 year, 9 months

3
3
0 / 0

[ext4] b1b4705d54: filebench.sum_bytes_mb/s -20.2% regression

by kernel test robot

Greeting, FYI, we noticed a -20.2% regression of filebench.sum_bytes_mb/s due to commit: commit: b1b4705d54abedfd69dcdf42779c521aa1e0fbd3 ("ext4: introduce direct I/O read using iomap infrastructure") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: filebench on test machine: 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory with following parameters: disk: 1HDD fs: ext4 test: fivestreamreaddirect.f cpufreq_governor: performance ucode: 0x27 If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/1HDD/ext4/x86_64-rhel-7.6/debian-x86_64-2019-11-14.cgz/lkp-hsw-d01/fivestreamreaddirect.f/filebench/0x27 commit: 09edf4d381 ("ext4: introduce new callback for IOMAP_REPORT") b1b4705d54 ("ext4: introduce direct I/O read using iomap infrastructure") 09edf4d381957b14 b1b4705d54abedfd69dcdf42779 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x :4 25% 1:4 dmesg.WARNING:stack_recursion :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:da :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:db :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:dc :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:dd :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:de :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:df :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ea :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:eb :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ec :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ed :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ee :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ef :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fa :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fb :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fc :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fd :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fe :4 25% 1:4 kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ff :4 25% 1:4 kmsg.file:libnss_files-#so_fault:filemap_fault_mmap:generic_file_mmap_readpage:simple_readpage :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a07fffffa0ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a17fffffa1ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a27fffffa2ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a37fffffa3ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a47fffffa4ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a57fffffa5ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a67fffffa6ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a77fffffa7ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a7fffff8aff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a7fffff9aff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a87fffffa8ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#a97fffffa9ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#aa7fffffaaff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ab7fffffabff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ac7fffffacff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ad7fffffadff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ae7fffffaeff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#af7fffffafff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b07fffffb0ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b17fffffb1ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b27fffffb2ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b37fffffb3ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b47fffffb4ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b57fffffb5ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b67fffffb6ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b77fffffb7ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b7fffff8bff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b7fffff9bff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b87fffffb8ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#b97fffffb9ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ba7fffffbaff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#bb7fffffbbff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#bc7fffffbcff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#bd7fffffbdff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#be7fffffbeff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#bf7fffffbfff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c07fffffc0ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c17fffffc1ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c27fffffc2ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c37fffffc3ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c47fffffc4ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c57fffffc5ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c67fffffc6ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c77fffffc7ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c7fffff8cff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c7fffff9cff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c87fffffc8ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#c97fffffc9ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ca7fffffcaff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#cb7fffffcbff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#cc7fffffccff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#cd7fffffcdff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ce7fffffceff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#cf7fffffcfff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d07fffffd0ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d17fffffd1ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d27fffffd2ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d37fffffd3ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d47fffffd4ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d57fffffd5ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d67fffffd6ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d77fffffd7ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d7fffff8dff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d7fffff9dff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d87fffffd8ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#d97fffffd9ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#da7fffffdaff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#db7fffffdbff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#dc7fffffdcff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#dd7fffffddff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#de7fffffdeff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#df7fffffdfff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e07fffffe0ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e17fffffe1ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e27fffffe2ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e37fffffe3ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e47fffffe4ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e57fffffe5ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e67fffffe6ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e77fffffe7ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e7fffff8eff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e7fffff9eff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e87fffffe8ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#e97fffffe9ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ea7fffffeaff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#eb7fffffebff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ec7fffffecff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ed7fffffedff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ee7fffffeeff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ef7fffffefff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f07ffffff0ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f17ffffff1ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f27ffffff2ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f37ffffff3ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f47ffffff4ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f57ffffff5ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f67ffffff6ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f77ffffff7ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f7fffff8fff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f7fffff9fff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f87ffffff8ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#f97ffffff9ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fa7ffffffaff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fb7ffffffbff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fc7ffffffcff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fd7ffffffdff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fe7ffffffeff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#ff7fffffffff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff80ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff81ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff82ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff83ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff84ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff85ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff86ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff87ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff88ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff89ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff90ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff91ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff92ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff93ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff94ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff95ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff96ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff97ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff98ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffff99ff :4 25% 1:4 kmsg.swap_info_get:Bad_swap_file_entry#fffffffffff %stddev %change %stddev \ | \ 58.30 ± 2% -20.2% 46.53 filebench.sum_bytes_mb/s 3503 ± 2% -20.2% 2796 filebench.sum_operations 58.39 ± 2% -20.2% 46.61 filebench.sum_operations/s 58.33 ± 2% -19.9% 46.75 filebench.sum_reads/s 84.34 ± 4% +27.1% 107.17 filebench.sum_time_ms/op 7174922 ± 2% -20.2% 5727328 filebench.time.file_system_inputs 10963 ± 2% -8.5% 10035 filebench.time.voluntary_context_switches 58.37 ± 3% -9.0 49.38 ± 3% mpstat.cpu.all.idle% 41.08 ± 5% +8.2 49.24 ± 2% mpstat.cpu.all.iowait% 3061 ± 6% +6.1% 3246 ± 4% turbostat.Bzy_MHz 5.48 ± 97% +260.7% 19.78 ± 30% turbostat.Pkg%pc2 27.33 ± 32% -52.0% 13.12 ± 19% sched_debug.cpu.nr_uninterruptible.max -67.67 -60.8% -26.50 sched_debug.cpu.nr_uninterruptible.min 28.48 ± 30% -57.9% 11.98 ± 23% sched_debug.cpu.nr_uninterruptible.stddev 8210 ± 44% +76.1% 14454 ± 6% softirqs.CPU0.RCU 8200 ± 40% +70.2% 13954 ± 5% softirqs.CPU2.RCU 9772 ± 36% +59.6% 15597 ± 15% softirqs.CPU3.RCU 108.33 ± 27% +113.5% 231.25 ± 26% interrupts.CPU1.RES:Rescheduling_interrupts 1032 -23.9% 786.00 ± 30% interrupts.CPU5.CAL:Function_call_interrupts 229.33 ± 46% +50.1% 344.25 ± 31% interrupts.CPU5.NMI:Non-maskable_interrupts 229.33 ± 46% +50.1% 344.25 ± 31% interrupts.CPU5.PMI:Performance_monitoring_interrupts 457.00 ± 61% -74.2% 117.75 ± 41% interrupts.CPU5.RES:Rescheduling_interrupts 53.33 ± 60% +206.6% 163.50 ± 42% interrupts.CPU7.RES:Rescheduling_interrupts 58.99 ± 3% -14.9% 50.18 ± 3% iostat.cpu.idle 40.36 ± 5% +19.8% 48.35 ± 2% iostat.cpu.iowait 0.18 +386.3% 0.87 ±137% iostat.cpu.user 33012 ± 2% -19.7% 26520 iostat.sda.rkB/s 40.60 ± 6% +16.2% 47.18 ± 4% iostat.sda.w/s 43038 ± 4% +17.1% 50399 ± 2% iostat.sda.wkB/s 58.67 ± 3% -15.6% 49.50 ± 3% vmstat.cpu.id 40.00 ± 5% +19.4% 47.75 ± 2% vmstat.cpu.wa 33010 ± 2% -19.7% 26518 vmstat.io.bi 42837 ± 4% +17.2% 50200 ± 2% vmstat.io.bo 5519500 -15.5% 4664152 vmstat.memory.cache 2183603 +39.2% 3039439 ± 2% vmstat.memory.free 1065500 -19.2% 861375 slabinfo.buffer_head.active_objs 27320 -19.1% 22093 slabinfo.buffer_head.active_slabs 1065521 -19.1% 861671 slabinfo.buffer_head.num_objs 27320 -19.1% 22093 slabinfo.buffer_head.num_slabs 1306 +18.4% 1546 ± 7% slabinfo.kmalloc-512.active_objs 1306 +18.4% 1546 ± 7% slabinfo.kmalloc-512.num_objs 28277 -9.3% 25651 slabinfo.radix_tree_node.active_objs 5388173 -15.3% 4561843 meminfo.Cached 989375 ± 2% -22.6% 765315 ± 3% meminfo.Dirty 4310025 -19.2% 3483318 meminfo.Inactive 4261041 -19.4% 3434223 meminfo.Inactive(file) 162676 -13.7% 140448 meminfo.KReclaimable 2151883 ± 2% +39.5% 3000927 ± 2% meminfo.MemFree 5902816 -14.4% 5053772 meminfo.Memused 162676 -13.7% 140448 meminfo.SReclaimable 197794 -11.3% 175489 meminfo.Slab 80314276 ± 5% +6.5% 85566891 ± 3% perf-stat.i.branch-instructions 4810221 ± 5% +7.5% 5172535 perf-stat.i.cache-references 7.022e+08 ± 3% +6.9% 7.509e+08 ± 2% perf-stat.i.cpu-cycles 3.891e+08 ± 5% +6.3% 4.135e+08 ± 3% perf-stat.i.instructions 195094 ± 2% +10.8% 216142 perf-stat.i.node-loads 79563934 ± 5% +6.5% 84757473 ± 3% perf-stat.ps.branch-instructions 4765201 ± 5% +7.5% 5123552 perf-stat.ps.cache-references 6.956e+08 ± 3% +6.9% 7.438e+08 ± 2% perf-stat.ps.cpu-cycles 3.855e+08 ± 5% +6.3% 4.096e+08 ± 3% perf-stat.ps.instructions 193276 ± 2% +10.8% 214098 perf-stat.ps.node-loads 4.114e+10 ± 5% +5.3% 4.333e+10 ± 3% perf-stat.total.instructions 247900 ± 3% -22.9% 191187 ± 3% proc-vmstat.nr_dirty 1349353 -15.5% 1140545 proc-vmstat.nr_file_pages 535810 ± 2% +40.1% 750444 ± 2% proc-vmstat.nr_free_pages 1067252 -19.6% 858356 proc-vmstat.nr_inactive_file 40723 -13.8% 35112 proc-vmstat.nr_slab_reclaimable 1110681 ± 2% +17.1% 1300126 proc-vmstat.nr_written 1067251 -19.6% 858384 proc-vmstat.nr_zone_inactive_file 249604 ± 3% -22.7% 193058 ± 3% proc-vmstat.nr_zone_write_pending 660225 -3.8% 635276 proc-vmstat.pgalloc_dma32 871298 +2.7% 894930 proc-vmstat.pgalloc_normal 3587465 ± 2% -20.2% 2863670 proc-vmstat.pgpgin 4645372 ± 4% +16.6% 5414779 proc-vmstat.pgpgout 13.16 ± 14% -6.6 6.58 ± 52% perf-profile.calltrace.cycles-pp.unlinkat 13.06 ± 15% -6.6 6.49 ± 53% perf-profile.calltrace.cycles-pp.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat 13.06 ± 15% -6.6 6.49 ± 53% perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat 13.06 ± 15% -6.6 6.49 ± 53% perf-profile.calltrace.cycles-pp.ext4_evict_inode.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.06 ± 15% -6.6 6.50 ± 52% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlinkat 13.06 ± 15% -6.6 6.50 ± 52% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat 13.02 ± 14% -6.5 6.48 ± 52% perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.ext4_evict_inode.evict.do_unlinkat.do_syscall_64 10.39 ± 36% -4.6 5.78 ± 8% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel 11.21 ± 33% -4.6 6.64 ± 7% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64 11.21 ± 33% -4.6 6.65 ± 7% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64 11.21 ± 33% -4.6 6.65 ± 7% perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64 5.71 ± 17% -2.8 2.95 ± 54% perf-profile.calltrace.cycles-pp.truncate_cleanup_page.truncate_inode_pages_range.ext4_evict_inode.evict.do_unlinkat 3.68 ± 14% -2.0 1.69 ± 50% perf-profile.calltrace.cycles-pp.release_pages.__pagevec_release.truncate_inode_pages_range.ext4_evict_inode.evict 3.70 ± 14% -2.0 1.74 ± 48% perf-profile.calltrace.cycles-pp.__pagevec_release.truncate_inode_pages_range.ext4_evict_inode.evict.do_unlinkat 2.90 ± 23% -1.6 1.32 ± 56% perf-profile.calltrace.cycles-pp.try_to_free_buffers.truncate_cleanup_page.truncate_inode_pages_range.ext4_evict_inode.evict 2.42 ± 20% -1.4 1.01 ± 75% perf-profile.calltrace.cycles-pp.delete_from_page_cache_batch.truncate_inode_pages_range.ext4_evict_inode.evict.do_unlinkat 2.02 ± 17% -1.3 0.68 ±104% perf-profile.calltrace.cycles-pp.free_unref_page_list.release_pages.__pagevec_release.truncate_inode_pages_range.ext4_evict_inode 1.33 ± 28% -0.9 0.45 ±100% perf-profile.calltrace.cycles-pp.free_buffer_head.try_to_free_buffers.truncate_cleanup_page.truncate_inode_pages_range.ext4_evict_inode 1.21 ± 22% -0.8 0.40 ±106% perf-profile.calltrace.cycles-pp.xas_store.delete_from_page_cache_batch.truncate_inode_pages_range.ext4_evict_inode.evict 1.17 ± 3% +0.4 1.57 ± 13% perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle 0.35 ±141% +0.7 1.07 ± 37% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 3.71 ± 34% +1.6 5.34 ± 14% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 64.92 ± 5% +9.2 74.17 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 63.80 ± 4% +9.4 73.19 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 69.95 ± 5% +10.4 80.37 ± 5% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 70.02 ± 5% +10.4 80.46 ± 5% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 70.02 ± 5% +10.4 80.46 ± 5% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 13.16 ± 14% -6.6 6.58 ± 52% perf-profile.children.cycles-pp.unlinkat 13.06 ± 15% -6.6 6.49 ± 53% perf-profile.children.cycles-pp.do_unlinkat 13.06 ± 15% -6.6 6.49 ± 53% perf-profile.children.cycles-pp.ext4_evict_inode 13.03 ± 14% -6.5 6.49 ± 53% perf-profile.children.cycles-pp.truncate_inode_pages_range 13.07 ± 15% -6.4 6.64 ± 53% perf-profile.children.cycles-pp.evict 11.21 ± 33% -4.6 6.65 ± 7% perf-profile.children.cycles-pp.start_kernel 5.71 ± 17% -2.7 2.99 ± 54% perf-profile.children.cycles-pp.truncate_cleanup_page 3.74 ± 13% -1.9 1.85 ± 49% perf-profile.children.cycles-pp.__pagevec_release 3.75 ± 12% -1.9 1.88 ± 49% perf-profile.children.cycles-pp.release_pages 2.91 ± 22% -1.5 1.40 ± 53% perf-profile.children.cycles-pp.try_to_free_buffers 2.43 ± 19% -1.3 1.12 ± 57% perf-profile.children.cycles-pp.delete_from_page_cache_batch 2.03 ± 17% -1.1 0.95 ± 58% perf-profile.children.cycles-pp.free_unref_page_list 1.34 ± 27% -0.7 0.65 ± 45% perf-profile.children.cycles-pp.free_buffer_head 1.23 ± 20% -0.6 0.64 ± 50% perf-profile.children.cycles-pp.xas_store 0.78 ± 15% -0.4 0.36 ± 52% perf-profile.children.cycles-pp.jbd2_journal_try_to_free_buffers 0.73 ± 16% -0.4 0.33 ± 52% perf-profile.children.cycles-pp.jbd2_journal_grab_journal_head 0.50 ± 39% -0.4 0.15 ± 53% perf-profile.children.cycles-pp.unaccount_page_cache_page 0.42 ± 24% -0.3 0.14 ± 93% perf-profile.children.cycles-pp.mem_cgroup_uncharge_list 0.76 ± 19% -0.2 0.53 ± 6% perf-profile.children.cycles-pp.lapic_next_deadline 0.32 ± 22% -0.2 0.14 ± 69% perf-profile.children.cycles-pp.free_unref_page_commit 0.25 ± 25% -0.2 0.08 ±102% perf-profile.children.cycles-pp.uncharge_page 0.19 ± 47% -0.1 0.07 ± 68% perf-profile.children.cycles-pp.__might_sleep 0.16 ± 28% -0.1 0.05 ±112% perf-profile.children.cycles-pp.__libc_fork 0.15 ± 22% -0.1 0.04 ±102% perf-profile.children.cycles-pp.xas_find 0.13 ± 12% -0.1 0.08 ± 34% perf-profile.children.cycles-pp.leave_mm 0.04 ± 71% +0.0 0.07 ± 14% perf-profile.children.cycles-pp.irq_work_needs_cpu 0.02 ±141% +0.1 0.09 ± 29% perf-profile.children.cycles-pp.__msecs_to_jiffies 0.04 ± 71% +0.1 0.12 ± 10% perf-profile.children.cycles-pp.cpuidle_not_available 0.09 ± 75% +0.1 0.18 ± 23% perf-profile.children.cycles-pp.__schedule 0.00 +0.1 0.10 ± 40% perf-profile.children.cycles-pp.link_path_walk 0.09 ± 41% +0.1 0.20 ± 19% perf-profile.children.cycles-pp.pm_qos_read_value 0.04 ± 71% +0.1 0.14 ± 52% perf-profile.children.cycles-pp.raise_softirq 0.04 ±141% +0.1 0.15 ± 33% perf-profile.children.cycles-pp.__test_set_page_writeback 0.00 +0.1 0.12 ± 17% perf-profile.children.cycles-pp.__delete_from_page_cache 0.05 ±141% +0.1 0.17 ± 60% perf-profile.children.cycles-pp.exit_mmap 0.19 ± 54% +0.1 0.32 ± 21% perf-profile.children.cycles-pp.hrtimer_get_next_event 0.05 ±141% +0.1 0.18 ± 64% perf-profile.children.cycles-pp.mmput 0.08 ± 30% +0.2 0.24 ± 33% perf-profile.children.cycles-pp.run_local_timers 0.05 ±141% +0.2 0.21 ± 41% perf-profile.children.cycles-pp.ext4_bio_write_page 0.14 ± 32% +0.2 0.30 ± 46% perf-profile.children.cycles-pp.update_rq_clock 0.15 ± 71% +0.2 0.33 ± 17% perf-profile.children.cycles-pp.rcu_idle_exit 0.38 ± 29% +0.2 0.56 ± 16% perf-profile.children.cycles-pp.sched_clock_cpu 0.14 ± 78% +0.2 0.33 ± 41% perf-profile.children.cycles-pp.mpage_map_and_submit_buffers 0.00 +0.2 0.20 ± 52% perf-profile.children.cycles-pp.invalidate_inode_pages2_range 0.07 ± 97% +0.2 0.30 ± 58% perf-profile.children.cycles-pp.__x64_sys_exit_group 0.07 ±141% +0.2 0.31 ± 37% perf-profile.children.cycles-pp.mpage_submit_page 0.38 ± 21% +0.3 0.64 ± 14% perf-profile.children.cycles-pp.cpuidle_governor_latency_req 0.08 ±101% +0.3 0.34 ± 51% perf-profile.children.cycles-pp.do_group_exit 0.08 ±101% +0.3 0.34 ± 51% perf-profile.children.cycles-pp.do_exit 0.23 ± 29% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.do_writepages 0.23 ± 29% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.ext4_writepages 0.22 ± 35% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.wb_workfn 0.22 ± 35% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.wb_writeback 0.22 ± 35% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.__writeback_inodes_wb 0.22 ± 35% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.writeback_sb_inodes 0.22 ± 35% +0.3 0.50 ± 34% perf-profile.children.cycles-pp.__writeback_single_inode 0.00 +0.3 0.32 ± 49% perf-profile.children.cycles-pp.ext4_file_read_iter 0.00 +0.3 0.32 ± 49% perf-profile.children.cycles-pp.iomap_dio_rw 1.30 ± 4% +0.4 1.72 ± 14% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.74 ± 41% +0.5 1.25 ± 33% perf-profile.children.cycles-pp.scheduler_tick 75.40 +4.7 80.05 ± 4% perf-profile.children.cycles-pp.cpuidle_enter 75.35 +4.7 80.02 ± 4% perf-profile.children.cycles-pp.cpuidle_enter_state 81.28 +5.9 87.18 ± 4% perf-profile.children.cycles-pp.do_idle 70.02 ± 5% +10.4 80.46 ± 5% perf-profile.children.cycles-pp.start_secondary 1.16 ± 22% -0.6 0.56 ± 63% perf-profile.self.cycles-pp.free_pcppages_bulk 0.73 ± 16% -0.4 0.33 ± 52% perf-profile.self.cycles-pp.jbd2_journal_grab_journal_head 0.46 ± 21% -0.3 0.16 ± 36% perf-profile.self.cycles-pp.xas_store 0.70 ± 5% -0.3 0.45 ± 38% perf-profile.self.cycles-pp.release_pages 0.29 ± 17% -0.2 0.07 ± 66% perf-profile.self.cycles-pp.free_unref_page_commit 0.32 ± 26% -0.2 0.13 ± 86% perf-profile.self.cycles-pp.workingset_update_node 0.18 ± 14% -0.1 0.06 ±100% perf-profile.self.cycles-pp.tick_nohz_tick_stopped 0.11 -0.1 0.05 ± 60% perf-profile.self.cycles-pp.leave_mm 0.04 ± 71% +0.0 0.07 ± 14% perf-profile.self.cycles-pp.irq_work_needs_cpu 0.00 +0.1 0.07 ± 14% perf-profile.self.cycles-pp.__msecs_to_jiffies 0.04 ± 71% +0.1 0.11 ± 26% perf-profile.self.cycles-pp.cpuidle_not_available 0.08 ± 81% +0.1 0.18 ± 23% perf-profile.self.cycles-pp.rcu_idle_exit 0.09 ± 41% +0.1 0.20 ± 19% perf-profile.self.cycles-pp.pm_qos_read_value 0.04 ± 71% +0.1 0.14 ± 52% perf-profile.self.cycles-pp.raise_softirq 0.04 ±141% +0.1 0.17 ± 41% perf-profile.self.cycles-pp.calc_global_load_tick 0.16 ± 12% +0.2 0.32 ± 27% perf-profile.self.cycles-pp.run_timer_softirq 0.08 ± 81% +0.2 0.26 ± 46% perf-profile.self.cycles-pp.update_rq_clock 0.17 ± 49% +0.2 0.36 ± 22% perf-profile.self.cycles-pp.load_balance 0.26 ± 8% +0.2 0.47 ± 40% perf-profile.self.cycles-pp.hrtimer_interrupt filebench.time.file_system_inputs 7.6e+06 +-+---------------------------------------------------------------+ 7.4e+06 +-++ .+ +.+ | |: ++ + +.+.+.+.++. .+ : : +.+.+. +.| 7.2e+06 +-+ : : + + .++ + : : .+ + + + | 7e+06 +-+ : : + + + +: +.+ + :+ + | 6.8e+06 +-+ + + + + +.+ : | 6.6e+06 +-+ +: | | + | 6.4e+06 +-+ | 6.2e+06 +-+ | 6e+06 +-+ | 5.8e+06 O-+ O O O O | | O OO O O O O O OO O O O OO O O OO O O O O | 5.6e+06 +-+ O O | 5.4e+06 +-+---------------------------------------------------------------+ filebench.sum_operations 3700 +-+------------------------------------------------------------------+ 3600 +-++ .+. +.+ | |: +. + +.+.+.+.++. .+ : : +.+.+. +.| 3500 +-+ + : + + .+.+ + : : : + + : | 3400 +-+ :: + + :+ : :.+.+ + + + : | | + + + + + +.+ + | 3300 +-+ :+ | 3200 +-+ + | 3100 +-+ | | | 3000 +-+ | 2900 +-+ | O O O O O O O O | 2800 +-O O OO O O O O O O O O O O O OO O O O O | 2700 +-+---------------------------------------------O--------------------+ filebench.sum_operations_s 62 +-+--------------------------------------------------------------------+ | + .+ .+ | 60 +-++ + +.+.+.+.+. .+ + : +. .+. +.| 58 +-+ +.+ : +.+ + .+.+ + : : + + + + : | |: + : + + :+ : :.+.+ + :+ : | 56 +-+ + + + + + +.+ + | 54 +-+ + + | | + | 52 +-+ | 50 +-+ | | | 48 +-+ O | 46 O-O O O O O O O O O O O O O O O OO O O O O O O O O O | | O O | 44 +-+--------------------------------------------------------------------+ filebench.sum_reads_s 62 +-+--------------------------------------------------------------------+ | + | 60 +-++ .++. .+. +.+ +. .+. .| 58 +-+ +.+ + +.+ +.+.+.+. .+.+ + : : + + + + + | |: + + + + :+ : : .+ + :+ + | 56 +-+ + + + +.+ + +.+ + | | : : | 54 +-+ : : | | + | 52 +-+ | 50 +-+ | | | 48 +-+ | O O O O O O O O O O | 46 +-O-O-O-O---OO-O-----O---O-O-O-O-O-OO-O-O-O-O-----O--------------------+ filebench.sum_bytes_mb_s 62 +-+--------------------------------------------------------------------+ | + | 60 +-++ +.++.+.+.+.+. +.+ +. .+ +.| 58 +-+ +.+ : +.+.+. .+. : : + + + : | |: + : + +. +. : : .+.+ +. +. : | 56 +-+ + + + + + +.+ + | 54 +-+ + + | | + | 52 +-+ | 50 +-+ | | | 48 +-+ O | 46 O-O O O O O O O O O O O O O O O O O O O O O O O O | | O O O O | 44 +-+--------------------------------------------------------------------+ filebench.sum_time_ms_op 110 +-+-O------O----------------------O--------------O--------------------+ | O O OO O O O O O OO O O O O O O O OO O O O O | 105 O-+ O | 100 +-+ | | | 95 +-+ | | .+. | 90 +-+ + .+. .+ + .+. .++ + | |: : : + .+.+.+ + + : +.+. + + + | 85 +-+ +. : +. +.+ :+ +. : +.+. + +.| 80 +-++ + +. .+. + : : + + | | + + + : : | 75 +-+ : : | | : | 70 +-+-------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 years, 2 months

5
10
0 / 0

[f2fs] fe1897eaa6: aim7.jobs-per-min -60.9% regression

by kernel test robot

Greeting, FYI, we noticed a -60.9% regression of aim7.jobs-per-min due to commit: commit: fe1897eaa6646f5a64a4cee0e6473ed9887d324b ("f2fs: fix to update time in lazytime mode") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: aim7 on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory with following parameters: disk: 4BRD_12G md: RAID1 fs: f2fs test: sync_disk_rw load: 200 cpufreq_governor: performance ucode: 0x42e test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system. test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/4BRD_12G/f2fs/x86_64-rhel-7.6/200/RAID1/debian-x86_64-2019-11-14.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e commit: b145b0eb20 (" ARM and x86 bugfixes of all kinds. The most visible one is that migrating") fe1897eaa6 ("f2fs: fix to update time in lazytime mode") b145b0eb2031a620 fe1897eaa6646f5a64a4cee0e64 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 1208 -60.9% 473.12 aim7.jobs-per-min 992.70 +155.6% 2537 aim7.time.elapsed_time 992.70 +155.6% 2537 aim7.time.elapsed_time.max 1.078e+08 +60.0% 1.724e+08 aim7.time.file_system_outputs 9646301 -4.2% 9238717 aim7.time.involuntary_context_switches 24107 ± 3% +58.2% 38130 ± 17% aim7.time.minor_page_faults 3295 +202.3% 9963 aim7.time.system_time 26.75 +51.4% 40.49 aim7.time.user_time 5.02e+08 +210.5% 1.559e+09 aim7.time.voluntary_context_switches 9294452 ± 4% -15.7% 7835497 ± 16% numa-numastat.node0.local_node 9298417 ± 4% -15.7% 7840991 ± 16% numa-numastat.node0.numa_hit 3.66 ± 3% +4.2 7.82 ± 18% mpstat.cpu.all.idle% 0.01 ± 32% -0.0 0.00 ± 31% mpstat.cpu.all.soft% 0.06 -0.0 0.04 mpstat.cpu.all.usr% 3.85 ± 3% +105.0% 7.89 ± 17% iostat.cpu.idle 89.80 -4.9% 85.36 iostat.cpu.iowait 6.29 +6.6% 6.70 iostat.cpu.system 23968 -48.4% 12356 iostat.md0.w/s 53780 -37.1% 33824 iostat.md0.wkB/s 89.00 -4.8% 84.75 vmstat.cpu.wa 53779 -37.1% 33833 vmstat.io.bo 29311046 -28.6% 20935254 vmstat.memory.free 1026388 +19.4% 1225080 vmstat.system.cs 103289 +8.5% 112038 vmstat.system.in 24013 ± 29% -58.5% 9976 ±120% numa-meminfo.node0.Dirty 119019 ± 10% +44.6% 172139 ± 13% numa-meminfo.node1.Active 3503 ± 56% +464.8% 19786 ± 38% numa-meminfo.node1.Active(file) 40477 ± 16% -38.5% 24896 ± 47% numa-meminfo.node1.Dirty 19966008 ± 2% -29.5% 14084594 ± 44% numa-meminfo.node1.MemFree 13014643 ± 4% +45.2% 18896052 ± 32% numa-meminfo.node1.MemUsed 12018 ± 41% +184.8% 34228 ± 35% numa-meminfo.node1.Shmem 2.09e+10 +127.4% 4.752e+10 ± 4% cpuidle.C1.time 2.558e+08 +140.1% 6.143e+08 ± 4% cpuidle.C1.usage 3.017e+09 ± 10% +301.2% 1.211e+10 ± 9% cpuidle.C1E.time 18225961 ± 9% +430.1% 96624314 ± 6% cpuidle.C1E.usage 4.627e+08 ± 14% +147.5% 1.145e+09 ± 20% cpuidle.C6.time 802166 ± 17% +186.9% 2301023 ± 26% cpuidle.C6.usage 1.799e+10 +148.9% 4.476e+10 cpuidle.POLL.time 2.616e+08 ± 2% +246.4% 9.063e+08 ± 2% cpuidle.POLL.usage 6017 ± 29% -58.5% 2496 ±120% numa-vmstat.node0.nr_dirty 5742 ± 29% -65.0% 2011 ±120% numa-vmstat.node0.nr_zone_write_pending 874.50 ± 56% +465.6% 4946 ± 38% numa-vmstat.node1.nr_active_file 10139 ± 16% -38.6% 6224 ± 47% numa-vmstat.node1.nr_dirty 4992144 ± 2% -29.5% 3521200 ± 44% numa-vmstat.node1.nr_free_pages 3003 ± 41% +184.9% 8556 ± 35% numa-vmstat.node1.nr_shmem 874.50 ± 56% +465.6% 4946 ± 38% numa-vmstat.node1.nr_zone_active_file 9775 ± 16% -47.9% 5089 ± 47% numa-vmstat.node1.nr_zone_write_pending 2.558e+08 +140.1% 6.143e+08 ± 4% turbostat.C1 43.77 -4.8 38.98 ± 3% turbostat.C1% 18225642 ± 9% +430.2% 96624037 ± 6% turbostat.C1E 6.31 ± 10% +3.6 9.94 ± 9% turbostat.C1E% 795117 ± 17% +188.4% 2292823 ± 26% turbostat.C6 0.38 ± 42% -63.6% 0.14 ± 13% turbostat.CPU%c6 1.028e+08 +176.7% 2.844e+08 turbostat.IRQ 0.23 ± 33% -69.1% 0.07 ± 22% turbostat.Pkg%pc2 275064 +12.3% 308846 ± 4% meminfo.Active 8036 ± 21% +261.7% 29072 ± 52% meminfo.Active(file) 64400 -45.9% 34850 meminfo.Dirty 16481 +10.8% 18264 meminfo.Inactive(anon) 147067 +12.8% 165858 ± 2% meminfo.KReclaimable 29120417 -28.8% 20740075 meminfo.MemAvailable 29335165 -28.6% 20936804 meminfo.MemFree 36511133 +23.0% 44909491 meminfo.Memused 147067 +12.8% 165858 ± 2% meminfo.SReclaimable 36567 ± 6% +37.2% 50173 meminfo.Shmem 51580 -60.8% 20220 meminfo.max_used_kB 4.27 ± 7% -13.9% 3.67 ± 4% perf-stat.i.MPKI 0.71 ± 3% -0.0 0.67 perf-stat.i.branch-miss-rate% 17.93 +3.5 21.41 ± 2% perf-stat.i.cache-miss-rate% 1029337 +19.1% 1226025 perf-stat.i.context-switches 11421 ± 4% +41.1% 16120 ± 6% perf-stat.i.cpu-migrations 0.42 ± 2% +0.0 0.45 ± 6% perf-stat.i.dTLB-load-miss-rate% 0.11 ± 27% -0.0 0.08 ± 3% perf-stat.i.dTLB-store-miss-rate% 84.04 ± 2% -2.3 81.75 ± 2% perf-stat.i.iTLB-load-miss-rate% 7375571 +7.3% 7917282 perf-stat.i.iTLB-load-misses 3.91 ± 7% -9.2% 3.56 ± 4% perf-stat.overall.MPKI 17.90 +4.1 21.96 ± 3% perf-stat.overall.cache-miss-rate% 0.41 ± 2% +0.0 0.45 ± 6% perf-stat.overall.dTLB-load-miss-rate% 0.10 ± 27% -0.0 0.08 ± 3% perf-stat.overall.dTLB-store-miss-rate% 83.99 ± 2% -2.3 81.72 ± 2% perf-stat.overall.iTLB-load-miss-rate% 1028292 +19.2% 1225532 perf-stat.ps.context-switches 11410 ± 4% +41.2% 16114 ± 6% perf-stat.ps.cpu-migrations 7368087 +7.4% 7914119 perf-stat.ps.iTLB-load-misses 2.558e+13 ± 2% +157.0% 6.575e+13 ± 3% perf-stat.total.instructions 9128 ± 2% -25.3% 6821 ± 4% slabinfo.dmaengine-unmap-16.active_objs 9175 ± 2% -24.4% 6938 ± 4% slabinfo.dmaengine-unmap-16.num_objs 28309 +12.5% 31840 slabinfo.f2fs_extent_node.active_objs 28704 +13.0% 32429 slabinfo.f2fs_extent_node.num_objs 964.00 ± 5% -13.3% 836.00 ± 6% slabinfo.kmem_cache_node.active_objs 1008 ± 5% -12.7% 880.00 ± 6% slabinfo.kmem_cache_node.num_objs 13017 -33.1% 8707 slabinfo.nat_entry.active_objs 13036 -33.2% 8707 slabinfo.nat_entry.num_objs 433.75 ± 9% +32.4% 574.50 ± 2% slabinfo.nfs_read_data.active_objs 433.75 ± 9% +32.4% 574.50 ± 2% slabinfo.nfs_read_data.num_objs 697.50 ± 3% +48.6% 1036 ± 33% slabinfo.numa_policy.active_objs 697.50 ± 3% +48.6% 1036 ± 33% slabinfo.numa_policy.num_objs 153098 +22.3% 187296 slabinfo.radix_tree_node.active_objs 2733 +22.3% 3344 slabinfo.radix_tree_node.active_slabs 153098 +22.3% 187296 slabinfo.radix_tree_node.num_objs 2733 +22.3% 3344 slabinfo.radix_tree_node.num_slabs 542.00 ± 10% +29.5% 702.00 ± 6% slabinfo.xfrm_state.active_objs 542.00 ± 10% +29.5% 702.00 ± 6% slabinfo.xfrm_state.num_objs 66757 +4.8% 69942 proc-vmstat.nr_active_anon 2009 ± 21% +261.7% 7267 ± 52% proc-vmstat.nr_active_file 96.00 +3.9% 99.75 proc-vmstat.nr_anon_transparent_hugepages 13466177 +60.0% 21542226 proc-vmstat.nr_dirtied 16154 -46.0% 8719 proc-vmstat.nr_dirty 726643 -28.8% 517541 proc-vmstat.nr_dirty_background_threshold 1455064 -28.8% 1036348 proc-vmstat.nr_dirty_threshold 311402 +2.4% 318957 proc-vmstat.nr_file_pages 7332853 -28.6% 5234441 proc-vmstat.nr_free_pages 4120 +10.8% 4565 proc-vmstat.nr_inactive_anon 13939 -1.2% 13772 proc-vmstat.nr_kernel_stack 2845 -1.5% 2803 proc-vmstat.nr_page_table_pages 9142 ± 6% +37.2% 12542 proc-vmstat.nr_shmem 36767 +12.8% 41463 ± 2% proc-vmstat.nr_slab_reclaimable 13387539 +60.3% 21463781 proc-vmstat.nr_written 66757 +4.8% 69942 proc-vmstat.nr_zone_active_anon 2009 ± 21% +261.7% 7267 ± 52% proc-vmstat.nr_zone_active_file 4120 +10.8% 4565 proc-vmstat.nr_zone_inactive_anon 15494 -54.2% 7089 proc-vmstat.nr_zone_write_pending 582146 ±102% +628.1% 4238850 ± 6% proc-vmstat.numa_foreign 13506 ± 3% +114.5% 28974 ± 31% proc-vmstat.numa_hint_faults 8913 ± 6% +148.6% 22154 ± 35% proc-vmstat.numa_hint_faults_local 582146 ±102% +628.1% 4238850 ± 6% proc-vmstat.numa_miss 597885 ± 99% +611.6% 4254585 ± 6% proc-vmstat.numa_other 1014174 ± 20% -46.1% 547135 ± 57% proc-vmstat.pgalloc_dma32 16157005 +22.4% 19769138 proc-vmstat.pgalloc_normal 2516271 +152.8% 6360808 proc-vmstat.pgfault 4866589 +64.4% 8002519 proc-vmstat.pgfree 53553588 +60.3% 85856803 proc-vmstat.pgpgout 1.55 ± 4% +0.1 1.65 perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.wait_on_page_bit.f2fs_wait_on_page_writeback 1.58 ± 4% +0.1 1.69 perf-profile.calltrace.cycles-pp.schedule.io_schedule.wait_on_page_bit.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback 1.60 ± 4% +0.1 1.71 perf-profile.calltrace.cycles-pp.io_schedule.wait_on_page_bit.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file 0.75 ± 2% +0.2 0.91 ± 2% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.70 ± 4% -0.3 0.36 ± 7% perf-profile.children.cycles-pp.__generic_file_write_iter 0.69 ± 4% -0.3 0.35 ± 7% perf-profile.children.cycles-pp.generic_perform_write 0.57 ± 5% -0.3 0.25 ± 6% perf-profile.children.cycles-pp.__filemap_fdatawrite_range 0.58 ± 5% -0.3 0.26 ± 7% perf-profile.children.cycles-pp.file_write_and_wait_range 0.57 ± 6% -0.3 0.25 ± 6% perf-profile.children.cycles-pp.do_writepages 0.57 ± 5% -0.3 0.25 ± 6% perf-profile.children.cycles-pp.f2fs_write_data_pages 0.64 ± 5% -0.3 0.34 ± 6% perf-profile.children.cycles-pp.f2fs_write_begin 0.57 ± 5% -0.2 0.32 ± 7% perf-profile.children.cycles-pp.f2fs_put_page 0.31 ± 10% -0.2 0.11 ± 7% perf-profile.children.cycles-pp.blk_finish_plug 0.31 ± 10% -0.2 0.11 ± 7% perf-profile.children.cycles-pp.blk_flush_plug_list 0.30 ± 11% -0.2 0.11 ± 6% perf-profile.children.cycles-pp.raid1_unplug 0.26 -0.1 0.14 ± 12% perf-profile.children.cycles-pp.f2fs_write_cache_pages 0.22 ± 20% -0.1 0.11 ± 41% perf-profile.children.cycles-pp.brd_do_bvec 0.14 ± 12% -0.1 0.03 ±100% perf-profile.children.cycles-pp.submit_bio_wait 0.21 ± 10% -0.1 0.11 ± 10% perf-profile.children.cycles-pp.submit_bio 0.23 ± 17% -0.1 0.13 ± 31% perf-profile.children.cycles-pp.worker_thread 0.20 ± 3% -0.1 0.11 ± 14% perf-profile.children.cycles-pp.__write_data_page 0.14 ± 15% -0.1 0.05 ± 9% perf-profile.children.cycles-pp.__submit_flush_wait 0.17 ± 12% -0.1 0.09 ± 12% perf-profile.children.cycles-pp.md_make_request 0.16 ± 11% -0.1 0.08 ± 14% perf-profile.children.cycles-pp.md_handle_request 0.15 ± 9% -0.1 0.07 ± 11% perf-profile.children.cycles-pp.raid1_make_request 0.14 ± 17% -0.1 0.06 ± 11% perf-profile.children.cycles-pp.f2fs_issue_flush 0.14 ± 6% -0.1 0.08 ± 19% perf-profile.children.cycles-pp.f2fs_do_write_data_page 0.10 ± 33% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.reschedule_interrupt 0.09 ± 37% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.scheduler_ipi 0.06 ± 6% +0.0 0.08 ± 10% perf-profile.children.cycles-pp.up_read 0.10 ± 4% +0.0 0.12 ± 13% perf-profile.children.cycles-pp.pick_next_task_idle 0.20 ± 2% +0.0 0.23 ± 5% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.16 ± 6% +0.0 0.19 ± 4% perf-profile.children.cycles-pp.irq_exit 0.01 ±173% +0.1 0.07 ± 7% perf-profile.children.cycles-pp.put_prev_task_fair 0.27 ± 3% +0.1 0.33 ± 4% perf-profile.children.cycles-pp.tick_nohz_next_event 0.35 ± 3% +0.1 0.42 ± 3% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 1.60 ± 4% +0.1 1.72 perf-profile.children.cycles-pp.io_schedule 0.78 ± 2% +0.1 0.92 ± 2% perf-profile.children.cycles-pp.menu_select 0.06 ± 6% +0.0 0.08 ± 10% perf-profile.self.cycles-pp.up_read 0.08 ± 5% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.cpuidle_enter_state 0.35 ± 5% +0.0 0.39 ± 5% perf-profile.self.cycles-pp.f2fs_wait_on_node_pages_writeback 0.35 ± 4% +0.1 0.41 ± 3% perf-profile.self.cycles-pp.menu_select 8096 ± 28% +278.9% 30679 ± 20% sched_debug.cfs_rq:/.MIN_vruntime.avg 158062 ± 17% +299.5% 631442 ± 8% sched_debug.cfs_rq:/.MIN_vruntime.max 32629 ± 22% +289.7% 127148 ± 13% sched_debug.cfs_rq:/.MIN_vruntime.stddev 44053 +192.3% 128774 sched_debug.cfs_rq:/.exec_clock.avg 140530 ± 9% +377.8% 671471 ± 29% sched_debug.cfs_rq:/.exec_clock.max 33108 ± 2% +109.9% 69484 ± 2% sched_debug.cfs_rq:/.exec_clock.min 20291 ± 7% +386.1% 98642 ± 21% sched_debug.cfs_rq:/.exec_clock.stddev 0.25 ± 82% -88.4% 0.03 ±131% sched_debug.cfs_rq:/.load_avg.min 8096 ± 28% +278.9% 30679 ± 20% sched_debug.cfs_rq:/.max_vruntime.avg 158062 ± 17% +299.5% 631442 ± 8% sched_debug.cfs_rq:/.max_vruntime.max 32629 ± 22% +289.7% 127148 ± 13% sched_debug.cfs_rq:/.max_vruntime.stddev 184170 +218.6% 586859 ± 2% sched_debug.cfs_rq:/.min_vruntime.avg 251516 ± 4% +254.5% 891702 ± 11% sched_debug.cfs_rq:/.min_vruntime.max 165614 +131.3% 383025 ± 2% sched_debug.cfs_rq:/.min_vruntime.min 15286 ± 9% +1057.3% 176903 ± 10% sched_debug.cfs_rq:/.min_vruntime.stddev 0.67 ± 6% +45.5% 0.98 ± 9% sched_debug.cfs_rq:/.nr_spread_over.avg 0.74 ± 20% +32.4% 0.98 ± 13% sched_debug.cfs_rq:/.nr_spread_over.stddev 3.43 ± 29% -60.7% 1.35 ±100% sched_debug.cfs_rq:/.removed.load_avg.avg 75.10 ± 34% -84.1% 11.91 ±100% sched_debug.cfs_rq:/.removed.load_avg.max 15.24 ± 23% -75.5% 3.74 ±100% sched_debug.cfs_rq:/.removed.load_avg.stddev 157.58 ± 29% -60.5% 62.19 ±100% sched_debug.cfs_rq:/.removed.runnable_sum.avg 3459 ± 34% -84.0% 552.75 ± 99% sched_debug.cfs_rq:/.removed.runnable_sum.max 700.79 ± 23% -75.4% 172.66 ±100% sched_debug.cfs_rq:/.removed.runnable_sum.stddev 34.81 ± 42% -87.0% 4.53 ±100% sched_debug.cfs_rq:/.removed.util_avg.max 5.89 ± 31% -79.0% 1.23 ±100% sched_debug.cfs_rq:/.removed.util_avg.stddev 59642 ± 58% +506.4% 361657 ± 61% sched_debug.cfs_rq:/.spread0.max 15286 ± 9% +1057.2% 176904 ± 10% sched_debug.cfs_rq:/.spread0.stddev 15.81 ± 15% +28.7% 20.34 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.avg 553.04 ± 15% +42.8% 789.60 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.max 83.37 ± 15% +37.7% 114.78 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.stddev 130721 -17.9% 107295 sched_debug.cpu.avg_idle.avg 32650 ± 7% -16.9% 27145 ± 6% sched_debug.cpu.avg_idle.min 509212 +151.7% 1281639 sched_debug.cpu.clock.avg 509215 +151.7% 1281642 sched_debug.cpu.clock.max 509209 +151.7% 1281636 sched_debug.cpu.clock.min 509212 +151.7% 1281639 sched_debug.cpu.clock_task.avg 509215 +151.7% 1281642 sched_debug.cpu.clock_task.max 509209 +151.7% 1281636 sched_debug.cpu.clock_task.min 480.69 ± 2% +89.2% 909.42 ± 3% sched_debug.cpu.curr->pid.avg 14037 +135.4% 33040 sched_debug.cpu.curr->pid.max 2146 ± 2% +125.6% 4843 sched_debug.cpu.curr->pid.stddev 0.12 ± 5% +10.2% 0.13 ± 3% sched_debug.cpu.nr_running.avg 10174129 +208.2% 31360192 sched_debug.cpu.nr_switches.avg 10849045 +290.1% 42321541 ± 3% sched_debug.cpu.nr_switches.max 564008 ± 15% +1719.4% 10261545 ± 8% sched_debug.cpu.nr_switches.stddev -87.16 +262.4% -315.83 sched_debug.cpu.nr_uninterruptible.min 66.84 ± 14% +143.1% 162.48 ± 21% sched_debug.cpu.nr_uninterruptible.stddev 10172460 +208.3% 31358619 sched_debug.cpu.sched_count.avg 10847113 +290.1% 42319816 ± 3% sched_debug.cpu.sched_count.max 563945 ± 15% +1719.6% 10261603 ± 8% sched_debug.cpu.sched_count.stddev 4957584 +211.1% 15425344 sched_debug.cpu.sched_goidle.avg 5286410 +291.7% 20709361 ± 3% sched_debug.cpu.sched_goidle.max 272603 ± 14% +1717.5% 4954528 ± 7% sched_debug.cpu.sched_goidle.stddev 5120024 +209.4% 15841794 sched_debug.cpu.ttwu_count.avg 58965381 ± 14% +567.2% 3.934e+08 ± 33% sched_debug.cpu.ttwu_count.max 132694 ± 2% -41.0% 78274 ± 5% sched_debug.cpu.ttwu_count.min 11266172 ± 8% +482.6% 65635267 ± 21% sched_debug.cpu.ttwu_count.stddev 114955 +26.1% 144930 ± 3% sched_debug.cpu.ttwu_local.max 83595 ± 4% -53.0% 39284 ± 30% sched_debug.cpu.ttwu_local.min 5967 ± 16% +460.9% 33468 ± 9% sched_debug.cpu.ttwu_local.stddev 509209 +151.7% 1281636 sched_debug.cpu_clk 506629 +152.5% 1279056 sched_debug.ktime 509627 +151.6% 1282053 sched_debug.sched_clk 18876 +138.7% 45051 ± 4% softirqs.CPU0.RCU 131251 ± 5% +153.2% 332324 ± 8% softirqs.CPU0.SCHED 327888 ± 3% +181.5% 923135 ± 10% softirqs.CPU0.TIMER 19003 ± 4% +121.7% 42132 ± 12% softirqs.CPU1.RCU 125587 ± 4% +130.3% 289267 ± 22% softirqs.CPU1.SCHED 331057 ± 2% +205.9% 1012636 ± 18% softirqs.CPU1.TIMER 19567 ± 5% +137.8% 46538 ± 2% softirqs.CPU10.RCU 131681 +161.3% 344047 softirqs.CPU10.SCHED 318774 +177.5% 884714 ± 8% softirqs.CPU10.TIMER 17876 ± 5% +158.8% 46258 softirqs.CPU11.RCU 128299 ± 4% +167.5% 343196 softirqs.CPU11.SCHED 320847 ± 3% +166.4% 854607 ± 8% softirqs.CPU11.TIMER 19129 ± 4% +121.1% 42292 ± 4% softirqs.CPU12.RCU 135251 +172.0% 367942 softirqs.CPU12.SCHED 310663 +201.8% 937514 ± 24% softirqs.CPU12.TIMER 17993 +143.9% 43882 ± 3% softirqs.CPU13.RCU 137747 ± 4% +157.5% 354715 ± 4% softirqs.CPU13.SCHED 359507 ± 23% +126.2% 813048 ± 13% softirqs.CPU13.TIMER 18173 +141.9% 43961 ± 3% softirqs.CPU14.RCU 134564 +163.8% 355019 ± 4% softirqs.CPU14.SCHED 310599 ± 2% +160.3% 808494 ± 13% softirqs.CPU14.TIMER 18180 +139.4% 43524 ± 3% softirqs.CPU15.RCU 134993 +160.8% 352061 ± 6% softirqs.CPU15.SCHED 308373 +164.7% 816152 ± 12% softirqs.CPU15.TIMER 19918 +144.6% 48716 softirqs.CPU16.RCU 134609 +162.6% 353418 ± 4% softirqs.CPU16.SCHED 307916 +163.0% 809772 ± 13% softirqs.CPU16.TIMER 19672 ± 2% +148.8% 48938 ± 3% softirqs.CPU17.RCU 132926 ± 2% +165.9% 353452 ± 4% softirqs.CPU17.SCHED 314316 +157.0% 807880 ± 13% softirqs.CPU17.TIMER 20021 ± 3% +141.2% 48298 ± 2% softirqs.CPU18.RCU 133347 ± 2% +164.0% 352038 ± 5% softirqs.CPU18.SCHED 312605 +158.6% 808358 ± 13% softirqs.CPU18.TIMER 19458 +139.3% 46570 ± 6% softirqs.CPU19.RCU 134001 +150.1% 335170 ± 14% softirqs.CPU19.SCHED 310521 +171.0% 841639 ± 14% softirqs.CPU19.TIMER 19131 ± 3% +138.3% 45586 ± 3% softirqs.CPU2.RCU 129111 ± 4% +154.4% 328419 ± 9% softirqs.CPU2.SCHED 329064 ± 3% +172.7% 897239 ± 10% softirqs.CPU2.TIMER 19249 ± 2% +145.5% 47257 ± 3% softirqs.CPU20.RCU 132532 +167.8% 354983 ± 3% softirqs.CPU20.SCHED 310659 +163.4% 818268 ± 13% softirqs.CPU20.TIMER 19473 +142.2% 47163 ± 4% softirqs.CPU21.RCU 133856 +166.0% 356005 ± 4% softirqs.CPU21.SCHED 308960 +164.3% 816637 ± 13% softirqs.CPU21.TIMER 19216 ± 2% +146.1% 47292 ± 3% softirqs.CPU22.RCU 133039 +165.4% 353140 ± 4% softirqs.CPU22.SCHED 307621 +166.0% 818260 ± 13% softirqs.CPU22.TIMER 19244 ± 2% +145.9% 47314 ± 2% softirqs.CPU23.RCU 133897 +163.1% 352326 ± 5% softirqs.CPU23.SCHED 310036 ± 2% +161.6% 811061 ± 13% softirqs.CPU23.TIMER 18954 ± 4% +143.9% 46235 ± 4% softirqs.CPU24.RCU 128754 ± 3% +157.8% 331927 ± 7% softirqs.CPU24.SCHED 323118 +169.7% 871609 ± 8% softirqs.CPU24.TIMER 18942 ± 6% +136.0% 44695 ± 3% softirqs.CPU25.RCU 129580 ± 3% +162.2% 339786 ± 3% softirqs.CPU25.SCHED 328907 +168.0% 881566 ± 7% softirqs.CPU25.TIMER 18913 ± 2% +147.2% 46757 ± 2% softirqs.CPU26.RCU 124714 ± 2% +176.4% 344692 softirqs.CPU26.SCHED 328463 ± 4% +170.2% 887463 ± 8% softirqs.CPU26.TIMER 19015 +137.2% 45107 ± 5% softirqs.CPU27.RCU 128551 ± 6% +152.0% 323916 ± 13% softirqs.CPU27.SCHED 325384 ± 2% +193.3% 954266 ± 13% softirqs.CPU27.TIMER 19094 +149.9% 47725 ± 2% softirqs.CPU28.RCU 131563 +161.9% 344579 ± 2% softirqs.CPU28.SCHED 319151 +176.7% 883105 ± 9% softirqs.CPU28.TIMER 19050 ± 4% +145.4% 46756 ± 2% softirqs.CPU29.RCU 124211 ± 6% +176.0% 342763 ± 2% softirqs.CPU29.SCHED 329526 ± 4% +158.9% 853085 ± 8% softirqs.CPU29.TIMER 22721 ± 8% +121.7% 50370 ± 4% softirqs.CPU3.RCU 132516 ± 2% +156.9% 340453 ± 4% softirqs.CPU3.SCHED 323552 ± 2% +174.3% 887612 ± 9% softirqs.CPU3.TIMER 19911 ± 3% +137.4% 47276 softirqs.CPU30.RCU 130778 +161.4% 341879 softirqs.CPU30.SCHED 317264 +181.0% 891407 ± 8% softirqs.CPU30.TIMER 19232 ± 4% +144.4% 47012 softirqs.CPU31.RCU 126109 ± 10% +174.9% 346628 ± 2% softirqs.CPU31.SCHED 329439 ± 4% +169.4% 887674 ± 9% softirqs.CPU31.TIMER 18139 ± 4% +141.4% 43795 ± 4% softirqs.CPU32.RCU 125566 ± 4% +170.9% 340112 ± 4% softirqs.CPU32.SCHED 328833 +184.0% 934027 ± 9% softirqs.CPU32.TIMER 19309 ± 10% +128.3% 44077 ± 3% softirqs.CPU33.RCU 128438 ± 3% +167.7% 343879 ± 2% softirqs.CPU33.SCHED 320949 +172.2% 873631 ± 9% softirqs.CPU33.TIMER 18041 ± 4% +146.1% 44399 ± 2% softirqs.CPU34.RCU 129791 +165.0% 343914 ± 2% softirqs.CPU34.SCHED 319403 ± 2% +177.3% 885711 ± 8% softirqs.CPU34.TIMER 17991 ± 5% +146.2% 44290 ± 2% softirqs.CPU35.RCU 137112 ± 7% +148.6% 340824 ± 3% softirqs.CPU35.SCHED 416552 ± 21% +119.5% 914435 ± 8% softirqs.CPU35.TIMER 20036 ± 4% +138.7% 47823 ± 4% softirqs.CPU36.RCU 138048 ± 3% +165.7% 366854 ± 7% softirqs.CPU36.SCHED 359442 ± 22% +196.1% 1064343 ± 19% softirqs.CPU36.TIMER 19421 ± 5% +153.4% 49203 ± 2% softirqs.CPU37.RCU 135462 +158.7% 350500 ± 6% softirqs.CPU37.SCHED 305531 +166.8% 815287 ± 12% softirqs.CPU37.TIMER 19963 +150.5% 50014 softirqs.CPU38.RCU 132257 +167.5% 353848 ± 5% softirqs.CPU38.SCHED 311979 ± 2% +160.3% 812118 ± 12% softirqs.CPU38.TIMER 20119 ± 2% +143.6% 49011 ± 2% softirqs.CPU39.RCU 135450 +161.9% 354713 ± 4% softirqs.CPU39.SCHED 307797 +164.3% 813377 ± 12% softirqs.CPU39.TIMER 20593 ± 9% +132.1% 47796 ± 5% softirqs.CPU4.RCU 129881 +159.2% 336643 ± 3% softirqs.CPU4.SCHED 321940 +175.0% 885297 ± 8% softirqs.CPU4.TIMER 19874 ± 2% +149.6% 49611 softirqs.CPU40.RCU 134660 +161.9% 352636 ± 5% softirqs.CPU40.SCHED 309870 +160.6% 807374 ± 13% softirqs.CPU40.TIMER 19897 ± 2% +147.2% 49186 ± 2% softirqs.CPU41.RCU 131978 ± 2% +167.4% 352864 ± 5% softirqs.CPU41.SCHED 315198 +157.6% 812013 ± 12% softirqs.CPU41.TIMER 19988 ± 2% +143.9% 48749 ± 2% softirqs.CPU42.RCU 133326 ± 2% +165.6% 354135 ± 4% softirqs.CPU42.SCHED 313042 +159.5% 812298 ± 12% softirqs.CPU42.TIMER 19723 ± 2% +141.9% 47704 ± 4% softirqs.CPU43.RCU 131944 +165.7% 350566 ± 5% softirqs.CPU43.SCHED 314428 +162.9% 826595 ± 13% softirqs.CPU43.TIMER 19824 ± 2% +143.0% 48177 ± 4% softirqs.CPU44.RCU 132047 ± 2% +157.8% 340362 ± 12% softirqs.CPU44.SCHED 312615 ± 2% +165.2% 829094 ± 12% softirqs.CPU44.TIMER 20140 ± 2% +138.6% 48049 ± 7% softirqs.CPU45.RCU 133846 +151.7% 336901 ± 14% softirqs.CPU45.SCHED 309086 ± 2% +170.8% 837094 ± 12% softirqs.CPU45.TIMER 20103 ± 3% +138.8% 48009 ± 3% softirqs.CPU46.RCU 133860 +154.1% 340108 ± 11% softirqs.CPU46.SCHED 311323 +166.1% 828498 ± 13% softirqs.CPU46.TIMER 19628 ± 2% +149.0% 48881 ± 2% softirqs.CPU47.RCU 132835 ± 2% +164.6% 351487 ± 5% softirqs.CPU47.SCHED 309264 +162.8% 812666 ± 13% softirqs.CPU47.TIMER 19773 ± 2% +139.0% 47263 ± 2% softirqs.CPU5.RCU 130891 ± 2% +163.7% 345185 softirqs.CPU5.SCHED 323979 ± 2% +182.2% 914392 ± 8% softirqs.CPU5.TIMER 19904 ± 3% +141.4% 48045 ± 4% softirqs.CPU6.RCU 131836 +160.9% 344009 softirqs.CPU6.SCHED 316150 +181.7% 890614 ± 8% softirqs.CPU6.TIMER 19077 ± 3% +145.1% 46751 ± 3% softirqs.CPU7.RCU 127906 ± 2% +157.6% 329548 ± 5% softirqs.CPU7.SCHED 325087 +181.6% 915494 ± 9% softirqs.CPU7.TIMER 20183 ± 2% +114.9% 43380 ± 10% softirqs.CPU8.RCU 131974 +118.3% 288124 ± 30% softirqs.CPU8.SCHED 319626 +190.1% 927161 ± 13% softirqs.CPU8.TIMER 19292 ± 3% +137.6% 45839 ± 3% softirqs.CPU9.RCU 129715 +158.9% 335788 ± 4% softirqs.CPU9.SCHED 321209 +178.0% 893035 ± 8% softirqs.CPU9.TIMER 931061 +140.9% 2242787 softirqs.RCU 6322319 +160.8% 16487173 softirqs.SCHED 15417545 +169.7% 41581783 ± 8% softirqs.TIMER 853.75 ± 12% +139.9% 2048 ± 7% interrupts.35:PCI-MSI.2621441-edge.eth0-TxRx-0 586.00 ± 14% +130.1% 1348 ± 5% interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1 651.00 ± 9% +136.3% 1538 ± 9% interrupts.37:PCI-MSI.2621443-edge.eth0-TxRx-2 565.25 ± 14% +129.7% 1298 interrupts.38:PCI-MSI.2621444-edge.eth0-TxRx-3 591.00 ± 14% +123.8% 1322 ± 2% interrupts.39:PCI-MSI.2621445-edge.eth0-TxRx-4 660.25 ± 13% +111.4% 1396 ± 7% interrupts.40:PCI-MSI.2621446-edge.eth0-TxRx-5 529.50 ± 4% +227.1% 1732 ± 14% interrupts.41:PCI-MSI.2621447-edge.eth0-TxRx-6 585.75 ± 16% +163.2% 1541 ± 10% interrupts.42:PCI-MSI.2621448-edge.eth0-TxRx-7 404968 +145.6% 994665 ± 3% interrupts.CAL:Function_call_interrupts 8396 ± 6% +133.7% 19627 ± 6% interrupts.CPU0.CAL:Function_call_interrupts 1988743 +155.3% 5077644 interrupts.CPU0.LOC:Local_timer_interrupts 8698 ± 5% +121.4% 19255 ± 2% interrupts.CPU1.CAL:Function_call_interrupts 1986989 +155.5% 5077206 interrupts.CPU1.LOC:Local_timer_interrupts 8487 ± 5% +132.3% 19713 ± 8% interrupts.CPU10.CAL:Function_call_interrupts 1988375 +155.4% 5077322 interrupts.CPU10.LOC:Local_timer_interrupts 8770 ± 5% +149.6% 21889 ± 3% interrupts.CPU11.CAL:Function_call_interrupts 1987964 +155.4% 5077484 interrupts.CPU11.LOC:Local_timer_interrupts 78678 ± 8% +541.4% 504605 ±122% interrupts.CPU11.RES:Rescheduling_interrupts 8731 ± 9% +119.5% 19167 ± 6% interrupts.CPU12.CAL:Function_call_interrupts 1988222 +155.4% 5077524 interrupts.CPU12.LOC:Local_timer_interrupts 7960 ± 3% +153.5% 20176 ± 9% interrupts.CPU13.CAL:Function_call_interrupts 1988667 +155.3% 5076359 interrupts.CPU13.LOC:Local_timer_interrupts 8518 ± 6% +155.0% 21723 ± 7% interrupts.CPU14.CAL:Function_call_interrupts 1987350 +155.5% 5077482 interrupts.CPU14.LOC:Local_timer_interrupts 37.00 ± 31% +124.3% 83.00 ± 24% interrupts.CPU14.TLB:TLB_shootdowns 8831 ± 4% +137.4% 20964 ± 10% interrupts.CPU15.CAL:Function_call_interrupts 1987501 +155.5% 5077129 interrupts.CPU15.LOC:Local_timer_interrupts 3708 ± 27% -63.0% 1370 ±103% interrupts.CPU15.NMI:Non-maskable_interrupts 3708 ± 27% -63.0% 1370 ±103% interrupts.CPU15.PMI:Performance_monitoring_interrupts 23.25 ± 24% +253.8% 82.25 ± 28% interrupts.CPU15.TLB:TLB_shootdowns 8708 ± 8% +132.7% 20268 ± 7% interrupts.CPU16.CAL:Function_call_interrupts 1988418 +155.3% 5076923 interrupts.CPU16.LOC:Local_timer_interrupts 8548 ± 7% +148.3% 21226 ± 7% interrupts.CPU17.CAL:Function_call_interrupts 1988416 +155.3% 5077317 interrupts.CPU17.LOC:Local_timer_interrupts 33.00 ± 18% +147.0% 81.50 ± 21% interrupts.CPU17.TLB:TLB_shootdowns 8456 ± 9% +157.0% 21731 ± 6% interrupts.CPU18.CAL:Function_call_interrupts 1988574 +155.3% 5077430 interrupts.CPU18.LOC:Local_timer_interrupts 30.25 ± 15% +181.8% 85.25 ± 25% interrupts.CPU18.TLB:TLB_shootdowns 8884 ± 6% +144.5% 21720 ± 3% interrupts.CPU19.CAL:Function_call_interrupts 1987747 +155.4% 5077260 interrupts.CPU19.LOC:Local_timer_interrupts 37.00 ± 19% +127.0% 84.00 ± 26% interrupts.CPU19.TLB:TLB_shootdowns 8298 ± 11% +145.3% 20352 ± 9% interrupts.CPU2.CAL:Function_call_interrupts 1988446 +155.4% 5077740 interrupts.CPU2.LOC:Local_timer_interrupts 8594 ± 5% +146.1% 21149 ± 5% interrupts.CPU20.CAL:Function_call_interrupts 1987855 +155.4% 5077226 interrupts.CPU20.LOC:Local_timer_interrupts 8376 ± 8% +151.7% 21084 ± 9% interrupts.CPU21.CAL:Function_call_interrupts 1988171 +155.4% 5077212 interrupts.CPU21.LOC:Local_timer_interrupts 41.25 ± 20% +106.1% 85.00 ± 27% interrupts.CPU21.TLB:TLB_shootdowns 9024 ± 2% +141.1% 21757 ± 3% interrupts.CPU22.CAL:Function_call_interrupts 1987613 +155.4% 5077125 interrupts.CPU22.LOC:Local_timer_interrupts 8095 ± 7% +154.9% 20633 ± 6% interrupts.CPU23.CAL:Function_call_interrupts 1989315 +155.2% 5077451 interrupts.CPU23.LOC:Local_timer_interrupts 853.75 ± 12% +139.9% 2048 ± 7% interrupts.CPU24.35:PCI-MSI.2621441-edge.eth0-TxRx-0 8463 ± 5% +147.0% 20901 ± 7% interrupts.CPU24.CAL:Function_call_interrupts 1988251 +155.4% 5077879 interrupts.CPU24.LOC:Local_timer_interrupts 74848 ± 8% +525.2% 467985 ±128% interrupts.CPU24.RES:Rescheduling_interrupts 586.00 ± 14% +130.1% 1348 ± 5% interrupts.CPU25.36:PCI-MSI.2621442-edge.eth0-TxRx-1 8156 ± 7% +170.0% 22019 ± 2% interrupts.CPU25.CAL:Function_call_interrupts 1988295 +155.3% 5077010 interrupts.CPU25.LOC:Local_timer_interrupts 930.75 ±123% +260.9% 3359 ± 15% interrupts.CPU25.NMI:Non-maskable_interrupts 930.75 ±123% +260.9% 3359 ± 15% interrupts.CPU25.PMI:Performance_monitoring_interrupts 72678 ± 7% +575.9% 491267 ±120% interrupts.CPU25.RES:Rescheduling_interrupts 651.00 ± 9% +136.3% 1538 ± 9% interrupts.CPU26.37:PCI-MSI.2621443-edge.eth0-TxRx-2 8538 ± 6% +141.5% 20625 ± 6% interrupts.CPU26.CAL:Function_call_interrupts 1988053 +155.4% 5077342 interrupts.CPU26.LOC:Local_timer_interrupts 75367 ± 8% +522.0% 468794 ±133% interrupts.CPU26.RES:Rescheduling_interrupts 565.25 ± 14% +129.7% 1298 interrupts.CPU27.38:PCI-MSI.2621444-edge.eth0-TxRx-3 8157 ± 5% +132.5% 18969 ± 3% interrupts.CPU27.CAL:Function_call_interrupts 1988437 +155.3% 5077428 interrupts.CPU27.LOC:Local_timer_interrupts 591.00 ± 14% +123.8% 1322 ± 2% interrupts.CPU28.39:PCI-MSI.2621445-edge.eth0-TxRx-4 8181 ± 6% +148.1% 20300 ± 3% interrupts.CPU28.CAL:Function_call_interrupts 1987807 +155.4% 5077738 interrupts.CPU28.LOC:Local_timer_interrupts 71071 ± 7% +581.0% 483987 ±128% interrupts.CPU28.RES:Rescheduling_interrupts 660.25 ± 13% +111.4% 1396 ± 7% interrupts.CPU29.40:PCI-MSI.2621446-edge.eth0-TxRx-5 8178 ± 5% +150.4% 20475 ± 7% interrupts.CPU29.CAL:Function_call_interrupts 1987852 +155.4% 5077382 interrupts.CPU29.LOC:Local_timer_interrupts 70205 ± 8% +585.5% 481286 ±127% interrupts.CPU29.RES:Rescheduling_interrupts 17.75 ± 36% +163.4% 46.75 ± 80% interrupts.CPU29.TLB:TLB_shootdowns 8684 ± 5% +151.7% 21860 ± 4% interrupts.CPU3.CAL:Function_call_interrupts 1988786 +155.3% 5077436 interrupts.CPU3.LOC:Local_timer_interrupts 80407 ± 4% +518.6% 497419 ±122% interrupts.CPU3.RES:Rescheduling_interrupts 529.50 ± 4% +227.1% 1732 ± 14% interrupts.CPU30.41:PCI-MSI.2621447-edge.eth0-TxRx-6 8415 ± 4% +138.0% 20031 ± 3% interrupts.CPU30.CAL:Function_call_interrupts 1988609 +155.3% 5077494 interrupts.CPU30.LOC:Local_timer_interrupts 74875 ± 8% +518.8% 463337 ±134% interrupts.CPU30.RES:Rescheduling_interrupts 585.75 ± 16% +163.2% 1541 ± 10% interrupts.CPU31.42:PCI-MSI.2621448-edge.eth0-TxRx-7 7917 +151.5% 19913 ± 3% interrupts.CPU31.CAL:Function_call_interrupts 1988677 +155.3% 5077502 interrupts.CPU31.LOC:Local_timer_interrupts 67596 ± 3% +596.5% 470844 ±134% interrupts.CPU31.RES:Rescheduling_interrupts 8230 ± 4% +161.9% 21551 ± 8% interrupts.CPU32.CAL:Function_call_interrupts 1988317 +155.4% 5077410 interrupts.CPU32.LOC:Local_timer_interrupts 70449 ± 5% +597.2% 491174 ±127% interrupts.CPU32.RES:Rescheduling_interrupts 8684 ± 6% +137.3% 20610 ± 6% interrupts.CPU33.CAL:Function_call_interrupts 1988440 +155.3% 5077281 interrupts.CPU33.LOC:Local_timer_interrupts 77401 ± 5% +495.8% 461151 ±134% interrupts.CPU33.RES:Rescheduling_interrupts 8412 ± 8% +150.3% 21061 ± 4% interrupts.CPU34.CAL:Function_call_interrupts 1988447 +155.4% 5077546 interrupts.CPU34.LOC:Local_timer_interrupts 75351 ± 8% +537.5% 480376 ±128% interrupts.CPU34.RES:Rescheduling_interrupts 7890 +147.7% 19543 ± 5% interrupts.CPU35.CAL:Function_call_interrupts 1988875 +155.3% 5077717 interrupts.CPU35.LOC:Local_timer_interrupts 70158 ± 2% +577.1% 475011 ±135% interrupts.CPU35.RES:Rescheduling_interrupts 7908 ± 2% +184.9% 22530 ± 3% interrupts.CPU36.CAL:Function_call_interrupts 1988403 +155.4% 5077694 interrupts.CPU36.LOC:Local_timer_interrupts 31.75 ± 20% +178.0% 88.25 ± 28% interrupts.CPU36.TLB:TLB_shootdowns 9078 ± 2% +133.8% 21229 ± 6% interrupts.CPU37.CAL:Function_call_interrupts 1986971 +155.5% 5076871 interrupts.CPU37.LOC:Local_timer_interrupts 34.75 ± 36% +154.7% 88.50 ± 25% interrupts.CPU37.TLB:TLB_shootdowns 8574 ± 7% +131.7% 19864 ± 3% interrupts.CPU38.CAL:Function_call_interrupts 1986780 +155.5% 5076769 interrupts.CPU38.LOC:Local_timer_interrupts 29.25 ± 35% +195.7% 86.50 ± 18% interrupts.CPU38.TLB:TLB_shootdowns 8184 ± 7% +158.0% 21112 ± 4% interrupts.CPU39.CAL:Function_call_interrupts 1986972 +155.5% 5077337 interrupts.CPU39.LOC:Local_timer_interrupts 8368 ± 8% +144.7% 20481 ± 12% interrupts.CPU4.CAL:Function_call_interrupts 1987774 +155.4% 5077506 interrupts.CPU4.LOC:Local_timer_interrupts 8165 ± 3% +163.7% 21528 ± 6% interrupts.CPU40.CAL:Function_call_interrupts 1987659 +155.4% 5077421 interrupts.CPU40.LOC:Local_timer_interrupts 32.00 ± 12% +154.7% 81.50 ± 29% interrupts.CPU40.TLB:TLB_shootdowns 8454 ± 6% +158.4% 21847 ± 8% interrupts.CPU41.CAL:Function_call_interrupts 1987504 +155.5% 5077467 interrupts.CPU41.LOC:Local_timer_interrupts 37.75 ± 45% +122.5% 84.00 ± 17% interrupts.CPU41.TLB:TLB_shootdowns 8529 ± 7% +136.4% 20164 ± 8% interrupts.CPU42.CAL:Function_call_interrupts 1988401 +155.4% 5077407 interrupts.CPU42.LOC:Local_timer_interrupts 32.75 ± 39% +171.8% 89.00 ± 37% interrupts.CPU42.TLB:TLB_shootdowns 8102 ± 5% +150.4% 20283 ± 9% interrupts.CPU43.CAL:Function_call_interrupts 1987414 +155.5% 5077903 interrupts.CPU43.LOC:Local_timer_interrupts 32.50 ± 18% +150.8% 81.50 ± 34% interrupts.CPU43.TLB:TLB_shootdowns 8206 ± 7% +157.3% 21112 ± 8% interrupts.CPU44.CAL:Function_call_interrupts 1987278 +155.5% 5077309 interrupts.CPU44.LOC:Local_timer_interrupts 37.00 ± 25% +119.6% 81.25 ± 24% interrupts.CPU44.TLB:TLB_shootdowns 8467 ± 7% +139.3% 20261 ± 5% interrupts.CPU45.CAL:Function_call_interrupts 1987858 +155.4% 5077234 interrupts.CPU45.LOC:Local_timer_interrupts 38.25 ± 22% +113.1% 81.50 ± 27% interrupts.CPU45.TLB:TLB_shootdowns 7773 ± 4% +162.0% 20367 ± 9% interrupts.CPU46.CAL:Function_call_interrupts 1988584 +155.3% 5077417 interrupts.CPU46.LOC:Local_timer_interrupts 37.50 ± 14% +142.7% 91.00 ± 34% interrupts.CPU46.TLB:TLB_shootdowns 8746 ± 4% +138.7% 20880 ± 8% interrupts.CPU47.CAL:Function_call_interrupts 1988495 +155.3% 5077204 interrupts.CPU47.LOC:Local_timer_interrupts 8722 ± 7% +134.1% 20414 ± 10% interrupts.CPU5.CAL:Function_call_interrupts 1987875 +155.4% 5077418 interrupts.CPU5.LOC:Local_timer_interrupts 8435 ± 8% +143.9% 20570 ± 11% interrupts.CPU6.CAL:Function_call_interrupts 1988389 +155.4% 5077630 interrupts.CPU6.LOC:Local_timer_interrupts 8837 ± 5% +139.1% 21133 ± 9% interrupts.CPU7.CAL:Function_call_interrupts 1987853 +155.4% 5077370 interrupts.CPU7.LOC:Local_timer_interrupts 8852 ± 5% +123.8% 19810 ± 3% interrupts.CPU8.CAL:Function_call_interrupts 1988702 +155.3% 5077569 interrupts.CPU8.LOC:Local_timer_interrupts 8273 ± 6% +151.1% 20771 ± 9% interrupts.CPU9.CAL:Function_call_interrupts 1988033 +155.4% 5077588 interrupts.CPU9.LOC:Local_timer_interrupts 70739 ± 6% +578.9% 480243 ±129% interrupts.CPU9.RES:Rescheduling_interrupts 95428175 +155.4% 2.437e+08 interrupts.LOC:Local_timer_interrupts 144.00 +133.3% 336.00 interrupts.MCP:Machine_check_polls 6773889 ± 8% +482.9% 39483682 ± 7% interrupts.RES:Rescheduling_interrupts 1544 ± 5% +91.5% 2957 ± 11% interrupts.TLB:TLB_shootdowns aim7.jobs-per-min 1300 +-+------------------------------------------------------------------+ 1200 +-+.++.+.+.+.+. +.+.+.+. +.+.+ +.++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.| | + + + | 1100 +-+ | 1000 +-+ | | | 900 +-+ | 800 +-+ | 700 +-+ | | | 600 +-+ | 500 +-+ O | | O O O O O O O | 400 O-O OO O O O O OO O O O OO O O O O OO O O O O | 300 +-+------------------------------------------------------------------+ aim7.time.user_time 55 +-+--------------------------------------------------------------------+ | | 50 +-+ O | | O | | | 45 +-+ | | O O O O O O O O O O | 40 O-O O OO O O O O O O O O O OO O O O O O | | | 35 +-+ | | | | | 30 +-+ | |.+.+.++.+.+.+.+.+.+.++.+.+.+.+.+.+.++.+.+.+.+.+.++.+. .+.+.+.+.++.+.+.| 25 +-+--------------------------------------------------------------------+ aim7.time.system_time 11000 +-+-----------------------------------------------------------------+ O O OO O O O OO O O OO O O O OO O O O OO O O | 10000 +-+ O OO O O OO O | 9000 +-+ | | | 8000 +-+ | | | 7000 +-+ | | | 6000 +-+ | 5000 +-+ | | | 4000 +-+ | | .++. .+. .+.+. .+.+. .++. .+ .+.+.+ .+. .++.+.| 3000 +-+-----------------------------------------------------------------+ aim7.time.elapsed_time 3500 +-+------------------------------------------------------------------+ O O OO O O O O OO O O OO O O O O OO O O O O | 3000 +-+ O | | | | O O O O O O | 2500 +-+ OO | | | 2000 +-+ | | | 1500 +-+ | | | | | 1000 +-+.++.+.+.+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.| | | 500 +-+------------------------------------------------------------------+ aim7.time.elapsed_time.max 3500 +-+------------------------------------------------------------------+ O O OO O O O O OO O O OO O O O O OO O O O O | 3000 +-+ O | | | | O O O O O O | 2500 +-+ OO | | | 2000 +-+ | | | 1500 +-+ | | | | | 1000 +-+.++.+.+.+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.| | | 500 +-+------------------------------------------------------------------+ aim7.time.voluntary_context_switches 1.8e+09 +-+---------------------------------------------------------------+ O O O O OO O OO O O OO O O OO O OO O O | 1.6e+09 +-O O O O O O OO O O O | | | 1.4e+09 +-+ | | | 1.2e+09 +-+ | | | 1e+09 +-+ | | | 8e+08 +-+ | | | 6e+08 +-+ | |.++.+.+.++.+.+.++.+.+.++.+.+.++.+.++.+.+.++.+.+.++.+.+.++.+.+.++.| 4e+08 +-+---------------------------------------------------------------+ aim7.time.file_system_outputs 1.8e+08 +-+---------------------------------------------------------------+ O OO O O OO O O OO O O OO O O OO O OO O O OO O O OO O O O | 1.7e+08 +-+ | 1.6e+08 +-+ | | | 1.5e+08 +-+ | | | 1.4e+08 +-+ | | | 1.3e+08 +-+ | 1.2e+08 +-+ | | | 1.1e+08 +-+ | |.++.+.+.++.+.+.++.+.+.++.+.+.++.+.++.+.+.++.+.+.++.+.+.++.+.+.++.| 1e+08 +-+---------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 years, 2 months

3
3
0 / 0

[btrfs] 8d47a0d8f7: fio.write_bw_MBps -28.6% regression

by kernel test robot

Greeting, FYI, we noticed a -28.6% regression of fio.write_bw_MBps due to commit: commit: 8d47a0d8f7947422dd359ac8e462687f81a7a137 ("btrfs: Do mandatory tree block check before submitting bio") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: fio-basic on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory with following parameters: disk: 2pmem fs: btrfs runtime: 200s nr_task: 50% time_based: tb rw: randwrite bs: 4k ioengine: libaio test_size: 100G cpufreq_governor: performance ucode: 0x3d test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. test-url: https://github.com/axboe/fio Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode: 4k/gcc-7/performance/2pmem/btrfs/libaio/x86_64-rhel-7.6/50%/debian-x86_64-2018-04-03.cgz/200s/randwrite/lkp-hsw-ep2/100G/fio-basic/tb/0x3d commit: ff2ac107fa ("btrfs: tree-checker: Remove comprehensive root owner check") 8d47a0d8f7 ("btrfs: Do mandatory tree block check before submitting bio") ff2ac107fae2440b 8d47a0d8f7947422dd359ac8e46 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.BUG:unable_to_handle_kernel :4 25% 1:4 dmesg.Kernel_panic-not_syncing:Fatal_exception_in_interrupt :4 25% 1:4 dmesg.Oops:#[##] :4 25% 1:4 dmesg.RIP:cpuidle_enter_state :4 25% 1:4 dmesg.RIP:native_write_msr :4 25% 1:4 dmesg.RIP:perf_prepare_sample 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 0.01 ± 21% +0.3 0.28 ± 20% fio.latency_1000ms% 4.14 ± 13% +1.8 5.95 ± 9% fio.latency_10ms% 8.73 ± 15% +3.0 11.68 ± 8% fio.latency_20ms% 1.91 ± 5% -0.4 1.52 ± 3% fio.latency_250ms% 66.81 -5.9 60.90 ± 2% fio.latency_4ms% 0.18 ± 15% +0.1 0.29 ± 16% fio.latency_750us% 1.017e+08 -28.5% 72701525 fio.time.file_system_outputs 109.75 -26.8% 80.33 ± 2% fio.time.percent_of_cpu_this_job_got 197.97 -26.4% 145.79 ± 2% fio.time.system_time 38.66 ± 4% -24.0% 29.39 ± 3% fio.time.user_time 12388015 -28.5% 8858878 fio.time.voluntary_context_switches 12711059 -28.5% 9083954 fio.workload 247.91 -28.6% 176.96 fio.write_bw_MBps 309248 ± 2% +71.7% 531114 ± 4% fio.write_clat_99%_us 17574 +40.0% 24609 fio.write_clat_mean_us 57989 ± 4% +82.2% 105661 fio.write_clat_stddev 63463 -28.6% 45302 fio.write_iops 564.61 +40.2% 791.54 fio.write_slat_mean_us 9375 ± 7% +96.3% 18406 fio.write_slat_stddev 21.25 +8.5% 23.06 ± 9% boot-time.dhcp 93.32 +1.4% 94.58 iostat.cpu.idle 6.43 -18.6% 5.23 ± 2% iostat.cpu.system 6.48 -1.2 5.27 ± 2% mpstat.cpu.all.sys% 0.24 ± 5% -0.1 0.18 ± 2% mpstat.cpu.all.usr% 93.00 +1.1% 94.00 vmstat.cpu.id 591200 -30.0% 413921 vmstat.io.bo 31653776 ± 2% -10.9% 28196588 ± 2% vmstat.memory.cache 11927341 ± 7% +29.8% 15486805 ± 4% vmstat.memory.free 5.00 -26.7% 3.67 ± 12% vmstat.procs.r 635910 ± 2% -27.6% 460384 vmstat.system.cs 7822384 ± 2% -25.0% 5870498 numa-numastat.node0.local_node 7839861 ± 2% -24.8% 5893785 numa-numastat.node0.numa_hit 224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node0.numa_miss 242465 ± 44% -84.3% 37964 ± 18% numa-numastat.node0.other_node 7762623 ± 12% -26.8% 5681358 ± 6% numa-numastat.node1.local_node 224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node1.numa_foreign 7768618 ± 12% -26.9% 5681573 ± 6% numa-numastat.node1.numa_hit 9.429e+08 ± 10% -35.4% 6.094e+08 cpuidle.C1.time 53504154 ± 6% -31.8% 36504731 cpuidle.C1.usage 1.093e+09 ± 22% -50.3% 5.435e+08 ± 3% cpuidle.C1E.time 16541925 ± 13% -45.0% 9103775 cpuidle.C1E.usage 5.366e+09 ± 36% +71.7% 9.214e+09 ± 18% cpuidle.C3.time 16163664 ± 33% +57.0% 25379738 ± 5% cpuidle.C3.usage 9697055 ± 16% -35.8% 6225168 ± 19% cpuidle.C6.usage 19313595 ± 17% -16.3% 16166123 cpuidle.POLL.time 10291838 ± 9% -26.8% 7537045 ± 7% meminfo.Active 10034782 ± 9% -27.4% 7280710 ± 7% meminfo.Active(file) 31524905 ± 2% -10.9% 28082249 ± 2% meminfo.Cached 115157 ± 4% +61.1% 185560 ± 3% meminfo.CmaFree 6430720 ± 8% +19.1% 7660885 ± 6% meminfo.DirectMap2M 534230 ± 5% -12.8% 465932 ± 6% meminfo.Dirty 11737322 ± 7% +30.2% 15281189 ± 4% meminfo.MemFree 37604344 ± 2% -9.4% 34060483 meminfo.Memused 806.50 ± 37% +137.4% 1914 ± 8% meminfo.Writeback 370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.active_objs 370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.num_objs 15147 ± 5% -8.3% 13897 ± 2% slabinfo.blkdev_ioc.active_objs 17839 ± 6% -11.7% 15745 ± 2% slabinfo.blkdev_ioc.num_objs 12178 ± 4% -7.4% 11275 ± 2% slabinfo.kmalloc-512.num_objs 7794 ± 3% +25.4% 9774 slabinfo.proc_inode_cache.active_objs 8017 ± 2% +23.4% 9895 slabinfo.proc_inode_cache.num_objs 2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.active_objs 2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.num_objs 263.75 ± 2% -15.1% 224.00 ± 2% turbostat.Avg_MHz 9.46 ± 2% -1.4 8.07 ± 2% turbostat.Busy% 53502248 ± 6% -31.8% 36503898 turbostat.C1 5.86 ± 8% -2.0 3.82 turbostat.C1% 16540396 ± 13% -45.0% 9102733 turbostat.C1E 6.80 ± 22% -3.4 3.40 ± 4% turbostat.C1E% 16163332 ± 33% +57.0% 25379351 ± 5% turbostat.C3 33.55 ± 36% +24.1 57.61 ± 18% turbostat.C3% 9689445 ± 16% -35.8% 6221033 ± 19% turbostat.C6 18.04 ± 26% +49.1% 26.90 ± 9% turbostat.CPU%c3 53.25 ± 3% +10.8% 59.00 ± 4% turbostat.CoreTmp 4.67 ± 14% +62.7% 7.59 ± 26% turbostat.Pkg%pc2 56.50 +9.7% 62.00 ± 2% turbostat.PkgTmp 11.84 -4.8% 11.28 turbostat.RAMWatt 7232478 ± 10% -37.2% 4543037 ± 22% numa-meminfo.node0.Active 7072564 ± 10% -37.6% 4410135 ± 23% numa-meminfo.node0.Active(file) 308167 ± 6% -24.7% 232152 ± 10% numa-meminfo.node0.Dirty 8784028 ± 13% +21.7% 10686327 ± 2% numa-meminfo.node0.Inactive 8669548 ± 12% +20.7% 10465444 ± 2% numa-meminfo.node0.Inactive(file) 572.50 ± 22% +115.9% 1236 ± 25% numa-meminfo.node0.Writeback 15098140 ± 3% -17.7% 12425670 ± 3% numa-meminfo.node1.FilePages 11599549 ± 7% -22.3% 9018051 ± 4% numa-meminfo.node1.Inactive 105067 ± 92% -99.3% 725.33 ± 51% numa-meminfo.node1.Inactive(anon) 11494482 ± 7% -21.6% 9017325 ± 4% numa-meminfo.node1.Inactive(file) 5504 ± 9% -12.5% 4819 ± 2% numa-meminfo.node1.KernelStack 6709938 ± 11% +45.8% 9781880 ± 5% numa-meminfo.node1.MemFree 18029136 ± 4% -17.0% 14957200 ± 3% numa-meminfo.node1.MemUsed 4427 ± 69% -77.5% 996.00 ± 8% numa-meminfo.node1.PageTables 857660 ± 32% -37.2% 538646 ± 8% numa-meminfo.node1.SUnreclaim 109232 ± 88% -96.1% 4274 ± 10% numa-meminfo.node1.Shmem 986740 ± 35% -40.0% 592452 ± 10% numa-meminfo.node1.Slab 575.21 ± 15% +30.1% 748.58 ± 2% sched_debug.cfs_rq:/.runnable_load_avg.max 100.45 ± 15% +28.6% 129.19 ± 5% sched_debug.cfs_rq:/.runnable_load_avg.stddev 830.19 ± 9% +15.7% 960.25 ± 6% sched_debug.cfs_rq:/.util_avg.max 29.97 ± 16% -25.2% 22.43 ± 31% sched_debug.cfs_rq:/.util_est_enqueued.avg 591.17 ± 13% -22.2% 459.83 ± 18% sched_debug.cfs_rq:/.util_est_enqueued.max 107.52 ± 13% -23.0% 82.84 ± 23% sched_debug.cfs_rq:/.util_est_enqueued.stddev 2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock.stddev 2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock_task.stddev 24.51 ± 7% +21.5% 29.77 ± 8% sched_debug.cpu.cpu_load[0].avg 541.29 ± 15% +38.3% 748.58 ± 2% sched_debug.cpu.cpu_load[0].max 93.97 ± 7% +27.8% 120.09 ± 2% sched_debug.cpu.cpu_load[0].stddev 519.08 ± 6% +40.5% 729.42 ± 4% sched_debug.cpu.cpu_load[1].max 87.45 ± 5% +28.5% 112.39 ± 4% sched_debug.cpu.cpu_load[1].stddev 480.83 ± 7% +39.5% 670.92 ± 13% sched_debug.cpu.cpu_load[2].max 80.92 ± 5% +25.7% 101.75 ± 8% sched_debug.cpu.cpu_load[2].stddev 408.83 ± 4% +54.0% 629.58 ± 19% sched_debug.cpu.cpu_load[3].max 71.62 ± 2% +30.7% 93.61 ± 11% sched_debug.cpu.cpu_load[3].stddev 381.50 ± 8% +66.7% 636.00 ± 18% sched_debug.cpu.cpu_load[4].max 64.40 ± 6% +39.7% 89.96 ± 12% sched_debug.cpu.cpu_load[4].stddev 106.75 ± 20% -88.4% 12.33 ± 10% proc-vmstat.kswapd_high_wmark_hit_quickly 2508660 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_active_file 33315360 -30.4% 23192301 proc-vmstat.nr_dirtied 133447 ± 5% -12.6% 116580 ± 6% proc-vmstat.nr_dirty 7881043 ± 2% -10.9% 7019533 ± 2% proc-vmstat.nr_file_pages 28790 ± 4% +61.1% 46392 ± 3% proc-vmstat.nr_free_cma 2934503 ± 7% +30.2% 3821369 ± 4% proc-vmstat.nr_free_pages 5044168 -3.4% 4872965 proc-vmstat.nr_inactive_file 430282 -6.0% 404589 proc-vmstat.nr_slab_unreclaimable 197.75 ± 38% +136.5% 467.67 ± 10% proc-vmstat.nr_writeback 33243480 -30.4% 23143756 proc-vmstat.nr_written 2508665 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_zone_active_file 5044399 -3.4% 4873026 proc-vmstat.nr_zone_inactive_file 134972 ± 5% -12.4% 118262 ± 5% proc-vmstat.nr_zone_write_pending 15632301 ± 5% -25.8% 11598535 ± 3% proc-vmstat.numa_hit 15608826 ± 5% -25.8% 11575029 ± 3% proc-vmstat.numa_local 2300530 ± 10% -31.1% 1585363 ± 10% proc-vmstat.pgactivate 16955745 -26.3% 12499611 ± 2% proc-vmstat.pgalloc_normal 17488000 -25.9% 12959812 ± 2% proc-vmstat.pgfree 1.33e+08 -30.4% 92575440 proc-vmstat.pgpgout 5728177 ± 6% -78.5% 1232433 ± 25% proc-vmstat.pgscan_kswapd 5727967 ± 6% -78.5% 1232244 ± 25% proc-vmstat.pgsteal_kswapd 32760 -7.6% 30285 proc-vmstat.slabs_scanned 1768211 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_active_file 10410188 ± 7% -32.5% 7023939 ± 7% numa-vmstat.node0.nr_dirtied 76995 ± 6% -24.6% 58035 ± 10% numa-vmstat.node0.nr_dirty 2167457 ± 12% +20.7% 2616147 ± 2% numa-vmstat.node0.nr_inactive_file 142.00 ± 24% +120.4% 313.00 ± 24% numa-vmstat.node0.nr_writeback 10308758 ± 7% -32.6% 6949497 ± 7% numa-vmstat.node0.nr_written 1768216 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_zone_active_file 2167563 ± 12% +20.7% 2616189 ± 2% numa-vmstat.node0.nr_zone_inactive_file 77786 ± 7% -24.3% 58899 ± 10% numa-vmstat.node0.nr_zone_write_pending 5889240 ± 3% -14.3% 5046874 ± 3% numa-vmstat.node0.numa_hit 5837851 ± 4% -14.0% 5023120 ± 3% numa-vmstat.node0.numa_local 68376 ± 48% -96.9% 2149 ± 46% numa-vmstat.node0.numa_miss 119766 ± 42% -78.4% 25904 ± 3% numa-vmstat.node0.numa_other 30.50 ± 38% -100.0% 0.00 numa-vmstat.node0.workingset_nodes 6829126 ± 6% -22.8% 5269898 ± 10% numa-vmstat.node1.nr_dirtied 3774645 ± 3% -17.7% 3106081 ± 3% numa-vmstat.node1.nr_file_pages 28796 ± 4% +61.1% 46405 ± 3% numa-vmstat.node1.nr_free_cma 1677332 ± 11% +45.8% 2445778 ± 5% numa-vmstat.node1.nr_free_pages 26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_inactive_anon 2873702 ± 7% -21.6% 2254116 ± 4% numa-vmstat.node1.nr_inactive_file 5506 ± 9% -12.5% 4820 ± 2% numa-vmstat.node1.nr_kernel_stack 29031 ± 85% -88.2% 3436 numa-vmstat.node1.nr_mapped 1106 ± 69% -77.5% 249.00 ± 8% numa-vmstat.node1.nr_page_table_pages 27308 ± 88% -96.1% 1068 ± 10% numa-vmstat.node1.nr_shmem 214418 ± 32% -37.2% 134663 ± 8% numa-vmstat.node1.nr_slab_unreclaimable 75.00 ± 35% +145.3% 184.00 ± 18% numa-vmstat.node1.nr_writeback 6761224 ± 6% -23.1% 5202586 ± 10% numa-vmstat.node1.nr_written 26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_zone_inactive_anon 2873810 ± 7% -21.6% 2254135 ± 4% numa-vmstat.node1.nr_zone_inactive_file 68381 ± 48% -96.9% 2152 ± 46% numa-vmstat.node1.numa_foreign 5035283 ± 15% -26.9% 3679318 ± 7% numa-vmstat.node1.numa_hit 4905249 ± 15% -28.2% 3522229 ± 8% numa-vmstat.node1.numa_local 18.43 ± 6% -28.3% 13.21 ± 3% perf-stat.i.MPKI 2.205e+09 ± 3% +10.5% 2.437e+09 perf-stat.i.branch-instructions 1.83 ± 5% -0.6 1.24 ± 7% perf-stat.i.branch-miss-rate% 37019551 ± 3% -29.4% 26140844 ± 5% perf-stat.i.branch-misses 28932020 ± 6% -31.1% 19933819 ± 3% perf-stat.i.cache-misses 2.03e+08 ± 4% -24.8% 1.526e+08 perf-stat.i.cache-references 645248 ± 2% -27.4% 468301 perf-stat.i.context-switches 1.97 ± 7% -22.1% 1.53 ± 2% perf-stat.i.cpi 1.983e+10 ± 3% -15.1% 1.684e+10 perf-stat.i.cpu-cycles 1.03 ± 5% -0.4 0.64 ± 10% perf-stat.i.dTLB-load-miss-rate% 31879502 ± 6% -37.9% 19807598 ± 9% perf-stat.i.dTLB-load-misses 3.201e+09 ± 3% +18.8% 3.804e+09 perf-stat.i.dTLB-loads 0.13 ± 7% -0.0 0.09 ± 10% perf-stat.i.dTLB-store-miss-rate% 2516181 ± 10% -28.3% 1805272 ± 10% perf-stat.i.dTLB-store-misses 2.143e+09 ± 3% +19.8% 2.568e+09 ± 2% perf-stat.i.dTLB-stores 13781489 ± 3% -28.4% 9866239 ± 3% perf-stat.i.iTLB-loads 1.164e+10 ± 3% +21.6% 1.415e+10 perf-stat.i.instructions 8435 ± 17% +42.5% 12023 ± 14% perf-stat.i.instructions-per-iTLB-miss 0.57 ± 3% +43.6% 0.82 perf-stat.i.ipc 16050361 ± 9% -30.7% 11127721 ± 3% perf-stat.i.node-load-misses 8120610 ± 9% -31.3% 5580996 perf-stat.i.node-loads 2591649 ± 13% -31.1% 1785932 ± 6% perf-stat.i.node-store-misses 1809775 ± 12% -31.2% 1244360 ± 2% perf-stat.i.node-stores 17.45 ± 4% -38.2% 10.79 perf-stat.overall.MPKI 1.68 ± 3% -0.6 1.07 ± 5% perf-stat.overall.branch-miss-rate% 1.70 ± 3% -30.2% 1.19 perf-stat.overall.cpi 689.69 ± 9% +22.6% 845.48 ± 2% perf-stat.overall.cycles-between-cache-misses 0.99 ± 5% -0.5 0.52 ± 10% perf-stat.overall.dTLB-load-miss-rate% 0.12 ± 8% -0.0 0.07 ± 10% perf-stat.overall.dTLB-store-miss-rate% 10.81 ± 14% +2.3 13.09 ± 10% perf-stat.overall.iTLB-load-miss-rate% 7119 ± 14% +35.9% 9679 ± 13% perf-stat.overall.instructions-per-iTLB-miss 0.59 ± 3% +43.1% 0.84 perf-stat.overall.ipc 202912 ± 2% +68.8% 342615 ± 3% perf-stat.overall.path-length 2.195e+09 ± 3% +10.5% 2.425e+09 perf-stat.ps.branch-instructions 36847010 ± 3% -29.4% 26015686 ± 5% perf-stat.ps.branch-misses 28799092 ± 6% -31.1% 19837306 ± 3% perf-stat.ps.cache-misses 2.021e+08 ± 4% -24.8% 1.519e+08 perf-stat.ps.cache-references 642266 ± 2% -27.4% 465968 perf-stat.ps.context-switches 1.974e+10 ± 3% -15.1% 1.676e+10 perf-stat.ps.cpu-cycles 31729472 ± 6% -37.9% 19715594 ± 9% perf-stat.ps.dTLB-load-misses 3.186e+09 ± 3% +18.8% 3.785e+09 perf-stat.ps.dTLB-loads 2504443 ± 10% -28.2% 1797154 ± 10% perf-stat.ps.dTLB-store-misses 2.133e+09 ± 3% +19.8% 2.556e+09 ± 2% perf-stat.ps.dTLB-stores 13717699 ± 3% -28.4% 9816979 ± 3% perf-stat.ps.iTLB-loads 1.159e+10 ± 3% +21.5% 1.408e+10 perf-stat.ps.instructions 15979371 ± 9% -30.7% 11073468 ± 3% perf-stat.ps.node-load-misses 8081205 ± 9% -31.3% 5554114 perf-stat.ps.node-loads 2580198 ± 13% -31.1% 1777169 ± 6% perf-stat.ps.node-store-misses 1801590 ± 12% -31.3% 1238321 ± 2% perf-stat.ps.node-stores 2.579e+12 +20.6% 3.111e+12 ± 2% perf-stat.total.instructions 36149 ± 4% -26.2% 26686 ± 5% softirqs.CPU0.RCU 37255 ± 3% -26.8% 27256 ± 2% softirqs.CPU1.RCU 37834 ± 5% -26.5% 27801 ± 9% softirqs.CPU10.RCU 39075 ± 3% -24.5% 29494 ± 2% softirqs.CPU11.RCU 38137 ± 6% -23.1% 29340 ± 4% softirqs.CPU12.RCU 37480 ± 3% -20.5% 29797 ± 2% softirqs.CPU13.RCU 37690 ± 5% -28.7% 26861 ± 4% softirqs.CPU14.RCU 33863 ± 9% -29.5% 23885 ± 5% softirqs.CPU15.RCU 34306 ± 9% -31.1% 23641 ± 12% softirqs.CPU16.RCU 34374 ± 9% -29.0% 24402 ± 7% softirqs.CPU17.RCU 44013 ± 9% -31.7% 30059 softirqs.CPU18.RCU 43114 ± 8% -29.9% 30219 softirqs.CPU19.RCU 36276 ± 8% -23.3% 27825 ± 5% softirqs.CPU2.RCU 67049 ± 7% +39.1% 93267 ± 18% softirqs.CPU2.TIMER 39721 ± 13% -31.4% 27233 ± 3% softirqs.CPU20.RCU 42340 ± 5% -28.6% 30215 ± 6% softirqs.CPU21.RCU 40896 ± 7% -30.3% 28506 ± 2% softirqs.CPU22.RCU 42422 ± 7% -33.2% 28357 ± 2% softirqs.CPU23.RCU 42330 ± 9% -33.5% 28150 ± 5% softirqs.CPU24.RCU 41406 ± 9% -31.4% 28398 softirqs.CPU25.RCU 39754 ± 5% -29.4% 28063 softirqs.CPU26.RCU 40207 ± 6% -27.9% 28995 ± 12% softirqs.CPU27.RCU 40351 ± 9% -29.4% 28487 ± 3% softirqs.CPU28.RCU 41880 ± 5% -29.6% 29472 softirqs.CPU29.RCU 36293 ± 4% -17.7% 29864 ± 7% softirqs.CPU3.RCU 39341 ± 9% -34.3% 25847 softirqs.CPU30.RCU 39687 ± 9% -33.5% 26400 ± 2% softirqs.CPU31.RCU 38062 ± 14% -39.9% 22878 ± 10% softirqs.CPU32.RCU 39769 ± 13% -33.8% 26314 ± 5% softirqs.CPU33.RCU 39504 ± 10% -37.2% 24824 ± 6% softirqs.CPU34.RCU 39591 ± 11% -38.6% 24300 ± 5% softirqs.CPU35.RCU 40398 ± 8% -27.0% 29488 ± 12% softirqs.CPU36.RCU 39464 ± 12% -25.3% 29478 ± 10% softirqs.CPU37.RCU 41177 ± 9% -34.0% 27190 ± 19% softirqs.CPU38.RCU 38734 ± 13% -24.8% 29145 ± 5% softirqs.CPU39.RCU 36804 ± 5% -23.0% 28354 softirqs.CPU4.RCU 40421 ± 12% -28.8% 28776 ± 6% softirqs.CPU40.RCU 40371 ± 11% -31.1% 27834 ± 8% softirqs.CPU41.RCU 40633 ± 10% -29.9% 28469 ± 10% softirqs.CPU42.RCU 41018 ± 9% -33.6% 27252 ± 6% softirqs.CPU43.RCU 38181 ± 12% -27.0% 27856 ± 10% softirqs.CPU44.RCU 39605 ± 12% -32.3% 26803 ± 12% softirqs.CPU45.RCU 42686 ± 11% -29.7% 30016 ± 9% softirqs.CPU47.RCU 40433 ± 9% -31.8% 27586 ± 11% softirqs.CPU48.RCU 38371 ± 12% -23.0% 29528 ± 7% softirqs.CPU49.RCU 37172 ± 2% -25.9% 27562 ± 3% softirqs.CPU5.RCU 39660 ± 12% -27.5% 28752 ± 11% softirqs.CPU50.RCU 40375 ± 10% -26.7% 29582 ± 3% softirqs.CPU51.RCU 40568 ± 9% -28.8% 28886 ± 6% softirqs.CPU52.RCU 39967 ± 10% -27.3% 29073 ± 11% softirqs.CPU53.RCU 40874 ± 11% -37.1% 25725 ± 6% softirqs.CPU54.RCU 41227 ± 14% -37.9% 25606 ± 6% softirqs.CPU55.RCU 38856 ± 13% -34.7% 25380 ± 8% softirqs.CPU56.RCU 39618 ± 13% -39.1% 24119 ± 4% softirqs.CPU57.RCU 39412 ± 12% -36.6% 24970 ± 3% softirqs.CPU58.RCU 41572 ± 14% -41.3% 24399 ± 10% softirqs.CPU59.RCU 36927 ± 3% -24.4% 27923 ± 22% softirqs.CPU6.RCU 37057 ± 10% -31.8% 25261 ± 2% softirqs.CPU60.RCU 36584 ± 8% -27.8% 26419 ± 2% softirqs.CPU61.RCU 35859 ± 4% -28.2% 25729 ± 5% softirqs.CPU62.RCU 35849 ± 7% -33.2% 23933 ± 5% softirqs.CPU63.RCU 35379 ± 6% -26.4% 26041 ± 4% softirqs.CPU64.RCU 36908 ± 6% -26.5% 27110 ± 2% softirqs.CPU65.RCU 37595 ± 3% -26.1% 27783 ± 4% softirqs.CPU66.RCU 36327 ± 2% -27.0% 26533 ± 2% softirqs.CPU67.RCU 35136 ± 7% -31.7% 23998 ± 9% softirqs.CPU68.RCU 35638 ± 4% -28.6% 25435 ± 4% softirqs.CPU69.RCU 37268 ± 6% -25.4% 27786 ± 4% softirqs.CPU7.RCU 36693 ± 4% -29.2% 25986 ± 5% softirqs.CPU70.RCU 36859 ± 3% -28.8% 26260 ± 4% softirqs.CPU71.RCU 37539 ± 7% -27.7% 27144 ± 4% softirqs.CPU8.RCU 37584 ± 4% -26.0% 27802 ± 15% softirqs.CPU9.RCU 2795313 ± 7% -29.7% 1964363 ± 4% softirqs.RCU 249.50 ± 76% -50.4% 123.67 ± 2% interrupts.41:PCI-MSI.1572870-edge.eth0-TxRx-6 154.50 ± 13% -10.9% 137.67 ± 10% interrupts.42:PCI-MSI.1572871-edge.eth0-TxRx-7 150.50 ± 43% -28.0% 108.33 interrupts.84:PCI-MSI.1572913-edge.eth0-TxRx-49 10393 ± 10% -26.4% 7653 ± 10% interrupts.CPU1.RES:Rescheduling_interrupts 12518 ± 15% -31.0% 8642 ± 21% interrupts.CPU10.RES:Rescheduling_interrupts 945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.NMI:Non-maskable_interrupts 945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.PMI:Performance_monitoring_interrupts 13068 ± 18% -30.6% 9075 ± 23% interrupts.CPU11.RES:Rescheduling_interrupts 11949 ± 11% -49.8% 6001 ± 29% interrupts.CPU12.RES:Rescheduling_interrupts 13711 ± 8% -19.4% 11052 ± 24% interrupts.CPU13.RES:Rescheduling_interrupts 13361 ± 6% -38.8% 8170 ± 25% interrupts.CPU14.RES:Rescheduling_interrupts 7628 ± 12% -29.2% 5401 ± 7% interrupts.CPU18.RES:Rescheduling_interrupts 7909 ± 25% -30.4% 5501 ± 37% interrupts.CPU19.RES:Rescheduling_interrupts 1482 ± 39% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts 1482 ± 39% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts 2116 +23.2% 2606 ± 10% interrupts.CPU20.CAL:Function_call_interrupts 194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.NMI:Non-maskable_interrupts 194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.PMI:Performance_monitoring_interrupts 2096 ± 4% +18.0% 2474 ± 9% interrupts.CPU21.CAL:Function_call_interrupts 11313 ± 20% -56.4% 4935 ± 23% interrupts.CPU21.RES:Rescheduling_interrupts 2111 ± 3% +20.1% 2534 ± 11% interrupts.CPU22.CAL:Function_call_interrupts 2109 ± 4% +22.6% 2585 ± 10% interrupts.CPU23.CAL:Function_call_interrupts 10305 ± 16% -40.7% 6115 ± 17% interrupts.CPU23.RES:Rescheduling_interrupts 2128 ± 2% +17.8% 2507 ± 10% interrupts.CPU24.CAL:Function_call_interrupts 12271 ± 20% -39.0% 7488 ± 29% interrupts.CPU24.RES:Rescheduling_interrupts 2123 +19.7% 2541 ± 7% interrupts.CPU25.CAL:Function_call_interrupts 9780 ± 21% -32.9% 6563 ± 25% interrupts.CPU25.RES:Rescheduling_interrupts 980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.NMI:Non-maskable_interrupts 980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.PMI:Performance_monitoring_interrupts 11163 ± 28% -49.8% 5605 ± 13% interrupts.CPU27.RES:Rescheduling_interrupts 11071 ± 12% -56.3% 4835 ± 26% interrupts.CPU29.RES:Rescheduling_interrupts 728.50 ± 66% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts 728.50 ± 66% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts 10599 ± 10% -38.7% 6501 ± 16% interrupts.CPU31.RES:Rescheduling_interrupts 2213 ± 4% +14.2% 2527 ± 7% interrupts.CPU32.CAL:Function_call_interrupts 8862 ± 16% -39.6% 5354 ± 28% interrupts.CPU32.RES:Rescheduling_interrupts 2156 ± 6% +20.5% 2599 ± 9% interrupts.CPU33.CAL:Function_call_interrupts 2169 ± 5% +19.9% 2601 ± 9% interrupts.CPU34.CAL:Function_call_interrupts 10483 ± 18% -48.8% 5371 ± 33% interrupts.CPU34.RES:Rescheduling_interrupts 2049 +24.4% 2549 ± 8% interrupts.CPU35.CAL:Function_call_interrupts 10929 ± 15% -49.0% 5578 ± 14% interrupts.CPU35.RES:Rescheduling_interrupts 1993 ± 4% +28.5% 2561 ± 12% interrupts.CPU36.CAL:Function_call_interrupts 161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.NMI:Non-maskable_interrupts 161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.PMI:Performance_monitoring_interrupts 1987 ± 4% +30.5% 2594 ± 13% interrupts.CPU37.CAL:Function_call_interrupts 1863 ± 10% +35.7% 2529 ± 14% interrupts.CPU38.CAL:Function_call_interrupts 1984 ± 6% +26.5% 2509 ± 12% interrupts.CPU41.CAL:Function_call_interrupts 12830 ± 30% -26.4% 9445 ± 9% interrupts.CPU43.RES:Rescheduling_interrupts 2067 ± 2% +27.4% 2632 ± 9% interrupts.CPU44.CAL:Function_call_interrupts 2066 ± 3% +27.2% 2629 ± 9% interrupts.CPU45.CAL:Function_call_interrupts 2158 ± 3% +19.6% 2582 ± 9% interrupts.CPU46.CAL:Function_call_interrupts 2036 ± 6% +30.3% 2652 ± 10% interrupts.CPU47.CAL:Function_call_interrupts 150.50 ± 43% -28.0% 108.33 interrupts.CPU49.84:PCI-MSI.1572913-edge.eth0-TxRx-49 11823 ± 12% -36.9% 7460 ± 18% interrupts.CPU49.RES:Rescheduling_interrupts 1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.NMI:Non-maskable_interrupts 1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.PMI:Performance_monitoring_interrupts 11748 ± 22% -40.0% 7048 ± 27% interrupts.CPU5.RES:Rescheduling_interrupts 2102 ± 5% +19.6% 2515 ± 7% interrupts.CPU52.CAL:Function_call_interrupts 2115 ± 6% +19.5% 2527 ± 7% interrupts.CPU53.CAL:Function_call_interrupts 11318 ± 17% -20.4% 9004 ± 26% interrupts.CPU53.RES:Rescheduling_interrupts 2177 ± 5% +17.4% 2556 ± 9% interrupts.CPU54.CAL:Function_call_interrupts 10968 ± 23% -62.3% 4134 ± 21% interrupts.CPU54.RES:Rescheduling_interrupts 2161 ± 5% +20.2% 2598 ± 9% interrupts.CPU55.CAL:Function_call_interrupts 9334 ± 19% -55.4% 4158 ± 34% interrupts.CPU55.RES:Rescheduling_interrupts 1443 ± 22% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts 1443 ± 22% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts 11377 ± 31% -56.9% 4907 ± 43% interrupts.CPU56.RES:Rescheduling_interrupts 601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.NMI:Non-maskable_interrupts 601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.PMI:Performance_monitoring_interrupts 8295 ± 31% -47.7% 4340 ± 24% interrupts.CPU57.RES:Rescheduling_interrupts 9182 ± 37% -59.7% 3701 ± 46% interrupts.CPU58.RES:Rescheduling_interrupts 249.50 ± 76% -50.4% 123.67 ± 2% interrupts.CPU6.41:PCI-MSI.1572870-edge.eth0-TxRx-6 12473 ± 27% -44.4% 6940 ± 13% interrupts.CPU6.RES:Rescheduling_interrupts 7886 ± 12% -42.2% 4555 ± 31% interrupts.CPU60.RES:Rescheduling_interrupts 971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.NMI:Non-maskable_interrupts 971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.PMI:Performance_monitoring_interrupts 9253 ± 26% -56.4% 4033 ± 16% interrupts.CPU61.RES:Rescheduling_interrupts 2103 ± 5% +19.1% 2505 ± 11% interrupts.CPU63.CAL:Function_call_interrupts 2059 ± 2% +20.6% 2483 ± 10% interrupts.CPU64.CAL:Function_call_interrupts 8545 ± 17% -41.2% 5025 ± 8% interrupts.CPU65.RES:Rescheduling_interrupts 7378 ± 23% -51.0% 3614 ± 30% interrupts.CPU68.RES:Rescheduling_interrupts 8573 ± 28% -50.1% 4277 ± 6% interrupts.CPU69.RES:Rescheduling_interrupts 154.50 ± 13% -10.9% 137.67 ± 10% interrupts.CPU7.42:PCI-MSI.1572871-edge.eth0-TxRx-7 9910 ± 28% -51.2% 4839 ± 40% interrupts.CPU70.RES:Rescheduling_interrupts 8836 ± 26% -52.7% 4180 ± 6% interrupts.CPU71.RES:Rescheduling_interrupts 911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.NMI:Non-maskable_interrupts 911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.PMI:Performance_monitoring_interrupts 12326 ± 15% -32.3% 8350 ± 10% interrupts.CPU8.RES:Rescheduling_interrupts 1392 ± 58% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts 1392 ± 58% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts 60185 ± 19% -30.5% 41822 ± 11% interrupts.NMI:Non-maskable_interrupts 60185 ± 19% -30.5% 41822 ± 11% interrupts.PMI:Performance_monitoring_interrupts 749419 -31.9% 510225 ± 3% interrupts.RES:Rescheduling_interrupts 15.05 ± 6% -4.1 10.97 ± 20% perf-profile.calltrace.cycles-pp.normal_work_helper.process_one_work.worker_thread.kthread.ret_from_fork 12.66 ± 5% -4.0 8.68 ± 20% perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread.kthread 5.99 ± 8% -1.6 4.37 ± 6% perf-profile.calltrace.cycles-pp.io_submit 5.83 ± 8% -1.6 4.25 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.io_submit 5.82 ± 8% -1.6 4.24 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.81 ± 8% -1.6 4.23 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.76 ± 8% -1.6 4.19 ± 6% perf-profile.calltrace.cycles-pp.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.65 ± 6% -1.6 4.08 ± 20% perf-profile.calltrace.cycles-pp.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread 4.52 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread 6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback 6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes 5.58 ± 8% -1.5 4.07 ± 6% perf-profile.calltrace.cycles-pp.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.50 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work 5.44 ± 8% -1.5 3.96 ± 7% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64 6.63 ± 8% -1.5 5.17 ± 10% perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode 5.34 ± 9% -1.4 3.90 ± 6% perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit 5.55 ± 8% -1.2 4.32 ± 10% perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages 4.98 ± 8% -1.1 3.86 ± 10% perf-profile.calltrace.cycles-pp.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages 5.00 ± 8% -1.1 3.88 ± 10% perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages 2.80 ± 4% -1.0 1.78 ± 22% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work 1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.__btrfs_cow_block.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io 1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper 2.32 ± 3% -0.6 1.72 ± 18% perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper 0.91 ± 8% -0.5 0.42 ± 73% perf-profile.calltrace.cycles-pp.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper 0.88 ± 8% -0.5 0.41 ± 74% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io 0.83 ± 5% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums 1.90 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread 1.89 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work 0.82 ± 4% -0.4 0.42 ± 71% perf-profile.calltrace.cycles-pp.run_one_async_done.normal_work_helper.process_one_work.worker_thread.kthread 0.81 ± 4% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.btrfs_map_bio.run_one_async_done.normal_work_helper.process_one_work.worker_thread 0.74 ± 6% -0.4 0.36 ± 70% perf-profile.calltrace.cycles-pp.push_leaf_right.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks 1.19 ± 3% -0.4 0.83 ± 16% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io 1.00 ± 6% -0.4 0.65 ± 7% perf-profile.calltrace.cycles-pp.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.27 ± 5% -0.4 0.92 ± 7% perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.25 ± 6% -0.3 0.90 ± 7% perf-profile.calltrace.cycles-pp.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 0.93 ± 8% -0.3 0.59 ± 7% perf-profile.calltrace.cycles-pp.__set_extent_bit.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter 0.94 ± 7% -0.3 0.60 ± 6% perf-profile.calltrace.cycles-pp.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write 0.70 ± 8% -0.3 0.37 ± 70% perf-profile.calltrace.cycles-pp.__lookup_extent_mapping.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter 0.79 ± 6% -0.3 0.47 ± 71% perf-profile.calltrace.cycles-pp.setup_items_for_insert.btrfs_duplicate_item.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper 0.71 ± 21% -0.3 0.43 ± 74% perf-profile.calltrace.cycles-pp.__btrfs_free_extent.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space 1.15 ± 11% -0.3 0.88 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.15 ± 11% -0.3 0.88 ± 8% perf-profile.calltrace.cycles-pp.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 1.43 ± 13% -0.3 1.17 ± 5% perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.06 ± 5% -0.3 0.81 ± 7% perf-profile.calltrace.cycles-pp.btrfs_drop_extent_cache.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.30 ± 15% -0.2 1.06 ± 4% perf-profile.calltrace.cycles-pp.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write 1.03 ± 13% -0.2 0.79 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range 1.20 ± 4% -0.2 0.96 ± 15% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 1.24 ± 3% -0.2 1.01 ± 15% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.05 ± 8% -0.2 0.82 ± 12% perf-profile.calltrace.cycles-pp.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages 0.88 ± 13% -0.2 0.71 ± 2% perf-profile.calltrace.cycles-pp.wait_reserve_ticket.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter 1.01 ± 9% -0.2 0.84 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.01 ± 10% -0.2 0.85 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 0.72 ± 7% -0.2 0.56 ± 6% perf-profile.calltrace.cycles-pp.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write 0.75 ± 8% -0.1 0.60 ± 10% perf-profile.calltrace.cycles-pp.submit_extent_page.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages 0.92 ± 15% +0.2 1.14 ± 3% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 1.01 ± 15% +0.2 1.25 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 1.15 ± 15% +0.3 1.44 ± 6% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.27 ±100% +0.4 0.63 ± 4% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 1.74 ± 9% +0.4 2.11 ± 2% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 2.63 ± 10% +0.6 3.26 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle 0.00 +0.8 0.75 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook 2.37 ± 19% +0.8 3.19 ± 5% perf-profile.calltrace.cycles-pp.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread.ret_from_fork 2.33 ± 19% +0.8 3.17 ± 5% perf-profile.calltrace.cycles-pp.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread 0.00 +0.9 0.90 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook 4.71 ± 11% +1.0 5.74 perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry 5.04 ± 10% +1.2 6.19 perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 0.00 +1.3 1.30 ± 19% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range 0.00 +1.4 1.36 ± 19% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents 0.29 ±101% +1.4 1.67 ± 18% perf-profile.calltrace.cycles-pp.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space 0.00 +1.5 1.52 ± 18% perf-profile.calltrace.cycles-pp.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio 0.00 +2.2 2.20 ± 14% perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work.worker_thread 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work 49.45 ± 2% +4.3 53.79 ± 4% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.calltrace.cycles-pp.secondary_startup_64 1.07 ± 18% +5.0 6.06 ± 20% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback 0.93 ± 18% +5.0 5.95 ± 20% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes 61.26 ± 2% +5.1 66.36 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 61.27 ± 2% +5.1 66.37 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 61.21 ± 2% +5.1 66.32 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.33 ±100% +5.3 5.68 ± 20% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode 55.45 ± 2% +5.4 60.88 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.00 +6.4 6.39 ± 13% perf-profile.calltrace.cycles-pp.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page 0.00 +6.5 6.50 ± 13% perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb 0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages 0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages 15.05 ± 6% -4.1 10.97 ± 20% perf-profile.children.cycles-pp.normal_work_helper 12.66 ± 5% -4.0 8.68 ± 20% perf-profile.children.cycles-pp.btrfs_finish_ordered_io 7.84 ± 5% -2.4 5.39 ± 10% perf-profile.children.cycles-pp.btrfs_search_slot 7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_writepages 7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_write_cache_pages 7.03 ± 7% -1.6 5.38 ± 12% perf-profile.children.cycles-pp.__extent_writepage 6.19 ± 8% -1.6 4.56 ± 7% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 6.17 ± 8% -1.6 4.55 ± 7% perf-profile.children.cycles-pp.do_syscall_64 6.01 ± 8% -1.6 4.39 ± 7% perf-profile.children.cycles-pp.io_submit 5.81 ± 8% -1.6 4.23 ± 6% perf-profile.children.cycles-pp.__x64_sys_io_submit 5.76 ± 8% -1.6 4.19 ± 6% perf-profile.children.cycles-pp.io_submit_one 5.66 ± 5% -1.6 4.09 ± 20% perf-profile.children.cycles-pp.btrfs_mark_extent_written 4.52 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.add_pending_csums 5.59 ± 8% -1.5 4.07 ± 6% perf-profile.children.cycles-pp.aio_write 4.51 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.btrfs_csum_file_blocks 5.44 ± 8% -1.5 3.96 ± 7% perf-profile.children.cycles-pp.btrfs_file_write_iter 5.34 ± 9% -1.4 3.90 ± 6% perf-profile.children.cycles-pp.btrfs_buffered_write 5.88 ± 7% -1.4 4.47 ± 11% perf-profile.children.cycles-pp.writepage_delalloc 5.30 ± 8% -1.3 4.01 ± 11% perf-profile.children.cycles-pp.btrfs_run_delalloc_range 5.28 ± 8% -1.3 3.99 ± 11% perf-profile.children.cycles-pp.run_delalloc_nocow 2.55 ± 2% -0.9 1.67 ± 16% perf-profile.children.cycles-pp.btrfs_cow_block 2.54 ± 2% -0.9 1.66 ± 16% perf-profile.children.cycles-pp.__btrfs_cow_block 2.15 ± 13% -0.8 1.31 ± 8% perf-profile.children.cycles-pp.__lookup_extent_mapping 1.87 ± 12% -0.8 1.04 ± 12% perf-profile.children.cycles-pp.__etree_search 0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.printk 0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.vprintk_emit 2.60 ± 2% -0.7 1.92 ± 14% perf-profile.children.cycles-pp.btrfs_insert_empty_items 3.00 ± 5% -0.7 2.33 ± 6% perf-profile.children.cycles-pp._raw_spin_lock 1.89 ± 2% -0.7 1.23 ± 7% perf-profile.children.cycles-pp.__set_extent_bit 2.78 ± 4% -0.6 2.14 ± 14% perf-profile.children.cycles-pp.__schedule 1.57 ± 2% -0.5 1.02 ± 9% perf-profile.children.cycles-pp.lock_extent_bits 1.19 ± 14% -0.5 0.70 ± 11% perf-profile.children.cycles-pp.__clear_extent_bit 1.51 ± 7% -0.5 1.04 ± 9% perf-profile.children.cycles-pp.read_block_for_search 2.08 ± 5% -0.5 1.62 ± 18% perf-profile.children.cycles-pp.setup_items_for_insert 2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.btrfs_run_delayed_refs 2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.__btrfs_run_delayed_refs 1.60 ± 6% -0.4 1.20 ± 12% perf-profile.children.cycles-pp.schedule 1.95 ± 11% -0.4 1.58 ± 12% perf-profile.children.cycles-pp.try_to_wake_up 1.31 ± 6% -0.4 0.94 ± 9% perf-profile.children.cycles-pp.create_io_em 0.99 ± 4% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush 1.00 ± 6% -0.4 0.65 ± 7% perf-profile.children.cycles-pp.lock_and_cleanup_extent_if_need 1.27 ± 9% -0.4 0.92 ± 11% perf-profile.children.cycles-pp.__wake_up_common_lock 0.98 ± 3% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.btrfs_alloc_tree_block 1.27 ± 5% -0.4 0.92 ± 7% perf-profile.children.cycles-pp.btrfs_dirty_pages 1.34 ± 7% -0.3 0.99 ± 10% perf-profile.children.cycles-pp.btrfs_map_bio 1.11 ± 9% -0.3 0.77 ± 11% perf-profile.children.cycles-pp.generic_bin_search 1.16 ± 6% -0.3 0.82 ± 9% perf-profile.children.cycles-pp.find_extent_buffer 0.91 ± 7% -0.3 0.57 ± 21% perf-profile.children.cycles-pp.btrfs_lookup_csum 1.21 ± 11% -0.3 0.90 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_csums_range 1.21 ± 11% -0.3 0.91 ± 9% perf-profile.children.cycles-pp.csum_exist_in_range 1.10 ± 10% -0.3 0.80 ± 9% perf-profile.children.cycles-pp.__wake_up_common 0.37 ± 70% -0.3 0.07 ± 18% perf-profile.children.cycles-pp.test_range_bit 1.61 ± 4% -0.3 1.32 ± 15% perf-profile.children.cycles-pp.push_leaf_right 0.60 ± 4% -0.3 0.31 ± 25% perf-profile.children.cycles-pp.btrfs_extend_item 1.02 ± 9% -0.3 0.73 ± 10% perf-profile.children.cycles-pp.autoremove_wake_function 1.12 ± 4% -0.3 0.84 ± 9% perf-profile.children.cycles-pp.btrfs_drop_extent_cache 0.82 ± 6% -0.3 0.55 ± 4% perf-profile.children.cycles-pp.btrfs_release_path 0.85 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_extent_buffer_full 0.99 ± 4% -0.3 0.73 ± 13% perf-profile.children.cycles-pp.btrfs_set_token_32 1.44 ± 13% -0.3 1.18 ± 5% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata 0.84 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_page 0.87 ± 11% -0.3 0.62 ± 11% perf-profile.children.cycles-pp.pagecache_get_page 0.84 ± 7% -0.2 0.60 ± 11% perf-profile.children.cycles-pp.kmem_cache_alloc 1.12 ± 7% -0.2 0.88 ± 17% perf-profile.children.cycles-pp.__extent_writepage_io 1.30 ± 15% -0.2 1.06 ± 5% perf-profile.children.cycles-pp.reserve_metadata_bytes 0.82 ± 4% -0.2 0.58 ± 13% perf-profile.children.cycles-pp.run_one_async_done 1.25 ± 3% -0.2 1.01 ± 15% perf-profile.children.cycles-pp.schedule_idle 0.34 ± 53% -0.2 0.10 ± 28% perf-profile.children.cycles-pp.delay_tsc 0.70 ± 13% -0.2 0.47 ± 11% perf-profile.children.cycles-pp.prepare_pages 0.50 ± 22% -0.2 0.28 ± 13% perf-profile.children.cycles-pp.clear_state_bit 0.84 ± 5% -0.2 0.63 ± 6% perf-profile.children.cycles-pp.btrfs_get_extent 0.54 ± 11% -0.2 0.34 ± 4% perf-profile.children.cycles-pp.btrfs_free_path 1.16 ± 11% -0.2 0.96 ± 19% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.53 ± 9% -0.2 0.33 ± 18% perf-profile.children.cycles-pp.unpin_extent_cache 1.07 ± 10% -0.2 0.88 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_file_extent 0.50 ± 4% -0.2 0.31 ± 15% perf-profile.children.cycles-pp.clear_extent_bit 0.45 ± 13% -0.2 0.26 ± 11% perf-profile.children.cycles-pp.kmem_cache_free 0.40 ± 14% -0.2 0.21 ± 50% perf-profile.children.cycles-pp.__writeback_inodes_wb 0.64 ± 7% -0.2 0.45 ± 10% perf-profile.children.cycles-pp.dequeue_task_fair 0.50 ± 8% -0.2 0.33 ± 23% perf-profile.children.cycles-pp.btrfs_add_delayed_tree_ref 0.88 ± 13% -0.2 0.71 ± 2% perf-profile.children.cycles-pp.wait_reserve_ticket 0.52 ± 9% -0.2 0.35 ± 7% perf-profile.children.cycles-pp.__radix_tree_lookup 0.52 ± 8% -0.2 0.36 ± 8% perf-profile.children.cycles-pp.xas_load 0.78 ± 19% -0.2 0.62 ± 23% perf-profile.children.cycles-pp.__btrfs_free_extent 0.56 ± 17% -0.1 0.42 ± 15% perf-profile.children.cycles-pp.btrfs_del_items 0.51 ± 5% -0.1 0.36 ± 9% perf-profile.children.cycles-pp.dequeue_entity 0.39 ± 7% -0.1 0.24 ± 6% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers 0.82 ± 14% -0.1 0.68 ± 18% perf-profile.children.cycles-pp.ttwu_do_activate 0.55 ± 8% -0.1 0.41 ± 8% perf-profile.children.cycles-pp.btrfs_submit_bio_hook 0.53 ± 5% -0.1 0.40 ± 13% perf-profile.children.cycles-pp.update_load_avg 0.40 ± 8% -0.1 0.27 ± 27% perf-profile.children.cycles-pp.add_delayed_ref_head 0.35 ± 13% -0.1 0.22 ± 3% perf-profile.children.cycles-pp.set_extent_bit 0.52 ± 9% -0.1 0.39 ± 7% perf-profile.children.cycles-pp.btrfs_wq_submit_bio 0.35 ± 3% -0.1 0.23 ± 32% perf-profile.children.cycles-pp.queued_write_lock_slowpath 0.33 ± 6% -0.1 0.21 ± 15% perf-profile.children.cycles-pp.__slab_alloc 0.63 ± 19% -0.1 0.51 ± 13% perf-profile.children.cycles-pp.sched_ttwu_pending 0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode 0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode_fallback 0.57 ± 5% -0.1 0.46 ± 10% perf-profile.children.cycles-pp.find_lock_delalloc_range 0.45 ± 5% -0.1 0.34 ± 9% perf-profile.children.cycles-pp.btrfs_cross_ref_exist 0.29 ± 3% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.free_extent_buffer 0.38 ± 3% -0.1 0.26 ± 8% perf-profile.children.cycles-pp.btrfs_mark_buffer_dirty 0.46 ± 5% -0.1 0.35 ± 13% perf-profile.children.cycles-pp.find_delalloc_range 0.41 ± 9% -0.1 0.30 ± 2% perf-profile.children.cycles-pp.btrfs_copy_from_user 0.40 ± 9% -0.1 0.29 ± 2% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 0.38 ± 10% -0.1 0.27 perf-profile.children.cycles-pp.find_get_entry 0.31 ± 12% -0.1 0.20 ± 31% perf-profile.children.cycles-pp.btrfs_remove_ordered_extent 0.30 ± 6% -0.1 0.20 ± 14% perf-profile.children.cycles-pp.___slab_alloc 0.26 ± 8% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.merge_state 0.38 ± 7% -0.1 0.28 ± 3% perf-profile.children.cycles-pp.copyin 0.26 ± 17% -0.1 0.15 ± 26% perf-profile.children.cycles-pp.btrfs_delayed_update_inode 0.25 ± 4% -0.1 0.15 ± 5% perf-profile.children.cycles-pp.insert_state 0.40 ± 13% -0.1 0.30 ± 13% perf-profile.children.cycles-pp.extent_clear_unlock_delalloc 0.36 ± 9% -0.1 0.26 ± 3% perf-profile.children.cycles-pp.alloc_extent_state 0.38 ± 7% -0.1 0.29 ± 3% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.39 ± 3% -0.1 0.29 ± 14% perf-profile.children.cycles-pp.alloc_extent_buffer 0.31 ± 4% -0.1 0.22 ± 11% perf-profile.children.cycles-pp.set_extent_buffer_dirty 0.26 ± 11% -0.1 0.17 ± 23% perf-profile.children.cycles-pp.unlock_up 0.32 ± 3% -0.1 0.24 ± 13% perf-profile.children.cycles-pp.__btrfs_map_block 0.23 ± 13% -0.1 0.15 ± 28% perf-profile.children.cycles-pp.btrfs_add_delayed_data_ref 0.25 ± 12% -0.1 0.17 ± 2% perf-profile.children.cycles-pp.__slab_free 0.47 ± 6% -0.1 0.39 ± 15% perf-profile.children.cycles-pp.__push_leaf_right 0.26 ± 9% -0.1 0.18 ± 19% perf-profile.children.cycles-pp.__test_set_page_writeback 0.32 ± 11% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.btrfs_add_ordered_extent 0.15 ± 8% -0.1 0.07 ± 6% perf-profile.children.cycles-pp.__xa_set_mark 0.23 ± 7% -0.1 0.15 ± 3% perf-profile.children.cycles-pp.release_extent_buffer 0.18 ± 8% -0.1 0.11 ± 26% perf-profile.children.cycles-pp.alloc_extent_map 0.32 ± 10% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.__btrfs_add_ordered_extent 0.32 ± 17% -0.1 0.24 ± 10% perf-profile.children.cycles-pp.btrfs_tree_read_lock_atomic 0.20 ± 13% -0.1 0.13 ± 27% perf-profile.children.cycles-pp.btrfs_inc_extent_ref 0.14 ± 21% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.find_ref_head 0.13 ± 32% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.rb_erase_cached 0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_submit_bio_start 0.53 ± 12% -0.1 0.46 ± 16% perf-profile.children.cycles-pp.btrfs_tree_read_lock 0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_csum_one_bio 0.17 ± 16% -0.1 0.10 ± 8% perf-profile.children.cycles-pp.rb_erase 0.21 ± 7% -0.1 0.14 ± 16% perf-profile.children.cycles-pp.run_one_async_start 0.22 ± 3% -0.1 0.15 ± 22% perf-profile.children.cycles-pp.btrfs_try_tree_write_lock 0.21 ± 9% -0.1 0.15 ± 21% perf-profile.children.cycles-pp.btrfs_leaf_free_space 0.15 ± 23% -0.1 0.09 ± 35% perf-profile.children.cycles-pp.free_extent_map 0.20 ± 9% -0.1 0.14 ± 20% perf-profile.children.cycles-pp.leaf_space_used 0.19 ± 8% -0.1 0.13 ± 13% perf-profile.children.cycles-pp.check_delayed_ref 0.17 ± 13% -0.1 0.12 ± 31% perf-profile.children.cycles-pp.get_page_from_freelist 0.13 ± 8% -0.1 0.08 ± 26% perf-profile.children.cycles-pp.__update_load_avg_se 0.17 ± 2% -0.1 0.12 ± 18% perf-profile.children.cycles-pp.btrfs_set_range_writeback 0.15 ± 8% -0.1 0.09 ± 20% perf-profile.children.cycles-pp.xas_set_mark 0.18 ± 6% -0.1 0.13 ± 16% perf-profile.children.cycles-pp.___might_sleep 0.20 ± 11% -0.1 0.15 ± 35% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.10 ± 9% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_unlock 0.14 ± 11% -0.1 0.09 ± 28% perf-profile.children.cycles-pp.btrfs_reserve_extent 0.10 ± 33% -0.1 0.05 ± 72% perf-profile.children.cycles-pp.__btrfs_release_delayed_node 0.22 ± 9% -0.0 0.17 ± 25% perf-profile.children.cycles-pp.mark_extent_buffer_accessed 0.09 ± 13% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking 0.18 ± 8% -0.0 0.14 ± 24% perf-profile.children.cycles-pp.__list_del_entry_valid 0.09 ± 13% -0.0 0.04 ± 73% perf-profile.children.cycles-pp.verify_parent_transid 0.11 ± 19% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.22 ± 5% -0.0 0.18 ± 12% perf-profile.children.cycles-pp.check_committed_ref 0.11 ± 19% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.select_idle_sibling 0.14 ± 11% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.do_io_getevents 0.15 ± 7% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.__x64_sys_io_getevents 0.08 ± 10% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_release_extent_buffer_pages 0.17 ± 18% -0.0 0.13 ± 9% perf-profile.children.cycles-pp.clear_page_dirty_for_io 0.14 ± 5% -0.0 0.10 ± 25% perf-profile.children.cycles-pp.block_group_cache_tree_search 0.18 ± 4% -0.0 0.14 ± 11% perf-profile.children.cycles-pp.btrfs_get_chunk_map 0.11 ± 13% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.btrfs_buffer_uptodate 0.15 ± 15% -0.0 0.11 ± 19% perf-profile.children.cycles-pp.xas_find_marked 0.12 ± 15% -0.0 0.08 ± 20% perf-profile.children.cycles-pp.btrfs_unlock_up_safe 0.12 ± 14% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.btrfs_set_path_blocking 0.11 ± 11% -0.0 0.07 ± 28% perf-profile.children.cycles-pp.find_free_extent 0.24 ± 5% -0.0 0.20 ± 6% perf-profile.children.cycles-pp.btrfs_root_node 0.07 ± 6% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.extent_mergeable 0.13 ± 12% -0.0 0.10 ± 9% perf-profile.children.cycles-pp.__switch_to 0.10 ± 18% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.rb_insert_color_cached 0.07 ± 14% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.rb_prev 0.11 ± 11% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.get_io_u 0.10 ± 14% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.available_idle_cpu 0.17 ± 10% -0.0 0.13 ± 3% perf-profile.children.cycles-pp._raw_write_lock 0.10 ± 11% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.__might_sleep 0.17 ± 7% -0.0 0.14 ± 12% perf-profile.children.cycles-pp._raw_read_lock 0.17 ± 10% -0.0 0.14 ± 12% perf-profile.children.cycles-pp.can_overcommit 0.10 ± 11% -0.0 0.07 perf-profile.children.cycles-pp.__list_add_valid 0.09 ± 7% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.aio_read_events 0.11 ± 10% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.___perf_sw_event 0.11 ± 9% -0.0 0.09 ± 18% perf-profile.children.cycles-pp.get_alloc_profile 0.11 ± 12% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.pick_next_task_idle 0.10 ± 13% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.read_events 0.08 ± 6% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.replace_extent_mapping 0.10 ± 5% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.menu_reflect 0.07 ± 17% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.05 ± 58% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.rcu_irq_enter 0.19 ± 11% +0.1 0.26 ± 17% perf-profile.children.cycles-pp.native_irq_return_iret 0.20 ± 14% +0.1 0.27 ± 8% perf-profile.children.cycles-pp.btrfs_comp_cpu_keys 0.38 ± 9% +0.1 0.46 perf-profile.children.cycles-pp.load_balance 0.30 ± 8% +0.1 0.38 ± 15% perf-profile.children.cycles-pp.irq_enter 0.21 ± 24% +0.1 0.30 ± 9% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.22 ± 3% +0.1 0.32 ± 9% perf-profile.children.cycles-pp.lapic_next_deadline 0.52 ± 12% +0.1 0.63 ± 3% perf-profile.children.cycles-pp.rebalance_domains 1.90 ± 10% +0.3 2.23 ± 2% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.23 ± 12% +0.4 0.67 ± 15% perf-profile.children.cycles-pp.read_extent_buffer 2.88 ± 11% +0.5 3.41 perf-profile.children.cycles-pp.hrtimer_interrupt 0.09 ± 12% +0.7 0.83 ± 15% perf-profile.children.cycles-pp.btrfs_get_token_8 2.37 ± 19% +0.8 3.19 ± 5% perf-profile.children.cycles-pp.btrfs_async_reclaim_metadata_space 2.33 ± 19% +0.8 3.17 ± 5% perf-profile.children.cycles-pp.flush_space 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_and_wait_transaction 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_marked_extents 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.__filemap_fdatawrite_range 0.42 ± 43% +1.3 1.67 ± 18% perf-profile.children.cycles-pp.btrfs_commit_transaction 0.25 ± 5% +1.9 2.17 ± 10% perf-profile.children.cycles-pp.btrfs_get_token_64 1.71 ± 5% +2.1 3.79 ± 5% perf-profile.children.cycles-pp.btrfs_get_token_32 0.43 ± 8% +2.4 2.84 ± 10% perf-profile.children.cycles-pp.map_private_extent_buffer 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_writeback 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.writeback_sb_inodes 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.__writeback_single_inode 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_workfn 50.06 ± 2% +3.8 53.82 ± 4% perf-profile.children.cycles-pp.intel_idle 61.99 ± 2% +4.4 66.39 ± 3% perf-profile.children.cycles-pp.do_idle 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.secondary_startup_64 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.cpu_startup_entry 8.51 ± 7% +4.5 13.04 ± 12% perf-profile.children.cycles-pp.do_writepages 56.43 ± 3% +4.9 61.30 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state 61.27 ± 2% +5.1 66.37 ± 3% perf-profile.children.cycles-pp.start_secondary 1.43 ± 7% +6.2 7.64 ± 13% perf-profile.children.cycles-pp.submit_extent_page 1.20 ± 17% +6.2 7.45 ± 13% perf-profile.children.cycles-pp.btree_write_cache_pages 1.05 ± 17% +6.3 7.31 ± 13% perf-profile.children.cycles-pp.write_one_eb 1.11 ± 7% +6.3 7.37 ± 13% perf-profile.children.cycles-pp.submit_one_bio 0.55 ± 16% +6.4 6.95 ± 13% perf-profile.children.cycles-pp.btree_submit_bio_hook 0.00 +6.5 6.48 ± 13% perf-profile.children.cycles-pp.check_leaf 0.00 +6.5 6.53 ± 13% perf-profile.children.cycles-pp.btree_csum_one_bio 2.12 ± 11% -0.8 1.29 ± 7% perf-profile.self.cycles-pp.__lookup_extent_mapping 1.83 ± 12% -0.8 1.03 ± 13% perf-profile.self.cycles-pp.__etree_search 2.06 ± 5% -0.5 1.59 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 0.81 ± 9% -0.3 0.52 ± 10% perf-profile.self.cycles-pp.generic_bin_search 0.84 ± 7% -0.3 0.58 ± 17% perf-profile.self.cycles-pp.copy_page 0.33 ± 54% -0.2 0.10 ± 28% perf-profile.self.cycles-pp.delay_tsc 0.89 ± 9% -0.2 0.69 ± 14% perf-profile.self.cycles-pp.btrfs_set_token_32 0.81 ± 4% -0.2 0.63 ± 14% perf-profile.self.cycles-pp.__schedule 0.51 ± 9% -0.2 0.35 ± 6% perf-profile.self.cycles-pp.__radix_tree_lookup 0.45 ± 8% -0.1 0.32 ± 10% perf-profile.self.cycles-pp.xas_load 0.43 ± 4% -0.1 0.31 ± 6% perf-profile.self.cycles-pp.find_extent_buffer 0.77 ± 5% -0.1 0.66 ± 11% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.31 ± 10% -0.1 0.20 ± 28% perf-profile.self.cycles-pp.add_delayed_ref_head 0.39 ± 16% -0.1 0.29 ± 16% perf-profile.self.cycles-pp.try_to_wake_up 0.31 ± 4% -0.1 0.20 ± 26% perf-profile.self.cycles-pp.queued_write_lock_slowpath 0.38 ± 7% -0.1 0.29 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.20 ± 17% -0.1 0.11 ± 14% perf-profile.self.cycles-pp.kmem_cache_free 0.25 ± 11% -0.1 0.17 ± 2% perf-profile.self.cycles-pp.__slab_free 0.16 ± 9% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.free_extent_buffer 0.28 ± 17% -0.1 0.20 ± 11% perf-profile.self.cycles-pp.btrfs_search_slot 0.31 ± 4% -0.1 0.24 ± 20% perf-profile.self.cycles-pp.kmem_cache_alloc 0.14 ± 21% -0.1 0.06 ± 19% perf-profile.self.cycles-pp.find_ref_head 0.16 ± 10% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.insert_state 0.12 ± 8% -0.1 0.06 ± 72% perf-profile.self.cycles-pp.__update_load_avg_se 0.16 ± 15% -0.1 0.10 ± 8% perf-profile.self.cycles-pp.rb_erase 0.10 ± 9% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_unlock 0.15 ± 13% -0.1 0.10 ± 12% perf-profile.self.cycles-pp.___slab_alloc 0.28 ± 13% -0.1 0.22 ± 5% perf-profile.self.cycles-pp.queued_read_lock_slowpath 0.15 ± 21% -0.1 0.09 ± 35% perf-profile.self.cycles-pp.free_extent_map 0.18 ± 6% -0.1 0.12 ± 11% perf-profile.self.cycles-pp.update_load_avg 0.10 ± 11% -0.1 0.04 ± 70% perf-profile.self.cycles-pp.btrfs_extend_item 0.15 ± 23% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.add_extent_mapping 0.15 ± 8% -0.1 0.09 ± 20% perf-profile.self.cycles-pp.xas_set_mark 0.18 ± 6% -0.1 0.13 ± 16% perf-profile.self.cycles-pp.___might_sleep 0.18 ± 6% -0.0 0.13 ± 22% perf-profile.self.cycles-pp.__list_del_entry_valid 0.18 ± 4% -0.0 0.14 ± 22% perf-profile.self.cycles-pp.pick_next_task_fair 0.09 ± 13% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.verify_parent_transid 0.09 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_read_unlock_blocking 0.16 ± 13% -0.0 0.12 ± 24% perf-profile.self.cycles-pp.mark_page_accessed 0.12 ± 14% -0.0 0.08 ± 20% perf-profile.self.cycles-pp.dequeue_task_fair 0.10 ± 19% -0.0 0.07 ± 18% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.08 ± 14% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.unlock_up 0.15 ± 15% -0.0 0.11 ± 19% perf-profile.self.cycles-pp.xas_find_marked 0.24 ± 6% -0.0 0.20 ± 8% perf-profile.self.cycles-pp.btrfs_root_node 0.17 ± 6% -0.0 0.13 ± 10% perf-profile.self.cycles-pp._raw_read_lock 0.10 ± 13% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.btrfs_csum_one_bio 0.16 ± 7% -0.0 0.13 ± 9% perf-profile.self.cycles-pp.dequeue_entity 0.07 -0.0 0.04 ± 71% perf-profile.self.cycles-pp.io_u_queued_complete 0.08 -0.0 0.05 ± 70% perf-profile.self.cycles-pp.mutex_lock 0.11 ± 11% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.get_io_u 0.16 ± 9% -0.0 0.13 perf-profile.self.cycles-pp._raw_write_lock 0.07 ± 17% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.btrfs_tree_read_lock 0.12 ± 8% -0.0 0.09 ± 13% perf-profile.self.cycles-pp.__switch_to 0.10 ± 11% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.__clear_extent_bit 0.07 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.rb_prev 0.10 ± 15% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.find_get_pages_range_tag 0.10 ± 12% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.available_idle_cpu 0.10 ± 11% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.__list_add_valid 0.08 ± 13% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.__might_sleep 0.12 ± 17% -0.0 0.09 ± 15% perf-profile.self.cycles-pp.release_extent_buffer 0.09 ± 17% -0.0 0.07 ± 14% perf-profile.self.cycles-pp.___perf_sw_event 0.08 ± 5% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.__push_leaf_right 0.11 ± 7% -0.0 0.09 ± 9% perf-profile.self.cycles-pp.set_extent_buffer_dirty 0.11 ± 4% -0.0 0.08 ± 11% perf-profile.self.cycles-pp.__set_extent_bit 0.09 ± 7% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.hrtimer_interrupt 0.07 ± 17% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.07 ± 7% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.load_balance 0.01 ±173% +0.1 0.07 ± 17% perf-profile.self.cycles-pp.lapic_next_deadline 0.00 +0.1 0.06 ± 23% perf-profile.self.cycles-pp.find_next_and_bit 0.19 ± 11% +0.1 0.26 ± 17% perf-profile.self.cycles-pp.native_irq_return_iret 0.19 ± 24% +0.1 0.28 ± 7% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.50 ± 8% +0.2 0.69 ± 17% perf-profile.self.cycles-pp.cpuidle_enter_state 0.07 ± 11% +0.4 0.46 ± 17% perf-profile.self.cycles-pp.btrfs_get_token_8 0.22 ± 13% +0.4 0.66 ± 16% perf-profile.self.cycles-pp.read_extent_buffer 0.00 +0.6 0.65 ± 14% perf-profile.self.cycles-pp.check_leaf 0.15 ± 11% +1.0 1.13 ± 11% perf-profile.self.cycles-pp.btrfs_get_token_64 1.48 ± 7% +1.0 2.51 ± 5% perf-profile.self.cycles-pp.btrfs_get_token_32 0.41 ± 8% +2.2 2.61 ± 9% perf-profile.self.cycles-pp.map_private_extent_buffer 49.99 ± 2% +3.7 53.74 ± 4% perf-profile.self.cycles-pp.intel_idle fio.write_bw_MBps 300 +-+-------------------------------------------------------------------+ | | 250 +-++.+ +. .+.. .+..+..+.+.. .+..+.. .+ | | : : +. +..+ + +..+ +. + | | : : : : : : | 200 +-+ : : : : : : O O | O O O: O :O O O O O O O O O O: O:O O: O: O O O O O 150 +-+ : : : : : : | | : : : : : : | 100 +-+ : : : : : : | | : : : : : : | | : : : : : : | 50 +-+ :: :: :: | | : : : | 0 +-+---------------------------------------------O-------------O-------+ fio.write_iops 70000 +-+-----------------------------------------------------------------+ |..+.+ +..+..+.+.. .+.+..+.+..+ +.. +..+.+..+..+ | 60000 +-+ : : +. : : + : | | : : : : : : | 50000 +-+ : : : : : : | O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O 40000 +-+ : O: : : : : | | : : : : : : | 30000 +-+ : : : : : : | | : : : : : : | 20000 +-+ : : : : : : | | :: :: :: | 10000 +-+ : : : | | : : : | 0 +-+--------------------------------------------O------------O-------+ fio.write_clat_mean_us 30000 +-+-----------------------------------------------------------------+ | | 25000 O-+O O O O O O O O O O O O O O O O O O O | | O O O O O | | 20000 +-+ | |..+.+ +..+..+.+..+..+.+..+.+..+ +..+ +..+.+..+..+ | 15000 +-+ : : : : : : | | : : : : : : | 10000 +-+ : : : : : : | | : : : : : : | | : : : : : : | 5000 +-+ : : : : :: | | : : : | 0 +-+--------------------------------------------O------------O-------+ fio.write_clat_stddev 160000 +-+----------------------------------------------------------------+ | | 140000 O-+O O O O O O O O O O O O O O | 120000 +-+ O O | | | 100000 +-+ O O O O O O O O | | 80000 +-+ | | .+.. | 60000 +-++.+ +..+.+. +.+..+.+..+..+ +.+ +..+..+.+..+ | 40000 +-+ : : : : : : | | : : : : : : | 20000 +-+ : : : : : : | | :: :: :: | 0 +-+-------------------------------------------O------------O-------+ fio.write_slat_mean_us 900 +-+-------------------------------------------------------------------+ O O O O O O O O O O O O O O O O O O | 800 +-+O O O O O O O 700 +-+ | | | 600 +-++.+ +.+..+..+..+.+..+..+.+..+ +..+ +..+..+..+.+ | 500 +-+ : : : : : : | | : : : : : : | 400 +-+ : : : : : : | 300 +-+ : : : : : : | | : : : : : : | 200 +-+ : : : : : : | 100 +-+ : : :: :: | | : : : | 0 +-+---------------------------------------------O-------------O-------+ fio.write_slat_stddev 25000 +-+-----O---------O-------------------------------------------------+ O O O O O O O O O O O O O O O | | | 20000 +-+ | | O O O O O O O O | | 15000 +-+ | | | 10000 +-++. .+.. .+.. .+..+ .+ .+..+ | |. + +..+..+ +..+ + : +. : +..+.+. | | : : : : : : | 5000 +-+ : : : : : : | | : : : : : : | | :: : :: | 0 +-+--------------------------------------------O------------O-------+ fio.latency_1000ms_ 0.4 +-+------------------------------------------------------------------+ | | 0.35 +-+ O | 0.3 +-+ | | O O | 0.25 +-+ O O | | O O O 0.2 O-+ O O O O O O O O O O | | O O | 0.15 +-+O O O O | 0.1 +-+ | | | 0.05 +-+ | | .+. .+..+..+.+..+.. .+. .+.+..+.. | 0 +-+---------------------------------------------O------------O-------+ fio.workload 1.4e+07 +-+---------------------------------------------------------------+ |..+.+ +..+.+..+.+..+.+..+.+..+ +.. +.+..+.+..+ | 1.2e+07 +-+ : : : : + : | | : : : : : : | 1e+07 +-+ : : : : : : | O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O 8e+06 +-+ : O: : : : : | | : : : : : : | 6e+06 +-+ : : : : : : | | : : : : : : | 4e+06 +-+ : : : : : : | | :: :: :: | 2e+06 +-+ : : : | | : : : | 0 +-+-------------------------------------------O-----------O-------+ fio.time.voluntary_context_switches 1.4e+07 +-+---------------------------------------------------------------+ | | 1.2e+07 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ | | : : : : : : | 1e+07 +-+ : : : : : : | | O O: : O O O : : :O : O O O O O O O 8e+06 O-+ : O:O O O O O O O: O:O O: : | | : : : : : : | 6e+06 +-+ : : : : : : | | : : : : : : | 4e+06 +-+ : : : : : : | | : : : : :: | 2e+06 +-+ : : :: | | : : : | 0 +-+-------------------------------------------O-----------O-------+ fio.time.file_system_outputs 1.2e+08 +-+---------------------------------------------------------------+ | | 1e+08 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ | | : : : : : : | | : : : : : : | 8e+07 +-+ : : : : : : O O O O O O O O: O:O O O O O O O O O O: O:O O:O : O O | 6e+07 +-+ : : : : : : | | : : : : : : | 4e+07 +-+ : : : : : : | | : : : : : : | | : : : : :: | 2e+07 +-+ : : :: | | : : : | 0 +-+-------------------------------------------O-----------O-------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 years, 2 months

4
4
0 / 0

[sched/numa] f6183ef98b: phoronix-test-suite.aom-av1.0.frames_per_second -25.0% regression

by kernel test robot

Greeting, FYI, we noticed a -25.0% regression of phoronix-test-suite.aom-av1.0.frames_per_second due to commit: commit: f6183ef98bba39a5de563d4ab4bc889ff2b600d7 ("sched/numa: replace runnable_load_avg by load_avg") https://git.kernel.org/cgit/linux/kernel/git/mel/linux.git sched-lbnuma-rewrite-v2r6 in testcase: phoronix-test-suite on test machine: 16 threads Intel(R) Xeon(R) CPU X5570 @ 2.93GHz with 48G memory with following parameters: test: aom-av1-1.2.0 cpufreq_governor: performance ucode: 0x1d test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added. test-url: http://www.phoronix-test-suite.com/ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-phoronix/lkp-nhm-2ep1/aom-av1-1.2.0/phoronix-test-suite/0x1d commit: 91d7637912 ("sched/fair: reorder enqueue/dequeue_task_fair path") f6183ef98b ("sched/numa: replace runnable_load_avg by load_avg") 91d76379120e4640 f6183ef98bba39a5de563d4ab4b ---------------- --------------------------- %stddev %change %stddev \ | \ 0.04 -25.0% 0.03 phoronix-test-suite.aom-av1.0.frames_per_second 1648 +24.6% 2055 phoronix-test-suite.time.elapsed_time 1648 +24.6% 2055 phoronix-test-suite.time.elapsed_time.max 1935 -19.6% 1556 meminfo.max_used_kB 1.518e+10 ± 29% +39.9% 2.124e+10 cpuidle.C1E.time 33131701 ± 34% +35.9% 45011103 ± 2% cpuidle.C1E.usage 672.00 -100.0% 0.00 slabinfo.dmaengine-unmap-16.active_objs 672.00 -100.0% 0.00 slabinfo.dmaengine-unmap-16.num_objs 67.50 -2.2% 66.00 vmstat.cpu.id 31.00 +6.5% 33.00 vmstat.cpu.us 467.75 +5.0% 491.00 proc-vmstat.nr_mlock 484.75 +4.8% 508.00 proc-vmstat.nr_unevictable 484.75 +4.8% 508.00 proc-vmstat.nr_zone_unevictable 3492646 ± 7% +67.4% 5846384 ± 3% proc-vmstat.numa_hint_faults 2478506 ± 6% +127.8% 5646434 ± 4% proc-vmstat.numa_hint_faults_local 3650509 +13.5% 4141674 proc-vmstat.numa_hit 76734 +98.1% 151980 ± 2% proc-vmstat.numa_huge_pte_updates 3647139 +13.5% 4138249 proc-vmstat.numa_local 3408776 ± 7% -80.4% 667824 ± 8% proc-vmstat.numa_pages_migrated 42828088 +95.3% 83628988 ± 2% proc-vmstat.numa_pte_updates 7766914 ± 3% -25.6% 5776215 proc-vmstat.pgalloc_normal 7771023 ± 3% +43.5% 11147904 ± 2% proc-vmstat.pgfault 7716195 ± 3% -25.8% 5726088 proc-vmstat.pgfree 1152 ± 19% -88.9% 128.00 ±173% proc-vmstat.pgmigrate_fail 3408776 ± 7% -80.4% 667824 ± 8% proc-vmstat.pgmigrate_success 0.13 ±173% +0.5 0.63 ± 7% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 0.14 ±173% +0.5 0.67 ± 8% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 4.40 ± 47% +3.5 7.87 ± 16% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel 4.40 ± 47% +3.5 7.87 ± 16% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64 4.41 ± 47% +3.5 7.94 ± 16% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64 4.41 ± 47% +3.5 7.94 ± 16% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64 4.41 ± 47% +3.5 7.94 ± 16% perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64 0.05 ± 67% +0.1 0.12 ± 17% perf-profile.children.cycles-pp.ktime_get 0.01 ±173% +0.1 0.10 ± 27% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.00 +0.1 0.10 ± 36% perf-profile.children.cycles-pp.rcu_core 0.32 ± 86% +0.5 0.85 ± 54% perf-profile.children.cycles-pp.__libc_start_main 0.60 ± 46% +0.6 1.19 ± 41% perf-profile.children.cycles-pp.do_syscall_64 0.60 ± 46% +0.6 1.19 ± 41% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 0.34 ± 82% +0.6 0.94 ± 51% perf-profile.children.cycles-pp.vfs_read 0.34 ± 82% +0.6 0.94 ± 51% perf-profile.children.cycles-pp.ksys_read 0.29 ± 96% +0.6 0.90 ± 53% perf-profile.children.cycles-pp.perf_event_read 0.29 ± 96% +0.6 0.90 ± 53% perf-profile.children.cycles-pp.smp_call_function_single 0.30 ± 96% +0.6 0.91 ± 53% perf-profile.children.cycles-pp.__libc_read 0.29 ± 96% +0.6 0.91 ± 52% perf-profile.children.cycles-pp.perf_read 4.41 ± 47% +3.5 7.94 ± 16% perf-profile.children.cycles-pp.start_kernel 0.01 ±173% +0.1 0.07 ± 19% perf-profile.self.cycles-pp.cpuidle_enter_state 0.29 ± 98% +0.6 0.90 ± 53% perf-profile.self.cycles-pp.smp_call_function_single 161465 ± 7% +66.5% 268796 ± 16% softirqs.CPU0.SCHED 154149 ± 4% +57.3% 242412 ± 5% softirqs.CPU1.SCHED 217783 ± 10% +17.4% 255655 ± 3% softirqs.CPU10.RCU 685538 ± 6% +16.3% 797133 ± 3% softirqs.CPU10.TIMER 647482 +29.6% 839256 ± 14% softirqs.CPU11.TIMER 215207 ± 9% +19.8% 257893 ± 5% softirqs.CPU12.RCU 676241 ± 4% +20.5% 814890 softirqs.CPU12.TIMER 218513 ± 8% +17.8% 257505 ± 4% softirqs.CPU14.RCU 665815 ± 5% +20.2% 800244 ± 3% softirqs.CPU14.TIMER 217764 ± 10% +16.0% 252579 ± 2% softirqs.CPU15.RCU 671996 ± 2% +18.0% 792743 ± 10% softirqs.CPU15.TIMER 216810 ± 10% +16.2% 251864 ± 3% softirqs.CPU2.RCU 168050 ± 8% +53.2% 257377 ± 4% softirqs.CPU2.SCHED 687608 ± 7% +13.9% 783359 ± 5% softirqs.CPU2.TIMER 214659 ± 10% +17.4% 252030 ± 2% softirqs.CPU3.RCU 201381 ± 8% +26.8% 255401 ± 10% softirqs.CPU3.SCHED 218567 ± 9% +15.1% 251588 ± 4% softirqs.CPU4.RCU 162228 ± 14% +39.7% 226690 ± 21% softirqs.CPU4.SCHED 686728 ± 7% +15.4% 792699 ± 4% softirqs.CPU4.TIMER 598763 ± 4% +33.1% 796983 ± 15% softirqs.CPU5.TIMER 219495 ± 10% +15.8% 254260 ± 4% softirqs.CPU6.RCU 685621 ± 6% +15.4% 790913 ± 4% softirqs.CPU6.TIMER 213675 ± 10% +18.1% 252314 ± 2% softirqs.CPU7.RCU 607580 ± 4% +27.7% 776035 ± 7% softirqs.CPU7.TIMER 217328 ± 10% +17.9% 256261 ± 4% softirqs.CPU8.RCU 672277 ± 8% +18.4% 795798 ± 3% softirqs.CPU8.TIMER 662696 ± 15% +22.2% 809936 ± 6% softirqs.CPU9.TIMER 3471815 ± 10% +16.7% 4051741 ± 2% softirqs.RCU 2788089 +21.0% 3372673 ± 2% softirqs.SCHED 10668573 ± 2% +17.5% 12539415 ± 5% softirqs.TIMER 250541 +31.8% 330137 ± 3% sched_debug.cfs_rq:/.exec_clock.avg 446877 ± 7% +68.2% 751493 ± 18% sched_debug.cfs_rq:/.exec_clock.max 112331 ± 17% +122.7% 250151 ± 35% sched_debug.cfs_rq:/.exec_clock.stddev 1517645 +35.0% 2049553 ± 3% sched_debug.cfs_rq:/.min_vruntime.avg 2633991 ± 6% +65.2% 4352247 ± 18% sched_debug.cfs_rq:/.min_vruntime.max 684251 ± 18% +118.2% 1492888 ± 37% sched_debug.cfs_rq:/.min_vruntime.stddev 0.46 ± 2% +8.1% 0.50 ± 4% sched_debug.cfs_rq:/.nr_running.avg -292907 -321.0% 647293 ± 49% sched_debug.cfs_rq:/.spread0.avg 823435 ± 35% +258.3% 2949987 ± 32% sched_debug.cfs_rq:/.spread0.max 684251 ± 18% +118.2% 1492888 ± 37% sched_debug.cfs_rq:/.spread0.stddev 389.08 +10.3% 429.08 ± 3% sched_debug.cfs_rq:/.util_avg.avg 282.99 ± 2% +13.1% 320.08 ± 6% sched_debug.cfs_rq:/.util_est_enqueued.avg 383.98 ± 2% +8.6% 416.84 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev 842328 +25.8% 1059503 sched_debug.cpu.clock.avg 842329 +25.8% 1059503 sched_debug.cpu.clock.max 842327 +25.8% 1059502 sched_debug.cpu.clock.min 1.02 ± 69% -49.5% 0.51 sched_debug.cpu.clock.stddev 842328 +25.8% 1059503 sched_debug.cpu.clock_task.avg 842329 +25.8% 1059503 sched_debug.cpu.clock_task.max 842327 +25.8% 1059502 sched_debug.cpu.clock_task.min 1.02 ± 69% -49.5% 0.51 sched_debug.cpu.clock_task.stddev 6159 ± 3% +29.3% 7966 ± 6% sched_debug.cpu.curr->pid.avg 21434 +25.0% 26795 sched_debug.cpu.curr->pid.max 7691 ± 2% +26.8% 9755 sched_debug.cpu.curr->pid.stddev 57507 ± 2% +27.2% 73171 sched_debug.cpu.nr_switches.avg 98634 ± 8% +51.9% 149817 ± 22% sched_debug.cpu.nr_switches.max 21105 ± 18% +85.3% 39113 ± 41% sched_debug.cpu.nr_switches.stddev 52692 ± 2% +29.9% 68458 sched_debug.cpu.sched_count.avg 94966 ± 8% +50.3% 142769 ± 25% sched_debug.cpu.sched_count.max 25787 ± 2% +27.6% 32917 sched_debug.cpu.sched_goidle.avg 47100 ± 8% +48.8% 70064 ± 26% sched_debug.cpu.sched_goidle.max 25109 ± 2% +28.2% 32200 sched_debug.cpu.ttwu_count.avg 47736 ± 4% +55.9% 74430 ± 9% sched_debug.cpu.ttwu_count.max 10778 ± 13% +80.8% 19490 ± 29% sched_debug.cpu.ttwu_count.stddev 11888 ± 4% +30.1% 15462 sched_debug.cpu.ttwu_local.avg 842327 +25.8% 1059502 sched_debug.cpu_clk 839741 +25.9% 1056916 sched_debug.ktime 843284 +25.8% 1060461 sched_debug.sched_clk 1307 +27.3% 1663 ± 10% interrupts.35:PCI-MSI.524289-edge.eth0-rx-0 884.50 ± 2% +45.9% 1290 ± 26% interrupts.42:PCI-MSI.524296-edge.eth0-tx-3 1810554 ± 9% -24.9% 1359040 interrupts.CAL:Function_call_interrupts 128994 ± 13% -54.5% 58688 ± 11% interrupts.CPU0.CAL:Function_call_interrupts 3136833 ± 9% +30.6% 4097106 interrupts.CPU0.LOC:Local_timer_interrupts 147624 ± 12% -63.5% 53903 ± 7% interrupts.CPU0.TLB:TLB_shootdowns 131316 ± 12% -58.5% 54545 ± 3% interrupts.CPU1.CAL:Function_call_interrupts 3196967 ± 6% +28.8% 4117877 interrupts.CPU1.LOC:Local_timer_interrupts 1316 ± 44% +134.5% 3086 ± 54% interrupts.CPU1.RES:Rescheduling_interrupts 150414 ± 17% -65.7% 51585 ± 14% interrupts.CPU1.TLB:TLB_shootdowns 3214233 ± 5% +28.6% 4132526 interrupts.CPU10.LOC:Local_timer_interrupts 9879 ± 80% -85.4% 1445 ± 45% interrupts.CPU10.RES:Rescheduling_interrupts 3177700 ± 6% +29.8% 4126192 interrupts.CPU11.LOC:Local_timer_interrupts 559.50 ± 46% +253.3% 1976 ± 51% interrupts.CPU11.RES:Rescheduling_interrupts 1307 +27.3% 1663 ± 10% interrupts.CPU12.35:PCI-MSI.524289-edge.eth0-rx-0 3234073 ± 4% +27.7% 4130544 interrupts.CPU12.LOC:Local_timer_interrupts 9543 ± 47% -86.8% 1263 ± 55% interrupts.CPU12.RES:Rescheduling_interrupts 3178037 ± 7% +29.8% 4124112 interrupts.CPU13.LOC:Local_timer_interrupts 3234950 ± 4% +27.7% 4130372 interrupts.CPU14.LOC:Local_timer_interrupts 7845 ± 61% -86.3% 1074 ± 77% interrupts.CPU14.RES:Rescheduling_interrupts 3177518 ± 7% +29.8% 4123846 interrupts.CPU15.LOC:Local_timer_interrupts 115381 ± 14% -46.8% 61375 ± 38% interrupts.CPU2.CAL:Function_call_interrupts 3211923 ± 4% +28.3% 4122113 interrupts.CPU2.LOC:Local_timer_interrupts 128739 ± 17% -57.7% 54470 ± 48% interrupts.CPU2.TLB:TLB_shootdowns 884.50 ± 2% +45.9% 1290 ± 26% interrupts.CPU3.42:PCI-MSI.524296-edge.eth0-tx-3 3189990 ± 6% +29.1% 4119429 interrupts.CPU3.LOC:Local_timer_interrupts 1680 ± 89% +222.4% 5418 ± 50% interrupts.CPU3.RES:Rescheduling_interrupts 3207702 ± 5% +28.7% 4126732 interrupts.CPU4.LOC:Local_timer_interrupts 132646 ± 21% -46.4% 71034 ± 60% interrupts.CPU4.TLB:TLB_shootdowns 3185259 ± 7% +29.4% 4121092 interrupts.CPU5.LOC:Local_timer_interrupts 1237 ± 56% +203.8% 3757 ± 37% interrupts.CPU5.RES:Rescheduling_interrupts 3200005 ± 5% +29.0% 4128870 interrupts.CPU6.LOC:Local_timer_interrupts 3187267 ± 7% +29.4% 4125551 interrupts.CPU7.LOC:Local_timer_interrupts 802.00 ± 53% +280.5% 3051 ± 26% interrupts.CPU7.RES:Rescheduling_interrupts 3266308 ± 2% +26.5% 4132734 interrupts.CPU8.LOC:Local_timer_interrupts 23706 ± 90% -82.0% 4262 ± 66% interrupts.CPU8.RES:Rescheduling_interrupts 3163000 ± 8% +30.5% 4128371 interrupts.CPU9.LOC:Local_timer_interrupts 51161770 ± 6% +29.0% 65987472 interrupts.LOC:Local_timer_interrupts 1944653 ± 10% -31.9% 1325214 interrupts.TLB:TLB_shootdowns 4.79 +19.6% 5.73 perf-stat.i.MPKI 1.596e+09 -19.9% 1.278e+09 perf-stat.i.branch-instructions 1.66 +0.1 1.76 perf-stat.i.branch-miss-rate% 26901389 -14.4% 23036625 perf-stat.i.branch-misses 1.39e+08 -2.8% 1.351e+08 perf-stat.i.cache-references 0.61 ± 6% +17.9% 0.72 perf-stat.i.cpi 16.50 ± 5% +145.5% 40.52 ± 2% perf-stat.i.cpu-migrations 5919 -3.6% 5706 ± 2% perf-stat.i.cycles-between-cache-misses 0.05 ± 2% +0.0 0.07 perf-stat.i.dTLB-load-miss-rate% 3285384 ± 3% +15.2% 3784496 perf-stat.i.dTLB-load-misses 7.201e+09 -19.5% 5.793e+09 perf-stat.i.dTLB-loads 0.20 ± 2% +0.0 0.22 ± 4% perf-stat.i.dTLB-store-miss-rate% 3641778 ± 2% -7.6% 3366197 ± 5% perf-stat.i.dTLB-store-misses 1.925e+09 -19.5% 1.549e+09 perf-stat.i.dTLB-stores 0.01 ± 5% +0.0 0.02 ± 2% perf-stat.i.iTLB-load-miss-rate% 2872660 ± 2% +25.9% 3617061 ± 2% perf-stat.i.iTLB-load-misses 2.816e+10 -19.7% 2.261e+10 perf-stat.i.iTLB-loads 2.819e+10 -19.8% 2.261e+10 perf-stat.i.instructions 10654 ± 3% -27.1% 7764 ± 2% perf-stat.i.instructions-per-iTLB-miss 1.70 ± 4% -14.9% 1.45 perf-stat.i.ipc 0.08 ± 2% -20.0% 0.07 ± 2% perf-stat.i.major-faults 4638 ± 3% +15.4% 5354 ± 2% perf-stat.i.minor-faults 4638 ± 3% +15.4% 5354 ± 2% perf-stat.i.page-faults 4.93 +21.2% 5.97 perf-stat.overall.MPKI 1.69 +0.1 1.80 perf-stat.overall.branch-miss-rate% 2.48 ± 4% +0.2 2.66 ± 2% perf-stat.overall.cache-miss-rate% 0.58 ± 2% +25.2% 0.73 perf-stat.overall.cpi 0.05 ± 3% +0.0 0.07 perf-stat.overall.dTLB-load-miss-rate% 0.19 ± 2% +0.0 0.22 ± 5% perf-stat.overall.dTLB-store-miss-rate% 0.01 +0.0 0.02 ± 2% perf-stat.overall.iTLB-load-miss-rate% 9815 ± 2% -36.3% 6253 ± 2% perf-stat.overall.instructions-per-iTLB-miss 1.72 ± 2% -20.2% 1.37 perf-stat.overall.ipc 1.595e+09 -19.9% 1.278e+09 perf-stat.ps.branch-instructions 26891393 -14.3% 23033727 perf-stat.ps.branch-misses 1.389e+08 -2.8% 1.35e+08 perf-stat.ps.cache-references 16.50 ± 5% +145.4% 40.50 ± 2% perf-stat.ps.cpu-migrations 3283187 ± 3% +15.2% 3782278 perf-stat.ps.dTLB-load-misses 7.196e+09 -19.5% 5.79e+09 perf-stat.ps.dTLB-loads 3639422 ± 2% -7.6% 3364334 ± 5% perf-stat.ps.dTLB-store-misses 1.924e+09 -19.5% 1.548e+09 perf-stat.ps.dTLB-stores 2870820 ± 2% +25.9% 3614797 ± 2% perf-stat.ps.iTLB-load-misses 2.814e+10 -19.7% 2.26e+10 perf-stat.ps.iTLB-loads 2.817e+10 -19.8% 2.259e+10 perf-stat.ps.instructions 0.08 ± 2% -20.0% 0.07 ± 2% perf-stat.ps.major-faults 4636 ± 3% +15.5% 5353 ± 2% perf-stat.ps.minor-faults 4636 ± 3% +15.5% 5353 ± 2% perf-stat.ps.page-faults phoronix-test-suite.aom-av1.0.frames_per_second 0.04 +-------------------------------------------------------------------+ | | 0.035 |-+ | 0.03 |-O O O O O O O O O O O O O O O O O O O O O O O O O O O | | | 0.025 |-+ | | | 0.02 |-+ | | | 0.015 |-+ | 0.01 |-+ | | | 0.005 |-+ | | | 0 +-------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 years, 3 months

5
11
0 / 0

[drm/i915/gt] 36511637bd: WARNING:at_drivers/gpu/drm/i915/gt/intel_workarounds.c:#intel_gt_init_workarounds[i915]

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: 36511637bd8526b63943cc06f40bf612de73080c ("drm/i915/gt: Apply Enable Boot Fetch to MBC control register") https://github.com/0day-ci/linux/commits/UPDATE-20200209-084341/Chris-Wil... in testcase: suspend_stress with following parameters: mode: mem iterations: 10 on test machine: 4 threads BroadWell with 8G memory caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> kern :warn : [ 201.333927] WARNING: CPU: 1 PID: 281 at drivers/gpu/drm/i915/gt/intel_workarounds.c:979 intel_gt_init_workarounds+0xc4/0x350 [i915] kern :warn : [ 201.334108] Modules linked in: i915(+) crct10dif_pclmul(+) crc32_pclmul crc32c_intel ghash_clmulni_intel iwlwifi btusb btrtl btbcm btintel snd_hda_codec_realtek(+) snd_hda_codec_generic drm_kms_helper ledtrig_audio ahci aesni_intel syscopyarea libahci sysfillrect crypto_simd snd_hda_intel snd_intel_dspcfg cryptd snd_hda_codec glue_helper bluetooth sysimgblt snd_hda_core fb_sys_fops pcspkr snd_hwdep libata ir_rc6_decoder cfg80211 snd_pcm ecdh_generic i2c_i801 lpc_ich joydev rc_rc6_mce ecc drm nuvoton_cir rfkill snd_timer snd mei_me mei soundcore rc_core video acpi_pad intel_pmc_core ip_tables kern :warn : [ 201.334837] CPU: 1 PID: 281 Comm: systemd-udevd Not tainted 5.5.0-rc7-02125-g36511637bd8526 #2 kern :warn : [ 201.334957] Hardware name: /NUC5i3RYB, BIOS RYBDWi35.86A.0363.2017.0316.1028 03/16/2017 kern :warn : [ 201.335176] RIP: 0010:intel_gt_init_workarounds+0xc4/0x350 [i915] kern :warn : [ 201.335266] Code: 80 fa 07 0f 84 55 02 00 00 80 fa 06 0f 84 4c 02 00 00 80 fa 05 76 60 48 c7 c6 49 4f 9d c0 48 c7 c7 62 4f 9d c0 e8 4c 9c 80 c0 <0f> 0b eb 49 48 89 ee e8 d0 f1 ff ff f6 83 8b 06 00 00 40 49 89 e4 kern :warn : [ 201.335524] RSP: 0018:ffffc90000247a08 EFLAGS: 00010282 kern :warn : [ 201.335599] RAX: 0000000000000000 RBX: ffff8881f9800000 RCX: 0000000000000007 kern :warn : [ 201.335700] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff888256899b30 kern :warn : [ 201.335801] RBP: ffff8881f9806fa0 R08: 0000000000000a26 R09: 000000000000003b kern :debug : [ 201.335841] initcall crct10dif_intel_mod_init+0x0/0x1000 [crct10dif_pclmul] returned 0 after 7181 usecs kern :warn : [ 201.335901] R10: 0000000000000010 R11: ffffc900002478a8 R12: ffff888107c67000 kern :warn : [ 201.335904] R13: ffff8881f9807758 R14: ffff8881f98007b0 R15: ffff8881f9807948 kern :warn : [ 201.336233] FS: 00007f175bad78c0(0000) GS:ffff888256880000(0000) knlGS:0000000000000000 kern :warn : [ 201.336346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kern :warn : [ 201.336428] CR2: 00007f9ecc0e7520 CR3: 0000000202216003 CR4: 00000000003606e0 kern :warn : [ 201.336527] Call Trace: kern :warn : [ 201.336662] i915_driver_probe+0xdb1/0x15a0 [i915] kern :warn : [ 201.336739] ? __kernfs_new_node+0x174/0x1e0 kern :warn : [ 201.336804] ? _cond_resched+0x19/0x30 kern :warn : [ 201.336927] i915_pci_probe+0x54/0x170 [i915] kern :warn : [ 201.336996] local_pci_probe+0x42/0x90 kern :warn : [ 201.337053] ? _cond_resched+0x19/0x30 kern :warn : [ 201.337110] pci_device_probe+0x10b/0x1c0 kern :warn : [ 201.337171] really_probe+0xef/0x420 kern :warn : [ 201.337226] driver_probe_device+0x61/0x120 kern :warn : [ 201.337289] device_driver_attach+0x4f/0x60 kern :warn : [ 201.337353] __driver_attach+0x9a/0x140 kern :warn : [ 201.337411] ? device_driver_attach+0x60/0x60 kern :warn : [ 201.337475] bus_for_each_dev+0x76/0xc0 kern :warn : [ 201.337533] ? klist_add_tail+0x3b/0x70 kern :warn : [ 201.337591] bus_add_driver+0x144/0x220 kern :warn : [ 201.337649] driver_register+0x5b/0xf0 kern :warn : [ 201.337777] i915_init+0x5b/0x6b [i915] kern :warn : [ 201.337835] ? 0xffffffffc064b000 kern :warn : [ 201.337887] do_one_initcall+0x46/0x220 kern :warn : [ 201.337947] ? _cond_resched+0x19/0x30 kern :warn : [ 201.338005] ? kmem_cache_alloc_trace+0x3b/0x230 kern :warn : [ 201.338074] do_init_module+0x5b/0x21d kern :warn : [ 201.338132] load_module+0x1b8f/0x2010 kern :warn : [ 201.338191] ? ima_post_read_file+0xe2/0x120 kern :warn : [ 201.338256] ? __do_sys_finit_module+0xe9/0x110 kern :warn : [ 201.338323] __do_sys_finit_module+0xe9/0x110 kern :warn : [ 201.338391] do_syscall_64+0x5b/0x1f0 kern :warn : [ 201.338448] entry_SYSCALL_64_after_hwframe+0x44/0xa9 kern :warn : [ 201.338522] RIP: 0033:0x7f175a954229 kern :warn : [ 201.338575] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 3f 4c 2b 00 f7 d8 64 89 01 48 kern :warn : [ 201.338829] RSP: 002b:00007ffd11f22ba8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 kern :warn : [ 201.338938] RAX: ffffffffffffffda RBX: 00005632e1713910 RCX: 00007f175a954229 kern :warn : [ 201.339039] RDX: 0000000000000000 RSI: 00007f175b26d265 RDI: 0000000000000013 kern :warn : [ 201.339140] RBP: 00007f175b26d265 R08: 0000000000000000 R09: 00007ffd11f23120 kern :warn : [ 201.339243] R10: 0000000000000013 R11: 0000000000000246 R12: 0000000000000000 kern :warn : [ 201.339345] R13: 00005632e170a0a0 R14: 0000000000020000 R15: 00005632df731cbc kern :warn : [ 201.339448] ---[ end trace 197ca55b67255237 ]--- To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml Thanks, lkp

2 years, 3 months

1
0
0 / 0

2c7abad740 ("kunit: test: create a single centralized executor .."): [ 409.679449] BUG: kernel NULL pointer dereference, address: 00000000

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://github.com/0day-ci/linux/commits/Brendan-Higgins/kunit-create-a-c... commit 2c7abad74010ee4f27b0b1f3c68353171e19967b Author: Alan Maguire <alan.maguire(a)oracle.com> AuthorDate: Thu Jan 30 15:08:08 2020 -0800 Commit: 0day robot <lkp(a)intel.com> CommitDate: Mon Feb 3 03:50:54 2020 +0800 kunit: test: create a single centralized executor for all tests Add a centralized executor to dispatch tests rather than relying on late_initcall to schedule each test suite separately. Centralized execution is for built-in tests only; modules will execute tests when loaded. Signed-off-by: Alan Maguire <alan.maguire(a)oracle.com> Co-developed-by: Iurii Zaikin <yzaikin(a)google.com> Signed-off-by: Iurii Zaikin <yzaikin(a)google.com> Co-developed-by: Brendan Higgins <brendanhiggins(a)google.com> Signed-off-by: Brendan Higgins <brendanhiggins(a)google.com> Reviewed-by: Stephen Boyd <sboyd(a)kernel.org> bf1af76d02 arch: um: add linker section for KUnit test suites 2c7abad740 kunit: test: create a single centralized executor for all tests 11385b32f9 Documentation: Add kunit_shutdown to kernel-parameters.txt +-----------------------------------------------------------+------------+------------+------------+ | | bf1af76d02 | 2c7abad740 | 11385b32f9 | +-----------------------------------------------------------+------------+------------+------------+ | boot_successes | 28 | 1 | 0 | | boot_failures | 7 | 10 | 7 | | WARNING:at_kernel/rcu/rcutorture.c:#rcutorture_oom_notify | 1 | | | | EIP:rcutorture_oom_notify | 1 | | | | Mem-Info | 1 | | | | BUG:kernel_NULL_pointer_dereference,address | 1 | 7 | 7 | | Oops:#[##] | 1 | 7 | 7 | | EIP:rcu_torture_fwd_cb_hist | 1 | | | | Kernel_panic-not_syncing:Fatal_exception | 3 | 7 | 7 | | kernel_BUG_at_mm/vmalloc.c | 2 | | | | invalid_opcode:#[##] | 2 | | | | EIP:alloc_vmap_area | 2 | | | | BUG:kernel_timeout_in_torture_test_stage | 4 | 3 | | | EIP:kunit_run_all_tests | 0 | 7 | 7 | +-----------------------------------------------------------+------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 409.675220] ok 7 - sysctl_test_dointvec_write_happy_single_positive [ 409.675796] ok 8 - sysctl_test_dointvec_write_happy_single_negative [ 409.676601] ok 9 - sysctl_test_api_dointvec_write_single_less_int_min [ 409.678166] ok 10 - sysctl_test_api_dointvec_write_single_greater_int_max [ 409.679009] ok 1 - sysctl_test [ 409.679449] BUG: kernel NULL pointer dereference, address: 00000000 [ 409.680224] #PF: supervisor read access in kernel mode [ 409.680846] #PF: error_code(0x0000) - not-present page [ 409.682123] *pdpt = 0000000000000000 *pde = f000ff53f000ff53 [ 409.682801] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC PTI [ 409.683475] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G T 5.5.0-08693-g2c7abad74010e #1 [ 409.684540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 [ 409.685584] EIP: kunit_run_all_tests+0x22/0x43 [ 409.686112] Code: 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 57 56 31 f6 53 bb b0 44 3a 5d eb 1c 8b 3b eb 0f e8 86 f7 ff ff 85 c0 b0 01 0f 45 f0 83 c7 04 <8b> 07 85 c0 75 eb 83 c3 04 81 fb bc 44 3a 5d 72 dc 89 f0 3c 01 19 [ 409.688277] EAX: 00000000 EBX: 5d3a44b4 ECX: 00000000 EDX: 00000001 [ 409.689019] ESI: 00000000 EDI: 00000000 EBP: 40073f28 ESP: 40073f1c [ 409.689841] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010293 [ 409.690633] CR0: 80050033 CR2: 00000000 CR3: 1d516000 CR4: 003406b0 [ 409.691386] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 409.692226] DR6: fffe0ff0 DR7: 00000400 [ 409.692690] Call Trace: [ 409.693002] ? kunit_try_catch_run+0xd1/0xd1 [ 409.693550] ? do_one_initcall+0xa4/0x1cd [ 409.694053] ? trace_initcall_level+0x7e/0x82 [ 409.694559] ? kernel_init_freeable+0x100/0x16f [ 409.695092] ? kernel_init_freeable+0x118/0x16f [ 409.695628] ? rest_init+0xdc/0xdc [ 409.696032] ? kernel_init+0x8/0xd5 [ 409.696452] ? ret_from_fork+0x2e/0x38 [ 409.696882] Modules linked in: [ 409.697290] CR2: 0000000000000000 [ 409.697628] ---[ end trace 045ceeef56b43c7f ]--- [ 409.697984] EIP: kunit_run_all_tests+0x22/0x43 # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start 7f777d6b2b6ba2646d7facc3c6201073ff3d91ed d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 -- git bisect good 213420b8b9d31ed07f465bfeecc4ac136b12eb2f # 20:37 G 10 0 1 4 Merge 'hwmon/hwmon' into devel-catchup-202002030406 git bisect bad 71e7078961cd57921eccfcea97a5ac01652a3372 # 21:38 B 0 1 17 0 Merge 'linux-review/Brendan-Higgins/kunit-create-a-centralized-executor-to-dispatch-all-KUnit-tests/20200203-035040' into devel-catchup-202002030406 git bisect good a2c8e1e21fa488760d9854643c459b9d13fa1d0e # 22:18 G 10 0 2 2 Merge 'linux-review/Yulia-Kartseva/runqslower-fix-Makefile/20200203-035351' into devel-catchup-202002030406 git bisect good 8f9cc5912605f2f915ba317d01d0076068f0a7e2 # 23:55 G 10 0 2 2 Merge 'linux-review/Vinicius-Costa-Gomes/igc-Add-initial-TSN-qdiscs-offloading/20200203-025747' into devel-catchup-202002030406 git bisect bad 0401f14bd224ed7f2d47b0b9f27bdfbb1552c399 # 00:29 B 0 1 17 0 init: main: add KUnit to kernel init git bisect good bf1af76d02c3998127b3164b1b5e8638f175a602 # 01:39 G 10 0 3 3 arch: um: add linker section for KUnit test suites git bisect bad 2c7abad74010ee4f27b0b1f3c68353171e19967b # 02:24 B 0 1 17 0 kunit: test: create a single centralized executor for all tests # first bad commit: [2c7abad74010ee4f27b0b1f3c68353171e19967b] kunit: test: create a single centralized executor for all tests git bisect good bf1af76d02c3998127b3164b1b5e8638f175a602 # 03:04 G 30 0 2 6 arch: um: add linker section for KUnit test suites # extra tests with debug options git bisect good 2c7abad74010ee4f27b0b1f3c68353171e19967b # 03:42 G 10 0 10 10 kunit: test: create a single centralized executor for all tests # extra tests on head commit of linux-review/Brendan-Higgins/kunit-create-a-centralized-executor-to-dispatch-all-KUnit-tests/20200203-035040 git bisect bad 11385b32f9e52707959f8566f2c719e66159d2e0 # 04:29 B 0 1 17 0 Documentation: Add kunit_shutdown to kernel-parameters.txt # bad: [11385b32f9e52707959f8566f2c719e66159d2e0] Documentation: Add kunit_shutdown to kernel-parameters.txt --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation

2 years, 3 months

3
3
0 / 0

[rcu] 76550daa4d: will-it-scale.per_thread_ops -10.5% regression

by kernel test robot

Greeting, FYI, we noticed a -10.5% regression of will-it-scale.per_thread_ops due to commit: commit: 76550daa4d1edaa8251460bd1d4a11b5df23c1c0 ("rcu: Support kfree_bulk() interface in kfree_rcu()") https://git.kernel.org/cgit/linux/kernel/git/paulmck/linux-rcu.git dev.2020.02.13c in testcase: will-it-scale on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory with following parameters: nr_task: 100% mode: thread test: open2 cpufreq_governor: performance ucode: 0x11 test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/open2/will-it-scale/0x11 commit: e8afd73e56 ("rcu: Don't flag non-starting GPs before GP kthread is running") 76550daa4d ("rcu: Support kfree_bulk() interface in kfree_rcu()") e8afd73e56e02b4d 76550daa4d1edaa8251460bd1d4 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 2:4 -50% :4 dmesg.WARNING:at_ip__fsnotify_parent/0x :4 25% 1:4 dmesg.WARNING:at_ip__slab_free/0x %stddev %change %stddev \ | \ 1021 -10.5% 914.50 will-it-scale.per_thread_ops 807081 -33.7% 534989 ± 2% will-it-scale.time.involuntary_context_switches 489.19 ± 9% -40.3% 292.19 ± 3% will-it-scale.time.user_time 294257 -10.5% 263455 will-it-scale.workload 1.68 ± 2% +30.8% 2.20 turbostat.RAMWatt 6340 -28.8% 4512 ± 2% vmstat.system.cs 0.09 ± 12% -0.0 0.06 ± 3% mpstat.cpu.all.soft% 0.71 ± 7% -0.2 0.49 ± 3% mpstat.cpu.all.usr% 999.25 ± 6% -17.3% 826.50 ± 13% slabinfo.skbuff_fclone_cache.active_objs 999.25 ± 6% -17.3% 826.50 ± 13% slabinfo.skbuff_fclone_cache.num_objs 19880 ± 5% -9.1% 18079 softirqs.CPU100.RCU 21159 ± 2% -6.7% 19750 ± 2% softirqs.CPU13.RCU 122401 ± 10% +23.7% 151373 ± 8% softirqs.CPU144.TIMER 21180 -8.5% 19379 softirqs.CPU15.RCU 21456 ± 7% -11.7% 18945 softirqs.CPU24.RCU 124140 ± 7% +12.3% 139469 ± 7% softirqs.CPU250.TIMER 20964 ± 11% -17.5% 17289 softirqs.CPU261.RCU 20945 -8.8% 19102 softirqs.CPU30.RCU 21274 -9.8% 19194 softirqs.CPU31.RCU 21024 ± 2% -10.3% 18853 softirqs.CPU33.RCU 20571 -8.9% 18746 softirqs.CPU35.RCU 21289 ± 2% -8.0% 19588 softirqs.CPU5.RCU 1199 ± 3% +11.2% 1332 ± 2% sched_debug.cfs_rq:/.exec_clock.stddev 969782 ± 8% -26.1% 716472 ± 22% sched_debug.cfs_rq:/.load.max 226.86 ± 6% -70.5% 66.95 ± 18% sched_debug.cfs_rq:/.nr_spread_over.avg 321.05 ± 5% -47.1% 169.95 ± 6% sched_debug.cfs_rq:/.nr_spread_over.max 107.45 ± 6% -68.8% 33.55 ± 26% sched_debug.cfs_rq:/.nr_spread_over.min 39.24 ± 17% -61.2% 15.24 ± 9% sched_debug.cfs_rq:/.nr_spread_over.stddev 969780 ± 8% -26.1% 716278 ± 22% sched_debug.cfs_rq:/.runnable_weight.max 1145 ± 21% -25.2% 856.65 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.max 169.65 ± 46% -80.8% 32.65 ± 79% sched_debug.cfs_rq:/.util_est_enqueued.min 871.94 ± 5% +23.4% 1075 ± 3% sched_debug.cpu.clock.stddev 871.94 ± 5% +23.4% 1075 ± 3% sched_debug.cpu.clock_task.stddev 5190 ± 11% +22.1% 6338 ± 6% sched_debug.cpu.curr->pid.max 271.88 ± 6% +30.5% 354.73 ± 9% sched_debug.cpu.curr->pid.stddev 0.00 ± 5% +23.8% 0.00 ± 2% sched_debug.cpu.next_balance.stddev 5624 -13.1% 4890 sched_debug.cpu.nr_switches.avg 2344 ± 3% -31.0% 1617 ± 3% sched_debug.cpu.nr_switches.min 2803 -25.7% 2082 ± 2% sched_debug.cpu.sched_count.avg 1906 -30.9% 1316 sched_debug.cpu.sched_count.min 1322 -27.8% 954.60 ± 2% sched_debug.cpu.ttwu_count.avg 919.25 -32.6% 619.90 sched_debug.cpu.ttwu_count.min 1255 -29.3% 888.47 ± 2% sched_debug.cpu.ttwu_local.avg 887.45 -32.9% 595.50 sched_debug.cpu.ttwu_local.min 8.476e+09 +1.3% 8.589e+09 perf-stat.i.branch-instructions 48929091 -3.8% 47071099 perf-stat.i.cache-misses 6396 -30.2% 4467 ± 2% perf-stat.i.context-switches 12.78 -1.4% 12.61 perf-stat.i.cpi 9065 +3.7% 9404 perf-stat.i.cycles-between-cache-misses 71922406 ± 2% -7.7% 66350699 ± 5% perf-stat.i.iTLB-load-misses 3.475e+10 +1.3% 3.519e+10 perf-stat.i.iTLB-loads 3.47e+10 +1.2% 3.512e+10 perf-stat.i.instructions 482.82 ± 2% +10.0% 531.13 ± 6% perf-stat.i.instructions-per-iTLB-miss 0.08 +1.4% 0.08 perf-stat.i.ipc 12.81 -1.3% 12.64 perf-stat.overall.cpi 9077 +3.8% 9423 perf-stat.overall.cycles-between-cache-misses 0.21 ± 2% -0.0 0.19 ± 5% perf-stat.overall.iTLB-load-miss-rate% 483.60 ± 2% +9.9% 531.72 ± 6% perf-stat.overall.instructions-per-iTLB-miss 0.08 +1.3% 0.08 perf-stat.overall.ipc 36101370 +13.0% 40812598 perf-stat.overall.path-length 8.46e+09 +1.2% 8.565e+09 perf-stat.ps.branch-instructions 48864485 -3.9% 46979669 perf-stat.ps.cache-misses 6244 -29.3% 4417 ± 2% perf-stat.ps.context-switches 71662456 ± 2% -7.8% 66078101 ± 5% perf-stat.ps.iTLB-load-misses 3.466e+10 +1.2% 3.507e+10 perf-stat.ps.iTLB-loads 3.463e+10 +1.1% 3.502e+10 perf-stat.ps.instructions 1.062e+13 +1.2% 1.075e+13 perf-stat.total.instructions 1.31 ± 2% -0.2 1.16 ± 4% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.26 ± 2% -0.1 1.13 ± 3% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64 48.05 +0.1 48.16 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__alloc_fd.do_sys_openat2.do_sys_open 48.09 +0.1 48.21 perf-profile.calltrace.cycles-pp._raw_spin_lock.__alloc_fd.do_sys_openat2.do_sys_open.do_syscall_64 48.16 +0.2 48.31 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__close_fd.__x64_sys_close.do_syscall_64 48.20 +0.2 48.37 perf-profile.calltrace.cycles-pp._raw_spin_lock.__close_fd.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 48.32 +0.2 48.49 perf-profile.calltrace.cycles-pp.__close_fd.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_close 48.24 +0.2 48.41 perf-profile.calltrace.cycles-pp.__alloc_fd.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 48.46 +0.2 48.64 perf-profile.calltrace.cycles-pp.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_close 1.32 ± 2% -0.2 1.16 ± 3% perf-profile.children.cycles-pp.do_filp_open 0.35 ± 25% -0.1 0.21 ± 14% perf-profile.children.cycles-pp.update_curr 1.27 ± 2% -0.1 1.14 ± 4% perf-profile.children.cycles-pp.path_openat 0.44 ± 2% -0.1 0.37 ± 2% perf-profile.children.cycles-pp.alloc_empty_file 0.19 ± 5% -0.1 0.12 ± 4% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.33 ± 3% -0.1 0.27 ± 5% perf-profile.children.cycles-pp.__fput 0.50 ± 3% -0.1 0.44 ± 5% perf-profile.children.cycles-pp.exit_to_usermode_loop 0.15 ± 4% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.security_file_alloc 0.45 ± 3% -0.1 0.40 ± 4% perf-profile.children.cycles-pp.task_work_run 0.39 ± 2% -0.0 0.35 ± 2% perf-profile.children.cycles-pp.__alloc_file 0.32 -0.0 0.29 ± 3% perf-profile.children.cycles-pp.link_path_walk 0.30 ± 5% -0.0 0.27 ± 4% perf-profile.children.cycles-pp.rcu_do_batch 0.30 ± 4% -0.0 0.28 ± 4% perf-profile.children.cycles-pp.rcu_core 0.22 -0.0 0.19 ± 2% perf-profile.children.cycles-pp.kmem_cache_free 0.12 ± 5% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.inode_permission 0.06 ± 7% +0.0 0.07 ± 5% perf-profile.children.cycles-pp.___might_sleep 0.06 ± 11% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.run_timer_softirq 0.26 ± 2% +0.0 0.29 ± 3% perf-profile.children.cycles-pp.kmem_cache_alloc 0.10 ± 9% +0.0 0.14 ± 8% perf-profile.children.cycles-pp.enqueue_hrtimer 0.08 ± 10% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.timerqueue_add 0.04 ± 57% +0.1 0.09 ± 11% perf-profile.children.cycles-pp.expand_files 0.05 ± 8% +0.1 0.11 ± 7% perf-profile.children.cycles-pp.perf_event_task_tick 0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.__might_sleep 0.00 +0.1 0.06 ± 20% perf-profile.children.cycles-pp.locks_remove_posix 0.11 ± 7% +0.1 0.17 ± 17% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.08 ± 15% +0.1 0.15 ± 19% perf-profile.children.cycles-pp.rcu_irq_enter 99.41 +0.1 99.49 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 99.37 +0.1 99.46 perf-profile.children.cycles-pp.do_syscall_64 48.32 +0.2 48.49 perf-profile.children.cycles-pp.__close_fd 48.27 +0.2 48.44 perf-profile.children.cycles-pp.__alloc_fd 48.46 +0.2 48.64 perf-profile.children.cycles-pp.__x64_sys_close 96.38 +0.3 96.64 perf-profile.children.cycles-pp._raw_spin_lock 96.31 +0.3 96.59 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.19 ± 5% -0.1 0.12 ± 4% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.11 ± 4% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.__alloc_file 0.21 ± 6% -0.0 0.18 ± 4% perf-profile.self.cycles-pp.file_free_rcu 0.08 ± 5% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.inode_permission 0.15 ± 3% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.kmem_cache_free 0.07 +0.0 0.09 perf-profile.self.cycles-pp.do_syscall_64 0.03 ±100% +0.0 0.06 ± 6% perf-profile.self.cycles-pp.run_timer_softirq 0.06 ± 6% +0.0 0.11 ± 8% perf-profile.self.cycles-pp.timerqueue_add 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.__might_sleep 0.05 ± 8% +0.1 0.11 ± 7% perf-profile.self.cycles-pp.perf_event_task_tick 0.00 +0.1 0.06 ± 20% perf-profile.self.cycles-pp.locks_remove_posix 0.11 ± 4% +0.1 0.17 ± 17% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.00 +0.1 0.06 ± 17% perf-profile.self.cycles-pp.expand_files 0.08 ± 15% +0.1 0.15 ± 19% perf-profile.self.cycles-pp.rcu_irq_enter 396.75 ± 31% +53.1% 607.25 ± 12% interrupts.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2 6719 ± 24% -42.5% 3865 interrupts.CPU0.NMI:Non-maskable_interrupts 6719 ± 24% -42.5% 3865 interrupts.CPU0.PMI:Performance_monitoring_interrupts 1541 ± 14% -38.1% 954.75 ± 32% interrupts.CPU0.RES:Rescheduling_interrupts 85.00 ± 34% -74.4% 21.75 ± 54% interrupts.CPU104.RES:Rescheduling_interrupts 67.75 ±109% -80.8% 13.00 ± 14% interrupts.CPU105.RES:Rescheduling_interrupts 5717 ± 32% -33.3% 3814 interrupts.CPU106.NMI:Non-maskable_interrupts 5717 ± 32% -33.3% 3814 interrupts.CPU106.PMI:Performance_monitoring_interrupts 3828 +73.2% 6630 ± 24% interrupts.CPU109.NMI:Non-maskable_interrupts 3828 +73.2% 6630 ± 24% interrupts.CPU109.PMI:Performance_monitoring_interrupts 396.75 ± 31% +53.1% 607.25 ± 12% interrupts.CPU12.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2 5694 ± 32% -33.7% 3777 interrupts.CPU131.NMI:Non-maskable_interrupts 5694 ± 32% -33.7% 3777 interrupts.CPU131.PMI:Performance_monitoring_interrupts 4725 ± 34% +58.8% 7506 interrupts.CPU173.NMI:Non-maskable_interrupts 4725 ± 34% +58.8% 7506 interrupts.CPU173.PMI:Performance_monitoring_interrupts 6578 ± 24% -42.9% 3756 interrupts.CPU179.NMI:Non-maskable_interrupts 6578 ± 24% -42.9% 3756 interrupts.CPU179.PMI:Performance_monitoring_interrupts 5716 ± 33% -17.2% 4735 ± 35% interrupts.CPU183.NMI:Non-maskable_interrupts 5716 ± 33% -17.2% 4735 ± 35% interrupts.CPU183.PMI:Performance_monitoring_interrupts 4686 ± 34% +41.7% 6639 ± 24% interrupts.CPU187.NMI:Non-maskable_interrupts 4686 ± 34% +41.7% 6639 ± 24% interrupts.CPU187.PMI:Performance_monitoring_interrupts 63.00 ± 81% -66.3% 21.25 ± 91% interrupts.CPU187.RES:Rescheduling_interrupts 5799 ± 32% -17.4% 4787 ± 33% interrupts.CPU20.NMI:Non-maskable_interrupts 5799 ± 32% -17.4% 4787 ± 33% interrupts.CPU20.PMI:Performance_monitoring_interrupts 3802 +48.6% 5649 ± 31% interrupts.CPU207.NMI:Non-maskable_interrupts 3802 +48.6% 5649 ± 31% interrupts.CPU207.PMI:Performance_monitoring_interrupts 51.25 ± 96% -82.4% 9.00 ± 7% interrupts.CPU211.RES:Rescheduling_interrupts 5692 ± 30% -17.6% 4690 ± 32% interrupts.CPU215.NMI:Non-maskable_interrupts 5692 ± 30% -17.6% 4690 ± 32% interrupts.CPU215.PMI:Performance_monitoring_interrupts 70.00 ±116% -80.7% 13.50 ± 49% interrupts.CPU217.RES:Rescheduling_interrupts 7.25 ± 35% +903.4% 72.75 ± 85% interrupts.CPU222.RES:Rescheduling_interrupts 3786 +49.4% 5655 ± 32% interrupts.CPU224.NMI:Non-maskable_interrupts 3786 +49.4% 5655 ± 32% interrupts.CPU224.PMI:Performance_monitoring_interrupts 3778 +48.2% 5599 ± 31% interrupts.CPU227.NMI:Non-maskable_interrupts 3778 +48.2% 5599 ± 31% interrupts.CPU227.PMI:Performance_monitoring_interrupts 66.75 ± 97% -86.1% 9.25 ± 20% interrupts.CPU228.RES:Rescheduling_interrupts 4710 ± 33% +60.1% 7539 interrupts.CPU240.NMI:Non-maskable_interrupts 4710 ± 33% +60.1% 7539 interrupts.CPU240.PMI:Performance_monitoring_interrupts 18.75 ± 40% +618.7% 134.75 ± 69% interrupts.CPU240.RES:Rescheduling_interrupts 12.50 ± 30% +582.0% 85.25 ± 81% interrupts.CPU25.RES:Rescheduling_interrupts 4672 ± 33% +39.4% 6515 ± 23% interrupts.CPU284.NMI:Non-maskable_interrupts 4672 ± 33% +39.4% 6515 ± 23% interrupts.CPU284.PMI:Performance_monitoring_interrupts 48.75 ±123% -78.5% 10.50 ± 24% interrupts.CPU284.RES:Rescheduling_interrupts 6705 ± 24% -28.7% 4781 ± 34% interrupts.CPU5.NMI:Non-maskable_interrupts 6705 ± 24% -28.7% 4781 ± 34% interrupts.CPU5.PMI:Performance_monitoring_interrupts 3801 +73.9% 6612 ± 24% interrupts.CPU52.NMI:Non-maskable_interrupts 3801 +73.9% 6612 ± 24% interrupts.CPU52.PMI:Performance_monitoring_interrupts 4731 ± 34% +40.3% 6638 ± 24% interrupts.CPU53.NMI:Non-maskable_interrupts 4731 ± 34% +40.3% 6638 ± 24% interrupts.CPU53.PMI:Performance_monitoring_interrupts 3808 +49.4% 5689 ± 32% interrupts.CPU54.NMI:Non-maskable_interrupts 3808 +49.4% 5689 ± 32% interrupts.CPU54.PMI:Performance_monitoring_interrupts 5676 ± 32% -16.8% 4720 ± 34% interrupts.CPU59.NMI:Non-maskable_interrupts 5676 ± 32% -16.8% 4720 ± 34% interrupts.CPU59.PMI:Performance_monitoring_interrupts 3778 +50.0% 5666 ± 32% interrupts.CPU63.NMI:Non-maskable_interrupts 3778 +50.0% 5666 ± 32% interrupts.CPU63.PMI:Performance_monitoring_interrupts 5739 ± 32% -18.0% 4706 ± 33% interrupts.CPU72.NMI:Non-maskable_interrupts 5739 ± 32% -18.0% 4706 ± 33% interrupts.CPU72.PMI:Performance_monitoring_interrupts 4750 ± 33% +59.4% 7571 interrupts.CPU87.NMI:Non-maskable_interrupts 4750 ± 33% +59.4% 7571 interrupts.CPU87.PMI:Performance_monitoring_interrupts 5720 ± 33% -34.0% 3778 interrupts.CPU98.NMI:Non-maskable_interrupts 5720 ± 33% -34.0% 3778 interrupts.CPU98.PMI:Performance_monitoring_interrupts will-it-scale.per_thread_ops 1040 +--------------------------------------------------------------------+ | +.. .+.| 1020 |-+ .+ + +.+ | 1000 |-+ .+. .+ + .+.. +.+ .+ .+..+.+. + | |.+.+.+. + + + + .+.+..+ + .+.+.+ +. + | 980 |-+ + + + + | 960 |-+ | | | 940 |-+ | 920 |-+ O O O O O O | | O O O O O O O | 900 |-+ O O O O O O O O | 880 |-+ O O O O O | | | 860 +--------------------------------------------------------------------+ will-it-scale.workload 300000 +------------------------------------------------------------------+ 295000 |-+ .| | + .+.+.+.+ | 290000 |-+ + : .+.+ + | 285000 |-+ .+.+.+..+ : .+. .+.+. .+.+. .+.+ + .. | |.+.+ + + +.+.+. +.+.+ +.+ | 280000 |-+ | 275000 |-+ | 270000 |-+ | | O O | 265000 |-+ O O O O O O | 260000 |-O O O O O O O O | | O O O O O O O O | 255000 |-+ O O | 250000 +------------------------------------------------------------------+ will-it-scale.time.involuntary_context_switches 850000 +------------------------------------------------------------------+ | .+.. +.+ +. + +.+.| 800000 |-+. .+.+ +. + : : + +. .+.+ +. .+.+.+.+ + + + | |+ + + : : + + +. + + + + + + | 750000 |-+ + + + +. .. | | + | 700000 |-+ | | | 650000 |-+ | | | 600000 |-+ | | | 550000 |-+ O O O O O O O | | O O O O O O O O O O O O O O O O O O O | 500000 +------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 years, 3 months

2
1
0 / 0

[mm/workingset] 323c95f095: fio.read_bw_MBps 19.5% improvement

by kernel test robot

Greeting, FYI, we noticed a 19.5% improvement of fio.read_bw_MBps due to commit: commit: 323c95f0957c8a7d59cd9d05f43f19dfc9bbc584 ("[PATCH 3/9] mm/workingset: extend the workingset detection for anon LRU") url: https://github.com/0day-ci/linux/commits/js1304-gmail-com/workingset-prot... in testcase: fio-basic on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory with following parameters: disk: 2pmem fs: ext4 runtime: 200s nr_task: 50% time_based: tb rw: randrw bs: 2M ioengine: mmap test_size: 200G cpufreq_governor: performance ucode: 0x500002c test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. test-url: https://github.com/axboe/fio Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode: 2M/gcc-7/performance/2pmem/ext4/mmap/x86_64-rhel-7.6/50%/debian-x86_64-20191114.cgz/200s/randrw/lkp-csl-2sp6/200G/fio-basic/tb/0x500002c commit: fffae40d0e ("mm/vmscan: protect the workingset on anonymous LRU") 323c95f095 ("mm/workingset: extend the workingset detection for anon LRU") fffae40d0ee3fd1e 323c95f0957c8a7d59cd9d05f43 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at_ip___perf_sw_event/0x :4 25% 1:4 dmesg.WARNING:at_ip_generic_make_request/0x 2:4 45% 4:4 perf-profile.calltrace.cycles-pp.sync_regs.error_entry 2:4 50% 4:4 perf-profile.calltrace.cycles-pp.error_entry 2:4 53% 4:4 perf-profile.children.cycles-pp.error_entry 0:4 6% 0:4 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 0.28 ± 11% -0.1 0.16 ± 28% fio.latency_100ms% 8.36 ± 8% -5.9 2.50 ± 5% fio.latency_10ms% 6.10 ± 17% -5.3 0.82 ± 6% fio.latency_20ms% 0.00 ±173% +0.0 0.01 ± 10% fio.latency_250ms% 0.71 ± 16% +2.8 3.56 ± 4% fio.latency_250us% 13.45 ± 5% +9.4 22.85 ± 2% fio.latency_2ms% 29.90 ± 2% -7.3 22.58 ± 2% fio.latency_4ms% 8.43 ± 7% +9.4 17.84 ± 4% fio.latency_500us% 7.43 ± 3% -3.3 4.13 ± 12% fio.latency_750us% 4336 ± 2% +19.5% 5182 ± 2% fio.read_bw_MBps 7430144 ± 16% -64.3% 2654208 fio.read_clat_90%_us 11501568 ± 9% -70.8% 3358720 fio.read_clat_95%_us 17170432 ± 8% -57.7% 7258112 ± 4% fio.read_clat_99%_us 2960419 ± 7% -53.3% 1381107 fio.read_clat_mean_us 4105907 ± 14% -45.6% 2232352 ± 11% fio.read_clat_stddev 2168 ± 2% +19.5% 2591 ± 2% fio.read_iops 1.942e+09 ± 2% -22.0% 1.514e+09 ± 2% fio.time.file_system_inputs 1.783e+09 ± 2% +19.5% 2.131e+09 ± 2% fio.time.file_system_outputs 216950 ± 12% -48.8% 111051 ± 16% fio.time.involuntary_context_switches 2.427e+08 ± 2% -22.0% 1.892e+08 ± 2% fio.time.major_page_faults 1.271e+08 ± 5% +66.8% 2.12e+08 ± 2% fio.time.minor_page_faults 1619 ± 9% -33.6% 1074 ± 2% fio.time.percent_of_cpu_this_job_got 2851 ± 9% -40.1% 1708 ± 2% fio.time.system_time 404.54 ± 6% +11.4% 450.81 ± 2% fio.time.user_time 869238 ± 2% +19.5% 1038826 ± 2% fio.workload 4350 ± 2% +19.5% 5199 ± 2% fio.write_bw_MBps 38404096 -12.9% 33456128 fio.write_clat_90%_us 39452672 -14.0% 33947648 fio.write_clat_95%_us 45744128 -19.5% 36831232 ± 3% fio.write_clat_99%_us 17993869 ± 4% -9.2% 16330675 ± 2% fio.write_clat_mean_us 16656042 ± 5% -6.2% 15615331 fio.write_clat_stddev 2175 ± 2% +19.5% 2599 ± 2% fio.write_iops 31.58 ± 4% +5.8 37.39 mpstat.cpu.all.iowait% 18.54 ± 7% -6.5 12.04 ± 2% mpstat.cpu.all.sys% 1.008e+08 ± 10% -37.9% 62633861 ± 18% numa-numastat.node0.local_node 1.008e+08 ± 9% -37.9% 62658517 ± 18% numa-numastat.node0.numa_hit 48.29 +1.1% 48.80 iostat.cpu.idle 31.27 ± 4% +18.4% 37.03 iostat.cpu.iowait 18.38 ± 7% -35.0% 11.94 ± 2% iostat.cpu.system 618.00 ± 6% -27.0% 451.25 turbostat.Avg_MHz 22.14 ± 6% -6.0 16.17 turbostat.Busy% 1.361e+08 ± 7% -26.6% 99961074 ± 3% turbostat.IRQ 224.76 -4.7% 214.16 turbostat.PkgWatt 4580495 +85.7% 8505839 slabinfo.buffer_head.active_objs 119518 +82.7% 218335 slabinfo.buffer_head.active_slabs 4661221 +82.7% 8515085 slabinfo.buffer_head.num_objs 119518 +82.7% 218335 slabinfo.buffer_head.num_slabs 19148 ± 3% +9.2% 20917 ± 3% slabinfo.kmalloc-512.num_objs 30.75 ± 4% +19.5% 36.75 vmstat.cpu.wa 4753706 ± 2% -21.9% 3710468 ± 2% vmstat.io.bi 4361808 ± 2% +19.7% 5220006 ± 2% vmstat.io.bo 619.75 ± 8% +86.0% 1153 ± 5% vmstat.memory.buff 31.25 ± 4% +15.2% 36.00 vmstat.procs.b 19.00 ± 9% -28.9% 13.50 ± 3% vmstat.procs.r 409038 ± 5% -18.7% 332463 ± 2% vmstat.system.in 9785202 +221.1% 31421653 meminfo.Active 60190 ± 8% -34.3% 39569 ± 3% meminfo.Active(anon) 9725011 +222.7% 31382083 meminfo.Active(file) 619.25 ± 7% +86.7% 1156 ± 5% meminfo.Buffers 32198569 -68.1% 10269890 meminfo.Inactive 28407610 -77.2% 6464247 meminfo.Inactive(file) 650015 +60.3% 1041779 meminfo.KReclaimable 650015 +60.3% 1041779 meminfo.SReclaimable 829409 +47.3% 1221631 meminfo.Slab 2564 ± 4% +42.5% 3654 ± 4% meminfo.Writeback 4673366 ± 9% +204.3% 14223125 numa-meminfo.node0.Active 4670584 ± 9% +204.4% 14218202 numa-meminfo.node0.Active(file) 3328794 ± 5% -10.7% 2972288 numa-meminfo.node0.Dirty 16103425 ± 3% -60.3% 6392546 ± 2% numa-meminfo.node0.Inactive 13636450 ± 5% -77.2% 3114750 ± 3% numa-meminfo.node0.Inactive(file) 361437 ± 2% +64.5% 594674 ± 2% numa-meminfo.node0.KReclaimable 361437 ± 2% +64.5% 594674 ± 2% numa-meminfo.node0.SReclaimable 454217 +51.0% 686089 ± 2% numa-meminfo.node0.Slab 5117666 ± 8% +236.1% 17201387 numa-meminfo.node1.Active 57417 ± 8% -39.6% 34652 ± 7% numa-meminfo.node1.Active(anon) 5060248 ± 8% +239.2% 17166734 numa-meminfo.node1.Active(file) 16086677 ± 3% -76.0% 3867738 ± 5% numa-meminfo.node1.Inactive 14767296 ± 5% -77.4% 3343591 ± 2% numa-meminfo.node1.Inactive(file) 288589 ± 3% +54.9% 446992 ± 3% numa-meminfo.node1.KReclaimable 288589 ± 3% +54.9% 446992 ± 3% numa-meminfo.node1.SReclaimable 375202 ± 2% +42.7% 535430 ± 2% numa-meminfo.node1.Slab 1310 ± 9% +57.4% 2062 ± 5% numa-meminfo.node1.Writeback 0.02 ± 38% +151.1% 0.04 ± 4% sched_debug.cfs_rq:/.nr_spread_over.avg 0.14 ± 30% +62.3% 0.22 ± 11% sched_debug.cfs_rq:/.nr_spread_over.stddev 702.00 ± 3% +20.3% 844.48 ± 11% sched_debug.cfs_rq:/.runnable_load_avg.max 8658 ± 41% -88.5% 994.62 ±366% sched_debug.cfs_rq:/.spread0.avg -5693 +147.0% -14059 sched_debug.cfs_rq:/.spread0.min 280.56 ± 11% -20.9% 221.88 ± 5% sched_debug.cfs_rq:/.util_avg.avg 16889 ± 8% +31.2% 22166 ± 13% sched_debug.cpu.nr_switches.avg 0.21 ± 23% +44.0% 0.30 ± 7% sched_debug.cpu.nr_uninterruptible.avg 15025 ± 9% +35.5% 20353 ± 14% sched_debug.cpu.sched_count.avg 16991 ± 21% +31.0% 22263 ± 16% sched_debug.cpu.sched_count.stddev 6755 ± 11% +39.0% 9388 ± 14% sched_debug.cpu.sched_goidle.avg 478.75 ± 16% +37.8% 659.79 ± 19% sched_debug.cpu.sched_goidle.min 8464 ± 21% +31.2% 11108 ± 16% sched_debug.cpu.sched_goidle.stddev 7465 ± 9% +35.5% 10114 ± 14% sched_debug.cpu.ttwu_count.avg 3184 ± 15% +79.5% 5715 ± 14% sched_debug.cpu.ttwu_local.max 349.50 ± 15% +32.8% 464.02 ± 16% sched_debug.cpu.ttwu_local.min 516.10 ± 9% +58.8% 819.50 ± 16% sched_debug.cpu.ttwu_local.stddev 135179 +16.5% 157474 ± 8% sched_debug.ktime 20801 ± 9% +15.2% 23973 ± 3% softirqs.CPU13.SCHED 21834 ± 5% +11.4% 24317 ± 4% softirqs.CPU15.SCHED 20068 ± 13% +26.6% 25416 ± 2% softirqs.CPU18.SCHED 21227 ± 8% +18.3% 25107 ± 2% softirqs.CPU19.SCHED 21493 ± 6% +14.2% 24542 ± 4% softirqs.CPU21.SCHED 21659 ± 5% +17.5% 25448 ± 3% softirqs.CPU22.SCHED 21989 ± 4% +17.2% 25773 ± 2% softirqs.CPU23.SCHED 20132 ± 14% +23.2% 24812 ± 2% softirqs.CPU25.SCHED 20294 ± 16% +23.7% 25112 ± 5% softirqs.CPU26.SCHED 21996 ± 2% +14.2% 25115 ± 2% softirqs.CPU3.SCHED 21985 ± 8% +17.2% 25765 ± 3% softirqs.CPU31.SCHED 21974 ± 6% +14.6% 25184 softirqs.CPU37.SCHED 21943 +12.4% 24665 ± 3% softirqs.CPU40.SCHED 21177 ± 9% +17.8% 24954 ± 2% softirqs.CPU41.SCHED 21555 ± 4% +16.1% 25018 softirqs.CPU43.SCHED 21565 ± 2% +15.0% 24804 softirqs.CPU45.SCHED 22158 ± 6% +14.9% 25466 ± 3% softirqs.CPU50.SCHED 22571 ± 7% +13.4% 25597 softirqs.CPU51.SCHED 21477 ± 7% +17.8% 25310 softirqs.CPU53.SCHED 21648 ± 4% +15.7% 25037 ± 2% softirqs.CPU54.SCHED 22303 ± 4% +10.0% 24524 ± 5% softirqs.CPU57.SCHED 22250 ± 8% +13.7% 25302 ± 2% softirqs.CPU6.SCHED 21945 ± 6% +15.1% 25253 softirqs.CPU62.SCHED 20860 ± 9% +23.0% 25655 ± 3% softirqs.CPU64.SCHED 22491 ± 4% +7.5% 24180 ± 5% softirqs.CPU81.SCHED 19570 ± 21% +28.5% 25157 ± 2% softirqs.CPU83.SCHED 1167199 ± 9% +204.5% 3554443 numa-vmstat.node0.nr_active_file 833733 ± 5% -11.3% 739589 numa-vmstat.node0.nr_dirty 3409188 ± 5% -77.1% 779009 ± 3% numa-vmstat.node0.nr_inactive_file 83.50 ± 22% -60.8% 32.75 ± 16% numa-vmstat.node0.nr_isolated_file 90376 ± 2% +64.5% 148639 ± 2% numa-vmstat.node0.nr_slab_reclaimable 1394052 ± 7% +282.6% 5333857 ± 17% numa-vmstat.node0.nr_vmscan_immediate_reclaim 315.25 ± 3% +22.5% 386.25 ± 13% numa-vmstat.node0.nr_writeback 1167965 ± 9% +204.3% 3554646 numa-vmstat.node0.nr_zone_active_file 3409621 ± 5% -77.1% 780149 ± 3% numa-vmstat.node0.nr_zone_inactive_file 835951 ± 5% -11.3% 741848 numa-vmstat.node0.nr_zone_write_pending 62116218 ± 6% -31.7% 42426214 ± 23% numa-vmstat.node0.numa_hit 62089335 ± 6% -31.8% 42365222 ± 23% numa-vmstat.node0.numa_local 18221559 ± 10% -100.0% 0.00 numa-vmstat.node0.workingset_activate 13281 ± 7% -22.3% 10326 ± 8% numa-vmstat.node0.workingset_nodes 33316100 ± 6% -100.0% 0.00 numa-vmstat.node0.workingset_refault 10372861 ± 12% -100.0% 1.25 ± 66% numa-vmstat.node0.workingset_restore 14346 ± 8% -39.6% 8669 ± 7% numa-vmstat.node1.nr_active_anon 1263972 ± 8% +239.6% 4292221 numa-vmstat.node1.nr_active_file 61372170 ± 9% +31.3% 80554020 ± 8% numa-vmstat.node1.nr_dirtied 3692694 ± 5% -77.4% 835792 numa-vmstat.node1.nr_inactive_file 72150 ± 3% +54.8% 111693 ± 3% numa-vmstat.node1.nr_slab_reclaimable 1341628 ± 12% +750.8% 11415031 ± 6% numa-vmstat.node1.nr_vmscan_immediate_reclaim 331.25 ± 10% +66.0% 549.75 ± 9% numa-vmstat.node1.nr_writeback 60469655 ± 9% +31.7% 79613669 ± 8% numa-vmstat.node1.nr_written 14347 ± 8% -39.6% 8670 ± 7% numa-vmstat.node1.nr_zone_active_anon 1264780 ± 8% +239.4% 4292348 numa-vmstat.node1.nr_zone_active_file 3693150 ± 5% -77.3% 837009 numa-vmstat.node1.nr_zone_inactive_file 19519796 ± 8% -100.0% 0.00 numa-vmstat.node1.workingset_activate 12632 ± 5% -20.7% 10012 ± 4% numa-vmstat.node1.workingset_nodes 36195648 ± 4% -100.0% 0.00 numa-vmstat.node1.workingset_refault 11279550 ± 9% -100.0% 0.25 ±173% numa-vmstat.node1.workingset_restore 15.72 +11.1% 17.46 perf-stat.i.MPKI 7.221e+09 ± 3% -14.9% 6.144e+09 ± 2% perf-stat.i.branch-instructions 29447507 ± 3% -13.8% 25382627 ± 2% perf-stat.i.branch-misses 67.78 +2.0 69.77 perf-stat.i.cache-miss-rate% 1.55 ± 4% -14.7% 1.33 perf-stat.i.cpi 5.857e+10 ± 7% -27.4% 4.252e+10 perf-stat.i.cpu-cycles 148.26 ± 2% -12.2% 130.23 perf-stat.i.cpu-migrations 154.03 ± 3% -24.7% 115.95 perf-stat.i.cycles-between-cache-misses 0.20 ± 2% -0.0 0.15 ± 3% perf-stat.i.dTLB-load-miss-rate% 20033658 ± 4% -34.6% 13108913 perf-stat.i.dTLB-load-misses 9.948e+09 ± 3% -13.4% 8.617e+09 ± 3% perf-stat.i.dTLB-loads 0.37 ± 3% -0.1 0.28 ± 3% perf-stat.i.dTLB-store-miss-rate% 20362529 ± 4% -30.4% 14162666 ± 5% perf-stat.i.dTLB-store-misses 5.532e+09 ± 2% -10.6% 4.945e+09 ± 2% perf-stat.i.dTLB-stores 73.19 -1.4 71.76 perf-stat.i.iTLB-load-miss-rate% 12785996 ± 2% -10.5% 11440556 ± 3% perf-stat.i.iTLB-load-misses 4812663 ± 2% -6.0% 4526122 perf-stat.i.iTLB-loads 3.712e+10 ± 3% -14.2% 3.185e+10 ± 2% perf-stat.i.instructions 2895 -4.1% 2777 perf-stat.i.instructions-per-iTLB-miss 0.66 ± 4% +16.5% 0.77 perf-stat.i.ipc 1203000 ± 2% -22.0% 938451 ± 2% perf-stat.i.major-faults 632441 ± 5% +66.7% 1054315 ± 2% perf-stat.i.minor-faults 1835441 ± 3% +8.6% 1992767 ± 2% perf-stat.i.page-faults 15.80 +10.3% 17.43 perf-stat.overall.MPKI 67.75 +2.1 69.87 perf-stat.overall.cache-miss-rate% 1.58 ± 4% -15.3% 1.34 perf-stat.overall.cpi 147.21 ± 3% -25.5% 109.71 perf-stat.overall.cycles-between-cache-misses 0.20 ± 2% -0.0 0.15 ± 3% perf-stat.overall.dTLB-load-miss-rate% 0.37 ± 3% -0.1 0.29 ± 3% perf-stat.overall.dTLB-store-miss-rate% 72.65 -1.0 71.64 perf-stat.overall.iTLB-load-miss-rate% 2902 -4.1% 2784 perf-stat.overall.instructions-per-iTLB-miss 0.64 ± 4% +17.8% 0.75 perf-stat.overall.ipc 8598809 -28.3% 6163102 perf-stat.overall.path-length 7.185e+09 ± 3% -14.9% 6.114e+09 ± 2% perf-stat.ps.branch-instructions 29306041 ± 3% -13.8% 25261462 ± 2% perf-stat.ps.branch-misses 5.829e+10 ± 7% -27.4% 4.232e+10 perf-stat.ps.cpu-cycles 147.55 ± 2% -12.2% 129.61 perf-stat.ps.cpu-migrations 19935636 ± 4% -34.6% 13047562 perf-stat.ps.dTLB-load-misses 9.899e+09 ± 3% -13.4% 8.575e+09 ± 2% perf-stat.ps.dTLB-loads 20263384 ± 4% -30.4% 14096250 ± 5% perf-stat.ps.dTLB-store-misses 5.505e+09 ± 2% -10.6% 4.92e+09 ± 2% perf-stat.ps.dTLB-stores 12723152 ± 2% -10.5% 11384126 ± 3% perf-stat.ps.iTLB-load-misses 4789226 ± 2% -6.0% 4503341 perf-stat.ps.iTLB-loads 3.694e+10 ± 3% -14.2% 3.169e+10 ± 2% perf-stat.ps.instructions 1197099 ± 2% -22.0% 933922 ± 2% perf-stat.ps.major-faults 629469 ± 5% +66.7% 1049045 ± 2% perf-stat.ps.minor-faults 1826569 ± 3% +8.6% 1982969 ± 2% perf-stat.ps.page-faults 7.476e+12 ± 3% -14.4% 6.403e+12 ± 2% perf-stat.total.instructions 104876 ± 15% -55.2% 46978 ± 8% proc-vmstat.allocstall_movable 6670 ± 13% +90.1% 12681 ± 23% proc-vmstat.allocstall_normal 270.00 ± 34% -62.4% 101.50 ± 33% proc-vmstat.compact_fail 277.50 ± 33% -61.0% 108.25 ± 36% proc-vmstat.compact_stall 629.25 ± 35% +596.1% 4380 ± 8% proc-vmstat.kswapd_low_wmark_hit_quickly 15034 ± 8% -34.2% 9894 ± 3% proc-vmstat.nr_active_anon 2432763 +222.5% 7846521 proc-vmstat.nr_active_file 2.229e+08 ± 2% +19.5% 2.664e+08 ± 2% proc-vmstat.nr_dirtied 1732719 -3.1% 1679210 proc-vmstat.nr_dirty 280680 -4.7% 267623 proc-vmstat.nr_free_pages 7100824 -77.3% 1613637 proc-vmstat.nr_inactive_file 164.75 ± 9% -54.9% 74.25 ± 8% proc-vmstat.nr_isolated_file 9582341 -2.0% 9389351 proc-vmstat.nr_mapped 121263 -4.0% 116373 proc-vmstat.nr_shmem 162508 +60.3% 260428 proc-vmstat.nr_slab_reclaimable 5464481 ± 4% +578.9% 37099437 ± 2% proc-vmstat.nr_vmscan_immediate_reclaim 639.25 ± 3% +43.5% 917.50 ± 4% proc-vmstat.nr_writeback 2.217e+08 ± 2% +19.4% 2.646e+08 ± 2% proc-vmstat.nr_written 15036 ± 8% -34.2% 9895 ± 3% proc-vmstat.nr_zone_active_anon 2434338 +222.3% 7846813 proc-vmstat.nr_zone_active_file 7101719 -77.2% 1615989 proc-vmstat.nr_zone_inactive_file 1737241 -3.1% 1683939 proc-vmstat.nr_zone_write_pending 40016131 ± 6% -16.6% 33361886 ± 6% proc-vmstat.numa_foreign 25745506 ± 14% +56.4% 40263559 ± 5% proc-vmstat.numa_hint_faults 19974893 ± 14% +49.8% 29926746 ± 8% proc-vmstat.numa_hint_faults_local 2.039e+08 ± 2% -22.5% 1.581e+08 ± 2% proc-vmstat.numa_hit 1808 ± 2% -3.3% 1749 proc-vmstat.numa_huge_pte_updates 2.039e+08 ± 2% -22.5% 1.58e+08 ± 2% proc-vmstat.numa_local 40016131 ± 6% -16.6% 33361886 ± 6% proc-vmstat.numa_miss 40047410 ± 6% -16.6% 33393063 ± 6% proc-vmstat.numa_other 251563 ± 29% +420.2% 1308599 ± 10% proc-vmstat.numa_pages_migrated 635.00 ± 35% +596.9% 4425 ± 8% proc-vmstat.pageoutrun 1.127e+08 ± 2% -55.0% 50732767 proc-vmstat.pgactivate 9619324 ± 4% -37.5% 6013199 ± 7% proc-vmstat.pgalloc_dma32 2.354e+08 ± 2% -20.7% 1.868e+08 ± 2% proc-vmstat.pgalloc_normal 1.829e+08 ± 2% -97.1% 5298999 ± 4% proc-vmstat.pgdeactivate 7.145e+08 ± 3% +6.8% 7.627e+08 ± 2% proc-vmstat.pgfault 2.353e+08 ± 2% -22.4% 1.827e+08 ± 3% proc-vmstat.pgfree 2.427e+08 ± 2% -22.0% 1.892e+08 ± 2% proc-vmstat.pgmajfault 427089 ± 38% +278.2% 1615323 ± 11% proc-vmstat.pgmigrate_success 9.71e+08 ± 2% -22.0% 7.569e+08 ± 2% proc-vmstat.pgpgin 8.867e+08 ± 2% +19.4% 1.059e+09 ± 2% proc-vmstat.pgpgout 1.829e+08 ± 2% -97.1% 5298820 ± 4% proc-vmstat.pgrefill 5485949 ± 3% +574.1% 36983260 ± 2% proc-vmstat.pgrotated 1.984e+08 ± 12% -78.9% 41928600 ± 14% proc-vmstat.pgscan_direct 3.395e+08 ± 3% +11.0% 3.768e+08 proc-vmstat.pgscan_kswapd 89080449 ± 12% -87.6% 11029910 ± 8% proc-vmstat.pgsteal_direct 1.439e+08 ± 3% +17.0% 1.684e+08 ± 2% proc-vmstat.pgsteal_kswapd 37738291 ± 5% -100.0% 0.00 proc-vmstat.workingset_activate 25988 ± 2% -22.6% 20107 ± 2% proc-vmstat.workingset_nodes 69506838 ± 2% -100.0% 0.00 proc-vmstat.workingset_refault 21650634 ± 5% -100.0% 1.50 ± 33% proc-vmstat.workingset_restore 87.50 ±168% -100.0% 0.00 interrupts.114:PCI-MSI.31981647-edge.i40e-eth0-TxRx-78 2.75 ±110% +5090.9% 142.75 ±103% interrupts.61:PCI-MSI.31981594-edge.i40e-eth0-TxRx-25 50.25 ±153% -98.5% 0.75 ±110% interrupts.83:PCI-MSI.31981616-edge.i40e-eth0-TxRx-47 0.25 ±173% +28300.0% 71.00 ±159% interrupts.86:PCI-MSI.31981619-edge.i40e-eth0-TxRx-50 43979343 ± 10% -35.3% 28436814 ± 6% interrupts.CAL:Function_call_interrupts 309869 ± 12% -48.7% 158885 ± 43% interrupts.CPU0.CAL:Function_call_interrupts 1496 ± 20% -59.7% 603.25 ± 30% interrupts.CPU0.NMI:Non-maskable_interrupts 1496 ± 20% -59.7% 603.25 ± 30% interrupts.CPU0.PMI:Performance_monitoring_interrupts 711.25 ± 39% +120.0% 1564 ± 46% interrupts.CPU0.RES:Rescheduling_interrupts 378389 ± 11% -40.7% 224400 ± 36% interrupts.CPU0.TLB:TLB_shootdowns 564422 ± 6% -51.1% 276198 ± 18% interrupts.CPU11.CAL:Function_call_interrupts 675725 ± 13% -47.0% 358389 ± 15% interrupts.CPU11.TLB:TLB_shootdowns 1834 ± 31% -58.9% 754.25 ± 16% interrupts.CPU12.NMI:Non-maskable_interrupts 1834 ± 31% -58.9% 754.25 ± 16% interrupts.CPU12.PMI:Performance_monitoring_interrupts 483343 ± 18% -43.3% 273844 ± 35% interrupts.CPU13.CAL:Function_call_interrupts 4095 ± 34% -72.2% 1137 ± 55% interrupts.CPU13.NMI:Non-maskable_interrupts 4095 ± 34% -72.2% 1137 ± 55% interrupts.CPU13.PMI:Performance_monitoring_interrupts 662959 ± 25% -52.6% 313982 ± 30% interrupts.CPU13.TLB:TLB_shootdowns 560563 ± 24% -49.1% 285449 ± 9% interrupts.CPU14.CAL:Function_call_interrupts 693290 ± 34% -47.1% 366618 ± 5% interrupts.CPU14.TLB:TLB_shootdowns 558990 ± 21% -61.4% 215573 ± 24% interrupts.CPU15.CAL:Function_call_interrupts 2204 ± 31% -67.0% 727.50 ± 35% interrupts.CPU15.NMI:Non-maskable_interrupts 2204 ± 31% -67.0% 727.50 ± 35% interrupts.CPU15.PMI:Performance_monitoring_interrupts 655709 ± 30% -64.3% 233774 ± 18% interrupts.CPU15.TLB:TLB_shootdowns 531963 ± 30% -51.7% 257180 ± 61% interrupts.CPU16.CAL:Function_call_interrupts 3104 ± 49% -66.2% 1050 ± 71% interrupts.CPU16.NMI:Non-maskable_interrupts 3104 ± 49% -66.2% 1050 ± 71% interrupts.CPU16.PMI:Performance_monitoring_interrupts 758.50 ± 52% +58.1% 1199 ± 18% interrupts.CPU16.RES:Rescheduling_interrupts 624333 ± 30% -53.8% 288559 ± 54% interrupts.CPU16.TLB:TLB_shootdowns 414942 ± 20% -51.1% 203078 ± 32% interrupts.CPU18.CAL:Function_call_interrupts 480458 ± 21% -52.4% 228645 ± 25% interrupts.CPU18.TLB:TLB_shootdowns 1109 ± 44% +205.5% 3390 ± 41% interrupts.CPU2.RES:Rescheduling_interrupts 461027 ± 25% -57.0% 198359 ± 30% interrupts.CPU20.CAL:Function_call_interrupts 3798 ± 63% -81.0% 720.75 ± 86% interrupts.CPU20.NMI:Non-maskable_interrupts 3798 ± 63% -81.0% 720.75 ± 86% interrupts.CPU20.PMI:Performance_monitoring_interrupts 564227 ± 37% -62.5% 211819 ± 31% interrupts.CPU20.TLB:TLB_shootdowns 513743 ± 29% -48.3% 265392 ± 32% interrupts.CPU21.CAL:Function_call_interrupts 644566 ± 43% -57.1% 276789 ± 32% interrupts.CPU21.TLB:TLB_shootdowns 503293 ± 25% -63.1% 185627 ± 52% interrupts.CPU22.CAL:Function_call_interrupts 645000 ± 13% -67.5% 209945 ± 56% interrupts.CPU22.TLB:TLB_shootdowns 553939 ± 27% -61.8% 211825 ± 28% interrupts.CPU23.CAL:Function_call_interrupts 687580 ± 25% -67.1% 225909 ± 29% interrupts.CPU23.TLB:TLB_shootdowns 474.00 ± 52% +258.5% 1699 ± 57% interrupts.CPU24.RES:Rescheduling_interrupts 2.25 ±123% +6233.3% 142.50 ±103% interrupts.CPU25.61:PCI-MSI.31981594-edge.i40e-eth0-TxRx-25 4043 ± 63% -71.3% 1160 ± 31% interrupts.CPU25.NMI:Non-maskable_interrupts 4043 ± 63% -71.3% 1160 ± 31% interrupts.CPU25.PMI:Performance_monitoring_interrupts 451692 ± 11% -39.6% 272701 ± 25% interrupts.CPU26.CAL:Function_call_interrupts 815.25 ± 39% +110.4% 1715 ± 29% interrupts.CPU26.RES:Rescheduling_interrupts 584283 ± 11% -41.2% 343585 ± 33% interrupts.CPU26.TLB:TLB_shootdowns 517137 ± 11% -41.0% 305137 ± 22% interrupts.CPU27.CAL:Function_call_interrupts 2414 ± 26% -45.2% 1323 ± 26% interrupts.CPU27.NMI:Non-maskable_interrupts 2414 ± 26% -45.2% 1323 ± 26% interrupts.CPU27.PMI:Performance_monitoring_interrupts 614104 ± 7% -39.7% 370314 ± 19% interrupts.CPU27.TLB:TLB_shootdowns 441205 ± 12% -32.1% 299453 ± 20% interrupts.CPU28.CAL:Function_call_interrupts 2273 ± 24% -39.7% 1369 ± 19% interrupts.CPU28.NMI:Non-maskable_interrupts 2273 ± 24% -39.7% 1369 ± 19% interrupts.CPU28.PMI:Performance_monitoring_interrupts 544289 ± 4% -31.6% 372065 ± 21% interrupts.CPU28.TLB:TLB_shootdowns 520217 ± 28% -31.2% 357653 ± 14% interrupts.CPU29.CAL:Function_call_interrupts 3744 ± 63% -55.3% 1674 ± 27% interrupts.CPU29.NMI:Non-maskable_interrupts 3744 ± 63% -55.3% 1674 ± 27% interrupts.CPU29.PMI:Performance_monitoring_interrupts 814.25 ± 53% +128.8% 1863 ± 39% interrupts.CPU29.RES:Rescheduling_interrupts 613340 ± 29% -31.8% 418352 ± 11% interrupts.CPU29.TLB:TLB_shootdowns 516303 ± 28% -48.8% 264227 ± 29% interrupts.CPU3.CAL:Function_call_interrupts 1951 ± 32% -71.4% 558.25 ± 28% interrupts.CPU3.NMI:Non-maskable_interrupts 1951 ± 32% -71.4% 558.25 ± 28% interrupts.CPU3.PMI:Performance_monitoring_interrupts 649962 ± 17% -54.0% 298822 ± 40% interrupts.CPU3.TLB:TLB_shootdowns 518481 ± 11% -22.2% 403431 ± 14% interrupts.CPU30.CAL:Function_call_interrupts 678.50 ± 47% +154.1% 1724 ± 16% interrupts.CPU30.RES:Rescheduling_interrupts 740472 ± 16% -40.8% 438014 ± 16% interrupts.CPU30.TLB:TLB_shootdowns 497599 ± 13% -26.9% 363979 ± 2% interrupts.CPU31.CAL:Function_call_interrupts 560143 ± 11% -21.8% 438031 ± 10% interrupts.CPU31.TLB:TLB_shootdowns 919.75 ± 37% +68.1% 1546 ± 16% interrupts.CPU32.RES:Rescheduling_interrupts 526266 ± 22% -40.5% 312935 ± 27% interrupts.CPU33.CAL:Function_call_interrupts 2250 ± 15% -38.3% 1387 ± 3% interrupts.CPU33.NMI:Non-maskable_interrupts 2250 ± 15% -38.3% 1387 ± 3% interrupts.CPU33.PMI:Performance_monitoring_interrupts 959.25 ± 31% +197.4% 2852 ± 59% interrupts.CPU33.RES:Rescheduling_interrupts 589843 ± 17% -40.4% 351483 ± 9% interrupts.CPU34.CAL:Function_call_interrupts 724580 ± 19% -43.0% 412731 ± 9% interrupts.CPU34.TLB:TLB_shootdowns 3061 ± 51% -66.5% 1026 ± 54% interrupts.CPU35.NMI:Non-maskable_interrupts 3061 ± 51% -66.5% 1026 ± 54% interrupts.CPU35.PMI:Performance_monitoring_interrupts 527518 ± 26% -41.3% 309832 ± 20% interrupts.CPU36.CAL:Function_call_interrupts 1935 ± 29% -41.1% 1139 ± 25% interrupts.CPU36.NMI:Non-maskable_interrupts 1935 ± 29% -41.1% 1139 ± 25% interrupts.CPU36.PMI:Performance_monitoring_interrupts 605919 ± 27% -33.5% 402973 ± 36% interrupts.CPU36.TLB:TLB_shootdowns 1968 ± 19% -43.1% 1120 ± 33% interrupts.CPU37.NMI:Non-maskable_interrupts 1968 ± 19% -43.1% 1120 ± 33% interrupts.CPU37.PMI:Performance_monitoring_interrupts 552091 ± 17% -37.9% 343088 ± 20% interrupts.CPU38.CAL:Function_call_interrupts 741448 ± 25% -42.9% 423500 ± 21% interrupts.CPU38.TLB:TLB_shootdowns 893.75 ± 22% +66.8% 1490 ± 34% interrupts.CPU39.RES:Rescheduling_interrupts 651520 ± 10% -32.9% 437091 ± 33% interrupts.CPU39.TLB:TLB_shootdowns 443156 ± 27% -74.1% 114892 ± 21% interrupts.CPU4.CAL:Function_call_interrupts 536804 ± 32% -73.9% 139975 ± 23% interrupts.CPU4.TLB:TLB_shootdowns 2376 ± 9% -31.2% 1635 ± 25% interrupts.CPU40.NMI:Non-maskable_interrupts 2376 ± 9% -31.2% 1635 ± 25% interrupts.CPU40.PMI:Performance_monitoring_interrupts 634311 ± 13% -30.5% 441067 ± 19% interrupts.CPU42.TLB:TLB_shootdowns 605811 ± 24% -44.1% 338572 ± 5% interrupts.CPU45.CAL:Function_call_interrupts 824.00 ± 43% +169.8% 2223 ± 39% interrupts.CPU45.RES:Rescheduling_interrupts 789721 ± 28% -45.9% 427130 ± 12% interrupts.CPU45.TLB:TLB_shootdowns 527252 ± 26% -41.4% 309202 ± 18% interrupts.CPU46.CAL:Function_call_interrupts 747063 ± 34% -55.4% 333517 ± 18% interrupts.CPU46.TLB:TLB_shootdowns 50.00 ±154% -99.5% 0.25 ±173% interrupts.CPU47.83:PCI-MSI.31981616-edge.i40e-eth0-TxRx-47 552975 ± 11% -35.1% 358953 ± 23% interrupts.CPU47.CAL:Function_call_interrupts 698382 ± 5% -34.2% 459438 ± 31% interrupts.CPU47.TLB:TLB_shootdowns 668725 ± 25% -71.0% 193871 ± 59% interrupts.CPU48.CAL:Function_call_interrupts 2351 ± 23% -63.0% 869.00 ± 74% interrupts.CPU48.NMI:Non-maskable_interrupts 2351 ± 23% -63.0% 869.00 ± 74% interrupts.CPU48.PMI:Performance_monitoring_interrupts 1036 ± 39% +143.7% 2524 ± 59% interrupts.CPU48.RES:Rescheduling_interrupts 723105 ± 28% -72.2% 200997 ± 57% interrupts.CPU48.TLB:TLB_shootdowns 681246 ± 37% -69.7% 206357 ± 66% interrupts.CPU49.CAL:Function_call_interrupts 734750 ± 36% -70.8% 214575 ± 68% interrupts.CPU49.TLB:TLB_shootdowns 532908 ± 33% -54.9% 240375 ± 53% interrupts.CPU50.CAL:Function_call_interrupts 622804 ± 27% -59.6% 251480 ± 56% interrupts.CPU50.TLB:TLB_shootdowns 2612 ± 56% -64.4% 929.00 ± 37% interrupts.CPU54.RES:Rescheduling_interrupts 1525 ± 19% -45.9% 825.25 ± 38% interrupts.CPU6.NMI:Non-maskable_interrupts 1525 ± 19% -45.9% 825.25 ± 38% interrupts.CPU6.PMI:Performance_monitoring_interrupts 658.25 ± 61% +183.8% 1868 ± 51% interrupts.CPU60.RES:Rescheduling_interrupts 1826 ± 29% -66.0% 620.75 ± 53% interrupts.CPU61.NMI:Non-maskable_interrupts 1826 ± 29% -66.0% 620.75 ± 53% interrupts.CPU61.PMI:Performance_monitoring_interrupts 2182 ± 35% -53.4% 1017 ± 20% interrupts.CPU62.NMI:Non-maskable_interrupts 2182 ± 35% -53.4% 1017 ± 20% interrupts.CPU62.PMI:Performance_monitoring_interrupts 479189 ± 47% -50.5% 237071 ± 37% interrupts.CPU64.TLB:TLB_shootdowns 417282 ± 22% -36.7% 263997 ± 22% interrupts.CPU7.CAL:Function_call_interrupts 826.50 ± 48% +73.7% 1435 ± 27% interrupts.CPU7.RES:Rescheduling_interrupts 533501 ± 13% -45.4% 291545 ± 28% interrupts.CPU7.TLB:TLB_shootdowns 618059 ± 18% -53.0% 290711 ± 27% interrupts.CPU72.CAL:Function_call_interrupts 680178 ± 18% -53.6% 315576 ± 36% interrupts.CPU72.TLB:TLB_shootdowns 2597 ± 16% -43.7% 1463 ± 24% interrupts.CPU73.NMI:Non-maskable_interrupts 2597 ± 16% -43.7% 1463 ± 24% interrupts.CPU73.PMI:Performance_monitoring_interrupts 459.50 ± 41% +431.0% 2440 ± 26% interrupts.CPU73.RES:Rescheduling_interrupts 521.75 ± 55% +281.9% 1992 ± 21% interrupts.CPU74.RES:Rescheduling_interrupts 2656 ± 21% -45.7% 1441 ± 33% interrupts.CPU75.NMI:Non-maskable_interrupts 2656 ± 21% -45.7% 1441 ± 33% interrupts.CPU75.PMI:Performance_monitoring_interrupts 524.75 ± 54% +141.9% 1269 ± 32% interrupts.CPU75.RES:Rescheduling_interrupts 87.00 ±169% -100.0% 0.00 interrupts.CPU78.114:PCI-MSI.31981647-edge.i40e-eth0-TxRx-78 650.00 ± 35% +102.7% 1317 ± 31% interrupts.CPU78.RES:Rescheduling_interrupts 903.25 ± 25% +76.1% 1590 ± 14% interrupts.CPU84.NMI:Non-maskable_interrupts 903.25 ± 25% +76.1% 1590 ± 14% interrupts.CPU84.PMI:Performance_monitoring_interrupts 421.00 ± 38% +147.6% 1042 ± 39% interrupts.CPU85.RES:Rescheduling_interrupts 807.25 ± 29% +56.5% 1263 ± 24% interrupts.CPU89.RES:Rescheduling_interrupts 521370 ± 3% -46.6% 278562 ± 38% interrupts.CPU9.CAL:Function_call_interrupts 613194 ± 20% -46.4% 328688 ± 32% interrupts.CPU9.TLB:TLB_shootdowns 440.00 ± 33% +163.4% 1158 ± 26% interrupts.CPU93.RES:Rescheduling_interrupts 208.75 ± 9% -40.4% 124.50 ± 3% interrupts.IWI:IRQ_work_interrupts 204364 ± 7% -31.0% 140939 ± 2% interrupts.NMI:Non-maskable_interrupts 204364 ± 7% -31.0% 140939 ± 2% interrupts.PMI:Performance_monitoring_interrupts 111435 ± 10% +35.2% 150649 ± 6% interrupts.RES:Rescheduling_interrupts 52728394 ± 10% -38.7% 32297958 ± 4% interrupts.TLB:TLB_shootdowns 22.74 ± 16% -18.0 4.75 ± 6% perf-profile.calltrace.cycles-pp.pagecache_get_page.filemap_fault.ext4_filemap_fault.__do_fault.handle_pte_fault 29.75 ± 13% -16.9 12.83 ± 2% perf-profile.calltrace.cycles-pp.filemap_fault.ext4_filemap_fault.__do_fault.handle_pte_fault.__handle_mm_fault 30.06 ± 13% -16.9 13.17 ± 2% perf-profile.calltrace.cycles-pp.__do_fault.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault 30.04 ± 13% -16.9 13.15 ± 2% perf-profile.calltrace.cycles-pp.ext4_filemap_fault.__do_fault.handle_pte_fault.__handle_mm_fault.handle_mm_fault 17.51 ± 18% -16.4 1.14 ± 28% perf-profile.calltrace.cycles-pp.__alloc_pages_slowpath.__alloc_pages_nodemask.pagecache_get_page.filemap_fault.ext4_filemap_fault 18.25 ± 17% -16.1 2.16 ± 12% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.filemap_fault.ext4_filemap_fault.__do_fault 15.88 ± 19% -15.2 0.66 ± 61% perf-profile.calltrace.cycles-pp.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.pagecache_get_page.filemap_fault 15.86 ± 19% -15.2 0.66 ± 61% perf-profile.calltrace.cycles-pp.shrink_node.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask 15.86 ± 19% -15.2 0.66 ± 61% perf-profile.calltrace.cycles-pp.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.pagecache_get_page 15.78 ± 19% -15.1 0.64 ± 61% perf-profile.calltrace.cycles-pp.shrink_lruvec.shrink_node.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath 37.45 ± 11% -14.3 23.12 ± 2% perf-profile.calltrace.cycles-pp.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault 38.00 ± 11% -14.0 24.01 ± 2% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_page_fault.page_fault 38.59 ± 11% -13.7 24.93 ± 2% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_page_fault.page_fault 39.53 ± 11% -13.0 26.49 ± 2% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 39.80 ± 11% -12.8 26.95 ± 2% perf-profile.calltrace.cycles-pp.page_fault 12.67 ± 19% -12.0 0.63 ± 61% perf-profile.calltrace.cycles-pp.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages.try_to_free_pages 6.30 ± 14% -6.2 0.13 ±173% perf-profile.calltrace.cycles-pp.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages 6.03 ± 27% -6.0 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node 5.53 ± 25% -5.5 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node.do_try_to_free_pages 4.11 ± 12% -2.1 2.05 ± 3% perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.filemap_fault.ext4_filemap_fault.__do_fault 2.75 ± 28% -2.0 0.72 ± 12% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.pagevec_lru_move_fn.deactivate_file_page.invalidate_mapping_pages 2.85 ± 27% -2.0 0.83 ± 10% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.pagevec_lru_move_fn.deactivate_file_page.invalidate_mapping_pages.generic_fadvise 3.32 ± 9% -2.0 1.35 ± 14% perf-profile.calltrace.cycles-pp.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node 3.19 ± 23% -1.8 1.35 ± 7% perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.deactivate_file_page.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64 3.36 ± 22% -1.7 1.70 ± 5% perf-profile.calltrace.cycles-pp.deactivate_file_page.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64 2.33 ± 17% -1.3 0.98 ± 2% perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.filemap_fault 2.39 ± 16% -1.3 1.07 ± 3% perf-profile.calltrace.cycles-pp.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.filemap_fault.ext4_filemap_fault 2.16 ± 8% -0.7 1.50 ± 11% perf-profile.calltrace.cycles-pp.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node 1.90 ± 9% -0.6 1.29 ± 12% perf-profile.calltrace.cycles-pp.rmap_walk_file.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec 1.69 ± 8% -0.4 1.33 ± 4% perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.flush_tlb_mm_range.ptep_clear_flush.page_mkclean_one.rmap_walk_file 1.75 ± 7% -0.3 1.44 ± 3% perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.ptep_clear_flush.page_mkclean_one.rmap_walk_file.page_mkclean 1.49 ± 8% -0.3 1.19 ± 6% perf-profile.calltrace.cycles-pp.smp_call_function_single.on_each_cpu_cond_mask.flush_tlb_mm_range.ptep_clear_flush.page_mkclean_one 1.82 ± 7% -0.3 1.54 ± 4% perf-profile.calltrace.cycles-pp.ptep_clear_flush.page_mkclean_one.rmap_walk_file.page_mkclean.clear_page_dirty_for_io 0.61 ± 8% +0.3 0.89 ± 8% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.pagecache_get_page.filemap_fault.ext4_filemap_fault 2.60 ± 8% +0.4 3.01 ± 3% perf-profile.calltrace.cycles-pp.filemap_map_pages.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault 2.08 ± 6% +0.4 2.51 ± 4% perf-profile.calltrace.cycles-pp.page_mkclean_one.rmap_walk_file.page_mkclean.clear_page_dirty_for_io.mpage_submit_page 0.73 ± 11% +0.4 1.16 ± 16% perf-profile.calltrace.cycles-pp.isolate_lru_pages.shrink_inactive_list.shrink_lruvec.shrink_node.balance_pgdat 0.93 ± 9% +0.5 1.38 ± 4% perf-profile.calltrace.cycles-pp.pagevec_lookup_entries.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64 0.92 ± 9% +0.5 1.38 ± 3% perf-profile.calltrace.cycles-pp.find_get_entries.pagevec_lookup_entries.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64 0.77 ± 7% +0.5 1.24 ± 5% perf-profile.calltrace.cycles-pp.get_io_u 0.00 +0.6 0.61 ± 5% perf-profile.calltrace.cycles-pp.fault_dirty_shared_page.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault 2.46 ± 10% +0.6 3.08 ± 4% perf-profile.calltrace.cycles-pp.page_mkclean.clear_page_dirty_for_io.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map 2.23 ± 6% +0.6 2.85 ± 4% perf-profile.calltrace.cycles-pp.rmap_walk_file.page_mkclean.clear_page_dirty_for_io.mpage_submit_page.mpage_process_page_bufs 0.00 +0.6 0.63 ± 9% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.22 ± 10% +0.7 1.87 ± 4% perf-profile.calltrace.cycles-pp.block_page_mkwrite.ext4_page_mkwrite.do_page_mkwrite.do_wp_page.handle_pte_fault 4.32 ± 6% +0.7 4.98 ± 4% perf-profile.calltrace.cycles-pp.__memcpy_mcsafe.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio 4.41 ± 6% +0.7 5.08 ± 4% perf-profile.calltrace.cycles-pp.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio.ext4_mpage_readpages 3.50 ± 5% +0.7 4.20 ± 3% perf-profile.calltrace.cycles-pp.clear_page_dirty_for_io.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages 0.00 +0.8 0.77 ± 8% perf-profile.calltrace.cycles-pp.__set_page_dirty.mark_buffer_dirty.__block_commit_write.block_page_mkwrite.ext4_page_mkwrite 5.00 ± 6% +0.8 5.78 ± 4% perf-profile.calltrace.cycles-pp.pmem_make_request.generic_make_request.submit_bio.ext4_mpage_readpages.filemap_fault 0.39 ± 57% +0.8 1.20 ± 5% perf-profile.calltrace.cycles-pp.__block_commit_write.block_page_mkwrite.ext4_page_mkwrite.do_page_mkwrite.do_wp_page 0.00 +0.9 0.90 ± 4% perf-profile.calltrace.cycles-pp.__test_set_page_writeback.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map 5.42 ± 6% +0.9 6.33 ± 4% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.ext4_mpage_readpages.filemap_fault.ext4_filemap_fault 5.46 ± 6% +0.9 6.38 ± 4% perf-profile.calltrace.cycles-pp.submit_bio.ext4_mpage_readpages.filemap_fault.ext4_filemap_fault.__do_fault 0.00 +0.9 0.94 ± 23% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.75 ± 9% +1.0 1.78 ± 19% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 1.42 ± 10% +1.0 2.46 ± 3% perf-profile.calltrace.cycles-pp.ext4_page_mkwrite.do_page_mkwrite.do_wp_page.handle_pte_fault.__handle_mm_fault 1.44 ± 10% +1.1 2.50 ± 3% perf-profile.calltrace.cycles-pp.do_page_mkwrite.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault 6.32 ± 5% +1.1 7.40 ± 4% perf-profile.calltrace.cycles-pp.ext4_mpage_readpages.filemap_fault.ext4_filemap_fault.__do_fault.handle_pte_fault 0.42 ± 57% +1.1 1.53 ± 4% perf-profile.calltrace.cycles-pp.mark_buffer_dirty.__block_commit_write.block_page_mkwrite.ext4_page_mkwrite.do_page_mkwrite 0.83 ± 8% +1.1 1.95 ± 19% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 0.65 ± 4% +1.2 1.87 ± 11% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.ext4_finish_bio.ext4_end_bio.pmem_make_request 0.71 ± 5% +1.7 2.40 ± 3% perf-profile.calltrace.cycles-pp.end_page_writeback.ext4_finish_bio.ext4_end_bio.pmem_make_request.generic_make_request 1.92 ± 5% +1.7 3.61 perf-profile.calltrace.cycles-pp.ext4_finish_bio.ext4_end_bio.pmem_make_request.generic_make_request.submit_bio 1.93 ± 5% +1.7 3.63 perf-profile.calltrace.cycles-pp.ext4_end_bio.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit 1.93 ± 8% +1.8 3.68 ± 3% perf-profile.calltrace.cycles-pp.do_wp_page.handle_pte_fault.__handle_mm_fault.handle_mm_fault.do_page_fault 8.26 ± 9% +1.8 10.04 ± 5% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.posix_fadvise 8.26 ± 9% +1.8 10.04 ± 5% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise 8.24 ± 9% +1.8 10.02 ± 5% perf-profile.calltrace.cycles-pp.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise 8.24 ± 9% +1.8 10.02 ± 5% perf-profile.calltrace.cycles-pp.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe.posix_fadvise 8.24 ± 9% +1.8 10.02 ± 5% perf-profile.calltrace.cycles-pp.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.28 ± 9% +1.8 10.07 ± 5% perf-profile.calltrace.cycles-pp.posix_fadvise 3.07 ± 9% +2.1 5.21 ± 8% perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__filemap_fdatawrite_range 3.29 ± 9% +2.3 5.56 ± 8% perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__filemap_fdatawrite_range.generic_fadvise 3.30 ± 9% +2.3 5.59 ± 8% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64 3.30 ± 9% +2.3 5.59 ± 8% perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64 3.30 ± 9% +2.3 5.59 ± 8% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64 4.45 ± 11% +2.4 6.80 ± 13% perf-profile.calltrace.cycles-pp.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node.balance_pgdat 3.45 ± 16% +3.3 6.74 ± 3% perf-profile.calltrace.cycles-pp.__memcpy_flushcache.write_pmem.pmem_do_bvec.pmem_make_request.generic_make_request 3.49 ± 16% +3.3 6.81 ± 3% perf-profile.calltrace.cycles-pp.write_pmem.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio 3.51 ± 16% +3.3 6.86 ± 3% perf-profile.calltrace.cycles-pp.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit 7.42 ± 7% +4.7 12.16 ± 2% perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__writeback_single_inode 7.77 ± 6% +5.0 12.77 perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes 7.94 ± 7% +5.0 12.96 perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 7.96 ± 7% +5.0 13.00 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback 7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb 7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork 7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread 7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work.worker_thread 7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work 7.79 ± 6% +5.1 12.85 perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn 5.48 ± 12% +5.1 10.57 ± 2% perf-profile.calltrace.cycles-pp.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit.ext4_bio_write_page 5.49 ± 12% +5.1 10.58 ± 2% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.ext4_io_submit.ext4_bio_write_page.mpage_submit_page 5.50 ± 12% +5.1 10.58 ± 2% perf-profile.calltrace.cycles-pp.submit_bio.ext4_io_submit.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs 5.50 ± 12% +5.1 10.58 ± 2% perf-profile.calltrace.cycles-pp.ext4_io_submit.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map 6.60 ± 11% +6.0 12.63 ± 2% perf-profile.calltrace.cycles-pp.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages 15.73 ± 8% +6.1 21.88 ± 5% perf-profile.calltrace.cycles-pp.ret_from_fork 15.73 ± 8% +6.1 21.88 ± 5% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 10.14 ± 7% +6.8 16.91 ± 2% perf-profile.calltrace.cycles-pp.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages 22.84 ± 16% -18.1 4.78 ± 6% perf-profile.children.cycles-pp.pagecache_get_page 29.77 ± 13% -16.9 12.85 ± 2% perf-profile.children.cycles-pp.filemap_fault 30.04 ± 13% -16.9 13.15 ± 2% perf-profile.children.cycles-pp.ext4_filemap_fault 30.07 ± 13% -16.9 13.18 ± 2% perf-profile.children.cycles-pp.__do_fault 18.08 ± 18% -16.8 1.31 ± 26% perf-profile.children.cycles-pp.__alloc_pages_slowpath 18.83 ± 17% -16.5 2.35 ± 14% perf-profile.children.cycles-pp.__alloc_pages_nodemask 16.42 ± 19% -15.5 0.89 ± 37% perf-profile.children.cycles-pp.try_to_free_pages 16.40 ± 19% -15.5 0.89 ± 37% perf-profile.children.cycles-pp.do_try_to_free_pages 24.16 ± 14% -14.4 9.73 ± 11% perf-profile.children.cycles-pp.shrink_node 24.07 ± 14% -14.4 9.67 ± 11% perf-profile.children.cycles-pp.shrink_lruvec 37.48 ± 11% -14.3 23.17 ± 2% perf-profile.children.cycles-pp.handle_pte_fault 38.05 ± 11% -14.0 24.07 ± 2% perf-profile.children.cycles-pp.__handle_mm_fault 38.64 ± 11% -13.6 25.00 ± 2% perf-profile.children.cycles-pp.handle_mm_fault 39.58 ± 11% -13.0 26.58 ± 2% perf-profile.children.cycles-pp.do_page_fault 39.84 ± 11% -12.8 27.01 ± 2% perf-profile.children.cycles-pp.page_fault 15.30 ± 22% -12.8 2.48 ± 12% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 19.34 ± 14% -9.8 9.52 ± 10% perf-profile.children.cycles-pp.shrink_inactive_list 8.85 ± 23% -8.1 0.72 ± 23% perf-profile.children.cycles-pp._raw_spin_lock_irq 4.72 ± 13% -4.6 0.11 ± 68% perf-profile.children.cycles-pp.shrink_active_list 6.92 ± 18% -4.0 2.88 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 11.04 ± 10% -3.8 7.26 ± 11% perf-profile.children.cycles-pp.shrink_page_list 5.65 ± 20% -3.0 2.67 ± 3% perf-profile.children.cycles-pp.pagevec_lru_move_fn 4.12 ± 12% -2.1 2.06 ± 3% perf-profile.children.cycles-pp.add_to_page_cache_lru 3.42 ± 10% -1.9 1.51 ± 13% perf-profile.children.cycles-pp.__remove_mapping 3.37 ± 22% -1.7 1.70 ± 5% perf-profile.children.cycles-pp.deactivate_file_page 3.11 ± 15% -1.5 1.57 ± 5% perf-profile.children.cycles-pp.on_each_cpu_cond_mask 2.41 ± 16% -1.3 1.08 ± 2% perf-profile.children.cycles-pp.__lru_cache_add 2.96 ± 10% -1.3 1.66 ± 9% perf-profile.children.cycles-pp.page_referenced 1.43 ± 25% -1.2 0.23 ± 11% perf-profile.children.cycles-pp.try_to_unmap_flush 1.43 ± 25% -1.2 0.23 ± 11% perf-profile.children.cycles-pp.arch_tlbbatch_flush 1.29 ± 24% -1.0 0.31 ± 15% perf-profile.children.cycles-pp.smp_call_function_many_cond 1.64 ± 10% -0.7 0.92 ± 8% perf-profile.children.cycles-pp.page_referenced_one 0.80 ± 9% -0.7 0.14 ± 23% perf-profile.children.cycles-pp.workingset_refault 0.80 ± 13% -0.6 0.18 ± 15% perf-profile.children.cycles-pp.wake_all_kswapds 0.78 ± 14% -0.6 0.17 ± 16% perf-profile.children.cycles-pp.wakeup_kswapd 1.80 ± 10% -0.6 1.23 ± 5% perf-profile.children.cycles-pp.smp_call_function_single 1.73 ± 11% -0.5 1.22 ± 15% perf-profile.children.cycles-pp.isolate_lru_pages 1.19 ± 20% -0.5 0.68 ± 29% perf-profile.children.cycles-pp.create_empty_buffers 1.21 ± 19% -0.5 0.71 ± 28% perf-profile.children.cycles-pp.create_page_buffers 1.74 ± 10% -0.5 1.25 ± 5% perf-profile.children.cycles-pp.page_vma_mapped_walk 0.85 ± 26% -0.4 0.44 ± 46% perf-profile.children.cycles-pp.alloc_page_buffers 1.61 ± 10% -0.4 1.22 ± 9% perf-profile.children.cycles-pp.__list_del_entry_valid 0.80 ± 28% -0.4 0.40 ± 50% perf-profile.children.cycles-pp.alloc_buffer_head 0.64 ± 32% -0.4 0.27 ± 71% perf-profile.children.cycles-pp.__slab_alloc 1.54 ± 17% -0.4 1.17 ± 16% perf-profile.children.cycles-pp.__block_write_begin_int 0.86 ± 26% -0.4 0.49 ± 40% perf-profile.children.cycles-pp.kmem_cache_alloc 0.63 ± 32% -0.4 0.27 ± 73% perf-profile.children.cycles-pp.___slab_alloc 0.81 ± 10% -0.4 0.46 ± 16% perf-profile.children.cycles-pp.move_pages_to_lru 0.48 ± 43% -0.3 0.15 ±138% perf-profile.children.cycles-pp.new_slab 0.35 ± 10% -0.3 0.09 ± 15% perf-profile.children.cycles-pp.advance_inactive_age 0.40 ± 8% -0.2 0.19 ± 9% perf-profile.children.cycles-pp.workingset_eviction 0.49 ± 8% -0.2 0.30 ± 13% perf-profile.children.cycles-pp.try_to_free_buffers 0.35 ± 10% -0.2 0.19 ± 21% perf-profile.children.cycles-pp.jbd2_journal_try_to_free_buffers 0.33 ± 8% -0.2 0.18 ± 22% perf-profile.children.cycles-pp.jbd2_journal_grab_journal_head 0.37 ± 9% -0.1 0.26 ± 4% perf-profile.children.cycles-pp.page_add_file_rmap 0.32 ± 13% -0.1 0.20 ± 18% perf-profile.children.cycles-pp.page_evictable 0.28 ± 12% -0.1 0.19 ± 19% perf-profile.children.cycles-pp.page_remove_rmap 0.14 ± 6% -0.1 0.07 ± 12% perf-profile.children.cycles-pp.vma_interval_tree_iter_first 0.09 ± 11% -0.1 0.03 ±100% perf-profile.children.cycles-pp.get_partial_node 0.17 ± 10% -0.1 0.11 ± 15% perf-profile.children.cycles-pp.drop_buffers 0.19 ± 13% -0.1 0.12 ± 12% perf-profile.children.cycles-pp.free_buffer_head 0.24 ± 10% -0.0 0.19 ± 3% perf-profile.children.cycles-pp.ext4_da_get_block_prep 0.15 ± 10% -0.0 0.10 ± 12% perf-profile.children.cycles-pp.mem_cgroup_page_lruvec 0.12 ± 14% -0.0 0.08 ± 15% perf-profile.children.cycles-pp.__slab_free 0.10 ± 11% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.ptep_clear_flush_young 0.06 ± 9% +0.0 0.07 ± 5% perf-profile.children.cycles-pp.__indirect_thunk_start 0.06 ± 11% +0.0 0.09 ± 12% perf-profile.children.cycles-pp.generic_update_time 0.08 ± 11% +0.0 0.11 ± 8% perf-profile.children.cycles-pp.fput_many 0.14 ± 7% +0.0 0.17 ± 7% perf-profile.children.cycles-pp.mempool_alloc 0.13 ± 6% +0.0 0.17 ± 6% perf-profile.children.cycles-pp._find_next_bit 0.09 ± 24% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.try_to_wake_up 0.08 ± 5% +0.0 0.11 ± 14% perf-profile.children.cycles-pp.task_tick_fair 0.07 ± 19% +0.0 0.10 ± 10% perf-profile.children.cycles-pp.io_serial_in 0.04 ± 58% +0.0 0.08 ± 10% perf-profile.children.cycles-pp.ext4_dirty_inode 0.06 ± 9% +0.0 0.09 ± 8% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.09 ± 19% +0.0 0.13 ± 18% perf-profile.children.cycles-pp.wait_for_xmitr 0.09 ± 19% +0.0 0.13 ± 18% perf-profile.children.cycles-pp.serial8250_console_putchar 0.06 ± 7% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.__unlock_page_memcg 0.07 ± 6% +0.0 0.11 ± 6% perf-profile.children.cycles-pp.pmd_devmap_trans_unstable 0.09 ± 5% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.bio_add_page 0.11 ± 7% +0.0 0.15 ± 10% perf-profile.children.cycles-pp.PageHuge 0.10 ± 21% +0.0 0.14 ± 16% perf-profile.children.cycles-pp.serial8250_console_write 0.07 ± 7% +0.0 0.11 ± 13% perf-profile.children.cycles-pp.schedule 0.07 ± 5% +0.0 0.12 ± 11% perf-profile.children.cycles-pp.ext4_convert_inline_data 0.13 ± 14% +0.0 0.18 ± 10% perf-profile.children.cycles-pp.blk_throtl_bio 0.06 ± 20% +0.0 0.11 ± 7% perf-profile.children.cycles-pp.native_write_msr 0.17 ± 6% +0.0 0.22 ± 8% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.10 ± 12% +0.0 0.15 ± 8% perf-profile.children.cycles-pp.read_tsc 0.01 ±173% +0.1 0.06 ± 13% perf-profile.children.cycles-pp.update_load_avg 0.04 ± 57% +0.1 0.09 ± 13% perf-profile.children.cycles-pp.__next_timer_interrupt 0.11 ± 17% +0.1 0.16 ± 6% perf-profile.children.cycles-pp.irq_work_run_list 0.01 ±173% +0.1 0.07 ± 23% perf-profile.children.cycles-pp.pick_next_task_fair 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.__bio_try_merge_page 0.10 ± 17% +0.1 0.15 ± 7% perf-profile.children.cycles-pp.console_unlock 0.06 ± 9% +0.1 0.11 ± 11% perf-profile.children.cycles-pp.page_mapped 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.sched_clock_cpu 0.22 ± 9% +0.1 0.28 perf-profile.children.cycles-pp.bio_alloc_bioset 0.43 ± 9% +0.1 0.49 ± 3% perf-profile.children.cycles-pp.__mod_memcg_state 0.10 ± 7% +0.1 0.16 ± 5% perf-profile.children.cycles-pp.vmacache_find 0.08 ± 16% +0.1 0.14 ± 6% perf-profile.children.cycles-pp.lru_deactivate_file_fn 0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp._raw_spin_trylock 0.01 ±173% +0.1 0.07 ± 10% perf-profile.children.cycles-pp.mem_cgroup_wb_domain 0.12 ± 13% +0.1 0.18 ± 6% perf-profile.children.cycles-pp.xas_clear_mark 0.08 ± 10% +0.1 0.14 ± 5% perf-profile.children.cycles-pp.down_read_trylock 0.03 ±100% +0.1 0.09 ± 9% perf-profile.children.cycles-pp.lapic_next_deadline 0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.note_gp_changes 0.06 ± 13% +0.1 0.12 ± 11% perf-profile.children.cycles-pp.__sb_start_write 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.ext4_inode_journal_mode 0.32 ± 11% +0.1 0.39 ± 3% perf-profile.children.cycles-pp.find_get_entry 0.00 +0.1 0.07 ± 17% perf-profile.children.cycles-pp.migrate_pages 0.04 ± 57% +0.1 0.11 ± 15% perf-profile.children.cycles-pp.load_balance 0.00 +0.1 0.07 ± 6% perf-profile.children.cycles-pp.current_time 0.00 +0.1 0.07 ± 6% perf-profile.children.cycles-pp.run_timer_softirq 0.08 +0.1 0.15 ± 10% perf-profile.children.cycles-pp.balance_dirty_pages 0.07 ± 13% +0.1 0.14 ± 6% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.04 ± 58% +0.1 0.11 ± 11% perf-profile.children.cycles-pp.inc_zone_page_state 0.01 ±173% +0.1 0.09 ± 10% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 0.11 ± 3% +0.1 0.19 ± 4% perf-profile.children.cycles-pp.find_vma 0.01 ±173% +0.1 0.09 ± 9% perf-profile.children.cycles-pp.xas_set_mark 0.23 ± 7% +0.1 0.31 ± 5% perf-profile.children.cycles-pp.xas_find 0.07 ± 12% +0.1 0.15 ± 7% perf-profile.children.cycles-pp.__fprop_inc_percpu_max 0.21 ± 7% +0.1 0.29 ± 5% perf-profile.children.cycles-pp.__might_sleep 0.29 ± 11% +0.1 0.36 ± 9% perf-profile.children.cycles-pp._cond_resched 0.00 +0.1 0.08 ± 19% perf-profile.children.cycles-pp.find_busiest_group 0.18 ± 6% +0.1 0.26 ± 4% perf-profile.children.cycles-pp.xas_start 0.13 ± 6% +0.1 0.22 ± 14% perf-profile.children.cycles-pp.scheduler_tick 0.12 ± 15% +0.1 0.20 ± 7% perf-profile.children.cycles-pp.__mark_inode_dirty 0.13 ± 12% +0.1 0.21 ± 6% perf-profile.children.cycles-pp.file_update_time 0.00 +0.1 0.09 ± 18% perf-profile.children.cycles-pp.migrate_misplaced_page 0.04 ± 57% +0.1 0.13 ± 19% perf-profile.children.cycles-pp.rebalance_domains 0.08 ± 5% +0.1 0.18 ± 8% perf-profile.children.cycles-pp.invalidate_inode_page 0.04 ± 58% +0.1 0.14 ± 3% perf-profile.children.cycles-pp.finish_mkwrite_fault 0.35 ± 8% +0.1 0.45 ± 4% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.12 ± 16% +0.1 0.23 ± 22% perf-profile.children.cycles-pp.tick_nohz_next_event 0.26 ± 7% +0.1 0.38 ± 8% perf-profile.children.cycles-pp.flush_tlb_func_common 0.12 ± 6% +0.1 0.24 ± 3% perf-profile.children.cycles-pp.set_page_dirty 0.35 ± 6% +0.1 0.47 ± 5% perf-profile.children.cycles-pp.generic_make_request_checks 0.20 ± 7% +0.1 0.31 ± 6% perf-profile.children.cycles-pp.native_flush_tlb_one_user 0.14 ± 13% +0.1 0.26 ± 8% perf-profile.children.cycles-pp.dec_zone_page_state 0.45 ± 6% +0.1 0.57 ± 7% perf-profile.children.cycles-pp.___might_sleep 0.14 ± 17% +0.1 0.27 ± 22% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 0.21 ± 2% +0.1 0.33 ± 9% perf-profile.children.cycles-pp.___perf_sw_event 0.14 ± 10% +0.1 0.27 ± 5% perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited 0.11 ± 7% +0.1 0.24 ± 6% perf-profile.children.cycles-pp.__xa_set_mark 0.14 ± 23% +0.1 0.27 ± 11% perf-profile.children.cycles-pp.rcu_core 0.11 ± 6% +0.1 0.25 ± 6% perf-profile.children.cycles-pp.__xa_clear_mark 0.03 ±100% +0.1 0.17 ± 42% perf-profile.children.cycles-pp.tick_irq_enter 0.04 ± 58% +0.2 0.20 ± 35% perf-profile.children.cycles-pp.irq_enter 0.00 +0.2 0.17 ± 6% perf-profile.children.cycles-pp.pagevec_move_tail_fn 0.18 ± 15% +0.2 0.36 ± 5% perf-profile.children.cycles-pp.__pagevec_release 0.31 ± 2% +0.2 0.51 ± 7% perf-profile.children.cycles-pp.__perf_sw_event 0.24 ± 11% +0.2 0.43 ± 15% perf-profile.children.cycles-pp.update_process_times 0.52 ± 7% +0.2 0.72 ± 6% perf-profile.children.cycles-pp.account_page_dirtied 0.24 ± 12% +0.2 0.45 ± 16% perf-profile.children.cycles-pp.tick_sched_handle 0.33 ± 2% +0.2 0.55 ± 7% perf-profile.children.cycles-pp.find_get_pages_range_tag 0.33 ± 2% +0.2 0.55 ± 6% perf-profile.children.cycles-pp.pagevec_lookup_range_tag 0.94 ± 7% +0.2 1.17 ± 2% perf-profile.children.cycles-pp.unlock_page 0.05 ± 59% +0.2 0.29 ± 8% perf-profile.children.cycles-pp.pagevec_move_tail 0.53 ± 9% +0.2 0.76 ± 5% perf-profile.children.cycles-pp.page_mapping 0.29 ± 7% +0.3 0.56 ± 18% perf-profile.children.cycles-pp.tick_sched_timer 0.22 ± 13% +0.3 0.49 ± 22% perf-profile.children.cycles-pp.clockevents_program_event 0.05 ± 59% +0.3 0.33 ± 5% perf-profile.children.cycles-pp.rotate_reclaimable_page 0.29 ± 12% +0.3 0.57 ± 14% perf-profile.children.cycles-pp.__softirqentry_text_start 0.42 ± 13% +0.3 0.72 ± 4% perf-profile.children.cycles-pp.release_pages 2.77 ± 5% +0.3 3.08 ± 3% perf-profile.children.cycles-pp.page_mkclean 0.43 ± 6% +0.3 0.78 ± 4% perf-profile.children.cycles-pp.percpu_counter_add_batch 0.39 ± 6% +0.3 0.74 ± 15% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.39 ± 8% +0.4 0.75 ± 18% perf-profile.children.cycles-pp.irq_exit 0.28 ± 15% +0.4 0.64 ± 8% perf-profile.children.cycles-pp.menu_select 0.75 ± 7% +0.4 1.16 ± 5% perf-profile.children.cycles-pp.__set_page_dirty 2.63 ± 8% +0.4 3.04 ± 3% perf-profile.children.cycles-pp.filemap_map_pages 0.72 ± 8% +0.4 1.14 ± 3% perf-profile.children.cycles-pp.xas_load 0.42 ± 5% +0.4 0.86 ± 5% perf-profile.children.cycles-pp.fault_dirty_shared_page 0.39 ± 7% +0.4 0.84 ± 15% perf-profile.children.cycles-pp.ktime_get 0.93 ± 9% +0.5 1.38 ± 4% perf-profile.children.cycles-pp.find_get_entries 0.61 ± 7% +0.5 1.07 ± 4% perf-profile.children.cycles-pp.sync_regs 0.93 ± 9% +0.5 1.38 ± 3% perf-profile.children.cycles-pp.pagevec_lookup_entries 0.78 ± 7% +0.5 1.26 ± 5% perf-profile.children.cycles-pp.get_io_u 0.46 ± 7% +0.5 0.97 ± 4% perf-profile.children.cycles-pp.__set_page_dirty_buffers 0.96 ± 8% +0.6 1.54 ± 3% perf-profile.children.cycles-pp.mark_buffer_dirty 2.93 ± 12% +0.6 3.55 ± 4% perf-profile.children.cycles-pp.block_page_mkwrite 0.70 ± 4% +0.6 1.33 ± 2% perf-profile.children.cycles-pp.__test_set_page_writeback 4.37 ± 6% +0.7 5.02 ± 4% perf-profile.children.cycles-pp.__memcpy_mcsafe 1.09 ± 9% +0.7 1.76 ± 5% perf-profile.children.cycles-pp.native_irq_return_iret 0.69 ± 8% +0.7 1.39 ± 15% perf-profile.children.cycles-pp.hrtimer_interrupt 1.11 ± 8% +0.7 1.81 ± 3% perf-profile.children.cycles-pp.__block_commit_write 3.53 ± 5% +0.7 4.27 ± 3% perf-profile.children.cycles-pp.clear_page_dirty_for_io 3.40 ± 12% +1.0 4.42 ± 3% perf-profile.children.cycles-pp.ext4_page_mkwrite 3.42 ± 12% +1.0 4.46 ± 3% perf-profile.children.cycles-pp.do_page_mkwrite 1.00 ± 6% +1.1 2.06 ± 3% perf-profile.children.cycles-pp.test_clear_page_writeback 6.32 ± 5% +1.1 7.41 ± 4% perf-profile.children.cycles-pp.ext4_mpage_readpages 1.15 ± 8% +1.2 2.36 ± 14% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 1.26 ± 7% +1.3 2.59 ± 13% perf-profile.children.cycles-pp.apic_timer_interrupt 1.07 ± 7% +1.3 2.42 ± 3% perf-profile.children.cycles-pp.end_page_writeback 1.94 ± 5% +1.7 3.66 perf-profile.children.cycles-pp.ext4_end_bio 1.94 ± 5% +1.7 3.66 perf-profile.children.cycles-pp.ext4_finish_bio 8.51 ± 9% +1.7 10.24 ± 5% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 8.51 ± 9% +1.7 10.24 ± 5% perf-profile.children.cycles-pp.do_syscall_64 1.94 ± 8% +1.8 3.69 ± 4% perf-profile.children.cycles-pp.do_wp_page 8.24 ± 9% +1.8 10.02 ± 5% perf-profile.children.cycles-pp.__x64_sys_fadvise64 8.24 ± 9% +1.8 10.02 ± 5% perf-profile.children.cycles-pp.ksys_fadvise64_64 8.24 ± 9% +1.8 10.02 ± 5% perf-profile.children.cycles-pp.generic_fadvise 8.28 ± 9% +1.8 10.07 ± 5% perf-profile.children.cycles-pp.posix_fadvise 3.30 ± 9% +2.3 5.59 ± 8% perf-profile.children.cycles-pp.__filemap_fdatawrite_range 3.47 ± 16% +3.3 6.81 ± 3% perf-profile.children.cycles-pp.__memcpy_flushcache 3.50 ± 16% +3.4 6.88 ± 3% perf-profile.children.cycles-pp.write_pmem 7.95 ± 7% +4.1 12.00 perf-profile.children.cycles-pp.pmem_do_bvec 7.94 ± 7% +5.0 12.96 perf-profile.children.cycles-pp.process_one_work 7.96 ± 7% +5.0 13.00 perf-profile.children.cycles-pp.worker_thread 7.79 ± 6% +5.1 12.85 perf-profile.children.cycles-pp.wb_workfn 7.79 ± 6% +5.1 12.85 perf-profile.children.cycles-pp.wb_writeback 7.79 ± 6% +5.1 12.85 perf-profile.children.cycles-pp.__writeback_inodes_wb 7.79 ± 6% +5.1 12.85 perf-profile.children.cycles-pp.writeback_sb_inodes 7.79 ± 6% +5.1 12.85 perf-profile.children.cycles-pp.__writeback_single_inode 5.52 ± 12% +5.2 10.67 ± 2% perf-profile.children.cycles-pp.ext4_io_submit 10.52 ± 7% +5.9 16.44 perf-profile.children.cycles-pp.pmem_make_request 6.60 ± 11% +6.0 12.63 ± 2% perf-profile.children.cycles-pp.ext4_bio_write_page 10.95 ± 7% +6.1 17.01 perf-profile.children.cycles-pp.generic_make_request 10.98 ± 6% +6.1 17.05 perf-profile.children.cycles-pp.submit_bio 15.73 ± 8% +6.1 21.88 ± 5% perf-profile.children.cycles-pp.ret_from_fork 15.73 ± 8% +6.1 21.88 ± 5% perf-profile.children.cycles-pp.kthread 10.14 ± 7% +6.8 16.92 ± 2% perf-profile.children.cycles-pp.mpage_submit_page 10.50 ± 7% +6.9 17.36 ± 2% perf-profile.children.cycles-pp.mpage_process_page_bufs 11.06 ± 7% +7.3 18.36 ± 2% perf-profile.children.cycles-pp.mpage_prepare_extent_to_map 11.09 ± 7% +7.4 18.45 ± 2% perf-profile.children.cycles-pp.do_writepages 11.09 ± 7% +7.4 18.45 ± 2% perf-profile.children.cycles-pp.ext4_writepages 15.29 ± 22% -12.8 2.47 ± 12% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 1.16 ± 26% -1.0 0.20 ± 21% perf-profile.self.cycles-pp.smp_call_function_many_cond 0.76 ± 13% -0.6 0.17 ± 17% perf-profile.self.cycles-pp.wakeup_kswapd 1.66 ± 10% -0.6 1.09 ± 5% perf-profile.self.cycles-pp.smp_call_function_single 1.60 ± 10% -0.4 1.21 ± 9% perf-profile.self.cycles-pp.__list_del_entry_valid 0.42 ± 8% -0.3 0.10 ± 23% perf-profile.self.cycles-pp.workingset_refault 0.34 ± 10% -0.2 0.09 ± 15% perf-profile.self.cycles-pp.advance_inactive_age 0.99 ± 9% -0.2 0.79 ± 7% perf-profile.self.cycles-pp.page_vma_mapped_walk 0.74 ± 10% -0.2 0.55 ± 5% perf-profile.self.cycles-pp.get_page_from_freelist 0.43 ± 9% -0.2 0.25 ± 16% perf-profile.self.cycles-pp.move_pages_to_lru 0.62 ± 9% -0.2 0.47 ± 7% perf-profile.self.cycles-pp.__mod_lruvec_state 0.32 ± 9% -0.1 0.17 ± 25% perf-profile.self.cycles-pp.jbd2_journal_grab_journal_head 0.37 ± 9% -0.1 0.24 ± 18% perf-profile.self.cycles-pp.__remove_mapping 0.37 ± 10% -0.1 0.26 ± 10% perf-profile.self.cycles-pp.page_referenced_one 0.23 ± 12% -0.1 0.13 ± 17% perf-profile.self.cycles-pp.page_evictable 0.17 ± 6% -0.1 0.10 ± 11% perf-profile.self.cycles-pp.workingset_eviction 0.16 ± 9% -0.1 0.10 ± 11% perf-profile.self.cycles-pp.page_add_file_rmap 0.14 ± 15% -0.1 0.08 ± 6% perf-profile.self.cycles-pp.pagecache_get_page 0.17 ± 12% -0.1 0.11 ± 15% perf-profile.self.cycles-pp.drop_buffers 0.12 ± 5% -0.1 0.06 ± 14% perf-profile.self.cycles-pp.vma_interval_tree_iter_first 0.26 ± 11% -0.1 0.20 ± 9% perf-profile.self.cycles-pp.isolate_lru_pages 0.12 ± 11% -0.1 0.06 ± 13% perf-profile.self.cycles-pp.page_referenced 0.14 ± 8% -0.1 0.09 ± 13% perf-profile.self.cycles-pp.mem_cgroup_page_lruvec 0.12 ± 11% -0.1 0.07 ± 12% perf-profile.self.cycles-pp.page_remove_rmap 0.12 ± 14% -0.0 0.08 ± 15% perf-profile.self.cycles-pp.__slab_free 0.10 ± 11% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.ptep_clear_flush_young 0.08 ± 10% -0.0 0.06 perf-profile.self.cycles-pp.ext4_da_get_block_prep 0.10 ± 7% +0.0 0.12 ± 3% perf-profile.self.cycles-pp.rcu_all_qs 0.06 ± 6% +0.0 0.08 ± 5% perf-profile.self.cycles-pp.__lru_cache_add 0.07 ± 17% +0.0 0.10 ± 8% perf-profile.self.cycles-pp.xas_find 0.07 ± 10% +0.0 0.10 ± 8% perf-profile.self.cycles-pp.fput_many 0.12 ± 8% +0.0 0.16 ± 7% perf-profile.self.cycles-pp._find_next_bit 0.15 ± 8% +0.0 0.18 ± 7% perf-profile.self.cycles-pp.alloc_set_pte 0.08 ± 10% +0.0 0.12 ± 7% perf-profile.self.cycles-pp.generic_make_request_checks 0.08 ± 14% +0.0 0.11 ± 6% perf-profile.self.cycles-pp.pmem_do_bvec 0.06 ± 14% +0.0 0.09 ± 4% perf-profile.self.cycles-pp.lru_deactivate_file_fn 0.07 ± 19% +0.0 0.10 ± 10% perf-profile.self.cycles-pp.io_serial_in 0.08 ± 19% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.free_unref_page_list 0.09 ± 11% +0.0 0.13 ± 11% perf-profile.self.cycles-pp.PageHuge 0.06 ± 6% +0.0 0.10 ± 7% perf-profile.self.cycles-pp.page_mkclean_one 0.07 ± 6% +0.0 0.11 ± 8% perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.41 ± 6% +0.0 0.45 ± 8% perf-profile.self.cycles-pp.filemap_fault 0.12 ± 13% +0.0 0.15 ± 7% perf-profile.self.cycles-pp.xas_store 0.10 ± 4% +0.0 0.14 ± 10% perf-profile.self.cycles-pp.ptep_clear_flush 0.06 ± 7% +0.0 0.10 ± 11% perf-profile.self.cycles-pp.pmd_devmap_trans_unstable 0.04 ± 57% +0.0 0.08 ± 5% perf-profile.self.cycles-pp.__unlock_page_memcg 0.10 ± 12% +0.0 0.14 ± 9% perf-profile.self.cycles-pp.read_tsc 0.08 ± 14% +0.0 0.12 ± 4% perf-profile.self.cycles-pp.down_read_trylock 0.07 +0.0 0.12 ± 9% perf-profile.self.cycles-pp.ext4_convert_inline_data 0.07 ± 11% +0.0 0.12 ± 5% perf-profile.self.cycles-pp.flush_tlb_mm_range 0.06 ± 6% +0.0 0.11 ± 6% perf-profile.self.cycles-pp.set_page_dirty 0.07 ± 17% +0.0 0.11 ± 9% perf-profile.self.cycles-pp.pagevec_lru_move_fn 0.06 ± 13% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.__mark_inode_dirty 0.06 ± 20% +0.0 0.11 ± 7% perf-profile.self.cycles-pp.native_write_msr 0.42 ± 8% +0.1 0.47 ± 3% perf-profile.self.cycles-pp.__mod_memcg_state 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.__indirect_thunk_start 0.12 ± 13% +0.1 0.18 ± 14% perf-profile.self.cycles-pp._cond_resched 0.10 ± 9% +0.1 0.15 ± 4% perf-profile.self.cycles-pp.vmacache_find 0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.note_gp_changes 0.11 ± 11% +0.1 0.17 ± 10% perf-profile.self.cycles-pp.xas_clear_mark 0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.write_pmem 0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp._raw_spin_trylock 0.01 ±173% +0.1 0.07 ± 10% perf-profile.self.cycles-pp.mem_cgroup_wb_domain 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.pagevec_move_tail_fn 0.00 +0.1 0.06 ± 13% perf-profile.self.cycles-pp.__sb_start_write 0.00 +0.1 0.06 ± 17% perf-profile.self.cycles-pp.balance_dirty_pages 0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp.ext4_inode_journal_mode 0.18 ± 8% +0.1 0.25 ± 5% perf-profile.self.cycles-pp.__might_sleep 0.04 ± 59% +0.1 0.11 ± 9% perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited 0.13 ± 8% +0.1 0.20 ± 10% perf-profile.self.cycles-pp.pmem_make_request 0.01 ±173% +0.1 0.08 ± 13% perf-profile.self.cycles-pp.xas_set_mark 0.03 ±100% +0.1 0.10 ± 11% perf-profile.self.cycles-pp.inc_zone_page_state 0.03 ±100% +0.1 0.10 ± 11% perf-profile.self.cycles-pp.block_page_mkwrite 0.01 ±173% +0.1 0.09 ± 10% perf-profile.self.cycles-pp.page_mapped 0.00 +0.1 0.07 ± 5% perf-profile.self.cycles-pp.invalidate_inode_page 0.10 ± 10% +0.1 0.17 ± 6% perf-profile.self.cycles-pp.__perf_sw_event 0.10 ± 10% +0.1 0.17 ± 10% perf-profile.self.cycles-pp.page_fault 0.00 +0.1 0.07 ± 22% perf-profile.self.cycles-pp.fault_dirty_shared_page 0.08 ± 8% +0.1 0.16 ± 7% perf-profile.self.cycles-pp.__block_commit_write 0.00 +0.1 0.08 ± 11% perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.09 ± 13% +0.1 0.16 ± 11% perf-profile.self.cycles-pp.mark_buffer_dirty 0.16 ± 5% +0.1 0.24 ± 4% perf-profile.self.cycles-pp.xas_start 0.01 ±173% +0.1 0.09 ± 8% perf-profile.self.cycles-pp.ext4_page_mkwrite 0.35 ± 6% +0.1 0.44 ± 7% perf-profile.self.cycles-pp.mpage_process_page_bufs 0.17 ± 2% +0.1 0.27 ± 11% perf-profile.self.cycles-pp.___perf_sw_event 0.11 ± 9% +0.1 0.22 ± 5% perf-profile.self.cycles-pp.mpage_prepare_extent_to_map 0.12 ± 13% +0.1 0.23 ± 8% perf-profile.self.cycles-pp.dec_zone_page_state 0.44 ± 7% +0.1 0.55 ± 6% perf-profile.self.cycles-pp.___might_sleep 0.20 ± 7% +0.1 0.31 ± 6% perf-profile.self.cycles-pp.native_flush_tlb_one_user 0.28 ± 7% +0.1 0.41 ± 4% perf-profile.self.cycles-pp.clear_page_dirty_for_io 0.23 ± 6% +0.1 0.36 ± 4% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.22 ± 4% +0.1 0.36 ± 8% perf-profile.self.cycles-pp.do_wp_page 0.20 ± 6% +0.1 0.35 ± 4% perf-profile.self.cycles-pp.do_page_fault 0.10 ± 12% +0.1 0.25 ± 5% perf-profile.self.cycles-pp.cpuidle_enter_state 0.17 ± 13% +0.2 0.32 ± 8% perf-profile.self.cycles-pp.deactivate_file_page 0.26 ± 4% +0.2 0.43 ± 7% perf-profile.self.cycles-pp.handle_pte_fault 0.22 ± 14% +0.2 0.39 ± 7% perf-profile.self.cycles-pp.invalidate_mapping_pages 0.09 ± 9% +0.2 0.27 ± 3% perf-profile.self.cycles-pp.__block_write_begin_int 0.29 ± 3% +0.2 0.48 ± 6% perf-profile.self.cycles-pp.find_get_pages_range_tag 0.21 ± 6% +0.2 0.41 ± 3% perf-profile.self.cycles-pp.__test_set_page_writeback 0.49 ± 8% +0.2 0.71 ± 5% perf-profile.self.cycles-pp.page_mapping 0.12 ± 27% +0.2 0.35 ± 3% perf-profile.self.cycles-pp.menu_select 0.90 ± 7% +0.2 1.14 ± 2% perf-profile.self.cycles-pp.unlock_page 0.25 ± 5% +0.2 0.49 ± 4% perf-profile.self.cycles-pp.ext4_bio_write_page 0.33 ± 8% +0.2 0.58 ± 6% perf-profile.self.cycles-pp.percpu_counter_add_batch 0.17 ± 11% +0.3 0.43 ± 8% perf-profile.self.cycles-pp.__set_page_dirty_buffers 0.36 ± 12% +0.3 0.63 ± 4% perf-profile.self.cycles-pp.release_pages 0.35 ± 9% +0.3 0.64 ± 4% perf-profile.self.cycles-pp.handle_mm_fault 0.51 ± 6% +0.3 0.81 ± 8% perf-profile.self.cycles-pp.__handle_mm_fault 0.22 ± 6% +0.3 0.53 ± 7% perf-profile.self.cycles-pp.test_clear_page_writeback 1.61 ± 7% +0.3 1.94 ± 3% perf-profile.self.cycles-pp.filemap_map_pages 0.54 ± 9% +0.3 0.87 ± 5% perf-profile.self.cycles-pp.xas_load 0.86 ± 5% +0.4 1.23 ± 4% perf-profile.self.cycles-pp.ext4_finish_bio 0.69 ± 9% +0.4 1.06 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.86 ± 9% +0.4 1.25 ± 4% perf-profile.self.cycles-pp.find_get_entries 0.30 ± 8% +0.4 0.70 ± 17% perf-profile.self.cycles-pp.ktime_get 0.61 ± 6% +0.4 1.06 ± 4% perf-profile.self.cycles-pp.sync_regs 0.76 ± 7% +0.5 1.23 ± 5% perf-profile.self.cycles-pp.get_io_u 4.28 ± 6% +0.6 4.92 ± 4% perf-profile.self.cycles-pp.__memcpy_mcsafe 1.09 ± 9% +0.7 1.76 ± 5% perf-profile.self.cycles-pp.native_irq_return_iret 3.43 ± 16% +3.3 6.72 ± 3% perf-profile.self.cycles-pp.__memcpy_flushcache fio.write_bw_MBps 6000 +--------------------------------------------------------------------+ | O O | 5000 |-+ O O O O O O O O O O O O O O O O O O O O O O O | | O .+. .+. | |.+.+.+.+.+.+.+ + +.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.| 4000 |-+ | | | 3000 |-+ | | | 2000 |-+ | | | | | 1000 |-+ | | | 0 +--------------------------------------------------------------------+ fio.read_clat_mean_us 3.5e+06 +-----------------------------------------------------------------+ |.+ .+ .+ .+. +. | 3e+06 |-++ .+.+ .+.+.+.+.+ + .+ + .++.+ +.+. + +.| | +.+.+.+.+ +.+ +.+.+ +.+.+ + | 2.5e+06 |-+ | | | 2e+06 |-+ | | | 1.5e+06 |-+ O O O O O O O O OO O O | | O O O O OO O O O O O O O O | 1e+06 |-+ | | | 500000 |-+ | | | 0 +-----------------------------------------------------------------+ fio.read_clat_90__us 9e+06 +-------------------------------------------------------------------+ | + +. : + | 8e+06 |.+ .+. + : + +.+.+ .+.+ +. .+.+ : :| 7e+06 |-+: +.+ +.+ : .+.+.+ : +.+ : +.+.+. + + + : :| | : + + : + : + + + | 6e+06 |-+ + + + | 5e+06 |-+ | | | 4e+06 |-+ | 3e+06 |-+ | | O O O O O O O O O O O O O O O O O O O O O O O O O O | 2e+06 |-+ | 1e+06 |-+ | | | 0 +-------------------------------------------------------------------+ fio.read_clat_95__us 1.4e+07 +-----------------------------------------------------------------+ |.+ .+ .+ +.+. +. | 1.2e+07 |-++ +. .+.+.+.+.+ + .+ + .++. + +. + + | | +.+.+.+. + ++.+ +.+.+ +.+.+ + + +| 1e+07 |-+ + | | | 8e+06 |-+ | | | 6e+06 |-+ | | | 4e+06 |-+ O | | O O O O O OO O O O O O O O O O O O O O O OO O O | 2e+06 |-+ | | | 0 +-----------------------------------------------------------------+ fio.read_clat_99__us 2.5e+07 +-----------------------------------------------------------------+ | | | +. | 2e+07 |.+. + +. .+. .+. ++. + +.+ +. | | +. : + + +.+.+.+ +. .+ +. + + + + +.| | +.+.+. : ++.+ +.+ +.+ + | 1.5e+07 |-+ + | | | 1e+07 |-+ | | O | | O O O O O O O O O O O OO O O | 5e+06 |-+ O O O O O O O O O O | | | | | 0 +-----------------------------------------------------------------+ fio.latency_500us_ 25 +----------------------------------------------------------------------+ | | | | 20 |-+ O O O O O O O O O O O O | | O O O O O O O O O O | | O O O O | 15 |-+ | | | 10 |-+ | | .+.+..+. .+.+.+. .+. .+.+. .+. .+. .| |.+.+.+ + +.+.+.+.+.+..+.+ +.+.+.+ + +..+.+ +.+ | 5 |-+ | | | | | 0 +----------------------------------------------------------------------+ fio.latency_2ms_ 25 +----------------------------------------------------------------------+ | O O O O O O O O O O | | O O O O O O O O O O O O O | 20 |-+ O | | | | | 15 |.+. .+.+.+..+.+.+.+.+.+.+.+.+.+.+.. .+.+.+.+. .+.+.+. .+.+..+.+. .+.+.| | + + + + + | 10 |-+ | | | | | 5 |-+ | | | | | 0 +----------------------------------------------------------------------+ fio.latency_4ms_ 35 +----------------------------------------------------------------------+ | | 30 |.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+. .+.+. .+.+.+.+.| | + +..+ | 25 |-+ O O O O O O O O O | | O O O O O O O O O O O O O O O O O | 20 |-+ | | | 15 |-+ | | | 10 |-+ | | | 5 |-+ | | | 0 +----------------------------------------------------------------------+ fio.latency_10ms_ 10 +----------------------------------------------------------------------+ 9 |-+ .+ + +.+ | | .+ + .+. + + .+.+.. + :| 8 |.+. .+.+.+..+ +.+ + +.+.+.. .+.+.+.+. .+.+.+.+ +.+.+ :| 7 |-+ + + + | | | 6 |-+ | 5 |-+ | 4 |-+ | | | 3 |-+ O O O O O O O O | 2 |-+ O O O O O O O O O O O O O O O O O O | | | 1 |-+ | 0 +----------------------------------------------------------------------+ fio.latency_20ms_ 8 +-----------------------------------------------------------------------+ | +. | 7 |.+ .+ +. .+ .+ : + | 6 |-+: + : +. .. +.+ : .+ + .+. .+.+.+..+. : :| | : .+..+ : : + + : .+ + .+ + + :| 5 |-+ +.+ + : +.+ +.+ +.+ | | + | 4 |-+ | | | 3 |-+ | 2 |-+ | | | 1 |-+ O | | O O O O O O O O O O O O O O O O O O O O O O O O O | 0 +-----------------------------------------------------------------------+ fio.time.system_time 3500 +--------------------------------------------------------------------+ | + +. | 3000 |-+ + + .+. .+. .+ : + | |.+. .+.+.+. .+ +.+.+.+ +.+.+.. .+.+.+.+. .+.+.+.+.+ + + : +| 2500 |-+ + + + + + | | | 2000 |-+ | | O O O O O O O O O O O O O O O O O O O O O O O O O | 1500 |-+ O | | | 1000 |-+ | | | 500 |-+ | | | 0 +--------------------------------------------------------------------+ fio.time.percent_of_cpu_this_job_got 2000 +--------------------------------------------------------------------+ 1800 |-+ +. | | .+. .+. .+. .+ : + | 1600 |.+. .+.+.+.+.+ +.+.+.+ +.+.+.. .+.+.+.+. .+.+.+.+.+ + + : +| 1400 |-+ + + + + | | | 1200 |-+ O O O O O O O O O O O O O | 1000 |-+ O O O O O O O O O O O O O | 800 |-+ | | | 600 |-+ | 400 |-+ | | | 200 |-+ | 0 +--------------------------------------------------------------------+ fio.time.major_page_faults 3e+08 +-----------------------------------------------------------------+ | | 2.5e+08 |-+ .+ .+. .+.+.| |.+.+.+.+.+.+.+ +.+.+.+ +.+.+.+.+.+.+.+.+.+.+.++.+. .+.+.+ | | + | 2e+08 |-+ O O O O OO O O O O O O O O O O O O O O O OO O O | | O | 1.5e+08 |-+ | | | 1e+08 |-+ | | | | | 5e+07 |-+ | | | 0 +-----------------------------------------------------------------+ fio.time.minor_page_faults 2.5e+08 +-----------------------------------------------------------------+ | O O O O O | | O O OO O O O O O O O O O O O O OO O O | 2e+08 |-+ O | | | | | 1.5e+08 |-+ .+ .+. .+.+. .+. | |.+.+.+.+.+.+.+ + +.+.+.+.+.+.+.+.+.+.+.+.+.+.++ +.+.+ +.| 1e+08 |-+ | | | | | 5e+07 |-+ | | | | | 0 +-----------------------------------------------------------------+ fio.time.file_system_inputs 2.5e+09 +-----------------------------------------------------------------+ | | | + | 2e+09 |.+ .+.+.+.+. + +.+. .+.+.+. .+.+.+. .+.+. .+. .+. .+.+.| | +.+ + +.+ +.+ +.+ ++ +.+ + | | O O O O OO O O O O O O O O O O O | 1.5e+09 |-+ O O O O O O O O O | | | 1e+09 |-+ | | | | | 5e+08 |-+ | | | | | 0 +-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 years, 3 months

2
1
0 / 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

LKP February 2020