Under construction

About CPU hotplug

One main interest of SMP systems compared to single core systems is the energy efficiency (DMIPS/mW) which enables a system either to do more with the same power consumption or reduce the power consumption of a Use Case. Based on this assumption, the cpu hotplug feature might not be very useful (from a power consumption point of view) for a SMP system. Nevertheless, this hypotheses is no more valid when the cpu load becomes very low. In this case, the power consumption is mainly linked to the number of wakeup and the time spent in retention (or a deeper C-state). The cpu hotplug could be interesting because it can help to go faster in retention state by simplifying the entry condition.

Arm CPU hotplug

The current cpu hotplug implementation of arm platforms are based on the same skeleton. We have 3 main function :

  • platform_cpu_kill and platform_cpu_disable are identical across all platform.
  • platform_cpu_die is very similar and we can extract a common skeleton :
    • check that we are on the right cpu.
    • complete the shutdown request
    • prepare to enter the power-down state (including cache flush)
    • loop in the power down state until we plug the cpu (manage the spurious wakeups)
    • leave the power-down state

Some code factorization have been added in the next release : the completion has been put in the common arm code. The platform_cpu_kill just need to return 1 unless some specific platform stuff and the platform_cpu_die just need to deal with the powerdown sequence of the core.

Using CPU hotplug on 2.6.35

Unplugging a cpu saves power but it also takes time, so we must find the best trade off between the waste of power for plugging/unplugging a cpu and improvement of the power consumption or the performance. Before doing any assumption of the best policy, we need to get some measurement of the latency, the idle state residency improvement and the power consumption improvement.

  • Latency for plugging/unplugging a cpu: use some tracepoints.
  • Idle improvement : the time spent by the CA-9 in the deepest sleep state should be improved by cpu hotplug
  • Power consumption: static power improvement linked to cpu hotplug

CPU hotplug latency

We need to measure several steps in the cpu plug/inplug sequence

  • time spent for the complete plug/unplug sequence
  • time spent in the architecture part (arm common part)

Add tracepoints in the cpu hotplug code for profiling: A proposal has been sent to LKML

The example below shows a plug latency time on an arm platform

# tracer: nop
#
#           TASK-PID    CPU#    TIMESTAMP  FUNCTION
#              | |       |          |         |
           <...>-3619  [000]   146.532388: cpu_hotplug_up_start: cpuid=1
           <...>-3619  [000]   146.548780: cpu_hotplug_arch_up_start: cpuid=1
           <...>-3619  [000]   146.549012: cpu_hotplug_arch_up_end: cpuid=1
           <...>-3619  [000]   146.588855: cpu_hotplug_up_end: cpuid=1

The latency measurements must be done in both idle and loaded use cases.

The results that are shown below, are base on 220 latency measurements which have been made with different cpu loads. We have used cyclictest for simulating the cpu load and each 20 tests, we have increased the number of running cyclictest processes.

The script is available here: test_cpu_hotplug_latency.sh

Step

Min

Max

cpu up

cpu up

95450us

4164866us

arch up

168us

451us

cpu down

cpu down

54796us

1526454us

arch disable

107us

194us

arch die

23us

1963us

The complete results are available here: cpu_hotlpug_results.dat

The graphs below show the cpu_up and the cpu_down duration when we are increasing the cpu load

cpu_hotplug_up_stat.png cpu_hotplug_down_stat.png

For the cpu_up sequence, the duration increases with the cpu load, but the architecture part remains short whatever the cpu load is. The cpu down sequence seems to be not impacted by the cpu load. Nevertheless, we can see some random large duration. A 1st analyses shows that the cpu_notify list is responsible of the duration increase. More precisely, the cpu_up sequence is waiting for the creation of some threads as shown is the graph below.

cpu_hotplug_up_task_scheduling.png

The 1st remark is that we must take care of removing the useless threads like the kconservative workqueue which is not used in our example but which slows down the plug sequence (ondemand governor is used instead).

Then, the only way to improve the cpu plug duration is to ensure that the threads will have enough cpu bandwidth during their creation. The cgroup could be a solution to ensure such bandwidth for the creation of thread. As a proof of concept, we have make a simple test with cgroup and cpu hotplug : we have created a cgroup with lot of cpu bandwidth. This cgroup is empty except for the cpu plug sequence where we add kthreadd to the cgroup. This ensures that the newly created threads will have enough bandwidth. Then, we move all tasks back to the default group at the end of the sequence.

The results show an improvement of the cpu plug duration with a maximum duration of 165ms against more than 4sec previously

Step

Min

Max

cpu up

cpu up

63339us

165150us

arch up

120us

1546us

The complete results are available here: to add

The graph below shows the cpu_up duration when we are increasing the cpu load

cpu_hotplug_up_cgroup_stat.png

This simple test shows that we can take advantage of the cgroup to minimize cpu plug duration.

Idle Improvement

Most of dual-core SoC can reach the retention state (or a deeper state) only when both cores are ready to enter this state. As the cores run asynchronously, they wait each others in a WFI state before entering retention state. When a core is unplugged, the system can enter the retention state as soon as the plugged core wants to enter the retention state. Unplugging a core improves the time spent in retention state by removing the step during which a core stays in WFI. On the other hand, the use of a single core will increase run time. We need to check which use-case could take advantage of using only one core.

The results below shows cpuidle statistics results for an idle use case of 80 sec with 1 and 2 cpus

The script is available here: to add

State

%

avg duration

avg wakeup

MP

WFI

4%

62ms

8/s

Retention

95%

119ms

4/s

UP

WFI

1%

5ms

2/s

Retention

98%

222ms

4/s

Power consumption

CPU hotplug on latest kernel

Several ongoing developments have been started to improve the duration of CPU plug/unplug sequence

workqueue: reimplement CPU hotplug to keep idle workers [Tejun Heo]

Tejun's patches are available in is git tree: git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git review-wq-hotplug. We have tested the plug latency on a dual cortex-A9 platform with both the kernel v3.5 and the patches which parks the kthreads instead of destroying them when a CPU is unplugged.

The graph below shows the improvement achieved with the patches. We save the time to create kthreads when a CPU is plugged.

cpu_hotplug_up_wq.png

Other

studying sched_mc for ARM

References

WorkingGroups/PowerManagement/Archives/Hotplug (last modified 2013-08-21 11:07:20)