2019-06-04 08:11:33 +00:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0-only */
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
2013-06-19 08:49:33 +00:00
|
|
|
* linux/include/linux/cpufreq.h
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
2013-06-19 08:49:33 +00:00
|
|
|
* Copyright (C) 2001 Russell King
|
|
|
|
* (C) 2002 - 2003 Dominik Brodowski <linux@brodo.de>
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
#ifndef _LINUX_CPUFREQ_H
|
|
|
|
#define _LINUX_CPUFREQ_H
|
|
|
|
|
2014-01-09 15:08:43 +00:00
|
|
|
#include <linux/clk.h>
|
2013-08-06 17:23:03 +00:00
|
|
|
#include <linux/cpumask.h>
|
|
|
|
#include <linux/completion.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <linux/kobject.h>
|
2013-08-06 17:23:03 +00:00
|
|
|
#include <linux/notifier.h>
|
cpufreq: Use per-policy frequency QoS
Replace the CPU device PM QoS used for the management of min and max
frequency constraints in cpufreq (and its users) with per-policy
frequency QoS to avoid problems with cpufreq policies covering
more then one CPU.
Namely, a cpufreq driver is registered with the subsys interface
which calls cpufreq_add_dev() for each CPU, starting from CPU0, so
currently the PM QoS notifiers are added to the first CPU in the
policy (i.e. CPU0 in the majority of cases).
In turn, when the cpufreq driver is unregistered, the subsys interface
doing that calls cpufreq_remove_dev() for each CPU, starting from CPU0,
and the PM QoS notifiers are only removed when cpufreq_remove_dev() is
called for the last CPU in the policy, say CPUx, which as a rule is
not CPU0 if the policy covers more than one CPU. Then, the PM QoS
notifiers cannot be removed, because CPUx does not have them, and
they are still there in the device PM QoS notifiers list of CPU0,
which prevents new PM QoS notifiers from being registered for CPU0
on the next attempt to register the cpufreq driver.
The same issue occurs when the first CPU in the policy goes offline
before unregistering the driver.
After this change it does not matter which CPU is the policy CPU at
the driver registration time and whether or not it is online all the
time, because the frequency QoS is per policy and not per CPU.
Fixes: 67d874c3b2c6 ("cpufreq: Register notifiers with the PM QoS framework")
Reported-by: Dmitry Osipenko <digetx@gmail.com>
Tested-by: Dmitry Osipenko <digetx@gmail.com>
Reported-by: Sudeep Holla <sudeep.holla@arm.com>
Tested-by: Sudeep Holla <sudeep.holla@arm.com>
Diagnosed-by: Viresh Kumar <viresh.kumar@linaro.org>
Link: https://lore.kernel.org/linux-pm/5ad2624194baa2f53acc1f1e627eb7684c577a19.1562210705.git.viresh.kumar@linaro.org/T/#md2d89e95906b8c91c15f582146173dce2e86e99f
Link: https://lore.kernel.org/linux-pm/20191017094612.6tbkwoq4harsjcqv@vireshk-i7/T/#m30d48cc23b9a80467fbaa16e30f90b3828a5a29b
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
2019-10-16 10:47:06 +00:00
|
|
|
#include <linux/pm_qos.h>
|
cpufreq: Make sure frequency transitions are serialized
Whenever we change the frequency of a CPU, we call the PRECHANGE and POSTCHANGE
notifiers. They must be serialized, i.e. PRECHANGE and POSTCHANGE notifiers
should strictly alternate, thereby preventing two different sets of PRECHANGE or
POSTCHANGE notifiers from interleaving arbitrarily.
The following examples illustrate why this is important:
Scenario 1:
-----------
A thread reading the value of cpuinfo_cur_freq, will call
__cpufreq_cpu_get()->cpufreq_out_of_sync()->cpufreq_notify_transition()
The ondemand governor can decide to change the frequency of the CPU at the same
time and hence it can end up sending the notifications via ->target().
If the notifiers are not serialized, the following sequence can occur:
- PRECHANGE Notification for freq A (from cpuinfo_cur_freq)
- PRECHANGE Notification for freq B (from target())
- Freq changed by target() to B
- POSTCHANGE Notification for freq B
- POSTCHANGE Notification for freq A
We can see from the above that the last POSTCHANGE Notification happens for freq
A but the hardware is set to run at freq B.
Where would we break then?: adjust_jiffies() in cpufreq.c & cpufreq_callback()
in arch/arm/kernel/smp.c (which also adjusts the jiffies). All the
loops_per_jiffy calculations will get messed up.
Scenario 2:
-----------
The governor calls __cpufreq_driver_target() to change the frequency. At the
same time, if we change scaling_{min|max}_freq from sysfs, it will end up
calling the governor's CPUFREQ_GOV_LIMITS notification, which will also call
__cpufreq_driver_target(). And hence we end up issuing concurrent calls to
->target().
Typically, platforms have the following logic in their ->target() routines:
(Eg: cpufreq-cpu0, omap, exynos, etc)
A. If new freq is more than old: Increase voltage
B. Change freq
C. If new freq is less than old: decrease voltage
Now, if the two concurrent calls to ->target() are X and Y, where X is trying to
increase the freq and Y is trying to decrease it, we get the following race
condition:
X.A: voltage gets increased for larger freq
Y.A: nothing happens
Y.B: freq gets decreased
Y.C: voltage gets decreased
X.B: freq gets increased
X.C: nothing happens
Thus we can end up setting a freq which is not supported by the voltage we have
set. That will probably make the clock to the CPU unstable and the system might
not work properly anymore.
This patch introduces a set of synchronization primitives to serialize frequency
transitions, which are to be used as shown below:
cpufreq_freq_transition_begin();
//Perform the frequency change
cpufreq_freq_transition_end();
The _begin() call sends the PRECHANGE notification whereas the _end() call sends
the POSTCHANGE notification. Also, all the necessary synchronization is handled
within these calls. In particular, even drivers which set the ASYNC_NOTIFICATION
flag can also use these APIs for performing frequency transitions (ie., you can
call _begin() from one task, and call the corresponding _end() from a different
task).
The actual synchronization underneath is not that complicated:
The key challenge is to allow drivers to begin the transition from one thread
and end it in a completely different thread (this is to enable drivers that do
asynchronous POSTCHANGE notification from bottom-halves, to also use the same
interface).
To achieve this, a 'transition_ongoing' flag, a 'transition_lock' spinlock and a
wait-queue are added per-policy. The flag and the wait-queue are used in
conjunction to create an "uninterrupted flow" from _begin() to _end(). The
spinlock is used to ensure that only one such "flow" is in flight at any given
time. Put together, this provides us all the necessary synchronization.
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
2014-03-24 08:05:44 +00:00
|
|
|
#include <linux/spinlock.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <linux/sysfs.h>
|
|
|
|
|
|
|
|
/*********************************************************************
|
2013-08-06 17:23:04 +00:00
|
|
|
* CPUFREQ INTERFACE *
|
2005-04-16 22:20:36 +00:00
|
|
|
*********************************************************************/
|
2013-08-06 17:23:04 +00:00
|
|
|
/*
|
|
|
|
* Frequency values here are CPU kHz
|
|
|
|
*
|
2005-06-01 02:03:47 +00:00
|
|
|
* Maximum transition latency is in nanoseconds - if it's unknown,
|
2005-04-16 22:20:36 +00:00
|
|
|
* CPUFREQ_ETERNAL shall be used.
|
|
|
|
*/
|
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
#define CPUFREQ_ETERNAL (-1)
|
|
|
|
#define CPUFREQ_NAME_LEN 16
|
2017-02-03 09:56:25 +00:00
|
|
|
/* Print length for names. Extra 1 space for accommodating '\n' in prints */
|
2013-08-06 17:23:04 +00:00
|
|
|
#define CPUFREQ_NAME_PLEN (CPUFREQ_NAME_LEN + 1)
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
struct cpufreq_governor;
|
|
|
|
|
2016-06-27 10:34:07 +00:00
|
|
|
enum cpufreq_table_sorting {
|
|
|
|
CPUFREQ_TABLE_UNSORTED,
|
|
|
|
CPUFREQ_TABLE_SORTED_ASCENDING,
|
|
|
|
CPUFREQ_TABLE_SORTED_DESCENDING
|
|
|
|
};
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
struct cpufreq_cpuinfo {
|
|
|
|
unsigned int max_freq;
|
|
|
|
unsigned int min_freq;
|
2011-04-28 23:42:53 +00:00
|
|
|
|
|
|
|
/* in 10^(-9) s = nanoseconds */
|
|
|
|
unsigned int transition_latency;
|
2005-04-16 22:20:36 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
struct cpufreq_policy {
|
2013-01-31 02:03:53 +00:00
|
|
|
/* CPUs sharing clock, require sw coordination */
|
|
|
|
cpumask_var_t cpus; /* Online CPUs only */
|
|
|
|
cpumask_var_t related_cpus; /* Online + Offline CPUs */
|
cpufreq: Avoid attempts to create duplicate symbolic links
After commit 87549141d516 (cpufreq: Stop migrating sysfs files on
hotplug) there is a problem with CPUs that share cpufreq policy
objects with other CPUs and are initially offline.
Say CPU1 shares a policy with CPU0 which is online and is registered
first. As part of the registration process, cpufreq_add_dev() is
called for it. It creates the policy object and a symbolic link
to it from the CPU1's sysfs directory. If CPU1 is registered
subsequently and it is offline at that time, cpufreq_add_dev() will
attempt to create a symbolic link to the policy object for it, but
that link is present already, so a warning about that will be
triggered.
To avoid that warning, make cpufreq use an additional CPU mask
containing related CPUs that are actually present for each policy
object. That mask is initialized when the policy object is populated
after its creation (for the first online CPU using it) and it includes
CPUs from the "policy CPUs" mask returned by the cpufreq driver's
->init() callback that are physically present at that time. Symbolic
links to the policy are created only for the CPUs in that mask.
If cpufreq_add_dev() is invoked for an offline CPU, it checks the
new mask and only creates the symlink if the CPU was not in it (the
CPU is added to the mask at the same time).
In turn, cpufreq_remove_dev() drops the given CPU from the new mask,
removes its symlink to the policy object and returns, unless it is
the CPU owning the policy object. In that case, the policy object
is moved to a new CPU's sysfs directory or deleted if the CPU being
removed was the last user of the policy.
While at it, notice that cpufreq_remove_dev() can't fail, because
its return value is ignored, so make it ignore return values from
__cpufreq_remove_dev_prepare() and __cpufreq_remove_dev_finish()
and prevent these functions from aborting on errors returned by
__cpufreq_governor(). Also drop the now unused sif argument from
them.
Fixes: 87549141d516 (cpufreq: Stop migrating sysfs files on hotplug)
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Reported-and-tested-by: Russell King <linux@arm.linux.org.uk>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
2015-07-26 00:07:47 +00:00
|
|
|
cpumask_var_t real_cpus; /* Related and present */
|
2013-01-31 02:03:53 +00:00
|
|
|
|
2013-02-01 06:40:02 +00:00
|
|
|
unsigned int shared_type; /* ACPI: ANY or ALL affected CPUs
|
2005-12-14 20:05:00 +00:00
|
|
|
should set cpufreq */
|
2015-05-18 05:13:31 +00:00
|
|
|
unsigned int cpu; /* cpu managing this policy, must be online */
|
|
|
|
|
2014-01-09 15:08:43 +00:00
|
|
|
struct clk *clk;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct cpufreq_cpuinfo cpuinfo;/* see above */
|
|
|
|
|
|
|
|
unsigned int min; /* in kHz */
|
|
|
|
unsigned int max; /* in kHz */
|
|
|
|
unsigned int cur; /* in kHz, only needed if cpufreq
|
|
|
|
* governors are used */
|
2014-06-02 17:19:28 +00:00
|
|
|
unsigned int restore_freq; /* = policy->cur before transition */
|
2014-03-04 03:00:27 +00:00
|
|
|
unsigned int suspend_freq; /* freq to set during suspend */
|
|
|
|
|
2011-04-28 23:42:53 +00:00
|
|
|
unsigned int policy; /* see above */
|
2015-12-02 00:52:14 +00:00
|
|
|
unsigned int last_policy; /* policy before unplug */
|
2005-04-16 22:20:36 +00:00
|
|
|
struct cpufreq_governor *governor; /* see below */
|
2013-03-27 15:58:57 +00:00
|
|
|
void *governor_data;
|
2015-05-12 06:52:34 +00:00
|
|
|
char last_governor[CPUFREQ_NAME_LEN]; /* last governor used */
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
struct work_struct update; /* if update_policy() needs to be
|
|
|
|
* called, but you're in IRQ context */
|
|
|
|
|
cpufreq: Use per-policy frequency QoS
Replace the CPU device PM QoS used for the management of min and max
frequency constraints in cpufreq (and its users) with per-policy
frequency QoS to avoid problems with cpufreq policies covering
more then one CPU.
Namely, a cpufreq driver is registered with the subsys interface
which calls cpufreq_add_dev() for each CPU, starting from CPU0, so
currently the PM QoS notifiers are added to the first CPU in the
policy (i.e. CPU0 in the majority of cases).
In turn, when the cpufreq driver is unregistered, the subsys interface
doing that calls cpufreq_remove_dev() for each CPU, starting from CPU0,
and the PM QoS notifiers are only removed when cpufreq_remove_dev() is
called for the last CPU in the policy, say CPUx, which as a rule is
not CPU0 if the policy covers more than one CPU. Then, the PM QoS
notifiers cannot be removed, because CPUx does not have them, and
they are still there in the device PM QoS notifiers list of CPU0,
which prevents new PM QoS notifiers from being registered for CPU0
on the next attempt to register the cpufreq driver.
The same issue occurs when the first CPU in the policy goes offline
before unregistering the driver.
After this change it does not matter which CPU is the policy CPU at
the driver registration time and whether or not it is online all the
time, because the frequency QoS is per policy and not per CPU.
Fixes: 67d874c3b2c6 ("cpufreq: Register notifiers with the PM QoS framework")
Reported-by: Dmitry Osipenko <digetx@gmail.com>
Tested-by: Dmitry Osipenko <digetx@gmail.com>
Reported-by: Sudeep Holla <sudeep.holla@arm.com>
Tested-by: Sudeep Holla <sudeep.holla@arm.com>
Diagnosed-by: Viresh Kumar <viresh.kumar@linaro.org>
Link: https://lore.kernel.org/linux-pm/5ad2624194baa2f53acc1f1e627eb7684c577a19.1562210705.git.viresh.kumar@linaro.org/T/#md2d89e95906b8c91c15f582146173dce2e86e99f
Link: https://lore.kernel.org/linux-pm/20191017094612.6tbkwoq4harsjcqv@vireshk-i7/T/#m30d48cc23b9a80467fbaa16e30f90b3828a5a29b
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
2019-10-16 10:47:06 +00:00
|
|
|
struct freq_constraints constraints;
|
|
|
|
struct freq_qos_request *min_freq_req;
|
|
|
|
struct freq_qos_request *max_freq_req;
|
|
|
|
|
2014-03-10 09:23:33 +00:00
|
|
|
struct cpufreq_frequency_table *freq_table;
|
2016-06-27 10:34:07 +00:00
|
|
|
enum cpufreq_table_sorting freq_table_sorted;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-08-06 17:23:08 +00:00
|
|
|
struct list_head policy_list;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct kobject kobj;
|
|
|
|
struct completion kobj_unregister;
|
2013-10-18 13:40:15 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The rules for this semaphore:
|
|
|
|
* - Any routine that wants to read from the policy structure will
|
|
|
|
* do a down_read on this semaphore.
|
|
|
|
* - Any routine that will write to the policy structure and/or may take away
|
|
|
|
* the policy altogether (eg. CPU hotplug), will hold this lock in write
|
|
|
|
* mode before doing so.
|
|
|
|
*/
|
|
|
|
struct rw_semaphore rwsem;
|
cpufreq: Make sure frequency transitions are serialized
Whenever we change the frequency of a CPU, we call the PRECHANGE and POSTCHANGE
notifiers. They must be serialized, i.e. PRECHANGE and POSTCHANGE notifiers
should strictly alternate, thereby preventing two different sets of PRECHANGE or
POSTCHANGE notifiers from interleaving arbitrarily.
The following examples illustrate why this is important:
Scenario 1:
-----------
A thread reading the value of cpuinfo_cur_freq, will call
__cpufreq_cpu_get()->cpufreq_out_of_sync()->cpufreq_notify_transition()
The ondemand governor can decide to change the frequency of the CPU at the same
time and hence it can end up sending the notifications via ->target().
If the notifiers are not serialized, the following sequence can occur:
- PRECHANGE Notification for freq A (from cpuinfo_cur_freq)
- PRECHANGE Notification for freq B (from target())
- Freq changed by target() to B
- POSTCHANGE Notification for freq B
- POSTCHANGE Notification for freq A
We can see from the above that the last POSTCHANGE Notification happens for freq
A but the hardware is set to run at freq B.
Where would we break then?: adjust_jiffies() in cpufreq.c & cpufreq_callback()
in arch/arm/kernel/smp.c (which also adjusts the jiffies). All the
loops_per_jiffy calculations will get messed up.
Scenario 2:
-----------
The governor calls __cpufreq_driver_target() to change the frequency. At the
same time, if we change scaling_{min|max}_freq from sysfs, it will end up
calling the governor's CPUFREQ_GOV_LIMITS notification, which will also call
__cpufreq_driver_target(). And hence we end up issuing concurrent calls to
->target().
Typically, platforms have the following logic in their ->target() routines:
(Eg: cpufreq-cpu0, omap, exynos, etc)
A. If new freq is more than old: Increase voltage
B. Change freq
C. If new freq is less than old: decrease voltage
Now, if the two concurrent calls to ->target() are X and Y, where X is trying to
increase the freq and Y is trying to decrease it, we get the following race
condition:
X.A: voltage gets increased for larger freq
Y.A: nothing happens
Y.B: freq gets decreased
Y.C: voltage gets decreased
X.B: freq gets increased
X.C: nothing happens
Thus we can end up setting a freq which is not supported by the voltage we have
set. That will probably make the clock to the CPU unstable and the system might
not work properly anymore.
This patch introduces a set of synchronization primitives to serialize frequency
transitions, which are to be used as shown below:
cpufreq_freq_transition_begin();
//Perform the frequency change
cpufreq_freq_transition_end();
The _begin() call sends the PRECHANGE notification whereas the _end() call sends
the POSTCHANGE notification. Also, all the necessary synchronization is handled
within these calls. In particular, even drivers which set the ASYNC_NOTIFICATION
flag can also use these APIs for performing frequency transitions (ie., you can
call _begin() from one task, and call the corresponding _end() from a different
task).
The actual synchronization underneath is not that complicated:
The key challenge is to allow drivers to begin the transition from one thread
and end it in a completely different thread (this is to enable drivers that do
asynchronous POSTCHANGE notification from bottom-halves, to also use the same
interface).
To achieve this, a 'transition_ongoing' flag, a 'transition_lock' spinlock and a
wait-queue are added per-policy. The flag and the wait-queue are used in
conjunction to create an "uninterrupted flow" from _begin() to _end(). The
spinlock is used to ensure that only one such "flow" is in flight at any given
time. Put together, this provides us all the necessary synchronization.
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
2014-03-24 08:05:44 +00:00
|
|
|
|
2016-03-30 01:47:49 +00:00
|
|
|
/*
|
|
|
|
* Fast switch flags:
|
|
|
|
* - fast_switch_possible should be set by the driver if it can
|
|
|
|
* guarantee that frequency can be changed on any CPU sharing the
|
|
|
|
* policy and that the change will affect all of the policy CPUs then.
|
|
|
|
* - fast_switch_enabled is to be set by governors that support fast
|
2017-02-03 09:56:25 +00:00
|
|
|
* frequency switching with the help of cpufreq_enable_fast_switch().
|
2016-03-30 01:47:49 +00:00
|
|
|
*/
|
|
|
|
bool fast_switch_possible;
|
|
|
|
bool fast_switch_enabled;
|
|
|
|
|
2020-11-10 17:26:37 +00:00
|
|
|
/*
|
|
|
|
* Set if the CPUFREQ_GOV_STRICT_TARGET flag is set for the current
|
|
|
|
* governor.
|
|
|
|
*/
|
|
|
|
bool strict_target;
|
|
|
|
|
2017-04-10 22:20:41 +00:00
|
|
|
/*
|
|
|
|
* Preferred average time interval between consecutive invocations of
|
|
|
|
* the driver to set the frequency for this policy. To be set by the
|
|
|
|
* scaling driver (0, which is the default, means no preference).
|
|
|
|
*/
|
|
|
|
unsigned int transition_delay_us;
|
|
|
|
|
2017-07-28 06:46:39 +00:00
|
|
|
/*
|
|
|
|
* Remote DVFS flag (Not added to the driver structure as we don't want
|
|
|
|
* to access another structure from scheduler hotpath).
|
|
|
|
*
|
|
|
|
* Should be set if CPUs can do DVFS on behalf of other CPUs from
|
|
|
|
* different cpufreq policies.
|
|
|
|
*/
|
|
|
|
bool dvfs_possible_from_any_cpu;
|
|
|
|
|
2016-07-13 20:25:25 +00:00
|
|
|
/* Cached frequency lookup from cpufreq_driver_resolve_freq. */
|
|
|
|
unsigned int cached_target_freq;
|
2020-07-30 03:29:40 +00:00
|
|
|
unsigned int cached_resolved_idx;
|
2016-07-13 20:25:25 +00:00
|
|
|
|
cpufreq: Make sure frequency transitions are serialized
Whenever we change the frequency of a CPU, we call the PRECHANGE and POSTCHANGE
notifiers. They must be serialized, i.e. PRECHANGE and POSTCHANGE notifiers
should strictly alternate, thereby preventing two different sets of PRECHANGE or
POSTCHANGE notifiers from interleaving arbitrarily.
The following examples illustrate why this is important:
Scenario 1:
-----------
A thread reading the value of cpuinfo_cur_freq, will call
__cpufreq_cpu_get()->cpufreq_out_of_sync()->cpufreq_notify_transition()
The ondemand governor can decide to change the frequency of the CPU at the same
time and hence it can end up sending the notifications via ->target().
If the notifiers are not serialized, the following sequence can occur:
- PRECHANGE Notification for freq A (from cpuinfo_cur_freq)
- PRECHANGE Notification for freq B (from target())
- Freq changed by target() to B
- POSTCHANGE Notification for freq B
- POSTCHANGE Notification for freq A
We can see from the above that the last POSTCHANGE Notification happens for freq
A but the hardware is set to run at freq B.
Where would we break then?: adjust_jiffies() in cpufreq.c & cpufreq_callback()
in arch/arm/kernel/smp.c (which also adjusts the jiffies). All the
loops_per_jiffy calculations will get messed up.
Scenario 2:
-----------
The governor calls __cpufreq_driver_target() to change the frequency. At the
same time, if we change scaling_{min|max}_freq from sysfs, it will end up
calling the governor's CPUFREQ_GOV_LIMITS notification, which will also call
__cpufreq_driver_target(). And hence we end up issuing concurrent calls to
->target().
Typically, platforms have the following logic in their ->target() routines:
(Eg: cpufreq-cpu0, omap, exynos, etc)
A. If new freq is more than old: Increase voltage
B. Change freq
C. If new freq is less than old: decrease voltage
Now, if the two concurrent calls to ->target() are X and Y, where X is trying to
increase the freq and Y is trying to decrease it, we get the following race
condition:
X.A: voltage gets increased for larger freq
Y.A: nothing happens
Y.B: freq gets decreased
Y.C: voltage gets decreased
X.B: freq gets increased
X.C: nothing happens
Thus we can end up setting a freq which is not supported by the voltage we have
set. That will probably make the clock to the CPU unstable and the system might
not work properly anymore.
This patch introduces a set of synchronization primitives to serialize frequency
transitions, which are to be used as shown below:
cpufreq_freq_transition_begin();
//Perform the frequency change
cpufreq_freq_transition_end();
The _begin() call sends the PRECHANGE notification whereas the _end() call sends
the POSTCHANGE notification. Also, all the necessary synchronization is handled
within these calls. In particular, even drivers which set the ASYNC_NOTIFICATION
flag can also use these APIs for performing frequency transitions (ie., you can
call _begin() from one task, and call the corresponding _end() from a different
task).
The actual synchronization underneath is not that complicated:
The key challenge is to allow drivers to begin the transition from one thread
and end it in a completely different thread (this is to enable drivers that do
asynchronous POSTCHANGE notification from bottom-halves, to also use the same
interface).
To achieve this, a 'transition_ongoing' flag, a 'transition_lock' spinlock and a
wait-queue are added per-policy. The flag and the wait-queue are used in
conjunction to create an "uninterrupted flow" from _begin() to _end(). The
spinlock is used to ensure that only one such "flow" is in flight at any given
time. Put together, this provides us all the necessary synchronization.
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
2014-03-24 08:05:44 +00:00
|
|
|
/* Synchronization for frequency transitions */
|
|
|
|
bool transition_ongoing; /* Tracks transition status */
|
|
|
|
spinlock_t transition_lock;
|
|
|
|
wait_queue_head_t transition_wait;
|
cpufreq: Catch double invocations of cpufreq_freq_transition_begin/end
Some cpufreq drivers were redundantly invoking the _begin() and _end()
APIs around frequency transitions, and this double invocation (one from
the cpufreq core and the other from the cpufreq driver) used to result
in a self-deadlock, leading to system hangs during boot. (The _begin()
API makes contending callers wait until the previous invocation is
complete. Hence, the cpufreq driver would end up waiting on itself!).
Now all such drivers have been fixed, but debugging this issue was not
very straight-forward (even lockdep didn't catch this). So let us add a
debug infrastructure to the cpufreq core to catch such issues more easily
in the future.
We add a new field called 'transition_task' to the policy structure, to keep
track of the task which is performing the frequency transition. Using this
field, we make note of this task during _begin() and print a warning if we
find a case where the same task is calling _begin() again, before completing
the previous frequency transition using the corresponding _end().
We have left out ASYNC_NOTIFICATION drivers from this debug infrastructure
for 2 reasons:
1. At the moment, we have no way to avoid a particular scenario where this
debug infrastructure can emit false-positive warnings for such drivers.
The scenario is depicted below:
Task A Task B
/* 1st freq transition */
Invoke _begin() {
...
...
}
Change the frequency
/* 2nd freq transition */
Invoke _begin() {
... //waiting for B to
... //finish _end() for
... //the 1st transition
... | Got interrupt for successful
... | change of frequency (1st one).
... |
... | /* 1st freq transition */
... | Invoke _end() {
... | ...
... V }
...
...
}
This scenario is actually deadlock-free because, once Task A changes the
frequency, it is Task B's responsibility to invoke the corresponding
_end() for the 1st frequency transition. Hence it is perfectly legal for
Task A to go ahead and attempt another frequency transition in the meantime.
(Of course it won't be able to proceed until Task B finishes the 1st _end(),
but this doesn't cause a deadlock or a hang).
The debug infrastructure cannot handle this scenario and will treat it as
a deadlock and print a warning. To avoid this, we exclude such drivers
from the purview of this code.
2. Luckily, we don't _need_ this infrastructure for ASYNC_NOTIFICATION drivers
at all! The cpufreq core does not automatically invoke the _begin() and
_end() APIs during frequency transitions in such drivers. Thus, the driver
alone is responsible for invoking _begin()/_end() and hence there shouldn't
be any conflicts which lead to double invocations. So, we can skip these
drivers, since the probability that such drivers will hit this problem is
extremely low, as outlined above.
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
2014-05-05 07:22:39 +00:00
|
|
|
struct task_struct *transition_task; /* Task which is doing the transition */
|
2014-08-28 05:52:23 +00:00
|
|
|
|
2015-01-13 06:04:00 +00:00
|
|
|
/* cpufreq-stats */
|
|
|
|
struct cpufreq_stats *stats;
|
|
|
|
|
2014-08-28 05:52:23 +00:00
|
|
|
/* For cpufreq driver's internal use */
|
|
|
|
void *driver_data;
|
2019-01-30 05:22:01 +00:00
|
|
|
|
|
|
|
/* Pointer to the cooling device if used for thermal mitigation */
|
|
|
|
struct thermal_cooling_device *cdev;
|
2019-07-08 10:57:52 +00:00
|
|
|
|
|
|
|
struct notifier_block nb_min;
|
|
|
|
struct notifier_block nb_max;
|
2005-04-16 22:20:36 +00:00
|
|
|
};
|
|
|
|
|
cpufreq: Avoid creating excessively large stack frames
In the process of modifying a cpufreq policy, the cpufreq core makes
a copy of it including all of the internals which is stored on the
CPU stack. Because struct cpufreq_policy is relatively large, this
may cause the size of the stack frame to exceed the 2 KB limit and
so the GCC complains when -Wframe-larger-than= is used.
In fact, it is not necessary to copy the entire policy structure
in order to modify it, however.
First, because cpufreq_set_policy() obtains the min and max policy
limits from frequency QoS now, it is not necessary to pass the limits
to it from the callers. The only things that need to be passed to it
from there are the new governor pointer or (if there is a built-in
governor in the driver) the "policy" value representing the governor
choice. They both can be passed as individual arguments, though, so
make cpufreq_set_policy() take them this way and rework its callers
accordingly. This avoids making copies of cpufreq policies in the
callers of cpufreq_set_policy().
Second, cpufreq_set_policy() still needs to pass the new policy
data to the ->verify() callback of the cpufreq driver whose task
is to sanitize the min and max policy limits. It still does not
need to make a full copy of struct cpufreq_policy for this purpose,
but it needs to pass a few items from it to the driver in case they
are needed (different drivers have different needs in that respect
and all of them have to be covered). For this reason, introduce
struct cpufreq_policy_data to hold copies of the members of
struct cpufreq_policy used by the existing ->verify() driver
callbacks and pass a pointer to a temporary structure of that
type to ->verify() (instead of passing a pointer to full struct
cpufreq_policy to it).
While at it, notice that intel_pstate and longrun don't really need
to verify the "policy" value in struct cpufreq_policy, so drop those
check from them to avoid copying "policy" into struct
cpufreq_policy_data (which allows it to be slightly smaller).
Also while at it fix up white space in a couple of places and make
cpufreq_set_policy() static (as it can be so).
Fixes: 3000ce3c52f8 ("cpufreq: Use per-policy frequency QoS")
Link: https://lore.kernel.org/linux-pm/CAMuHMdX6-jb1W8uC2_237m8ctCpsnGp=JCxqt8pCWVqNXHmkVg@mail.gmail.com
Reported-by: kbuild test robot <lkp@intel.com>
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: 5.4+ <stable@vger.kernel.org> # 5.4+
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
2020-01-26 22:40:11 +00:00
|
|
|
/*
|
|
|
|
* Used for passing new cpufreq policy data to the cpufreq driver's ->verify()
|
|
|
|
* callback for sanitization. That callback is only expected to modify the min
|
|
|
|
* and max values, if necessary, and specifically it must not update the
|
|
|
|
* frequency table.
|
|
|
|
*/
|
|
|
|
struct cpufreq_policy_data {
|
|
|
|
struct cpufreq_cpuinfo cpuinfo;
|
|
|
|
struct cpufreq_frequency_table *freq_table;
|
|
|
|
unsigned int cpu;
|
|
|
|
unsigned int min; /* in kHz */
|
|
|
|
unsigned int max; /* in kHz */
|
|
|
|
};
|
|
|
|
|
2019-04-29 09:33:58 +00:00
|
|
|
struct cpufreq_freqs {
|
|
|
|
struct cpufreq_policy *policy;
|
|
|
|
unsigned int old;
|
|
|
|
unsigned int new;
|
|
|
|
u8 flags; /* flags of cpufreq_driver, see below. */
|
|
|
|
};
|
|
|
|
|
2013-02-01 06:40:02 +00:00
|
|
|
/* Only for ACPI */
|
2006-06-26 04:34:43 +00:00
|
|
|
#define CPUFREQ_SHARED_TYPE_NONE (0) /* None */
|
|
|
|
#define CPUFREQ_SHARED_TYPE_HW (1) /* HW does needed coordination */
|
|
|
|
#define CPUFREQ_SHARED_TYPE_ALL (2) /* All dependent CPUs should set freq */
|
|
|
|
#define CPUFREQ_SHARED_TYPE_ANY (3) /* Freq can be set from any dependent CPU*/
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-10-08 08:56:11 +00:00
|
|
|
#ifdef CONFIG_CPU_FREQ
|
2015-09-16 00:17:49 +00:00
|
|
|
struct cpufreq_policy *cpufreq_cpu_get_raw(unsigned int cpu);
|
2013-08-06 17:23:04 +00:00
|
|
|
struct cpufreq_policy *cpufreq_cpu_get(unsigned int cpu);
|
2013-08-06 17:23:05 +00:00
|
|
|
void cpufreq_cpu_put(struct cpufreq_policy *policy);
|
2013-10-08 08:56:11 +00:00
|
|
|
#else
|
2015-09-16 00:17:49 +00:00
|
|
|
static inline struct cpufreq_policy *cpufreq_cpu_get_raw(unsigned int cpu)
|
|
|
|
{
|
|
|
|
return NULL;
|
|
|
|
}
|
2013-10-08 08:56:11 +00:00
|
|
|
static inline struct cpufreq_policy *cpufreq_cpu_get(unsigned int cpu)
|
|
|
|
{
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
static inline void cpufreq_cpu_put(struct cpufreq_policy *policy) { }
|
|
|
|
#endif
|
2013-08-06 17:23:04 +00:00
|
|
|
|
2019-03-26 11:19:52 +00:00
|
|
|
static inline bool policy_is_inactive(struct cpufreq_policy *policy)
|
|
|
|
{
|
|
|
|
return cpumask_empty(policy->cpus);
|
|
|
|
}
|
|
|
|
|
2013-01-31 09:44:40 +00:00
|
|
|
static inline bool policy_is_shared(struct cpufreq_policy *policy)
|
|
|
|
{
|
|
|
|
return cpumask_weight(policy->cpus) > 1;
|
|
|
|
}
|
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
#ifdef CONFIG_CPU_FREQ
|
|
|
|
unsigned int cpufreq_get(unsigned int cpu);
|
|
|
|
unsigned int cpufreq_quick_get(unsigned int cpu);
|
|
|
|
unsigned int cpufreq_quick_get_max(unsigned int cpu);
|
2020-03-05 09:06:25 +00:00
|
|
|
unsigned int cpufreq_get_hw_max_freq(unsigned int cpu);
|
2013-08-06 17:23:04 +00:00
|
|
|
void disable_cpufreq(void);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy);
|
2019-03-26 11:19:52 +00:00
|
|
|
|
|
|
|
struct cpufreq_policy *cpufreq_cpu_acquire(unsigned int cpu);
|
|
|
|
void cpufreq_cpu_release(struct cpufreq_policy *policy);
|
2013-08-06 17:23:04 +00:00
|
|
|
int cpufreq_get_policy(struct cpufreq_policy *policy, unsigned int cpu);
|
2019-07-04 07:36:22 +00:00
|
|
|
void refresh_frequency_limits(struct cpufreq_policy *policy);
|
2016-11-18 12:59:21 +00:00
|
|
|
void cpufreq_update_policy(unsigned int cpu);
|
2019-03-26 11:15:13 +00:00
|
|
|
void cpufreq_update_limits(unsigned int cpu);
|
2013-08-06 17:23:04 +00:00
|
|
|
bool have_governor_per_policy(void);
|
2020-09-01 20:55:47 +00:00
|
|
|
bool cpufreq_supports_freq_invariance(void);
|
2013-08-06 17:23:04 +00:00
|
|
|
struct kobject *get_governor_parent_kobj(struct cpufreq_policy *policy);
|
2016-03-30 01:47:49 +00:00
|
|
|
void cpufreq_enable_fast_switch(struct cpufreq_policy *policy);
|
2016-04-07 21:38:46 +00:00
|
|
|
void cpufreq_disable_fast_switch(struct cpufreq_policy *policy);
|
2013-08-06 17:23:04 +00:00
|
|
|
#else
|
|
|
|
static inline unsigned int cpufreq_get(unsigned int cpu)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2013-08-06 17:23:04 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
static inline unsigned int cpufreq_quick_get(unsigned int cpu)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
static inline unsigned int cpufreq_quick_get_max(unsigned int cpu)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
2020-03-05 09:06:25 +00:00
|
|
|
static inline unsigned int cpufreq_get_hw_max_freq(unsigned int cpu)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
2020-09-01 20:55:47 +00:00
|
|
|
static inline bool cpufreq_supports_freq_invariance(void)
|
|
|
|
{
|
|
|
|
return false;
|
|
|
|
}
|
2013-08-06 17:23:04 +00:00
|
|
|
static inline void disable_cpufreq(void) { }
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif
|
|
|
|
|
2016-05-31 20:14:44 +00:00
|
|
|
#ifdef CONFIG_CPU_FREQ_STAT
|
|
|
|
void cpufreq_stats_create_table(struct cpufreq_policy *policy);
|
|
|
|
void cpufreq_stats_free_table(struct cpufreq_policy *policy);
|
|
|
|
void cpufreq_stats_record_transition(struct cpufreq_policy *policy,
|
|
|
|
unsigned int new_freq);
|
|
|
|
#else
|
|
|
|
static inline void cpufreq_stats_create_table(struct cpufreq_policy *policy) { }
|
|
|
|
static inline void cpufreq_stats_free_table(struct cpufreq_policy *policy) { }
|
|
|
|
static inline void cpufreq_stats_record_transition(struct cpufreq_policy *policy,
|
|
|
|
unsigned int new_freq) { }
|
|
|
|
#endif /* CONFIG_CPU_FREQ_STAT */
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*********************************************************************
|
2013-08-06 17:23:04 +00:00
|
|
|
* CPUFREQ DRIVER INTERFACE *
|
2005-04-16 22:20:36 +00:00
|
|
|
*********************************************************************/
|
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
#define CPUFREQ_RELATION_L 0 /* lowest frequency at or above target */
|
|
|
|
#define CPUFREQ_RELATION_H 1 /* highest frequency below or at target */
|
2014-06-30 16:59:33 +00:00
|
|
|
#define CPUFREQ_RELATION_C 2 /* closest frequency to target */
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
struct freq_attr {
|
|
|
|
struct attribute attr;
|
|
|
|
ssize_t (*show)(struct cpufreq_policy *, char *);
|
|
|
|
ssize_t (*store)(struct cpufreq_policy *, const char *, size_t count);
|
2005-04-16 22:20:36 +00:00
|
|
|
};
|
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
#define cpufreq_freq_attr_ro(_name) \
|
|
|
|
static struct freq_attr _name = \
|
|
|
|
__ATTR(_name, 0444, show_##_name, NULL)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
#define cpufreq_freq_attr_ro_perm(_name, _perm) \
|
|
|
|
static struct freq_attr _name = \
|
|
|
|
__ATTR(_name, _perm, show_##_name, NULL)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
#define cpufreq_freq_attr_rw(_name) \
|
|
|
|
static struct freq_attr _name = \
|
|
|
|
__ATTR(_name, 0644, show_##_name, store_##_name)
|
|
|
|
|
2016-11-07 18:02:23 +00:00
|
|
|
#define cpufreq_freq_attr_wo(_name) \
|
|
|
|
static struct freq_attr _name = \
|
|
|
|
__ATTR(_name, 0200, NULL, store_##_name)
|
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
#define define_one_global_ro(_name) \
|
2019-01-25 07:23:07 +00:00
|
|
|
static struct kobj_attribute _name = \
|
2013-08-06 17:23:04 +00:00
|
|
|
__ATTR(_name, 0444, show_##_name, NULL)
|
|
|
|
|
|
|
|
#define define_one_global_rw(_name) \
|
2019-01-25 07:23:07 +00:00
|
|
|
static struct kobj_attribute _name = \
|
2013-08-06 17:23:04 +00:00
|
|
|
__ATTR(_name, 0644, show_##_name, store_##_name)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
|
|
|
|
struct cpufreq_driver {
|
2014-11-27 00:37:49 +00:00
|
|
|
char name[CPUFREQ_NAME_LEN];
|
2020-10-23 15:35:19 +00:00
|
|
|
u16 flags;
|
2014-11-27 00:37:49 +00:00
|
|
|
void *driver_data;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* needed by all drivers */
|
2014-11-27 00:37:49 +00:00
|
|
|
int (*init)(struct cpufreq_policy *policy);
|
cpufreq: Avoid creating excessively large stack frames
In the process of modifying a cpufreq policy, the cpufreq core makes
a copy of it including all of the internals which is stored on the
CPU stack. Because struct cpufreq_policy is relatively large, this
may cause the size of the stack frame to exceed the 2 KB limit and
so the GCC complains when -Wframe-larger-than= is used.
In fact, it is not necessary to copy the entire policy structure
in order to modify it, however.
First, because cpufreq_set_policy() obtains the min and max policy
limits from frequency QoS now, it is not necessary to pass the limits
to it from the callers. The only things that need to be passed to it
from there are the new governor pointer or (if there is a built-in
governor in the driver) the "policy" value representing the governor
choice. They both can be passed as individual arguments, though, so
make cpufreq_set_policy() take them this way and rework its callers
accordingly. This avoids making copies of cpufreq policies in the
callers of cpufreq_set_policy().
Second, cpufreq_set_policy() still needs to pass the new policy
data to the ->verify() callback of the cpufreq driver whose task
is to sanitize the min and max policy limits. It still does not
need to make a full copy of struct cpufreq_policy for this purpose,
but it needs to pass a few items from it to the driver in case they
are needed (different drivers have different needs in that respect
and all of them have to be covered). For this reason, introduce
struct cpufreq_policy_data to hold copies of the members of
struct cpufreq_policy used by the existing ->verify() driver
callbacks and pass a pointer to a temporary structure of that
type to ->verify() (instead of passing a pointer to full struct
cpufreq_policy to it).
While at it, notice that intel_pstate and longrun don't really need
to verify the "policy" value in struct cpufreq_policy, so drop those
check from them to avoid copying "policy" into struct
cpufreq_policy_data (which allows it to be slightly smaller).
Also while at it fix up white space in a couple of places and make
cpufreq_set_policy() static (as it can be so).
Fixes: 3000ce3c52f8 ("cpufreq: Use per-policy frequency QoS")
Link: https://lore.kernel.org/linux-pm/CAMuHMdX6-jb1W8uC2_237m8ctCpsnGp=JCxqt8pCWVqNXHmkVg@mail.gmail.com
Reported-by: kbuild test robot <lkp@intel.com>
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: 5.4+ <stable@vger.kernel.org> # 5.4+
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
2020-01-26 22:40:11 +00:00
|
|
|
int (*verify)(struct cpufreq_policy_data *policy);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* define one out of two */
|
2014-11-27 00:37:49 +00:00
|
|
|
int (*setpolicy)(struct cpufreq_policy *policy);
|
2014-06-02 17:19:28 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* On failure, should always restore frequency to policy->restore_freq
|
|
|
|
* (i.e. old freq).
|
|
|
|
*/
|
2014-11-27 00:37:49 +00:00
|
|
|
int (*target)(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq,
|
|
|
|
unsigned int relation); /* Deprecated */
|
|
|
|
int (*target_index)(struct cpufreq_policy *policy,
|
|
|
|
unsigned int index);
|
2016-03-30 01:47:49 +00:00
|
|
|
unsigned int (*fast_switch)(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq);
|
2016-07-13 20:25:25 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Caches and returns the lowest driver-supported frequency greater than
|
|
|
|
* or equal to the target frequency, subject to any driver limitations.
|
|
|
|
* Does not set the frequency. Only to be implemented for drivers with
|
|
|
|
* target().
|
|
|
|
*/
|
|
|
|
unsigned int (*resolve_freq)(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq);
|
|
|
|
|
2014-06-02 17:19:28 +00:00
|
|
|
/*
|
|
|
|
* Only for drivers with target_index() and CPUFREQ_ASYNC_NOTIFICATION
|
|
|
|
* unset.
|
|
|
|
*
|
|
|
|
* get_intermediate should return a stable intermediate frequency
|
|
|
|
* platform wants to switch to and target_intermediate() should set CPU
|
2020-05-13 14:18:54 +00:00
|
|
|
* to that frequency, before jumping to the frequency corresponding
|
2014-06-02 17:19:28 +00:00
|
|
|
* to 'index'. Core will take care of sending notifications and driver
|
|
|
|
* doesn't have to handle them in target_intermediate() or
|
|
|
|
* target_index().
|
|
|
|
*
|
|
|
|
* Drivers can return '0' from get_intermediate() in case they don't
|
|
|
|
* wish to switch to intermediate frequency for some target frequency.
|
|
|
|
* In that case core will directly call ->target_index().
|
|
|
|
*/
|
2014-11-27 00:37:49 +00:00
|
|
|
unsigned int (*get_intermediate)(struct cpufreq_policy *policy,
|
|
|
|
unsigned int index);
|
|
|
|
int (*target_intermediate)(struct cpufreq_policy *policy,
|
|
|
|
unsigned int index);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* should be defined, if possible */
|
2014-11-27 00:37:49 +00:00
|
|
|
unsigned int (*get)(unsigned int cpu);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2019-03-26 11:15:13 +00:00
|
|
|
/* Called to update policy limits on firmware notifications. */
|
|
|
|
void (*update_limits)(unsigned int cpu);
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* optional */
|
2014-11-27 00:37:49 +00:00
|
|
|
int (*bios_limit)(int cpu, unsigned int *limit);
|
2008-08-04 18:59:07 +00:00
|
|
|
|
2019-02-12 11:06:04 +00:00
|
|
|
int (*online)(struct cpufreq_policy *policy);
|
|
|
|
int (*offline)(struct cpufreq_policy *policy);
|
2014-11-27 00:37:49 +00:00
|
|
|
int (*exit)(struct cpufreq_policy *policy);
|
|
|
|
void (*stop_cpu)(struct cpufreq_policy *policy);
|
|
|
|
int (*suspend)(struct cpufreq_policy *policy);
|
|
|
|
int (*resume)(struct cpufreq_policy *policy);
|
2014-11-27 00:37:51 +00:00
|
|
|
|
|
|
|
/* Will be called after the driver is fully initialized */
|
|
|
|
void (*ready)(struct cpufreq_policy *policy);
|
|
|
|
|
2014-11-27 00:37:49 +00:00
|
|
|
struct freq_attr **attr;
|
2013-12-20 14:24:49 +00:00
|
|
|
|
|
|
|
/* platform specific boost support code */
|
2014-11-27 00:37:49 +00:00
|
|
|
bool boost_enabled;
|
2020-05-30 02:08:30 +00:00
|
|
|
int (*set_boost)(struct cpufreq_policy *policy, int state);
|
2005-04-16 22:20:36 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
/* flags */
|
2019-01-21 08:47:37 +00:00
|
|
|
|
|
|
|
/* driver isn't removed even if all ->init() calls failed */
|
|
|
|
#define CPUFREQ_STICKY BIT(0)
|
|
|
|
|
|
|
|
/* loops_per_jiffy or other kernel "constants" aren't affected by frequency transitions */
|
|
|
|
#define CPUFREQ_CONST_LOOPS BIT(1)
|
|
|
|
|
|
|
|
/* don't warn on suspend/resume speed mismatches */
|
|
|
|
#define CPUFREQ_PM_NO_WARN BIT(2)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-10-02 08:43:18 +00:00
|
|
|
/*
|
|
|
|
* This should be set by platforms having multiple clock-domains, i.e.
|
|
|
|
* supporting multiple policies. With this sysfs directories of governor would
|
|
|
|
* be created in cpu/cpu<num>/cpufreq/ directory and so they can use the same
|
|
|
|
* governor with different tunables for different clusters.
|
|
|
|
*/
|
2019-01-21 08:47:37 +00:00
|
|
|
#define CPUFREQ_HAVE_GOVERNOR_PER_POLICY BIT(3)
|
2013-10-02 08:43:18 +00:00
|
|
|
|
2013-10-29 13:26:06 +00:00
|
|
|
/*
|
|
|
|
* Driver will do POSTCHANGE notifications from outside of their ->target()
|
|
|
|
* routine and so must set cpufreq_driver->flags with this flag, so that core
|
|
|
|
* can handle them specially.
|
|
|
|
*/
|
2019-01-21 08:47:37 +00:00
|
|
|
#define CPUFREQ_ASYNC_NOTIFICATION BIT(4)
|
2013-10-29 13:26:06 +00:00
|
|
|
|
2013-12-03 05:50:45 +00:00
|
|
|
/*
|
|
|
|
* Set by drivers which want cpufreq core to check if CPU is running at a
|
|
|
|
* frequency present in freq-table exposed by the driver. For these drivers if
|
|
|
|
* CPU is found running at an out of table freq, we will try to set it to a freq
|
|
|
|
* from the table. And if that fails, we will stop further boot process by
|
|
|
|
* issuing a BUG_ON().
|
|
|
|
*/
|
2019-01-21 08:47:37 +00:00
|
|
|
#define CPUFREQ_NEED_INITIAL_FREQ_CHECK BIT(5)
|
2013-12-03 05:50:45 +00:00
|
|
|
|
2017-07-19 10:12:48 +00:00
|
|
|
/*
|
|
|
|
* Set by drivers to disallow use of governors with "dynamic_switching" flag
|
|
|
|
* set.
|
|
|
|
*/
|
2019-01-21 08:47:37 +00:00
|
|
|
#define CPUFREQ_NO_AUTO_DYNAMIC_SWITCHING BIT(6)
|
2017-07-19 10:12:48 +00:00
|
|
|
|
2019-01-30 05:22:01 +00:00
|
|
|
/*
|
|
|
|
* Set by drivers that want the core to automatically register the cpufreq
|
|
|
|
* driver as a thermal cooling device.
|
|
|
|
*/
|
|
|
|
#define CPUFREQ_IS_COOLING_DEV BIT(7)
|
|
|
|
|
2020-10-23 15:35:19 +00:00
|
|
|
/*
|
|
|
|
* Set by drivers that need to update internale upper and lower boundaries along
|
|
|
|
* with the target frequency and so the core and governors should also invoke
|
|
|
|
* the diver if the target frequency does not change, but the policy min or max
|
|
|
|
* may have changed.
|
|
|
|
*/
|
|
|
|
#define CPUFREQ_NEED_UPDATE_LIMITS BIT(8)
|
|
|
|
|
2007-02-26 22:55:48 +00:00
|
|
|
int cpufreq_register_driver(struct cpufreq_driver *driver_data);
|
|
|
|
int cpufreq_unregister_driver(struct cpufreq_driver *driver_data);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2020-10-23 15:35:46 +00:00
|
|
|
bool cpufreq_driver_test_flags(u16 flags);
|
2013-08-06 17:23:04 +00:00
|
|
|
const char *cpufreq_get_current_driver(void);
|
2014-10-19 09:30:27 +00:00
|
|
|
void *cpufreq_get_driver_data(void);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2019-06-25 11:32:41 +00:00
|
|
|
static inline int cpufreq_thermal_control_enabled(struct cpufreq_driver *drv)
|
|
|
|
{
|
|
|
|
return IS_ENABLED(CONFIG_CPU_THERMAL) &&
|
|
|
|
(drv->flags & CPUFREQ_IS_COOLING_DEV);
|
|
|
|
}
|
|
|
|
|
cpufreq: Avoid creating excessively large stack frames
In the process of modifying a cpufreq policy, the cpufreq core makes
a copy of it including all of the internals which is stored on the
CPU stack. Because struct cpufreq_policy is relatively large, this
may cause the size of the stack frame to exceed the 2 KB limit and
so the GCC complains when -Wframe-larger-than= is used.
In fact, it is not necessary to copy the entire policy structure
in order to modify it, however.
First, because cpufreq_set_policy() obtains the min and max policy
limits from frequency QoS now, it is not necessary to pass the limits
to it from the callers. The only things that need to be passed to it
from there are the new governor pointer or (if there is a built-in
governor in the driver) the "policy" value representing the governor
choice. They both can be passed as individual arguments, though, so
make cpufreq_set_policy() take them this way and rework its callers
accordingly. This avoids making copies of cpufreq policies in the
callers of cpufreq_set_policy().
Second, cpufreq_set_policy() still needs to pass the new policy
data to the ->verify() callback of the cpufreq driver whose task
is to sanitize the min and max policy limits. It still does not
need to make a full copy of struct cpufreq_policy for this purpose,
but it needs to pass a few items from it to the driver in case they
are needed (different drivers have different needs in that respect
and all of them have to be covered). For this reason, introduce
struct cpufreq_policy_data to hold copies of the members of
struct cpufreq_policy used by the existing ->verify() driver
callbacks and pass a pointer to a temporary structure of that
type to ->verify() (instead of passing a pointer to full struct
cpufreq_policy to it).
While at it, notice that intel_pstate and longrun don't really need
to verify the "policy" value in struct cpufreq_policy, so drop those
check from them to avoid copying "policy" into struct
cpufreq_policy_data (which allows it to be slightly smaller).
Also while at it fix up white space in a couple of places and make
cpufreq_set_policy() static (as it can be so).
Fixes: 3000ce3c52f8 ("cpufreq: Use per-policy frequency QoS")
Link: https://lore.kernel.org/linux-pm/CAMuHMdX6-jb1W8uC2_237m8ctCpsnGp=JCxqt8pCWVqNXHmkVg@mail.gmail.com
Reported-by: kbuild test robot <lkp@intel.com>
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: 5.4+ <stable@vger.kernel.org> # 5.4+
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
2020-01-26 22:40:11 +00:00
|
|
|
static inline void cpufreq_verify_within_limits(struct cpufreq_policy_data *policy,
|
|
|
|
unsigned int min,
|
|
|
|
unsigned int max)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
if (policy->min < min)
|
|
|
|
policy->min = min;
|
|
|
|
if (policy->max < min)
|
|
|
|
policy->max = min;
|
|
|
|
if (policy->min > max)
|
|
|
|
policy->min = max;
|
|
|
|
if (policy->max > max)
|
|
|
|
policy->max = max;
|
|
|
|
if (policy->min > policy->max)
|
|
|
|
policy->min = policy->max;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2013-10-02 08:43:19 +00:00
|
|
|
static inline void
|
cpufreq: Avoid creating excessively large stack frames
In the process of modifying a cpufreq policy, the cpufreq core makes
a copy of it including all of the internals which is stored on the
CPU stack. Because struct cpufreq_policy is relatively large, this
may cause the size of the stack frame to exceed the 2 KB limit and
so the GCC complains when -Wframe-larger-than= is used.
In fact, it is not necessary to copy the entire policy structure
in order to modify it, however.
First, because cpufreq_set_policy() obtains the min and max policy
limits from frequency QoS now, it is not necessary to pass the limits
to it from the callers. The only things that need to be passed to it
from there are the new governor pointer or (if there is a built-in
governor in the driver) the "policy" value representing the governor
choice. They both can be passed as individual arguments, though, so
make cpufreq_set_policy() take them this way and rework its callers
accordingly. This avoids making copies of cpufreq policies in the
callers of cpufreq_set_policy().
Second, cpufreq_set_policy() still needs to pass the new policy
data to the ->verify() callback of the cpufreq driver whose task
is to sanitize the min and max policy limits. It still does not
need to make a full copy of struct cpufreq_policy for this purpose,
but it needs to pass a few items from it to the driver in case they
are needed (different drivers have different needs in that respect
and all of them have to be covered). For this reason, introduce
struct cpufreq_policy_data to hold copies of the members of
struct cpufreq_policy used by the existing ->verify() driver
callbacks and pass a pointer to a temporary structure of that
type to ->verify() (instead of passing a pointer to full struct
cpufreq_policy to it).
While at it, notice that intel_pstate and longrun don't really need
to verify the "policy" value in struct cpufreq_policy, so drop those
check from them to avoid copying "policy" into struct
cpufreq_policy_data (which allows it to be slightly smaller).
Also while at it fix up white space in a couple of places and make
cpufreq_set_policy() static (as it can be so).
Fixes: 3000ce3c52f8 ("cpufreq: Use per-policy frequency QoS")
Link: https://lore.kernel.org/linux-pm/CAMuHMdX6-jb1W8uC2_237m8ctCpsnGp=JCxqt8pCWVqNXHmkVg@mail.gmail.com
Reported-by: kbuild test robot <lkp@intel.com>
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: 5.4+ <stable@vger.kernel.org> # 5.4+
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
2020-01-26 22:40:11 +00:00
|
|
|
cpufreq_verify_within_cpu_limits(struct cpufreq_policy_data *policy)
|
2013-10-02 08:43:19 +00:00
|
|
|
{
|
|
|
|
cpufreq_verify_within_limits(policy, policy->cpuinfo.min_freq,
|
cpufreq: Avoid creating excessively large stack frames
In the process of modifying a cpufreq policy, the cpufreq core makes
a copy of it including all of the internals which is stored on the
CPU stack. Because struct cpufreq_policy is relatively large, this
may cause the size of the stack frame to exceed the 2 KB limit and
so the GCC complains when -Wframe-larger-than= is used.
In fact, it is not necessary to copy the entire policy structure
in order to modify it, however.
First, because cpufreq_set_policy() obtains the min and max policy
limits from frequency QoS now, it is not necessary to pass the limits
to it from the callers. The only things that need to be passed to it
from there are the new governor pointer or (if there is a built-in
governor in the driver) the "policy" value representing the governor
choice. They both can be passed as individual arguments, though, so
make cpufreq_set_policy() take them this way and rework its callers
accordingly. This avoids making copies of cpufreq policies in the
callers of cpufreq_set_policy().
Second, cpufreq_set_policy() still needs to pass the new policy
data to the ->verify() callback of the cpufreq driver whose task
is to sanitize the min and max policy limits. It still does not
need to make a full copy of struct cpufreq_policy for this purpose,
but it needs to pass a few items from it to the driver in case they
are needed (different drivers have different needs in that respect
and all of them have to be covered). For this reason, introduce
struct cpufreq_policy_data to hold copies of the members of
struct cpufreq_policy used by the existing ->verify() driver
callbacks and pass a pointer to a temporary structure of that
type to ->verify() (instead of passing a pointer to full struct
cpufreq_policy to it).
While at it, notice that intel_pstate and longrun don't really need
to verify the "policy" value in struct cpufreq_policy, so drop those
check from them to avoid copying "policy" into struct
cpufreq_policy_data (which allows it to be slightly smaller).
Also while at it fix up white space in a couple of places and make
cpufreq_set_policy() static (as it can be so).
Fixes: 3000ce3c52f8 ("cpufreq: Use per-policy frequency QoS")
Link: https://lore.kernel.org/linux-pm/CAMuHMdX6-jb1W8uC2_237m8ctCpsnGp=JCxqt8pCWVqNXHmkVg@mail.gmail.com
Reported-by: kbuild test robot <lkp@intel.com>
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: 5.4+ <stable@vger.kernel.org> # 5.4+
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
2020-01-26 22:40:11 +00:00
|
|
|
policy->cpuinfo.max_freq);
|
2013-10-02 08:43:19 +00:00
|
|
|
}
|
|
|
|
|
2014-03-04 03:00:26 +00:00
|
|
|
#ifdef CONFIG_CPU_FREQ
|
|
|
|
void cpufreq_suspend(void);
|
|
|
|
void cpufreq_resume(void);
|
2014-03-04 03:00:27 +00:00
|
|
|
int cpufreq_generic_suspend(struct cpufreq_policy *policy);
|
2014-03-04 03:00:26 +00:00
|
|
|
#else
|
|
|
|
static inline void cpufreq_suspend(void) {}
|
|
|
|
static inline void cpufreq_resume(void) {}
|
|
|
|
#endif
|
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
/*********************************************************************
|
|
|
|
* CPUFREQ NOTIFIER INTERFACE *
|
|
|
|
*********************************************************************/
|
2010-03-31 19:56:46 +00:00
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
#define CPUFREQ_TRANSITION_NOTIFIER (0)
|
|
|
|
#define CPUFREQ_POLICY_NOTIFIER (1)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
/* Transition notifiers */
|
|
|
|
#define CPUFREQ_PRECHANGE (0)
|
|
|
|
#define CPUFREQ_POSTCHANGE (1)
|
2010-03-31 19:56:46 +00:00
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
/* Policy Notifiers */
|
2019-07-23 06:14:09 +00:00
|
|
|
#define CPUFREQ_CREATE_POLICY (0)
|
|
|
|
#define CPUFREQ_REMOVE_POLICY (1)
|
2010-03-31 19:56:46 +00:00
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
#ifdef CONFIG_CPU_FREQ
|
|
|
|
int cpufreq_register_notifier(struct notifier_block *nb, unsigned int list);
|
|
|
|
int cpufreq_unregister_notifier(struct notifier_block *nb, unsigned int list);
|
2010-03-31 19:56:46 +00:00
|
|
|
|
cpufreq: Make sure frequency transitions are serialized
Whenever we change the frequency of a CPU, we call the PRECHANGE and POSTCHANGE
notifiers. They must be serialized, i.e. PRECHANGE and POSTCHANGE notifiers
should strictly alternate, thereby preventing two different sets of PRECHANGE or
POSTCHANGE notifiers from interleaving arbitrarily.
The following examples illustrate why this is important:
Scenario 1:
-----------
A thread reading the value of cpuinfo_cur_freq, will call
__cpufreq_cpu_get()->cpufreq_out_of_sync()->cpufreq_notify_transition()
The ondemand governor can decide to change the frequency of the CPU at the same
time and hence it can end up sending the notifications via ->target().
If the notifiers are not serialized, the following sequence can occur:
- PRECHANGE Notification for freq A (from cpuinfo_cur_freq)
- PRECHANGE Notification for freq B (from target())
- Freq changed by target() to B
- POSTCHANGE Notification for freq B
- POSTCHANGE Notification for freq A
We can see from the above that the last POSTCHANGE Notification happens for freq
A but the hardware is set to run at freq B.
Where would we break then?: adjust_jiffies() in cpufreq.c & cpufreq_callback()
in arch/arm/kernel/smp.c (which also adjusts the jiffies). All the
loops_per_jiffy calculations will get messed up.
Scenario 2:
-----------
The governor calls __cpufreq_driver_target() to change the frequency. At the
same time, if we change scaling_{min|max}_freq from sysfs, it will end up
calling the governor's CPUFREQ_GOV_LIMITS notification, which will also call
__cpufreq_driver_target(). And hence we end up issuing concurrent calls to
->target().
Typically, platforms have the following logic in their ->target() routines:
(Eg: cpufreq-cpu0, omap, exynos, etc)
A. If new freq is more than old: Increase voltage
B. Change freq
C. If new freq is less than old: decrease voltage
Now, if the two concurrent calls to ->target() are X and Y, where X is trying to
increase the freq and Y is trying to decrease it, we get the following race
condition:
X.A: voltage gets increased for larger freq
Y.A: nothing happens
Y.B: freq gets decreased
Y.C: voltage gets decreased
X.B: freq gets increased
X.C: nothing happens
Thus we can end up setting a freq which is not supported by the voltage we have
set. That will probably make the clock to the CPU unstable and the system might
not work properly anymore.
This patch introduces a set of synchronization primitives to serialize frequency
transitions, which are to be used as shown below:
cpufreq_freq_transition_begin();
//Perform the frequency change
cpufreq_freq_transition_end();
The _begin() call sends the PRECHANGE notification whereas the _end() call sends
the POSTCHANGE notification. Also, all the necessary synchronization is handled
within these calls. In particular, even drivers which set the ASYNC_NOTIFICATION
flag can also use these APIs for performing frequency transitions (ie., you can
call _begin() from one task, and call the corresponding _end() from a different
task).
The actual synchronization underneath is not that complicated:
The key challenge is to allow drivers to begin the transition from one thread
and end it in a completely different thread (this is to enable drivers that do
asynchronous POSTCHANGE notification from bottom-halves, to also use the same
interface).
To achieve this, a 'transition_ongoing' flag, a 'transition_lock' spinlock and a
wait-queue are added per-policy. The flag and the wait-queue are used in
conjunction to create an "uninterrupted flow" from _begin() to _end(). The
spinlock is used to ensure that only one such "flow" is in flight at any given
time. Put together, this provides us all the necessary synchronization.
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
2014-03-24 08:05:44 +00:00
|
|
|
void cpufreq_freq_transition_begin(struct cpufreq_policy *policy,
|
|
|
|
struct cpufreq_freqs *freqs);
|
|
|
|
void cpufreq_freq_transition_end(struct cpufreq_policy *policy,
|
|
|
|
struct cpufreq_freqs *freqs, int transition_failed);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
#else /* CONFIG_CPU_FREQ */
|
|
|
|
static inline int cpufreq_register_notifier(struct notifier_block *nb,
|
|
|
|
unsigned int list)
|
2009-10-26 23:49:29 +00:00
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
2013-08-06 17:23:04 +00:00
|
|
|
static inline int cpufreq_unregister_notifier(struct notifier_block *nb,
|
|
|
|
unsigned int list)
|
2005-12-02 18:43:20 +00:00
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
2013-08-06 17:23:04 +00:00
|
|
|
#endif /* !CONFIG_CPU_FREQ */
|
|
|
|
|
|
|
|
/**
|
|
|
|
* cpufreq_scale - "old * mult / div" calculation for large values (32-bit-arch
|
|
|
|
* safe)
|
|
|
|
* @old: old value
|
|
|
|
* @div: divisor
|
|
|
|
* @mult: multiplier
|
|
|
|
*
|
|
|
|
*
|
|
|
|
* new = old * mult / div
|
|
|
|
*/
|
|
|
|
static inline unsigned long cpufreq_scale(unsigned long old, u_int div,
|
|
|
|
u_int mult)
|
2011-06-28 17:59:12 +00:00
|
|
|
{
|
2013-08-06 17:23:04 +00:00
|
|
|
#if BITS_PER_LONG == 32
|
|
|
|
u64 result = ((u64) old) * ((u64) mult);
|
|
|
|
do_div(result, div);
|
|
|
|
return (unsigned long) result;
|
|
|
|
|
|
|
|
#elif BITS_PER_LONG == 64
|
|
|
|
unsigned long result = old * ((u64) mult);
|
|
|
|
result /= div;
|
|
|
|
return result;
|
2005-12-02 18:43:20 +00:00
|
|
|
#endif
|
2013-08-06 17:23:04 +00:00
|
|
|
}
|
2005-12-02 18:43:20 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*********************************************************************
|
2013-08-06 17:23:04 +00:00
|
|
|
* CPUFREQ GOVERNORS *
|
2005-04-16 22:20:36 +00:00
|
|
|
*********************************************************************/
|
|
|
|
|
cpufreq: Avoid creating excessively large stack frames
In the process of modifying a cpufreq policy, the cpufreq core makes
a copy of it including all of the internals which is stored on the
CPU stack. Because struct cpufreq_policy is relatively large, this
may cause the size of the stack frame to exceed the 2 KB limit and
so the GCC complains when -Wframe-larger-than= is used.
In fact, it is not necessary to copy the entire policy structure
in order to modify it, however.
First, because cpufreq_set_policy() obtains the min and max policy
limits from frequency QoS now, it is not necessary to pass the limits
to it from the callers. The only things that need to be passed to it
from there are the new governor pointer or (if there is a built-in
governor in the driver) the "policy" value representing the governor
choice. They both can be passed as individual arguments, though, so
make cpufreq_set_policy() take them this way and rework its callers
accordingly. This avoids making copies of cpufreq policies in the
callers of cpufreq_set_policy().
Second, cpufreq_set_policy() still needs to pass the new policy
data to the ->verify() callback of the cpufreq driver whose task
is to sanitize the min and max policy limits. It still does not
need to make a full copy of struct cpufreq_policy for this purpose,
but it needs to pass a few items from it to the driver in case they
are needed (different drivers have different needs in that respect
and all of them have to be covered). For this reason, introduce
struct cpufreq_policy_data to hold copies of the members of
struct cpufreq_policy used by the existing ->verify() driver
callbacks and pass a pointer to a temporary structure of that
type to ->verify() (instead of passing a pointer to full struct
cpufreq_policy to it).
While at it, notice that intel_pstate and longrun don't really need
to verify the "policy" value in struct cpufreq_policy, so drop those
check from them to avoid copying "policy" into struct
cpufreq_policy_data (which allows it to be slightly smaller).
Also while at it fix up white space in a couple of places and make
cpufreq_set_policy() static (as it can be so).
Fixes: 3000ce3c52f8 ("cpufreq: Use per-policy frequency QoS")
Link: https://lore.kernel.org/linux-pm/CAMuHMdX6-jb1W8uC2_237m8ctCpsnGp=JCxqt8pCWVqNXHmkVg@mail.gmail.com
Reported-by: kbuild test robot <lkp@intel.com>
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: 5.4+ <stable@vger.kernel.org> # 5.4+
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
2020-01-26 22:40:11 +00:00
|
|
|
#define CPUFREQ_POLICY_UNKNOWN (0)
|
2013-08-06 17:23:04 +00:00
|
|
|
/*
|
|
|
|
* If (cpufreq_driver->target) exists, the ->governor decides what frequency
|
|
|
|
* within the limits is used. If (cpufreq_driver->setpolicy> exists, these
|
|
|
|
* two generic policies are available:
|
|
|
|
*/
|
|
|
|
#define CPUFREQ_POLICY_POWERSAVE (1)
|
|
|
|
#define CPUFREQ_POLICY_PERFORMANCE (2)
|
|
|
|
|
2016-03-22 01:51:56 +00:00
|
|
|
/*
|
|
|
|
* The polling frequency depends on the capability of the processor. Default
|
|
|
|
* polling frequency is 1000 times the transition latency of the processor. The
|
|
|
|
* ondemand governor will work on any processor with transition latency <= 10ms,
|
|
|
|
* using appropriate sampling rate.
|
|
|
|
*/
|
|
|
|
#define LATENCY_MULTIPLIER (1000)
|
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
struct cpufreq_governor {
|
|
|
|
char name[CPUFREQ_NAME_LEN];
|
2016-06-02 21:24:15 +00:00
|
|
|
int (*init)(struct cpufreq_policy *policy);
|
|
|
|
void (*exit)(struct cpufreq_policy *policy);
|
|
|
|
int (*start)(struct cpufreq_policy *policy);
|
|
|
|
void (*stop)(struct cpufreq_policy *policy);
|
|
|
|
void (*limits)(struct cpufreq_policy *policy);
|
2013-08-06 17:23:04 +00:00
|
|
|
ssize_t (*show_setspeed) (struct cpufreq_policy *policy,
|
|
|
|
char *buf);
|
|
|
|
int (*store_setspeed) (struct cpufreq_policy *policy,
|
|
|
|
unsigned int freq);
|
|
|
|
struct list_head governor_list;
|
|
|
|
struct module *owner;
|
2020-11-10 17:25:57 +00:00
|
|
|
u8 flags;
|
2013-08-06 17:23:04 +00:00
|
|
|
};
|
|
|
|
|
2020-11-10 17:25:57 +00:00
|
|
|
/* Governor flags */
|
|
|
|
|
|
|
|
/* For governors which change frequency dynamically by themselves */
|
|
|
|
#define CPUFREQ_GOV_DYNAMIC_SWITCHING BIT(0)
|
|
|
|
|
2020-11-10 17:26:10 +00:00
|
|
|
/* For governors wanting the target frequency to be set exactly */
|
|
|
|
#define CPUFREQ_GOV_STRICT_TARGET BIT(1)
|
|
|
|
|
2020-11-10 17:25:57 +00:00
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
/* Pass a target to the cpufreq driver */
|
2016-03-30 01:47:49 +00:00
|
|
|
unsigned int cpufreq_driver_fast_switch(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq);
|
2013-08-06 17:23:04 +00:00
|
|
|
int cpufreq_driver_target(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq,
|
|
|
|
unsigned int relation);
|
|
|
|
int __cpufreq_driver_target(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq,
|
|
|
|
unsigned int relation);
|
2016-07-13 20:25:25 +00:00
|
|
|
unsigned int cpufreq_driver_resolve_freq(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq);
|
2017-07-19 10:12:42 +00:00
|
|
|
unsigned int cpufreq_policy_transition_delay_us(struct cpufreq_policy *policy);
|
2013-08-06 17:23:04 +00:00
|
|
|
int cpufreq_register_governor(struct cpufreq_governor *governor);
|
|
|
|
void cpufreq_unregister_governor(struct cpufreq_governor *governor);
|
cpufreq: intel_pstate: Implement passive mode with HWP enabled
Allow intel_pstate to work in the passive mode with HWP enabled and
make it set the HWP minimum performance limit (HWP floor) to the
P-state value given by the target frequency supplied by the cpufreq
governor, so as to prevent the HWP algorithm and the CPU scheduler
from working against each other, at least when the schedutil governor
is in use, and update the intel_pstate documentation accordingly.
Among other things, this allows utilization clamps to be taken
into account, at least to a certain extent, when intel_pstate is
in use and makes it more likely that sufficient capacity for
deadline tasks will be provided.
After this change, the resulting behavior of an HWP system with
intel_pstate in the passive mode should be close to the behavior
of the analogous non-HWP system with intel_pstate in the passive
mode, except that the HWP algorithm is generally allowed to make the
CPU run at a frequency above the floor P-state set by intel_pstate in
the entire available range of P-states, while without HWP a CPU can
run in a P-state above the requested one if the latter falls into the
range of turbo P-states (referred to as the turbo range) or if the
P-states of all CPUs in one package are coordinated with each other
at the hardware level.
[Note that in principle the HWP floor may not be taken into account
by the processor if it falls into the turbo range, in which case the
processor has a license to choose any P-state, either below or above
the HWP floor, just like a non-HWP processor in the case when the
target P-state falls into the turbo range.]
With this change applied, intel_pstate in the passive mode assumes
complete control over the HWP request MSR and concurrent changes of
that MSR (eg. via the direct MSR access interface) are overridden by
it.
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
Reviewed-by: Francisco Jerez <currojerez@riseup.net>
2020-08-06 12:03:55 +00:00
|
|
|
int cpufreq_start_governor(struct cpufreq_policy *policy);
|
|
|
|
void cpufreq_stop_governor(struct cpufreq_policy *policy);
|
2013-08-06 17:23:04 +00:00
|
|
|
|
2020-06-29 08:24:59 +00:00
|
|
|
#define cpufreq_governor_init(__governor) \
|
|
|
|
static int __init __governor##_init(void) \
|
|
|
|
{ \
|
|
|
|
return cpufreq_register_governor(&__governor); \
|
|
|
|
} \
|
|
|
|
core_initcall(__governor##_init)
|
|
|
|
|
|
|
|
#define cpufreq_governor_exit(__governor) \
|
|
|
|
static void __exit __governor##_exit(void) \
|
|
|
|
{ \
|
|
|
|
return cpufreq_unregister_governor(&__governor); \
|
|
|
|
} \
|
|
|
|
module_exit(__governor##_exit)
|
|
|
|
|
2016-02-05 01:37:42 +00:00
|
|
|
struct cpufreq_governor *cpufreq_default_governor(void);
|
|
|
|
struct cpufreq_governor *cpufreq_fallback_governor(void);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2016-05-18 12:25:31 +00:00
|
|
|
static inline void cpufreq_policy_apply_limits(struct cpufreq_policy *policy)
|
|
|
|
{
|
|
|
|
if (policy->max < policy->cur)
|
|
|
|
__cpufreq_driver_target(policy, policy->max, CPUFREQ_RELATION_H);
|
|
|
|
else if (policy->min > policy->cur)
|
|
|
|
__cpufreq_driver_target(policy, policy->min, CPUFREQ_RELATION_L);
|
|
|
|
}
|
|
|
|
|
2016-03-22 01:50:45 +00:00
|
|
|
/* Governor attribute set */
|
|
|
|
struct gov_attr_set {
|
|
|
|
struct kobject kobj;
|
|
|
|
struct list_head policy_list;
|
|
|
|
struct mutex update_lock;
|
|
|
|
int usage_count;
|
|
|
|
};
|
|
|
|
|
|
|
|
/* sysfs ops for cpufreq governors */
|
|
|
|
extern const struct sysfs_ops governor_sysfs_ops;
|
|
|
|
|
|
|
|
void gov_attr_set_init(struct gov_attr_set *attr_set, struct list_head *list_node);
|
|
|
|
void gov_attr_set_get(struct gov_attr_set *attr_set, struct list_head *list_node);
|
|
|
|
unsigned int gov_attr_set_put(struct gov_attr_set *attr_set, struct list_head *list_node);
|
|
|
|
|
|
|
|
/* Governor sysfs attribute */
|
|
|
|
struct governor_attr {
|
|
|
|
struct attribute attr;
|
|
|
|
ssize_t (*show)(struct gov_attr_set *attr_set, char *buf);
|
|
|
|
ssize_t (*store)(struct gov_attr_set *attr_set, const char *buf,
|
|
|
|
size_t count);
|
|
|
|
};
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*********************************************************************
|
|
|
|
* FREQUENCY TABLE HELPERS *
|
|
|
|
*********************************************************************/
|
|
|
|
|
2014-03-28 13:41:47 +00:00
|
|
|
/* Special Values of .frequency field */
|
2014-06-27 21:09:39 +00:00
|
|
|
#define CPUFREQ_ENTRY_INVALID ~0u
|
|
|
|
#define CPUFREQ_TABLE_END ~1u
|
2014-03-28 13:41:47 +00:00
|
|
|
/* Special Values of .flags field */
|
|
|
|
#define CPUFREQ_BOOST_FREQ (1 << 0)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
struct cpufreq_frequency_table {
|
2014-03-28 13:41:47 +00:00
|
|
|
unsigned int flags;
|
2013-03-30 10:55:15 +00:00
|
|
|
unsigned int driver_data; /* driver specific data, not used by core */
|
2005-04-16 22:20:36 +00:00
|
|
|
unsigned int frequency; /* kHz - doesn't need to be in ascending
|
|
|
|
* order */
|
|
|
|
};
|
|
|
|
|
2014-05-05 13:33:50 +00:00
|
|
|
#if defined(CONFIG_CPU_FREQ) && defined(CONFIG_PM_OPP)
|
|
|
|
int dev_pm_opp_init_cpufreq_table(struct device *dev,
|
|
|
|
struct cpufreq_frequency_table **table);
|
|
|
|
void dev_pm_opp_free_cpufreq_table(struct device *dev,
|
|
|
|
struct cpufreq_frequency_table **table);
|
|
|
|
#else
|
|
|
|
static inline int dev_pm_opp_init_cpufreq_table(struct device *dev,
|
|
|
|
struct cpufreq_frequency_table
|
|
|
|
**table)
|
|
|
|
{
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void dev_pm_opp_free_cpufreq_table(struct device *dev,
|
|
|
|
struct cpufreq_frequency_table
|
|
|
|
**table)
|
|
|
|
{
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2014-04-25 20:15:23 +00:00
|
|
|
/*
|
|
|
|
* cpufreq_for_each_entry - iterate over a cpufreq_frequency_table
|
|
|
|
* @pos: the cpufreq_frequency_table * to use as a loop cursor.
|
|
|
|
* @table: the cpufreq_frequency_table * to iterate over.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#define cpufreq_for_each_entry(pos, table) \
|
|
|
|
for (pos = table; pos->frequency != CPUFREQ_TABLE_END; pos++)
|
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
/*
|
|
|
|
* cpufreq_for_each_entry_idx - iterate over a cpufreq_frequency_table
|
|
|
|
* with index
|
|
|
|
* @pos: the cpufreq_frequency_table * to use as a loop cursor.
|
|
|
|
* @table: the cpufreq_frequency_table * to iterate over.
|
|
|
|
* @idx: the table entry currently being processed
|
|
|
|
*/
|
|
|
|
|
|
|
|
#define cpufreq_for_each_entry_idx(pos, table, idx) \
|
|
|
|
for (pos = table, idx = 0; pos->frequency != CPUFREQ_TABLE_END; \
|
|
|
|
pos++, idx++)
|
|
|
|
|
2014-04-25 20:15:23 +00:00
|
|
|
/*
|
|
|
|
* cpufreq_for_each_valid_entry - iterate over a cpufreq_frequency_table
|
|
|
|
* excluding CPUFREQ_ENTRY_INVALID frequencies.
|
|
|
|
* @pos: the cpufreq_frequency_table * to use as a loop cursor.
|
|
|
|
* @table: the cpufreq_frequency_table * to iterate over.
|
|
|
|
*/
|
|
|
|
|
2016-02-25 23:22:57 +00:00
|
|
|
#define cpufreq_for_each_valid_entry(pos, table) \
|
|
|
|
for (pos = table; pos->frequency != CPUFREQ_TABLE_END; pos++) \
|
|
|
|
if (pos->frequency == CPUFREQ_ENTRY_INVALID) \
|
|
|
|
continue; \
|
|
|
|
else
|
2014-04-25 20:15:23 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
/*
|
|
|
|
* cpufreq_for_each_valid_entry_idx - iterate with index over a cpufreq
|
|
|
|
* frequency_table excluding CPUFREQ_ENTRY_INVALID frequencies.
|
|
|
|
* @pos: the cpufreq_frequency_table * to use as a loop cursor.
|
|
|
|
* @table: the cpufreq_frequency_table * to iterate over.
|
|
|
|
* @idx: the table entry currently being processed
|
|
|
|
*/
|
|
|
|
|
|
|
|
#define cpufreq_for_each_valid_entry_idx(pos, table, idx) \
|
|
|
|
cpufreq_for_each_entry_idx(pos, table, idx) \
|
|
|
|
if (pos->frequency == CPUFREQ_ENTRY_INVALID) \
|
|
|
|
continue; \
|
|
|
|
else
|
|
|
|
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
int cpufreq_frequency_table_cpuinfo(struct cpufreq_policy *policy,
|
|
|
|
struct cpufreq_frequency_table *table);
|
|
|
|
|
cpufreq: Avoid creating excessively large stack frames
In the process of modifying a cpufreq policy, the cpufreq core makes
a copy of it including all of the internals which is stored on the
CPU stack. Because struct cpufreq_policy is relatively large, this
may cause the size of the stack frame to exceed the 2 KB limit and
so the GCC complains when -Wframe-larger-than= is used.
In fact, it is not necessary to copy the entire policy structure
in order to modify it, however.
First, because cpufreq_set_policy() obtains the min and max policy
limits from frequency QoS now, it is not necessary to pass the limits
to it from the callers. The only things that need to be passed to it
from there are the new governor pointer or (if there is a built-in
governor in the driver) the "policy" value representing the governor
choice. They both can be passed as individual arguments, though, so
make cpufreq_set_policy() take them this way and rework its callers
accordingly. This avoids making copies of cpufreq policies in the
callers of cpufreq_set_policy().
Second, cpufreq_set_policy() still needs to pass the new policy
data to the ->verify() callback of the cpufreq driver whose task
is to sanitize the min and max policy limits. It still does not
need to make a full copy of struct cpufreq_policy for this purpose,
but it needs to pass a few items from it to the driver in case they
are needed (different drivers have different needs in that respect
and all of them have to be covered). For this reason, introduce
struct cpufreq_policy_data to hold copies of the members of
struct cpufreq_policy used by the existing ->verify() driver
callbacks and pass a pointer to a temporary structure of that
type to ->verify() (instead of passing a pointer to full struct
cpufreq_policy to it).
While at it, notice that intel_pstate and longrun don't really need
to verify the "policy" value in struct cpufreq_policy, so drop those
check from them to avoid copying "policy" into struct
cpufreq_policy_data (which allows it to be slightly smaller).
Also while at it fix up white space in a couple of places and make
cpufreq_set_policy() static (as it can be so).
Fixes: 3000ce3c52f8 ("cpufreq: Use per-policy frequency QoS")
Link: https://lore.kernel.org/linux-pm/CAMuHMdX6-jb1W8uC2_237m8ctCpsnGp=JCxqt8pCWVqNXHmkVg@mail.gmail.com
Reported-by: kbuild test robot <lkp@intel.com>
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: 5.4+ <stable@vger.kernel.org> # 5.4+
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
2020-01-26 22:40:11 +00:00
|
|
|
int cpufreq_frequency_table_verify(struct cpufreq_policy_data *policy,
|
2005-04-16 22:20:36 +00:00
|
|
|
struct cpufreq_frequency_table *table);
|
cpufreq: Avoid creating excessively large stack frames
In the process of modifying a cpufreq policy, the cpufreq core makes
a copy of it including all of the internals which is stored on the
CPU stack. Because struct cpufreq_policy is relatively large, this
may cause the size of the stack frame to exceed the 2 KB limit and
so the GCC complains when -Wframe-larger-than= is used.
In fact, it is not necessary to copy the entire policy structure
in order to modify it, however.
First, because cpufreq_set_policy() obtains the min and max policy
limits from frequency QoS now, it is not necessary to pass the limits
to it from the callers. The only things that need to be passed to it
from there are the new governor pointer or (if there is a built-in
governor in the driver) the "policy" value representing the governor
choice. They both can be passed as individual arguments, though, so
make cpufreq_set_policy() take them this way and rework its callers
accordingly. This avoids making copies of cpufreq policies in the
callers of cpufreq_set_policy().
Second, cpufreq_set_policy() still needs to pass the new policy
data to the ->verify() callback of the cpufreq driver whose task
is to sanitize the min and max policy limits. It still does not
need to make a full copy of struct cpufreq_policy for this purpose,
but it needs to pass a few items from it to the driver in case they
are needed (different drivers have different needs in that respect
and all of them have to be covered). For this reason, introduce
struct cpufreq_policy_data to hold copies of the members of
struct cpufreq_policy used by the existing ->verify() driver
callbacks and pass a pointer to a temporary structure of that
type to ->verify() (instead of passing a pointer to full struct
cpufreq_policy to it).
While at it, notice that intel_pstate and longrun don't really need
to verify the "policy" value in struct cpufreq_policy, so drop those
check from them to avoid copying "policy" into struct
cpufreq_policy_data (which allows it to be slightly smaller).
Also while at it fix up white space in a couple of places and make
cpufreq_set_policy() static (as it can be so).
Fixes: 3000ce3c52f8 ("cpufreq: Use per-policy frequency QoS")
Link: https://lore.kernel.org/linux-pm/CAMuHMdX6-jb1W8uC2_237m8ctCpsnGp=JCxqt8pCWVqNXHmkVg@mail.gmail.com
Reported-by: kbuild test robot <lkp@intel.com>
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: 5.4+ <stable@vger.kernel.org> # 5.4+
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
2020-01-26 22:40:11 +00:00
|
|
|
int cpufreq_generic_frequency_table_verify(struct cpufreq_policy_data *policy);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2016-06-27 10:34:07 +00:00
|
|
|
int cpufreq_table_index_unsorted(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq,
|
|
|
|
unsigned int relation);
|
2013-12-03 05:50:46 +00:00
|
|
|
int cpufreq_frequency_table_get_index(struct cpufreq_policy *policy,
|
|
|
|
unsigned int freq);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-08-06 17:23:04 +00:00
|
|
|
ssize_t cpufreq_show_cpus(const struct cpumask *mask, char *buf);
|
|
|
|
|
2013-12-20 14:24:49 +00:00
|
|
|
#ifdef CONFIG_CPU_FREQ
|
|
|
|
int cpufreq_boost_trigger_state(int state);
|
|
|
|
int cpufreq_boost_enabled(void);
|
2015-07-29 10:53:09 +00:00
|
|
|
int cpufreq_enable_boost_support(void);
|
|
|
|
bool policy_has_boost_freq(struct cpufreq_policy *policy);
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
/* Find lowest freq at or above target in a table in ascending order */
|
|
|
|
static inline int cpufreq_table_find_index_al(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq)
|
|
|
|
{
|
|
|
|
struct cpufreq_frequency_table *table = policy->freq_table;
|
2018-01-30 05:42:37 +00:00
|
|
|
struct cpufreq_frequency_table *pos;
|
2016-06-27 10:34:07 +00:00
|
|
|
unsigned int freq;
|
2018-01-30 05:42:37 +00:00
|
|
|
int idx, best = -1;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
cpufreq_for_each_valid_entry_idx(pos, table, idx) {
|
2016-10-12 03:15:05 +00:00
|
|
|
freq = pos->frequency;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
if (freq >= target_freq)
|
2018-01-30 05:42:37 +00:00
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
best = idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
return best;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Find lowest freq at or above target in a table in descending order */
|
|
|
|
static inline int cpufreq_table_find_index_dl(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq)
|
|
|
|
{
|
|
|
|
struct cpufreq_frequency_table *table = policy->freq_table;
|
2018-01-30 05:42:37 +00:00
|
|
|
struct cpufreq_frequency_table *pos;
|
2016-06-27 10:34:07 +00:00
|
|
|
unsigned int freq;
|
2018-01-30 05:42:37 +00:00
|
|
|
int idx, best = -1;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
cpufreq_for_each_valid_entry_idx(pos, table, idx) {
|
2016-10-12 03:15:05 +00:00
|
|
|
freq = pos->frequency;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
if (freq == target_freq)
|
2018-01-30 05:42:37 +00:00
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
if (freq > target_freq) {
|
2018-01-30 05:42:37 +00:00
|
|
|
best = idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* No freq found above target_freq */
|
2018-01-30 05:42:37 +00:00
|
|
|
if (best == -1)
|
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
return best;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
return best;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Works only on sorted freq-tables */
|
|
|
|
static inline int cpufreq_table_find_index_l(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq)
|
|
|
|
{
|
|
|
|
target_freq = clamp_val(target_freq, policy->min, policy->max);
|
|
|
|
|
|
|
|
if (policy->freq_table_sorted == CPUFREQ_TABLE_SORTED_ASCENDING)
|
|
|
|
return cpufreq_table_find_index_al(policy, target_freq);
|
|
|
|
else
|
|
|
|
return cpufreq_table_find_index_dl(policy, target_freq);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Find highest freq at or below target in a table in ascending order */
|
|
|
|
static inline int cpufreq_table_find_index_ah(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq)
|
|
|
|
{
|
|
|
|
struct cpufreq_frequency_table *table = policy->freq_table;
|
2018-01-30 05:42:37 +00:00
|
|
|
struct cpufreq_frequency_table *pos;
|
2016-06-27 10:34:07 +00:00
|
|
|
unsigned int freq;
|
2018-01-30 05:42:37 +00:00
|
|
|
int idx, best = -1;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
cpufreq_for_each_valid_entry_idx(pos, table, idx) {
|
2016-10-12 03:15:05 +00:00
|
|
|
freq = pos->frequency;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
if (freq == target_freq)
|
2018-01-30 05:42:37 +00:00
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
if (freq < target_freq) {
|
2018-01-30 05:42:37 +00:00
|
|
|
best = idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* No freq found below target_freq */
|
2018-01-30 05:42:37 +00:00
|
|
|
if (best == -1)
|
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
return best;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
return best;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Find highest freq at or below target in a table in descending order */
|
|
|
|
static inline int cpufreq_table_find_index_dh(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq)
|
|
|
|
{
|
|
|
|
struct cpufreq_frequency_table *table = policy->freq_table;
|
2018-01-30 05:42:37 +00:00
|
|
|
struct cpufreq_frequency_table *pos;
|
2016-06-27 10:34:07 +00:00
|
|
|
unsigned int freq;
|
2018-01-30 05:42:37 +00:00
|
|
|
int idx, best = -1;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
cpufreq_for_each_valid_entry_idx(pos, table, idx) {
|
2016-10-12 03:15:05 +00:00
|
|
|
freq = pos->frequency;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
if (freq <= target_freq)
|
2018-01-30 05:42:37 +00:00
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
best = idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
return best;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Works only on sorted freq-tables */
|
|
|
|
static inline int cpufreq_table_find_index_h(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq)
|
|
|
|
{
|
|
|
|
target_freq = clamp_val(target_freq, policy->min, policy->max);
|
|
|
|
|
|
|
|
if (policy->freq_table_sorted == CPUFREQ_TABLE_SORTED_ASCENDING)
|
|
|
|
return cpufreq_table_find_index_ah(policy, target_freq);
|
|
|
|
else
|
|
|
|
return cpufreq_table_find_index_dh(policy, target_freq);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Find closest freq to target in a table in ascending order */
|
|
|
|
static inline int cpufreq_table_find_index_ac(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq)
|
|
|
|
{
|
|
|
|
struct cpufreq_frequency_table *table = policy->freq_table;
|
2018-01-30 05:42:37 +00:00
|
|
|
struct cpufreq_frequency_table *pos;
|
2016-06-27 10:34:07 +00:00
|
|
|
unsigned int freq;
|
2018-01-30 05:42:37 +00:00
|
|
|
int idx, best = -1;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
cpufreq_for_each_valid_entry_idx(pos, table, idx) {
|
2016-10-12 03:15:05 +00:00
|
|
|
freq = pos->frequency;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
if (freq == target_freq)
|
2018-01-30 05:42:37 +00:00
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
if (freq < target_freq) {
|
2018-01-30 05:42:37 +00:00
|
|
|
best = idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* No freq found below target_freq */
|
2018-01-30 05:42:37 +00:00
|
|
|
if (best == -1)
|
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
/* Choose the closest freq */
|
2018-01-30 05:42:37 +00:00
|
|
|
if (target_freq - table[best].frequency > freq - target_freq)
|
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
return best;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
return best;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Find closest freq to target in a table in descending order */
|
|
|
|
static inline int cpufreq_table_find_index_dc(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq)
|
|
|
|
{
|
|
|
|
struct cpufreq_frequency_table *table = policy->freq_table;
|
2018-01-30 05:42:37 +00:00
|
|
|
struct cpufreq_frequency_table *pos;
|
2016-06-27 10:34:07 +00:00
|
|
|
unsigned int freq;
|
2018-01-30 05:42:37 +00:00
|
|
|
int idx, best = -1;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
cpufreq_for_each_valid_entry_idx(pos, table, idx) {
|
2016-10-12 03:15:05 +00:00
|
|
|
freq = pos->frequency;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
if (freq == target_freq)
|
2018-01-30 05:42:37 +00:00
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
if (freq > target_freq) {
|
2018-01-30 05:42:37 +00:00
|
|
|
best = idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* No freq found above target_freq */
|
2018-01-30 05:42:37 +00:00
|
|
|
if (best == -1)
|
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
|
|
|
/* Choose the closest freq */
|
2018-01-30 05:42:37 +00:00
|
|
|
if (table[best].frequency - target_freq > target_freq - freq)
|
|
|
|
return idx;
|
2016-06-27 10:34:07 +00:00
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
return best;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
2018-01-30 05:42:37 +00:00
|
|
|
return best;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Works only on sorted freq-tables */
|
|
|
|
static inline int cpufreq_table_find_index_c(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq)
|
|
|
|
{
|
|
|
|
target_freq = clamp_val(target_freq, policy->min, policy->max);
|
|
|
|
|
|
|
|
if (policy->freq_table_sorted == CPUFREQ_TABLE_SORTED_ASCENDING)
|
|
|
|
return cpufreq_table_find_index_ac(policy, target_freq);
|
|
|
|
else
|
|
|
|
return cpufreq_table_find_index_dc(policy, target_freq);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline int cpufreq_frequency_table_target(struct cpufreq_policy *policy,
|
|
|
|
unsigned int target_freq,
|
|
|
|
unsigned int relation)
|
|
|
|
{
|
|
|
|
if (unlikely(policy->freq_table_sorted == CPUFREQ_TABLE_UNSORTED))
|
|
|
|
return cpufreq_table_index_unsorted(policy, target_freq,
|
|
|
|
relation);
|
|
|
|
|
|
|
|
switch (relation) {
|
|
|
|
case CPUFREQ_RELATION_L:
|
|
|
|
return cpufreq_table_find_index_l(policy, target_freq);
|
|
|
|
case CPUFREQ_RELATION_H:
|
|
|
|
return cpufreq_table_find_index_h(policy, target_freq);
|
|
|
|
case CPUFREQ_RELATION_C:
|
|
|
|
return cpufreq_table_find_index_c(policy, target_freq);
|
|
|
|
default:
|
2020-08-27 05:24:16 +00:00
|
|
|
WARN_ON_ONCE(1);
|
|
|
|
return 0;
|
2016-06-27 10:34:07 +00:00
|
|
|
}
|
|
|
|
}
|
2017-04-25 10:27:15 +00:00
|
|
|
|
|
|
|
static inline int cpufreq_table_count_valid_entries(const struct cpufreq_policy *policy)
|
|
|
|
{
|
|
|
|
struct cpufreq_frequency_table *pos;
|
|
|
|
int count = 0;
|
|
|
|
|
|
|
|
if (unlikely(!policy->freq_table))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
cpufreq_for_each_valid_entry(pos, policy->freq_table)
|
|
|
|
count++;
|
|
|
|
|
|
|
|
return count;
|
|
|
|
}
|
2013-12-20 14:24:49 +00:00
|
|
|
#else
|
|
|
|
static inline int cpufreq_boost_trigger_state(int state)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
static inline int cpufreq_boost_enabled(void)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
2015-07-29 10:53:09 +00:00
|
|
|
|
|
|
|
static inline int cpufreq_enable_boost_support(void)
|
|
|
|
{
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline bool policy_has_boost_freq(struct cpufreq_policy *policy)
|
|
|
|
{
|
|
|
|
return false;
|
|
|
|
}
|
2013-12-20 14:24:49 +00:00
|
|
|
#endif
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2018-12-03 09:56:21 +00:00
|
|
|
#if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL)
|
|
|
|
void sched_cpufreq_governor_change(struct cpufreq_policy *policy,
|
|
|
|
struct cpufreq_governor *old_gov);
|
|
|
|
#else
|
|
|
|
static inline void sched_cpufreq_governor_change(struct cpufreq_policy *policy,
|
|
|
|
struct cpufreq_governor *old_gov) { }
|
|
|
|
#endif
|
|
|
|
|
x86 / CPU: Always show current CPU frequency in /proc/cpuinfo
After commit 890da9cf0983 (Revert "x86: do not use cpufreq_quick_get()
for /proc/cpuinfo "cpu MHz"") the "cpu MHz" number in /proc/cpuinfo
on x86 can be either the nominal CPU frequency (which is constant)
or the frequency most recently requested by a scaling governor in
cpufreq, depending on the cpufreq configuration. That is somewhat
inconsistent and is different from what it was before 4.13, so in
order to restore the previous behavior, make it report the current
CPU frequency like the scaling_cur_freq sysfs file in cpufreq.
To that end, modify the /proc/cpuinfo implementation on x86 to use
aperfmperf_snapshot_khz() to snapshot the APERF and MPERF feedback
registers, if available, and use their values to compute the CPU
frequency to be reported as "cpu MHz".
However, do that carefully enough to avoid accumulating delays that
lead to unacceptable access times for /proc/cpuinfo on systems with
many CPUs. Run aperfmperf_snapshot_khz() once on all CPUs
asynchronously at the /proc/cpuinfo open time, add a single delay
upfront (if necessary) at that point and simply compute the current
frequency while running show_cpuinfo() for each individual CPU.
Also, to avoid slowing down /proc/cpuinfo accesses too much, reduce
the default delay between consecutive APERF and MPERF reads to 10 ms,
which should be sufficient to get large enough numbers for the
frequency computation in all cases.
Fixes: 890da9cf0983 (Revert "x86: do not use cpufreq_quick_get() for /proc/cpuinfo "cpu MHz"")
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Ingo Molnar <mingo@kernel.org>
2017-11-15 01:13:40 +00:00
|
|
|
extern void arch_freq_prepare_all(void);
|
2017-06-24 05:11:52 +00:00
|
|
|
extern unsigned int arch_freq_get_on_cpu(int cpu);
|
|
|
|
|
2020-09-24 12:30:15 +00:00
|
|
|
#ifndef arch_set_freq_scale
|
|
|
|
static __always_inline
|
|
|
|
void arch_set_freq_scale(const struct cpumask *cpus,
|
|
|
|
unsigned long cur_freq,
|
|
|
|
unsigned long max_freq)
|
|
|
|
{
|
|
|
|
}
|
|
|
|
#endif
|
2017-09-26 16:41:07 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* the following are really really optional */
|
|
|
|
extern struct freq_attr cpufreq_freq_attr_scaling_available_freqs;
|
2015-08-07 11:59:16 +00:00
|
|
|
extern struct freq_attr cpufreq_freq_attr_scaling_boost_freqs;
|
2013-10-03 14:57:55 +00:00
|
|
|
extern struct freq_attr *cpufreq_generic_attr[];
|
2018-02-22 05:59:44 +00:00
|
|
|
int cpufreq_table_validate_and_sort(struct cpufreq_policy *policy);
|
2013-06-27 07:08:54 +00:00
|
|
|
|
2014-01-09 15:08:43 +00:00
|
|
|
unsigned int cpufreq_generic_get(unsigned int cpu);
|
2019-07-16 04:06:08 +00:00
|
|
|
void cpufreq_generic_init(struct cpufreq_policy *policy,
|
2013-10-03 14:59:07 +00:00
|
|
|
struct cpufreq_frequency_table *table,
|
|
|
|
unsigned int transition_latency);
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif /* _LINUX_CPUFREQ_H */
|