mirrors
/
linux-stable
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1
0
Fork 0
Code Issues Releases Wiki Activity

Perf events changes in this cycle were: 

- Improve Intel uncore PMU support:
 
      - Parse uncore 'discovery tables' - a new hardware capability enumeration method
        introduced on the latest Intel platforms. This table is in a well-defined PCI
        namespace location and is read via MMIO. It is organized in an rbtree.
 
        These uncore tables will allow the discovery of standard counter blocks, but
        fancier counters still need to be enumerated explicitly.
 
      - Add Alder Lake support
 
      - Improve IIO stacks to PMON mapping support on Skylake servers
 
  - Add Intel Alder Lake PMU support - which requires the introduction of 'hybrid' CPUs
    and PMUs. Alder Lake is a mix of Golden Cove ('big') and Gracemont ('small' - Atom derived)
    cores.
 
    The CPU-side feature set is entirely symmetrical - but on the PMU side there's
    core type dependent PMU functionality.
 
  - Reduce data loss with CPU level hardware tracing on Intel PT / AUX profiling, by
    fixing the AUX allocation watermark logic.
 
  - Improve ring buffer allocation on NUMA systems
 
  - Put 'struct perf_event' into their separate kmem_cache pool
 
  - Add support for synchronous signals for select perf events. The immediate motivation
    is to support low-overhead sampling-based race detection for user-space code. The
    feature consists of the following main changes:
 
     - Add thread-only event inheritance via perf_event_attr::inherit_thread, which limits
       inheritance of events to CLONE_THREAD.
 
     - Add the ability for events to not leak through exec(), via perf_event_attr::remove_on_exec.
 
     - Allow the generation of SIGTRAP via perf_event_attr::sigtrap, extend siginfo with an u64
       ::si_perf, and add the breakpoint information to ::si_addr and ::si_perf if the event is
       PERF_TYPE_BREAKPOINT.
 
    The siginfo support is adequate for breakpoints right now - but the new field can be used
    to introduce support for other types of metadata passed over siginfo as well.
 
  - Misc fixes, cleanups and smaller updates.
 
 Signed-off-by: Ingo Molnar <mingo@kernel.org>
 -----BEGIN PGP SIGNATURE-----
 
 iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmCJGpERHG1pbmdvQGtl
 cm5lbC5vcmcACgkQEnMQ0APhK1j9zBAAuVbG2snV6SBSdXLhQcM66N3NckOXvSY5
 QjjhQcuwJQEK/NJB3266K5d8qSmdyRBsWf3GCsrmyBT67P1V28K44Pu7oCV0UDtf
 mpVRjEP0oR7hNsANSSgo8Fa4ZD7H5waX7dK7925Tvw8By3mMoZoddiD/84WJHhxO
 NDF+GRFaRj+/dpbhV8cdCoXTjYdkC36vYuZs3b9lu0tS9D/AJgsNy7TinLvO02Cs
 5peP+2y29dgvCXiGBiuJtEA6JyGnX3nUJCvfOZZ/DWDc3fdduARlRrc5Aiq4n/wY
 UdSkw1VTZBlZ1wMSdmHQVeC5RIH3uWUtRoNqy0Yc90lBm55AQ0EENwIfWDUDC5zy
 USdBqWTNWKMBxlEilUIyqKPQK8LW/31TRzqy8BWKPNcZt5yP5YS1SjAJRDDjSwL/
 I+OBw1vjLJamYh8oNiD5b+VLqNQba81jFASfv+HVWcULumnY6ImECCpkg289Fkpi
 BVR065boifJDlyENXFbvTxyMBXQsZfA+EhtxG7ju2Ni+TokBbogyCb3L2injPt9g
 7jjtTOqmfad4gX1WSc+215iYZMkgECcUd9E+BfOseEjBohqlo7yNKIfYnT8mE/Xq
 nb7eHjyvLiE8tRtZ+7SjsujOMHv9LhWFAbSaxU/kEVzpkp0zyd6mnnslDKaaHLhz
 goUMOL/D0lg=
 =NhQ7
 -----END PGP SIGNATURE-----

Merge tag 'perf-core-2021-04-28' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull perf event updates from Ingo Molnar:

 - Improve Intel uncore PMU support:

     - Parse uncore 'discovery tables' - a new hardware capability
       enumeration method introduced on the latest Intel platforms. This
       table is in a well-defined PCI namespace location and is read via
       MMIO. It is organized in an rbtree.

       These uncore tables will allow the discovery of standard counter
       blocks, but fancier counters still need to be enumerated
       explicitly.

     - Add Alder Lake support

     - Improve IIO stacks to PMON mapping support on Skylake servers

 - Add Intel Alder Lake PMU support - which requires the introduction of
   'hybrid' CPUs and PMUs. Alder Lake is a mix of Golden Cove ('big')
   and Gracemont ('small' - Atom derived) cores.

   The CPU-side feature set is entirely symmetrical - but on the PMU
   side there's core type dependent PMU functionality.

 - Reduce data loss with CPU level hardware tracing on Intel PT / AUX
   profiling, by fixing the AUX allocation watermark logic.

 - Improve ring buffer allocation on NUMA systems

 - Put 'struct perf_event' into their separate kmem_cache pool

 - Add support for synchronous signals for select perf events. The
   immediate motivation is to support low-overhead sampling-based race
   detection for user-space code. The feature consists of the following
   main changes:

     - Add thread-only event inheritance via
       perf_event_attr::inherit_thread, which limits inheritance of
       events to CLONE_THREAD.

     - Add the ability for events to not leak through exec(), via
       perf_event_attr::remove_on_exec.

     - Allow the generation of SIGTRAP via perf_event_attr::sigtrap,
       extend siginfo with an u64 ::si_perf, and add the breakpoint
       information to ::si_addr and ::si_perf if the event is
       PERF_TYPE_BREAKPOINT.

   The siginfo support is adequate for breakpoints right now - but the
   new field can be used to introduce support for other types of
   metadata passed over siginfo as well.

 - Misc fixes, cleanups and smaller updates.

* tag 'perf-core-2021-04-28' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (53 commits)
  signal, perf: Add missing TRAP_PERF case in siginfo_layout()
  signal, perf: Fix siginfo_t by avoiding u64 on 32-bit architectures
  perf/x86: Allow for 8<num_fixed_counters<16
  perf/x86/rapl: Add support for Intel Alder Lake
  perf/x86/cstate: Add Alder Lake CPU support
  perf/x86/msr: Add Alder Lake CPU support
  perf/x86/intel/uncore: Add Alder Lake support
  perf: Extend PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE
  perf/x86/intel: Add Alder Lake Hybrid support
  perf/x86: Support filter_match callback
  perf/x86/intel: Add attr_update for Hybrid PMUs
  perf/x86: Add structures for the attributes of Hybrid PMUs
  perf/x86: Register hybrid PMUs
  perf/x86: Factor out x86_pmu_show_pmu_cap
  perf/x86: Remove temporary pmu assignment in event_init
  perf/x86/intel: Factor out intel_pmu_check_extra_regs
  perf/x86/intel: Factor out intel_pmu_check_event_constraints
  perf/x86/intel: Factor out intel_pmu_check_num_counters
  perf/x86: Hybrid PMU support for extra_regs
  perf/x86: Hybrid PMU support for event constraints
  ...
This commit is contained in:
Linus Torvalds 2021-04-28 13:03:44 -07:00
parent 03b2cd72aa ed8e50800b
commit 42dec9a936
42 changed files with 3058 additions and 420 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
arch/m68k/kernel/signal.c
View File

@ -622,6 +622,9 @@ static inline void siginfo_build_tests(void)
/* _sigfault._addr_pkey */
BUILD_BUG_ON(offsetof(siginfo_t, si_pkey) != 0x12);
/* _sigfault._perf */
BUILD_BUG_ON(offsetof(siginfo_t, si_perf) != 0x10);
/* _sigpoll */
BUILD_BUG_ON(offsetof(siginfo_t, si_band) != 0x0c);
BUILD_BUG_ON(offsetof(siginfo_t, si_fd) != 0x10);

6
arch/x86/events/amd/iommu.c
View File

@ -81,12 +81,12 @@ static struct attribute_group amd_iommu_events_group = {
};
struct amd_iommu_event_desc {
struct kobj_attribute attr;
struct device_attribute attr;
const char *event;
};
static ssize_t _iommu_event_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
static ssize_t _iommu_event_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct amd_iommu_event_desc *event =
container_of(attr, struct amd_iommu_event_desc, attr);

6
arch/x86/events/amd/uncore.c
View File

@ -275,14 +275,14 @@ static struct attribute_group amd_uncore_attr_group = {
};
#define DEFINE_UNCORE_FORMAT_ATTR(_var, _name, _format) \
static ssize_t __uncore_##_var##_show(struct kobject *kobj, \
struct kobj_attribute *attr, \
static ssize_t __uncore_##_var##_show(struct device *dev, \
struct device_attribute *attr, \
char *page) \
{ \
BUILD_BUG_ON(sizeof(_format) >= PAGE_SIZE); \
return sprintf(page, _format "\n"); \
} \
static struct kobj_attribute format_attr_##_var = \
static struct device_attribute format_attr_##_var = \
__ATTR(_name, 0444, __uncore_##_var##_show, NULL)
DEFINE_UNCORE_FORMAT_ATTR(event12, event, "config:0-7,32-35");

332
arch/x86/events/core.c
View File

@ -45,13 +45,16 @@
#include "perf_event.h"
struct x86_pmu x86_pmu __read_mostly;
static struct pmu pmu;
DEFINE_PER_CPU(struct cpu_hw_events, cpu_hw_events) = {
.enabled = 1,
.pmu = &pmu,
};
DEFINE_STATIC_KEY_FALSE(rdpmc_never_available_key);
DEFINE_STATIC_KEY_FALSE(rdpmc_always_available_key);
DEFINE_STATIC_KEY_FALSE(perf_is_hybrid);
/*
* This here uses DEFINE_STATIC_CALL_NULL() to get a static_call defined
@ -151,15 +154,16 @@ again:
*/
static int x86_pmu_extra_regs(u64 config, struct perf_event *event)
{
struct extra_reg *extra_regs = hybrid(event->pmu, extra_regs);
struct hw_perf_event_extra *reg;
struct extra_reg *er;
reg = &event->hw.extra_reg;
if (!x86_pmu.extra_regs)
if (!extra_regs)
return 0;
for (er = x86_pmu.extra_regs; er->msr; er++) {
for (er = extra_regs; er->msr; er++) {
if (er->event != (config & er->config_mask))
continue;
if (event->attr.config1 & ~er->valid_mask)
@ -182,16 +186,29 @@ static DEFINE_MUTEX(pmc_reserve_mutex);
#ifdef CONFIG_X86_LOCAL_APIC
static inline int get_possible_num_counters(void)
{
int i, num_counters = x86_pmu.num_counters;
if (!is_hybrid())
return num_counters;
for (i = 0; i < x86_pmu.num_hybrid_pmus; i++)
num_counters = max_t(int, num_counters, x86_pmu.hybrid_pmu[i].num_counters);
return num_counters;
}
static bool reserve_pmc_hardware(void)
{
int i;
int i, num_counters = get_possible_num_counters();
for (i = 0; i < x86_pmu.num_counters; i++) {
for (i = 0; i < num_counters; i++) {
if (!reserve_perfctr_nmi(x86_pmu_event_addr(i)))
goto perfctr_fail;
}
for (i = 0; i < x86_pmu.num_counters; i++) {
for (i = 0; i < num_counters; i++) {
if (!reserve_evntsel_nmi(x86_pmu_config_addr(i)))
goto eventsel_fail;
}
@ -202,7 +219,7 @@ eventsel_fail:
for (i--; i >= 0; i--)
release_evntsel_nmi(x86_pmu_config_addr(i));
i = x86_pmu.num_counters;
i = num_counters;
perfctr_fail:
for (i--; i >= 0; i--)
@ -213,9 +230,9 @@ perfctr_fail:
static void release_pmc_hardware(void)
{
int i;
int i, num_counters = get_possible_num_counters();
for (i = 0; i < x86_pmu.num_counters; i++) {
for (i = 0; i < num_counters; i++) {
release_perfctr_nmi(x86_pmu_event_addr(i));
release_evntsel_nmi(x86_pmu_config_addr(i));
}
@ -228,7 +245,7 @@ static void release_pmc_hardware(void) {}
#endif
static bool check_hw_exists(void)
bool check_hw_exists(struct pmu *pmu, int num_counters, int num_counters_fixed)
{
u64 val, val_fail = -1, val_new= ~0;
int i, reg, reg_fail = -1, ret = 0;
@ -239,7 +256,7 @@ static bool check_hw_exists(void)
* Check to see if the BIOS enabled any of the counters, if so
* complain and bail.
*/
for (i = 0; i < x86_pmu.num_counters; i++) {
for (i = 0; i < num_counters; i++) {
reg = x86_pmu_config_addr(i);
ret = rdmsrl_safe(reg, &val);
if (ret)
@ -253,15 +270,15 @@ static bool check_hw_exists(void)
}
}
if (x86_pmu.num_counters_fixed) {
if (num_counters_fixed) {
reg = MSR_ARCH_PERFMON_FIXED_CTR_CTRL;
ret = rdmsrl_safe(reg, &val);
if (ret)
goto msr_fail;
for (i = 0; i < x86_pmu.num_counters_fixed; i++) {
if (fixed_counter_disabled(i))
for (i = 0; i < num_counters_fixed; i++) {
if (fixed_counter_disabled(i, pmu))
continue;
if (val & (0x03 << i*4)) {
if (val & (0x03ULL << i*4)) {
bios_fail = 1;
val_fail = val;
reg_fail = reg;
@ -360,8 +377,7 @@ set_ext_hw_attr(struct hw_perf_event *hwc, struct perf_event *event)
return -EINVAL;
cache_result = array_index_nospec(cache_result, PERF_COUNT_HW_CACHE_RESULT_MAX);
val = hw_cache_event_ids[cache_type][cache_op][cache_result];
val = hybrid_var(event->pmu, hw_cache_event_ids)[cache_type][cache_op][cache_result];
if (val == 0)
return -ENOENT;
@ -369,7 +385,7 @@ set_ext_hw_attr(struct hw_perf_event *hwc, struct perf_event *event)
return -EINVAL;
hwc->config |= val;
attr->config1 = hw_cache_extra_regs[cache_type][cache_op][cache_result];
attr->config1 = hybrid_var(event->pmu, hw_cache_extra_regs)[cache_type][cache_op][cache_result];
return x86_pmu_extra_regs(val, event);
}
@ -462,7 +478,7 @@ int x86_setup_perfctr(struct perf_event *event)
local64_set(&hwc->period_left, hwc->sample_period);
}
if (attr->type == PERF_TYPE_RAW)
if (attr->type == event->pmu->type)
return x86_pmu_extra_regs(event->attr.config, event);
if (attr->type == PERF_TYPE_HW_CACHE)
@ -597,7 +613,7 @@ int x86_pmu_hw_config(struct perf_event *event)
if (!event->attr.exclude_kernel)
event->hw.config |= ARCH_PERFMON_EVENTSEL_OS;
if (event->attr.type == PERF_TYPE_RAW)
if (event->attr.type == event->pmu->type)
event->hw.config |= event->attr.config & X86_RAW_EVENT_MASK;
if (event->attr.sample_period && x86_pmu.limit_period) {
@ -724,16 +740,33 @@ void x86_pmu_enable_all(int added)
}
}
static struct pmu pmu;
static inline int is_x86_event(struct perf_event *event)
{
return event->pmu == &pmu;
int i;
if (!is_hybrid())
return event->pmu == &pmu;
for (i = 0; i < x86_pmu.num_hybrid_pmus; i++) {
if (event->pmu == &x86_pmu.hybrid_pmu[i].pmu)
return true;
}
return false;
}
struct pmu *x86_get_pmu(void)
struct pmu *x86_get_pmu(unsigned int cpu)
{
return &pmu;
struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
/*
* All CPUs of the hybrid type have been offline.
* The x86_get_pmu() should not be invoked.
*/
if (WARN_ON_ONCE(!cpuc->pmu))
return &pmu;
return cpuc->pmu;
}
/*
* Event scheduler state:
@ -936,6 +969,7 @@ EXPORT_SYMBOL_GPL(perf_assign_events);
int x86_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign)
{
int num_counters = hybrid(cpuc->pmu, num_counters);
struct event_constraint *c;
struct perf_event *e;
int n0, i, wmin, wmax, unsched = 0;
@ -1011,7 +1045,7 @@ int x86_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign)
/* slow path */
if (i != n) {
int gpmax = x86_pmu.num_counters;
int gpmax = num_counters;
/*
* Do not allow scheduling of more than half the available
@ -1032,7 +1066,7 @@ int x86_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign)
* the extra Merge events needed by large increment events.
*/
if (x86_pmu.flags & PMU_FL_PAIR) {
gpmax = x86_pmu.num_counters - cpuc->n_pair;
gpmax = num_counters - cpuc->n_pair;
WARN_ON(gpmax <= 0);
}
@ -1096,8 +1130,9 @@ static void del_nr_metric_event(struct cpu_hw_events *cpuc,
static int collect_event(struct cpu_hw_events *cpuc, struct perf_event *event,
int max_count, int n)
{
union perf_capabilities intel_cap = hybrid(cpuc->pmu, intel_cap);
if (x86_pmu.intel_cap.perf_metrics && add_nr_metric_event(cpuc, event))
if (intel_cap.perf_metrics && add_nr_metric_event(cpuc, event))
return -EINVAL;
if (n >= max_count + cpuc->n_metric)
@ -1118,10 +1153,12 @@ static int collect_event(struct cpu_hw_events *cpuc, struct perf_event *event,
*/
static int collect_events(struct cpu_hw_events *cpuc, struct perf_event *leader, bool dogrp)
{
int num_counters = hybrid(cpuc->pmu, num_counters);
int num_counters_fixed = hybrid(cpuc->pmu, num_counters_fixed);
struct perf_event *event;
int n, max_count;
max_count = x86_pmu.num_counters + x86_pmu.num_counters_fixed;
max_count = num_counters + num_counters_fixed;
/* current number of events already accepted */
n = cpuc->n_events;
@ -1480,7 +1517,6 @@ static void x86_pmu_start(struct perf_event *event, int flags)
cpuc->events[idx] = event;
__set_bit(idx, cpuc->active_mask);
__set_bit(idx, cpuc->running);
static_call(x86_pmu_enable)(event);
perf_event_update_userpage(event);
}
@ -1489,18 +1525,19 @@ void perf_event_print_debug(void)
{
u64 ctrl, status, overflow, pmc_ctrl, pmc_count, prev_left, fixed;
u64 pebs, debugctl;
struct cpu_hw_events *cpuc;
int cpu = smp_processor_id();
struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
int num_counters = hybrid(cpuc->pmu, num_counters);
int num_counters_fixed = hybrid(cpuc->pmu, num_counters_fixed);
struct event_constraint *pebs_constraints = hybrid(cpuc->pmu, pebs_constraints);
unsigned long flags;
int cpu, idx;
int idx;
if (!x86_pmu.num_counters)
if (!num_counters)
return;
local_irq_save(flags);
cpu = smp_processor_id();
cpuc = &per_cpu(cpu_hw_events, cpu);
if (x86_pmu.version >= 2) {
rdmsrl(MSR_CORE_PERF_GLOBAL_CTRL, ctrl);
rdmsrl(MSR_CORE_PERF_GLOBAL_STATUS, status);
@ -1512,7 +1549,7 @@ void perf_event_print_debug(void)
pr_info("CPU#%d: status: %016llx\n", cpu, status);
pr_info("CPU#%d: overflow: %016llx\n", cpu, overflow);
pr_info("CPU#%d: fixed: %016llx\n", cpu, fixed);
if (x86_pmu.pebs_constraints) {
if (pebs_constraints) {
rdmsrl(MSR_IA32_PEBS_ENABLE, pebs);
pr_info("CPU#%d: pebs: %016llx\n", cpu, pebs);
}
@ -1523,7 +1560,7 @@ void perf_event_print_debug(void)
}
pr_info("CPU#%d: active: %016llx\n", cpu, *(u64 *)cpuc->active_mask);
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
for (idx = 0; idx < num_counters; idx++) {
rdmsrl(x86_pmu_config_addr(idx), pmc_ctrl);
rdmsrl(x86_pmu_event_addr(idx), pmc_count);
@ -1536,8 +1573,8 @@ void perf_event_print_debug(void)
pr_info("CPU#%d: gen-PMC%d left: %016llx\n",
cpu, idx, prev_left);
}
for (idx = 0; idx < x86_pmu.num_counters_fixed; idx++) {
if (fixed_counter_disabled(idx))
for (idx = 0; idx < num_counters_fixed; idx++) {
if (fixed_counter_disabled(idx, cpuc->pmu))
continue;
rdmsrl(MSR_ARCH_PERFMON_FIXED_CTR0 + idx, pmc_count);
@ -1573,6 +1610,7 @@ void x86_pmu_stop(struct perf_event *event, int flags)
static void x86_pmu_del(struct perf_event *event, int flags)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
union perf_capabilities intel_cap = hybrid(cpuc->pmu, intel_cap);
int i;
/*
@ -1612,7 +1650,7 @@ static void x86_pmu_del(struct perf_event *event, int flags)
}
cpuc->event_constraint[i-1] = NULL;
--cpuc->n_events;
if (x86_pmu.intel_cap.perf_metrics)
if (intel_cap.perf_metrics)
del_nr_metric_event(cpuc, event);
perf_event_update_userpage(event);
@ -1822,6 +1860,49 @@ ssize_t events_ht_sysfs_show(struct device *dev, struct device_attribute *attr,
pmu_attr->event_str_noht);
}
ssize_t events_hybrid_sysfs_show(struct device *dev,
struct device_attribute *attr,
char *page)
{
struct perf_pmu_events_hybrid_attr *pmu_attr =
container_of(attr, struct perf_pmu_events_hybrid_attr, attr);
struct x86_hybrid_pmu *pmu;
const char *str, *next_str;
int i;
if (hweight64(pmu_attr->pmu_type) == 1)
return sprintf(page, "%s", pmu_attr->event_str);
/*
* Hybrid PMUs may support the same event name, but with different
* event encoding, e.g., the mem-loads event on an Atom PMU has
* different event encoding from a Core PMU.
*
* The event_str includes all event encodings. Each event encoding
* is divided by ";". The order of the event encodings must follow
* the order of the hybrid PMU index.
*/
pmu = container_of(dev_get_drvdata(dev), struct x86_hybrid_pmu, pmu);
str = pmu_attr->event_str;
for (i = 0; i < x86_pmu.num_hybrid_pmus; i++) {
if (!(x86_pmu.hybrid_pmu[i].cpu_type & pmu_attr->pmu_type))
continue;
if (x86_pmu.hybrid_pmu[i].cpu_type & pmu->cpu_type) {
next_str = strchr(str, ';');
if (next_str)
return snprintf(page, next_str - str + 1, "%s", str);
else
return sprintf(page, "%s", str);
}
str = strchr(str, ';');
str++;
}
return 0;
}
EXPORT_SYMBOL_GPL(events_hybrid_sysfs_show);
EVENT_ATTR(cpu-cycles, CPU_CYCLES );
EVENT_ATTR(instructions, INSTRUCTIONS );
EVENT_ATTR(cache-references, CACHE_REFERENCES );
@ -1948,6 +2029,37 @@ static void _x86_pmu_read(struct perf_event *event)
x86_perf_event_update(event);
}
void x86_pmu_show_pmu_cap(int num_counters, int num_counters_fixed,
u64 intel_ctrl)
{
pr_info("... version: %d\n", x86_pmu.version);
pr_info("... bit width: %d\n", x86_pmu.cntval_bits);
pr_info("... generic registers: %d\n", num_counters);
pr_info("... value mask: %016Lx\n", x86_pmu.cntval_mask);
pr_info("... max period: %016Lx\n", x86_pmu.max_period);
pr_info("... fixed-purpose events: %lu\n",
hweight64((((1ULL << num_counters_fixed) - 1)
<< INTEL_PMC_IDX_FIXED) & intel_ctrl));
pr_info("... event mask: %016Lx\n", intel_ctrl);
}
/*
* The generic code is not hybrid friendly. The hybrid_pmu->pmu
* of the first registered PMU is unconditionally assigned to
* each possible cpuctx->ctx.pmu.
* Update the correct hybrid PMU to the cpuctx->ctx.pmu.
*/
void x86_pmu_update_cpu_context(struct pmu *pmu, int cpu)
{
struct perf_cpu_context *cpuctx;
if (!pmu->pmu_cpu_context)
return;
cpuctx = per_cpu_ptr(pmu->pmu_cpu_context, cpu);
cpuctx->ctx.pmu = pmu;
}
static int __init init_hw_perf_events(void)
{
struct x86_pmu_quirk *quirk;
@ -1981,7 +2093,7 @@ static int __init init_hw_perf_events(void)
pmu_check_apic();
/* sanity check that the hardware exists or is emulated */
if (!check_hw_exists())
if (!check_hw_exists(&pmu, x86_pmu.num_counters, x86_pmu.num_counters_fixed))
return 0;
pr_cont("%s PMU driver.\n", x86_pmu.name);
@ -2008,15 +2120,11 @@ static int __init init_hw_perf_events(void)
pmu.attr_update = x86_pmu.attr_update;
pr_info("... version: %d\n", x86_pmu.version);
pr_info("... bit width: %d\n", x86_pmu.cntval_bits);
pr_info("... generic registers: %d\n", x86_pmu.num_counters);
pr_info("... value mask: %016Lx\n", x86_pmu.cntval_mask);
pr_info("... max period: %016Lx\n", x86_pmu.max_period);
pr_info("... fixed-purpose events: %lu\n",
hweight64((((1ULL << x86_pmu.num_counters_fixed) - 1)
<< INTEL_PMC_IDX_FIXED) & x86_pmu.intel_ctrl));
pr_info("... event mask: %016Lx\n", x86_pmu.intel_ctrl);
if (!is_hybrid()) {
x86_pmu_show_pmu_cap(x86_pmu.num_counters,
x86_pmu.num_counters_fixed,
x86_pmu.intel_ctrl);
}
if (!x86_pmu.read)
x86_pmu.read = _x86_pmu_read;
@ -2046,9 +2154,46 @@ static int __init init_hw_perf_events(void)
if (err)
goto out1;
err = perf_pmu_register(&pmu, "cpu", PERF_TYPE_RAW);
if (err)
goto out2;
if (!is_hybrid()) {
err = perf_pmu_register(&pmu, "cpu", PERF_TYPE_RAW);
if (err)
goto out2;
} else {
u8 cpu_type = get_this_hybrid_cpu_type();
struct x86_hybrid_pmu *hybrid_pmu;
int i, j;
if (!cpu_type && x86_pmu.get_hybrid_cpu_type)
cpu_type = x86_pmu.get_hybrid_cpu_type();
for (i = 0; i < x86_pmu.num_hybrid_pmus; i++) {
hybrid_pmu = &x86_pmu.hybrid_pmu[i];
hybrid_pmu->pmu = pmu;
hybrid_pmu->pmu.type = -1;
hybrid_pmu->pmu.attr_update = x86_pmu.attr_update;
hybrid_pmu->pmu.capabilities |= PERF_PMU_CAP_HETEROGENEOUS_CPUS;
hybrid_pmu->pmu.capabilities |= PERF_PMU_CAP_EXTENDED_HW_TYPE;
err = perf_pmu_register(&hybrid_pmu->pmu, hybrid_pmu->name,
(hybrid_pmu->cpu_type == hybrid_big) ? PERF_TYPE_RAW : -1);
if (err)
break;
if (cpu_type == hybrid_pmu->cpu_type)
x86_pmu_update_cpu_context(&hybrid_pmu->pmu, raw_smp_processor_id());
}
if (i < x86_pmu.num_hybrid_pmus) {
for (j = 0; j < i; j++)
perf_pmu_unregister(&x86_pmu.hybrid_pmu[j].pmu);
pr_warn("Failed to register hybrid PMUs\n");
kfree(x86_pmu.hybrid_pmu);
x86_pmu.hybrid_pmu = NULL;
x86_pmu.num_hybrid_pmus = 0;
goto out2;
}
}
return 0;
@ -2173,16 +2318,27 @@ static void free_fake_cpuc(struct cpu_hw_events *cpuc)
kfree(cpuc);
}
static struct cpu_hw_events *allocate_fake_cpuc(void)
static struct cpu_hw_events *allocate_fake_cpuc(struct pmu *event_pmu)
{
struct cpu_hw_events *cpuc;
int cpu = raw_smp_processor_id();
int cpu;
cpuc = kzalloc(sizeof(*cpuc), GFP_KERNEL);
if (!cpuc)
return ERR_PTR(-ENOMEM);
cpuc->is_fake = 1;
if (is_hybrid()) {
struct x86_hybrid_pmu *h_pmu;
h_pmu = hybrid_pmu(event_pmu);
if (cpumask_empty(&h_pmu->supported_cpus))
goto error;
cpu = cpumask_first(&h_pmu->supported_cpus);
} else
cpu = raw_smp_processor_id();
cpuc->pmu = event_pmu;
if (intel_cpuc_prepare(cpuc, cpu))
goto error;
@ -2201,7 +2357,7 @@ static int validate_event(struct perf_event *event)
struct event_constraint *c;
int ret = 0;
fake_cpuc = allocate_fake_cpuc();
fake_cpuc = allocate_fake_cpuc(event->pmu);
if (IS_ERR(fake_cpuc))
return PTR_ERR(fake_cpuc);
@ -2235,7 +2391,27 @@ static int validate_group(struct perf_event *event)
struct cpu_hw_events *fake_cpuc;
int ret = -EINVAL, n;
fake_cpuc = allocate_fake_cpuc();
/*
* Reject events from different hybrid PMUs.
*/
if (is_hybrid()) {
struct perf_event *sibling;
struct pmu *pmu = NULL;
if (is_x86_event(leader))
pmu = leader->pmu;
for_each_sibling_event(sibling, leader) {
if (!is_x86_event(sibling))
continue;
if (!pmu)
pmu = sibling->pmu;
else if (pmu != sibling->pmu)
return ret;
}
}
fake_cpuc = allocate_fake_cpuc(event->pmu);
if (IS_ERR(fake_cpuc))
return PTR_ERR(fake_cpuc);
/*
@ -2263,35 +2439,26 @@ out:
static int x86_pmu_event_init(struct perf_event *event)
{
struct pmu *tmp;
struct x86_hybrid_pmu *pmu = NULL;
int err;
switch (event->attr.type) {
case PERF_TYPE_RAW:
case PERF_TYPE_HARDWARE:
case PERF_TYPE_HW_CACHE:
break;
default:
if ((event->attr.type != event->pmu->type) &&
(event->attr.type != PERF_TYPE_HARDWARE) &&
(event->attr.type != PERF_TYPE_HW_CACHE))
return -ENOENT;
if (is_hybrid() && (event->cpu != -1)) {
pmu = hybrid_pmu(event->pmu);
if (!cpumask_test_cpu(event->cpu, &pmu->supported_cpus))
return -ENOENT;
}
err = __x86_pmu_event_init(event);
if (!err) {
/*
* we temporarily connect event to its pmu
* such that validate_group() can classify
* it as an x86 event using is_x86_event()
*/
tmp = event->pmu;
event->pmu = &pmu;
if (event->group_leader != event)
err = validate_group(event);
else
err = validate_event(event);
event->pmu = tmp;
}
if (err) {
if (event->destroy)
@ -2475,6 +2642,14 @@ static int x86_pmu_aux_output_match(struct perf_event *event)
return 0;
}
static int x86_pmu_filter_match(struct perf_event *event)
{
if (x86_pmu.filter_match)
return x86_pmu.filter_match(event);
return 1;
}
static struct pmu pmu = {
.pmu_enable = x86_pmu_enable,
.pmu_disable = x86_pmu_disable,
@ -2502,6 +2677,8 @@ static struct pmu pmu = {
.check_period = x86_pmu_check_period,
.aux_output_match = x86_pmu_aux_output_match,
.filter_match = x86_pmu_filter_match,
};
void arch_perf_update_userpage(struct perf_event *event,
@ -2770,6 +2947,11 @@ unsigned long perf_misc_flags(struct pt_regs *regs)
void perf_get_x86_pmu_capability(struct x86_pmu_capability *cap)
{
cap->version = x86_pmu.version;
/*
* KVM doesn't support the hybrid PMU yet.
* Return the common value in global x86_pmu,
* which available for all cores.
*/
cap->num_counters_gp = x86_pmu.num_counters;
cap->num_counters_fixed = x86_pmu.num_counters_fixed;
cap->bit_width_gp = x86_pmu.cntval_bits;

2
arch/x86/events/intel/Makefile
View File

@ -3,6 +3,6 @@ obj-$(CONFIG_CPU_SUP_INTEL) += core.o bts.o
obj-$(CONFIG_CPU_SUP_INTEL) += ds.o knc.o
obj-$(CONFIG_CPU_SUP_INTEL) += lbr.o p4.o p6.o pt.o
obj-$(CONFIG_PERF_EVENTS_INTEL_UNCORE) += intel-uncore.o
intel-uncore-objs := uncore.o uncore_nhmex.o uncore_snb.o uncore_snbep.o
intel-uncore-objs := uncore.o uncore_nhmex.o uncore_snb.o uncore_snbep.o uncore_discovery.o
obj-$(CONFIG_PERF_EVENTS_INTEL_CSTATE) += intel-cstate.o
intel-cstate-objs := cstate.o

689
arch/x86/events/intel/core.c
View File

@ -2076,6 +2076,14 @@ static struct extra_reg intel_tnt_extra_regs[] __read_mostly = {
EVENT_EXTRA_END
};
static struct extra_reg intel_grt_extra_regs[] __read_mostly = {
/* must define OFFCORE_RSP_X first, see intel_fixup_er() */
INTEL_UEVENT_EXTRA_REG(0x01b7, MSR_OFFCORE_RSP_0, 0x3fffffffffull, RSP_0),
INTEL_UEVENT_EXTRA_REG(0x02b7, MSR_OFFCORE_RSP_1, 0x3fffffffffull, RSP_1),
INTEL_UEVENT_PEBS_LDLAT_EXTRA_REG(0x5d0),
EVENT_EXTRA_END
};
#define KNL_OT_L2_HITE BIT_ULL(19) /* Other Tile L2 Hit */
#define KNL_OT_L2_HITF BIT_ULL(20) /* Other Tile L2 Hit */
#define KNL_MCDRAM_LOCAL BIT_ULL(21)
@ -2153,10 +2161,11 @@ static void intel_pmu_disable_all(void)
static void __intel_pmu_enable_all(int added, bool pmi)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl);
intel_pmu_lbr_enable_all(pmi);
wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL,
x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask);
intel_ctrl & ~cpuc->intel_ctrl_guest_mask);
if (test_bit(INTEL_PMC_IDX_FIXED_BTS, cpuc->active_mask)) {
struct perf_event *event =
@ -2429,6 +2438,16 @@ static int icl_set_topdown_event_period(struct perf_event *event)
return 0;
}
static int adl_set_topdown_event_period(struct perf_event *event)
{
struct x86_hybrid_pmu *pmu = hybrid_pmu(event->pmu);
if (pmu->cpu_type != hybrid_big)
return 0;
return icl_set_topdown_event_period(event);
}
static inline u64 icl_get_metrics_event_value(u64 metric, u64 slots, int idx)
{
u32 val;
@ -2569,6 +2588,17 @@ static u64 icl_update_topdown_event(struct perf_event *event)
x86_pmu.num_topdown_events - 1);
}
static u64 adl_update_topdown_event(struct perf_event *event)
{
struct x86_hybrid_pmu *pmu = hybrid_pmu(event->pmu);
if (pmu->cpu_type != hybrid_big)
return 0;
return icl_update_topdown_event(event);
}
static void intel_pmu_read_topdown_event(struct perf_event *event)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
@ -2709,22 +2739,25 @@ int intel_pmu_save_and_restart(struct perf_event *event)
static void intel_pmu_reset(void)
{
struct debug_store *ds = __this_cpu_read(cpu_hw_events.ds);
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int num_counters_fixed = hybrid(cpuc->pmu, num_counters_fixed);
int num_counters = hybrid(cpuc->pmu, num_counters);
unsigned long flags;
int idx;
if (!x86_pmu.num_counters)
if (!num_counters)
return;
local_irq_save(flags);
pr_info("clearing PMU state on CPU#%d\n", smp_processor_id());
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
for (idx = 0; idx < num_counters; idx++) {
wrmsrl_safe(x86_pmu_config_addr(idx), 0ull);
wrmsrl_safe(x86_pmu_event_addr(idx), 0ull);
}
for (idx = 0; idx < x86_pmu.num_counters_fixed; idx++) {
if (fixed_counter_disabled(idx))
for (idx = 0; idx < num_counters_fixed; idx++) {
if (fixed_counter_disabled(idx, cpuc->pmu))
continue;
wrmsrl_safe(MSR_ARCH_PERFMON_FIXED_CTR0 + idx, 0ull);
}
@ -2753,6 +2786,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int bit;
int handled = 0;
u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl);
inc_irq_stat(apic_perf_irqs);
@ -2798,7 +2832,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
handled++;
x86_pmu.drain_pebs(regs, &data);
status &= x86_pmu.intel_ctrl | GLOBAL_STATUS_TRACE_TOPAPMI;
status &= intel_ctrl | GLOBAL_STATUS_TRACE_TOPAPMI;
/*
* PMI throttle may be triggered, which stops the PEBS event.
@ -2961,8 +2995,10 @@ intel_vlbr_constraints(struct perf_event *event)
return NULL;
}
static int intel_alt_er(int idx, u64 config)
static int intel_alt_er(struct cpu_hw_events *cpuc,
int idx, u64 config)
{
struct extra_reg *extra_regs = hybrid(cpuc->pmu, extra_regs);
int alt_idx = idx;
if (!(x86_pmu.flags & PMU_FL_HAS_RSP_1))
@ -2974,7 +3010,7 @@ static int intel_alt_er(int idx, u64 config)
if (idx == EXTRA_REG_RSP_1)
alt_idx = EXTRA_REG_RSP_0;
if (config & ~x86_pmu.extra_regs[alt_idx].valid_mask)
if (config & ~extra_regs[alt_idx].valid_mask)
return idx;
return alt_idx;
@ -2982,15 +3018,16 @@ static int intel_alt_er(int idx, u64 config)
static void intel_fixup_er(struct perf_event *event, int idx)
{
struct extra_reg *extra_regs = hybrid(event->pmu, extra_regs);
event->hw.extra_reg.idx = idx;
if (idx == EXTRA_REG_RSP_0) {
event->hw.config &= ~INTEL_ARCH_EVENT_MASK;
event->hw.config |= x86_pmu.extra_regs[EXTRA_REG_RSP_0].event;
event->hw.config |= extra_regs[EXTRA_REG_RSP_0].event;
event->hw.extra_reg.reg = MSR_OFFCORE_RSP_0;
} else if (idx == EXTRA_REG_RSP_1) {
event->hw.config &= ~INTEL_ARCH_EVENT_MASK;
event->hw.config |= x86_pmu.extra_regs[EXTRA_REG_RSP_1].event;
event->hw.config |= extra_regs[EXTRA_REG_RSP_1].event;
event->hw.extra_reg.reg = MSR_OFFCORE_RSP_1;
}
}
@ -3066,7 +3103,7 @@ again:
*/
c = NULL;
} else {
idx = intel_alt_er(idx, reg->config);
idx = intel_alt_er(cpuc, idx, reg->config);
if (idx != reg->idx) {
raw_spin_unlock_irqrestore(&era->lock, flags);
goto again;
@ -3131,10 +3168,11 @@ struct event_constraint *
x86_get_event_constraints(struct cpu_hw_events *cpuc, int idx,
struct perf_event *event)
{
struct event_constraint *event_constraints = hybrid(cpuc->pmu, event_constraints);
struct event_constraint *c;
if (x86_pmu.event_constraints) {
for_each_event_constraint(c, x86_pmu.event_constraints) {
if (event_constraints) {
for_each_event_constraint(c, event_constraints) {
if (constraint_match(c, event->hw.config)) {
event->hw.flags |= c->flags;
return c;
@ -3142,7 +3180,7 @@ x86_get_event_constraints(struct cpu_hw_events *cpuc, int idx,
}
}
return &unconstrained;
return &hybrid_var(cpuc->pmu, unconstrained);
}
static struct event_constraint *
@ -3646,6 +3684,23 @@ static inline bool is_mem_loads_aux_event(struct perf_event *event)
return (event->attr.config & INTEL_ARCH_EVENT_MASK) == X86_CONFIG(.event=0x03, .umask=0x82);
}
static inline bool require_mem_loads_aux_event(struct perf_event *event)
{
if (!(x86_pmu.flags & PMU_FL_MEM_LOADS_AUX))
return false;
if (is_hybrid())
return hybrid_pmu(event->pmu)->cpu_type == hybrid_big;
return true;
}
static inline bool intel_pmu_has_cap(struct perf_event *event, int idx)
{
union perf_capabilities *intel_cap = &hybrid(event->pmu, intel_cap);
return test_bit(idx, (unsigned long *)&intel_cap->capabilities);
}
static int intel_pmu_hw_config(struct perf_event *event)
{
@ -3702,7 +3757,8 @@ static int intel_pmu_hw_config(struct perf_event *event)
event->hw.flags |= PERF_X86_EVENT_PEBS_VIA_PT;
}
if (event->attr.type != PERF_TYPE_RAW)
if ((event->attr.type == PERF_TYPE_HARDWARE) ||
(event->attr.type == PERF_TYPE_HW_CACHE))
return 0;
/*
@ -3715,7 +3771,7 @@ static int intel_pmu_hw_config(struct perf_event *event)
* with a slots event as group leader. When the slots event
* is used in a metrics group, it too cannot support sampling.
*/
if (x86_pmu.intel_cap.perf_metrics && is_topdown_event(event)) {
if (intel_pmu_has_cap(event, PERF_CAP_METRICS_IDX) && is_topdown_event(event)) {
if (event->attr.config1 || event->attr.config2)
return -EINVAL;
@ -3766,7 +3822,7 @@ static int intel_pmu_hw_config(struct perf_event *event)
* event. The rule is to simplify the implementation of the check.
* That's because perf cannot have a complete group at the moment.
*/
if (x86_pmu.flags & PMU_FL_MEM_LOADS_AUX &&
if (require_mem_loads_aux_event(event) &&
(event->attr.sample_type & PERF_SAMPLE_DATA_SRC) &&
is_mem_loads_event(event)) {
struct perf_event *leader = event->group_leader;
@ -3801,10 +3857,11 @@ static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_guest_switch_msr *arr = cpuc->guest_switch_msrs;
u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl);
arr[0].msr = MSR_CORE_PERF_GLOBAL_CTRL;
arr[0].host = x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask;
arr[0].guest = x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_host_mask;
arr[0].host = intel_ctrl & ~cpuc->intel_ctrl_guest_mask;
arr[0].guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask;
if (x86_pmu.flags & PMU_FL_PEBS_ALL)
arr[0].guest &= ~cpuc->pebs_enabled;
else
@ -4042,6 +4099,39 @@ tfa_get_event_constraints(struct cpu_hw_events *cpuc, int idx,
return c;
}
static struct event_constraint *
adl_get_event_constraints(struct cpu_hw_events *cpuc, int idx,
struct perf_event *event)
{
struct x86_hybrid_pmu *pmu = hybrid_pmu(event->pmu);
if (pmu->cpu_type == hybrid_big)
return spr_get_event_constraints(cpuc, idx, event);
else if (pmu->cpu_type == hybrid_small)
return tnt_get_event_constraints(cpuc, idx, event);
WARN_ON(1);
return &emptyconstraint;
}
static int adl_hw_config(struct perf_event *event)
{
struct x86_hybrid_pmu *pmu = hybrid_pmu(event->pmu);
if (pmu->cpu_type == hybrid_big)
return hsw_hw_config(event);
else if (pmu->cpu_type == hybrid_small)
return intel_pmu_hw_config(event);
WARN_ON(1);
return -EOPNOTSUPP;
}
static u8 adl_get_hybrid_cpu_type(void)
{
return hybrid_big;
}
/*
* Broadwell:
*
@ -4145,7 +4235,7 @@ int intel_cpuc_prepare(struct cpu_hw_events *cpuc, int cpu)
{
cpuc->pebs_record_size = x86_pmu.pebs_record_size;
if (x86_pmu.extra_regs || x86_pmu.lbr_sel_map) {
if (is_hybrid() || x86_pmu.extra_regs || x86_pmu.lbr_sel_map) {
cpuc->shared_regs = allocate_shared_regs(cpu);
if (!cpuc->shared_regs)
goto err;
@ -4199,12 +4289,62 @@ static void flip_smm_bit(void *data)
}
}
static bool init_hybrid_pmu(int cpu)
{
struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
u8 cpu_type = get_this_hybrid_cpu_type();
struct x86_hybrid_pmu *pmu = NULL;
int i;
if (!cpu_type && x86_pmu.get_hybrid_cpu_type)
cpu_type = x86_pmu.get_hybrid_cpu_type();
for (i = 0; i < x86_pmu.num_hybrid_pmus; i++) {
if (x86_pmu.hybrid_pmu[i].cpu_type == cpu_type) {
pmu = &x86_pmu.hybrid_pmu[i];
break;
}
}
if (WARN_ON_ONCE(!pmu || (pmu->pmu.type == -1))) {
cpuc->pmu = NULL;
return false;
}
/* Only check and dump the PMU information for the first CPU */
if (!cpumask_empty(&pmu->supported_cpus))
goto end;
if (!check_hw_exists(&pmu->pmu, pmu->num_counters, pmu->num_counters_fixed))
return false;
pr_info("%s PMU driver: ", pmu->name);
if (pmu->intel_cap.pebs_output_pt_available)
pr_cont("PEBS-via-PT ");
pr_cont("\n");
x86_pmu_show_pmu_cap(pmu->num_counters, pmu->num_counters_fixed,
pmu->intel_ctrl);
end:
cpumask_set_cpu(cpu, &pmu->supported_cpus);
cpuc->pmu = &pmu->pmu;
x86_pmu_update_cpu_context(&pmu->pmu, cpu);
return true;
}
static void intel_pmu_cpu_starting(int cpu)
{
struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
int core_id = topology_core_id(cpu);
int i;
if (is_hybrid() && !init_hybrid_pmu(cpu))
return;
init_debug_store_on_cpu(cpu);
/*
* Deal with CPUs that don't clear their LBRs on power-up.
@ -4222,8 +4362,16 @@ static void intel_pmu_cpu_starting(int cpu)
if (x86_pmu.version > 1)
flip_smm_bit(&x86_pmu.attr_freeze_on_smi);
/* Disable perf metrics if any added CPU doesn't support it. */
if (x86_pmu.intel_cap.perf_metrics) {
/*
* Disable perf metrics if any added CPU doesn't support it.
*
* Turn off the check for a hybrid architecture, because the
* architecture MSR, MSR_IA32_PERF_CAPABILITIES, only indicate
* the architecture features. The perf metrics is a model-specific
* feature for now. The corresponding bit should always be 0 on
* a hybrid platform, e.g., Alder Lake.
*/
if (!is_hybrid() && x86_pmu.intel_cap.perf_metrics) {
union perf_capabilities perf_cap;
rdmsrl(MSR_IA32_PERF_CAPABILITIES, perf_cap.capabilities);
@ -4310,7 +4458,12 @@ void intel_cpuc_finish(struct cpu_hw_events *cpuc)
static void intel_pmu_cpu_dead(int cpu)
{
intel_cpuc_finish(&per_cpu(cpu_hw_events, cpu));
struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
intel_cpuc_finish(cpuc);
if (is_hybrid() && cpuc->pmu)
cpumask_clear_cpu(cpu, &hybrid_pmu(cpuc->pmu)->supported_cpus);
}
static void intel_pmu_sched_task(struct perf_event_context *ctx,
@ -4339,6 +4492,14 @@ static int intel_pmu_aux_output_match(struct perf_event *event)
return is_intel_pt_event(event);
}
static int intel_pmu_filter_match(struct perf_event *event)
{
struct x86_hybrid_pmu *pmu = hybrid_pmu(event->pmu);
unsigned int cpu = smp_processor_id();
return cpumask_test_cpu(cpu, &pmu->supported_cpus);
}
PMU_FORMAT_ATTR(offcore_rsp, "config1:0-63");
PMU_FORMAT_ATTR(ldlat, "config1:0-15");
@ -4879,7 +5040,7 @@ static void update_tfa_sched(void *ignored)
* and if so force schedule out for all event types all contexts
*/
if (test_bit(3, cpuc->active_mask))
perf_pmu_resched(x86_get_pmu());
perf_pmu_resched(x86_get_pmu(smp_processor_id()));
}
static ssize_t show_sysctl_tfa(struct device *cdev,
@ -5041,8 +5202,299 @@ static const struct attribute_group *attr_update[] = {
NULL,
};
EVENT_ATTR_STR_HYBRID(slots, slots_adl, "event=0x00,umask=0x4", hybrid_big);
EVENT_ATTR_STR_HYBRID(topdown-retiring, td_retiring_adl, "event=0xc2,umask=0x0;event=0x00,umask=0x80", hybrid_big_small);
EVENT_ATTR_STR_HYBRID(topdown-bad-spec, td_bad_spec_adl, "event=0x73,umask=0x0;event=0x00,umask=0x81", hybrid_big_small);
EVENT_ATTR_STR_HYBRID(topdown-fe-bound, td_fe_bound_adl, "event=0x71,umask=0x0;event=0x00,umask=0x82", hybrid_big_small);
EVENT_ATTR_STR_HYBRID(topdown-be-bound, td_be_bound_adl, "event=0x74,umask=0x0;event=0x00,umask=0x83", hybrid_big_small);
EVENT_ATTR_STR_HYBRID(topdown-heavy-ops, td_heavy_ops_adl, "event=0x00,umask=0x84", hybrid_big);
EVENT_ATTR_STR_HYBRID(topdown-br-mispredict, td_br_mis_adl, "event=0x00,umask=0x85", hybrid_big);
EVENT_ATTR_STR_HYBRID(topdown-fetch-lat, td_fetch_lat_adl, "event=0x00,umask=0x86", hybrid_big);
EVENT_ATTR_STR_HYBRID(topdown-mem-bound, td_mem_bound_adl, "event=0x00,umask=0x87", hybrid_big);
static struct attribute *adl_hybrid_events_attrs[] = {
EVENT_PTR(slots_adl),
EVENT_PTR(td_retiring_adl),
EVENT_PTR(td_bad_spec_adl),
EVENT_PTR(td_fe_bound_adl),
EVENT_PTR(td_be_bound_adl),
EVENT_PTR(td_heavy_ops_adl),
EVENT_PTR(td_br_mis_adl),
EVENT_PTR(td_fetch_lat_adl),
EVENT_PTR(td_mem_bound_adl),
NULL,
};
/* Must be in IDX order */
EVENT_ATTR_STR_HYBRID(mem-loads, mem_ld_adl, "event=0xd0,umask=0x5,ldlat=3;event=0xcd,umask=0x1,ldlat=3", hybrid_big_small);
EVENT_ATTR_STR_HYBRID(mem-stores, mem_st_adl, "event=0xd0,umask=0x6;event=0xcd,umask=0x2", hybrid_big_small);
EVENT_ATTR_STR_HYBRID(mem-loads-aux, mem_ld_aux_adl, "event=0x03,umask=0x82", hybrid_big);
static struct attribute *adl_hybrid_mem_attrs[] = {
EVENT_PTR(mem_ld_adl),
EVENT_PTR(mem_st_adl),
EVENT_PTR(mem_ld_aux_adl),
NULL,
};
EVENT_ATTR_STR_HYBRID(tx-start, tx_start_adl, "event=0xc9,umask=0x1", hybrid_big);
EVENT_ATTR_STR_HYBRID(tx-commit, tx_commit_adl, "event=0xc9,umask=0x2", hybrid_big);
EVENT_ATTR_STR_HYBRID(tx-abort, tx_abort_adl, "event=0xc9,umask=0x4", hybrid_big);
EVENT_ATTR_STR_HYBRID(tx-conflict, tx_conflict_adl, "event=0x54,umask=0x1", hybrid_big);
EVENT_ATTR_STR_HYBRID(cycles-t, cycles_t_adl, "event=0x3c,in_tx=1", hybrid_big);
EVENT_ATTR_STR_HYBRID(cycles-ct, cycles_ct_adl, "event=0x3c,in_tx=1,in_tx_cp=1", hybrid_big);
EVENT_ATTR_STR_HYBRID(tx-capacity-read, tx_capacity_read_adl, "event=0x54,umask=0x80", hybrid_big);
EVENT_ATTR_STR_HYBRID(tx-capacity-write, tx_capacity_write_adl, "event=0x54,umask=0x2", hybrid_big);
static struct attribute *adl_hybrid_tsx_attrs[] = {
EVENT_PTR(tx_start_adl),
EVENT_PTR(tx_abort_adl),
EVENT_PTR(tx_commit_adl),
EVENT_PTR(tx_capacity_read_adl),
EVENT_PTR(tx_capacity_write_adl),
EVENT_PTR(tx_conflict_adl),
EVENT_PTR(cycles_t_adl),
EVENT_PTR(cycles_ct_adl),
NULL,
};
FORMAT_ATTR_HYBRID(in_tx, hybrid_big);
FORMAT_ATTR_HYBRID(in_tx_cp, hybrid_big);
FORMAT_ATTR_HYBRID(offcore_rsp, hybrid_big_small);
FORMAT_ATTR_HYBRID(ldlat, hybrid_big_small);
FORMAT_ATTR_HYBRID(frontend, hybrid_big);
static struct attribute *adl_hybrid_extra_attr_rtm[] = {
FORMAT_HYBRID_PTR(in_tx),
FORMAT_HYBRID_PTR(in_tx_cp),
FORMAT_HYBRID_PTR(offcore_rsp),
FORMAT_HYBRID_PTR(ldlat),
FORMAT_HYBRID_PTR(frontend),
NULL,
};
static struct attribute *adl_hybrid_extra_attr[] = {
FORMAT_HYBRID_PTR(offcore_rsp),
FORMAT_HYBRID_PTR(ldlat),
FORMAT_HYBRID_PTR(frontend),
NULL,
};
static bool is_attr_for_this_pmu(struct kobject *kobj, struct attribute *attr)
{
struct device *dev = kobj_to_dev(kobj);
struct x86_hybrid_pmu *pmu =
container_of(dev_get_drvdata(dev), struct x86_hybrid_pmu, pmu);
struct perf_pmu_events_hybrid_attr *pmu_attr =
container_of(attr, struct perf_pmu_events_hybrid_attr, attr.attr);
return pmu->cpu_type & pmu_attr->pmu_type;
}
static umode_t hybrid_events_is_visible(struct kobject *kobj,
struct attribute *attr, int i)
{
return is_attr_for_this_pmu(kobj, attr) ? attr->mode : 0;
}
static inline int hybrid_find_supported_cpu(struct x86_hybrid_pmu *pmu)
{
int cpu = cpumask_first(&pmu->supported_cpus);
return (cpu >= nr_cpu_ids) ? -1 : cpu;
}
static umode_t hybrid_tsx_is_visible(struct kobject *kobj,
struct attribute *attr, int i)
{
struct device *dev = kobj_to_dev(kobj);
struct x86_hybrid_pmu *pmu =
container_of(dev_get_drvdata(dev), struct x86_hybrid_pmu, pmu);
int cpu = hybrid_find_supported_cpu(pmu);
return (cpu >= 0) && is_attr_for_this_pmu(kobj, attr) && cpu_has(&cpu_data(cpu), X86_FEATURE_RTM) ? attr->mode : 0;
}
static umode_t hybrid_format_is_visible(struct kobject *kobj,
struct attribute *attr, int i)
{
struct device *dev = kobj_to_dev(kobj);
struct x86_hybrid_pmu *pmu =
container_of(dev_get_drvdata(dev), struct x86_hybrid_pmu, pmu);
struct perf_pmu_format_hybrid_attr *pmu_attr =
container_of(attr, struct perf_pmu_format_hybrid_attr, attr.attr);
int cpu = hybrid_find_supported_cpu(pmu);
return (cpu >= 0) && (pmu->cpu_type & pmu_attr->pmu_type) ? attr->mode : 0;
}
static struct attribute_group hybrid_group_events_td = {
.name = "events",
.is_visible = hybrid_events_is_visible,
};
static struct attribute_group hybrid_group_events_mem = {
.name = "events",
.is_visible = hybrid_events_is_visible,
};
static struct attribute_group hybrid_group_events_tsx = {
.name = "events",
.is_visible = hybrid_tsx_is_visible,
};
static struct attribute_group hybrid_group_format_extra = {
.name = "format",
.is_visible = hybrid_format_is_visible,
};
static ssize_t intel_hybrid_get_attr_cpus(struct device *dev,
struct device_attribute *attr,
char *buf)
{
struct x86_hybrid_pmu *pmu =
container_of(dev_get_drvdata(dev), struct x86_hybrid_pmu, pmu);
return cpumap_print_to_pagebuf(true, buf, &pmu->supported_cpus);
}
static DEVICE_ATTR(cpus, S_IRUGO, intel_hybrid_get_attr_cpus, NULL);
static struct attribute *intel_hybrid_cpus_attrs[] = {
&dev_attr_cpus.attr,
NULL,
};
static struct attribute_group hybrid_group_cpus = {
.attrs = intel_hybrid_cpus_attrs,
};
static const struct attribute_group *hybrid_attr_update[] = {
&hybrid_group_events_td,
&hybrid_group_events_mem,
&hybrid_group_events_tsx,
&group_caps_gen,
&group_caps_lbr,
&hybrid_group_format_extra,
&group_default,
&hybrid_group_cpus,
NULL,
};
static struct attribute *empty_attrs;
static void intel_pmu_check_num_counters(int *num_counters,
int *num_counters_fixed,
u64 *intel_ctrl, u64 fixed_mask)
{
if (*num_counters > INTEL_PMC_MAX_GENERIC) {
WARN(1, KERN_ERR "hw perf events %d > max(%d), clipping!",
*num_counters, INTEL_PMC_MAX_GENERIC);
*num_counters = INTEL_PMC_MAX_GENERIC;
}
*intel_ctrl = (1ULL << *num_counters) - 1;
if (*num_counters_fixed > INTEL_PMC_MAX_FIXED) {
WARN(1, KERN_ERR "hw perf events fixed %d > max(%d), clipping!",
*num_counters_fixed, INTEL_PMC_MAX_FIXED);
*num_counters_fixed = INTEL_PMC_MAX_FIXED;
}
*intel_ctrl |= fixed_mask << INTEL_PMC_IDX_FIXED;
}
static void intel_pmu_check_event_constraints(struct event_constraint *event_constraints,
int num_counters,
int num_counters_fixed,
u64 intel_ctrl)
{
struct event_constraint *c;
if (!event_constraints)
return;
/*
* event on fixed counter2 (REF_CYCLES) only works on this
* counter, so do not extend mask to generic counters
*/
for_each_event_constraint(c, event_constraints) {
/*
* Don't extend the topdown slots and metrics
* events to the generic counters.
*/
if (c->idxmsk64 & INTEL_PMC_MSK_TOPDOWN) {
/*
* Disable topdown slots and metrics events,
* if slots event is not in CPUID.
*/
if (!(INTEL_PMC_MSK_FIXED_SLOTS & intel_ctrl))
c->idxmsk64 = 0;
c->weight = hweight64(c->idxmsk64);
continue;
}
if (c->cmask == FIXED_EVENT_FLAGS) {
/* Disabled fixed counters which are not in CPUID */
c->idxmsk64 &= intel_ctrl;
if (c->idxmsk64 != INTEL_PMC_MSK_FIXED_REF_CYCLES)
c->idxmsk64 |= (1ULL << num_counters) - 1;
}
c->idxmsk64 &=
~(~0ULL << (INTEL_PMC_IDX_FIXED + num_counters_fixed));
c->weight = hweight64(c->idxmsk64);
}
}
static void intel_pmu_check_extra_regs(struct extra_reg *extra_regs)
{
struct extra_reg *er;
/*
* Access extra MSR may cause #GP under certain circumstances.
* E.g. KVM doesn't support offcore event
* Check all extra_regs here.
*/
if (!extra_regs)
return;
for (er = extra_regs; er->msr; er++) {
er->extra_msr_access = check_msr(er->msr, 0x11UL);
/* Disable LBR select mapping */
if ((er->idx == EXTRA_REG_LBR) && !er->extra_msr_access)
x86_pmu.lbr_sel_map = NULL;
}
}
static void intel_pmu_check_hybrid_pmus(u64 fixed_mask)
{
struct x86_hybrid_pmu *pmu;
int i;
for (i = 0; i < x86_pmu.num_hybrid_pmus; i++) {
pmu = &x86_pmu.hybrid_pmu[i];
intel_pmu_check_num_counters(&pmu->num_counters,
&pmu->num_counters_fixed,
&pmu->intel_ctrl,
fixed_mask);
if (pmu->intel_cap.perf_metrics) {
pmu->intel_ctrl |= 1ULL << GLOBAL_CTRL_EN_PERF_METRICS;
pmu->intel_ctrl |= INTEL_PMC_MSK_FIXED_SLOTS;
}
if (pmu->intel_cap.pebs_output_pt_available)
pmu->pmu.capabilities |= PERF_PMU_CAP_AUX_OUTPUT;
intel_pmu_check_event_constraints(pmu->event_constraints,
pmu->num_counters,
pmu->num_counters_fixed,
pmu->intel_ctrl);
intel_pmu_check_extra_regs(pmu->extra_regs);
}
}
__init int intel_pmu_init(void)
{
struct attribute **extra_skl_attr = &empty_attrs;
@ -5053,12 +5505,11 @@ __init int intel_pmu_init(void)
union cpuid10_edx edx;
union cpuid10_eax eax;
union cpuid10_ebx ebx;
struct event_constraint *c;
unsigned int fixed_mask;
struct extra_reg *er;
bool pmem = false;
int version, i;
char *name;
struct x86_hybrid_pmu *pmu;
if (!cpu_has(&boot_cpu_data, X86_FEATURE_ARCH_PERFMON)) {
switch (boot_cpu_data.x86) {
@ -5653,6 +6104,99 @@ __init int intel_pmu_init(void)
name = "sapphire_rapids";
break;
case INTEL_FAM6_ALDERLAKE:
case INTEL_FAM6_ALDERLAKE_L:
/*
* Alder Lake has 2 types of CPU, core and atom.
*
* Initialize the common PerfMon capabilities here.
*/
x86_pmu.hybrid_pmu = kcalloc(X86_HYBRID_NUM_PMUS,
sizeof(struct x86_hybrid_pmu),
GFP_KERNEL);
if (!x86_pmu.hybrid_pmu)
return -ENOMEM;
static_branch_enable(&perf_is_hybrid);
x86_pmu.num_hybrid_pmus = X86_HYBRID_NUM_PMUS;
x86_pmu.late_ack = true;
x86_pmu.pebs_aliases = NULL;
x86_pmu.pebs_prec_dist = true;
x86_pmu.pebs_block = true;
x86_pmu.flags |= PMU_FL_HAS_RSP_1;
x86_pmu.flags |= PMU_FL_NO_HT_SHARING;
x86_pmu.flags |= PMU_FL_PEBS_ALL;
x86_pmu.flags |= PMU_FL_INSTR_LATENCY;
x86_pmu.flags |= PMU_FL_MEM_LOADS_AUX;
x86_pmu.lbr_pt_coexist = true;
intel_pmu_pebs_data_source_skl(false);
x86_pmu.num_topdown_events = 8;
x86_pmu.update_topdown_event = adl_update_topdown_event;
x86_pmu.set_topdown_event_period = adl_set_topdown_event_period;
x86_pmu.filter_match = intel_pmu_filter_match;
x86_pmu.get_event_constraints = adl_get_event_constraints;
x86_pmu.hw_config = adl_hw_config;
x86_pmu.limit_period = spr_limit_period;
x86_pmu.get_hybrid_cpu_type = adl_get_hybrid_cpu_type;
/*
* The rtm_abort_event is used to check whether to enable GPRs
* for the RTM abort event. Atom doesn't have the RTM abort
* event. There is no harmful to set it in the common
* x86_pmu.rtm_abort_event.
*/
x86_pmu.rtm_abort_event = X86_CONFIG(.event=0xc9, .umask=0x04);
td_attr = adl_hybrid_events_attrs;
mem_attr = adl_hybrid_mem_attrs;
tsx_attr = adl_hybrid_tsx_attrs;
extra_attr = boot_cpu_has(X86_FEATURE_RTM) ?
adl_hybrid_extra_attr_rtm : adl_hybrid_extra_attr;
/* Initialize big core specific PerfMon capabilities.*/
pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_CORE_IDX];
pmu->name = "cpu_core";
pmu->cpu_type = hybrid_big;
pmu->num_counters = x86_pmu.num_counters + 2;
pmu->num_counters_fixed = x86_pmu.num_counters_fixed + 1;
pmu->max_pebs_events = min_t(unsigned, MAX_PEBS_EVENTS, pmu->num_counters);
pmu->unconstrained = (struct event_constraint)
__EVENT_CONSTRAINT(0, (1ULL << pmu->num_counters) - 1,
0, pmu->num_counters, 0, 0);
pmu->intel_cap.capabilities = x86_pmu.intel_cap.capabilities;
pmu->intel_cap.perf_metrics = 1;
pmu->intel_cap.pebs_output_pt_available = 0;
memcpy(pmu->hw_cache_event_ids, spr_hw_cache_event_ids, sizeof(pmu->hw_cache_event_ids));
memcpy(pmu->hw_cache_extra_regs, spr_hw_cache_extra_regs, sizeof(pmu->hw_cache_extra_regs));
pmu->event_constraints = intel_spr_event_constraints;
pmu->pebs_constraints = intel_spr_pebs_event_constraints;
pmu->extra_regs = intel_spr_extra_regs;
/* Initialize Atom core specific PerfMon capabilities.*/
pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_ATOM_IDX];
pmu->name = "cpu_atom";
pmu->cpu_type = hybrid_small;
pmu->num_counters = x86_pmu.num_counters;
pmu->num_counters_fixed = x86_pmu.num_counters_fixed;
pmu->max_pebs_events = x86_pmu.max_pebs_events;
pmu->unconstrained = (struct event_constraint)
__EVENT_CONSTRAINT(0, (1ULL << pmu->num_counters) - 1,
0, pmu->num_counters, 0, 0);
pmu->intel_cap.capabilities = x86_pmu.intel_cap.capabilities;
pmu->intel_cap.perf_metrics = 0;
pmu->intel_cap.pebs_output_pt_available = 1;
memcpy(pmu->hw_cache_event_ids, glp_hw_cache_event_ids, sizeof(pmu->hw_cache_event_ids));
memcpy(pmu->hw_cache_extra_regs, tnt_hw_cache_extra_regs, sizeof(pmu->hw_cache_extra_regs));
pmu->hw_cache_event_ids[C(ITLB)][C(OP_READ)][C(RESULT_ACCESS)] = -1;
pmu->event_constraints = intel_slm_event_constraints;
pmu->pebs_constraints = intel_grt_pebs_event_constraints;
pmu->extra_regs = intel_grt_extra_regs;
pr_cont("Alderlake Hybrid events, ");
name = "alderlake_hybrid";
break;
default:
switch (x86_pmu.version) {
case 1:
@ -5673,68 +6217,36 @@ __init int intel_pmu_init(void)
snprintf(pmu_name_str, sizeof(pmu_name_str), "%s", name);
if (!is_hybrid()) {
group_events_td.attrs = td_attr;
group_events_mem.attrs = mem_attr;
group_events_tsx.attrs = tsx_attr;
group_format_extra.attrs = extra_attr;
group_format_extra_skl.attrs = extra_skl_attr;
group_events_td.attrs = td_attr;
group_events_mem.attrs = mem_attr;
group_events_tsx.attrs = tsx_attr;
group_format_extra.attrs = extra_attr;
group_format_extra_skl.attrs = extra_skl_attr;
x86_pmu.attr_update = attr_update;
} else {
hybrid_group_events_td.attrs = td_attr;
hybrid_group_events_mem.attrs = mem_attr;
hybrid_group_events_tsx.attrs = tsx_attr;
hybrid_group_format_extra.attrs = extra_attr;
x86_pmu.attr_update = attr_update;
if (x86_pmu.num_counters > INTEL_PMC_MAX_GENERIC) {
WARN(1, KERN_ERR "hw perf events %d > max(%d), clipping!",
x86_pmu.num_counters, INTEL_PMC_MAX_GENERIC);
x86_pmu.num_counters = INTEL_PMC_MAX_GENERIC;
}
x86_pmu.intel_ctrl = (1ULL << x86_pmu.num_counters) - 1;
if (x86_pmu.num_counters_fixed > INTEL_PMC_MAX_FIXED) {
WARN(1, KERN_ERR "hw perf events fixed %d > max(%d), clipping!",
x86_pmu.num_counters_fixed, INTEL_PMC_MAX_FIXED);
x86_pmu.num_counters_fixed = INTEL_PMC_MAX_FIXED;
x86_pmu.attr_update = hybrid_attr_update;
}
x86_pmu.intel_ctrl |= (u64)fixed_mask << INTEL_PMC_IDX_FIXED;
intel_pmu_check_num_counters(&x86_pmu.num_counters,
&x86_pmu.num_counters_fixed,
&x86_pmu.intel_ctrl,
(u64)fixed_mask);
/* AnyThread may be deprecated on arch perfmon v5 or later */
if (x86_pmu.intel_cap.anythread_deprecated)
x86_pmu.format_attrs = intel_arch_formats_attr;
if (x86_pmu.event_constraints) {
/*
* event on fixed counter2 (REF_CYCLES) only works on this
* counter, so do not extend mask to generic counters
*/
for_each_event_constraint(c, x86_pmu.event_constraints) {
/*
* Don't extend the topdown slots and metrics
* events to the generic counters.
*/
if (c->idxmsk64 & INTEL_PMC_MSK_TOPDOWN) {
/*
* Disable topdown slots and metrics events,
* if slots event is not in CPUID.
*/
if (!(INTEL_PMC_MSK_FIXED_SLOTS & x86_pmu.intel_ctrl))
c->idxmsk64 = 0;
c->weight = hweight64(c->idxmsk64);
continue;
}
if (c->cmask == FIXED_EVENT_FLAGS) {
/* Disabled fixed counters which are not in CPUID */
c->idxmsk64 &= x86_pmu.intel_ctrl;
if (c->idxmsk64 != INTEL_PMC_MSK_FIXED_REF_CYCLES)
c->idxmsk64 |= (1ULL << x86_pmu.num_counters) - 1;
}
c->idxmsk64 &=
~(~0ULL << (INTEL_PMC_IDX_FIXED + x86_pmu.num_counters_fixed));
c->weight = hweight64(c->idxmsk64);
}
}
intel_pmu_check_event_constraints(x86_pmu.event_constraints,
x86_pmu.num_counters,
x86_pmu.num_counters_fixed,
x86_pmu.intel_ctrl);
/*
* Access LBR MSR may cause #GP under certain circumstances.
* E.g. KVM doesn't support LBR MSR
@ -5752,19 +6264,7 @@ __init int intel_pmu_init(void)
if (x86_pmu.lbr_nr)
pr_cont("%d-deep LBR, ", x86_pmu.lbr_nr);
/*
* Access extra MSR may cause #GP under certain circumstances.
* E.g. KVM doesn't support offcore event
* Check all extra_regs here.
*/
if (x86_pmu.extra_regs) {
for (er = x86_pmu.extra_regs; er->msr; er++) {
er->extra_msr_access = check_msr(er->msr, 0x11UL);
/* Disable LBR select mapping */
if ((er->idx == EXTRA_REG_LBR) && !er->extra_msr_access)
x86_pmu.lbr_sel_map = NULL;
}
}
intel_pmu_check_extra_regs(x86_pmu.extra_regs);
/* Support full width counters using alternative MSR range */
if (x86_pmu.intel_cap.full_width_write) {
@ -5773,9 +6273,12 @@ __init int intel_pmu_init(void)
pr_cont("full-width counters, ");
}
if (x86_pmu.intel_cap.perf_metrics)
if (!is_hybrid() && x86_pmu.intel_cap.perf_metrics)
x86_pmu.intel_ctrl |= 1ULL << GLOBAL_CTRL_EN_PERF_METRICS;
if (is_hybrid())
intel_pmu_check_hybrid_pmus((u64)fixed_mask);
return 0;
}

39
arch/x86/events/intel/cstate.c
View File

@ -40,7 +40,7 @@
* Model specific counters:
* MSR_CORE_C1_RES: CORE C1 Residency Counter
* perf code: 0x00
* Available model: SLM,AMT,GLM,CNL,TNT
* Available model: SLM,AMT,GLM,CNL,TNT,ADL
* Scope: Core (each processor core has a MSR)
* MSR_CORE_C3_RESIDENCY: CORE C3 Residency Counter
* perf code: 0x01
@ -51,46 +51,49 @@
* perf code: 0x02
* Available model: SLM,AMT,NHM,WSM,SNB,IVB,HSW,BDW,
* SKL,KNL,GLM,CNL,KBL,CML,ICL,TGL,
* TNT,RKL
* TNT,RKL,ADL
* Scope: Core
* MSR_CORE_C7_RESIDENCY: CORE C7 Residency Counter
* perf code: 0x03
* Available model: SNB,IVB,HSW,BDW,SKL,CNL,KBL,CML,
* ICL,TGL,RKL
* ICL,TGL,RKL,ADL
* Scope: Core
* MSR_PKG_C2_RESIDENCY: Package C2 Residency Counter.
* perf code: 0x00
* Available model: SNB,IVB,HSW,BDW,SKL,KNL,GLM,CNL,
* KBL,CML,ICL,TGL,TNT,RKL
* KBL,CML,ICL,TGL,TNT,RKL,ADL
* Scope: Package (physical package)
* MSR_PKG_C3_RESIDENCY: Package C3 Residency Counter.
* perf code: 0x01
* Available model: NHM,WSM,SNB,IVB,HSW,BDW,SKL,KNL,
* GLM,CNL,KBL,CML,ICL,TGL,TNT,RKL
* GLM,CNL,KBL,CML,ICL,TGL,TNT,RKL,
* ADL
* Scope: Package (physical package)
* MSR_PKG_C6_RESIDENCY: Package C6 Residency Counter.
* perf code: 0x02
* Available model: SLM,AMT,NHM,WSM,SNB,IVB,HSW,BDW,
* SKL,KNL,GLM,CNL,KBL,CML,ICL,TGL,
* TNT,RKL
* TNT,RKL,ADL
* Scope: Package (physical package)
* MSR_PKG_C7_RESIDENCY: Package C7 Residency Counter.
* perf code: 0x03
* Available model: NHM,WSM,SNB,IVB,HSW,BDW,SKL,CNL,
* KBL,CML,ICL,TGL,RKL
* KBL,CML,ICL,TGL,RKL,ADL
* Scope: Package (physical package)
* MSR_PKG_C8_RESIDENCY: Package C8 Residency Counter.
* perf code: 0x04
* Available model: HSW ULT,KBL,CNL,CML,ICL,TGL,RKL
* Available model: HSW ULT,KBL,CNL,CML,ICL,TGL,RKL,
* ADL
* Scope: Package (physical package)
* MSR_PKG_C9_RESIDENCY: Package C9 Residency Counter.
* perf code: 0x05
* Available model: HSW ULT,KBL,CNL,CML,ICL,TGL,RKL
* Available model: HSW ULT,KBL,CNL,CML,ICL,TGL,RKL,
* ADL
* Scope: Package (physical package)
* MSR_PKG_C10_RESIDENCY: Package C10 Residency Counter.
* perf code: 0x06
* Available model: HSW ULT,KBL,GLM,CNL,CML,ICL,TGL,
* TNT,RKL
* TNT,RKL,ADL
* Scope: Package (physical package)
*
*/
@ -563,6 +566,20 @@ static const struct cstate_model icl_cstates __initconst = {
BIT(PERF_CSTATE_PKG_C10_RES),
};
static const struct cstate_model adl_cstates __initconst = {
.core_events = BIT(PERF_CSTATE_CORE_C1_RES) |
BIT(PERF_CSTATE_CORE_C6_RES) |
BIT(PERF_CSTATE_CORE_C7_RES),
.pkg_events = BIT(PERF_CSTATE_PKG_C2_RES) |
BIT(PERF_CSTATE_PKG_C3_RES) |
BIT(PERF_CSTATE_PKG_C6_RES) |
BIT(PERF_CSTATE_PKG_C7_RES) |
BIT(PERF_CSTATE_PKG_C8_RES) |
BIT(PERF_CSTATE_PKG_C9_RES) |
BIT(PERF_CSTATE_PKG_C10_RES),
};
static const struct cstate_model slm_cstates __initconst = {
.core_events = BIT(PERF_CSTATE_CORE_C1_RES) |
BIT(PERF_CSTATE_CORE_C6_RES),
@ -650,6 +667,8 @@ static const struct x86_cpu_id intel_cstates_match[] __initconst = {
X86_MATCH_INTEL_FAM6_MODEL(TIGERLAKE_L, &icl_cstates),
X86_MATCH_INTEL_FAM6_MODEL(TIGERLAKE, &icl_cstates),
X86_MATCH_INTEL_FAM6_MODEL(ROCKETLAKE, &icl_cstates),
X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE, &adl_cstates),
X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE_L, &adl_cstates),
{ },
};
MODULE_DEVICE_TABLE(x86cpu, intel_cstates_match);

32
arch/x86/events/intel/ds.c
View File

@ -779,6 +779,13 @@ struct event_constraint intel_glm_pebs_event_constraints[] = {
EVENT_CONSTRAINT_END
};
struct event_constraint intel_grt_pebs_event_constraints[] = {
/* Allow all events as PEBS with no flags */
INTEL_PLD_CONSTRAINT(0x5d0, 0xf),
INTEL_PSD_CONSTRAINT(0x6d0, 0xf),
EVENT_CONSTRAINT_END
};
struct event_constraint intel_nehalem_pebs_event_constraints[] = {
INTEL_PLD_CONSTRAINT(0x100b, 0xf), /* MEM_INST_RETIRED.* */
INTEL_FLAGS_EVENT_CONSTRAINT(0x0f, 0xf), /* MEM_UNCORE_RETIRED.* */
@ -959,13 +966,14 @@ struct event_constraint intel_spr_pebs_event_constraints[] = {
struct event_constraint *intel_pebs_constraints(struct perf_event *event)
{
struct event_constraint *pebs_constraints = hybrid(event->pmu, pebs_constraints);
struct event_constraint *c;
if (!event->attr.precise_ip)
return NULL;
if (x86_pmu.pebs_constraints) {
for_each_event_constraint(c, x86_pmu.pebs_constraints) {
if (pebs_constraints) {
for_each_event_constraint(c, pebs_constraints) {
if (constraint_match(c, event->hw.config)) {
event->hw.flags |= c->flags;
return c;
@ -1007,6 +1015,8 @@ void intel_pmu_pebs_sched_task(struct perf_event_context *ctx, bool sched_in)
static inline void pebs_update_threshold(struct cpu_hw_events *cpuc)
{
struct debug_store *ds = cpuc->ds;
int max_pebs_events = hybrid(cpuc->pmu, max_pebs_events);
int num_counters_fixed = hybrid(cpuc->pmu, num_counters_fixed);
u64 threshold;
int reserved;
@ -1014,9 +1024,9 @@ static inline void pebs_update_threshold(struct cpu_hw_events *cpuc)
return;
if (x86_pmu.flags & PMU_FL_PEBS_ALL)
reserved = x86_pmu.max_pebs_events + x86_pmu.num_counters_fixed;
reserved = max_pebs_events + num_counters_fixed;
else
reserved = x86_pmu.max_pebs_events;
reserved = max_pebs_events;
if (cpuc->n_pebs == cpuc->n_large_pebs) {
threshold = ds->pebs_absolute_maximum -
@ -2071,6 +2081,8 @@ static void intel_pmu_drain_pebs_icl(struct pt_regs *iregs, struct perf_sample_d
{
short counts[INTEL_PMC_IDX_FIXED + MAX_FIXED_PEBS_EVENTS] = {};
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int max_pebs_events = hybrid(cpuc->pmu, max_pebs_events);
int num_counters_fixed = hybrid(cpuc->pmu, num_counters_fixed);
struct debug_store *ds = cpuc->ds;
struct perf_event *event;
void *base, *at, *top;
@ -2085,9 +2097,9 @@ static void intel_pmu_drain_pebs_icl(struct pt_regs *iregs, struct perf_sample_d
ds->pebs_index = ds->pebs_buffer_base;
mask = ((1ULL << x86_pmu.max_pebs_events) - 1) |
(((1ULL << x86_pmu.num_counters_fixed) - 1) << INTEL_PMC_IDX_FIXED);
size = INTEL_PMC_IDX_FIXED + x86_pmu.num_counters_fixed;
mask = ((1ULL << max_pebs_events) - 1) |
(((1ULL << num_counters_fixed) - 1) << INTEL_PMC_IDX_FIXED);
size = INTEL_PMC_IDX_FIXED + num_counters_fixed;
if (unlikely(base >= top)) {
intel_pmu_pebs_event_update_no_drain(cpuc, size);
@ -2191,7 +2203,7 @@ void __init intel_ds_init(void)
PERF_SAMPLE_TIME;
x86_pmu.flags |= PMU_FL_PEBS_ALL;
pebs_qual = "-baseline";
x86_get_pmu()->capabilities |= PERF_PMU_CAP_EXTENDED_REGS;
x86_get_pmu(smp_processor_id())->capabilities |= PERF_PMU_CAP_EXTENDED_REGS;
} else {
/* Only basic record supported */
x86_pmu.large_pebs_flags &=
@ -2204,9 +2216,9 @@ void __init intel_ds_init(void)
}
pr_cont("PEBS fmt4%c%s, ", pebs_type, pebs_qual);
if (x86_pmu.intel_cap.pebs_output_pt_available) {
if (!is_hybrid() && x86_pmu.intel_cap.pebs_output_pt_available) {
pr_cont("PEBS-via-PT, ");
x86_get_pmu()->capabilities |= PERF_PMU_CAP_AUX_OUTPUT;
x86_get_pmu(smp_processor_id())->capabilities |= PERF_PMU_CAP_AUX_OUTPUT;
}
break;

9
arch/x86/events/intel/lbr.c
View File

@ -705,7 +705,7 @@ void intel_pmu_lbr_add(struct perf_event *event)
void release_lbr_buffers(void)
{
struct kmem_cache *kmem_cache = x86_get_pmu()->task_ctx_cache;
struct kmem_cache *kmem_cache;
struct cpu_hw_events *cpuc;
int cpu;
@ -714,6 +714,7 @@ void release_lbr_buffers(void)
for_each_possible_cpu(cpu) {
cpuc = per_cpu_ptr(&cpu_hw_events, cpu);
kmem_cache = x86_get_pmu(cpu)->task_ctx_cache;
if (kmem_cache && cpuc->lbr_xsave) {
kmem_cache_free(kmem_cache, cpuc->lbr_xsave);
cpuc->lbr_xsave = NULL;
@ -1609,7 +1610,7 @@ void intel_pmu_lbr_init_hsw(void)
x86_pmu.lbr_sel_mask = LBR_SEL_MASK;
x86_pmu.lbr_sel_map = hsw_lbr_sel_map;
x86_get_pmu()->task_ctx_cache = create_lbr_kmem_cache(size, 0);
x86_get_pmu(smp_processor_id())->task_ctx_cache = create_lbr_kmem_cache(size, 0);
if (lbr_from_signext_quirk_needed())
static_branch_enable(&lbr_from_quirk_key);
@ -1629,7 +1630,7 @@ __init void intel_pmu_lbr_init_skl(void)
x86_pmu.lbr_sel_mask = LBR_SEL_MASK;
x86_pmu.lbr_sel_map = hsw_lbr_sel_map;
x86_get_pmu()->task_ctx_cache = create_lbr_kmem_cache(size, 0);
x86_get_pmu(smp_processor_id())->task_ctx_cache = create_lbr_kmem_cache(size, 0);
/*
* SW branch filter usage:
@ -1726,7 +1727,7 @@ static bool is_arch_lbr_xsave_available(void)
void __init intel_pmu_arch_lbr_init(void)
{
struct pmu *pmu = x86_get_pmu();
struct pmu *pmu = x86_get_pmu(smp_processor_id());
union cpuid28_eax eax;
union cpuid28_ebx ebx;
union cpuid28_ecx ecx;

16
arch/x86/events/intel/p4.c
View File

@ -947,7 +947,7 @@ static void p4_pmu_enable_pebs(u64 config)
(void)wrmsrl_safe(MSR_P4_PEBS_MATRIX_VERT, (u64)bind->metric_vert);
}
static void p4_pmu_enable_event(struct perf_event *event)
static void __p4_pmu_enable_event(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
int thread = p4_ht_config_thread(hwc->config);
@ -983,6 +983,16 @@ static void p4_pmu_enable_event(struct perf_event *event)
(cccr & ~P4_CCCR_RESERVED) | P4_CCCR_ENABLE);
}
static DEFINE_PER_CPU(unsigned long [BITS_TO_LONGS(X86_PMC_IDX_MAX)], p4_running);
static void p4_pmu_enable_event(struct perf_event *event)
{
int idx = event->hw.idx;
__set_bit(idx, per_cpu(p4_running, smp_processor_id()));
__p4_pmu_enable_event(event);
}
static void p4_pmu_enable_all(int added)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
@ -992,7 +1002,7 @@ static void p4_pmu_enable_all(int added)
struct perf_event *event = cpuc->events[idx];
if (!test_bit(idx, cpuc->active_mask))
continue;
p4_pmu_enable_event(event);
__p4_pmu_enable_event(event);
}
}
@ -1012,7 +1022,7 @@ static int p4_pmu_handle_irq(struct pt_regs *regs)
if (!test_bit(idx, cpuc->active_mask)) {
/* catch in-flight IRQs */
if (__test_and_clear_bit(idx, cpuc->running))
if (__test_and_clear_bit(idx, per_cpu(p4_running, smp_processor_id())))
handled++;
continue;
}

207
arch/x86/events/intel/uncore.c
View File

@ -4,8 +4,13 @@
#include <asm/cpu_device_id.h>
#include <asm/intel-family.h>
#include "uncore.h"
#include "uncore_discovery.h"
static struct intel_uncore_type *empty_uncore[] = { NULL, };
static bool uncore_no_discover;
module_param(uncore_no_discover, bool, 0);
MODULE_PARM_DESC(uncore_no_discover, "Don't enable the Intel uncore PerfMon discovery mechanism "
"(default: enable the discovery mechanism).");
struct intel_uncore_type *empty_uncore[] = { NULL, };
struct intel_uncore_type **uncore_msr_uncores = empty_uncore;
struct intel_uncore_type **uncore_pci_uncores = empty_uncore;
struct intel_uncore_type **uncore_mmio_uncores = empty_uncore;
@ -48,6 +53,18 @@ int uncore_pcibus_to_dieid(struct pci_bus *bus)
return die_id;
}
int uncore_die_to_segment(int die)
{
struct pci_bus *bus = NULL;
/* Find first pci bus which attributes to specified die. */
while ((bus = pci_find_next_bus(bus)) &&
(die != uncore_pcibus_to_dieid(bus)))
;
return bus ? pci_domain_nr(bus) : -EINVAL;
}
static void uncore_free_pcibus_map(void)
{
struct pci2phy_map *map, *tmp;
@ -829,6 +846,34 @@ static const struct attribute_group uncore_pmu_attr_group = {
.attrs = uncore_pmu_attrs,
};
static void uncore_get_pmu_name(struct intel_uncore_pmu *pmu)
{
struct intel_uncore_type *type = pmu->type;
/*
* No uncore block name in discovery table.
* Use uncore_type_&typeid_&boxid as name.
*/
if (!type->name) {
if (type->num_boxes == 1)
sprintf(pmu->name, "uncore_type_%u", type->type_id);
else {
sprintf(pmu->name, "uncore_type_%u_%d",
type->type_id, type->box_ids[pmu->pmu_idx]);
}
return;
}
if (type->num_boxes == 1) {
if (strlen(type->name) > 0)
sprintf(pmu->name, "uncore_%s", type->name);
else
sprintf(pmu->name, "uncore");
} else
sprintf(pmu->name, "uncore_%s_%d", type->name, pmu->pmu_idx);
}
static int uncore_pmu_register(struct intel_uncore_pmu *pmu)
{
int ret;
@ -855,15 +900,7 @@ static int uncore_pmu_register(struct intel_uncore_pmu *pmu)
pmu->pmu.attr_update = pmu->type->attr_update;
}
if (pmu->type->num_boxes == 1) {
if (strlen(pmu->type->name) > 0)
sprintf(pmu->name, "uncore_%s", pmu->type->name);
else
sprintf(pmu->name, "uncore");
} else {
sprintf(pmu->name, "uncore_%s_%d", pmu->type->name,
pmu->pmu_idx);
}
uncore_get_pmu_name(pmu);
ret = perf_pmu_register(&pmu->pmu, pmu->name, -1);
if (!ret)
@ -904,6 +941,10 @@ static void uncore_type_exit(struct intel_uncore_type *type)
kfree(type->pmus);
type->pmus = NULL;
}
if (type->box_ids) {
kfree(type->box_ids);
type->box_ids = NULL;
}
kfree(type->events_group);
type->events_group = NULL;
}
@ -1003,10 +1044,37 @@ static int uncore_pci_get_dev_die_info(struct pci_dev *pdev, int *die)
return 0;
}
static struct intel_uncore_pmu *
uncore_pci_find_dev_pmu_from_types(struct pci_dev *pdev)
{
struct intel_uncore_type **types = uncore_pci_uncores;
struct intel_uncore_type *type;
u64 box_ctl;
int i, die;
for (; *types; types++) {
type = *types;
for (die = 0; die < __uncore_max_dies; die++) {
for (i = 0; i < type->num_boxes; i++) {
if (!type->box_ctls[die])
continue;
box_ctl = type->box_ctls[die] + type->pci_offsets[i];
if (pdev->devfn == UNCORE_DISCOVERY_PCI_DEVFN(box_ctl) &&
pdev->bus->number == UNCORE_DISCOVERY_PCI_BUS(box_ctl) &&
pci_domain_nr(pdev->bus) == UNCORE_DISCOVERY_PCI_DOMAIN(box_ctl))
return &type->pmus[i];
}
}
}
return NULL;
}
/*
* Find the PMU of a PCI device.
* @pdev: The PCI device.
* @ids: The ID table of the available PCI devices with a PMU.
* If NULL, search the whole uncore_pci_uncores.
*/
static struct intel_uncore_pmu *
uncore_pci_find_dev_pmu(struct pci_dev *pdev, const struct pci_device_id *ids)
@ -1016,6 +1084,9 @@ uncore_pci_find_dev_pmu(struct pci_dev *pdev, const struct pci_device_id *ids)
kernel_ulong_t data;
unsigned int devfn;
if (!ids)
return uncore_pci_find_dev_pmu_from_types(pdev);
while (ids && ids->vendor) {
if ((ids->vendor == pdev->vendor) &&
(ids->device == pdev->device)) {
@ -1174,7 +1245,8 @@ static void uncore_pci_remove(struct pci_dev *pdev)
}
static int uncore_bus_notify(struct notifier_block *nb,
unsigned long action, void *data)
unsigned long action, void *data,
const struct pci_device_id *ids)
{
struct device *dev = data;
struct pci_dev *pdev = to_pci_dev(dev);
@ -1185,7 +1257,7 @@ static int uncore_bus_notify(struct notifier_block *nb,
if (action != BUS_NOTIFY_DEL_DEVICE)
return NOTIFY_DONE;
pmu = uncore_pci_find_dev_pmu(pdev, uncore_pci_sub_driver->id_table);
pmu = uncore_pci_find_dev_pmu(pdev, ids);
if (!pmu)
return NOTIFY_DONE;
@ -1197,8 +1269,15 @@ static int uncore_bus_notify(struct notifier_block *nb,
return NOTIFY_OK;
}
static struct notifier_block uncore_notifier = {
.notifier_call = uncore_bus_notify,
static int uncore_pci_sub_bus_notify(struct notifier_block *nb,
unsigned long action, void *data)
{
return uncore_bus_notify(nb, action, data,
uncore_pci_sub_driver->id_table);
}
static struct notifier_block uncore_pci_sub_notifier = {
.notifier_call = uncore_pci_sub_bus_notify,
};
static void uncore_pci_sub_driver_init(void)
@ -1239,13 +1318,55 @@ static void uncore_pci_sub_driver_init(void)
ids++;
}
if (notify && bus_register_notifier(&pci_bus_type, &uncore_notifier))
if (notify && bus_register_notifier(&pci_bus_type, &uncore_pci_sub_notifier))
notify = false;
if (!notify)
uncore_pci_sub_driver = NULL;
}
static int uncore_pci_bus_notify(struct notifier_block *nb,
unsigned long action, void *data)
{
return uncore_bus_notify(nb, action, data, NULL);
}
static struct notifier_block uncore_pci_notifier = {
.notifier_call = uncore_pci_bus_notify,
};
static void uncore_pci_pmus_register(void)
{
struct intel_uncore_type **types = uncore_pci_uncores;
struct intel_uncore_type *type;
struct intel_uncore_pmu *pmu;
struct pci_dev *pdev;
u64 box_ctl;
int i, die;
for (; *types; types++) {
type = *types;
for (die = 0; die < __uncore_max_dies; die++) {
for (i = 0; i < type->num_boxes; i++) {
if (!type->box_ctls[die])
continue;
box_ctl = type->box_ctls[die] + type->pci_offsets[i];
pdev = pci_get_domain_bus_and_slot(UNCORE_DISCOVERY_PCI_DOMAIN(box_ctl),
UNCORE_DISCOVERY_PCI_BUS(box_ctl),
UNCORE_DISCOVERY_PCI_DEVFN(box_ctl));
if (!pdev)
continue;
pmu = &type->pmus[i];
uncore_pci_pmu_register(pdev, type, pmu, die);
}
}
}
bus_register_notifier(&pci_bus_type, &uncore_pci_notifier);
}
static int __init uncore_pci_init(void)
{
size_t size;
@ -1262,12 +1383,15 @@ static int __init uncore_pci_init(void)
if (ret)
goto errtype;
uncore_pci_driver->probe = uncore_pci_probe;
uncore_pci_driver->remove = uncore_pci_remove;
if (uncore_pci_driver) {
uncore_pci_driver->probe = uncore_pci_probe;
uncore_pci_driver->remove = uncore_pci_remove;
ret = pci_register_driver(uncore_pci_driver);
if (ret)
goto errtype;
ret = pci_register_driver(uncore_pci_driver);
if (ret)
goto errtype;
} else
uncore_pci_pmus_register();
if (uncore_pci_sub_driver)
uncore_pci_sub_driver_init();
@ -1290,8 +1414,11 @@ static void uncore_pci_exit(void)
if (pcidrv_registered) {
pcidrv_registered = false;
if (uncore_pci_sub_driver)
bus_unregister_notifier(&pci_bus_type, &uncore_notifier);
pci_unregister_driver(uncore_pci_driver);
bus_unregister_notifier(&pci_bus_type, &uncore_pci_sub_notifier);
if (uncore_pci_driver)
pci_unregister_driver(uncore_pci_driver);
else
bus_unregister_notifier(&pci_bus_type, &uncore_pci_notifier);
uncore_types_exit(uncore_pci_uncores);
kfree(uncore_extra_pci_dev);
uncore_free_pcibus_map();
@ -1625,6 +1752,11 @@ static const struct intel_uncore_init_fun rkl_uncore_init __initconst = {
.pci_init = skl_uncore_pci_init,
};
static const struct intel_uncore_init_fun adl_uncore_init __initconst = {
.cpu_init = adl_uncore_cpu_init,
.mmio_init = tgl_uncore_mmio_init,
};
static const struct intel_uncore_init_fun icx_uncore_init __initconst = {
.cpu_init = icx_uncore_cpu_init,
.pci_init = icx_uncore_pci_init,
@ -1637,6 +1769,12 @@ static const struct intel_uncore_init_fun snr_uncore_init __initconst = {
.mmio_init = snr_uncore_mmio_init,
};
static const struct intel_uncore_init_fun generic_uncore_init __initconst = {
.cpu_init = intel_uncore_generic_uncore_cpu_init,
.pci_init = intel_uncore_generic_uncore_pci_init,
.mmio_init = intel_uncore_generic_uncore_mmio_init,
};
static const struct x86_cpu_id intel_uncore_match[] __initconst = {
X86_MATCH_INTEL_FAM6_MODEL(NEHALEM_EP, &nhm_uncore_init),
X86_MATCH_INTEL_FAM6_MODEL(NEHALEM, &nhm_uncore_init),
@ -1673,6 +1811,8 @@ static const struct x86_cpu_id intel_uncore_match[] __initconst = {
X86_MATCH_INTEL_FAM6_MODEL(TIGERLAKE_L, &tgl_l_uncore_init),
X86_MATCH_INTEL_FAM6_MODEL(TIGERLAKE, &tgl_uncore_init),
X86_MATCH_INTEL_FAM6_MODEL(ROCKETLAKE, &rkl_uncore_init),
X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE, &adl_uncore_init),
X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE_L, &adl_uncore_init),
X86_MATCH_INTEL_FAM6_MODEL(ATOM_TREMONT_D, &snr_uncore_init),
{},
};
@ -1684,17 +1824,21 @@ static int __init intel_uncore_init(void)
struct intel_uncore_init_fun *uncore_init;
int pret = 0, cret = 0, mret = 0, ret;
id = x86_match_cpu(intel_uncore_match);
if (!id)
return -ENODEV;
if (boot_cpu_has(X86_FEATURE_HYPERVISOR))
return -ENODEV;
__uncore_max_dies =
topology_max_packages() * topology_max_die_per_package();
uncore_init = (struct intel_uncore_init_fun *)id->driver_data;
id = x86_match_cpu(intel_uncore_match);
if (!id) {
if (!uncore_no_discover && intel_uncore_has_discovery_tables())
uncore_init = (struct intel_uncore_init_fun *)&generic_uncore_init;
else
return -ENODEV;
} else
uncore_init = (struct intel_uncore_init_fun *)id->driver_data;
if (uncore_init->pci_init) {
pret = uncore_init->pci_init();
if (!pret)
@ -1711,8 +1855,10 @@ static int __init intel_uncore_init(void)
mret = uncore_mmio_init();
}
if (cret && pret && mret)
return -ENODEV;
if (cret && pret && mret) {
ret = -ENODEV;
goto free_discovery;
}
/* Install hotplug callbacks to setup the targets for each package */
ret = cpuhp_setup_state(CPUHP_AP_PERF_X86_UNCORE_ONLINE,
@ -1727,6 +1873,8 @@ err:
uncore_types_exit(uncore_msr_uncores);
uncore_types_exit(uncore_mmio_uncores);
uncore_pci_exit();
free_discovery:
intel_uncore_clear_discovery_tables();
return ret;
}
module_init(intel_uncore_init);
@ -1737,5 +1885,6 @@ static void __exit intel_uncore_exit(void)
uncore_types_exit(uncore_msr_uncores);
uncore_types_exit(uncore_mmio_uncores);
uncore_pci_exit();
intel_uncore_clear_discovery_tables();
}
module_exit(intel_uncore_exit);

20
arch/x86/events/intel/uncore.h
View File

@ -42,6 +42,7 @@ struct intel_uncore_pmu;
struct intel_uncore_box;
struct uncore_event_desc;
struct freerunning_counters;
struct intel_uncore_topology;
struct intel_uncore_type {
const char *name;
@ -50,6 +51,7 @@ struct intel_uncore_type {
int perf_ctr_bits;
int fixed_ctr_bits;
int num_freerunning_types;
int type_id;
unsigned perf_ctr;
unsigned event_ctl;
unsigned event_mask;
@ -57,6 +59,7 @@ struct intel_uncore_type {
unsigned fixed_ctr;
unsigned fixed_ctl;
unsigned box_ctl;
u64 *box_ctls; /* Unit ctrl addr of the first box of each die */
union {
unsigned msr_offset;
unsigned mmio_offset;
@ -65,7 +68,12 @@ struct intel_uncore_type {
unsigned num_shared_regs:8;
unsigned single_fixed:1;
unsigned pair_ctr_ctl:1;
unsigned *msr_offsets;
union {
unsigned *msr_offsets;
unsigned *pci_offsets;
unsigned *mmio_offsets;
};
unsigned *box_ids;
struct event_constraint unconstrainted;
struct event_constraint *constraints;
struct intel_uncore_pmu *pmus;
@ -80,7 +88,7 @@ struct intel_uncore_type {
* to identify which platform component each PMON block of that type is
* supposed to monitor.
*/
u64 *topology;
struct intel_uncore_topology *topology;
/*
* Optional callbacks for managing mapping of Uncore units to PMONs
*/
@ -169,6 +177,11 @@ struct freerunning_counters {
unsigned *box_offsets;
};
struct intel_uncore_topology {
u64 configuration;
int segment;
};
struct pci2phy_map {
struct list_head list;
int segment;
@ -177,6 +190,7 @@ struct pci2phy_map {
struct pci2phy_map *__find_pci2phy_map(int segment);
int uncore_pcibus_to_dieid(struct pci_bus *bus);
int uncore_die_to_segment(int die);
ssize_t uncore_event_show(struct device *dev,
struct device_attribute *attr, char *buf);
@ -547,6 +561,7 @@ uncore_get_constraint(struct intel_uncore_box *box, struct perf_event *event);
void uncore_put_constraint(struct intel_uncore_box *box, struct perf_event *event);
u64 uncore_shared_reg_config(struct intel_uncore_box *box, int idx);
extern struct intel_uncore_type *empty_uncore[];
extern struct intel_uncore_type **uncore_msr_uncores;
extern struct intel_uncore_type **uncore_pci_uncores;
extern struct intel_uncore_type **uncore_mmio_uncores;
@ -567,6 +582,7 @@ void snb_uncore_cpu_init(void);
void nhm_uncore_cpu_init(void);
void skl_uncore_cpu_init(void);
void icl_uncore_cpu_init(void);
void adl_uncore_cpu_init(void);
void tgl_uncore_cpu_init(void);
void tgl_uncore_mmio_init(void);
void tgl_l_uncore_mmio_init(void);

622
arch/x86/events/intel/uncore_discovery.c Normal file
View File

@ -0,0 +1,622 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Support Intel uncore PerfMon discovery mechanism.
* Copyright(c) 2021 Intel Corporation.
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include "uncore.h"
#include "uncore_discovery.h"
static struct rb_root discovery_tables = RB_ROOT;
static int num_discovered_types[UNCORE_ACCESS_MAX];
static bool has_generic_discovery_table(void)
{
struct pci_dev *dev;
int dvsec;
dev = pci_get_device(PCI_VENDOR_ID_INTEL, UNCORE_DISCOVERY_TABLE_DEVICE, NULL);
if (!dev)
return false;
/* A discovery table device has the unique capability ID. */
dvsec = pci_find_next_ext_capability(dev, 0, UNCORE_EXT_CAP_ID_DISCOVERY);
pci_dev_put(dev);
if (dvsec)
return true;
return false;
}
static int logical_die_id;
static int get_device_die_id(struct pci_dev *dev)
{
int cpu, node = pcibus_to_node(dev->bus);
/*
* If the NUMA info is not available, assume that the logical die id is
* continuous in the order in which the discovery table devices are
* detected.
*/
if (node < 0)
return logical_die_id++;
for_each_cpu(cpu, cpumask_of_node(node)) {
struct cpuinfo_x86 *c = &cpu_data(cpu);
if (c->initialized && cpu_to_node(cpu) == node)
return c->logical_die_id;
}
/*
* All CPUs of a node may be offlined. For this case,
* the PCI and MMIO type of uncore blocks which are
* enumerated by the device will be unavailable.
*/
return -1;
}
#define __node_2_type(cur) \
rb_entry((cur), struct intel_uncore_discovery_type, node)
static inline int __type_cmp(const void *key, const struct rb_node *b)
{
struct intel_uncore_discovery_type *type_b = __node_2_type(b);
const u16 *type_id = key;
if (type_b->type > *type_id)
return -1;
else if (type_b->type < *type_id)
return 1;
return 0;
}
static inline struct intel_uncore_discovery_type *
search_uncore_discovery_type(u16 type_id)
{
struct rb_node *node = rb_find(&type_id, &discovery_tables, __type_cmp);
return (node) ? __node_2_type(node) : NULL;
}
static inline bool __type_less(struct rb_node *a, const struct rb_node *b)
{
return (__node_2_type(a)->type < __node_2_type(b)->type);
}
static struct intel_uncore_discovery_type *
add_uncore_discovery_type(struct uncore_unit_discovery *unit)
{
struct intel_uncore_discovery_type *type;
if (unit->access_type >= UNCORE_ACCESS_MAX) {
pr_warn("Unsupported access type %d\n", unit->access_type);
return NULL;
}
type = kzalloc(sizeof(struct intel_uncore_discovery_type), GFP_KERNEL);
if (!type)
return NULL;
type->box_ctrl_die = kcalloc(__uncore_max_dies, sizeof(u64), GFP_KERNEL);
if (!type->box_ctrl_die)
goto free_type;
type->access_type = unit->access_type;
num_discovered_types[type->access_type]++;
type->type = unit->box_type;
rb_add(&type->node, &discovery_tables, __type_less);
return type;
free_type:
kfree(type);
return NULL;
}
static struct intel_uncore_discovery_type *
get_uncore_discovery_type(struct uncore_unit_discovery *unit)
{
struct intel_uncore_discovery_type *type;
type = search_uncore_discovery_type(unit->box_type);
if (type)
return type;
return add_uncore_discovery_type(unit);
}
static void
uncore_insert_box_info(struct uncore_unit_discovery *unit,
int die, bool parsed)
{
struct intel_uncore_discovery_type *type;
unsigned int *box_offset, *ids;
int i;
if (WARN_ON_ONCE(!unit->ctl || !unit->ctl_offset || !unit->ctr_offset))
return;
if (parsed) {
type = search_uncore_discovery_type(unit->box_type);
if (WARN_ON_ONCE(!type))
return;
/* Store the first box of each die */
if (!type->box_ctrl_die[die])
type->box_ctrl_die[die] = unit->ctl;
return;
}
type = get_uncore_discovery_type(unit);
if (!type)
return;
box_offset = kcalloc(type->num_boxes + 1, sizeof(unsigned int), GFP_KERNEL);
if (!box_offset)
return;
ids = kcalloc(type->num_boxes + 1, sizeof(unsigned int), GFP_KERNEL);
if (!ids)
goto free_box_offset;
/* Store generic information for the first box */
if (!type->num_boxes) {
type->box_ctrl = unit->ctl;
type->box_ctrl_die[die] = unit->ctl;
type->num_counters = unit->num_regs;
type->counter_width = unit->bit_width;
type->ctl_offset = unit->ctl_offset;
type->ctr_offset = unit->ctr_offset;
*ids = unit->box_id;
goto end;
}
for (i = 0; i < type->num_boxes; i++) {
ids[i] = type->ids[i];
box_offset[i] = type->box_offset[i];
if (WARN_ON_ONCE(unit->box_id == ids[i]))
goto free_ids;
}
ids[i] = unit->box_id;
box_offset[i] = unit->ctl - type->box_ctrl;
kfree(type->ids);
kfree(type->box_offset);
end:
type->ids = ids;
type->box_offset = box_offset;
type->num_boxes++;
return;
free_ids:
kfree(ids);
free_box_offset:
kfree(box_offset);
}
static int parse_discovery_table(struct pci_dev *dev, int die,
u32 bar_offset, bool *parsed)
{
struct uncore_global_discovery global;
struct uncore_unit_discovery unit;
void __iomem *io_addr;
resource_size_t addr;
unsigned long size;
u32 val;
int i;
pci_read_config_dword(dev, bar_offset, &val);
if (val & UNCORE_DISCOVERY_MASK)
return -EINVAL;
addr = (resource_size_t)(val & ~UNCORE_DISCOVERY_MASK);
size = UNCORE_DISCOVERY_GLOBAL_MAP_SIZE;
io_addr = ioremap(addr, size);
if (!io_addr)
return -ENOMEM;
/* Read Global Discovery State */
memcpy_fromio(&global, io_addr, sizeof(struct uncore_global_discovery));
if (uncore_discovery_invalid_unit(global)) {
pr_info("Invalid Global Discovery State: 0x%llx 0x%llx 0x%llx\n",
global.table1, global.ctl, global.table3);
iounmap(io_addr);
return -EINVAL;
}
iounmap(io_addr);
size = (1 + global.max_units) * global.stride * 8;
io_addr = ioremap(addr, size);
if (!io_addr)
return -ENOMEM;
/* Parsing Unit Discovery State */
for (i = 0; i < global.max_units; i++) {
memcpy_fromio(&unit, io_addr + (i + 1) * (global.stride * 8),
sizeof(struct uncore_unit_discovery));
if (uncore_discovery_invalid_unit(unit))
continue;
if (unit.access_type >= UNCORE_ACCESS_MAX)
continue;
uncore_insert_box_info(&unit, die, *parsed);
}
*parsed = true;
iounmap(io_addr);
return 0;
}
bool intel_uncore_has_discovery_tables(void)
{
u32 device, val, entry_id, bar_offset;
int die, dvsec = 0, ret = true;
struct pci_dev *dev = NULL;
bool parsed = false;
if (has_generic_discovery_table())
device = UNCORE_DISCOVERY_TABLE_DEVICE;
else
device = PCI_ANY_ID;
/*
* Start a new search and iterates through the list of
* the discovery table devices.
*/
while ((dev = pci_get_device(PCI_VENDOR_ID_INTEL, device, dev)) != NULL) {
while ((dvsec = pci_find_next_ext_capability(dev, dvsec, UNCORE_EXT_CAP_ID_DISCOVERY))) {
pci_read_config_dword(dev, dvsec + UNCORE_DISCOVERY_DVSEC_OFFSET, &val);
entry_id = val & UNCORE_DISCOVERY_DVSEC_ID_MASK;
if (entry_id != UNCORE_DISCOVERY_DVSEC_ID_PMON)
continue;
pci_read_config_dword(dev, dvsec + UNCORE_DISCOVERY_DVSEC2_OFFSET, &val);
if (val & ~UNCORE_DISCOVERY_DVSEC2_BIR_MASK) {
ret = false;
goto err;
}
bar_offset = UNCORE_DISCOVERY_BIR_BASE +
(val & UNCORE_DISCOVERY_DVSEC2_BIR_MASK) * UNCORE_DISCOVERY_BIR_STEP;
die = get_device_die_id(dev);
if (die < 0)
continue;
parse_discovery_table(dev, die, bar_offset, &parsed);
}
}
/* None of the discovery tables are available */
if (!parsed)
ret = false;
err:
pci_dev_put(dev);
return ret;
}
void intel_uncore_clear_discovery_tables(void)
{
struct intel_uncore_discovery_type *type, *next;
rbtree_postorder_for_each_entry_safe(type, next, &discovery_tables, node) {
kfree(type->box_ctrl_die);
kfree(type);
}
}
DEFINE_UNCORE_FORMAT_ATTR(event, event, "config:0-7");
DEFINE_UNCORE_FORMAT_ATTR(umask, umask, "config:8-15");
DEFINE_UNCORE_FORMAT_ATTR(edge, edge, "config:18");
DEFINE_UNCORE_FORMAT_ATTR(inv, inv, "config:23");
DEFINE_UNCORE_FORMAT_ATTR(thresh, thresh, "config:24-31");
static struct attribute *generic_uncore_formats_attr[] = {
&format_attr_event.attr,
&format_attr_umask.attr,
&format_attr_edge.attr,
&format_attr_inv.attr,
&format_attr_thresh.attr,
NULL,
};
static const struct attribute_group generic_uncore_format_group = {
.name = "format",
.attrs = generic_uncore_formats_attr,
};
static void intel_generic_uncore_msr_init_box(struct intel_uncore_box *box)
{
wrmsrl(uncore_msr_box_ctl(box), GENERIC_PMON_BOX_CTL_INT);
}
static void intel_generic_uncore_msr_disable_box(struct intel_uncore_box *box)
{
wrmsrl(uncore_msr_box_ctl(box), GENERIC_PMON_BOX_CTL_FRZ);
}
static void intel_generic_uncore_msr_enable_box(struct intel_uncore_box *box)
{
wrmsrl(uncore_msr_box_ctl(box), 0);
}
static void intel_generic_uncore_msr_enable_event(struct intel_uncore_box *box,
struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
wrmsrl(hwc->config_base, hwc->config);
}
static void intel_generic_uncore_msr_disable_event(struct intel_uncore_box *box,
struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
wrmsrl(hwc->config_base, 0);
}
static struct intel_uncore_ops generic_uncore_msr_ops = {
.init_box = intel_generic_uncore_msr_init_box,
.disable_box = intel_generic_uncore_msr_disable_box,
.enable_box = intel_generic_uncore_msr_enable_box,
.disable_event = intel_generic_uncore_msr_disable_event,
.enable_event = intel_generic_uncore_msr_enable_event,
.read_counter = uncore_msr_read_counter,
};
static void intel_generic_uncore_pci_init_box(struct intel_uncore_box *box)
{
struct pci_dev *pdev = box->pci_dev;
int box_ctl = uncore_pci_box_ctl(box);
__set_bit(UNCORE_BOX_FLAG_CTL_OFFS8, &box->flags);
pci_write_config_dword(pdev, box_ctl, GENERIC_PMON_BOX_CTL_INT);
}
static void intel_generic_uncore_pci_disable_box(struct intel_uncore_box *box)
{
struct pci_dev *pdev = box->pci_dev;
int box_ctl = uncore_pci_box_ctl(box);
pci_write_config_dword(pdev, box_ctl, GENERIC_PMON_BOX_CTL_FRZ);
}
static void intel_generic_uncore_pci_enable_box(struct intel_uncore_box *box)
{
struct pci_dev *pdev = box->pci_dev;
int box_ctl = uncore_pci_box_ctl(box);
pci_write_config_dword(pdev, box_ctl, 0);
}
static void intel_generic_uncore_pci_enable_event(struct intel_uncore_box *box,
struct perf_event *event)
{
struct pci_dev *pdev = box->pci_dev;
struct hw_perf_event *hwc = &event->hw;
pci_write_config_dword(pdev, hwc->config_base, hwc->config);
}
static void intel_generic_uncore_pci_disable_event(struct intel_uncore_box *box,
struct perf_event *event)
{
struct pci_dev *pdev = box->pci_dev;
struct hw_perf_event *hwc = &event->hw;
pci_write_config_dword(pdev, hwc->config_base, 0);
}
static u64 intel_generic_uncore_pci_read_counter(struct intel_uncore_box *box,
struct perf_event *event)
{
struct pci_dev *pdev = box->pci_dev;
struct hw_perf_event *hwc = &event->hw;
u64 count = 0;
pci_read_config_dword(pdev, hwc->event_base, (u32 *)&count);
pci_read_config_dword(pdev, hwc->event_base + 4, (u32 *)&count + 1);
return count;
}
static struct intel_uncore_ops generic_uncore_pci_ops = {
.init_box = intel_generic_uncore_pci_init_box,
.disable_box = intel_generic_uncore_pci_disable_box,
.enable_box = intel_generic_uncore_pci_enable_box,
.disable_event = intel_generic_uncore_pci_disable_event,
.enable_event = intel_generic_uncore_pci_enable_event,
.read_counter = intel_generic_uncore_pci_read_counter,
};
#define UNCORE_GENERIC_MMIO_SIZE 0x4000
static unsigned int generic_uncore_mmio_box_ctl(struct intel_uncore_box *box)
{
struct intel_uncore_type *type = box->pmu->type;
if (!type->box_ctls || !type->box_ctls[box->dieid] || !type->mmio_offsets)
return 0;
return type->box_ctls[box->dieid] + type->mmio_offsets[box->pmu->pmu_idx];
}
static void intel_generic_uncore_mmio_init_box(struct intel_uncore_box *box)
{
unsigned int box_ctl = generic_uncore_mmio_box_ctl(box);
struct intel_uncore_type *type = box->pmu->type;
resource_size_t addr;
if (!box_ctl) {
pr_warn("Uncore type %d box %d: Invalid box control address.\n",
type->type_id, type->box_ids[box->pmu->pmu_idx]);
return;
}
addr = box_ctl;
box->io_addr = ioremap(addr, UNCORE_GENERIC_MMIO_SIZE);
if (!box->io_addr) {
pr_warn("Uncore type %d box %d: ioremap error for 0x%llx.\n",
type->type_id, type->box_ids[box->pmu->pmu_idx],
(unsigned long long)addr);
return;
}
writel(GENERIC_PMON_BOX_CTL_INT, box->io_addr);
}
static void intel_generic_uncore_mmio_disable_box(struct intel_uncore_box *box)
{
if (!box->io_addr)
return;
writel(GENERIC_PMON_BOX_CTL_FRZ, box->io_addr);
}
static void intel_generic_uncore_mmio_enable_box(struct intel_uncore_box *box)
{
if (!box->io_addr)
return;
writel(0, box->io_addr);
}
static void intel_generic_uncore_mmio_enable_event(struct intel_uncore_box *box,
struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
if (!box->io_addr)
return;
writel(hwc->config, box->io_addr + hwc->config_base);
}
static void intel_generic_uncore_mmio_disable_event(struct intel_uncore_box *box,
struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
if (!box->io_addr)
return;
writel(0, box->io_addr + hwc->config_base);
}
static struct intel_uncore_ops generic_uncore_mmio_ops = {
.init_box = intel_generic_uncore_mmio_init_box,
.exit_box = uncore_mmio_exit_box,
.disable_box = intel_generic_uncore_mmio_disable_box,
.enable_box = intel_generic_uncore_mmio_enable_box,
.disable_event = intel_generic_uncore_mmio_disable_event,
.enable_event = intel_generic_uncore_mmio_enable_event,
.read_counter = uncore_mmio_read_counter,
};
static bool uncore_update_uncore_type(enum uncore_access_type type_id,
struct intel_uncore_type *uncore,
struct intel_uncore_discovery_type *type)
{
uncore->type_id = type->type;
uncore->num_boxes = type->num_boxes;
uncore->num_counters = type->num_counters;
uncore->perf_ctr_bits = type->counter_width;
uncore->box_ids = type->ids;
switch (type_id) {
case UNCORE_ACCESS_MSR:
uncore->ops = &generic_uncore_msr_ops;
uncore->perf_ctr = (unsigned int)type->box_ctrl + type->ctr_offset;
uncore->event_ctl = (unsigned int)type->box_ctrl + type->ctl_offset;
uncore->box_ctl = (unsigned int)type->box_ctrl;
uncore->msr_offsets = type->box_offset;
break;
case UNCORE_ACCESS_PCI:
uncore->ops = &generic_uncore_pci_ops;
uncore->perf_ctr = (unsigned int)UNCORE_DISCOVERY_PCI_BOX_CTRL(type->box_ctrl) + type->ctr_offset;
uncore->event_ctl = (unsigned int)UNCORE_DISCOVERY_PCI_BOX_CTRL(type->box_ctrl) + type->ctl_offset;
uncore->box_ctl = (unsigned int)UNCORE_DISCOVERY_PCI_BOX_CTRL(type->box_ctrl);
uncore->box_ctls = type->box_ctrl_die;
uncore->pci_offsets = type->box_offset;
break;
case UNCORE_ACCESS_MMIO:
uncore->ops = &generic_uncore_mmio_ops;
uncore->perf_ctr = (unsigned int)type->ctr_offset;
uncore->event_ctl = (unsigned int)type->ctl_offset;
uncore->box_ctl = (unsigned int)type->box_ctrl;
uncore->box_ctls = type->box_ctrl_die;
uncore->mmio_offsets = type->box_offset;
uncore->mmio_map_size = UNCORE_GENERIC_MMIO_SIZE;
break;
default:
return false;
}
return true;
}
static struct intel_uncore_type **
intel_uncore_generic_init_uncores(enum uncore_access_type type_id)
{
struct intel_uncore_discovery_type *type;
struct intel_uncore_type **uncores;
struct intel_uncore_type *uncore;
struct rb_node *node;
int i = 0;
uncores = kcalloc(num_discovered_types[type_id] + 1,
sizeof(struct intel_uncore_type *), GFP_KERNEL);
if (!uncores)
return empty_uncore;
for (node = rb_first(&discovery_tables); node; node = rb_next(node)) {
type = rb_entry(node, struct intel_uncore_discovery_type, node);
if (type->access_type != type_id)
continue;
uncore = kzalloc(sizeof(struct intel_uncore_type), GFP_KERNEL);
if (!uncore)
break;
uncore->event_mask = GENERIC_PMON_RAW_EVENT_MASK;
uncore->format_group = &generic_uncore_format_group;
if (!uncore_update_uncore_type(type_id, uncore, type)) {
kfree(uncore);
continue;
}
uncores[i++] = uncore;
}
return uncores;
}
void intel_uncore_generic_uncore_cpu_init(void)
{
uncore_msr_uncores = intel_uncore_generic_init_uncores(UNCORE_ACCESS_MSR);
}
int intel_uncore_generic_uncore_pci_init(void)
{
uncore_pci_uncores = intel_uncore_generic_init_uncores(UNCORE_ACCESS_PCI);
return 0;
}
void intel_uncore_generic_uncore_mmio_init(void)
{
uncore_mmio_uncores = intel_uncore_generic_init_uncores(UNCORE_ACCESS_MMIO);
}

131
arch/x86/events/intel/uncore_discovery.h Normal file
View File

@ -0,0 +1,131 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/* Generic device ID of a discovery table device */
#define UNCORE_DISCOVERY_TABLE_DEVICE 0x09a7
/* Capability ID for a discovery table device */
#define UNCORE_EXT_CAP_ID_DISCOVERY 0x23
/* First DVSEC offset */
#define UNCORE_DISCOVERY_DVSEC_OFFSET 0x8
/* Mask of the supported discovery entry type */
#define UNCORE_DISCOVERY_DVSEC_ID_MASK 0xffff
/* PMON discovery entry type ID */
#define UNCORE_DISCOVERY_DVSEC_ID_PMON 0x1
/* Second DVSEC offset */
#define UNCORE_DISCOVERY_DVSEC2_OFFSET 0xc
/* Mask of the discovery table BAR offset */
#define UNCORE_DISCOVERY_DVSEC2_BIR_MASK 0x7
/* Discovery table BAR base offset */
#define UNCORE_DISCOVERY_BIR_BASE 0x10
/* Discovery table BAR step */
#define UNCORE_DISCOVERY_BIR_STEP 0x4
/* Mask of the discovery table offset */
#define UNCORE_DISCOVERY_MASK 0xf
/* Global discovery table size */
#define UNCORE_DISCOVERY_GLOBAL_MAP_SIZE 0x20
#define UNCORE_DISCOVERY_PCI_DOMAIN(data) ((data >> 28) & 0x7)
#define UNCORE_DISCOVERY_PCI_BUS(data) ((data >> 20) & 0xff)
#define UNCORE_DISCOVERY_PCI_DEVFN(data) ((data >> 12) & 0xff)
#define UNCORE_DISCOVERY_PCI_BOX_CTRL(data) (data & 0xfff)
#define uncore_discovery_invalid_unit(unit) \
(!unit.table1 || !unit.ctl || !unit.table3 || \
unit.table1 == -1ULL || unit.ctl == -1ULL || \
unit.table3 == -1ULL)
#define GENERIC_PMON_CTL_EV_SEL_MASK 0x000000ff
#define GENERIC_PMON_CTL_UMASK_MASK 0x0000ff00
#define GENERIC_PMON_CTL_EDGE_DET (1 << 18)
#define GENERIC_PMON_CTL_INVERT (1 << 23)
#define GENERIC_PMON_CTL_TRESH_MASK 0xff000000
#define GENERIC_PMON_RAW_EVENT_MASK (GENERIC_PMON_CTL_EV_SEL_MASK | \
GENERIC_PMON_CTL_UMASK_MASK | \
GENERIC_PMON_CTL_EDGE_DET | \
GENERIC_PMON_CTL_INVERT | \
GENERIC_PMON_CTL_TRESH_MASK)
#define GENERIC_PMON_BOX_CTL_FRZ (1 << 0)
#define GENERIC_PMON_BOX_CTL_RST_CTRL (1 << 8)
#define GENERIC_PMON_BOX_CTL_RST_CTRS (1 << 9)
#define GENERIC_PMON_BOX_CTL_INT (GENERIC_PMON_BOX_CTL_RST_CTRL | \
GENERIC_PMON_BOX_CTL_RST_CTRS)
enum uncore_access_type {
UNCORE_ACCESS_MSR = 0,
UNCORE_ACCESS_MMIO,
UNCORE_ACCESS_PCI,
UNCORE_ACCESS_MAX,
};
struct uncore_global_discovery {
union {
u64 table1;
struct {
u64 type : 8,
stride : 8,
max_units : 10,
__reserved_1 : 36,
access_type : 2;
};
};
u64 ctl; /* Global Control Address */
union {
u64 table3;
struct {
u64 status_offset : 8,
num_status : 16,
__reserved_2 : 40;
};
};
};
struct uncore_unit_discovery {
union {
u64 table1;
struct {
u64 num_regs : 8,
ctl_offset : 8,
bit_width : 8,
ctr_offset : 8,
status_offset : 8,
__reserved_1 : 22,
access_type : 2;
};
};
u64 ctl; /* Unit Control Address */
union {
u64 table3;
struct {
u64 box_type : 16,
box_id : 16,
__reserved_2 : 32;
};
};
};
struct intel_uncore_discovery_type {
struct rb_node node;
enum uncore_access_type access_type;
u64 box_ctrl; /* Unit ctrl addr of the first box */
u64 *box_ctrl_die; /* Unit ctrl addr of the first box of each die */
u16 type; /* Type ID of the uncore block */
u8 num_counters;
u8 counter_width;
u8 ctl_offset; /* Counter Control 0 offset */
u8 ctr_offset; /* Counter 0 offset */
u16 num_boxes; /* number of boxes for the uncore block */
unsigned int *ids; /* Box IDs */
unsigned int *box_offset; /* Box offset */
};
bool intel_uncore_has_discovery_tables(void);
void intel_uncore_clear_discovery_tables(void);
void intel_uncore_generic_uncore_cpu_init(void);
int intel_uncore_generic_uncore_pci_init(void);
void intel_uncore_generic_uncore_mmio_init(void);

131
arch/x86/events/intel/uncore_snb.c
View File

@ -62,6 +62,8 @@
#define PCI_DEVICE_ID_INTEL_TGL_H_IMC 0x9a36
#define PCI_DEVICE_ID_INTEL_RKL_1_IMC 0x4c43
#define PCI_DEVICE_ID_INTEL_RKL_2_IMC 0x4c53
#define PCI_DEVICE_ID_INTEL_ADL_1_IMC 0x4660
#define PCI_DEVICE_ID_INTEL_ADL_2_IMC 0x4641
/* SNB event control */
#define SNB_UNC_CTL_EV_SEL_MASK 0x000000ff
@ -131,12 +133,33 @@
#define ICL_UNC_ARB_PER_CTR 0x3b1
#define ICL_UNC_ARB_PERFEVTSEL 0x3b3
/* ADL uncore global control */
#define ADL_UNC_PERF_GLOBAL_CTL 0x2ff0
#define ADL_UNC_FIXED_CTR_CTRL 0x2fde
#define ADL_UNC_FIXED_CTR 0x2fdf
/* ADL Cbo register */
#define ADL_UNC_CBO_0_PER_CTR0 0x2002
#define ADL_UNC_CBO_0_PERFEVTSEL0 0x2000
#define ADL_UNC_CTL_THRESHOLD 0x3f000000
#define ADL_UNC_RAW_EVENT_MASK (SNB_UNC_CTL_EV_SEL_MASK | \
SNB_UNC_CTL_UMASK_MASK | \
SNB_UNC_CTL_EDGE_DET | \
SNB_UNC_CTL_INVERT | \
ADL_UNC_CTL_THRESHOLD)
/* ADL ARB register */
#define ADL_UNC_ARB_PER_CTR0 0x2FD2
#define ADL_UNC_ARB_PERFEVTSEL0 0x2FD0
#define ADL_UNC_ARB_MSR_OFFSET 0x8
DEFINE_UNCORE_FORMAT_ATTR(event, event, "config:0-7");
DEFINE_UNCORE_FORMAT_ATTR(umask, umask, "config:8-15");
DEFINE_UNCORE_FORMAT_ATTR(edge, edge, "config:18");
DEFINE_UNCORE_FORMAT_ATTR(inv, inv, "config:23");
DEFINE_UNCORE_FORMAT_ATTR(cmask5, cmask, "config:24-28");
DEFINE_UNCORE_FORMAT_ATTR(cmask8, cmask, "config:24-31");
DEFINE_UNCORE_FORMAT_ATTR(threshold, threshold, "config:24-29");
/* Sandy Bridge uncore support */
static void snb_uncore_msr_enable_event(struct intel_uncore_box *box, struct perf_event *event)
@ -422,6 +445,106 @@ void tgl_uncore_cpu_init(void)
skl_uncore_msr_ops.init_box = rkl_uncore_msr_init_box;
}
static void adl_uncore_msr_init_box(struct intel_uncore_box *box)
{
if (box->pmu->pmu_idx == 0)
wrmsrl(ADL_UNC_PERF_GLOBAL_CTL, SNB_UNC_GLOBAL_CTL_EN);
}
static void adl_uncore_msr_enable_box(struct intel_uncore_box *box)
{
wrmsrl(ADL_UNC_PERF_GLOBAL_CTL, SNB_UNC_GLOBAL_CTL_EN);
}
static void adl_uncore_msr_disable_box(struct intel_uncore_box *box)
{
if (box->pmu->pmu_idx == 0)
wrmsrl(ADL_UNC_PERF_GLOBAL_CTL, 0);
}
static void adl_uncore_msr_exit_box(struct intel_uncore_box *box)
{
if (box->pmu->pmu_idx == 0)
wrmsrl(ADL_UNC_PERF_GLOBAL_CTL, 0);
}
static struct intel_uncore_ops adl_uncore_msr_ops = {
.init_box = adl_uncore_msr_init_box,
.enable_box = adl_uncore_msr_enable_box,
.disable_box = adl_uncore_msr_disable_box,
.exit_box = adl_uncore_msr_exit_box,
.disable_event = snb_uncore_msr_disable_event,
.enable_event = snb_uncore_msr_enable_event,
.read_counter = uncore_msr_read_counter,
};
static struct attribute *adl_uncore_formats_attr[] = {
&format_attr_event.attr,
&format_attr_umask.attr,
&format_attr_edge.attr,
&format_attr_inv.attr,
&format_attr_threshold.attr,
NULL,
};
static const struct attribute_group adl_uncore_format_group = {
.name = "format",
.attrs = adl_uncore_formats_attr,
};
static struct intel_uncore_type adl_uncore_cbox = {
.name = "cbox",
.num_counters = 2,
.perf_ctr_bits = 44,
.perf_ctr = ADL_UNC_CBO_0_PER_CTR0,
.event_ctl = ADL_UNC_CBO_0_PERFEVTSEL0,
.event_mask = ADL_UNC_RAW_EVENT_MASK,
.msr_offset = ICL_UNC_CBO_MSR_OFFSET,
.ops = &adl_uncore_msr_ops,
.format_group = &adl_uncore_format_group,
};
static struct intel_uncore_type adl_uncore_arb = {
.name = "arb",
.num_counters = 2,
.num_boxes = 2,
.perf_ctr_bits = 44,
.perf_ctr = ADL_UNC_ARB_PER_CTR0,
.event_ctl = ADL_UNC_ARB_PERFEVTSEL0,
.event_mask = SNB_UNC_RAW_EVENT_MASK,
.msr_offset = ADL_UNC_ARB_MSR_OFFSET,
.constraints = snb_uncore_arb_constraints,
.ops = &adl_uncore_msr_ops,
.format_group = &snb_uncore_format_group,
};
static struct intel_uncore_type adl_uncore_clockbox = {
.name = "clock",
.num_counters = 1,
.num_boxes = 1,
.fixed_ctr_bits = 48,
.fixed_ctr = ADL_UNC_FIXED_CTR,
.fixed_ctl = ADL_UNC_FIXED_CTR_CTRL,
.single_fixed = 1,
.event_mask = SNB_UNC_CTL_EV_SEL_MASK,
.format_group = &icl_uncore_clock_format_group,
.ops = &adl_uncore_msr_ops,
.event_descs = icl_uncore_events,
};
static struct intel_uncore_type *adl_msr_uncores[] = {
&adl_uncore_cbox,
&adl_uncore_arb,
&adl_uncore_clockbox,
NULL,
};
void adl_uncore_cpu_init(void)
{
adl_uncore_cbox.num_boxes = icl_get_cbox_num();
uncore_msr_uncores = adl_msr_uncores;
}
enum {
SNB_PCI_UNCORE_IMC,
};
@ -1203,6 +1326,14 @@ static const struct pci_device_id tgl_uncore_pci_ids[] = {
PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_TGL_H_IMC),
.driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
},
{ /* IMC */
PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ADL_1_IMC),
.driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
},
{ /* IMC */
PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ADL_2_IMC),
.driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
},
{ /* end: all zeroes */ }
};

60
arch/x86/events/intel/uncore_snbep.c
View File

@ -3675,7 +3675,8 @@ static struct intel_uncore_ops skx_uncore_iio_ops = {
static inline u8 skx_iio_stack(struct intel_uncore_pmu *pmu, int die)
{
return pmu->type->topology[die] >> (pmu->pmu_idx * BUS_NUM_STRIDE);
return pmu->type->topology[die].configuration >>
(pmu->pmu_idx * BUS_NUM_STRIDE);
}
static umode_t
@ -3688,19 +3689,14 @@ skx_iio_mapping_visible(struct kobject *kobj, struct attribute *attr, int die)
}
static ssize_t skx_iio_mapping_show(struct device *dev,
struct device_attribute *attr, char *buf)
struct device_attribute *attr, char *buf)
{
struct pci_bus *bus = pci_find_next_bus(NULL);
struct intel_uncore_pmu *uncore_pmu = dev_to_uncore_pmu(dev);
struct intel_uncore_pmu *pmu = dev_to_uncore_pmu(dev);
struct dev_ext_attribute *ea = to_dev_ext_attribute(attr);
long die = (long)ea->var;
/*
* Current implementation is for single segment configuration hence it's
* safe to take the segment value from the first available root bus.
*/
return sprintf(buf, "%04x:%02x\n", pci_domain_nr(bus),
skx_iio_stack(uncore_pmu, die));
return sprintf(buf, "%04x:%02x\n", pmu->type->topology[die].segment,
skx_iio_stack(pmu, die));
}
static int skx_msr_cpu_bus_read(int cpu, u64 *topology)
@ -3737,34 +3733,32 @@ static int die_to_cpu(int die)
static int skx_iio_get_topology(struct intel_uncore_type *type)
{
int i, ret;
struct pci_bus *bus = NULL;
int die, ret = -EPERM;
/*
* Verified single-segment environments only; disabled for multiple
* segment topologies for now except VMD domains.
* VMD domains start at 0x10000 to not clash with ACPI _SEG domains.
*/
while ((bus = pci_find_next_bus(bus))
&& (!pci_domain_nr(bus) || pci_domain_nr(bus) > 0xffff))
;
if (bus)
return -EPERM;
type->topology = kcalloc(uncore_max_dies(), sizeof(u64), GFP_KERNEL);
type->topology = kcalloc(uncore_max_dies(), sizeof(*type->topology),
GFP_KERNEL);
if (!type->topology)
return -ENOMEM;
for (i = 0; i < uncore_max_dies(); i++) {
ret = skx_msr_cpu_bus_read(die_to_cpu(i), &type->topology[i]);
if (ret) {
kfree(type->topology);
type->topology = NULL;
return ret;
}
for (die = 0; die < uncore_max_dies(); die++) {
ret = skx_msr_cpu_bus_read(die_to_cpu(die),
&type->topology[die].configuration);
if (ret)
break;
ret = uncore_die_to_segment(die);
if (ret < 0)
break;
type->topology[die].segment = ret;
}
return 0;
if (ret < 0) {
kfree(type->topology);
type->topology = NULL;
}
return ret;
}
static struct attribute_group skx_iio_mapping_group = {
@ -3785,7 +3779,7 @@ static int skx_iio_set_mapping(struct intel_uncore_type *type)
struct dev_ext_attribute *eas = NULL;
ret = skx_iio_get_topology(type);
if (ret)
if (ret < 0)
goto clear_attr_update;
ret = -ENOMEM;

2
arch/x86/events/msr.c
View File

@ -100,6 +100,8 @@ static bool test_intel(int idx, void *data)
case INTEL_FAM6_TIGERLAKE_L:
case INTEL_FAM6_TIGERLAKE:
case INTEL_FAM6_ROCKETLAKE:
case INTEL_FAM6_ALDERLAKE:
case INTEL_FAM6_ALDERLAKE_L:
if (idx == PERF_MSR_SMI || idx == PERF_MSR_PPERF)
return true;
break;

119
arch/x86/events/perf_event.h
View File

@ -15,6 +15,7 @@
#include <linux/perf_event.h>
#include <asm/intel_ds.h>
#include <asm/cpu.h>
/* To enable MSR tracing please use the generic trace points. */
@ -228,7 +229,6 @@ struct cpu_hw_events {
*/
struct perf_event *events[X86_PMC_IDX_MAX]; /* in counter order */
unsigned long active_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
unsigned long running[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
int enabled;
int n_events; /* the # of events in the below arrays */
@ -327,6 +327,8 @@ struct cpu_hw_events {
int n_pair; /* Large increment events */
void *kfree_on_online[X86_PERF_KFREE_MAX];
struct pmu *pmu;
};
#define __EVENT_CONSTRAINT_RANGE(c, e, n, m, w, o, f) { \
@ -630,6 +632,71 @@ enum {
x86_lbr_exclusive_max,
};
struct x86_hybrid_pmu {
struct pmu pmu;
const char *name;
u8 cpu_type;
cpumask_t supported_cpus;
union perf_capabilities intel_cap;
u64 intel_ctrl;
int max_pebs_events;
int num_counters;
int num_counters_fixed;
struct event_constraint unconstrained;
u64 hw_cache_event_ids
[PERF_COUNT_HW_CACHE_MAX]
[PERF_COUNT_HW_CACHE_OP_MAX]
[PERF_COUNT_HW_CACHE_RESULT_MAX];
u64 hw_cache_extra_regs
[PERF_COUNT_HW_CACHE_MAX]
[PERF_COUNT_HW_CACHE_OP_MAX]
[PERF_COUNT_HW_CACHE_RESULT_MAX];
struct event_constraint *event_constraints;
struct event_constraint *pebs_constraints;
struct extra_reg *extra_regs;
};
static __always_inline struct x86_hybrid_pmu *hybrid_pmu(struct pmu *pmu)
{
return container_of(pmu, struct x86_hybrid_pmu, pmu);
}
extern struct static_key_false perf_is_hybrid;
#define is_hybrid() static_branch_unlikely(&perf_is_hybrid)
#define hybrid(_pmu, _field) \
(*({ \
typeof(&x86_pmu._field) __Fp = &x86_pmu._field; \
\
if (is_hybrid() && (_pmu)) \
__Fp = &hybrid_pmu(_pmu)->_field; \
\
__Fp; \
}))
#define hybrid_var(_pmu, _var) \
(*({ \
typeof(&_var) __Fp = &_var; \
\
if (is_hybrid() && (_pmu)) \
__Fp = &hybrid_pmu(_pmu)->_var; \
\
__Fp; \
}))
enum hybrid_pmu_type {
hybrid_big = 0x40,
hybrid_small = 0x20,
hybrid_big_small = hybrid_big | hybrid_small,
};
#define X86_HYBRID_PMU_ATOM_IDX 0
#define X86_HYBRID_PMU_CORE_IDX 1
#define X86_HYBRID_NUM_PMUS 2
/*
* struct x86_pmu - generic x86 pmu
*/
@ -816,6 +883,19 @@ struct x86_pmu {
int (*check_period) (struct perf_event *event, u64 period);
int (*aux_output_match) (struct perf_event *event);
int (*filter_match)(struct perf_event *event);
/*
* Hybrid support
*
* Most PMU capabilities are the same among different hybrid PMUs.
* The global x86_pmu saves the architecture capabilities, which
* are available for all PMUs. The hybrid_pmu only includes the
* unique capabilities.
*/
int num_hybrid_pmus;
struct x86_hybrid_pmu *hybrid_pmu;
u8 (*get_hybrid_cpu_type) (void);
};
struct x86_perf_task_context_opt {
@ -905,7 +985,23 @@ static struct perf_pmu_events_ht_attr event_attr_##v = { \
.event_str_ht = ht, \
}
struct pmu *x86_get_pmu(void);
#define EVENT_ATTR_STR_HYBRID(_name, v, str, _pmu) \
static struct perf_pmu_events_hybrid_attr event_attr_##v = { \
.attr = __ATTR(_name, 0444, events_hybrid_sysfs_show, NULL),\
.id = 0, \
.event_str = str, \
.pmu_type = _pmu, \
}
#define FORMAT_HYBRID_PTR(_id) (&format_attr_hybrid_##_id.attr.attr)
#define FORMAT_ATTR_HYBRID(_name, _pmu) \
static struct perf_pmu_format_hybrid_attr format_attr_hybrid_##_name = {\
.attr = __ATTR_RO(_name), \
.pmu_type = _pmu, \
}
struct pmu *x86_get_pmu(unsigned int cpu);
extern struct x86_pmu x86_pmu __read_mostly;
static __always_inline struct x86_perf_task_context_opt *task_context_opt(void *ctx)
@ -964,6 +1060,9 @@ static inline int x86_pmu_rdpmc_index(int index)
return x86_pmu.rdpmc_index ? x86_pmu.rdpmc_index(index) : index;
}
bool check_hw_exists(struct pmu *pmu, int num_counters,
int num_counters_fixed);
int x86_add_exclusive(unsigned int what);
void x86_del_exclusive(unsigned int what);
@ -1027,6 +1126,11 @@ void x86_pmu_enable_event(struct perf_event *event);
int x86_pmu_handle_irq(struct pt_regs *regs);
void x86_pmu_show_pmu_cap(int num_counters, int num_counters_fixed,
u64 intel_ctrl);
void x86_pmu_update_cpu_context(struct pmu *pmu, int cpu);
extern struct event_constraint emptyconstraint;
extern struct event_constraint unconstrained;
@ -1067,10 +1171,15 @@ ssize_t events_sysfs_show(struct device *dev, struct device_attribute *attr,
char *page);
ssize_t events_ht_sysfs_show(struct device *dev, struct device_attribute *attr,
char *page);
ssize_t events_hybrid_sysfs_show(struct device *dev,
struct device_attribute *attr,
char *page);
static inline bool fixed_counter_disabled(int i)
static inline bool fixed_counter_disabled(int i, struct pmu *pmu)
{
return !(x86_pmu.intel_ctrl >> (i + INTEL_PMC_IDX_FIXED));
u64 intel_ctrl = hybrid(pmu, intel_ctrl);
return !(intel_ctrl >> (i + INTEL_PMC_IDX_FIXED));
}
#ifdef CONFIG_CPU_SUP_AMD
@ -1154,6 +1263,8 @@ extern struct event_constraint intel_glm_pebs_event_constraints[];
extern struct event_constraint intel_glp_pebs_event_constraints[];
extern struct event_constraint intel_grt_pebs_event_constraints[];
extern struct event_constraint intel_nehalem_pebs_event_constraints[];
extern struct event_constraint intel_westmere_pebs_event_constraints[];

2
arch/x86/events/rapl.c
View File

@ -800,6 +800,8 @@ static const struct x86_cpu_id rapl_model_match[] __initconst = {
X86_MATCH_INTEL_FAM6_MODEL(ICELAKE_X, &model_hsx),
X86_MATCH_INTEL_FAM6_MODEL(COMETLAKE_L, &model_skl),
X86_MATCH_INTEL_FAM6_MODEL(COMETLAKE, &model_skl),
X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE, &model_skl),
X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE_L, &model_skl),
X86_MATCH_INTEL_FAM6_MODEL(SAPPHIRERAPIDS_X, &model_spr),
X86_MATCH_VENDOR_FAM(AMD, 0x17, &model_amd_fam17h),
X86_MATCH_VENDOR_FAM(HYGON, 0x18, &model_amd_fam17h),

6
arch/x86/include/asm/cpu.h
View File

@ -46,6 +46,7 @@ extern void switch_to_sld(unsigned long tifn);
extern bool handle_user_split_lock(struct pt_regs *regs, long error_code);
extern bool handle_guest_split_lock(unsigned long ip);
extern void handle_bus_lock(struct pt_regs *regs);
u8 get_this_hybrid_cpu_type(void);
#else
static inline void __init sld_setup(struct cpuinfo_x86 *c) {}
static inline void switch_to_sld(unsigned long tifn) {}
@ -60,6 +61,11 @@ static inline bool handle_guest_split_lock(unsigned long ip)
}
static inline void handle_bus_lock(struct pt_regs *regs) {}
static inline u8 get_this_hybrid_cpu_type(void)
{
return 0;
}
#endif
#ifdef CONFIG_IA32_FEAT_CTL
void init_ia32_feat_ctl(struct cpuinfo_x86 *c);

1
arch/x86/include/asm/cpufeatures.h
View File

@ -379,6 +379,7 @@
#define X86_FEATURE_MD_CLEAR (18*32+10) /* VERW clears CPU buffers */
#define X86_FEATURE_TSX_FORCE_ABORT (18*32+13) /* "" TSX_FORCE_ABORT */
#define X86_FEATURE_SERIALIZE (18*32+14) /* SERIALIZE instruction */
#define X86_FEATURE_HYBRID_CPU (18*32+15) /* "" This part has CPUs of more than one type */
#define X86_FEATURE_TSXLDTRK (18*32+16) /* TSX Suspend Load Address Tracking */
#define X86_FEATURE_PCONFIG (18*32+18) /* Intel PCONFIG */
#define X86_FEATURE_ARCH_LBR (18*32+19) /* Intel ARCH LBR */

3
arch/x86/include/asm/msr-index.h
View File

@ -185,6 +185,9 @@
#define MSR_PEBS_DATA_CFG 0x000003f2
#define MSR_IA32_DS_AREA 0x00000600
#define MSR_IA32_PERF_CAPABILITIES 0x00000345
#define PERF_CAP_METRICS_IDX 15
#define PERF_CAP_PT_IDX 16
#define MSR_PEBS_LD_LAT_THRESHOLD 0x000003f6
#define MSR_IA32_RTIT_CTL 0x00000570

16
arch/x86/kernel/cpu/intel.c
View File

@ -1268,3 +1268,19 @@ void __init sld_setup(struct cpuinfo_x86 *c)
sld_state_setup();
sld_state_show();
}
#define X86_HYBRID_CPU_TYPE_ID_SHIFT 24
/**
* get_this_hybrid_cpu_type() - Get the type of this hybrid CPU
*
* Returns the CPU type [31:24] (i.e., Atom or Core) of a CPU in
* a hybrid processor. If the processor is not hybrid, returns 0.
*/
u8 get_this_hybrid_cpu_type(void)
{
if (!cpu_feature_enabled(X86_FEATURE_HYBRID_CPU))
return 0;
return cpuid_eax(0x0000001a) >> X86_HYBRID_CPU_TYPE_ID_SHIFT;
}

5
arch/x86/kernel/signal_compat.c
View File

@ -29,7 +29,7 @@ static inline void signal_compat_build_tests(void)
BUILD_BUG_ON(NSIGFPE != 15);
BUILD_BUG_ON(NSIGSEGV != 9);
BUILD_BUG_ON(NSIGBUS != 5);
BUILD_BUG_ON(NSIGTRAP != 5);
BUILD_BUG_ON(NSIGTRAP != 6);
BUILD_BUG_ON(NSIGCHLD != 6);
BUILD_BUG_ON(NSIGSYS != 2);
@ -138,6 +138,9 @@ static inline void signal_compat_build_tests(void)
BUILD_BUG_ON(offsetof(siginfo_t, si_pkey) != 0x20);
BUILD_BUG_ON(offsetof(compat_siginfo_t, si_pkey) != 0x14);
BUILD_BUG_ON(offsetof(siginfo_t, si_perf) != 0x18);
BUILD_BUG_ON(offsetof(compat_siginfo_t, si_perf) != 0x10);
CHECK_CSI_OFFSET(_sigpoll);
CHECK_CSI_SIZE (_sigpoll, 2*sizeof(int));
CHECK_SI_SIZE (_sigpoll, 4*sizeof(int));

4
fs/signalfd.c
View File

@ -134,6 +134,10 @@ static int signalfd_copyinfo(struct signalfd_siginfo __user *uinfo,
#endif
new.ssi_addr_lsb = (short) kinfo->si_addr_lsb;
break;
case SIL_PERF_EVENT:
new.ssi_addr = (long) kinfo->si_addr;
new.ssi_perf = kinfo->si_perf;
break;
case SIL_CHLD:
new.ssi_pid = kinfo->si_pid;
new.ssi_uid = kinfo->si_uid;

2
include/linux/compat.h
View File

@ -236,6 +236,8 @@ typedef struct compat_siginfo {
char _dummy_pkey[__COMPAT_ADDR_BND_PKEY_PAD];
u32 _pkey;
} _addr_pkey;
/* used when si_code=TRAP_PERF */
compat_ulong_t _perf;
};
} _sigfault;

78
include/linux/perf_event.h
View File

@ -260,15 +260,16 @@ struct perf_event;
/**
* pmu::capabilities flags
*/
#define PERF_PMU_CAP_NO_INTERRUPT 0x01
#define PERF_PMU_CAP_NO_NMI 0x02
#define PERF_PMU_CAP_AUX_NO_SG 0x04
#define PERF_PMU_CAP_EXTENDED_REGS 0x08
#define PERF_PMU_CAP_EXCLUSIVE 0x10
#define PERF_PMU_CAP_ITRACE 0x20
#define PERF_PMU_CAP_HETEROGENEOUS_CPUS 0x40
#define PERF_PMU_CAP_NO_EXCLUDE 0x80
#define PERF_PMU_CAP_AUX_OUTPUT 0x100
#define PERF_PMU_CAP_NO_INTERRUPT 0x0001
#define PERF_PMU_CAP_NO_NMI 0x0002
#define PERF_PMU_CAP_AUX_NO_SG 0x0004
#define PERF_PMU_CAP_EXTENDED_REGS 0x0008
#define PERF_PMU_CAP_EXCLUSIVE 0x0010
#define PERF_PMU_CAP_ITRACE 0x0020
#define PERF_PMU_CAP_HETEROGENEOUS_CPUS 0x0040
#define PERF_PMU_CAP_NO_EXCLUDE 0x0080
#define PERF_PMU_CAP_AUX_OUTPUT 0x0100
#define PERF_PMU_CAP_EXTENDED_HW_TYPE 0x0200
struct perf_output_handle;
@ -607,6 +608,7 @@ struct swevent_hlist {
#define PERF_ATTACH_TASK_DATA 0x08
#define PERF_ATTACH_ITRACE 0x10
#define PERF_ATTACH_SCHED_CB 0x20
#define PERF_ATTACH_CHILD 0x40
struct perf_cgroup;
struct perf_buffer;
@ -734,6 +736,7 @@ struct perf_event {
int pending_wakeup;
int pending_kill;
int pending_disable;
unsigned long pending_addr; /* SIGTRAP */
struct irq_work pending;
atomic_t event_limit;
@ -957,7 +960,7 @@ extern void __perf_event_task_sched_in(struct task_struct *prev,
struct task_struct *task);
extern void __perf_event_task_sched_out(struct task_struct *prev,
struct task_struct *next);
extern int perf_event_init_task(struct task_struct *child);
extern int perf_event_init_task(struct task_struct *child, u64 clone_flags);
extern void perf_event_exit_task(struct task_struct *child);
extern void perf_event_free_task(struct task_struct *task);
extern void perf_event_delayed_put(struct task_struct *task);
@ -1176,30 +1179,24 @@ DECLARE_PER_CPU(struct pt_regs, __perf_regs[4]);
* which is guaranteed by us not actually scheduling inside other swevents
* because those disable preemption.
*/
static __always_inline void
perf_sw_event_sched(u32 event_id, u64 nr, u64 addr)
static __always_inline void __perf_sw_event_sched(u32 event_id, u64 nr, u64 addr)
{
if (static_key_false(&perf_swevent_enabled[event_id])) {
struct pt_regs *regs = this_cpu_ptr(&__perf_regs[0]);
struct pt_regs *regs = this_cpu_ptr(&__perf_regs[0]);
perf_fetch_caller_regs(regs);
___perf_sw_event(event_id, nr, regs, addr);
}
perf_fetch_caller_regs(regs);
___perf_sw_event(event_id, nr, regs, addr);
}
extern struct static_key_false perf_sched_events;
static __always_inline bool
perf_sw_migrate_enabled(void)
static __always_inline bool __perf_sw_enabled(int swevt)
{
if (static_key_false(&perf_swevent_enabled[PERF_COUNT_SW_CPU_MIGRATIONS]))
return true;
return false;
return static_key_false(&perf_swevent_enabled[swevt]);
}
static inline void perf_event_task_migrate(struct task_struct *task)
{
if (perf_sw_migrate_enabled())
if (__perf_sw_enabled(PERF_COUNT_SW_CPU_MIGRATIONS))
task->sched_migrated = 1;
}
@ -1209,11 +1206,9 @@ static inline void perf_event_task_sched_in(struct task_struct *prev,
if (static_branch_unlikely(&perf_sched_events))
__perf_event_task_sched_in(prev, task);
if (perf_sw_migrate_enabled() && task->sched_migrated) {
struct pt_regs *regs = this_cpu_ptr(&__perf_regs[0]);
perf_fetch_caller_regs(regs);
___perf_sw_event(PERF_COUNT_SW_CPU_MIGRATIONS, 1, regs, 0);
if (__perf_sw_enabled(PERF_COUNT_SW_CPU_MIGRATIONS) &&
task->sched_migrated) {
__perf_sw_event_sched(PERF_COUNT_SW_CPU_MIGRATIONS, 1, 0);
task->sched_migrated = 0;
}
}
@ -1221,7 +1216,15 @@ static inline void perf_event_task_sched_in(struct task_struct *prev,
static inline void perf_event_task_sched_out(struct task_struct *prev,
struct task_struct *next)
{
perf_sw_event_sched(PERF_COUNT_SW_CONTEXT_SWITCHES, 1, 0);
if (__perf_sw_enabled(PERF_COUNT_SW_CONTEXT_SWITCHES))
__perf_sw_event_sched(PERF_COUNT_SW_CONTEXT_SWITCHES, 1, 0);
#ifdef CONFIG_CGROUP_PERF
if (__perf_sw_enabled(PERF_COUNT_SW_CGROUP_SWITCHES) &&
perf_cgroup_from_task(prev, NULL) !=
perf_cgroup_from_task(next, NULL))
__perf_sw_event_sched(PERF_COUNT_SW_CGROUP_SWITCHES, 1, 0);
#endif
if (static_branch_unlikely(&perf_sched_events))
__perf_event_task_sched_out(prev, next);
@ -1448,7 +1451,8 @@ perf_event_task_sched_in(struct task_struct *prev,
static inline void
perf_event_task_sched_out(struct task_struct *prev,
struct task_struct *next) { }
static inline int perf_event_init_task(struct task_struct *child) { return 0; }
static inline int perf_event_init_task(struct task_struct *child,
u64 clone_flags) { return 0; }
static inline void perf_event_exit_task(struct task_struct *child) { }
static inline void perf_event_free_task(struct task_struct *task) { }
static inline void perf_event_delayed_put(struct task_struct *task) { }
@ -1477,8 +1481,6 @@ static inline int perf_event_refresh(struct perf_event *event, int refresh)
static inline void
perf_sw_event(u32 event_id, u64 nr, struct pt_regs *regs, u64 addr) { }
static inline void
perf_sw_event_sched(u32 event_id, u64 nr, u64 addr) { }
static inline void
perf_bp_event(struct perf_event *event, void *data) { }
static inline int perf_register_guest_info_callbacks
@ -1548,6 +1550,18 @@ struct perf_pmu_events_ht_attr {
const char *event_str_noht;
};
struct perf_pmu_events_hybrid_attr {
struct device_attribute attr;
u64 id;
const char *event_str;
u64 pmu_type;
};
struct perf_pmu_format_hybrid_attr {
struct device_attribute attr;
u64 pmu_type;
};
ssize_t perf_event_sysfs_show(struct device *dev, struct device_attribute *attr,
char *page);

1
include/linux/signal.h
View File

@ -43,6 +43,7 @@ enum siginfo_layout {
SIL_FAULT_MCEERR,
SIL_FAULT_BNDERR,
SIL_FAULT_PKUERR,
SIL_PERF_EVENT,
SIL_CHLD,
SIL_RT,
SIL_SYS,

6
include/uapi/asm-generic/siginfo.h
View File

@ -91,6 +91,8 @@ union __sifields {
char _dummy_pkey[__ADDR_BND_PKEY_PAD];
__u32 _pkey;
} _addr_pkey;
/* used when si_code=TRAP_PERF */
unsigned long _perf;
};
} _sigfault;
@ -155,6 +157,7 @@ typedef struct siginfo {
#define si_lower _sifields._sigfault._addr_bnd._lower
#define si_upper _sifields._sigfault._addr_bnd._upper
#define si_pkey _sifields._sigfault._addr_pkey._pkey
#define si_perf _sifields._sigfault._perf
#define si_band _sifields._sigpoll._band
#define si_fd _sifields._sigpoll._fd
#define si_call_addr _sifields._sigsys._call_addr
@ -253,7 +256,8 @@ typedef struct siginfo {
#define TRAP_BRANCH 3 /* process taken branch trap */
#define TRAP_HWBKPT 4 /* hardware breakpoint/watchpoint */
#define TRAP_UNK 5 /* undiagnosed trap */
#define NSIGTRAP 5
#define TRAP_PERF 6 /* perf event with sigtrap=1 */
#define NSIGTRAP 6
/*
* There is an additional set of SIGTRAP si_codes used by ptrace

28
include/uapi/linux/perf_event.h
View File

@ -37,6 +37,21 @@ enum perf_type_id {
PERF_TYPE_MAX, /* non-ABI */
};
/*
* attr.config layout for type PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE
* PERF_TYPE_HARDWARE: 0xEEEEEEEE000000AA
* AA: hardware event ID
* EEEEEEEE: PMU type ID
* PERF_TYPE_HW_CACHE: 0xEEEEEEEE00DDCCBB
* BB: hardware cache ID
* CC: hardware cache op ID
* DD: hardware cache op result ID
* EEEEEEEE: PMU type ID
* If the PMU type ID is 0, the PERF_TYPE_RAW will be applied.
*/
#define PERF_PMU_TYPE_SHIFT 32
#define PERF_HW_EVENT_MASK 0xffffffff
/*
* Generalized performance event event_id types, used by the
* attr.event_id parameter of the sys_perf_event_open()
@ -112,6 +127,7 @@ enum perf_sw_ids {
PERF_COUNT_SW_EMULATION_FAULTS = 8,
PERF_COUNT_SW_DUMMY = 9,
PERF_COUNT_SW_BPF_OUTPUT = 10,
PERF_COUNT_SW_CGROUP_SWITCHES = 11,
PERF_COUNT_SW_MAX, /* non-ABI */
};
@ -311,6 +327,7 @@ enum perf_event_read_format {
#define PERF_ATTR_SIZE_VER4 104 /* add: sample_regs_intr */
#define PERF_ATTR_SIZE_VER5 112 /* add: aux_watermark */
#define PERF_ATTR_SIZE_VER6 120 /* add: aux_sample_size */
#define PERF_ATTR_SIZE_VER7 128 /* add: sig_data */
/*
* Hardware event_id to monitor via a performance monitoring event:
@ -389,7 +406,10 @@ struct perf_event_attr {
cgroup : 1, /* include cgroup events */
text_poke : 1, /* include text poke events */
build_id : 1, /* use build id in mmap2 events */
__reserved_1 : 29;
inherit_thread : 1, /* children only inherit if cloned with CLONE_THREAD */
remove_on_exec : 1, /* event is removed from task on exec */
sigtrap : 1, /* send synchronous SIGTRAP on event */
__reserved_1 : 26;
union {
__u32 wakeup_events; /* wakeup every n events */
@ -441,6 +461,12 @@ struct perf_event_attr {
__u16 __reserved_2;
__u32 aux_sample_size;
__u32 __reserved_3;
/*
* User provided data if sigtrap=1, passed back to user via
* siginfo_t::si_perf, e.g. to permit user to identify the event.
*/
__u64 sig_data;
};
/*

4
include/uapi/linux/signalfd.h
View File

@ -39,6 +39,8 @@ struct signalfd_siginfo {
__s32 ssi_syscall;
__u64 ssi_call_addr;
__u32 ssi_arch;
__u32 __pad3;
__u64 ssi_perf;
/*
* Pad strcture to 128 bytes. Remember to update the
@ -49,7 +51,7 @@ struct signalfd_siginfo {
* comes out of a read(2) and we really don't want to have
* a compat on read(2).
*/
__u8 __pad[28];
__u8 __pad[16];
};

351
kernel/events/core.c
View File

@ -405,6 +405,7 @@ static LIST_HEAD(pmus);
static DEFINE_MUTEX(pmus_lock);
static struct srcu_struct pmus_srcu;
static cpumask_var_t perf_online_mask;
static struct kmem_cache *perf_event_cache;
/*
* perf event paranoia level:
@ -2204,6 +2205,26 @@ out:
perf_event__header_size(leader);
}
static void sync_child_event(struct perf_event *child_event);
static void perf_child_detach(struct perf_event *event)
{
struct perf_event *parent_event = event->parent;
if (!(event->attach_state & PERF_ATTACH_CHILD))
return;
event->attach_state &= ~PERF_ATTACH_CHILD;
if (WARN_ON_ONCE(!parent_event))
return;
lockdep_assert_held(&parent_event->child_mutex);
sync_child_event(event);
list_del_init(&event->child_list);
}
static bool is_orphaned_event(struct perf_event *event)
{
return event->state == PERF_EVENT_STATE_DEAD;
@ -2311,6 +2332,7 @@ group_sched_out(struct perf_event *group_event,
}
#define DETACH_GROUP 0x01UL
#define DETACH_CHILD 0x02UL
/*
* Cross CPU call to remove a performance event
@ -2334,6 +2356,8 @@ __perf_remove_from_context(struct perf_event *event,
event_sched_out(event, cpuctx, ctx);
if (flags & DETACH_GROUP)
perf_group_detach(event);
if (flags & DETACH_CHILD)
perf_child_detach(event);
list_del_event(event, ctx);
if (!ctx->nr_events && ctx->is_active) {
@ -2362,25 +2386,21 @@ static void perf_remove_from_context(struct perf_event *event, unsigned long fla
lockdep_assert_held(&ctx->mutex);
event_function_call(event, __perf_remove_from_context, (void *)flags);
/*
* The above event_function_call() can NO-OP when it hits
* TASK_TOMBSTONE. In that case we must already have been detached
* from the context (by perf_event_exit_event()) but the grouping
* might still be in-tact.
* Because of perf_event_exit_task(), perf_remove_from_context() ought
* to work in the face of TASK_TOMBSTONE, unlike every other
* event_function_call() user.
*/
WARN_ON_ONCE(event->attach_state & PERF_ATTACH_CONTEXT);
if ((flags & DETACH_GROUP) &&
(event->attach_state & PERF_ATTACH_GROUP)) {
/*
* Since in that case we cannot possibly be scheduled, simply
* detach now.
*/
raw_spin_lock_irq(&ctx->lock);
perf_group_detach(event);
raw_spin_lock_irq(&ctx->lock);
if (!ctx->is_active) {
__perf_remove_from_context(event, __get_cpu_context(ctx),
ctx, (void *)flags);
raw_spin_unlock_irq(&ctx->lock);
return;
}
raw_spin_unlock_irq(&ctx->lock);
event_function_call(event, __perf_remove_from_context, (void *)flags);
}
/*
@ -3180,16 +3200,36 @@ static int perf_event_modify_breakpoint(struct perf_event *bp,
static int perf_event_modify_attr(struct perf_event *event,
struct perf_event_attr *attr)
{
int (*func)(struct perf_event *, struct perf_event_attr *);
struct perf_event *child;
int err;
if (event->attr.type != attr->type)
return -EINVAL;
switch (event->attr.type) {
case PERF_TYPE_BREAKPOINT:
return perf_event_modify_breakpoint(event, attr);
func = perf_event_modify_breakpoint;
break;
default:
/* Place holder for future additions. */
return -EOPNOTSUPP;
}
WARN_ON_ONCE(event->ctx->parent_ctx);
mutex_lock(&event->child_mutex);
err = func(event, attr);
if (err)
goto out;
list_for_each_entry(child, &event->child_list, child_list) {
err = func(child, attr);
if (err)
goto out;
}
out:
mutex_unlock(&event->child_mutex);
return err;
}
static void ctx_sched_out(struct perf_event_context *ctx,
@ -4208,6 +4248,57 @@ out:
put_ctx(clone_ctx);
}
static void perf_remove_from_owner(struct perf_event *event);
static void perf_event_exit_event(struct perf_event *event,
struct perf_event_context *ctx);
/*
* Removes all events from the current task that have been marked
* remove-on-exec, and feeds their values back to parent events.
*/
static void perf_event_remove_on_exec(int ctxn)
{
struct perf_event_context *ctx, *clone_ctx = NULL;
struct perf_event *event, *next;
LIST_HEAD(free_list);
unsigned long flags;
bool modified = false;
ctx = perf_pin_task_context(current, ctxn);
if (!ctx)
return;
mutex_lock(&ctx->mutex);
if (WARN_ON_ONCE(ctx->task != current))
goto unlock;
list_for_each_entry_safe(event, next, &ctx->event_list, event_entry) {
if (!event->attr.remove_on_exec)
continue;
if (!is_kernel_event(event))
perf_remove_from_owner(event);
modified = true;
perf_event_exit_event(event, ctx);
}
raw_spin_lock_irqsave(&ctx->lock, flags);
if (modified)
clone_ctx = unclone_ctx(ctx);
--ctx->pin_count;
raw_spin_unlock_irqrestore(&ctx->lock, flags);
unlock:
mutex_unlock(&ctx->mutex);
put_ctx(ctx);
if (clone_ctx)
put_ctx(clone_ctx);
}
struct perf_read_data {
struct perf_event *event;
bool group;
@ -4611,7 +4702,7 @@ static void free_event_rcu(struct rcu_head *head)
if (event->ns)
put_pid_ns(event->ns);
perf_event_free_filter(event);
kfree(event);
kmem_cache_free(perf_event_cache, event);
}
static void ring_buffer_attach(struct perf_event *event,
@ -6301,6 +6392,33 @@ void perf_event_wakeup(struct perf_event *event)
}
}
static void perf_sigtrap(struct perf_event *event)
{
struct kernel_siginfo info;
/*
* We'd expect this to only occur if the irq_work is delayed and either
* ctx->task or current has changed in the meantime. This can be the
* case on architectures that do not implement arch_irq_work_raise().
*/
if (WARN_ON_ONCE(event->ctx->task != current))
return;
/*
* perf_pending_event() can race with the task exiting.
*/
if (current->flags & PF_EXITING)
return;
clear_siginfo(&info);
info.si_signo = SIGTRAP;
info.si_code = TRAP_PERF;
info.si_errno = event->attr.type;
info.si_perf = event->attr.sig_data;
info.si_addr = (void __user *)event->pending_addr;
force_sig_info(&info);
}
static void perf_pending_event_disable(struct perf_event *event)
{
int cpu = READ_ONCE(event->pending_disable);
@ -6310,6 +6428,13 @@ static void perf_pending_event_disable(struct perf_event *event)
if (cpu == smp_processor_id()) {
WRITE_ONCE(event->pending_disable, -1);
if (event->attr.sigtrap) {
perf_sigtrap(event);
atomic_set_release(&event->event_limit, 1); /* rearm event */
return;
}
perf_event_disable_local(event);
return;
}
@ -7520,18 +7645,18 @@ void perf_event_exec(void)
struct perf_event_context *ctx;
int ctxn;
rcu_read_lock();
for_each_task_context_nr(ctxn) {
ctx = current->perf_event_ctxp[ctxn];
if (!ctx)
continue;
perf_event_enable_on_exec(ctxn);
perf_event_remove_on_exec(ctxn);
perf_iterate_ctx(ctx, perf_event_addr_filters_exec, NULL,
true);
rcu_read_lock();
ctx = rcu_dereference(current->perf_event_ctxp[ctxn]);
if (ctx) {
perf_iterate_ctx(ctx, perf_event_addr_filters_exec,
NULL, true);
}
rcu_read_unlock();
}
rcu_read_unlock();
}
struct remote_output {
@ -9012,6 +9137,7 @@ static int __perf_event_overflow(struct perf_event *event,
if (events && atomic_dec_and_test(&event->event_limit)) {
ret = 1;
event->pending_kill = POLL_HUP;
event->pending_addr = data->addr;
perf_event_disable_inatomic(event);
}
@ -11094,6 +11220,7 @@ static int perf_try_init_event(struct pmu *pmu, struct perf_event *event)
static struct pmu *perf_init_event(struct perf_event *event)
{
bool extended_type = false;
int idx, type, ret;
struct pmu *pmu;
@ -11112,16 +11239,27 @@ static struct pmu *perf_init_event(struct perf_event *event)
* are often aliases for PERF_TYPE_RAW.
*/
type = event->attr.type;
if (type == PERF_TYPE_HARDWARE || type == PERF_TYPE_HW_CACHE)
type = PERF_TYPE_RAW;
if (type == PERF_TYPE_HARDWARE || type == PERF_TYPE_HW_CACHE) {
type = event->attr.config >> PERF_PMU_TYPE_SHIFT;
if (!type) {
type = PERF_TYPE_RAW;
} else {
extended_type = true;
event->attr.config &= PERF_HW_EVENT_MASK;
}
}
again:
rcu_read_lock();
pmu = idr_find(&pmu_idr, type);
rcu_read_unlock();
if (pmu) {
if (event->attr.type != type && type != PERF_TYPE_RAW &&
!(pmu->capabilities & PERF_PMU_CAP_EXTENDED_HW_TYPE))
goto fail;
ret = perf_try_init_event(pmu, event);
if (ret == -ENOENT && event->attr.type != type) {
if (ret == -ENOENT && event->attr.type != type && !extended_type) {
type = event->attr.type;
goto again;
}
@ -11142,6 +11280,7 @@ again:
goto unlock;
}
}
fail:
pmu = ERR_PTR(-ENOENT);
unlock:
srcu_read_unlock(&pmus_srcu, idx);
@ -11287,13 +11426,20 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
struct perf_event *event;
struct hw_perf_event *hwc;
long err = -EINVAL;
int node;
if ((unsigned)cpu >= nr_cpu_ids) {
if (!task || cpu != -1)
return ERR_PTR(-EINVAL);
}
if (attr->sigtrap && !task) {
/* Requires a task: avoid signalling random tasks. */
return ERR_PTR(-EINVAL);
}
event = kzalloc(sizeof(*event), GFP_KERNEL);
node = (cpu >= 0) ? cpu_to_node(cpu) : -1;
event = kmem_cache_alloc_node(perf_event_cache, GFP_KERNEL | __GFP_ZERO,
node);
if (!event)
return ERR_PTR(-ENOMEM);
@ -11338,6 +11484,9 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
event->state = PERF_EVENT_STATE_INACTIVE;
if (event->attr.sigtrap)
atomic_set(&event->event_limit, 1);
if (task) {
event->attach_state = PERF_ATTACH_TASK;
/*
@ -11497,7 +11646,7 @@ err_ns:
put_pid_ns(event->ns);
if (event->hw.target)
put_task_struct(event->hw.target);
kfree(event);
kmem_cache_free(perf_event_cache, event);
return ERR_PTR(err);
}
@ -11610,6 +11759,15 @@ static int perf_copy_attr(struct perf_event_attr __user *uattr,
(attr->sample_type & PERF_SAMPLE_WEIGHT_STRUCT))
return -EINVAL;
if (!attr->inherit && attr->inherit_thread)
return -EINVAL;
if (attr->remove_on_exec && attr->enable_on_exec)
return -EINVAL;
if (attr->sigtrap && !attr->remove_on_exec)
return -EINVAL;
out:
return ret;
@ -11829,12 +11987,12 @@ SYSCALL_DEFINE5(perf_event_open,
return err;
}
err = security_locked_down(LOCKDOWN_PERF);
if (err && (attr.sample_type & PERF_SAMPLE_REGS_INTR))
/* REGS_INTR can leak data, lockdown must prevent this */
return err;
err = 0;
/* REGS_INTR can leak data, lockdown must prevent this */
if (attr.sample_type & PERF_SAMPLE_REGS_INTR) {
err = security_locked_down(LOCKDOWN_PERF);
if (err)
return err;
}
/*
* In cgroup mode, the pid argument is used to pass the fd
@ -12373,14 +12531,17 @@ void perf_pmu_migrate_context(struct pmu *pmu, int src_cpu, int dst_cpu)
}
EXPORT_SYMBOL_GPL(perf_pmu_migrate_context);
static void sync_child_event(struct perf_event *child_event,
struct task_struct *child)
static void sync_child_event(struct perf_event *child_event)
{
struct perf_event *parent_event = child_event->parent;
u64 child_val;
if (child_event->attr.inherit_stat)
perf_event_read_event(child_event, child);
if (child_event->attr.inherit_stat) {
struct task_struct *task = child_event->ctx->task;
if (task && task != TASK_TOMBSTONE)
perf_event_read_event(child_event, task);
}
child_val = perf_event_count(child_event);
@ -12395,60 +12556,53 @@ static void sync_child_event(struct perf_event *child_event,
}
static void
perf_event_exit_event(struct perf_event *child_event,
struct perf_event_context *child_ctx,
struct task_struct *child)
perf_event_exit_event(struct perf_event *event, struct perf_event_context *ctx)
{
struct perf_event *parent_event = child_event->parent;
struct perf_event *parent_event = event->parent;
unsigned long detach_flags = 0;
if (parent_event) {
/*
* Do not destroy the 'original' grouping; because of the
* context switch optimization the original events could've
* ended up in a random child task.
*
* If we were to destroy the original group, all group related
* operations would cease to function properly after this
* random child dies.
*
* Do destroy all inherited groups, we don't care about those
* and being thorough is better.
*/
detach_flags = DETACH_GROUP | DETACH_CHILD;
mutex_lock(&parent_event->child_mutex);
}
perf_remove_from_context(event, detach_flags);
raw_spin_lock_irq(&ctx->lock);
if (event->state > PERF_EVENT_STATE_EXIT)
perf_event_set_state(event, PERF_EVENT_STATE_EXIT);
raw_spin_unlock_irq(&ctx->lock);
/*
* Do not destroy the 'original' grouping; because of the context
* switch optimization the original events could've ended up in a
* random child task.
*
* If we were to destroy the original group, all group related
* operations would cease to function properly after this random
* child dies.
*
* Do destroy all inherited groups, we don't care about those
* and being thorough is better.
* Child events can be freed.
*/
raw_spin_lock_irq(&child_ctx->lock);
WARN_ON_ONCE(child_ctx->is_active);
if (parent_event)
perf_group_detach(child_event);
list_del_event(child_event, child_ctx);
perf_event_set_state(child_event, PERF_EVENT_STATE_EXIT); /* is_event_hup() */
raw_spin_unlock_irq(&child_ctx->lock);
if (parent_event) {
mutex_unlock(&parent_event->child_mutex);
/*
* Kick perf_poll() for is_event_hup();
*/
perf_event_wakeup(parent_event);
free_event(event);
put_event(parent_event);
return;
}
/*
* Parent events are governed by their filedesc, retain them.
*/
if (!parent_event) {
perf_event_wakeup(child_event);
return;
}
/*
* Child events can be cleaned up.
*/
sync_child_event(child_event, child);
/*
* Remove this event from the parent's list
*/
WARN_ON_ONCE(parent_event->ctx->parent_ctx);
mutex_lock(&parent_event->child_mutex);
list_del_init(&child_event->child_list);
mutex_unlock(&parent_event->child_mutex);
/*
* Kick perf_poll() for is_event_hup().
*/
perf_event_wakeup(parent_event);
free_event(child_event);
put_event(parent_event);
perf_event_wakeup(event);
}
static void perf_event_exit_task_context(struct task_struct *child, int ctxn)
@ -12505,7 +12659,7 @@ static void perf_event_exit_task_context(struct task_struct *child, int ctxn)
perf_event_task(child, child_ctx, 0);
list_for_each_entry_safe(child_event, next, &child_ctx->event_list, event_entry)
perf_event_exit_event(child_event, child_ctx, child);
perf_event_exit_event(child_event, child_ctx);
mutex_unlock(&child_ctx->mutex);
@ -12765,6 +12919,7 @@ inherit_event(struct perf_event *parent_event,
*/
raw_spin_lock_irqsave(&child_ctx->lock, flags);
add_event_to_ctx(child_event, child_ctx);
child_event->attach_state |= PERF_ATTACH_CHILD;
raw_spin_unlock_irqrestore(&child_ctx->lock, flags);
/*
@ -12833,12 +12988,15 @@ static int
inherit_task_group(struct perf_event *event, struct task_struct *parent,
struct perf_event_context *parent_ctx,
struct task_struct *child, int ctxn,
int *inherited_all)
u64 clone_flags, int *inherited_all)
{
int ret;
struct perf_event_context *child_ctx;
if (!event->attr.inherit) {
if (!event->attr.inherit ||
(event->attr.inherit_thread && !(clone_flags & CLONE_THREAD)) ||
/* Do not inherit if sigtrap and signal handlers were cleared. */
(event->attr.sigtrap && (clone_flags & CLONE_CLEAR_SIGHAND))) {
*inherited_all = 0;
return 0;
}
@ -12870,7 +13028,8 @@ inherit_task_group(struct perf_event *event, struct task_struct *parent,
/*
* Initialize the perf_event context in task_struct
*/
static int perf_event_init_context(struct task_struct *child, int ctxn)
static int perf_event_init_context(struct task_struct *child, int ctxn,
u64 clone_flags)
{
struct perf_event_context *child_ctx, *parent_ctx;
struct perf_event_context *cloned_ctx;
@ -12910,7 +13069,8 @@ static int perf_event_init_context(struct task_struct *child, int ctxn)
*/
perf_event_groups_for_each(event, &parent_ctx->pinned_groups) {
ret = inherit_task_group(event, parent, parent_ctx,
child, ctxn, &inherited_all);
child, ctxn, clone_flags,
&inherited_all);
if (ret)
goto out_unlock;
}
@ -12926,7 +13086,8 @@ static int perf_event_init_context(struct task_struct *child, int ctxn)
perf_event_groups_for_each(event, &parent_ctx->flexible_groups) {
ret = inherit_task_group(event, parent, parent_ctx,
child, ctxn, &inherited_all);
child, ctxn, clone_flags,
&inherited_all);
if (ret)
goto out_unlock;
}
@ -12968,7 +13129,7 @@ out_unlock:
/*
* Initialize the perf_event context in task_struct
*/
int perf_event_init_task(struct task_struct *child)
int perf_event_init_task(struct task_struct *child, u64 clone_flags)
{
int ctxn, ret;
@ -12977,7 +13138,7 @@ int perf_event_init_task(struct task_struct *child)
INIT_LIST_HEAD(&child->perf_event_list);
for_each_task_context_nr(ctxn) {
ret = perf_event_init_context(child, ctxn);
ret = perf_event_init_context(child, ctxn, clone_flags);
if (ret) {
perf_event_free_task(child);
return ret;
@ -13130,6 +13291,8 @@ void __init perf_event_init(void)
ret = init_hw_breakpoint();
WARN(ret, "hw_breakpoint initialization failed with: %d", ret);
perf_event_cache = KMEM_CACHE(perf_event, SLAB_PANIC);
/*
* Build time assertion that we keep the data_head at the intended
* location. IOW, validation we got the __reserved[] size right.

43
kernel/events/ring_buffer.c
View File

@ -674,21 +674,26 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
if (!has_aux(event))
return -EOPNOTSUPP;
/*
* We need to start with the max_order that fits in nr_pages,
* not the other way around, hence ilog2() and not get_order.
*/
max_order = ilog2(nr_pages);
/*
* PMU requests more than one contiguous chunks of memory
* for SW double buffering
*/
if (!overwrite) {
if (!max_order)
return -EINVAL;
/*
* Watermark defaults to half the buffer, and so does the
* max_order, to aid PMU drivers in double buffering.
*/
if (!watermark)
watermark = nr_pages << (PAGE_SHIFT - 1);
max_order--;
/*
* Use aux_watermark as the basis for chunking to
* help PMU drivers honor the watermark.
*/
max_order = get_order(watermark);
} else {
/*
* We need to start with the max_order that fits in nr_pages,
* not the other way around, hence ilog2() and not get_order.
*/
max_order = ilog2(nr_pages);
watermark = 0;
}
rb->aux_pages = kcalloc_node(nr_pages, sizeof(void *), GFP_KERNEL,
@ -743,9 +748,6 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
rb->aux_overwrite = overwrite;
rb->aux_watermark = watermark;
if (!rb->aux_watermark && !rb->aux_overwrite)
rb->aux_watermark = nr_pages << (PAGE_SHIFT - 1);
out:
if (!ret)
rb->aux_pgoff = pgoff;
@ -804,7 +806,7 @@ struct perf_buffer *rb_alloc(int nr_pages, long watermark, int cpu, int flags)
{
struct perf_buffer *rb;
unsigned long size;
int i;
int i, node;
size = sizeof(struct perf_buffer);
size += nr_pages * sizeof(void *);
@ -812,7 +814,8 @@ struct perf_buffer *rb_alloc(int nr_pages, long watermark, int cpu, int flags)
if (order_base_2(size) >= PAGE_SHIFT+MAX_ORDER)
goto fail;
rb = kzalloc(size, GFP_KERNEL);
node = (cpu == -1) ? cpu : cpu_to_node(cpu);
rb = kzalloc_node(size, GFP_KERNEL, node);
if (!rb)
goto fail;
@ -906,11 +909,13 @@ struct perf_buffer *rb_alloc(int nr_pages, long watermark, int cpu, int flags)
struct perf_buffer *rb;
unsigned long size;
void *all_buf;
int node;
size = sizeof(struct perf_buffer);
size += sizeof(void *);
rb = kzalloc(size, GFP_KERNEL);
node = (cpu == -1) ? cpu : cpu_to_node(cpu);
rb = kzalloc_node(size, GFP_KERNEL, node);
if (!rb)
goto fail;

2
kernel/fork.c
View File

@ -2084,7 +2084,7 @@ static __latent_entropy struct task_struct *copy_process(
if (retval)
goto bad_fork_cleanup_policy;
retval = perf_event_init_task(p);
retval = perf_event_init_task(p, clone_flags);
if (retval)
goto bad_fork_cleanup_policy;
retval = audit_alloc(p);

13
kernel/signal.c
View File

@ -1197,6 +1197,7 @@ static inline bool has_si_pid_and_uid(struct kernel_siginfo *info)
case SIL_FAULT_MCEERR:
case SIL_FAULT_BNDERR:
case SIL_FAULT_PKUERR:
case SIL_PERF_EVENT:
case SIL_SYS:
ret = false;
break;
@ -2529,6 +2530,7 @@ static void hide_si_addr_tag_bits(struct ksignal *ksig)
case SIL_FAULT_MCEERR:
case SIL_FAULT_BNDERR:
case SIL_FAULT_PKUERR:
case SIL_PERF_EVENT:
ksig->info.si_addr = arch_untagged_si_addr(
ksig->info.si_addr, ksig->sig, ksig->info.si_code);
break;
@ -3210,6 +3212,8 @@ enum siginfo_layout siginfo_layout(unsigned sig, int si_code)
else if ((sig == SIGSEGV) && (si_code == SEGV_PKUERR))
layout = SIL_FAULT_PKUERR;
#endif
else if ((sig == SIGTRAP) && (si_code == TRAP_PERF))
layout = SIL_PERF_EVENT;
}
else if (si_code <= NSIGPOLL)
layout = SIL_POLL;
@ -3339,6 +3343,10 @@ void copy_siginfo_to_external32(struct compat_siginfo *to,
#endif
to->si_pkey = from->si_pkey;
break;
case SIL_PERF_EVENT:
to->si_addr = ptr_to_compat(from->si_addr);
to->si_perf = from->si_perf;
break;
case SIL_CHLD:
to->si_pid = from->si_pid;
to->si_uid = from->si_uid;
@ -3419,6 +3427,10 @@ static int post_copy_siginfo_from_user32(kernel_siginfo_t *to,
#endif
to->si_pkey = from->si_pkey;
break;
case SIL_PERF_EVENT:
to->si_addr = compat_ptr(from->si_addr);
to->si_perf = from->si_perf;
break;
case SIL_CHLD:
to->si_pid = from->si_pid;
to->si_uid = from->si_uid;
@ -4599,6 +4611,7 @@ static inline void siginfo_buildtime_checks(void)
CHECK_OFFSET(si_lower);
CHECK_OFFSET(si_upper);
CHECK_OFFSET(si_pkey);
CHECK_OFFSET(si_perf);
/* sigpoll */
CHECK_OFFSET(si_band);

6
tools/perf/arch/x86/util/intel-pt.c
View File

@ -776,6 +776,12 @@ static int intel_pt_recording_options(struct auxtrace_record *itr,
}
}
if (!opts->auxtrace_snapshot_mode && !opts->auxtrace_sample_mode) {
u32 aux_watermark = opts->auxtrace_mmap_pages * page_size / 4;
intel_pt_evsel->core.attr.aux_watermark = aux_watermark;
}
intel_pt_parse_terms(intel_pt_pmu->name, &intel_pt_pmu->format,
"tsc", &tsc_bit);

3
tools/testing/selftests/perf_events/.gitignore vendored Normal file
View File

@ -0,0 +1,3 @@
# SPDX-License-Identifier: GPL-2.0-only
sigtrap_threads
remove_on_exec

6
tools/testing/selftests/perf_events/Makefile Normal file
View File

@ -0,0 +1,6 @@
# SPDX-License-Identifier: GPL-2.0
CFLAGS += -Wl,-no-as-needed -Wall -I../../../../usr/include
LDFLAGS += -lpthread
TEST_GEN_PROGS := sigtrap_threads remove_on_exec
include ../lib.mk

1
tools/testing/selftests/perf_events/config Normal file
View File

@ -0,0 +1 @@
CONFIG_PERF_EVENTS=y

260
tools/testing/selftests/perf_events/remove_on_exec.c Normal file
View File

@ -0,0 +1,260 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Test for remove_on_exec.
*
* Copyright (C) 2021, Google LLC.
*/
#define _GNU_SOURCE
/* We need the latest siginfo from the kernel repo. */
#include <sys/types.h>
#include <asm/siginfo.h>
#define __have_siginfo_t 1
#define __have_sigval_t 1
#define __have_sigevent_t 1
#define __siginfo_t_defined
#define __sigval_t_defined
#define __sigevent_t_defined
#define _BITS_SIGINFO_CONSTS_H 1
#define _BITS_SIGEVENT_CONSTS_H 1
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <linux/perf_event.h>
#include <pthread.h>
#include <signal.h>
#include <sys/ioctl.h>
#include <sys/syscall.h>
#include <unistd.h>
#include "../kselftest_harness.h"
static volatile int signal_count;
static struct perf_event_attr make_event_attr(void)
{
struct perf_event_attr attr = {
.type = PERF_TYPE_HARDWARE,
.size = sizeof(attr),
.config = PERF_COUNT_HW_INSTRUCTIONS,
.sample_period = 1000,
.exclude_kernel = 1,
.exclude_hv = 1,
.disabled = 1,
.inherit = 1,
/*
* Children normally retain their inherited event on exec; with
* remove_on_exec, we'll remove their event, but the parent and
* any other non-exec'd children will keep their events.
*/
.remove_on_exec = 1,
.sigtrap = 1,
};
return attr;
}
static void sigtrap_handler(int signum, siginfo_t *info, void *ucontext)
{
if (info->si_code != TRAP_PERF) {
fprintf(stderr, "%s: unexpected si_code %d\n", __func__, info->si_code);
return;
}
signal_count++;
}
FIXTURE(remove_on_exec)
{
struct sigaction oldact;
int fd;
};
FIXTURE_SETUP(remove_on_exec)
{
struct perf_event_attr attr = make_event_attr();
struct sigaction action = {};
signal_count = 0;
/* Initialize sigtrap handler. */
action.sa_flags = SA_SIGINFO | SA_NODEFER;
action.sa_sigaction = sigtrap_handler;
sigemptyset(&action.sa_mask);
ASSERT_EQ(sigaction(SIGTRAP, &action, &self->oldact), 0);
/* Initialize perf event. */
self->fd = syscall(__NR_perf_event_open, &attr, 0, -1, -1, PERF_FLAG_FD_CLOEXEC);
ASSERT_NE(self->fd, -1);
}
FIXTURE_TEARDOWN(remove_on_exec)
{
close(self->fd);
sigaction(SIGTRAP, &self->oldact, NULL);
}
/* Verify event propagates to fork'd child. */
TEST_F(remove_on_exec, fork_only)
{
int status;
pid_t pid = fork();
if (pid == 0) {
ASSERT_EQ(signal_count, 0);
ASSERT_EQ(ioctl(self->fd, PERF_EVENT_IOC_ENABLE, 0), 0);
while (!signal_count);
_exit(42);
}
while (!signal_count); /* Child enables event. */
EXPECT_EQ(waitpid(pid, &status, 0), pid);
EXPECT_EQ(WEXITSTATUS(status), 42);
}
/*
* Verify that event does _not_ propagate to fork+exec'd child; event enabled
* after fork+exec.
*/
TEST_F(remove_on_exec, fork_exec_then_enable)
{
pid_t pid_exec, pid_only_fork;
int pipefd[2];
int tmp;
/*
* Non-exec child, to ensure exec does not affect inherited events of
* other children.
*/
pid_only_fork = fork();
if (pid_only_fork == 0) {
/* Block until parent enables event. */
while (!signal_count);
_exit(42);
}
ASSERT_NE(pipe(pipefd), -1);
pid_exec = fork();
if (pid_exec == 0) {
ASSERT_NE(dup2(pipefd[1], STDOUT_FILENO), -1);
close(pipefd[0]);
execl("/proc/self/exe", "exec_child", NULL);
_exit((perror("exec failed"), 1));
}
close(pipefd[1]);
ASSERT_EQ(waitpid(pid_exec, &tmp, WNOHANG), 0); /* Child is running. */
/* Wait for exec'd child to start spinning. */
EXPECT_EQ(read(pipefd[0], &tmp, sizeof(int)), sizeof(int));
EXPECT_EQ(tmp, 42);
close(pipefd[0]);
/* Now we can enable the event, knowing the child is doing work. */
EXPECT_EQ(ioctl(self->fd, PERF_EVENT_IOC_ENABLE, 0), 0);
/* If the event propagated to the exec'd child, it will exit normally... */
usleep(100000); /* ... give time for event to trigger (in case of bug). */
EXPECT_EQ(waitpid(pid_exec, &tmp, WNOHANG), 0); /* Should still be running. */
EXPECT_EQ(kill(pid_exec, SIGKILL), 0);
/* Verify removal from child did not affect this task's event. */
tmp = signal_count;
while (signal_count == tmp); /* Should not hang! */
/* Nor should it have affected the first child. */
EXPECT_EQ(waitpid(pid_only_fork, &tmp, 0), pid_only_fork);
EXPECT_EQ(WEXITSTATUS(tmp), 42);
}
/*
* Verify that event does _not_ propagate to fork+exec'd child; event enabled
* before fork+exec.
*/
TEST_F(remove_on_exec, enable_then_fork_exec)
{
pid_t pid_exec;
int tmp;
EXPECT_EQ(ioctl(self->fd, PERF_EVENT_IOC_ENABLE, 0), 0);
pid_exec = fork();
if (pid_exec == 0) {
execl("/proc/self/exe", "exec_child", NULL);
_exit((perror("exec failed"), 1));
}
/*
* The child may exit abnormally at any time if the event propagated and
* a SIGTRAP is sent before the handler was set up.
*/
usleep(100000); /* ... give time for event to trigger (in case of bug). */
EXPECT_EQ(waitpid(pid_exec, &tmp, WNOHANG), 0); /* Should still be running. */
EXPECT_EQ(kill(pid_exec, SIGKILL), 0);
/* Verify removal from child did not affect this task's event. */
tmp = signal_count;
while (signal_count == tmp); /* Should not hang! */
}
TEST_F(remove_on_exec, exec_stress)
{
pid_t pids[30];
int i, tmp;
for (i = 0; i < sizeof(pids) / sizeof(pids[0]); i++) {
pids[i] = fork();
if (pids[i] == 0) {
execl("/proc/self/exe", "exec_child", NULL);
_exit((perror("exec failed"), 1));
}
/* Some forked with event disabled, rest with enabled. */
if (i > 10)
EXPECT_EQ(ioctl(self->fd, PERF_EVENT_IOC_ENABLE, 0), 0);
}
usleep(100000); /* ... give time for event to trigger (in case of bug). */
for (i = 0; i < sizeof(pids) / sizeof(pids[0]); i++) {
/* All children should still be running. */
EXPECT_EQ(waitpid(pids[i], &tmp, WNOHANG), 0);
EXPECT_EQ(kill(pids[i], SIGKILL), 0);
}
/* Verify event is still alive. */
tmp = signal_count;
while (signal_count == tmp);
}
/* For exec'd child. */
static void exec_child(void)
{
struct sigaction action = {};
const int val = 42;
/* Set up sigtrap handler in case we erroneously receive a trap. */
action.sa_flags = SA_SIGINFO | SA_NODEFER;
action.sa_sigaction = sigtrap_handler;
sigemptyset(&action.sa_mask);
if (sigaction(SIGTRAP, &action, NULL))
_exit((perror("sigaction failed"), 1));
/* Signal parent that we're starting to spin. */
if (write(STDOUT_FILENO, &val, sizeof(int)) == -1)
_exit((perror("write failed"), 1));
/* Should hang here until killed. */
while (!signal_count);
}
#define main test_main
TEST_HARNESS_MAIN
#undef main
int main(int argc, char *argv[])
{
if (!strcmp(argv[0], "exec_child")) {
exec_child();
return 1;
}
return test_main(argc, argv);
}

1
tools/testing/selftests/perf_events/settings Normal file
View File

@ -0,0 +1 @@
timeout=120

210
tools/testing/selftests/perf_events/sigtrap_threads.c Normal file
View File

@ -0,0 +1,210 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Test for perf events with SIGTRAP across all threads.
*
* Copyright (C) 2021, Google LLC.
*/
#define _GNU_SOURCE
/* We need the latest siginfo from the kernel repo. */
#include <sys/types.h>
#include <asm/siginfo.h>
#define __have_siginfo_t 1
#define __have_sigval_t 1
#define __have_sigevent_t 1
#define __siginfo_t_defined
#define __sigval_t_defined
#define __sigevent_t_defined
#define _BITS_SIGINFO_CONSTS_H 1
#define _BITS_SIGEVENT_CONSTS_H 1
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <linux/hw_breakpoint.h>
#include <linux/perf_event.h>
#include <pthread.h>
#include <signal.h>
#include <sys/ioctl.h>
#include <sys/syscall.h>
#include <unistd.h>
#include "../kselftest_harness.h"
#define NUM_THREADS 5
/* Data shared between test body, threads, and signal handler. */
static struct {
int tids_want_signal; /* Which threads still want a signal. */
int signal_count; /* Sanity check number of signals received. */
volatile int iterate_on; /* Variable to set breakpoint on. */
siginfo_t first_siginfo; /* First observed siginfo_t. */
} ctx;
/* Unique value to check si_perf is correctly set from perf_event_attr::sig_data. */
#define TEST_SIG_DATA(addr) (~(unsigned long)(addr))
static struct perf_event_attr make_event_attr(bool enabled, volatile void *addr)
{
struct perf_event_attr attr = {
.type = PERF_TYPE_BREAKPOINT,
.size = sizeof(attr),
.sample_period = 1,
.disabled = !enabled,
.bp_addr = (unsigned long)addr,
.bp_type = HW_BREAKPOINT_RW,
.bp_len = HW_BREAKPOINT_LEN_1,
.inherit = 1, /* Children inherit events ... */
.inherit_thread = 1, /* ... but only cloned with CLONE_THREAD. */
.remove_on_exec = 1, /* Required by sigtrap. */
.sigtrap = 1, /* Request synchronous SIGTRAP on event. */
.sig_data = TEST_SIG_DATA(addr),
};
return attr;
}
static void sigtrap_handler(int signum, siginfo_t *info, void *ucontext)
{
if (info->si_code != TRAP_PERF) {
fprintf(stderr, "%s: unexpected si_code %d\n", __func__, info->si_code);
return;
}
/*
* The data in siginfo_t we're interested in should all be the same
* across threads.
*/
if (!__atomic_fetch_add(&ctx.signal_count, 1, __ATOMIC_RELAXED))
ctx.first_siginfo = *info;
__atomic_fetch_sub(&ctx.tids_want_signal, syscall(__NR_gettid), __ATOMIC_RELAXED);
}
static void *test_thread(void *arg)
{
pthread_barrier_t *barrier = (pthread_barrier_t *)arg;
pid_t tid = syscall(__NR_gettid);
int iter;
int i;
pthread_barrier_wait(barrier);
__atomic_fetch_add(&ctx.tids_want_signal, tid, __ATOMIC_RELAXED);
iter = ctx.iterate_on; /* read */
for (i = 0; i < iter - 1; i++) {
__atomic_fetch_add(&ctx.tids_want_signal, tid, __ATOMIC_RELAXED);
ctx.iterate_on = iter; /* idempotent write */
}
return NULL;
}
FIXTURE(sigtrap_threads)
{
struct sigaction oldact;
pthread_t threads[NUM_THREADS];
pthread_barrier_t barrier;
int fd;
};
FIXTURE_SETUP(sigtrap_threads)
{
struct perf_event_attr attr = make_event_attr(false, &ctx.iterate_on);
struct sigaction action = {};
int i;
memset(&ctx, 0, sizeof(ctx));
/* Initialize sigtrap handler. */
action.sa_flags = SA_SIGINFO | SA_NODEFER;
action.sa_sigaction = sigtrap_handler;
sigemptyset(&action.sa_mask);
ASSERT_EQ(sigaction(SIGTRAP, &action, &self->oldact), 0);
/* Initialize perf event. */
self->fd = syscall(__NR_perf_event_open, &attr, 0, -1, -1, PERF_FLAG_FD_CLOEXEC);
ASSERT_NE(self->fd, -1);
/* Spawn threads inheriting perf event. */
pthread_barrier_init(&self->barrier, NULL, NUM_THREADS + 1);
for (i = 0; i < NUM_THREADS; i++)
ASSERT_EQ(pthread_create(&self->threads[i], NULL, test_thread, &self->barrier), 0);
}
FIXTURE_TEARDOWN(sigtrap_threads)
{
pthread_barrier_destroy(&self->barrier);
close(self->fd);
sigaction(SIGTRAP, &self->oldact, NULL);
}
static void run_test_threads(struct __test_metadata *_metadata,
FIXTURE_DATA(sigtrap_threads) *self)
{
int i;
pthread_barrier_wait(&self->barrier);
for (i = 0; i < NUM_THREADS; i++)
ASSERT_EQ(pthread_join(self->threads[i], NULL), 0);
}
TEST_F(sigtrap_threads, remain_disabled)
{
run_test_threads(_metadata, self);
EXPECT_EQ(ctx.signal_count, 0);
EXPECT_NE(ctx.tids_want_signal, 0);
}
TEST_F(sigtrap_threads, enable_event)
{
EXPECT_EQ(ioctl(self->fd, PERF_EVENT_IOC_ENABLE, 0), 0);
run_test_threads(_metadata, self);
EXPECT_EQ(ctx.signal_count, NUM_THREADS);
EXPECT_EQ(ctx.tids_want_signal, 0);
EXPECT_EQ(ctx.first_siginfo.si_addr, &ctx.iterate_on);
EXPECT_EQ(ctx.first_siginfo.si_errno, PERF_TYPE_BREAKPOINT);
EXPECT_EQ(ctx.first_siginfo.si_perf, TEST_SIG_DATA(&ctx.iterate_on));
/* Check enabled for parent. */
ctx.iterate_on = 0;
EXPECT_EQ(ctx.signal_count, NUM_THREADS + 1);
}
/* Test that modification propagates to all inherited events. */
TEST_F(sigtrap_threads, modify_and_enable_event)
{
struct perf_event_attr new_attr = make_event_attr(true, &ctx.iterate_on);
EXPECT_EQ(ioctl(self->fd, PERF_EVENT_IOC_MODIFY_ATTRIBUTES, &new_attr), 0);
run_test_threads(_metadata, self);
EXPECT_EQ(ctx.signal_count, NUM_THREADS);
EXPECT_EQ(ctx.tids_want_signal, 0);
EXPECT_EQ(ctx.first_siginfo.si_addr, &ctx.iterate_on);
EXPECT_EQ(ctx.first_siginfo.si_errno, PERF_TYPE_BREAKPOINT);
EXPECT_EQ(ctx.first_siginfo.si_perf, TEST_SIG_DATA(&ctx.iterate_on));
/* Check enabled for parent. */
ctx.iterate_on = 0;
EXPECT_EQ(ctx.signal_count, NUM_THREADS + 1);
}
/* Stress test event + signal handling. */
TEST_F(sigtrap_threads, signal_stress)
{
ctx.iterate_on = 3000;
EXPECT_EQ(ioctl(self->fd, PERF_EVENT_IOC_ENABLE, 0), 0);
run_test_threads(_metadata, self);
EXPECT_EQ(ioctl(self->fd, PERF_EVENT_IOC_DISABLE, 0), 0);
EXPECT_EQ(ctx.signal_count, NUM_THREADS * ctx.iterate_on);
EXPECT_EQ(ctx.tids_want_signal, 0);
EXPECT_EQ(ctx.first_siginfo.si_addr, &ctx.iterate_on);
EXPECT_EQ(ctx.first_siginfo.si_errno, PERF_TYPE_BREAKPOINT);
EXPECT_EQ(ctx.first_siginfo.si_perf, TEST_SIG_DATA(&ctx.iterate_on));
}
TEST_HARNESS_MAIN