Suren Baghdasaryan 2c321f3f70 mm: change inlined allocation helpers to account at the call site ... Main goal of memory allocation profiling patchset is to provide accounting that is cheap enough to run in production. To achieve that we inject counters using codetags at the allocation call sites to account every time allocation is made. This injection allows us to perform accounting efficiently because injected counters are immediately available as opposed to the alternative methods, such as using _RET_IP_, which would require counter lookup and appropriate locking that makes accounting much more expensive. This method requires all allocation functions to inject separate counters at their call sites so that their callers can be individually accounted. Counter injection is implemented by allocation hooks which should wrap all allocation functions. Inlined functions which perform allocations but do not use allocation hooks are directly charged for the allocations they perform. In most cases these functions are just specialized allocation wrappers used from multiple places to allocate objects of a specific type. It would be more useful to do the accounting at their call sites instead. Instrument these helpers to do accounting at the call site. Simple inlined allocation wrappers are converted directly into macros. More complex allocators or allocators with documentation are converted into _noprof versions and allocation hooks are added. This allows memory allocation profiling mechanism to charge allocations to the callers of these functions. Link: https://lkml.kernel.org/r/20240415020731.1152108-1-surenb@google.com Signed-off-by: Suren Baghdasaryan <surenb@google.com> Acked-by: Jan Kara <jack@suse.cz> [jbd2] Cc: Anna Schumaker <anna@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Benjamin Tissoires <benjamin.tissoires@redhat.com> Cc: Christoph Lameter <cl@linux.com> Cc: David Rientjes <rientjes@google.com> Cc: David S. Miller <davem@davemloft.net> Cc: Dennis Zhou <dennis@kernel.org> Cc: Eric Dumazet <edumazet@google.com> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: Jakub Kicinski <kuba@kernel.org> Cc: Jakub Sitnicki <jakub@cloudflare.com> Cc: Jiri Kosina <jikos@kernel.org> Cc: Joerg Roedel <joro@8bytes.org> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Kent Overstreet <kent.overstreet@linux.dev> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Paolo Abeni <pabeni@redhat.com> Cc: Pekka Enberg <penberg@kernel.org> Cc: Tejun Heo <tj@kernel.org> Cc: Theodore Ts'o <tytso@mit.edu> Cc: Trond Myklebust <trond.myklebust@hammerspace.com> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>		2024-04-25 20:55:59 -07:00
..
9p	9p: Add additional debug flags and open modes	2023-03-27 02:33:48 +00:00
bluetooth	Bluetooth: SCO: Fix not validating setsockopt user input	2024-04-10 15:03:46 -04:00
caif	net: caif: Remove unused declaration cfsrvl_ctrlcmd()	2023-08-10 18:24:48 -07:00
iucv	s390/iucv: use new address translation helpers	2024-03-13 09:23:48 +01:00
mana	net: mana: Fix Rx DMA datasize and skb_over_panic	2024-04-03 19:32:03 -07:00
netfilter	Networking changes for 6.9.	2024-03-12 17:44:08 -07:00
netns	net: Namespace-ify sysctl_optmem_max	2023-12-15 11:01:27 +00:00
nfc	nfc: core: make nfc_class constant	2024-03-05 11:21:18 -08:00
page_pool	net: page_pool: fix recycle stats for system page_pool allocator	2024-02-19 12:30:27 -08:00
phonet	net: ioctl: Use kernel memory on protocol ioctl callbacks	2023-06-15 22:33:26 -07:00
sctp	sctp: preserve const qualifier in sctp_sk()	2024-02-05 11:08:06 +00:00
tc_act	net/sched: Retire ipt action	2024-01-02 12:41:16 +00:00
6lowpan.h
act_api.h	net/sched: Add helper macros with module names	2024-02-02 10:57:55 -08:00
addrconf.h	ipv6: fix race condition between ipv6_get_ifaddr and ipv6_del_addr	2024-04-09 17:09:05 -07:00
af_ieee802154.h
af_rxrpc.h	rxrpc, afs: Allow afs to pin rxrpc_peer objects	2023-12-24 15:22:50 +00:00
af_unix.h	Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net	2024-02-01 15:12:37 -08:00
af_vsock.h	virtio/vsock: send credit update during setting SO_RCVLOWAT	2023-12-15 10:37:35 +00:00
ah.h
amt.h
arp.h	neighbour: switch to standard rcu, instead of rcu_bh	2023-03-21 21:32:18 -07:00
atmclip.h
ax25.h	x25: preserve const qualifier in [a]x25_sk()	2023-03-18 12:23:34 +00:00
ax88796.h
bareudp.h
bond_3ad.h	bonding: Add independent control state machine	2024-02-06 13:17:54 +01:00
bond_alb.h
bond_options.h	bonding: Add independent control state machine	2024-02-06 13:17:54 +01:00
bonding.h	bonding: Add independent control state machine	2024-02-06 13:17:54 +01:00
bpf_sk_storage.h
busy_poll.h	net: add napi_busy_loop_rcu()	2024-02-09 10:01:09 -08:00
calipso.h
cfg80211-wext.h	wifi: cfg80211: Avoid clashing function prototypes	2022-11-16 11:31:47 +02:00
cfg80211.h	wifi: cfg80211: add a flag to disable wireless extensions	2024-03-25 15:23:06 +01:00
cfg802154.h	mac802154: fix llsec key resources release in mac802154_llsec_key_del	2024-03-06 21:01:26 +01:00
checksum.h	net: checksum: drop the linux/uaccess.h include	2023-01-27 11:19:46 +00:00
cipso_ipv4.h
cls_cgroup.h
codel.h	codel: fix kernel-doc notation warnings	2023-07-14 20:39:29 -07:00
codel_impl.h
codel_qdisc.h
compat.h
datalink.h	net: datalink: Remove unused declarations	2023-07-27 17:17:32 -07:00
dcbevent.h
dcbnl.h	net: dcb: add helper functions to retrieve PCP and DSCP rewrite maps	2023-01-20 09:33:22 +00:00
devlink.h	devlink: convert most of devlink_fmsg_*() to return void	2023-10-20 11:34:51 +01:00
dropreason-core.h	tcp: introduce dropreasons in receive path	2024-02-28 10:39:21 +00:00
dropreason.h	net: openvswitch: add last-action drop reason	2023-08-14 08:01:06 +01:00
dsa.h	ethtool: replace struct ethtool_eee with a new struct ethtool_keee on kernel side	2024-01-31 12:30:47 +00:00
dsa_stubs.h	net: dsa: Use conduit and user terms	2023-10-24 13:08:14 -07:00
dsfield.h
dst.h	net: dst: Make dst_destroy() static and return void.	2024-02-06 11:45:53 +01:00
dst_cache.h
dst_metadata.h	xfrm: interface: Add unstable helpers for setting/getting XFRM metadata from TC-BPF	2022-12-05 21:58:27 -08:00
dst_ops.h	net: remove unnecessary input parameter 'how' in ifdown function	2023-08-22 13:19:02 +02:00
eee.h	net: add helpers for EEE configuration	2024-03-05 19:21:17 -08:00
erspan.h
esp.h
espintcp.h
ethoc.h
failover.h
fib_notifier.h
fib_rules.h	fib: remove unnecessary input parameters in fib_default_rule_add	2024-01-03 16:42:48 -08:00
firewire.h
flow.h	inet: shrink struct flowi_common	2023-11-02 09:31:02 +01:00
flow_dissector.h	net: flow_dissector: Add IPSEC dissector	2023-08-02 10:09:31 +01:00
flow_offload.h	flow_offload: Annotate struct flow_action_entry with __counted_by	2023-10-06 11:37:02 +01:00
fou.h	bpf,fou: Add bpf_skb_{set,get}_fou_encap kfuncs	2023-04-12 16:40:39 -07:00
fq.h	net: fq: Remove unused typedef fq_flow_get_default_t	2023-08-08 15:58:23 -07:00
fq_impl.h	wifi: mac80211: add support for restricting netdev features per vif	2022-12-01 15:09:10 +01:00
garp.h
gen_stats.h
genetlink.h	genetlink: make info in GENL_REQ_ATTR_CHECK() const	2024-02-23 18:17:36 -08:00
geneve.h
gre.h
gro.h	net: gro: move two declarations to include/net/gro.h	2024-03-11 14:13:14 -07:00
gro_cells.h
gso.h	net: move gso declarations and functions to their own files	2023-06-10 00:11:41 -07:00
gtp.h
gue.h
handshake.h	net/handshake: Add helpers for parsing incoming TLS Alerts	2023-07-28 14:07:59 -07:00
hotdata.h	net: move rps_sock_flow_table to net_hotdata	2024-03-07 21:12:43 -08:00
hwbm.h
icmp.h
ieee80211_radiotap.h	wifi: radiotap: add bandwidth definition of EHT U-SIG	2023-10-12 15:14:27 +03:00
ieee802154_netdev.h	mac802154: Handle association requests from peers	2023-11-20 11:43:03 +01:00
if_inet6.h	ipv6: anycast: complete RCU handling of struct ifacaddr6	2024-02-26 18:40:34 -08:00
ife.h
inet6_connection_sock.h
inet6_hashtables.h	net: Fix slab-out-of-bounds in inet[6]_steal_sock	2023-08-15 13:57:51 -07:00
inet_common.h	net: factor out __inet_listen_sk() helper	2023-08-14 07:06:13 +01:00
inet_connection_sock.h	tcp: properly terminate timers for kernel sockets	2024-03-25 19:51:57 -07:00
inet_dscp.h
inet_ecn.h
inet_frag.h	inet: frags: eliminate kernel-doc warning	2023-07-14 20:39:29 -07:00
inet_hashtables.h	tcp: Remove dead code and fields for bhash2.	2023-12-22 22:15:35 +00:00
inet_sock.h	inet: Add getsockopt support for IP_ROUTER_ALERT and IPV6_ROUTER_ALERT	2024-03-06 12:37:06 +00:00
inet_timewait_sock.h	tcp: Remove dead code and fields for bhash2.	2023-12-22 22:15:35 +00:00
inetpeer.h
ioam6.h	net: ioam6: multicast event	2024-02-28 11:19:41 +00:00
ip.h	ipmr: fix kernel panic when forwarding mcast packets	2024-01-26 21:05:26 -08:00
ip6_checksum.h
ip6_fib.h	net/ipv6: Remove expired routes with a separated list of routes.	2024-02-12 10:24:12 +00:00
ip6_route.h	ipv6: annotate data-races around cnf.mtu6	2024-03-01 08:42:31 +00:00
ip6_tunnel.h
ip_fib.h	inet: allow ip_valid_fib_dump_req() to be called with RTNL or RCU	2024-02-26 11:46:12 +00:00
ip_tunnels.h	geneve: fix header validation in geneve[6]_xmit_skb	2024-04-08 11:51:04 +01:00
ip_vs.h	ipvs: Correct spelling in comments	2023-04-22 01:39:41 +02:00
ipcomp.h
ipconfig.h
ipv6.h	ipv6: annotate data-races around cnf.forwarding	2024-03-01 08:42:31 +00:00
ipv6_frag.h
ipv6_stubs.h	ipsec-next-2023-10-28	2023-10-30 14:36:57 -07:00
iw_handler.h	wifi: wext: Remove unused declaration dev_get_wireless_info()	2023-08-22 21:40:40 +02:00
kcm.h	kcm: Send multiple frags in one sendmsg()	2023-06-12 21:13:23 -07:00
l3mdev.h
lag.h
lapb.h
lib80211.h
llc.h
llc_c_ac.h	net: llc: Remove unused function declarations	2023-08-04 15:33:17 -07:00
llc_c_ev.h	net: llc: Remove unused function declarations	2023-08-04 15:33:17 -07:00
llc_c_st.h
llc_conn.h	llc: Check netns in llc_estab_match() and llc_listener_match().	2023-07-20 10:46:28 +02:00
llc_if.h
llc_pdu.h	llc: Drop support for ETH_P_TR_802_2.	2024-01-19 21:30:09 -08:00
llc_s_ac.h
llc_s_ev.h
llc_s_st.h
llc_sap.h
lwtunnel.h	lwt: Check LWTUNNEL_XMIT_CONTINUE strictly	2023-08-18 16:05:26 +02:00
mac80211.h	wifi: mac80211: introduce a feature flag for quiet in CSA	2024-03-04 14:33:58 +01:00
mac802154.h
macsec.h	net: macsec: introduce mdo_insert_tx_tag	2023-12-27 13:08:10 +00:00
mctp.h	Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net	2024-02-29 14:24:56 -08:00
mctpdevice.h
mip6.h
mld.h
mpls.h
mpls_iptunnel.h
mptcp.h	mptcp: add struct mptcp_sched_ops	2023-08-22 17:31:18 -07:00
mrp.h	mrp: introduce active flags to prevent UAF when applicant uninit	2022-11-18 12:14:55 +00:00
ncsi.h
ndisc.h	ndisc: Remove unused ndisc_ifinfo_sysctl_strategy() declaration	2023-08-07 08:53:55 +01:00
neighbour.h	neighbour: Fix __randomize_layout crash in struct neighbour	2023-11-28 12:18:29 +01:00
net_debug.h
net_failover.h
net_namespace.h	net: add exit_batch_rtnl() method	2024-02-07 18:55:10 -08:00
net_ratelimit.h
net_trackers.h
netdev_queues.h	netdev: add queue stat for alloc failures	2024-03-07 21:13:26 -08:00
netdev_rx_queue.h	net: Add queue and napi association	2023-12-04 18:04:05 -08:00
netevent.h
netkit.h	bpf, netkit: Add indirect call wrapper for fetching peer dev	2023-11-20 10:15:16 -08:00
netlabel.h	mm: change inlined allocation helpers to account at the call site	2024-04-25 20:55:59 -07:00
netlink.h	mm: change inlined allocation helpers to account at the call site	2024-04-25 20:55:59 -07:00
netmem.h	net: introduce abstraction for network memory	2024-02-20 09:22:58 +01:00
netprio_cgroup.h
netrom.h
nexthop.h	net: nexthop: Have all NH notifiers carry NH ID	2024-03-11 14:14:07 -07:00
nl802154.h	ieee802154: Give the user the association list	2023-11-20 11:43:45 +01:00
nsh.h	net: NSH: fix kernel-doc notation warning	2023-07-14 20:39:29 -07:00
p8022.h	net: 802: Remove unused function declarations	2023-08-04 15:33:50 -07:00
pie.h	pie: fix kernel-doc notation warning	2023-07-14 20:39:30 -07:00
ping.h	net/ipv4: ping_group_range: allow GID from 2147483648 to 4294967294	2023-06-02 09:55:22 +01:00
pkt_cls.h	net/sched: Add helper macros with module names	2024-02-02 10:57:55 -08:00
pkt_sched.h	net/sched: Add helper macros with module names	2024-02-02 10:57:55 -08:00
pptp.h
protocol.h	ipv6: move tcp_ipv6_hash_secret and udp_ipv6_hash_secret to net_hotdata	2024-03-07 21:12:43 -08:00
psample.h
psnap.h
raw.h	Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net	2023-04-06 12:01:20 -07:00
rawv6.h	ipv6: raw: constify raw_v6_match() socket argument	2023-03-17 08:56:37 +00:00
red.h
regulatory.h	wifi: cfg80211: save power spectral density(psd) of regulatory rule	2023-09-18 09:44:05 +02:00
request_sock.h	mm: change inlined allocation helpers to account at the call site	2024-04-25 20:55:59 -07:00
rose.h
route.h	ipv4: Set the routing scope properly in ip_route_output_ports().	2024-02-12 17:33:05 -08:00
rpl.h	ipv6: rpl: Remove pskb(_may)?_pull() in ipv6_rpl_srh_rcv().	2023-06-19 11:32:58 -07:00
rps.h	net: move rps_sock_flow_table to net_hotdata	2024-03-07 21:12:43 -08:00
rsi_91x.h	rsi: remove kernel-doc comment marker	2023-07-14 20:39:30 -07:00
rtnetlink.h	rtnetlink: add RTNL_FLAG_DUMP_UNLOCKED flag	2024-02-26 11:46:12 +00:00
rtnh.h
sch_generic.h	net: bql: fix building with BQL disabled	2024-03-01 08:46:15 +00:00
scm.h	af_unix: Try to run GC async.	2024-01-26 20:34:25 -08:00
secure_seq.h
seg6.h
seg6_hmac.h
seg6_local.h
selftests.h
slhc_vj.h
smc.h	net/smc: manage system EID in SMC stack instead of ISM driver	2023-12-26 20:24:33 +00:00
snmp.h
sock.h	tcp: properly terminate timers for kernel sockets	2024-03-25 19:51:57 -07:00
sock_reuseport.h
Space.h	net: appletalk: remove cops support	2023-10-04 11:49:20 -07:00
stp.h
strparser.h
switchdev.h	net: bridge: switchdev: Skip MDB replays of deferred events on offload	2024-02-16 09:36:37 +00:00
tc_wrapper.h	Merge branch 'x86/bugs' into x86/core, to pick up pending changes before dependent patches	2024-02-14 10:49:37 +01:00
tcp.h	tcp: make the dropreason really work when calling tcp_rcv_state_process()	2024-02-28 10:39:22 +00:00
tcp_ao.h	Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net	2024-01-04 18:06:46 -08:00
tcp_states.h	tcp: Dump bound-only sockets in inet_diag.	2023-12-04 14:45:26 -08:00
tcx.h	mm: change inlined allocation helpers to account at the call site	2024-04-25 20:55:59 -07:00
timewait_sock.h
tipc.h
tls.h	tls: fix race between async notify and socket close	2024-02-10 21:38:19 +00:00
tls_prot.h	net/tls: Add TLS Alert definitions	2023-07-28 14:07:59 -07:00
tls_toe.h
transp_v6.h	inet6: Remove unused function declaration udpv6_connect()	2023-08-01 15:06:27 -07:00
tso.h	net: tso: inline tso_count_descs()	2022-12-12 15:04:39 -08:00
tun_proto.h
udp.h	udp/udplite: Remove unused function declarations udp{,lite}_get_port()	2023-08-07 08:53:55 +01:00
udp_tunnel.h	ipv6: add new arguments to udp_tunnel6_dst_lookup()	2023-10-23 08:48:57 +01:00
udplite.h	udplite: fix various data-races	2023-09-14 16:16:36 +02:00
vsock_addr.h
vxlan.h	vxlan: add support for flowlabel inherit	2023-11-16 22:33:31 +00:00
wext.h
x25.h	x25: preserve const qualifier in [a]x25_sk()	2023-03-18 12:23:34 +00:00
x25device.h
xdp.h	net, xdp: Correct grammar	2023-12-14 16:38:59 +01:00
xdp_priv.h
xdp_sock.h	xsk: Don't assume metadata is always requested in TX completion	2024-03-19 13:47:29 +01:00
xdp_sock_drv.h	xsk: fix usage of multi-buffer BPF helpers for ZC XDP	2024-01-24 16:24:06 -08:00
xfrm.h	xfrm: get global statistics from the offloaded device	2024-02-05 16:45:49 -08:00
xsk_buff_pool.h	xsk: add functions to fill control buffer	2023-12-13 16:16:40 -08:00