linux-stable/include/net
Jesper Dangaard Brouer be5dba25b4 net: page_pool: use alloc_pages_bulk in refill code path
There are cases where the page_pool need to refill with pages from the
page allocator.  Some workloads cause the page_pool to release pages
instead of recycling these pages.

For these workload it can improve performance to bulk alloc pages from the
page-allocator to refill the alloc cache.

For XDP-redirect workload with 100G mlx5 driver (that use page_pool)
redirecting xdp_frame packets into a veth, that does XDP_PASS to create an
SKB from the xdp_frame, which then cannot return the page to the
page_pool.

Performance results under GitHub xdp-project[1]:
 [1] https://github.com/xdp-project/xdp-project/blob/master/areas/mem/page_pool06_alloc_pages_bulk.org

Mel: The patch "net: page_pool: convert to use alloc_pages_bulk_array
variant" was squashed with this patch. From the test page, the array
variant was superior with one of the test results as follows.

	Kernel		XDP stats       CPU     pps           Delta
	Baseline	XDP-RX CPU      total   3,771,046       n/a
	List		XDP-RX CPU      total   3,940,242    +4.49%
	Array		XDP-RX CPU      total   4,249,224   +12.68%

Link: https://lkml.kernel.org/r/20210325114228.27719-10-mgorman@techsingularity.net
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Reviewed-by: Alexander Lobakin <alobakin@pm.me>
Cc: Alexander Duyck <alexander.duyck@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Chuck Lever <chuck.lever@oracle.com>
Cc: David Miller <davem@davemloft.net>
Cc: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2021-04-30 11:20:43 -07:00
..
9p 9p: apply review requests for fid refcounting 2020-11-19 17:21:34 +01:00
bluetooth Bluetooth: Allow Microsoft extension to indicate curve validation 2021-04-08 12:26:34 +02:00
caif net: caif: Remove unused caif SPI driver 2020-09-29 14:02:53 -07:00
iucv net/af_iucv: don't track individual TX skbs for TRANS_HIPER sockets 2021-01-28 20:36:21 -08:00
netfilter netfilter: nftables: add catch-all set element support 2021-04-27 18:06:29 +02:00
netns netfilter: remove all xt_table anchors from struct net 2021-04-26 03:20:47 +02:00
nfc net: nfc: nci: drop nci_uart_ops::recv_buf 2021-03-10 09:34:07 +01:00
phonet
sctp sctp: bring inet(6)_skb_parm back to sctp_input_cb 2020-11-05 14:27:30 -08:00
tc_act net/sched: act_police: add support for packet-per-second policing 2021-03-13 14:18:09 -08:00
6lowpan.h
act_api.h net: sched: fix err handler in tcf_action_init() 2021-04-08 13:47:33 -07:00
addrconf.h net: bridge: mcast: fix broken length + header check for MRDv6 Adv. 2021-04-27 14:02:06 -07:00
af_ieee802154.h
af_rxrpc.h afs: Don't truncate iter during data fetch 2021-04-23 10:17:26 +01:00
af_unix.h
af_vsock.h
ah.h
arp.h
atmclip.h
ax25.h
ax88796.h
bareudp.h bareudp: Reverted support to enable & disable rx metadata collection 2020-07-21 18:30:47 -07:00
bond_3ad.h
bond_alb.h
bond_options.h
bonding.h net/bonding: Declare TLS RX device offload support 2021-01-18 20:48:40 -08:00
bpf_sk_storage.h bpf: struct sock is declared twice in bpf_sk_storage header 2021-03-26 17:43:55 +01:00
busy_poll.h net, xdp, xsk: fix __sk_mark_napi_id_once napi_id error 2020-12-01 15:51:19 +01:00
calipso.h
cfg80211-wext.h
cfg80211.h nl80211/cfg80211: add a flag to negotiate for LMR feedback in NDP ranging 2021-04-19 12:02:51 +02:00
cfg802154.h
checksum.h saner calling conventions for csum_and_copy_..._user() 2020-08-20 15:45:15 -04:00
cipso_ipv4.h cipso: Remove unused inline functions 2020-07-15 07:45:24 -07:00
cls_cgroup.h
codel.h
codel_impl.h
codel_qdisc.h
compat.h compat: always include linux/compat.h from net/compat.h 2020-11-23 13:31:54 -08:00
datalink.h
dcbevent.h
dcbnl.h
devlink.h devlink: Extend SF port attributes to have external attribute 2021-04-24 00:58:53 -07:00
dn.h
dn_dev.h
dn_fib.h
dn_neigh.h
dn_nsp.h
dn_route.h
dsa.h net: dsa: free skb->cb usage in core driver 2021-04-27 14:10:15 -07:00
dsfield.h
dst.h net: Consolidate common blackhole dst ops 2021-03-10 12:24:18 -08:00
dst_cache.h
dst_metadata.h
dst_ops.h
erspan.h
esp.h
espintcp.h
ethoc.h
failover.h
fib_notifier.h
fib_rules.h fib: use indirect call wrappers in the most common fib_rules_ops 2020-07-28 17:42:31 -07:00
firewire.h
flow.h flow: remove spi key from flowi struct 2021-04-19 12:25:11 +02:00
flow_dissector.h flow_dissector: constify raw input data argument 2021-03-14 14:46:32 -07:00
flow_offload.h net: flow_offload: add FLOW_ACTION_PPPOE_PUSH 2021-03-24 12:48:39 -07:00
fou.h
fq.h net/fq_impl: do not maintain a backlog-sorted list of flows 2021-01-21 13:33:45 +01:00
fq_impl.h net/fq_impl: do not maintain a backlog-sorted list of flows 2021-01-21 13:33:45 +01:00
garp.h
gen_stats.h
genetlink.h mptcp: avoid lock_fast usage in accept path 2021-02-12 16:31:46 -08:00
geneve.h
gre.h ip_gre: add csum offload support for gre header 2021-01-29 20:39:14 -08:00
gro.h gro: add combined call_gro_receive() + INDIRECT_CALL_INET() helper 2021-03-18 19:51:12 -07:00
gro_cells.h
gtp.h
gue.h GUE: Fix a typo 2020-06-22 21:12:44 -07:00
hwbm.h
icmp.h net: icmp: pass zeroed opts from icmp{,v6}_ndo_send before sending 2021-02-23 11:29:52 -08:00
ieee80211_radiotap.h mac80211: add radiotap flag to assure frames are not reordered 2020-11-06 11:01:01 +01:00
ieee802154_netdev.h
if_inet6.h mld: add mc_lock for protecting per-interface mld data 2021-03-26 15:14:56 -07:00
ife.h
ila.h
inet6_connection_sock.h
inet6_hashtables.h
inet_common.h bpf: Allow rewriting to ports under ip_unprivileged_port_start 2021-01-27 18:18:15 -08:00
inet_connection_sock.h tcp: relookup sock for RST+ACK packets handled by obsolete req sock 2021-03-15 14:34:29 -07:00
inet_ecn.h inet_ecn: Use csum16_add() helper for IP_ECN_set_* helpers 2020-12-14 18:38:58 -08:00
inet_frag.h inet: frags: batch fqdir destroy works 2020-12-12 15:08:54 -08:00
inet_hashtables.h tcp: fix race condition when creating child sockets from syncookies 2020-11-23 16:32:33 -08:00
inet_sock.h inet: remove inet_sk_copy_descendant() 2020-08-26 07:33:19 -07:00
inet_timewait_sock.h
inetpeer.h
ip.h inet: constify inet_sdif() argument 2020-11-10 17:56:54 -08:00
ip6_checksum.h tcp: remove indirect calls for icsk->icsk_af_ops->send_check 2020-06-20 17:47:53 -07:00
ip6_fib.h IPv6: Add "offload failed" indication to routes 2021-02-08 16:47:03 -08:00
ip6_route.h net: allow user to set metric on default route learned via Router Advertisement 2021-01-26 18:39:45 -08:00
ip6_tunnel.h
ip_fib.h IPv4: Add "offload failed" indication to routes 2021-02-08 16:47:03 -08:00
ip_tunnels.h Merge https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-11-19 19:08:46 -08:00
ip_vs.h netfilter: move handlers to net/ip_vs.h 2021-02-04 18:37:57 -08:00
ipcomp.h
ipconfig.h
ipv6.h seg6: add support for IPv4 decapsulation in ipv6_srh_rcv() 2021-03-11 16:09:21 -08:00
ipv6_frag.h ipv6: Remove dependency of ipv6_frag_thdr_truncated on ipv6 module 2020-11-19 10:49:50 -08:00
ipv6_stubs.h ipv6: add ipv6_dev_find to stubs 2021-03-30 13:29:39 -07:00
ipx.h
iw_handler.h
kcm.h
l3mdev.h l3mdev: add infrastructure for table to VRF mapping 2020-06-20 17:22:22 -07:00
lag.h
lapb.h net: lapb: Make "lapb_t1timer_running" able to detect an already running timer 2021-03-23 14:14:50 -07:00
lib80211.h
llc.h
llc_c_ac.h
llc_c_ev.h
llc_c_st.h
llc_conn.h
llc_if.h
llc_pdu.h
llc_s_ac.h
llc_s_ev.h
llc_s_st.h
llc_sap.h
lwtunnel.h
mac80211.h mac80211: make ieee80211_vif_to_wdev work when the vif isn't in the driver 2021-04-19 12:03:13 +02:00
mac802154.h
macsec.h
mip6.h
mld.h mld: add new workqueues for process mld events 2021-03-26 15:14:56 -07:00
mpls.h net: Make mpls_entry_encode() available for generic users 2020-05-29 21:20:20 -07:00
mpls_iptunnel.h
mptcp.h mptcp: use mptcp_addr_info in mptcp_out_options 2021-04-07 14:09:39 -07:00
mrp.h
ncsi.h
ndisc.h ipv6: ndisc: adjust ndisc_ifinfo_sysctl_change prototype 2020-08-24 06:40:07 -07:00
neighbour.h net: Exempt multicast addresses from five-second neighbor lifetime 2020-11-13 14:24:39 -08:00
net_failover.h
net_namespace.h net: dccp: use net_generic storage 2021-04-09 16:34:56 -07:00
net_ratelimit.h
netevent.h
netlabel.h
netlink.h treewide: rename nla_strlcpy to nla_strscpy. 2020-11-16 08:08:54 -08:00
netprio_cgroup.h
netrom.h
nexthop.h nexthop: Rename artifacts related to legacy multipath nexthop groups 2021-03-28 17:53:39 -07:00
nl802154.h
nsh.h
p8022.h
page_pool.h net: page_pool: use alloc_pages_bulk in refill code path 2021-04-30 11:20:43 -07:00
pie.h
ping.h
pkt_cls.h sch_htb: Hierarchical QoS hardware offload 2021-01-22 20:41:29 -08:00
pkt_sched.h net: add a helper to avoid issues with HW TX timestamping and SO_TXTIME 2021-03-10 12:45:16 -08:00
pptp.h
protocol.h
psample.h psample: Add additional metadata attributes 2021-03-14 15:00:43 -07:00
psnap.h
raw.h
rawv6.h
red.h sch_red: fix off-by-one checks in red_check_params() 2021-03-25 17:40:43 -07:00
regulatory.h net/wireless: regulatory.h: drop duplicate word in comment 2020-07-31 09:24:23 +02:00
request_sock.h tcp: bpf: Optionally store mac header in TCP_SAVE_SYN 2020-08-24 14:35:00 -07:00
rose.h
route.h lsm,selinux: pass flowi_common instead of flowi to the LSM hooks 2020-11-23 18:36:21 -05:00
rpl.h net: ipv6: Use struct_size() helper and kcalloc() 2020-06-23 20:27:09 -07:00
rsi_91x.h
rtnetlink.h ipv6: report errors for iftoken via netlink extack 2021-04-08 13:52:36 -07:00
rtnh.h
sch_generic.h net/sched: act_police: add support for packet-per-second policing 2021-03-13 14:18:09 -08:00
scm.h fs: Move __scm_install_fd() to __receive_fd() 2020-07-13 11:03:44 -07:00
secure_seq.h
seg6.h seg6: fix seg6_validate_srh() to avoid slab-out-of-bounds 2020-06-04 15:39:32 -07:00
seg6_hmac.h
seg6_local.h
selftests.h net: selftest: fix build issue if INET is disabled 2021-04-28 14:06:45 -07:00
slhc_vj.h
smc.h net/smc: introduce CHID callback for ISM devices 2020-09-28 15:19:03 -07:00
snmp.h
sock.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next 2021-04-25 18:02:32 -07:00
sock_reuseport.h
Space.h
stp.h
strparser.h
switchdev.h net: bridge: switchdev: include local flag in FDB notifications 2021-04-16 15:15:45 -07:00
tcp.h skmsg: Pass psock pointer to ->psock_update_sk_prot() 2021-04-12 17:34:27 +02:00
tcp_states.h
timewait_sock.h
tipc.h
tls.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-11-27 18:25:27 -08:00
tls_toe.h
transp_v6.h tcp: move ipv4_specific to tcp include file 2020-06-23 20:10:15 -07:00
tso.h net: tso: cache transport header length 2020-06-18 20:46:23 -07:00
tun_proto.h
udp.h skmsg: Pass psock pointer to ->psock_update_sk_prot() 2021-04-12 17:34:27 +02:00
udp_tunnel.h udp: call udp_encap_enable for v6 sockets when enabling encap 2021-02-04 18:37:14 -08:00
udplite.h
vsock_addr.h
vxlan.h net: sched: only keep the available bits when setting vxlan md->gbp 2020-09-14 16:49:39 -07:00
wext.h
x25.h
x25device.h
xdp.h net, veth: Alloc skb in bulk for ndo_xdp_xmit 2021-02-04 01:00:07 +01:00
xdp_priv.h
xdp_sock.h bpf, xdp: Make bpf_redirect_map() a map operation 2021-03-10 01:06:34 +01:00
xdp_sock_drv.h xsk: Introduce batched Tx descriptor interfaces 2020-11-17 22:07:40 +01:00
xfrm.h xfrm: Fix NULL pointer dereference on policy lookup 2021-03-24 10:00:24 +01:00
xsk_buff_pool.h xsk: Fix race in SKB mode transmit with shared cq 2020-12-18 16:10:21 +01:00