page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0
|
|
|
|
*
|
|
|
|
* page_pool.h
|
|
|
|
* Author: Jesper Dangaard Brouer <netoptimizer@brouer.com>
|
|
|
|
* Copyright (C) 2016 Red Hat, Inc.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/**
|
|
|
|
* DOC: page_pool allocator
|
|
|
|
*
|
|
|
|
* This page_pool allocator is optimized for the XDP mode that
|
|
|
|
* uses one-frame-per-page, but have fallbacks that act like the
|
|
|
|
* regular page allocator APIs.
|
|
|
|
*
|
|
|
|
* Basic use involve replacing alloc_pages() calls with the
|
|
|
|
* page_pool_alloc_pages() call. Drivers should likely use
|
|
|
|
* page_pool_dev_alloc_pages() replacing dev_alloc_pages().
|
|
|
|
*
|
xdp: tracking page_pool resources and safe removal
This patch is needed before we can allow drivers to use page_pool for
DMA-mappings. Today with page_pool and XDP return API, it is possible to
remove the page_pool object (from rhashtable), while there are still
in-flight packet-pages. This is safely handled via RCU and failed lookups in
__xdp_return() fallback to call put_page(), when page_pool object is gone.
In-case page is still DMA mapped, this will result in page note getting
correctly DMA unmapped.
To solve this, the page_pool is extended with tracking in-flight pages. And
XDP disconnect system queries page_pool and waits, via workqueue, for all
in-flight pages to be returned.
To avoid killing performance when tracking in-flight pages, the implement
use two (unsigned) counters, that in placed on different cache-lines, and
can be used to deduct in-flight packets. This is done by mapping the
unsigned "sequence" counters onto signed Two's complement arithmetic
operations. This is e.g. used by kernel's time_after macros, described in
kernel commit 1ba3aab3033b and 5a581b367b5, and also explained in RFC1982.
The trick is these two incrementing counters only need to be read and
compared, when checking if it's safe to free the page_pool structure. Which
will only happen when driver have disconnected RX/alloc side. Thus, on a
non-fast-path.
It is chosen that page_pool tracking is also enabled for the non-DMA
use-case, as this can be used for statistics later.
After this patch, using page_pool requires more strict resource "release",
e.g. via page_pool_release_page() that was introduced in this patchset, and
previous patches implement/fix this more strict requirement.
Drivers no-longer call page_pool_destroy(). Drivers already call
xdp_rxq_info_unreg() which call xdp_rxq_info_unreg_mem_model(), which will
attempt to disconnect the mem id, and if attempt fails schedule the
disconnect for later via delayed workqueue.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-06-18 13:05:47 +00:00
|
|
|
* API keeps track of in-flight pages, in-order to let API user know
|
|
|
|
* when it is safe to dealloactor page_pool object. Thus, API users
|
|
|
|
* must make sure to call page_pool_release_page() when a page is
|
|
|
|
* "leaving" the page_pool. Or call page_pool_put_page() where
|
|
|
|
* appropiate. For maintaining correct accounting.
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
*
|
xdp: tracking page_pool resources and safe removal
This patch is needed before we can allow drivers to use page_pool for
DMA-mappings. Today with page_pool and XDP return API, it is possible to
remove the page_pool object (from rhashtable), while there are still
in-flight packet-pages. This is safely handled via RCU and failed lookups in
__xdp_return() fallback to call put_page(), when page_pool object is gone.
In-case page is still DMA mapped, this will result in page note getting
correctly DMA unmapped.
To solve this, the page_pool is extended with tracking in-flight pages. And
XDP disconnect system queries page_pool and waits, via workqueue, for all
in-flight pages to be returned.
To avoid killing performance when tracking in-flight pages, the implement
use two (unsigned) counters, that in placed on different cache-lines, and
can be used to deduct in-flight packets. This is done by mapping the
unsigned "sequence" counters onto signed Two's complement arithmetic
operations. This is e.g. used by kernel's time_after macros, described in
kernel commit 1ba3aab3033b and 5a581b367b5, and also explained in RFC1982.
The trick is these two incrementing counters only need to be read and
compared, when checking if it's safe to free the page_pool structure. Which
will only happen when driver have disconnected RX/alloc side. Thus, on a
non-fast-path.
It is chosen that page_pool tracking is also enabled for the non-DMA
use-case, as this can be used for statistics later.
After this patch, using page_pool requires more strict resource "release",
e.g. via page_pool_release_page() that was introduced in this patchset, and
previous patches implement/fix this more strict requirement.
Drivers no-longer call page_pool_destroy(). Drivers already call
xdp_rxq_info_unreg() which call xdp_rxq_info_unreg_mem_model(), which will
attempt to disconnect the mem id, and if attempt fails schedule the
disconnect for later via delayed workqueue.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-06-18 13:05:47 +00:00
|
|
|
* API user must only call page_pool_put_page() once on a page, as it
|
|
|
|
* will either recycle the page, or in case of elevated refcnt, it
|
|
|
|
* will release the DMA mapping and in-flight state accounting. We
|
|
|
|
* hope to lift this requirement in the future.
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
*/
|
|
|
|
#ifndef _NET_PAGE_POOL_H
|
|
|
|
#define _NET_PAGE_POOL_H
|
|
|
|
|
|
|
|
#include <linux/mm.h> /* Needed by ptr_ring */
|
|
|
|
#include <linux/ptr_ring.h>
|
|
|
|
#include <linux/dma-direction.h>
|
|
|
|
|
2019-11-20 14:54:18 +00:00
|
|
|
#define PP_FLAG_DMA_MAP BIT(0) /* Should page_pool do the DMA
|
|
|
|
* map/unmap
|
|
|
|
*/
|
|
|
|
#define PP_FLAG_DMA_SYNC_DEV BIT(1) /* If set all pages that the driver gets
|
|
|
|
* from page_pool will be
|
|
|
|
* DMA-synced-for-device according to
|
|
|
|
* the length provided by the device
|
|
|
|
* driver.
|
|
|
|
* Please note DMA-sync-for-CPU is still
|
|
|
|
* device driver responsibility
|
|
|
|
*/
|
2021-08-06 02:46:20 +00:00
|
|
|
#define PP_FLAG_PAGE_FRAG BIT(2) /* for page frag feature */
|
|
|
|
#define PP_FLAG_ALL (PP_FLAG_DMA_MAP |\
|
|
|
|
PP_FLAG_DMA_SYNC_DEV |\
|
|
|
|
PP_FLAG_PAGE_FRAG)
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Fast allocation side cache array/stack
|
|
|
|
*
|
|
|
|
* The cache size and refill watermark is related to the network
|
|
|
|
* use-case. The NAPI budget is 64 packets. After a NAPI poll the RX
|
|
|
|
* ring is usually refilled and the max consumed elements will be 64,
|
|
|
|
* thus a natural max size of objects needed in the cache.
|
|
|
|
*
|
|
|
|
* Keeping room for more objects, is due to XDP_DROP use-case. As
|
|
|
|
* XDP_DROP allows the opportunity to recycle objects directly into
|
|
|
|
* this array, as it shares the same softirq/NAPI protection. If
|
|
|
|
* cache is already full (or partly full) then the XDP_DROP recycles
|
|
|
|
* would have to take a slower code path.
|
|
|
|
*/
|
|
|
|
#define PP_ALLOC_CACHE_SIZE 128
|
|
|
|
#define PP_ALLOC_CACHE_REFILL 64
|
|
|
|
struct pp_alloc_cache {
|
|
|
|
u32 count;
|
2021-04-30 06:02:07 +00:00
|
|
|
struct page *cache[PP_ALLOC_CACHE_SIZE];
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
struct page_pool_params {
|
|
|
|
unsigned int flags;
|
|
|
|
unsigned int order;
|
|
|
|
unsigned int pool_size;
|
|
|
|
int nid; /* Numa node id to allocate from pages from */
|
|
|
|
struct device *dev; /* device, for DMA pre-mapping purposes */
|
|
|
|
enum dma_data_direction dma_dir; /* DMA mapping direction */
|
2019-11-20 14:54:18 +00:00
|
|
|
unsigned int max_len; /* max DMA sync memory size */
|
|
|
|
unsigned int offset; /* DMA addr offset */
|
2022-01-03 15:08:07 +00:00
|
|
|
void (*init_callback)(struct page *page, void *arg);
|
|
|
|
void *init_arg;
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
};
|
|
|
|
|
2022-03-02 07:55:47 +00:00
|
|
|
#ifdef CONFIG_PAGE_POOL_STATS
|
|
|
|
struct page_pool_alloc_stats {
|
|
|
|
u64 fast; /* fast path allocations */
|
|
|
|
u64 slow; /* slow-path order 0 allocations */
|
|
|
|
u64 slow_high_order; /* slow-path high order allocations */
|
|
|
|
u64 empty; /* failed refills due to empty ptr ring, forcing
|
|
|
|
* slow path allocation
|
|
|
|
*/
|
|
|
|
u64 refill; /* allocations via successful refill */
|
|
|
|
u64 waive; /* failed refills due to numa zone mismatch */
|
|
|
|
};
|
2022-03-02 07:55:48 +00:00
|
|
|
|
|
|
|
struct page_pool_recycle_stats {
|
|
|
|
u64 cached; /* recycling placed page in the cache. */
|
|
|
|
u64 cache_full; /* cache was full */
|
|
|
|
u64 ring; /* recycling placed page back into ptr ring */
|
|
|
|
u64 ring_full; /* page was released from page-pool because
|
|
|
|
* PTR ring was full.
|
|
|
|
*/
|
|
|
|
u64 released_refcnt; /* page released because of elevated
|
|
|
|
* refcnt
|
|
|
|
*/
|
|
|
|
};
|
2022-03-02 07:55:49 +00:00
|
|
|
|
|
|
|
/* This struct wraps the above stats structs so users of the
|
|
|
|
* page_pool_get_stats API can pass a single argument when requesting the
|
|
|
|
* stats for the page pool.
|
|
|
|
*/
|
|
|
|
struct page_pool_stats {
|
|
|
|
struct page_pool_alloc_stats alloc_stats;
|
|
|
|
struct page_pool_recycle_stats recycle_stats;
|
|
|
|
};
|
|
|
|
|
2022-04-12 16:31:58 +00:00
|
|
|
int page_pool_ethtool_stats_get_count(void);
|
|
|
|
u8 *page_pool_ethtool_stats_get_strings(u8 *data);
|
|
|
|
u64 *page_pool_ethtool_stats_get(u64 *data, void *stats);
|
|
|
|
|
2022-03-02 07:55:49 +00:00
|
|
|
/*
|
|
|
|
* Drivers that wish to harvest page pool stats and report them to users
|
|
|
|
* (perhaps via ethtool, debugfs, or another mechanism) can allocate a
|
|
|
|
* struct page_pool_stats call page_pool_get_stats to get stats for the specified pool.
|
|
|
|
*/
|
|
|
|
bool page_pool_get_stats(struct page_pool *pool,
|
|
|
|
struct page_pool_stats *stats);
|
2022-04-12 16:31:58 +00:00
|
|
|
#else
|
|
|
|
|
|
|
|
static inline int page_pool_ethtool_stats_get_count(void)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline u8 *page_pool_ethtool_stats_get_strings(u8 *data)
|
|
|
|
{
|
|
|
|
return data;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline u64 *page_pool_ethtool_stats_get(u64 *data, void *stats)
|
|
|
|
{
|
|
|
|
return data;
|
|
|
|
}
|
|
|
|
|
2022-03-02 07:55:47 +00:00
|
|
|
#endif
|
|
|
|
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
struct page_pool {
|
|
|
|
struct page_pool_params p;
|
|
|
|
|
2019-11-14 22:13:00 +00:00
|
|
|
struct delayed_work release_dw;
|
|
|
|
void (*disconnect)(void *);
|
|
|
|
unsigned long defer_start;
|
|
|
|
unsigned long defer_warn;
|
|
|
|
|
|
|
|
u32 pages_state_hold_cnt;
|
2021-08-06 02:46:21 +00:00
|
|
|
unsigned int frag_offset;
|
|
|
|
struct page *frag_page;
|
|
|
|
long frag_users;
|
2022-03-02 07:55:47 +00:00
|
|
|
|
|
|
|
#ifdef CONFIG_PAGE_POOL_STATS
|
|
|
|
/* these stats are incremented while in softirq context */
|
|
|
|
struct page_pool_alloc_stats alloc_stats;
|
|
|
|
#endif
|
2022-01-03 15:08:08 +00:00
|
|
|
u32 xdp_mem_id;
|
xdp: tracking page_pool resources and safe removal
This patch is needed before we can allow drivers to use page_pool for
DMA-mappings. Today with page_pool and XDP return API, it is possible to
remove the page_pool object (from rhashtable), while there are still
in-flight packet-pages. This is safely handled via RCU and failed lookups in
__xdp_return() fallback to call put_page(), when page_pool object is gone.
In-case page is still DMA mapped, this will result in page note getting
correctly DMA unmapped.
To solve this, the page_pool is extended with tracking in-flight pages. And
XDP disconnect system queries page_pool and waits, via workqueue, for all
in-flight pages to be returned.
To avoid killing performance when tracking in-flight pages, the implement
use two (unsigned) counters, that in placed on different cache-lines, and
can be used to deduct in-flight packets. This is done by mapping the
unsigned "sequence" counters onto signed Two's complement arithmetic
operations. This is e.g. used by kernel's time_after macros, described in
kernel commit 1ba3aab3033b and 5a581b367b5, and also explained in RFC1982.
The trick is these two incrementing counters only need to be read and
compared, when checking if it's safe to free the page_pool structure. Which
will only happen when driver have disconnected RX/alloc side. Thus, on a
non-fast-path.
It is chosen that page_pool tracking is also enabled for the non-DMA
use-case, as this can be used for statistics later.
After this patch, using page_pool requires more strict resource "release",
e.g. via page_pool_release_page() that was introduced in this patchset, and
previous patches implement/fix this more strict requirement.
Drivers no-longer call page_pool_destroy(). Drivers already call
xdp_rxq_info_unreg() which call xdp_rxq_info_unreg_mem_model(), which will
attempt to disconnect the mem id, and if attempt fails schedule the
disconnect for later via delayed workqueue.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-06-18 13:05:47 +00:00
|
|
|
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
/*
|
|
|
|
* Data structure for allocation side
|
|
|
|
*
|
|
|
|
* Drivers allocation side usually already perform some kind
|
|
|
|
* of resource protection. Piggyback on this protection, and
|
|
|
|
* require driver to protect allocation side.
|
|
|
|
*
|
|
|
|
* For NIC drivers this means, allocate a page_pool per
|
|
|
|
* RX-queue. As the RX-queue is already protected by
|
|
|
|
* Softirq/BH scheduling and napi_schedule. NAPI schedule
|
|
|
|
* guarantee that a single napi_struct will only be scheduled
|
|
|
|
* on a single CPU (see napi_schedule).
|
|
|
|
*/
|
|
|
|
struct pp_alloc_cache alloc ____cacheline_aligned_in_smp;
|
|
|
|
|
|
|
|
/* Data structure for storing recycled pages.
|
|
|
|
*
|
|
|
|
* Returning/freeing pages is more complicated synchronization
|
|
|
|
* wise, because free's can happen on remote CPUs, with no
|
|
|
|
* association with allocation resource.
|
|
|
|
*
|
|
|
|
* Use ptr_ring, as it separates consumer and producer
|
|
|
|
* effeciently, it a way that doesn't bounce cache-lines.
|
|
|
|
*
|
|
|
|
* TODO: Implement bulk return pages into this structure.
|
|
|
|
*/
|
|
|
|
struct ptr_ring ring;
|
xdp: tracking page_pool resources and safe removal
This patch is needed before we can allow drivers to use page_pool for
DMA-mappings. Today with page_pool and XDP return API, it is possible to
remove the page_pool object (from rhashtable), while there are still
in-flight packet-pages. This is safely handled via RCU and failed lookups in
__xdp_return() fallback to call put_page(), when page_pool object is gone.
In-case page is still DMA mapped, this will result in page note getting
correctly DMA unmapped.
To solve this, the page_pool is extended with tracking in-flight pages. And
XDP disconnect system queries page_pool and waits, via workqueue, for all
in-flight pages to be returned.
To avoid killing performance when tracking in-flight pages, the implement
use two (unsigned) counters, that in placed on different cache-lines, and
can be used to deduct in-flight packets. This is done by mapping the
unsigned "sequence" counters onto signed Two's complement arithmetic
operations. This is e.g. used by kernel's time_after macros, described in
kernel commit 1ba3aab3033b and 5a581b367b5, and also explained in RFC1982.
The trick is these two incrementing counters only need to be read and
compared, when checking if it's safe to free the page_pool structure. Which
will only happen when driver have disconnected RX/alloc side. Thus, on a
non-fast-path.
It is chosen that page_pool tracking is also enabled for the non-DMA
use-case, as this can be used for statistics later.
After this patch, using page_pool requires more strict resource "release",
e.g. via page_pool_release_page() that was introduced in this patchset, and
previous patches implement/fix this more strict requirement.
Drivers no-longer call page_pool_destroy(). Drivers already call
xdp_rxq_info_unreg() which call xdp_rxq_info_unreg_mem_model(), which will
attempt to disconnect the mem id, and if attempt fails schedule the
disconnect for later via delayed workqueue.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-06-18 13:05:47 +00:00
|
|
|
|
2022-03-02 07:55:48 +00:00
|
|
|
#ifdef CONFIG_PAGE_POOL_STATS
|
|
|
|
/* recycle stats are per-cpu to avoid locking */
|
|
|
|
struct page_pool_recycle_stats __percpu *recycle_stats;
|
|
|
|
#endif
|
xdp: tracking page_pool resources and safe removal
This patch is needed before we can allow drivers to use page_pool for
DMA-mappings. Today with page_pool and XDP return API, it is possible to
remove the page_pool object (from rhashtable), while there are still
in-flight packet-pages. This is safely handled via RCU and failed lookups in
__xdp_return() fallback to call put_page(), when page_pool object is gone.
In-case page is still DMA mapped, this will result in page note getting
correctly DMA unmapped.
To solve this, the page_pool is extended with tracking in-flight pages. And
XDP disconnect system queries page_pool and waits, via workqueue, for all
in-flight pages to be returned.
To avoid killing performance when tracking in-flight pages, the implement
use two (unsigned) counters, that in placed on different cache-lines, and
can be used to deduct in-flight packets. This is done by mapping the
unsigned "sequence" counters onto signed Two's complement arithmetic
operations. This is e.g. used by kernel's time_after macros, described in
kernel commit 1ba3aab3033b and 5a581b367b5, and also explained in RFC1982.
The trick is these two incrementing counters only need to be read and
compared, when checking if it's safe to free the page_pool structure. Which
will only happen when driver have disconnected RX/alloc side. Thus, on a
non-fast-path.
It is chosen that page_pool tracking is also enabled for the non-DMA
use-case, as this can be used for statistics later.
After this patch, using page_pool requires more strict resource "release",
e.g. via page_pool_release_page() that was introduced in this patchset, and
previous patches implement/fix this more strict requirement.
Drivers no-longer call page_pool_destroy(). Drivers already call
xdp_rxq_info_unreg() which call xdp_rxq_info_unreg_mem_model(), which will
attempt to disconnect the mem id, and if attempt fails schedule the
disconnect for later via delayed workqueue.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-06-18 13:05:47 +00:00
|
|
|
atomic_t pages_state_release_cnt;
|
2019-07-08 21:34:28 +00:00
|
|
|
|
|
|
|
/* A page_pool is strictly tied to a single RX-queue being
|
|
|
|
* protected by NAPI, due to above pp_alloc_cache. This
|
|
|
|
* refcnt serves purpose is to simplify drivers error handling.
|
|
|
|
*/
|
|
|
|
refcount_t user_cnt;
|
2019-11-16 11:22:43 +00:00
|
|
|
|
|
|
|
u64 destroy_cnt;
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp);
|
|
|
|
|
|
|
|
static inline struct page *page_pool_dev_alloc_pages(struct page_pool *pool)
|
|
|
|
{
|
|
|
|
gfp_t gfp = (GFP_ATOMIC | __GFP_NOWARN);
|
|
|
|
|
|
|
|
return page_pool_alloc_pages(pool, gfp);
|
|
|
|
}
|
|
|
|
|
2021-08-06 02:46:21 +00:00
|
|
|
struct page *page_pool_alloc_frag(struct page_pool *pool, unsigned int *offset,
|
|
|
|
unsigned int size, gfp_t gfp);
|
|
|
|
|
|
|
|
static inline struct page *page_pool_dev_alloc_frag(struct page_pool *pool,
|
|
|
|
unsigned int *offset,
|
|
|
|
unsigned int size)
|
|
|
|
{
|
|
|
|
gfp_t gfp = (GFP_ATOMIC | __GFP_NOWARN);
|
|
|
|
|
|
|
|
return page_pool_alloc_frag(pool, offset, size, gfp);
|
|
|
|
}
|
|
|
|
|
2019-06-29 05:23:24 +00:00
|
|
|
/* get the stored dma direction. A driver might decide to treat this locally and
|
|
|
|
* avoid the extra cache line from page_pool to determine the direction
|
|
|
|
*/
|
|
|
|
static
|
|
|
|
inline enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool)
|
|
|
|
{
|
|
|
|
return pool->p.dma_dir;
|
|
|
|
}
|
|
|
|
|
2021-06-07 19:02:38 +00:00
|
|
|
bool page_pool_return_skb_page(struct page *page);
|
|
|
|
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
struct page_pool *page_pool_create(const struct page_pool_params *params);
|
|
|
|
|
2022-01-03 15:08:08 +00:00
|
|
|
struct xdp_mem_info;
|
|
|
|
|
2019-06-18 13:05:37 +00:00
|
|
|
#ifdef CONFIG_PAGE_POOL
|
2019-11-14 22:13:00 +00:00
|
|
|
void page_pool_destroy(struct page_pool *pool);
|
2022-01-03 15:08:08 +00:00
|
|
|
void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *),
|
|
|
|
struct xdp_mem_info *mem);
|
2020-02-20 07:41:55 +00:00
|
|
|
void page_pool_release_page(struct page_pool *pool, struct page *page);
|
2020-11-13 11:48:29 +00:00
|
|
|
void page_pool_put_page_bulk(struct page_pool *pool, void **data,
|
|
|
|
int count);
|
2019-11-14 22:13:00 +00:00
|
|
|
#else
|
2019-07-08 21:34:28 +00:00
|
|
|
static inline void page_pool_destroy(struct page_pool *pool)
|
|
|
|
{
|
2019-11-14 22:13:00 +00:00
|
|
|
}
|
2019-07-08 21:34:28 +00:00
|
|
|
|
2019-11-14 22:13:00 +00:00
|
|
|
static inline void page_pool_use_xdp_mem(struct page_pool *pool,
|
2022-01-03 15:08:08 +00:00
|
|
|
void (*disconnect)(void *),
|
|
|
|
struct xdp_mem_info *mem)
|
2019-11-14 22:13:00 +00:00
|
|
|
{
|
2019-07-08 21:34:28 +00:00
|
|
|
}
|
2020-02-20 07:41:55 +00:00
|
|
|
static inline void page_pool_release_page(struct page_pool *pool,
|
|
|
|
struct page *page)
|
|
|
|
{
|
|
|
|
}
|
2020-11-13 11:48:29 +00:00
|
|
|
|
|
|
|
static inline void page_pool_put_page_bulk(struct page_pool *pool, void **data,
|
|
|
|
int count)
|
|
|
|
{
|
|
|
|
}
|
2019-11-14 22:13:00 +00:00
|
|
|
#endif
|
2019-07-08 21:34:28 +00:00
|
|
|
|
2022-01-31 16:40:01 +00:00
|
|
|
void page_pool_put_defragged_page(struct page_pool *pool, struct page *page,
|
|
|
|
unsigned int dma_sync_size,
|
|
|
|
bool allow_direct);
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
|
2023-02-17 22:21:30 +00:00
|
|
|
/* pp_frag_count represents the number of writers who can update the page
|
|
|
|
* either by updating skb->data or via DMA mappings for the device.
|
|
|
|
* We can't rely on the page refcnt for that as we don't know who might be
|
|
|
|
* holding page references and we can't reliably destroy or sync DMA mappings
|
|
|
|
* of the fragments.
|
|
|
|
*
|
|
|
|
* When pp_frag_count reaches 0 we can either recycle the page if the page
|
|
|
|
* refcnt is 1 or return it back to the memory allocator and destroy any
|
|
|
|
* mappings we have.
|
|
|
|
*/
|
2022-01-31 16:40:01 +00:00
|
|
|
static inline void page_pool_fragment_page(struct page *page, long nr)
|
|
|
|
{
|
|
|
|
atomic_long_set(&page->pp_frag_count, nr);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline long page_pool_defrag_page(struct page *page, long nr)
|
|
|
|
{
|
|
|
|
long ret;
|
|
|
|
|
|
|
|
/* If nr == pp_frag_count then we have cleared all remaining
|
|
|
|
* references to the page. No need to actually overwrite it, instead
|
|
|
|
* we can leave this to be overwritten by the calling function.
|
|
|
|
*
|
|
|
|
* The main advantage to doing this is that an atomic_read is
|
|
|
|
* generally a much cheaper operation than an atomic update,
|
|
|
|
* especially when dealing with a page that may be partitioned
|
|
|
|
* into only 2 or 3 pieces.
|
|
|
|
*/
|
|
|
|
if (atomic_long_read(&page->pp_frag_count) == nr)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
ret = atomic_long_sub_return(nr, &page->pp_frag_count);
|
|
|
|
WARN_ON(ret < 0);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline bool page_pool_is_last_frag(struct page_pool *pool,
|
|
|
|
struct page *page)
|
|
|
|
{
|
|
|
|
/* If fragments aren't enabled or count is 0 we were the last user */
|
|
|
|
return !(pool->p.flags & PP_FLAG_PAGE_FRAG) ||
|
|
|
|
(page_pool_defrag_page(page, 1) == 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void page_pool_put_page(struct page_pool *pool,
|
|
|
|
struct page *page,
|
|
|
|
unsigned int dma_sync_size,
|
|
|
|
bool allow_direct)
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
{
|
2018-04-17 14:46:22 +00:00
|
|
|
/* When page_pool isn't compiled-in, net/core/xdp.c doesn't
|
|
|
|
* allow registering MEM_TYPE_PAGE_POOL, but shield linker.
|
|
|
|
*/
|
|
|
|
#ifdef CONFIG_PAGE_POOL
|
2022-01-31 16:40:01 +00:00
|
|
|
if (!page_pool_is_last_frag(pool, page))
|
|
|
|
return;
|
|
|
|
|
|
|
|
page_pool_put_defragged_page(pool, page, dma_sync_size, allow_direct);
|
2018-04-17 14:46:22 +00:00
|
|
|
#endif
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
}
|
2020-02-20 07:41:55 +00:00
|
|
|
|
2022-01-31 16:40:01 +00:00
|
|
|
/* Same as above but will try to sync the entire area pool->max_len */
|
|
|
|
static inline void page_pool_put_full_page(struct page_pool *pool,
|
|
|
|
struct page *page, bool allow_direct)
|
|
|
|
{
|
|
|
|
page_pool_put_page(pool, page, -1, allow_direct);
|
|
|
|
}
|
|
|
|
|
2020-02-20 07:41:55 +00:00
|
|
|
/* Same as above but the caller must guarantee safe context. e.g NAPI */
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
static inline void page_pool_recycle_direct(struct page_pool *pool,
|
|
|
|
struct page *page)
|
|
|
|
{
|
2020-02-20 07:41:55 +00:00
|
|
|
page_pool_put_full_page(pool, page, true);
|
2019-06-18 13:05:27 +00:00
|
|
|
}
|
|
|
|
|
2021-11-17 07:56:52 +00:00
|
|
|
#define PAGE_POOL_DMA_USE_PP_FRAG_COUNT \
|
|
|
|
(sizeof(dma_addr_t) > sizeof(unsigned long))
|
|
|
|
|
2019-06-18 13:05:12 +00:00
|
|
|
static inline dma_addr_t page_pool_get_dma_addr(struct page *page)
|
|
|
|
{
|
2021-11-17 07:56:52 +00:00
|
|
|
dma_addr_t ret = page->dma_addr;
|
|
|
|
|
|
|
|
if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT)
|
|
|
|
ret |= (dma_addr_t)page->dma_addr_upper << 16 << 16;
|
|
|
|
|
|
|
|
return ret;
|
2021-05-15 00:27:24 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline void page_pool_set_dma_addr(struct page *page, dma_addr_t addr)
|
|
|
|
{
|
2021-08-06 02:46:20 +00:00
|
|
|
page->dma_addr = addr;
|
2021-11-17 07:56:52 +00:00
|
|
|
if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT)
|
|
|
|
page->dma_addr_upper = upper_32_bits(addr);
|
2021-08-06 02:46:20 +00:00
|
|
|
}
|
|
|
|
|
2018-04-17 14:46:22 +00:00
|
|
|
static inline bool is_page_pool_compiled_in(void)
|
|
|
|
{
|
|
|
|
#ifdef CONFIG_PAGE_POOL
|
|
|
|
return true;
|
|
|
|
#else
|
|
|
|
return false;
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2019-07-08 21:34:28 +00:00
|
|
|
static inline bool page_pool_put(struct page_pool *pool)
|
|
|
|
{
|
|
|
|
return refcount_dec_and_test(&pool->user_cnt);
|
|
|
|
}
|
|
|
|
|
2019-11-20 00:15:17 +00:00
|
|
|
/* Caller must provide appropriate safe context, e.g. NAPI. */
|
|
|
|
void page_pool_update_nid(struct page_pool *pool, int new_nid);
|
|
|
|
static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid)
|
|
|
|
{
|
|
|
|
if (unlikely(pool->p.nid != new_nid))
|
|
|
|
page_pool_update_nid(pool, new_nid);
|
|
|
|
}
|
2020-11-13 11:48:29 +00:00
|
|
|
|
|
|
|
static inline void page_pool_ring_lock(struct page_pool *pool)
|
|
|
|
__acquires(&pool->ring.producer_lock)
|
|
|
|
{
|
2023-02-03 01:16:11 +00:00
|
|
|
if (in_softirq())
|
2020-11-13 11:48:29 +00:00
|
|
|
spin_lock(&pool->ring.producer_lock);
|
|
|
|
else
|
|
|
|
spin_lock_bh(&pool->ring.producer_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void page_pool_ring_unlock(struct page_pool *pool)
|
|
|
|
__releases(&pool->ring.producer_lock)
|
|
|
|
{
|
2023-02-03 01:16:11 +00:00
|
|
|
if (in_softirq())
|
2020-11-13 11:48:29 +00:00
|
|
|
spin_unlock(&pool->ring.producer_lock);
|
|
|
|
else
|
|
|
|
spin_unlock_bh(&pool->ring.producer_lock);
|
|
|
|
}
|
|
|
|
|
page_pool: refurbish version of page_pool code
Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
pages on DMA-TX completion time, which have good cross CPU
performance, given DMA-TX completion time can happen on a remote CPU.
Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
Adapted page_pool code to not depend the page allocator and
integration into struct page. The DMA mapping feature is kept,
even-though it will not be activated/used in this patchset.
[1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes, don't return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
V4: many small improvements and cleanups
- Add DOC comment section, that can be used by kernel-doc
- Improve fallback mode, to work better with refcnt based recycling
e.g. remove a WARN as pointed out by Tariq
e.g. quicker fallback if ptr_ring is empty.
V5: Fixed SPDX license as pointed out by Alexei
V6: Adjustments requested by Eric Dumazet
- Adjust ____cacheline_aligned_in_smp usage/placement
- Move rcu_head in struct page_pool
- Free pages quicker on destroy, minimize resources delayed an RCU period
- Remove code for forward/backward compat ABI interface
V8: Issues found by kbuild test robot
- Address sparse should be static warnings
- Only compile+link when a driver use/select page_pool,
mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-17 14:46:17 +00:00
|
|
|
#endif /* _NET_PAGE_POOL_H */
|