2019-06-04 08:11:33 +00:00
|
|
|
// SPDX-License-Identifier: GPL-2.0-only
|
2008-04-27 11:55:59 +00:00
|
|
|
/****************************************************************************
|
2013-08-29 22:32:48 +00:00
|
|
|
* Driver for Solarflare network controllers and boards
|
2008-04-27 11:55:59 +00:00
|
|
|
* Copyright 2005-2006 Fen Systems Ltd.
|
2013-08-29 22:32:48 +00:00
|
|
|
* Copyright 2005-2013 Solarflare Communications Inc.
|
2008-04-27 11:55:59 +00:00
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include <linux/socket.h>
|
|
|
|
|
#include <linux/in.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 08:04:11 +00:00
|
|
|
#include <linux/slab.h>
|
2008-04-27 11:55:59 +00:00
|
|
|
#include <linux/ip.h>
|
2013-09-03 16:22:23 +00:00
|
|
|
#include <linux/ipv6.h>
|
2008-04-27 11:55:59 +00:00
|
|
|
#include <linux/tcp.h>
|
|
|
|
|
#include <linux/udp.h>
|
2011-05-22 20:47:17 +00:00
|
|
|
#include <linux/prefetch.h>
|
2011-09-15 23:46:05 +00:00
|
|
|
#include <linux/moduleparam.h>
|
sfc: reuse pages to avoid DMA mapping/unmapping costs
On POWER systems, DMA mapping/unmapping operations are very expensive.
These changes reduce these costs by trying to reuse DMA mapped pages.
After all the buffers associated with a page have been processed and
passed up, the page is placed into a ring (if there is room). For
each page that is required for a refill operation, a page in the ring
is examined to determine if its page count has fallen to 1, ie. the
kernel has released its reference to these packets. If this is the
case, the page can be immediately added back into the RX descriptor
ring, without having to re-map it for DMA.
If the kernel is still holding a reference to this page, it is removed
from the ring and unmapped for DMA. Then a new page, which can
immediately be used by RX buffers in the descriptor ring, is allocated
and DMA mapped.
The time a page needs to spend in the recycle ring before the kernel
has released its page references is based on the number of buffers
that use this page. As large pages can hold more RX buffers, the RX
recycle ring can be shorter. This reduces memory usage on POWER
systems, while maintaining the performance gain achieved by recycling
pages, following the driver change to pack more than two RX buffers
into large pages.
When an IOMMU is not present, the recycle ring can be small to reduce
memory usage, since DMA mapping operations are inexpensive.
With a small recycle ring, attempting to refill the descriptor queue
with more buffers than the equivalent size of the recycle ring could
ultimately lead to memory leaks if page entries in the recycle ring
were overwritten. To prevent this, the check to see if the recycle
ring is full is changed to check if the next entry to be written is
NULL.
[bwh: Combine and rebase several commits so this is complete
before the following buffer-packing changes. Remove module
parameter.]
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
2013-02-13 10:54:41 +00:00
|
|
|
#include <linux/iommu.h>
|
2008-04-27 11:55:59 +00:00
|
|
|
#include <net/ip.h>
|
|
|
|
|
#include <net/checksum.h>
|
2019-10-31 10:23:23 +00:00
|
|
|
#include <net/xdp.h>
|
|
|
|
|
#include <linux/bpf_trace.h>
|
2008-04-27 11:55:59 +00:00
|
|
|
#include "net_driver.h"
|
|
|
|
|
#include "efx.h"
|
2020-01-08 16:10:32 +00:00
|
|
|
#include "rx_common.h"
|
2012-11-08 01:46:53 +00:00
|
|
|
#include "filter.h"
|
2009-11-29 15:12:08 +00:00
|
|
|
#include "nic.h"
|
2008-05-07 12:36:19 +00:00
|
|
|
#include "selftest.h"
|
2008-04-27 11:55:59 +00:00
|
|
|
#include "workarounds.h"
|
|
|
|
|
|
2013-02-13 10:54:41 +00:00
|
|
|
/* Preferred number of descriptors to fill at once */
|
|
|
|
|
#define EFX_RX_PREFERRED_BATCH 8U
|
2008-04-27 11:55:59 +00:00
|
|
|
|
2019-10-31 10:23:23 +00:00
|
|
|
/* Maximum rx prefix used by any architecture. */
|
|
|
|
|
#define EFX_MAX_RX_PREFIX_SIZE 16
|
|
|
|
|
|
2008-04-27 11:55:59 +00:00
|
|
|
/* Size of buffer allocated for skb header area. */
|
2013-04-08 11:55:58 +00:00
|
|
|
#define EFX_SKB_HEADERS 128u
|
2008-04-27 11:55:59 +00:00
|
|
|
|
2013-01-29 23:33:15 +00:00
|
|
|
/* Each packet can consume up to ceil(max_frame_len / buffer_size) buffers */
|
|
|
|
|
#define EFX_RX_MAX_FRAGS DIV_ROUND_UP(EFX_MAX_FRAME_LEN(EFX_MAX_MTU), \
|
|
|
|
|
EFX_RX_USR_BUF_SIZE)
|
|
|
|
|
|
sfc: reuse pages to avoid DMA mapping/unmapping costs
On POWER systems, DMA mapping/unmapping operations are very expensive.
These changes reduce these costs by trying to reuse DMA mapped pages.
After all the buffers associated with a page have been processed and
passed up, the page is placed into a ring (if there is room). For
each page that is required for a refill operation, a page in the ring
is examined to determine if its page count has fallen to 1, ie. the
kernel has released its reference to these packets. If this is the
case, the page can be immediately added back into the RX descriptor
ring, without having to re-map it for DMA.
If the kernel is still holding a reference to this page, it is removed
from the ring and unmapped for DMA. Then a new page, which can
immediately be used by RX buffers in the descriptor ring, is allocated
and DMA mapped.
The time a page needs to spend in the recycle ring before the kernel
has released its page references is based on the number of buffers
that use this page. As large pages can hold more RX buffers, the RX
recycle ring can be shorter. This reduces memory usage on POWER
systems, while maintaining the performance gain achieved by recycling
pages, following the driver change to pack more than two RX buffers
into large pages.
When an IOMMU is not present, the recycle ring can be small to reduce
memory usage, since DMA mapping operations are inexpensive.
With a small recycle ring, attempting to refill the descriptor queue
with more buffers than the equivalent size of the recycle ring could
ultimately lead to memory leaks if page entries in the recycle ring
were overwritten. To prevent this, the check to see if the recycle
ring is full is changed to check if the next entry to be written is
NULL.
[bwh: Combine and rebase several commits so this is complete
before the following buffer-packing changes. Remove module
parameter.]
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
2013-02-13 10:54:41 +00:00
|
|
|
static inline void efx_sync_rx_buffer(struct efx_nic *efx,
|
|
|
|
|
struct efx_rx_buffer *rx_buf,
|
|
|
|
|
unsigned int len)
|
|
|
|
|
{
|
|
|
|
|
dma_sync_single_for_cpu(&efx->pci_dev->dev, rx_buf->dma_addr, len,
|
|
|
|
|
DMA_FROM_DEVICE);
|
|
|
|
|
}
|
|
|
|
|
|
2008-09-01 11:47:12 +00:00
|
|
|
static void efx_rx_packet__check_len(struct efx_rx_queue *rx_queue,
|
|
|
|
|
struct efx_rx_buffer *rx_buf,
|
2013-01-11 12:26:21 +00:00
|
|
|
int len)
|
2008-04-27 11:55:59 +00:00
|
|
|
{
|
|
|
|
|
struct efx_nic *efx = rx_queue->efx;
|
|
|
|
|
unsigned max_len = rx_buf->len - efx->type->rx_buffer_padding;
|
|
|
|
|
|
|
|
|
|
if (likely(len <= max_len))
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
/* The packet must be discarded, but this is only a fatal error
|
|
|
|
|
* if the caller indicated it was
|
|
|
|
|
*/
|
2011-08-26 17:05:11 +00:00
|
|
|
rx_buf->flags |= EFX_RX_PKT_DISCARD;
|
2008-04-27 11:55:59 +00:00
|
|
|
|
sfc: separate out SFC4000 ("Falcon") support into new sfc-falcon driver
Rationale: The differences between Falcon and Siena are in many ways larger
than those between Siena and EF10 (despite Siena being nominally "Falcon-
architecture"); for instance, Falcon has no MCPU, so there is no MCDI.
Removing Falcon support from the sfc driver should simplify the latter,
and avoid the possibility of Falcon support being broken by changes to sfc
(which are rarely if ever tested on Falcon, it being end-of-lifed hardware).
The sfc-falcon driver created in this changeset is essentially a copy of the
sfc driver, but with Siena- and EF10-specific code, including MCDI, removed
and with the "efx_" identifier prefix changed to "ef4_" (for "EFX 4000-
series") to avoid collisions when both drivers are built-in.
This changeset removes Falcon from the sfc driver's PCI ID table; then in
sfc I've removed obvious Falcon-related code: I removed the Falcon NIC
functions, Falcon PHY code, and EFX_REV_FALCON_*, then fixed up everything
that referenced them.
Also, increment minor version of both drivers (to 4.1).
For now, CONFIG_SFC selects CONFIG_SFC_FALCON, so that updating old configs
doesn't cause Falcon support to disappear; but that should be undone at
some point in the future.
Signed-off-by: Edward Cree <ecree@solarflare.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-28 18:55:34 +00:00
|
|
|
if (net_ratelimit())
|
|
|
|
|
netif_err(efx, rx_err, efx->net_dev,
|
|
|
|
|
"RX queue %d overlength RX event (%#x > %#x)\n",
|
|
|
|
|
efx_rx_queue_index(rx_queue), len, max_len);
|
2008-04-27 11:55:59 +00:00
|
|
|
|
2010-09-10 06:41:36 +00:00
|
|
|
efx_rx_queue_channel(rx_queue)->n_rx_overlength++;
|
2008-04-27 11:55:59 +00:00
|
|
|
}
|
|
|
|
|
|
2013-01-29 23:33:15 +00:00
|
|
|
/* Allocate and construct an SKB around page fragments */
|
2013-01-11 12:26:21 +00:00
|
|
|
static struct sk_buff *efx_rx_mk_skb(struct efx_channel *channel,
|
|
|
|
|
struct efx_rx_buffer *rx_buf,
|
2013-01-29 23:33:15 +00:00
|
|
|
unsigned int n_frags,
|
2013-01-11 12:26:21 +00:00
|
|
|
u8 *eh, int hdr_len)
|
|
|
|
|
{
|
|
|
|
|
struct efx_nic *efx = channel->efx;
|
|
|
|
|
struct sk_buff *skb;
|
2009-10-29 07:21:24 +00:00
|
|
|
|
2013-01-11 12:26:21 +00:00
|
|
|
/* Allocate an SKB to store the headers */
|
2013-11-28 18:58:11 +00:00
|
|
|
skb = netdev_alloc_skb(efx->net_dev,
|
|
|
|
|
efx->rx_ip_align + efx->rx_prefix_size +
|
|
|
|
|
hdr_len);
|
2014-07-15 10:58:12 +00:00
|
|
|
if (unlikely(skb == NULL)) {
|
|
|
|
|
atomic_inc(&efx->n_rx_noskb_drops);
|
2013-01-11 12:26:21 +00:00
|
|
|
return NULL;
|
2014-07-15 10:58:12 +00:00
|
|
|
}
|
2013-01-11 12:26:21 +00:00
|
|
|
|
2016-12-02 15:51:33 +00:00
|
|
|
EFX_WARN_ON_ONCE_PARANOID(rx_buf->len < hdr_len);
|
2013-01-11 12:26:21 +00:00
|
|
|
|
2013-11-28 18:58:11 +00:00
|
|
|
memcpy(skb->data + efx->rx_ip_align, eh - efx->rx_prefix_size,
|
|
|
|
|
efx->rx_prefix_size + hdr_len);
|
|
|
|
|
skb_reserve(skb, efx->rx_ip_align + efx->rx_prefix_size);
|
|
|
|
|
__skb_put(skb, hdr_len);
|
2013-01-11 12:26:21 +00:00
|
|
|
|
2013-01-29 23:33:15 +00:00
|
|
|
/* Append the remaining page(s) onto the frag list */
|
2013-01-11 12:26:21 +00:00
|
|
|
if (rx_buf->len > hdr_len) {
|
2013-01-29 23:33:15 +00:00
|
|
|
rx_buf->page_offset += hdr_len;
|
|
|
|
|
rx_buf->len -= hdr_len;
|
|
|
|
|
|
|
|
|
|
for (;;) {
|
|
|
|
|
skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags,
|
|
|
|
|
rx_buf->page, rx_buf->page_offset,
|
|
|
|
|
rx_buf->len);
|
|
|
|
|
rx_buf->page = NULL;
|
|
|
|
|
skb->len += rx_buf->len;
|
|
|
|
|
skb->data_len += rx_buf->len;
|
|
|
|
|
if (skb_shinfo(skb)->nr_frags == n_frags)
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
rx_buf = efx_rx_buf_next(&channel->rx_queue, rx_buf);
|
|
|
|
|
}
|
2013-01-11 12:26:21 +00:00
|
|
|
} else {
|
|
|
|
|
__free_pages(rx_buf->page, efx->rx_buffer_order);
|
2013-01-29 23:33:15 +00:00
|
|
|
rx_buf->page = NULL;
|
|
|
|
|
n_frags = 0;
|
2009-10-29 07:21:24 +00:00
|
|
|
}
|
2013-01-11 12:26:21 +00:00
|
|
|
|
2013-01-29 23:33:15 +00:00
|
|
|
skb->truesize += n_frags * efx->rx_buffer_truesize;
|
2013-01-11 12:26:21 +00:00
|
|
|
|
|
|
|
|
/* Move past the ethernet header */
|
|
|
|
|
skb->protocol = eth_type_trans(skb, efx->net_dev);
|
|
|
|
|
|
2014-07-22 13:03:25 +00:00
|
|
|
skb_mark_napi_id(skb, &channel->napi_str);
|
|
|
|
|
|
2013-01-11 12:26:21 +00:00
|
|
|
return skb;
|
2008-04-27 11:55:59 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void efx_rx_packet(struct efx_rx_queue *rx_queue, unsigned int index,
|
2013-01-29 23:33:15 +00:00
|
|
|
unsigned int n_frags, unsigned int len, u16 flags)
|
2008-04-27 11:55:59 +00:00
|
|
|
{
|
|
|
|
|
struct efx_nic *efx = rx_queue->efx;
|
2010-09-10 06:41:36 +00:00
|
|
|
struct efx_channel *channel = efx_rx_queue_channel(rx_queue);
|
2008-04-27 11:55:59 +00:00
|
|
|
struct efx_rx_buffer *rx_buf;
|
|
|
|
|
|
2014-07-17 11:10:43 +00:00
|
|
|
rx_queue->rx_packets++;
|
|
|
|
|
|
2008-04-27 11:55:59 +00:00
|
|
|
rx_buf = efx_rx_buffer(rx_queue, index);
|
2013-03-07 16:31:17 +00:00
|
|
|
rx_buf->flags |= flags;
|
2008-04-27 11:55:59 +00:00
|
|
|
|
2013-01-29 23:33:15 +00:00
|
|
|
/* Validate the number of fragments and completed length */
|
|
|
|
|
if (n_frags == 1) {
|
2013-04-27 00:55:18 +00:00
|
|
|
if (!(flags & EFX_RX_PKT_PREFIX_LEN))
|
|
|
|
|
efx_rx_packet__check_len(rx_queue, rx_buf, len);
|
2013-01-29 23:33:15 +00:00
|
|
|
} else if (unlikely(n_frags > EFX_RX_MAX_FRAGS) ||
|
2013-03-08 10:18:28 +00:00
|
|
|
unlikely(len <= (n_frags - 1) * efx->rx_dma_len) ||
|
|
|
|
|
unlikely(len > n_frags * efx->rx_dma_len) ||
|
2013-01-29 23:33:15 +00:00
|
|
|
unlikely(!efx->rx_scatter)) {
|
|
|
|
|
/* If this isn't an explicit discard request, either
|
|
|
|
|
* the hardware or the driver is broken.
|
|
|
|
|
*/
|
|
|
|
|
WARN_ON(!(len == 0 && rx_buf->flags & EFX_RX_PKT_DISCARD));
|
|
|
|
|
rx_buf->flags |= EFX_RX_PKT_DISCARD;
|
|
|
|
|
}
|
2008-04-27 11:55:59 +00:00
|
|
|
|
2010-06-23 11:30:07 +00:00
|
|
|
netif_vdbg(efx, rx_status, efx->net_dev,
|
2013-01-29 23:33:15 +00:00
|
|
|
"RX queue %d received ids %x-%x len %d %s%s\n",
|
2010-09-10 06:41:36 +00:00
|
|
|
efx_rx_queue_index(rx_queue), index,
|
2013-01-29 23:33:15 +00:00
|
|
|
(index + n_frags - 1) & rx_queue->ptr_mask, len,
|
2011-08-26 17:05:11 +00:00
|
|
|
(rx_buf->flags & EFX_RX_PKT_CSUMMED) ? " [SUMMED]" : "",
|
|
|
|
|
(rx_buf->flags & EFX_RX_PKT_DISCARD) ? " [DISCARD]" : "");
|
2008-04-27 11:55:59 +00:00
|
|
|
|
2013-01-29 23:33:15 +00:00
|
|
|
/* Discard packet, if instructed to do so. Process the
|
|
|
|
|
* previous receive first.
|
|
|
|
|
*/
|
2011-08-26 17:05:11 +00:00
|
|
|
if (unlikely(rx_buf->flags & EFX_RX_PKT_DISCARD)) {
|
2013-01-29 23:33:15 +00:00
|
|
|
efx_rx_flush_packet(channel);
|
2013-07-04 22:48:46 +00:00
|
|
|
efx_discard_rx_packet(channel, rx_buf, n_frags);
|
2013-01-29 23:33:15 +00:00
|
|
|
return;
|
2008-04-27 11:55:59 +00:00
|
|
|
}
|
|
|
|
|
|
2013-04-27 00:55:18 +00:00
|
|
|
if (n_frags == 1 && !(flags & EFX_RX_PKT_PREFIX_LEN))
|
2013-01-29 23:33:15 +00:00
|
|
|
rx_buf->len = len;
|
|
|
|
|
|
sfc: reuse pages to avoid DMA mapping/unmapping costs
On POWER systems, DMA mapping/unmapping operations are very expensive.
These changes reduce these costs by trying to reuse DMA mapped pages.
After all the buffers associated with a page have been processed and
passed up, the page is placed into a ring (if there is room). For
each page that is required for a refill operation, a page in the ring
is examined to determine if its page count has fallen to 1, ie. the
kernel has released its reference to these packets. If this is the
case, the page can be immediately added back into the RX descriptor
ring, without having to re-map it for DMA.
If the kernel is still holding a reference to this page, it is removed
from the ring and unmapped for DMA. Then a new page, which can
immediately be used by RX buffers in the descriptor ring, is allocated
and DMA mapped.
The time a page needs to spend in the recycle ring before the kernel
has released its page references is based on the number of buffers
that use this page. As large pages can hold more RX buffers, the RX
recycle ring can be shorter. This reduces memory usage on POWER
systems, while maintaining the performance gain achieved by recycling
pages, following the driver change to pack more than two RX buffers
into large pages.
When an IOMMU is not present, the recycle ring can be small to reduce
memory usage, since DMA mapping operations are inexpensive.
With a small recycle ring, attempting to refill the descriptor queue
with more buffers than the equivalent size of the recycle ring could
ultimately lead to memory leaks if page entries in the recycle ring
were overwritten. To prevent this, the check to see if the recycle
ring is full is changed to check if the next entry to be written is
NULL.
[bwh: Combine and rebase several commits so this is complete
before the following buffer-packing changes. Remove module
parameter.]
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
2013-02-13 10:54:41 +00:00
|
|
|
/* Release and/or sync the DMA mapping - assumes all RX buffers
|
|
|
|
|
* consumed in-order per RX queue.
|
2008-04-27 11:55:59 +00:00
|
|
|
*/
|
sfc: reuse pages to avoid DMA mapping/unmapping costs
On POWER systems, DMA mapping/unmapping operations are very expensive.
These changes reduce these costs by trying to reuse DMA mapped pages.
After all the buffers associated with a page have been processed and
passed up, the page is placed into a ring (if there is room). For
each page that is required for a refill operation, a page in the ring
is examined to determine if its page count has fallen to 1, ie. the
kernel has released its reference to these packets. If this is the
case, the page can be immediately added back into the RX descriptor
ring, without having to re-map it for DMA.
If the kernel is still holding a reference to this page, it is removed
from the ring and unmapped for DMA. Then a new page, which can
immediately be used by RX buffers in the descriptor ring, is allocated
and DMA mapped.
The time a page needs to spend in the recycle ring before the kernel
has released its page references is based on the number of buffers
that use this page. As large pages can hold more RX buffers, the RX
recycle ring can be shorter. This reduces memory usage on POWER
systems, while maintaining the performance gain achieved by recycling
pages, following the driver change to pack more than two RX buffers
into large pages.
When an IOMMU is not present, the recycle ring can be small to reduce
memory usage, since DMA mapping operations are inexpensive.
With a small recycle ring, attempting to refill the descriptor queue
with more buffers than the equivalent size of the recycle ring could
ultimately lead to memory leaks if page entries in the recycle ring
were overwritten. To prevent this, the check to see if the recycle
ring is full is changed to check if the next entry to be written is
NULL.
[bwh: Combine and rebase several commits so this is complete
before the following buffer-packing changes. Remove module
parameter.]
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
2013-02-13 10:54:41 +00:00
|
|
|
efx_sync_rx_buffer(efx, rx_buf, rx_buf->len);
|
2008-04-27 11:55:59 +00:00
|
|
|
|
|
|
|
|
/* Prefetch nice and early so data will (hopefully) be in cache by
|
|
|
|
|
* the time we look at it.
|
|
|
|
|
*/
|
2013-01-29 23:33:15 +00:00
|
|
|
prefetch(efx_rx_buf_va(rx_buf));
|
2008-04-27 11:55:59 +00:00
|
|
|
|
2012-10-18 14:49:54 +00:00
|
|
|
rx_buf->page_offset += efx->rx_prefix_size;
|
|
|
|
|
rx_buf->len -= efx->rx_prefix_size;
|
2013-01-29 23:33:15 +00:00
|
|
|
|
|
|
|
|
if (n_frags > 1) {
|
|
|
|
|
/* Release/sync DMA mapping for additional fragments.
|
|
|
|
|
* Fix length for last fragment.
|
|
|
|
|
*/
|
|
|
|
|
unsigned int tail_frags = n_frags - 1;
|
|
|
|
|
|
|
|
|
|
for (;;) {
|
|
|
|
|
rx_buf = efx_rx_buf_next(rx_queue, rx_buf);
|
|
|
|
|
if (--tail_frags == 0)
|
|
|
|
|
break;
|
2013-03-08 10:18:28 +00:00
|
|
|
efx_sync_rx_buffer(efx, rx_buf, efx->rx_dma_len);
|
2013-01-29 23:33:15 +00:00
|
|
|
}
|
2013-03-08 10:18:28 +00:00
|
|
|
rx_buf->len = len - (n_frags - 1) * efx->rx_dma_len;
|
sfc: reuse pages to avoid DMA mapping/unmapping costs
On POWER systems, DMA mapping/unmapping operations are very expensive.
These changes reduce these costs by trying to reuse DMA mapped pages.
After all the buffers associated with a page have been processed and
passed up, the page is placed into a ring (if there is room). For
each page that is required for a refill operation, a page in the ring
is examined to determine if its page count has fallen to 1, ie. the
kernel has released its reference to these packets. If this is the
case, the page can be immediately added back into the RX descriptor
ring, without having to re-map it for DMA.
If the kernel is still holding a reference to this page, it is removed
from the ring and unmapped for DMA. Then a new page, which can
immediately be used by RX buffers in the descriptor ring, is allocated
and DMA mapped.
The time a page needs to spend in the recycle ring before the kernel
has released its page references is based on the number of buffers
that use this page. As large pages can hold more RX buffers, the RX
recycle ring can be shorter. This reduces memory usage on POWER
systems, while maintaining the performance gain achieved by recycling
pages, following the driver change to pack more than two RX buffers
into large pages.
When an IOMMU is not present, the recycle ring can be small to reduce
memory usage, since DMA mapping operations are inexpensive.
With a small recycle ring, attempting to refill the descriptor queue
with more buffers than the equivalent size of the recycle ring could
ultimately lead to memory leaks if page entries in the recycle ring
were overwritten. To prevent this, the check to see if the recycle
ring is full is changed to check if the next entry to be written is
NULL.
[bwh: Combine and rebase several commits so this is complete
before the following buffer-packing changes. Remove module
parameter.]
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
2013-02-13 10:54:41 +00:00
|
|
|
efx_sync_rx_buffer(efx, rx_buf, rx_buf->len);
|
2013-01-29 23:33:15 +00:00
|
|
|
}
|
2013-01-29 23:33:15 +00:00
|
|
|
|
2013-07-04 22:48:46 +00:00
|
|
|
/* All fragments have been DMA-synced, so recycle pages. */
|
sfc: reuse pages to avoid DMA mapping/unmapping costs
On POWER systems, DMA mapping/unmapping operations are very expensive.
These changes reduce these costs by trying to reuse DMA mapped pages.
After all the buffers associated with a page have been processed and
passed up, the page is placed into a ring (if there is room). For
each page that is required for a refill operation, a page in the ring
is examined to determine if its page count has fallen to 1, ie. the
kernel has released its reference to these packets. If this is the
case, the page can be immediately added back into the RX descriptor
ring, without having to re-map it for DMA.
If the kernel is still holding a reference to this page, it is removed
from the ring and unmapped for DMA. Then a new page, which can
immediately be used by RX buffers in the descriptor ring, is allocated
and DMA mapped.
The time a page needs to spend in the recycle ring before the kernel
has released its page references is based on the number of buffers
that use this page. As large pages can hold more RX buffers, the RX
recycle ring can be shorter. This reduces memory usage on POWER
systems, while maintaining the performance gain achieved by recycling
pages, following the driver change to pack more than two RX buffers
into large pages.
When an IOMMU is not present, the recycle ring can be small to reduce
memory usage, since DMA mapping operations are inexpensive.
With a small recycle ring, attempting to refill the descriptor queue
with more buffers than the equivalent size of the recycle ring could
ultimately lead to memory leaks if page entries in the recycle ring
were overwritten. To prevent this, the check to see if the recycle
ring is full is changed to check if the next entry to be written is
NULL.
[bwh: Combine and rebase several commits so this is complete
before the following buffer-packing changes. Remove module
parameter.]
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
2013-02-13 10:54:41 +00:00
|
|
|
rx_buf = efx_rx_buffer(rx_queue, index);
|
2013-07-04 22:48:46 +00:00
|
|
|
efx_recycle_rx_pages(channel, rx_buf, n_frags);
|
sfc: reuse pages to avoid DMA mapping/unmapping costs
On POWER systems, DMA mapping/unmapping operations are very expensive.
These changes reduce these costs by trying to reuse DMA mapped pages.
After all the buffers associated with a page have been processed and
passed up, the page is placed into a ring (if there is room). For
each page that is required for a refill operation, a page in the ring
is examined to determine if its page count has fallen to 1, ie. the
kernel has released its reference to these packets. If this is the
case, the page can be immediately added back into the RX descriptor
ring, without having to re-map it for DMA.
If the kernel is still holding a reference to this page, it is removed
from the ring and unmapped for DMA. Then a new page, which can
immediately be used by RX buffers in the descriptor ring, is allocated
and DMA mapped.
The time a page needs to spend in the recycle ring before the kernel
has released its page references is based on the number of buffers
that use this page. As large pages can hold more RX buffers, the RX
recycle ring can be shorter. This reduces memory usage on POWER
systems, while maintaining the performance gain achieved by recycling
pages, following the driver change to pack more than two RX buffers
into large pages.
When an IOMMU is not present, the recycle ring can be small to reduce
memory usage, since DMA mapping operations are inexpensive.
With a small recycle ring, attempting to refill the descriptor queue
with more buffers than the equivalent size of the recycle ring could
ultimately lead to memory leaks if page entries in the recycle ring
were overwritten. To prevent this, the check to see if the recycle
ring is full is changed to check if the next entry to be written is
NULL.
[bwh: Combine and rebase several commits so this is complete
before the following buffer-packing changes. Remove module
parameter.]
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
2013-02-13 10:54:41 +00:00
|
|
|
|
2008-04-27 11:55:59 +00:00
|
|
|
/* Pipeline receives so that we give time for packet headers to be
|
|
|
|
|
* prefetched into cache.
|
|
|
|
|
*/
|
2013-01-29 23:33:14 +00:00
|
|
|
efx_rx_flush_packet(channel);
|
2013-01-29 23:33:15 +00:00
|
|
|
channel->rx_pkt_n_frags = n_frags;
|
|
|
|
|
channel->rx_pkt_index = index;
|
2008-04-27 11:55:59 +00:00
|
|
|
}
|
|
|
|
|
|
2013-01-11 12:26:21 +00:00
|
|
|
static void efx_rx_deliver(struct efx_channel *channel, u8 *eh,
|
2013-01-29 23:33:15 +00:00
|
|
|
struct efx_rx_buffer *rx_buf,
|
|
|
|
|
unsigned int n_frags)
|
2012-01-23 22:41:30 +00:00
|
|
|
{
|
|
|
|
|
struct sk_buff *skb;
|
2013-01-11 12:26:21 +00:00
|
|
|
u16 hdr_len = min_t(u16, rx_buf->len, EFX_SKB_HEADERS);
|
2012-01-23 22:41:30 +00:00
|
|
|
|
2013-01-29 23:33:15 +00:00
|
|
|
skb = efx_rx_mk_skb(channel, rx_buf, n_frags, eh, hdr_len);
|
2013-01-11 12:26:21 +00:00
|
|
|
if (unlikely(skb == NULL)) {
|
2015-05-29 11:25:54 +00:00
|
|
|
struct efx_rx_queue *rx_queue;
|
|
|
|
|
|
|
|
|
|
rx_queue = efx_channel_get_rx_queue(channel);
|
|
|
|
|
efx_free_rx_buffers(rx_queue, rx_buf, n_frags);
|
2013-01-11 12:26:21 +00:00
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
skb_record_rx_queue(skb, channel->rx_queue.core_index);
|
2012-01-23 22:41:30 +00:00
|
|
|
|
|
|
|
|
/* Set the SKB flags */
|
|
|
|
|
skb_checksum_none_assert(skb);
|
2017-02-08 16:51:02 +00:00
|
|
|
if (likely(rx_buf->flags & EFX_RX_PKT_CSUMMED)) {
|
2013-04-08 11:49:48 +00:00
|
|
|
skb->ip_summed = CHECKSUM_UNNECESSARY;
|
2017-02-08 16:51:02 +00:00
|
|
|
skb->csum_level = !!(rx_buf->flags & EFX_RX_PKT_CSUM_LEVEL);
|
|
|
|
|
}
|
2012-01-23 22:41:30 +00:00
|
|
|
|
2013-11-18 12:54:41 +00:00
|
|
|
efx_rx_skb_attach_timestamp(channel, skb);
|
|
|
|
|
|
2012-07-18 08:52:11 +00:00
|
|
|
if (channel->type->receive_skb)
|
2013-03-05 20:13:54 +00:00
|
|
|
if (channel->type->receive_skb(channel, skb))
|
2013-01-11 12:26:21 +00:00
|
|
|
return;
|
2013-03-05 20:13:54 +00:00
|
|
|
|
|
|
|
|
/* Pass the packet up */
|
2018-07-02 15:12:53 +00:00
|
|
|
if (channel->rx_list != NULL)
|
|
|
|
|
/* Add to list, will pass up later */
|
|
|
|
|
list_add_tail(&skb->list, channel->rx_list);
|
|
|
|
|
else
|
|
|
|
|
/* No list, so pass it up now */
|
|
|
|
|
netif_receive_skb(skb);
|
2012-01-23 22:41:30 +00:00
|
|
|
}
|
|
|
|
|
|
2019-10-31 10:23:23 +00:00
|
|
|
/** efx_do_xdp: perform XDP processing on a received packet
|
|
|
|
|
*
|
|
|
|
|
* Returns true if packet should still be delivered.
|
|
|
|
|
*/
|
|
|
|
|
static bool efx_do_xdp(struct efx_nic *efx, struct efx_channel *channel,
|
|
|
|
|
struct efx_rx_buffer *rx_buf, u8 **ehp)
|
|
|
|
|
{
|
|
|
|
|
u8 rx_prefix[EFX_MAX_RX_PREFIX_SIZE];
|
|
|
|
|
struct efx_rx_queue *rx_queue;
|
|
|
|
|
struct bpf_prog *xdp_prog;
|
2019-10-31 10:24:12 +00:00
|
|
|
struct xdp_frame *xdpf;
|
2019-10-31 10:23:23 +00:00
|
|
|
struct xdp_buff xdp;
|
|
|
|
|
u32 xdp_act;
|
|
|
|
|
s16 offset;
|
|
|
|
|
int err;
|
|
|
|
|
|
|
|
|
|
rcu_read_lock();
|
|
|
|
|
xdp_prog = rcu_dereference(efx->xdp_prog);
|
|
|
|
|
if (!xdp_prog) {
|
|
|
|
|
rcu_read_unlock();
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
rx_queue = efx_channel_get_rx_queue(channel);
|
|
|
|
|
|
|
|
|
|
if (unlikely(channel->rx_pkt_n_frags > 1)) {
|
|
|
|
|
/* We can't do XDP on fragmented packets - drop. */
|
|
|
|
|
rcu_read_unlock();
|
|
|
|
|
efx_free_rx_buffers(rx_queue, rx_buf,
|
|
|
|
|
channel->rx_pkt_n_frags);
|
|
|
|
|
if (net_ratelimit())
|
|
|
|
|
netif_err(efx, rx_err, efx->net_dev,
|
|
|
|
|
"XDP is not possible with multiple receive fragments (%d)\n",
|
|
|
|
|
channel->rx_pkt_n_frags);
|
2019-10-31 10:24:23 +00:00
|
|
|
channel->n_rx_xdp_bad_drops++;
|
2019-10-31 10:23:23 +00:00
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
dma_sync_single_for_cpu(&efx->pci_dev->dev, rx_buf->dma_addr,
|
|
|
|
|
rx_buf->len, DMA_FROM_DEVICE);
|
|
|
|
|
|
|
|
|
|
/* Save the rx prefix. */
|
|
|
|
|
EFX_WARN_ON_PARANOID(efx->rx_prefix_size > EFX_MAX_RX_PREFIX_SIZE);
|
|
|
|
|
memcpy(rx_prefix, *ehp - efx->rx_prefix_size,
|
|
|
|
|
efx->rx_prefix_size);
|
|
|
|
|
|
|
|
|
|
xdp.data = *ehp;
|
|
|
|
|
xdp.data_hard_start = xdp.data - XDP_PACKET_HEADROOM;
|
|
|
|
|
|
|
|
|
|
/* No support yet for XDP metadata */
|
|
|
|
|
xdp_set_data_meta_invalid(&xdp);
|
|
|
|
|
xdp.data_end = xdp.data + rx_buf->len;
|
|
|
|
|
xdp.rxq = &rx_queue->xdp_rxq_info;
|
|
|
|
|
|
|
|
|
|
xdp_act = bpf_prog_run_xdp(xdp_prog, &xdp);
|
|
|
|
|
rcu_read_unlock();
|
|
|
|
|
|
|
|
|
|
offset = (u8 *)xdp.data - *ehp;
|
|
|
|
|
|
|
|
|
|
switch (xdp_act) {
|
|
|
|
|
case XDP_PASS:
|
|
|
|
|
/* Fix up rx prefix. */
|
|
|
|
|
if (offset) {
|
|
|
|
|
*ehp += offset;
|
|
|
|
|
rx_buf->page_offset += offset;
|
|
|
|
|
rx_buf->len -= offset;
|
|
|
|
|
memcpy(*ehp - efx->rx_prefix_size, rx_prefix,
|
|
|
|
|
efx->rx_prefix_size);
|
|
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case XDP_TX:
|
2019-10-31 10:24:12 +00:00
|
|
|
/* Buffer ownership passes to tx on success. */
|
|
|
|
|
xdpf = convert_to_xdp_frame(&xdp);
|
|
|
|
|
err = efx_xdp_tx_buffers(efx, 1, &xdpf, true);
|
|
|
|
|
if (unlikely(err != 1)) {
|
|
|
|
|
efx_free_rx_buffers(rx_queue, rx_buf, 1);
|
|
|
|
|
if (net_ratelimit())
|
|
|
|
|
netif_err(efx, rx_err, efx->net_dev,
|
|
|
|
|
"XDP TX failed (%d)\n", err);
|
2019-10-31 10:24:23 +00:00
|
|
|
channel->n_rx_xdp_bad_drops++;
|
2019-11-12 15:36:01 +00:00
|
|
|
trace_xdp_exception(efx->net_dev, xdp_prog, xdp_act);
|
2019-10-31 10:24:23 +00:00
|
|
|
} else {
|
|
|
|
|
channel->n_rx_xdp_tx++;
|
2019-10-31 10:24:12 +00:00
|
|
|
}
|
|
|
|
|
break;
|
2019-10-31 10:23:23 +00:00
|
|
|
|
|
|
|
|
case XDP_REDIRECT:
|
|
|
|
|
err = xdp_do_redirect(efx->net_dev, &xdp, xdp_prog);
|
|
|
|
|
if (unlikely(err)) {
|
|
|
|
|
efx_free_rx_buffers(rx_queue, rx_buf, 1);
|
|
|
|
|
if (net_ratelimit())
|
|
|
|
|
netif_err(efx, rx_err, efx->net_dev,
|
|
|
|
|
"XDP redirect failed (%d)\n", err);
|
2019-10-31 10:24:23 +00:00
|
|
|
channel->n_rx_xdp_bad_drops++;
|
2019-11-12 15:36:01 +00:00
|
|
|
trace_xdp_exception(efx->net_dev, xdp_prog, xdp_act);
|
2019-10-31 10:24:23 +00:00
|
|
|
} else {
|
|
|
|
|
channel->n_rx_xdp_redirect++;
|
2019-10-31 10:23:23 +00:00
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
default:
|
|
|
|
|
bpf_warn_invalid_xdp_action(xdp_act);
|
|
|
|
|
efx_free_rx_buffers(rx_queue, rx_buf, 1);
|
2019-10-31 10:24:23 +00:00
|
|
|
channel->n_rx_xdp_bad_drops++;
|
2019-11-12 15:36:01 +00:00
|
|
|
trace_xdp_exception(efx->net_dev, xdp_prog, xdp_act);
|
2019-10-31 10:23:23 +00:00
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case XDP_ABORTED:
|
|
|
|
|
trace_xdp_exception(efx->net_dev, xdp_prog, xdp_act);
|
|
|
|
|
/* Fall through */
|
|
|
|
|
case XDP_DROP:
|
|
|
|
|
efx_free_rx_buffers(rx_queue, rx_buf, 1);
|
2019-10-31 10:24:23 +00:00
|
|
|
channel->n_rx_xdp_drops++;
|
2019-10-31 10:23:23 +00:00
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return xdp_act == XDP_PASS;
|
|
|
|
|
}
|
|
|
|
|
|
2008-04-27 11:55:59 +00:00
|
|
|
/* Handle a received packet. Second half: Touches packet payload. */
|
2013-01-29 23:33:15 +00:00
|
|
|
void __efx_rx_packet(struct efx_channel *channel)
|
2008-04-27 11:55:59 +00:00
|
|
|
{
|
|
|
|
|
struct efx_nic *efx = channel->efx;
|
2013-01-29 23:33:15 +00:00
|
|
|
struct efx_rx_buffer *rx_buf =
|
|
|
|
|
efx_rx_buffer(&channel->rx_queue, channel->rx_pkt_index);
|
2013-01-29 23:33:15 +00:00
|
|
|
u8 *eh = efx_rx_buf_va(rx_buf);
|
2010-06-25 07:05:33 +00:00
|
|
|
|
2013-04-27 00:55:18 +00:00
|
|
|
/* Read length from the prefix if necessary. This already
|
|
|
|
|
* excludes the length of the prefix itself.
|
|
|
|
|
*/
|
|
|
|
|
if (rx_buf->flags & EFX_RX_PKT_PREFIX_LEN)
|
|
|
|
|
rx_buf->len = le16_to_cpup((__le16 *)
|
|
|
|
|
(eh + efx->rx_packet_len_offset));
|
|
|
|
|
|
2008-05-07 12:36:19 +00:00
|
|
|
/* If we're in loopback test, then pass the packet directly to the
|
|
|
|
|
* loopback layer, and free the rx_buf here
|
|
|
|
|
*/
|
|
|
|
|
if (unlikely(efx->loopback_selftest)) {
|
2015-05-29 11:25:54 +00:00
|
|
|
struct efx_rx_queue *rx_queue;
|
|
|
|
|
|
2011-02-24 23:45:16 +00:00
|
|
|
efx_loopback_rx_packet(efx, eh, rx_buf->len);
|
2015-05-29 11:25:54 +00:00
|
|
|
rx_queue = efx_channel_get_rx_queue(channel);
|
|
|
|
|
efx_free_rx_buffers(rx_queue, rx_buf,
|
|
|
|
|
channel->rx_pkt_n_frags);
|
2013-01-29 23:33:15 +00:00
|
|
|
goto out;
|
2008-05-07 12:36:19 +00:00
|
|
|
}
|
|
|
|
|
|
2019-10-31 10:23:23 +00:00
|
|
|
if (!efx_do_xdp(efx, channel, rx_buf, &eh))
|
|
|
|
|
goto out;
|
|
|
|
|
|
2011-04-05 14:00:02 +00:00
|
|
|
if (unlikely(!(efx->net_dev->features & NETIF_F_RXCSUM)))
|
2011-08-26 17:05:11 +00:00
|
|
|
rx_buf->flags &= ~EFX_RX_PKT_CSUMMED;
|
2011-04-01 21:20:06 +00:00
|
|
|
|
2017-02-03 01:13:19 +00:00
|
|
|
if ((rx_buf->flags & EFX_RX_PKT_TCP) && !channel->type->receive_skb)
|
2013-01-29 23:33:15 +00:00
|
|
|
efx_rx_packet_gro(channel, rx_buf, channel->rx_pkt_n_frags, eh);
|
2012-01-23 22:41:30 +00:00
|
|
|
else
|
2013-01-29 23:33:15 +00:00
|
|
|
efx_rx_deliver(channel, eh, rx_buf, channel->rx_pkt_n_frags);
|
|
|
|
|
out:
|
|
|
|
|
channel->rx_pkt_n_frags = 0;
|
2008-04-27 11:55:59 +00:00
|
|
|
}
|
|
|
|
|
|
2012-11-08 01:46:53 +00:00
|
|
|
#ifdef CONFIG_RFS_ACCEL
|
|
|
|
|
|
2018-03-27 16:41:59 +00:00
|
|
|
static void efx_filter_rfs_work(struct work_struct *data)
|
|
|
|
|
{
|
|
|
|
|
struct efx_async_filter_insertion *req = container_of(data, struct efx_async_filter_insertion,
|
|
|
|
|
work);
|
|
|
|
|
struct efx_nic *efx = netdev_priv(req->net_dev);
|
|
|
|
|
struct efx_channel *channel = efx_get_channel(efx, req->rxq_index);
|
2018-04-13 18:18:09 +00:00
|
|
|
int slot_idx = req - efx->rps_slot;
|
2018-04-24 16:09:30 +00:00
|
|
|
struct efx_arfs_rule *rule;
|
|
|
|
|
u16 arfs_id = 0;
|
2018-03-27 16:41:59 +00:00
|
|
|
int rc;
|
|
|
|
|
|
2018-04-13 18:17:22 +00:00
|
|
|
rc = efx->type->filter_insert(efx, &req->spec, true);
|
2018-04-27 14:08:41 +00:00
|
|
|
if (rc >= 0)
|
2019-11-22 17:57:03 +00:00
|
|
|
/* Discard 'priority' part of EF10+ filter ID (mcdi_filters) */
|
2018-04-27 14:08:41 +00:00
|
|
|
rc %= efx->type->max_rx_ip_filters;
|
2018-04-24 16:09:30 +00:00
|
|
|
if (efx->rps_hash_table) {
|
|
|
|
|
spin_lock_bh(&efx->rps_hash_lock);
|
|
|
|
|
rule = efx_rps_hash_find(efx, &req->spec);
|
|
|
|
|
/* The rule might have already gone, if someone else's request
|
|
|
|
|
* for the same spec was already worked and then expired before
|
|
|
|
|
* we got around to our work. In that case we have nothing
|
|
|
|
|
* tying us to an arfs_id, meaning that as soon as the filter
|
|
|
|
|
* is considered for expiry it will be removed.
|
|
|
|
|
*/
|
|
|
|
|
if (rule) {
|
|
|
|
|
if (rc < 0)
|
|
|
|
|
rule->filter_id = EFX_ARFS_FILTER_ID_ERROR;
|
|
|
|
|
else
|
|
|
|
|
rule->filter_id = rc;
|
|
|
|
|
arfs_id = rule->arfs_id;
|
|
|
|
|
}
|
|
|
|
|
spin_unlock_bh(&efx->rps_hash_lock);
|
|
|
|
|
}
|
2018-03-27 16:41:59 +00:00
|
|
|
if (rc >= 0) {
|
|
|
|
|
/* Remember this so we can check whether to expire the filter
|
|
|
|
|
* later.
|
|
|
|
|
*/
|
|
|
|
|
mutex_lock(&efx->rps_mutex);
|
2019-11-22 17:57:03 +00:00
|
|
|
if (channel->rps_flow_id[rc] == RPS_FLOW_ID_INVALID)
|
|
|
|
|
channel->rfs_filter_count++;
|
2018-03-27 16:41:59 +00:00
|
|
|
channel->rps_flow_id[rc] = req->flow_id;
|
|
|
|
|
mutex_unlock(&efx->rps_mutex);
|
|
|
|
|
|
|
|
|
|
if (req->spec.ether_type == htons(ETH_P_IP))
|
|
|
|
|
netif_info(efx, rx_status, efx->net_dev,
|
2018-04-24 16:09:30 +00:00
|
|
|
"steering %s %pI4:%u:%pI4:%u to queue %u [flow %u filter %d id %u]\n",
|
2018-03-27 16:41:59 +00:00
|
|
|
(req->spec.ip_proto == IPPROTO_TCP) ? "TCP" : "UDP",
|
|
|
|
|
req->spec.rem_host, ntohs(req->spec.rem_port),
|
|
|
|
|
req->spec.loc_host, ntohs(req->spec.loc_port),
|
2018-04-24 16:09:30 +00:00
|
|
|
req->rxq_index, req->flow_id, rc, arfs_id);
|
2018-03-27 16:41:59 +00:00
|
|
|
else
|
|
|
|
|
netif_info(efx, rx_status, efx->net_dev,
|
2018-04-24 16:09:30 +00:00
|
|
|
"steering %s [%pI6]:%u:[%pI6]:%u to queue %u [flow %u filter %d id %u]\n",
|
2018-03-27 16:41:59 +00:00
|
|
|
(req->spec.ip_proto == IPPROTO_TCP) ? "TCP" : "UDP",
|
|
|
|
|
req->spec.rem_host, ntohs(req->spec.rem_port),
|
|
|
|
|
req->spec.loc_host, ntohs(req->spec.loc_port),
|
2018-04-24 16:09:30 +00:00
|
|
|
req->rxq_index, req->flow_id, rc, arfs_id);
|
2019-11-22 17:57:27 +00:00
|
|
|
channel->n_rfs_succeeded++;
|
2019-11-22 17:57:19 +00:00
|
|
|
} else {
|
|
|
|
|
if (req->spec.ether_type == htons(ETH_P_IP))
|
|
|
|
|
netif_dbg(efx, rx_status, efx->net_dev,
|
|
|
|
|
"failed to steer %s %pI4:%u:%pI4:%u to queue %u [flow %u rc %d id %u]\n",
|
|
|
|
|
(req->spec.ip_proto == IPPROTO_TCP) ? "TCP" : "UDP",
|
|
|
|
|
req->spec.rem_host, ntohs(req->spec.rem_port),
|
|
|
|
|
req->spec.loc_host, ntohs(req->spec.loc_port),
|
|
|
|
|
req->rxq_index, req->flow_id, rc, arfs_id);
|
|
|
|
|
else
|
|
|
|
|
netif_dbg(efx, rx_status, efx->net_dev,
|
|
|
|
|
"failed to steer %s [%pI6]:%u:[%pI6]:%u to queue %u [flow %u rc %d id %u]\n",
|
|
|
|
|
(req->spec.ip_proto == IPPROTO_TCP) ? "TCP" : "UDP",
|
|
|
|
|
req->spec.rem_host, ntohs(req->spec.rem_port),
|
|
|
|
|
req->spec.loc_host, ntohs(req->spec.loc_port),
|
|
|
|
|
req->rxq_index, req->flow_id, rc, arfs_id);
|
2019-11-22 17:57:27 +00:00
|
|
|
channel->n_rfs_failed++;
|
2019-11-22 17:57:19 +00:00
|
|
|
/* We're overloading the NIC's filter tables, so let's do a
|
|
|
|
|
* chunk of extra expiry work.
|
|
|
|
|
*/
|
|
|
|
|
__efx_filter_rfs_expire(channel, min(channel->rfs_filter_count,
|
|
|
|
|
100u));
|
2018-03-27 16:41:59 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Release references */
|
2018-04-13 18:18:09 +00:00
|
|
|
clear_bit(slot_idx, &efx->rps_slot_map);
|
2018-03-27 16:41:59 +00:00
|
|
|
dev_put(req->net_dev);
|
|
|
|
|
}
|
|
|
|
|
|
2012-11-08 01:46:53 +00:00
|
|
|
int efx_filter_rfs(struct net_device *net_dev, const struct sk_buff *skb,
|
|
|
|
|
u16 rxq_index, u32 flow_id)
|
|
|
|
|
{
|
|
|
|
|
struct efx_nic *efx = netdev_priv(net_dev);
|
2018-03-27 16:41:59 +00:00
|
|
|
struct efx_async_filter_insertion *req;
|
2018-04-24 16:09:30 +00:00
|
|
|
struct efx_arfs_rule *rule;
|
2016-05-26 20:46:05 +00:00
|
|
|
struct flow_keys fk;
|
2018-04-13 18:18:09 +00:00
|
|
|
int slot_idx;
|
2018-04-24 16:09:30 +00:00
|
|
|
bool new;
|
2018-04-13 18:18:09 +00:00
|
|
|
int rc;
|
2012-11-08 01:46:53 +00:00
|
|
|
|
2018-04-13 18:18:09 +00:00
|
|
|
/* find a free slot */
|
|
|
|
|
for (slot_idx = 0; slot_idx < EFX_RPS_MAX_IN_FLIGHT; slot_idx++)
|
|
|
|
|
if (!test_and_set_bit(slot_idx, &efx->rps_slot_map))
|
|
|
|
|
break;
|
|
|
|
|
if (slot_idx >= EFX_RPS_MAX_IN_FLIGHT)
|
|
|
|
|
return -EBUSY;
|
2016-05-31 18:12:32 +00:00
|
|
|
|
2018-04-13 18:18:09 +00:00
|
|
|
if (flow_id == RPS_FLOW_ID_INVALID) {
|
|
|
|
|
rc = -EINVAL;
|
|
|
|
|
goto out_clear;
|
|
|
|
|
}
|
2012-11-08 01:46:53 +00:00
|
|
|
|
2018-04-13 18:18:09 +00:00
|
|
|
if (!skb_flow_dissect_flow_keys(skb, &fk, 0)) {
|
|
|
|
|
rc = -EPROTONOSUPPORT;
|
|
|
|
|
goto out_clear;
|
|
|
|
|
}
|
2012-11-08 01:46:53 +00:00
|
|
|
|
2018-04-13 18:18:09 +00:00
|
|
|
if (fk.basic.n_proto != htons(ETH_P_IP) && fk.basic.n_proto != htons(ETH_P_IPV6)) {
|
|
|
|
|
rc = -EPROTONOSUPPORT;
|
|
|
|
|
goto out_clear;
|
|
|
|
|
}
|
|
|
|
|
if (fk.control.flags & FLOW_DIS_IS_FRAGMENT) {
|
|
|
|
|
rc = -EPROTONOSUPPORT;
|
|
|
|
|
goto out_clear;
|
|
|
|
|
}
|
2018-03-27 16:41:59 +00:00
|
|
|
|
2018-04-13 18:18:09 +00:00
|
|
|
req = efx->rps_slot + slot_idx;
|
2018-03-27 16:41:59 +00:00
|
|
|
efx_filter_init_rx(&req->spec, EFX_FILTER_PRI_HINT,
|
2012-11-08 01:46:53 +00:00
|
|
|
efx->rx_scatter ? EFX_FILTER_FLAG_RX_SCATTER : 0,
|
|
|
|
|
rxq_index);
|
2018-03-27 16:41:59 +00:00
|
|
|
req->spec.match_flags =
|
2013-09-03 16:22:23 +00:00
|
|
|
EFX_FILTER_MATCH_ETHER_TYPE | EFX_FILTER_MATCH_IP_PROTO |
|
|
|
|
|
EFX_FILTER_MATCH_LOC_HOST | EFX_FILTER_MATCH_LOC_PORT |
|
|
|
|
|
EFX_FILTER_MATCH_REM_HOST | EFX_FILTER_MATCH_REM_PORT;
|
2018-03-27 16:41:59 +00:00
|
|
|
req->spec.ether_type = fk.basic.n_proto;
|
|
|
|
|
req->spec.ip_proto = fk.basic.ip_proto;
|
2016-05-26 20:46:05 +00:00
|
|
|
|
|
|
|
|
if (fk.basic.n_proto == htons(ETH_P_IP)) {
|
2018-03-27 16:41:59 +00:00
|
|
|
req->spec.rem_host[0] = fk.addrs.v4addrs.src;
|
|
|
|
|
req->spec.loc_host[0] = fk.addrs.v4addrs.dst;
|
2013-09-03 16:22:23 +00:00
|
|
|
} else {
|
2018-03-27 16:41:59 +00:00
|
|
|
memcpy(req->spec.rem_host, &fk.addrs.v6addrs.src,
|
|
|
|
|
sizeof(struct in6_addr));
|
|
|
|
|
memcpy(req->spec.loc_host, &fk.addrs.v6addrs.dst,
|
|
|
|
|
sizeof(struct in6_addr));
|
2013-09-03 16:22:23 +00:00
|
|
|
}
|
|
|
|
|
|
2018-03-27 16:41:59 +00:00
|
|
|
req->spec.rem_port = fk.ports.src;
|
|
|
|
|
req->spec.loc_port = fk.ports.dst;
|
2012-11-08 01:46:53 +00:00
|
|
|
|
2018-04-24 16:09:30 +00:00
|
|
|
if (efx->rps_hash_table) {
|
|
|
|
|
/* Add it to ARFS hash table */
|
|
|
|
|
spin_lock(&efx->rps_hash_lock);
|
|
|
|
|
rule = efx_rps_hash_add(efx, &req->spec, &new);
|
|
|
|
|
if (!rule) {
|
|
|
|
|
rc = -ENOMEM;
|
|
|
|
|
goto out_unlock;
|
|
|
|
|
}
|
|
|
|
|
if (new)
|
|
|
|
|
rule->arfs_id = efx->rps_next_id++ % RPS_NO_FILTER;
|
|
|
|
|
rc = rule->arfs_id;
|
|
|
|
|
/* Skip if existing or pending filter already does the right thing */
|
|
|
|
|
if (!new && rule->rxq_index == rxq_index &&
|
|
|
|
|
rule->filter_id >= EFX_ARFS_FILTER_ID_PENDING)
|
|
|
|
|
goto out_unlock;
|
|
|
|
|
rule->rxq_index = rxq_index;
|
|
|
|
|
rule->filter_id = EFX_ARFS_FILTER_ID_PENDING;
|
|
|
|
|
spin_unlock(&efx->rps_hash_lock);
|
|
|
|
|
} else {
|
|
|
|
|
/* Without an ARFS hash table, we just use arfs_id 0 for all
|
|
|
|
|
* filters. This means if multiple flows hash to the same
|
|
|
|
|
* flow_id, all but the most recently touched will be eligible
|
|
|
|
|
* for expiry.
|
|
|
|
|
*/
|
|
|
|
|
rc = 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Queue the request */
|
2018-03-27 16:41:59 +00:00
|
|
|
dev_hold(req->net_dev = net_dev);
|
|
|
|
|
INIT_WORK(&req->work, efx_filter_rfs_work);
|
|
|
|
|
req->rxq_index = rxq_index;
|
|
|
|
|
req->flow_id = flow_id;
|
|
|
|
|
schedule_work(&req->work);
|
2018-04-24 16:09:30 +00:00
|
|
|
return rc;
|
|
|
|
|
out_unlock:
|
|
|
|
|
spin_unlock(&efx->rps_hash_lock);
|
2018-04-13 18:18:09 +00:00
|
|
|
out_clear:
|
|
|
|
|
clear_bit(slot_idx, &efx->rps_slot_map);
|
|
|
|
|
return rc;
|
2012-11-08 01:46:53 +00:00
|
|
|
}
|
|
|
|
|
|
2019-11-22 17:57:03 +00:00
|
|
|
bool __efx_filter_rfs_expire(struct efx_channel *channel, unsigned int quota)
|
2012-11-08 01:46:53 +00:00
|
|
|
{
|
|
|
|
|
bool (*expire_one)(struct efx_nic *efx, u32 flow_id, unsigned int index);
|
2019-11-22 17:57:03 +00:00
|
|
|
struct efx_nic *efx = channel->efx;
|
|
|
|
|
unsigned int index, size, start;
|
2012-11-08 01:46:53 +00:00
|
|
|
u32 flow_id;
|
|
|
|
|
|
2018-03-27 16:41:59 +00:00
|
|
|
if (!mutex_trylock(&efx->rps_mutex))
|
2012-11-08 01:46:53 +00:00
|
|
|
return false;
|
|
|
|
|
expire_one = efx->type->filter_rfs_expire_one;
|
2019-11-22 17:57:03 +00:00
|
|
|
index = channel->rfs_expire_index;
|
|
|
|
|
start = index;
|
2012-11-08 01:46:53 +00:00
|
|
|
size = efx->type->max_rx_ip_filters;
|
2019-11-22 17:57:03 +00:00
|
|
|
while (quota) {
|
2016-05-31 18:12:32 +00:00
|
|
|
flow_id = channel->rps_flow_id[index];
|
|
|
|
|
|
2019-11-22 17:57:03 +00:00
|
|
|
if (flow_id != RPS_FLOW_ID_INVALID) {
|
|
|
|
|
quota--;
|
|
|
|
|
if (expire_one(efx, flow_id, index)) {
|
|
|
|
|
netif_info(efx, rx_status, efx->net_dev,
|
|
|
|
|
"expired filter %d [channel %u flow %u]\n",
|
|
|
|
|
index, channel->channel, flow_id);
|
|
|
|
|
channel->rps_flow_id[index] = RPS_FLOW_ID_INVALID;
|
|
|
|
|
channel->rfs_filter_count--;
|
|
|
|
|
}
|
2016-05-31 18:12:32 +00:00
|
|
|
}
|
2019-11-22 17:57:03 +00:00
|
|
|
if (++index == size)
|
2012-11-08 01:46:53 +00:00
|
|
|
index = 0;
|
2019-11-22 17:57:03 +00:00
|
|
|
/* If we were called with a quota that exceeds the total number
|
2019-11-22 17:57:19 +00:00
|
|
|
* of filters in the table (which shouldn't happen, but could
|
|
|
|
|
* if two callers race), ensure that we don't loop forever -
|
|
|
|
|
* stop when we've examined every row of the table.
|
2019-11-22 17:57:03 +00:00
|
|
|
*/
|
2019-11-22 17:57:19 +00:00
|
|
|
if (index == start)
|
2019-11-22 17:57:03 +00:00
|
|
|
break;
|
2012-11-08 01:46:53 +00:00
|
|
|
}
|
|
|
|
|
|
2019-11-22 17:57:03 +00:00
|
|
|
channel->rfs_expire_index = index;
|
2018-03-27 16:41:59 +00:00
|
|
|
mutex_unlock(&efx->rps_mutex);
|
2012-11-08 01:46:53 +00:00
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#endif /* CONFIG_RFS_ACCEL */
|