License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* INET An implementation of the TCP/IP protocol suite for the LINUX
|
|
|
|
* operating system. INET is implemented using the BSD Socket
|
|
|
|
* interface as the means of communication with the user level.
|
|
|
|
*
|
|
|
|
* The IP fragmentation functionality.
|
2007-02-09 14:24:47 +00:00
|
|
|
*
|
2005-04-16 22:20:36 +00:00
|
|
|
* Authors: Fred N. van Kempen <waltje@uWalt.NL.Mugnet.ORG>
|
2008-10-14 02:01:08 +00:00
|
|
|
* Alan Cox <alan@lxorguk.ukuu.org.uk>
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
|
|
|
* Fixes:
|
|
|
|
* Alan Cox : Split from ip.c , see ip_input.c for history.
|
|
|
|
* David S. Miller : Begin massive cleanup...
|
|
|
|
* Andi Kleen : Add sysctls.
|
|
|
|
* xxxx : Overlapfrag bug.
|
|
|
|
* Ultima : ip_expire() kernel panic.
|
|
|
|
* Bill Hawes : Frag accounting and evictor fixes.
|
|
|
|
* John McDonald : 0 length frag bug.
|
|
|
|
* Alexey Kuznetsov: SMP races, threading, cleanup.
|
|
|
|
* Patrick McHardy : LRU queue of frag heads for evictor.
|
|
|
|
*/
|
|
|
|
|
2012-03-12 07:03:32 +00:00
|
|
|
#define pr_fmt(fmt) "IPv4: " fmt
|
|
|
|
|
2005-12-14 07:14:27 +00:00
|
|
|
#include <linux/compiler.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <linux/module.h>
|
|
|
|
#include <linux/types.h>
|
|
|
|
#include <linux/mm.h>
|
|
|
|
#include <linux/jiffies.h>
|
|
|
|
#include <linux/skbuff.h>
|
|
|
|
#include <linux/list.h>
|
|
|
|
#include <linux/ip.h>
|
|
|
|
#include <linux/icmp.h>
|
|
|
|
#include <linux/netdevice.h>
|
|
|
|
#include <linux/jhash.h>
|
|
|
|
#include <linux/random.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 08:04:11 +00:00
|
|
|
#include <linux/slab.h>
|
2010-01-23 09:57:42 +00:00
|
|
|
#include <net/route.h>
|
|
|
|
#include <net/dst.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <net/sock.h>
|
|
|
|
#include <net/ip.h>
|
|
|
|
#include <net/icmp.h>
|
|
|
|
#include <net/checksum.h>
|
2005-12-14 07:14:27 +00:00
|
|
|
#include <net/inetpeer.h>
|
2007-10-15 09:24:19 +00:00
|
|
|
#include <net/inet_frag.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <linux/tcp.h>
|
|
|
|
#include <linux/udp.h>
|
|
|
|
#include <linux/inet.h>
|
|
|
|
#include <linux/netfilter_ipv4.h>
|
2011-01-05 07:52:55 +00:00
|
|
|
#include <net/inet_ecn.h>
|
2015-09-30 03:07:13 +00:00
|
|
|
#include <net/l3mdev.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* NOTE. Logic of IP defragmentation is parallel to corresponding IPv6
|
|
|
|
* code now. If you change something here, _PLEASE_ update ipv6/reassembly.c
|
|
|
|
* as well. Or notify me, at least. --ANK
|
|
|
|
*/
|
2014-08-01 10:29:48 +00:00
|
|
|
static const char ip_frag_cache_name[] = "ip4-frags";
|
2005-12-14 07:14:27 +00:00
|
|
|
|
2018-08-11 20:27:24 +00:00
|
|
|
/* Use skb->cb to track consecutive/adjacent fragments coming at
|
|
|
|
* the end of the queue. Nodes in the rb-tree queue will
|
|
|
|
* contain "runs" of one or more adjacent fragments.
|
|
|
|
*
|
|
|
|
* Invariants:
|
|
|
|
* - next_frag is NULL at the tail of a "run";
|
|
|
|
* - the head of a "run" has the sum of all fragment lengths in frag_run_len.
|
|
|
|
*/
|
|
|
|
struct ipfrag_skb_cb {
|
|
|
|
struct inet_skb_parm h;
|
|
|
|
struct sk_buff *next_frag;
|
|
|
|
int frag_run_len;
|
|
|
|
};
|
|
|
|
|
|
|
|
#define FRAG_CB(skb) ((struct ipfrag_skb_cb *)((skb)->cb))
|
|
|
|
|
|
|
|
static void ip4_frag_init_run(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
BUILD_BUG_ON(sizeof(struct ipfrag_skb_cb) > sizeof(skb->cb));
|
|
|
|
|
|
|
|
FRAG_CB(skb)->next_frag = NULL;
|
|
|
|
FRAG_CB(skb)->frag_run_len = skb->len;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Append skb to the last "run". */
|
|
|
|
static void ip4_frag_append_to_last_run(struct inet_frag_queue *q,
|
|
|
|
struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
RB_CLEAR_NODE(&skb->rbnode);
|
|
|
|
FRAG_CB(skb)->next_frag = NULL;
|
|
|
|
|
|
|
|
FRAG_CB(q->last_run_head)->frag_run_len += skb->len;
|
|
|
|
FRAG_CB(q->fragments_tail)->next_frag = skb;
|
|
|
|
q->fragments_tail = skb;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Create a new "run" with the skb. */
|
|
|
|
static void ip4_frag_create_run(struct inet_frag_queue *q, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
if (q->last_run_head)
|
|
|
|
rb_link_node(&skb->rbnode, &q->last_run_head->rbnode,
|
|
|
|
&q->last_run_head->rbnode.rb_right);
|
|
|
|
else
|
|
|
|
rb_link_node(&skb->rbnode, NULL, &q->rb_fragments.rb_node);
|
|
|
|
rb_insert_color(&skb->rbnode, &q->rb_fragments);
|
|
|
|
|
|
|
|
ip4_frag_init_run(skb);
|
|
|
|
q->fragments_tail = skb;
|
|
|
|
q->last_run_head = skb;
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Describe an entry in the "incomplete datagrams" queue. */
|
|
|
|
struct ipq {
|
2007-10-15 09:24:19 +00:00
|
|
|
struct inet_frag_queue q;
|
|
|
|
|
2011-01-05 07:52:55 +00:00
|
|
|
u8 ecn; /* RFC3168 support */
|
2015-05-22 14:32:51 +00:00
|
|
|
u16 max_df_size; /* largest frag with DF set seen */
|
2005-12-14 07:14:27 +00:00
|
|
|
int iif;
|
|
|
|
unsigned int rid;
|
|
|
|
struct inet_peer *peer;
|
2005-04-16 22:20:36 +00:00
|
|
|
};
|
|
|
|
|
2014-11-04 19:44:04 +00:00
|
|
|
static u8 ip4_frag_ecn(u8 tos)
|
2011-01-05 07:52:55 +00:00
|
|
|
{
|
2011-05-16 08:37:37 +00:00
|
|
|
return 1 << (tos & INET_ECN_MASK);
|
2011-01-05 07:52:55 +00:00
|
|
|
}
|
|
|
|
|
2007-10-15 09:31:52 +00:00
|
|
|
static struct inet_frags ip4_frags;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2018-08-11 20:27:25 +00:00
|
|
|
static int ip_frag_reasm(struct ipq *qp, struct sk_buff *skb,
|
|
|
|
struct sk_buff *prev_tail, struct net_device *dev);
|
2007-10-14 07:38:15 +00:00
|
|
|
|
2007-10-18 02:47:21 +00:00
|
|
|
|
2014-07-24 14:50:29 +00:00
|
|
|
static void ip4_frag_init(struct inet_frag_queue *q, const void *a)
|
2007-10-18 02:46:47 +00:00
|
|
|
{
|
|
|
|
struct ipq *qp = container_of(q, struct ipq, q);
|
2012-06-08 01:21:40 +00:00
|
|
|
struct netns_ipv4 *ipv4 = container_of(q->net, struct netns_ipv4,
|
|
|
|
frags);
|
|
|
|
struct net *net = container_of(ipv4, struct net, ipv4);
|
|
|
|
|
inet: frags: use rhashtables for reassembly units
Some applications still rely on IP fragmentation, and to be fair linux
reassembly unit is not working under any serious load.
It uses static hash tables of 1024 buckets, and up to 128 items per bucket (!!!)
A work queue is supposed to garbage collect items when host is under memory
pressure, and doing a hash rebuild, changing seed used in hash computations.
This work queue blocks softirqs for up to 25 ms when doing a hash rebuild,
occurring every 5 seconds if host is under fire.
Then there is the problem of sharing this hash table for all netns.
It is time to switch to rhashtables, and allocate one of them per netns
to speedup netns dismantle, since this is a critical metric these days.
Lookup is now using RCU. A followup patch will even remove
the refcount hold/release left from prior implementation and save
a couple of atomic operations.
Before this patch, 16 cpus (16 RX queue NIC) could not handle more
than 1 Mpps frags DDOS.
After the patch, I reach 9 Mpps without any tuning, and can use up to 2GB
of storage for the fragments (exact number depends on frags being evicted
after timeout)
$ grep FRAG /proc/net/sockstat
FRAG: inuse 1966916 memory 2140004608
A followup patch will change the limits for 64bit arches.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Florian Westphal <fw@strlen.de>
Cc: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Alexander Aring <alex.aring@gmail.com>
Cc: Stefan Schmidt <stefan@osg.samsung.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-03-31 19:58:49 +00:00
|
|
|
const struct frag_v4_compare_key *key = a;
|
2007-10-18 02:46:47 +00:00
|
|
|
|
inet: frags: use rhashtables for reassembly units
Some applications still rely on IP fragmentation, and to be fair linux
reassembly unit is not working under any serious load.
It uses static hash tables of 1024 buckets, and up to 128 items per bucket (!!!)
A work queue is supposed to garbage collect items when host is under memory
pressure, and doing a hash rebuild, changing seed used in hash computations.
This work queue blocks softirqs for up to 25 ms when doing a hash rebuild,
occurring every 5 seconds if host is under fire.
Then there is the problem of sharing this hash table for all netns.
It is time to switch to rhashtables, and allocate one of them per netns
to speedup netns dismantle, since this is a critical metric these days.
Lookup is now using RCU. A followup patch will even remove
the refcount hold/release left from prior implementation and save
a couple of atomic operations.
Before this patch, 16 cpus (16 RX queue NIC) could not handle more
than 1 Mpps frags DDOS.
After the patch, I reach 9 Mpps without any tuning, and can use up to 2GB
of storage for the fragments (exact number depends on frags being evicted
after timeout)
$ grep FRAG /proc/net/sockstat
FRAG: inuse 1966916 memory 2140004608
A followup patch will change the limits for 64bit arches.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Florian Westphal <fw@strlen.de>
Cc: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Alexander Aring <alex.aring@gmail.com>
Cc: Stefan Schmidt <stefan@osg.samsung.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-03-31 19:58:49 +00:00
|
|
|
q->key.v4 = *key;
|
|
|
|
qp->ecn = 0;
|
2016-02-15 10:11:31 +00:00
|
|
|
qp->peer = q->net->max_dist ?
|
inet: frags: use rhashtables for reassembly units
Some applications still rely on IP fragmentation, and to be fair linux
reassembly unit is not working under any serious load.
It uses static hash tables of 1024 buckets, and up to 128 items per bucket (!!!)
A work queue is supposed to garbage collect items when host is under memory
pressure, and doing a hash rebuild, changing seed used in hash computations.
This work queue blocks softirqs for up to 25 ms when doing a hash rebuild,
occurring every 5 seconds if host is under fire.
Then there is the problem of sharing this hash table for all netns.
It is time to switch to rhashtables, and allocate one of them per netns
to speedup netns dismantle, since this is a critical metric these days.
Lookup is now using RCU. A followup patch will even remove
the refcount hold/release left from prior implementation and save
a couple of atomic operations.
Before this patch, 16 cpus (16 RX queue NIC) could not handle more
than 1 Mpps frags DDOS.
After the patch, I reach 9 Mpps without any tuning, and can use up to 2GB
of storage for the fragments (exact number depends on frags being evicted
after timeout)
$ grep FRAG /proc/net/sockstat
FRAG: inuse 1966916 memory 2140004608
A followup patch will change the limits for 64bit arches.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Florian Westphal <fw@strlen.de>
Cc: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Alexander Aring <alex.aring@gmail.com>
Cc: Stefan Schmidt <stefan@osg.samsung.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-03-31 19:58:49 +00:00
|
|
|
inet_getpeer_v4(net->ipv4.peers, key->saddr, key->vif, 1) :
|
2015-08-27 23:07:03 +00:00
|
|
|
NULL;
|
2007-10-18 02:46:47 +00:00
|
|
|
}
|
|
|
|
|
2014-11-04 19:44:04 +00:00
|
|
|
static void ip4_frag_free(struct inet_frag_queue *q)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2007-10-15 09:39:14 +00:00
|
|
|
struct ipq *qp;
|
|
|
|
|
|
|
|
qp = container_of(q, struct ipq, q);
|
|
|
|
if (qp->peer)
|
|
|
|
inet_putpeer(qp->peer);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/* Destruction primitives. */
|
|
|
|
|
2014-11-04 19:44:04 +00:00
|
|
|
static void ipq_put(struct ipq *ipq)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2018-03-31 19:58:44 +00:00
|
|
|
inet_frag_put(&ipq->q);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Kill ipq entry. It is not destroyed immediately,
|
|
|
|
* because caller (and someone more) holds reference count.
|
|
|
|
*/
|
|
|
|
static void ipq_kill(struct ipq *ipq)
|
|
|
|
{
|
2018-03-31 19:58:44 +00:00
|
|
|
inet_frag_kill(&ipq->q);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2015-05-15 21:15:35 +00:00
|
|
|
static bool frag_expire_skip_icmp(u32 user)
|
|
|
|
{
|
|
|
|
return user == IP_DEFRAG_AF_PACKET ||
|
|
|
|
ip_defrag_user_in_between(user, IP_DEFRAG_CONNTRACK_IN,
|
2015-05-15 21:15:36 +00:00
|
|
|
__IP_DEFRAG_CONNTRACK_IN_END) ||
|
|
|
|
ip_defrag_user_in_between(user, IP_DEFRAG_CONNTRACK_BRIDGE_IN,
|
|
|
|
__IP_DEFRAG_CONNTRACK_BRIDGE_IN);
|
2015-05-15 21:15:35 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* Oops, a fragment queue timed out. Kill it and send an ICMP reply.
|
|
|
|
*/
|
2017-10-17 00:29:20 +00:00
|
|
|
static void ip_expire(struct timer_list *t)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2017-10-17 00:29:20 +00:00
|
|
|
struct inet_frag_queue *frag = from_timer(frag, t, timer);
|
2018-03-31 19:58:51 +00:00
|
|
|
const struct iphdr *iph;
|
2018-08-02 23:34:39 +00:00
|
|
|
struct sk_buff *head = NULL;
|
2008-07-17 03:19:08 +00:00
|
|
|
struct net *net;
|
2018-03-31 19:58:51 +00:00
|
|
|
struct ipq *qp;
|
|
|
|
int err;
|
2007-10-18 02:45:23 +00:00
|
|
|
|
2017-10-17 00:29:20 +00:00
|
|
|
qp = container_of(frag, struct ipq, q);
|
2008-07-17 03:19:08 +00:00
|
|
|
net = container_of(qp->q.net, struct net, ipv4.frags);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
inet: frag: release spinlock before calling icmp_send()
Dmitry reported a lockdep splat [1] (false positive) that we can fix
by releasing the spinlock before calling icmp_send() from ip_expire()
This is a false positive because sending an ICMP message can not
possibly re-enter the IP frag engine.
[1]
[ INFO: possible circular locking dependency detected ]
4.10.0+ #29 Not tainted
-------------------------------------------------------
modprobe/12392 is trying to acquire lock:
(_xmit_ETHER#2){+.-...}, at: [<ffffffff837a8182>] spin_lock
include/linux/spinlock.h:299 [inline]
(_xmit_ETHER#2){+.-...}, at: [<ffffffff837a8182>] __netif_tx_lock
include/linux/netdevice.h:3486 [inline]
(_xmit_ETHER#2){+.-...}, at: [<ffffffff837a8182>]
sch_direct_xmit+0x282/0x6d0 net/sched/sch_generic.c:180
but task is already holding lock:
(&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>] spin_lock
include/linux/spinlock.h:299 [inline]
(&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>]
ip_expire+0x51/0x6c0 net/ipv4/ip_fragment.c:201
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&(&q->lock)->rlock){+.-...}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x33/0x50 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:299 [inline]
ip_defrag+0x3a2/0x4130 net/ipv4/ip_fragment.c:669
ip_check_defrag+0x4e3/0x8b0 net/ipv4/ip_fragment.c:713
packet_rcv_fanout+0x282/0x800 net/packet/af_packet.c:1459
deliver_skb net/core/dev.c:1834 [inline]
dev_queue_xmit_nit+0x294/0xa90 net/core/dev.c:1890
xmit_one net/core/dev.c:2903 [inline]
dev_hard_start_xmit+0x16b/0xab0 net/core/dev.c:2923
sch_direct_xmit+0x31f/0x6d0 net/sched/sch_generic.c:182
__dev_xmit_skb net/core/dev.c:3092 [inline]
__dev_queue_xmit+0x13e5/0x1e60 net/core/dev.c:3358
dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
neigh_resolve_output+0x6b9/0xb10 net/core/neighbour.c:1308
neigh_output include/net/neighbour.h:478 [inline]
ip_finish_output2+0x8b8/0x15a0 net/ipv4/ip_output.c:228
ip_do_fragment+0x1d93/0x2720 net/ipv4/ip_output.c:672
ip_fragment.constprop.54+0x145/0x200 net/ipv4/ip_output.c:545
ip_finish_output+0x82d/0xe10 net/ipv4/ip_output.c:314
NF_HOOK_COND include/linux/netfilter.h:246 [inline]
ip_output+0x1f0/0x7a0 net/ipv4/ip_output.c:404
dst_output include/net/dst.h:486 [inline]
ip_local_out+0x95/0x170 net/ipv4/ip_output.c:124
ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1492
ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1512
raw_sendmsg+0x26de/0x3a00 net/ipv4/raw.c:655
inet_sendmsg+0x164/0x5b0 net/ipv4/af_inet.c:761
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
___sys_sendmsg+0x4a3/0x9f0 net/socket.c:1985
__sys_sendmmsg+0x25c/0x750 net/socket.c:2075
SYSC_sendmmsg net/socket.c:2106 [inline]
SyS_sendmmsg+0x35/0x60 net/socket.c:2101
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
return_from_SYSCALL_64+0x0/0x7a
-> #0 (_xmit_ETHER#2){+.-...}:
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x33/0x50 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:299 [inline]
__netif_tx_lock include/linux/netdevice.h:3486 [inline]
sch_direct_xmit+0x282/0x6d0 net/sched/sch_generic.c:180
__dev_xmit_skb net/core/dev.c:3092 [inline]
__dev_queue_xmit+0x13e5/0x1e60 net/core/dev.c:3358
dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
neigh_hh_output include/net/neighbour.h:468 [inline]
neigh_output include/net/neighbour.h:476 [inline]
ip_finish_output2+0xf6c/0x15a0 net/ipv4/ip_output.c:228
ip_finish_output+0xa29/0xe10 net/ipv4/ip_output.c:316
NF_HOOK_COND include/linux/netfilter.h:246 [inline]
ip_output+0x1f0/0x7a0 net/ipv4/ip_output.c:404
dst_output include/net/dst.h:486 [inline]
ip_local_out+0x95/0x170 net/ipv4/ip_output.c:124
ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1492
ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1512
icmp_push_reply+0x372/0x4d0 net/ipv4/icmp.c:394
icmp_send+0x156c/0x1c80 net/ipv4/icmp.c:754
ip_expire+0x40e/0x6c0 net/ipv4/ip_fragment.c:239
call_timer_fn+0x241/0x820 kernel/time/timer.c:1268
expire_timers kernel/time/timer.c:1307 [inline]
__run_timers+0x960/0xcf0 kernel/time/timer.c:1601
run_timer_softirq+0x21/0x80 kernel/time/timer.c:1614
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
invoke_softirq kernel/softirq.c:364 [inline]
irq_exit+0x1cc/0x200 kernel/softirq.c:405
exiting_irq arch/x86/include/asm/apic.h:657 [inline]
smp_apic_timer_interrupt+0x76/0xa0 arch/x86/kernel/apic/apic.c:962
apic_timer_interrupt+0x93/0xa0 arch/x86/entry/entry_64.S:707
__read_once_size include/linux/compiler.h:254 [inline]
atomic_read arch/x86/include/asm/atomic.h:26 [inline]
rcu_dynticks_curr_cpu_in_eqs kernel/rcu/tree.c:350 [inline]
__rcu_is_watching kernel/rcu/tree.c:1133 [inline]
rcu_is_watching+0x83/0x110 kernel/rcu/tree.c:1147
rcu_read_lock_held+0x87/0xc0 kernel/rcu/update.c:293
radix_tree_deref_slot include/linux/radix-tree.h:238 [inline]
filemap_map_pages+0x6d4/0x1570 mm/filemap.c:2335
do_fault_around mm/memory.c:3231 [inline]
do_read_fault mm/memory.c:3265 [inline]
do_fault+0xbd5/0x2080 mm/memory.c:3370
handle_pte_fault mm/memory.c:3600 [inline]
__handle_mm_fault+0x1062/0x2cb0 mm/memory.c:3714
handle_mm_fault+0x1e2/0x480 mm/memory.c:3751
__do_page_fault+0x4f6/0xb60 arch/x86/mm/fault.c:1397
do_page_fault+0x54/0x70 arch/x86/mm/fault.c:1460
page_fault+0x28/0x30 arch/x86/entry/entry_64.S:1011
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&(&q->lock)->rlock);
lock(_xmit_ETHER#2);
lock(&(&q->lock)->rlock);
lock(_xmit_ETHER#2);
*** DEADLOCK ***
10 locks held by modprobe/12392:
#0: (&mm->mmap_sem){++++++}, at: [<ffffffff81329758>]
__do_page_fault+0x2b8/0xb60 arch/x86/mm/fault.c:1336
#1: (rcu_read_lock){......}, at: [<ffffffff8188cab6>]
filemap_map_pages+0x1e6/0x1570 mm/filemap.c:2324
#2: (&(ptlock_ptr(page))->rlock#2){+.+...}, at: [<ffffffff81984a78>]
spin_lock include/linux/spinlock.h:299 [inline]
#2: (&(ptlock_ptr(page))->rlock#2){+.+...}, at: [<ffffffff81984a78>]
pte_alloc_one_map mm/memory.c:2944 [inline]
#2: (&(ptlock_ptr(page))->rlock#2){+.+...}, at: [<ffffffff81984a78>]
alloc_set_pte+0x13b8/0x1b90 mm/memory.c:3072
#3: (((&q->timer))){+.-...}, at: [<ffffffff81627e72>]
lockdep_copy_map include/linux/lockdep.h:175 [inline]
#3: (((&q->timer))){+.-...}, at: [<ffffffff81627e72>]
call_timer_fn+0x1c2/0x820 kernel/time/timer.c:1258
#4: (&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>] spin_lock
include/linux/spinlock.h:299 [inline]
#4: (&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>]
ip_expire+0x51/0x6c0 net/ipv4/ip_fragment.c:201
#5: (rcu_read_lock){......}, at: [<ffffffff8389a633>]
ip_expire+0x1b3/0x6c0 net/ipv4/ip_fragment.c:216
#6: (slock-AF_INET){+.-...}, at: [<ffffffff839b3313>] spin_trylock
include/linux/spinlock.h:309 [inline]
#6: (slock-AF_INET){+.-...}, at: [<ffffffff839b3313>] icmp_xmit_lock
net/ipv4/icmp.c:219 [inline]
#6: (slock-AF_INET){+.-...}, at: [<ffffffff839b3313>]
icmp_send+0x803/0x1c80 net/ipv4/icmp.c:681
#7: (rcu_read_lock_bh){......}, at: [<ffffffff838ab9a1>]
ip_finish_output2+0x2c1/0x15a0 net/ipv4/ip_output.c:198
#8: (rcu_read_lock_bh){......}, at: [<ffffffff836d1dee>]
__dev_queue_xmit+0x23e/0x1e60 net/core/dev.c:3324
#9: (dev->qdisc_running_key ?: &qdisc_running_key){+.....}, at:
[<ffffffff836d3a27>] dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
stack backtrace:
CPU: 0 PID: 12392 Comm: modprobe Not tainted 4.10.0+ #29
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:52
print_circular_bug+0x307/0x3b0 kernel/locking/lockdep.c:1204
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x33/0x50 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:299 [inline]
__netif_tx_lock include/linux/netdevice.h:3486 [inline]
sch_direct_xmit+0x282/0x6d0 net/sched/sch_generic.c:180
__dev_xmit_skb net/core/dev.c:3092 [inline]
__dev_queue_xmit+0x13e5/0x1e60 net/core/dev.c:3358
dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
neigh_hh_output include/net/neighbour.h:468 [inline]
neigh_output include/net/neighbour.h:476 [inline]
ip_finish_output2+0xf6c/0x15a0 net/ipv4/ip_output.c:228
ip_finish_output+0xa29/0xe10 net/ipv4/ip_output.c:316
NF_HOOK_COND include/linux/netfilter.h:246 [inline]
ip_output+0x1f0/0x7a0 net/ipv4/ip_output.c:404
dst_output include/net/dst.h:486 [inline]
ip_local_out+0x95/0x170 net/ipv4/ip_output.c:124
ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1492
ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1512
icmp_push_reply+0x372/0x4d0 net/ipv4/icmp.c:394
icmp_send+0x156c/0x1c80 net/ipv4/icmp.c:754
ip_expire+0x40e/0x6c0 net/ipv4/ip_fragment.c:239
call_timer_fn+0x241/0x820 kernel/time/timer.c:1268
expire_timers kernel/time/timer.c:1307 [inline]
__run_timers+0x960/0xcf0 kernel/time/timer.c:1601
run_timer_softirq+0x21/0x80 kernel/time/timer.c:1614
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
invoke_softirq kernel/softirq.c:364 [inline]
irq_exit+0x1cc/0x200 kernel/softirq.c:405
exiting_irq arch/x86/include/asm/apic.h:657 [inline]
smp_apic_timer_interrupt+0x76/0xa0 arch/x86/kernel/apic/apic.c:962
apic_timer_interrupt+0x93/0xa0 arch/x86/entry/entry_64.S:707
RIP: 0010:__read_once_size include/linux/compiler.h:254 [inline]
RIP: 0010:atomic_read arch/x86/include/asm/atomic.h:26 [inline]
RIP: 0010:rcu_dynticks_curr_cpu_in_eqs kernel/rcu/tree.c:350 [inline]
RIP: 0010:__rcu_is_watching kernel/rcu/tree.c:1133 [inline]
RIP: 0010:rcu_is_watching+0x83/0x110 kernel/rcu/tree.c:1147
RSP: 0000:ffff8801c391f120 EFLAGS: 00000a03 ORIG_RAX: ffffffffffffff10
RAX: dffffc0000000000 RBX: ffff8801c391f148 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000055edd4374000 RDI: ffff8801dbe1ae0c
RBP: ffff8801c391f1a0 R08: 0000000000000002 R09: 0000000000000000
R10: dffffc0000000000 R11: 0000000000000002 R12: 1ffff10038723e25
R13: ffff8801dbe1ae00 R14: ffff8801c391f680 R15: dffffc0000000000
</IRQ>
rcu_read_lock_held+0x87/0xc0 kernel/rcu/update.c:293
radix_tree_deref_slot include/linux/radix-tree.h:238 [inline]
filemap_map_pages+0x6d4/0x1570 mm/filemap.c:2335
do_fault_around mm/memory.c:3231 [inline]
do_read_fault mm/memory.c:3265 [inline]
do_fault+0xbd5/0x2080 mm/memory.c:3370
handle_pte_fault mm/memory.c:3600 [inline]
__handle_mm_fault+0x1062/0x2cb0 mm/memory.c:3714
handle_mm_fault+0x1e2/0x480 mm/memory.c:3751
__do_page_fault+0x4f6/0xb60 arch/x86/mm/fault.c:1397
do_page_fault+0x54/0x70 arch/x86/mm/fault.c:1460
page_fault+0x28/0x30 arch/x86/entry/entry_64.S:1011
RIP: 0033:0x7f83172f2786
RSP: 002b:00007fffe859ae80 EFLAGS: 00010293
RAX: 000055edd4373040 RBX: 00007f83175111c8 RCX: 000055edd4373238
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007f8317510970
RBP: 00007fffe859afd0 R08: 0000000000000009 R09: 0000000000000000
R10: 0000000000000064 R11: 0000000000000000 R12: 000055edd4373040
R13: 0000000000000000 R14: 00007fffe859afe8 R15: 0000000000000000
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Dmitry Vyukov <dvyukov@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-03-22 15:57:15 +00:00
|
|
|
rcu_read_lock();
|
2007-10-15 09:24:19 +00:00
|
|
|
spin_lock(&qp->q.lock);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2014-08-01 10:29:44 +00:00
|
|
|
if (qp->q.flags & INET_FRAG_COMPLETE)
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
ipq_kill(qp);
|
2016-04-27 23:44:35 +00:00
|
|
|
__IP_INC_STATS(net, IPSTATS_MIB_REASMFAILS);
|
2018-03-31 19:58:51 +00:00
|
|
|
__IP_INC_STATS(net, IPSTATS_MIB_REASMTIMEOUT);
|
2014-08-01 10:29:47 +00:00
|
|
|
|
2018-08-06 19:17:35 +00:00
|
|
|
if (!(qp->q.flags & INET_FRAG_FIRST_IN))
|
2018-03-31 19:58:51 +00:00
|
|
|
goto out;
|
2014-08-01 10:29:47 +00:00
|
|
|
|
2018-08-02 23:34:39 +00:00
|
|
|
/* sk_buff::dev and sk_buff::rbnode are unionized. So we
|
|
|
|
* pull the head out of the tree in order to be able to
|
|
|
|
* deal with head->dev.
|
|
|
|
*/
|
|
|
|
if (qp->q.fragments) {
|
|
|
|
head = qp->q.fragments;
|
|
|
|
qp->q.fragments = head->next;
|
|
|
|
} else {
|
|
|
|
head = skb_rb_first(&qp->q.rb_fragments);
|
|
|
|
if (!head)
|
|
|
|
goto out;
|
2018-08-11 20:27:25 +00:00
|
|
|
if (FRAG_CB(head)->next_frag)
|
|
|
|
rb_replace_node(&head->rbnode,
|
|
|
|
&FRAG_CB(head)->next_frag->rbnode,
|
|
|
|
&qp->q.rb_fragments);
|
|
|
|
else
|
|
|
|
rb_erase(&head->rbnode, &qp->q.rb_fragments);
|
2018-08-02 23:34:39 +00:00
|
|
|
memset(&head->rbnode, 0, sizeof(head->rbnode));
|
|
|
|
barrier();
|
|
|
|
}
|
|
|
|
if (head == qp->q.fragments_tail)
|
|
|
|
qp->q.fragments_tail = NULL;
|
|
|
|
|
|
|
|
sub_frag_mem_limit(qp->q.net, head->truesize);
|
|
|
|
|
2018-03-31 19:58:51 +00:00
|
|
|
head->dev = dev_get_by_index_rcu(net, qp->iif);
|
|
|
|
if (!head->dev)
|
|
|
|
goto out;
|
inet: frag: release spinlock before calling icmp_send()
Dmitry reported a lockdep splat [1] (false positive) that we can fix
by releasing the spinlock before calling icmp_send() from ip_expire()
This is a false positive because sending an ICMP message can not
possibly re-enter the IP frag engine.
[1]
[ INFO: possible circular locking dependency detected ]
4.10.0+ #29 Not tainted
-------------------------------------------------------
modprobe/12392 is trying to acquire lock:
(_xmit_ETHER#2){+.-...}, at: [<ffffffff837a8182>] spin_lock
include/linux/spinlock.h:299 [inline]
(_xmit_ETHER#2){+.-...}, at: [<ffffffff837a8182>] __netif_tx_lock
include/linux/netdevice.h:3486 [inline]
(_xmit_ETHER#2){+.-...}, at: [<ffffffff837a8182>]
sch_direct_xmit+0x282/0x6d0 net/sched/sch_generic.c:180
but task is already holding lock:
(&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>] spin_lock
include/linux/spinlock.h:299 [inline]
(&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>]
ip_expire+0x51/0x6c0 net/ipv4/ip_fragment.c:201
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&(&q->lock)->rlock){+.-...}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x33/0x50 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:299 [inline]
ip_defrag+0x3a2/0x4130 net/ipv4/ip_fragment.c:669
ip_check_defrag+0x4e3/0x8b0 net/ipv4/ip_fragment.c:713
packet_rcv_fanout+0x282/0x800 net/packet/af_packet.c:1459
deliver_skb net/core/dev.c:1834 [inline]
dev_queue_xmit_nit+0x294/0xa90 net/core/dev.c:1890
xmit_one net/core/dev.c:2903 [inline]
dev_hard_start_xmit+0x16b/0xab0 net/core/dev.c:2923
sch_direct_xmit+0x31f/0x6d0 net/sched/sch_generic.c:182
__dev_xmit_skb net/core/dev.c:3092 [inline]
__dev_queue_xmit+0x13e5/0x1e60 net/core/dev.c:3358
dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
neigh_resolve_output+0x6b9/0xb10 net/core/neighbour.c:1308
neigh_output include/net/neighbour.h:478 [inline]
ip_finish_output2+0x8b8/0x15a0 net/ipv4/ip_output.c:228
ip_do_fragment+0x1d93/0x2720 net/ipv4/ip_output.c:672
ip_fragment.constprop.54+0x145/0x200 net/ipv4/ip_output.c:545
ip_finish_output+0x82d/0xe10 net/ipv4/ip_output.c:314
NF_HOOK_COND include/linux/netfilter.h:246 [inline]
ip_output+0x1f0/0x7a0 net/ipv4/ip_output.c:404
dst_output include/net/dst.h:486 [inline]
ip_local_out+0x95/0x170 net/ipv4/ip_output.c:124
ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1492
ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1512
raw_sendmsg+0x26de/0x3a00 net/ipv4/raw.c:655
inet_sendmsg+0x164/0x5b0 net/ipv4/af_inet.c:761
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
___sys_sendmsg+0x4a3/0x9f0 net/socket.c:1985
__sys_sendmmsg+0x25c/0x750 net/socket.c:2075
SYSC_sendmmsg net/socket.c:2106 [inline]
SyS_sendmmsg+0x35/0x60 net/socket.c:2101
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
return_from_SYSCALL_64+0x0/0x7a
-> #0 (_xmit_ETHER#2){+.-...}:
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x33/0x50 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:299 [inline]
__netif_tx_lock include/linux/netdevice.h:3486 [inline]
sch_direct_xmit+0x282/0x6d0 net/sched/sch_generic.c:180
__dev_xmit_skb net/core/dev.c:3092 [inline]
__dev_queue_xmit+0x13e5/0x1e60 net/core/dev.c:3358
dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
neigh_hh_output include/net/neighbour.h:468 [inline]
neigh_output include/net/neighbour.h:476 [inline]
ip_finish_output2+0xf6c/0x15a0 net/ipv4/ip_output.c:228
ip_finish_output+0xa29/0xe10 net/ipv4/ip_output.c:316
NF_HOOK_COND include/linux/netfilter.h:246 [inline]
ip_output+0x1f0/0x7a0 net/ipv4/ip_output.c:404
dst_output include/net/dst.h:486 [inline]
ip_local_out+0x95/0x170 net/ipv4/ip_output.c:124
ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1492
ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1512
icmp_push_reply+0x372/0x4d0 net/ipv4/icmp.c:394
icmp_send+0x156c/0x1c80 net/ipv4/icmp.c:754
ip_expire+0x40e/0x6c0 net/ipv4/ip_fragment.c:239
call_timer_fn+0x241/0x820 kernel/time/timer.c:1268
expire_timers kernel/time/timer.c:1307 [inline]
__run_timers+0x960/0xcf0 kernel/time/timer.c:1601
run_timer_softirq+0x21/0x80 kernel/time/timer.c:1614
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
invoke_softirq kernel/softirq.c:364 [inline]
irq_exit+0x1cc/0x200 kernel/softirq.c:405
exiting_irq arch/x86/include/asm/apic.h:657 [inline]
smp_apic_timer_interrupt+0x76/0xa0 arch/x86/kernel/apic/apic.c:962
apic_timer_interrupt+0x93/0xa0 arch/x86/entry/entry_64.S:707
__read_once_size include/linux/compiler.h:254 [inline]
atomic_read arch/x86/include/asm/atomic.h:26 [inline]
rcu_dynticks_curr_cpu_in_eqs kernel/rcu/tree.c:350 [inline]
__rcu_is_watching kernel/rcu/tree.c:1133 [inline]
rcu_is_watching+0x83/0x110 kernel/rcu/tree.c:1147
rcu_read_lock_held+0x87/0xc0 kernel/rcu/update.c:293
radix_tree_deref_slot include/linux/radix-tree.h:238 [inline]
filemap_map_pages+0x6d4/0x1570 mm/filemap.c:2335
do_fault_around mm/memory.c:3231 [inline]
do_read_fault mm/memory.c:3265 [inline]
do_fault+0xbd5/0x2080 mm/memory.c:3370
handle_pte_fault mm/memory.c:3600 [inline]
__handle_mm_fault+0x1062/0x2cb0 mm/memory.c:3714
handle_mm_fault+0x1e2/0x480 mm/memory.c:3751
__do_page_fault+0x4f6/0xb60 arch/x86/mm/fault.c:1397
do_page_fault+0x54/0x70 arch/x86/mm/fault.c:1460
page_fault+0x28/0x30 arch/x86/entry/entry_64.S:1011
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&(&q->lock)->rlock);
lock(_xmit_ETHER#2);
lock(&(&q->lock)->rlock);
lock(_xmit_ETHER#2);
*** DEADLOCK ***
10 locks held by modprobe/12392:
#0: (&mm->mmap_sem){++++++}, at: [<ffffffff81329758>]
__do_page_fault+0x2b8/0xb60 arch/x86/mm/fault.c:1336
#1: (rcu_read_lock){......}, at: [<ffffffff8188cab6>]
filemap_map_pages+0x1e6/0x1570 mm/filemap.c:2324
#2: (&(ptlock_ptr(page))->rlock#2){+.+...}, at: [<ffffffff81984a78>]
spin_lock include/linux/spinlock.h:299 [inline]
#2: (&(ptlock_ptr(page))->rlock#2){+.+...}, at: [<ffffffff81984a78>]
pte_alloc_one_map mm/memory.c:2944 [inline]
#2: (&(ptlock_ptr(page))->rlock#2){+.+...}, at: [<ffffffff81984a78>]
alloc_set_pte+0x13b8/0x1b90 mm/memory.c:3072
#3: (((&q->timer))){+.-...}, at: [<ffffffff81627e72>]
lockdep_copy_map include/linux/lockdep.h:175 [inline]
#3: (((&q->timer))){+.-...}, at: [<ffffffff81627e72>]
call_timer_fn+0x1c2/0x820 kernel/time/timer.c:1258
#4: (&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>] spin_lock
include/linux/spinlock.h:299 [inline]
#4: (&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>]
ip_expire+0x51/0x6c0 net/ipv4/ip_fragment.c:201
#5: (rcu_read_lock){......}, at: [<ffffffff8389a633>]
ip_expire+0x1b3/0x6c0 net/ipv4/ip_fragment.c:216
#6: (slock-AF_INET){+.-...}, at: [<ffffffff839b3313>] spin_trylock
include/linux/spinlock.h:309 [inline]
#6: (slock-AF_INET){+.-...}, at: [<ffffffff839b3313>] icmp_xmit_lock
net/ipv4/icmp.c:219 [inline]
#6: (slock-AF_INET){+.-...}, at: [<ffffffff839b3313>]
icmp_send+0x803/0x1c80 net/ipv4/icmp.c:681
#7: (rcu_read_lock_bh){......}, at: [<ffffffff838ab9a1>]
ip_finish_output2+0x2c1/0x15a0 net/ipv4/ip_output.c:198
#8: (rcu_read_lock_bh){......}, at: [<ffffffff836d1dee>]
__dev_queue_xmit+0x23e/0x1e60 net/core/dev.c:3324
#9: (dev->qdisc_running_key ?: &qdisc_running_key){+.....}, at:
[<ffffffff836d3a27>] dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
stack backtrace:
CPU: 0 PID: 12392 Comm: modprobe Not tainted 4.10.0+ #29
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:52
print_circular_bug+0x307/0x3b0 kernel/locking/lockdep.c:1204
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x33/0x50 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:299 [inline]
__netif_tx_lock include/linux/netdevice.h:3486 [inline]
sch_direct_xmit+0x282/0x6d0 net/sched/sch_generic.c:180
__dev_xmit_skb net/core/dev.c:3092 [inline]
__dev_queue_xmit+0x13e5/0x1e60 net/core/dev.c:3358
dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
neigh_hh_output include/net/neighbour.h:468 [inline]
neigh_output include/net/neighbour.h:476 [inline]
ip_finish_output2+0xf6c/0x15a0 net/ipv4/ip_output.c:228
ip_finish_output+0xa29/0xe10 net/ipv4/ip_output.c:316
NF_HOOK_COND include/linux/netfilter.h:246 [inline]
ip_output+0x1f0/0x7a0 net/ipv4/ip_output.c:404
dst_output include/net/dst.h:486 [inline]
ip_local_out+0x95/0x170 net/ipv4/ip_output.c:124
ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1492
ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1512
icmp_push_reply+0x372/0x4d0 net/ipv4/icmp.c:394
icmp_send+0x156c/0x1c80 net/ipv4/icmp.c:754
ip_expire+0x40e/0x6c0 net/ipv4/ip_fragment.c:239
call_timer_fn+0x241/0x820 kernel/time/timer.c:1268
expire_timers kernel/time/timer.c:1307 [inline]
__run_timers+0x960/0xcf0 kernel/time/timer.c:1601
run_timer_softirq+0x21/0x80 kernel/time/timer.c:1614
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
invoke_softirq kernel/softirq.c:364 [inline]
irq_exit+0x1cc/0x200 kernel/softirq.c:405
exiting_irq arch/x86/include/asm/apic.h:657 [inline]
smp_apic_timer_interrupt+0x76/0xa0 arch/x86/kernel/apic/apic.c:962
apic_timer_interrupt+0x93/0xa0 arch/x86/entry/entry_64.S:707
RIP: 0010:__read_once_size include/linux/compiler.h:254 [inline]
RIP: 0010:atomic_read arch/x86/include/asm/atomic.h:26 [inline]
RIP: 0010:rcu_dynticks_curr_cpu_in_eqs kernel/rcu/tree.c:350 [inline]
RIP: 0010:__rcu_is_watching kernel/rcu/tree.c:1133 [inline]
RIP: 0010:rcu_is_watching+0x83/0x110 kernel/rcu/tree.c:1147
RSP: 0000:ffff8801c391f120 EFLAGS: 00000a03 ORIG_RAX: ffffffffffffff10
RAX: dffffc0000000000 RBX: ffff8801c391f148 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000055edd4374000 RDI: ffff8801dbe1ae0c
RBP: ffff8801c391f1a0 R08: 0000000000000002 R09: 0000000000000000
R10: dffffc0000000000 R11: 0000000000000002 R12: 1ffff10038723e25
R13: ffff8801dbe1ae00 R14: ffff8801c391f680 R15: dffffc0000000000
</IRQ>
rcu_read_lock_held+0x87/0xc0 kernel/rcu/update.c:293
radix_tree_deref_slot include/linux/radix-tree.h:238 [inline]
filemap_map_pages+0x6d4/0x1570 mm/filemap.c:2335
do_fault_around mm/memory.c:3231 [inline]
do_read_fault mm/memory.c:3265 [inline]
do_fault+0xbd5/0x2080 mm/memory.c:3370
handle_pte_fault mm/memory.c:3600 [inline]
__handle_mm_fault+0x1062/0x2cb0 mm/memory.c:3714
handle_mm_fault+0x1e2/0x480 mm/memory.c:3751
__do_page_fault+0x4f6/0xb60 arch/x86/mm/fault.c:1397
do_page_fault+0x54/0x70 arch/x86/mm/fault.c:1460
page_fault+0x28/0x30 arch/x86/entry/entry_64.S:1011
RIP: 0033:0x7f83172f2786
RSP: 002b:00007fffe859ae80 EFLAGS: 00010293
RAX: 000055edd4373040 RBX: 00007f83175111c8 RCX: 000055edd4373238
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007f8317510970
RBP: 00007fffe859afd0 R08: 0000000000000009 R09: 0000000000000000
R10: 0000000000000064 R11: 0000000000000000 R12: 000055edd4373040
R13: 0000000000000000 R14: 00007fffe859afe8 R15: 0000000000000000
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Dmitry Vyukov <dvyukov@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-03-22 15:57:15 +00:00
|
|
|
|
2010-01-23 09:57:42 +00:00
|
|
|
|
2018-03-31 19:58:51 +00:00
|
|
|
/* skb has no dst, perform route lookup again */
|
|
|
|
iph = ip_hdr(head);
|
|
|
|
err = ip_route_input_noref(head, iph->daddr, iph->saddr,
|
2012-07-26 11:14:38 +00:00
|
|
|
iph->tos, head->dev);
|
2018-03-31 19:58:51 +00:00
|
|
|
if (err)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
/* Only an end host needs to send an ICMP
|
|
|
|
* "Fragment Reassembly Timeout" message, per RFC792.
|
|
|
|
*/
|
|
|
|
if (frag_expire_skip_icmp(qp->q.key.v4.user) &&
|
|
|
|
(skb_rtable(head)->rt_type != RTN_LOCAL))
|
|
|
|
goto out;
|
|
|
|
|
2018-03-31 19:58:54 +00:00
|
|
|
spin_unlock(&qp->q.lock);
|
|
|
|
icmp_send(head, ICMP_TIME_EXCEEDED, ICMP_EXC_FRAGTIME, 0);
|
|
|
|
goto out_rcu_unlock;
|
2018-03-31 19:58:51 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
out:
|
2007-10-15 09:24:19 +00:00
|
|
|
spin_unlock(&qp->q.lock);
|
inet: frag: release spinlock before calling icmp_send()
Dmitry reported a lockdep splat [1] (false positive) that we can fix
by releasing the spinlock before calling icmp_send() from ip_expire()
This is a false positive because sending an ICMP message can not
possibly re-enter the IP frag engine.
[1]
[ INFO: possible circular locking dependency detected ]
4.10.0+ #29 Not tainted
-------------------------------------------------------
modprobe/12392 is trying to acquire lock:
(_xmit_ETHER#2){+.-...}, at: [<ffffffff837a8182>] spin_lock
include/linux/spinlock.h:299 [inline]
(_xmit_ETHER#2){+.-...}, at: [<ffffffff837a8182>] __netif_tx_lock
include/linux/netdevice.h:3486 [inline]
(_xmit_ETHER#2){+.-...}, at: [<ffffffff837a8182>]
sch_direct_xmit+0x282/0x6d0 net/sched/sch_generic.c:180
but task is already holding lock:
(&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>] spin_lock
include/linux/spinlock.h:299 [inline]
(&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>]
ip_expire+0x51/0x6c0 net/ipv4/ip_fragment.c:201
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&(&q->lock)->rlock){+.-...}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x33/0x50 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:299 [inline]
ip_defrag+0x3a2/0x4130 net/ipv4/ip_fragment.c:669
ip_check_defrag+0x4e3/0x8b0 net/ipv4/ip_fragment.c:713
packet_rcv_fanout+0x282/0x800 net/packet/af_packet.c:1459
deliver_skb net/core/dev.c:1834 [inline]
dev_queue_xmit_nit+0x294/0xa90 net/core/dev.c:1890
xmit_one net/core/dev.c:2903 [inline]
dev_hard_start_xmit+0x16b/0xab0 net/core/dev.c:2923
sch_direct_xmit+0x31f/0x6d0 net/sched/sch_generic.c:182
__dev_xmit_skb net/core/dev.c:3092 [inline]
__dev_queue_xmit+0x13e5/0x1e60 net/core/dev.c:3358
dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
neigh_resolve_output+0x6b9/0xb10 net/core/neighbour.c:1308
neigh_output include/net/neighbour.h:478 [inline]
ip_finish_output2+0x8b8/0x15a0 net/ipv4/ip_output.c:228
ip_do_fragment+0x1d93/0x2720 net/ipv4/ip_output.c:672
ip_fragment.constprop.54+0x145/0x200 net/ipv4/ip_output.c:545
ip_finish_output+0x82d/0xe10 net/ipv4/ip_output.c:314
NF_HOOK_COND include/linux/netfilter.h:246 [inline]
ip_output+0x1f0/0x7a0 net/ipv4/ip_output.c:404
dst_output include/net/dst.h:486 [inline]
ip_local_out+0x95/0x170 net/ipv4/ip_output.c:124
ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1492
ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1512
raw_sendmsg+0x26de/0x3a00 net/ipv4/raw.c:655
inet_sendmsg+0x164/0x5b0 net/ipv4/af_inet.c:761
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
___sys_sendmsg+0x4a3/0x9f0 net/socket.c:1985
__sys_sendmmsg+0x25c/0x750 net/socket.c:2075
SYSC_sendmmsg net/socket.c:2106 [inline]
SyS_sendmmsg+0x35/0x60 net/socket.c:2101
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
return_from_SYSCALL_64+0x0/0x7a
-> #0 (_xmit_ETHER#2){+.-...}:
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x33/0x50 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:299 [inline]
__netif_tx_lock include/linux/netdevice.h:3486 [inline]
sch_direct_xmit+0x282/0x6d0 net/sched/sch_generic.c:180
__dev_xmit_skb net/core/dev.c:3092 [inline]
__dev_queue_xmit+0x13e5/0x1e60 net/core/dev.c:3358
dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
neigh_hh_output include/net/neighbour.h:468 [inline]
neigh_output include/net/neighbour.h:476 [inline]
ip_finish_output2+0xf6c/0x15a0 net/ipv4/ip_output.c:228
ip_finish_output+0xa29/0xe10 net/ipv4/ip_output.c:316
NF_HOOK_COND include/linux/netfilter.h:246 [inline]
ip_output+0x1f0/0x7a0 net/ipv4/ip_output.c:404
dst_output include/net/dst.h:486 [inline]
ip_local_out+0x95/0x170 net/ipv4/ip_output.c:124
ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1492
ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1512
icmp_push_reply+0x372/0x4d0 net/ipv4/icmp.c:394
icmp_send+0x156c/0x1c80 net/ipv4/icmp.c:754
ip_expire+0x40e/0x6c0 net/ipv4/ip_fragment.c:239
call_timer_fn+0x241/0x820 kernel/time/timer.c:1268
expire_timers kernel/time/timer.c:1307 [inline]
__run_timers+0x960/0xcf0 kernel/time/timer.c:1601
run_timer_softirq+0x21/0x80 kernel/time/timer.c:1614
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
invoke_softirq kernel/softirq.c:364 [inline]
irq_exit+0x1cc/0x200 kernel/softirq.c:405
exiting_irq arch/x86/include/asm/apic.h:657 [inline]
smp_apic_timer_interrupt+0x76/0xa0 arch/x86/kernel/apic/apic.c:962
apic_timer_interrupt+0x93/0xa0 arch/x86/entry/entry_64.S:707
__read_once_size include/linux/compiler.h:254 [inline]
atomic_read arch/x86/include/asm/atomic.h:26 [inline]
rcu_dynticks_curr_cpu_in_eqs kernel/rcu/tree.c:350 [inline]
__rcu_is_watching kernel/rcu/tree.c:1133 [inline]
rcu_is_watching+0x83/0x110 kernel/rcu/tree.c:1147
rcu_read_lock_held+0x87/0xc0 kernel/rcu/update.c:293
radix_tree_deref_slot include/linux/radix-tree.h:238 [inline]
filemap_map_pages+0x6d4/0x1570 mm/filemap.c:2335
do_fault_around mm/memory.c:3231 [inline]
do_read_fault mm/memory.c:3265 [inline]
do_fault+0xbd5/0x2080 mm/memory.c:3370
handle_pte_fault mm/memory.c:3600 [inline]
__handle_mm_fault+0x1062/0x2cb0 mm/memory.c:3714
handle_mm_fault+0x1e2/0x480 mm/memory.c:3751
__do_page_fault+0x4f6/0xb60 arch/x86/mm/fault.c:1397
do_page_fault+0x54/0x70 arch/x86/mm/fault.c:1460
page_fault+0x28/0x30 arch/x86/entry/entry_64.S:1011
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&(&q->lock)->rlock);
lock(_xmit_ETHER#2);
lock(&(&q->lock)->rlock);
lock(_xmit_ETHER#2);
*** DEADLOCK ***
10 locks held by modprobe/12392:
#0: (&mm->mmap_sem){++++++}, at: [<ffffffff81329758>]
__do_page_fault+0x2b8/0xb60 arch/x86/mm/fault.c:1336
#1: (rcu_read_lock){......}, at: [<ffffffff8188cab6>]
filemap_map_pages+0x1e6/0x1570 mm/filemap.c:2324
#2: (&(ptlock_ptr(page))->rlock#2){+.+...}, at: [<ffffffff81984a78>]
spin_lock include/linux/spinlock.h:299 [inline]
#2: (&(ptlock_ptr(page))->rlock#2){+.+...}, at: [<ffffffff81984a78>]
pte_alloc_one_map mm/memory.c:2944 [inline]
#2: (&(ptlock_ptr(page))->rlock#2){+.+...}, at: [<ffffffff81984a78>]
alloc_set_pte+0x13b8/0x1b90 mm/memory.c:3072
#3: (((&q->timer))){+.-...}, at: [<ffffffff81627e72>]
lockdep_copy_map include/linux/lockdep.h:175 [inline]
#3: (((&q->timer))){+.-...}, at: [<ffffffff81627e72>]
call_timer_fn+0x1c2/0x820 kernel/time/timer.c:1258
#4: (&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>] spin_lock
include/linux/spinlock.h:299 [inline]
#4: (&(&q->lock)->rlock){+.-...}, at: [<ffffffff8389a4d1>]
ip_expire+0x51/0x6c0 net/ipv4/ip_fragment.c:201
#5: (rcu_read_lock){......}, at: [<ffffffff8389a633>]
ip_expire+0x1b3/0x6c0 net/ipv4/ip_fragment.c:216
#6: (slock-AF_INET){+.-...}, at: [<ffffffff839b3313>] spin_trylock
include/linux/spinlock.h:309 [inline]
#6: (slock-AF_INET){+.-...}, at: [<ffffffff839b3313>] icmp_xmit_lock
net/ipv4/icmp.c:219 [inline]
#6: (slock-AF_INET){+.-...}, at: [<ffffffff839b3313>]
icmp_send+0x803/0x1c80 net/ipv4/icmp.c:681
#7: (rcu_read_lock_bh){......}, at: [<ffffffff838ab9a1>]
ip_finish_output2+0x2c1/0x15a0 net/ipv4/ip_output.c:198
#8: (rcu_read_lock_bh){......}, at: [<ffffffff836d1dee>]
__dev_queue_xmit+0x23e/0x1e60 net/core/dev.c:3324
#9: (dev->qdisc_running_key ?: &qdisc_running_key){+.....}, at:
[<ffffffff836d3a27>] dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
stack backtrace:
CPU: 0 PID: 12392 Comm: modprobe Not tainted 4.10.0+ #29
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:52
print_circular_bug+0x307/0x3b0 kernel/locking/lockdep.c:1204
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x33/0x50 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:299 [inline]
__netif_tx_lock include/linux/netdevice.h:3486 [inline]
sch_direct_xmit+0x282/0x6d0 net/sched/sch_generic.c:180
__dev_xmit_skb net/core/dev.c:3092 [inline]
__dev_queue_xmit+0x13e5/0x1e60 net/core/dev.c:3358
dev_queue_xmit+0x17/0x20 net/core/dev.c:3423
neigh_hh_output include/net/neighbour.h:468 [inline]
neigh_output include/net/neighbour.h:476 [inline]
ip_finish_output2+0xf6c/0x15a0 net/ipv4/ip_output.c:228
ip_finish_output+0xa29/0xe10 net/ipv4/ip_output.c:316
NF_HOOK_COND include/linux/netfilter.h:246 [inline]
ip_output+0x1f0/0x7a0 net/ipv4/ip_output.c:404
dst_output include/net/dst.h:486 [inline]
ip_local_out+0x95/0x170 net/ipv4/ip_output.c:124
ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1492
ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1512
icmp_push_reply+0x372/0x4d0 net/ipv4/icmp.c:394
icmp_send+0x156c/0x1c80 net/ipv4/icmp.c:754
ip_expire+0x40e/0x6c0 net/ipv4/ip_fragment.c:239
call_timer_fn+0x241/0x820 kernel/time/timer.c:1268
expire_timers kernel/time/timer.c:1307 [inline]
__run_timers+0x960/0xcf0 kernel/time/timer.c:1601
run_timer_softirq+0x21/0x80 kernel/time/timer.c:1614
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
invoke_softirq kernel/softirq.c:364 [inline]
irq_exit+0x1cc/0x200 kernel/softirq.c:405
exiting_irq arch/x86/include/asm/apic.h:657 [inline]
smp_apic_timer_interrupt+0x76/0xa0 arch/x86/kernel/apic/apic.c:962
apic_timer_interrupt+0x93/0xa0 arch/x86/entry/entry_64.S:707
RIP: 0010:__read_once_size include/linux/compiler.h:254 [inline]
RIP: 0010:atomic_read arch/x86/include/asm/atomic.h:26 [inline]
RIP: 0010:rcu_dynticks_curr_cpu_in_eqs kernel/rcu/tree.c:350 [inline]
RIP: 0010:__rcu_is_watching kernel/rcu/tree.c:1133 [inline]
RIP: 0010:rcu_is_watching+0x83/0x110 kernel/rcu/tree.c:1147
RSP: 0000:ffff8801c391f120 EFLAGS: 00000a03 ORIG_RAX: ffffffffffffff10
RAX: dffffc0000000000 RBX: ffff8801c391f148 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000055edd4374000 RDI: ffff8801dbe1ae0c
RBP: ffff8801c391f1a0 R08: 0000000000000002 R09: 0000000000000000
R10: dffffc0000000000 R11: 0000000000000002 R12: 1ffff10038723e25
R13: ffff8801dbe1ae00 R14: ffff8801c391f680 R15: dffffc0000000000
</IRQ>
rcu_read_lock_held+0x87/0xc0 kernel/rcu/update.c:293
radix_tree_deref_slot include/linux/radix-tree.h:238 [inline]
filemap_map_pages+0x6d4/0x1570 mm/filemap.c:2335
do_fault_around mm/memory.c:3231 [inline]
do_read_fault mm/memory.c:3265 [inline]
do_fault+0xbd5/0x2080 mm/memory.c:3370
handle_pte_fault mm/memory.c:3600 [inline]
__handle_mm_fault+0x1062/0x2cb0 mm/memory.c:3714
handle_mm_fault+0x1e2/0x480 mm/memory.c:3751
__do_page_fault+0x4f6/0xb60 arch/x86/mm/fault.c:1397
do_page_fault+0x54/0x70 arch/x86/mm/fault.c:1460
page_fault+0x28/0x30 arch/x86/entry/entry_64.S:1011
RIP: 0033:0x7f83172f2786
RSP: 002b:00007fffe859ae80 EFLAGS: 00010293
RAX: 000055edd4373040 RBX: 00007f83175111c8 RCX: 000055edd4373238
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007f8317510970
RBP: 00007fffe859afd0 R08: 0000000000000009 R09: 0000000000000000
R10: 0000000000000064 R11: 0000000000000000 R12: 000055edd4373040
R13: 0000000000000000 R14: 00007fffe859afe8 R15: 0000000000000000
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Dmitry Vyukov <dvyukov@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-03-22 15:57:15 +00:00
|
|
|
out_rcu_unlock:
|
|
|
|
rcu_read_unlock();
|
2018-08-02 23:34:39 +00:00
|
|
|
if (head)
|
|
|
|
kfree_skb(head);
|
2007-10-15 09:41:09 +00:00
|
|
|
ipq_put(qp);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-10-18 02:47:21 +00:00
|
|
|
/* Find the correct entry in the "incomplete datagrams" queue for
|
|
|
|
* this IP datagram, and create new one, if nothing is found.
|
|
|
|
*/
|
2015-08-13 20:59:09 +00:00
|
|
|
static struct ipq *ip_find(struct net *net, struct iphdr *iph,
|
|
|
|
u32 user, int vif)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
inet: frags: use rhashtables for reassembly units
Some applications still rely on IP fragmentation, and to be fair linux
reassembly unit is not working under any serious load.
It uses static hash tables of 1024 buckets, and up to 128 items per bucket (!!!)
A work queue is supposed to garbage collect items when host is under memory
pressure, and doing a hash rebuild, changing seed used in hash computations.
This work queue blocks softirqs for up to 25 ms when doing a hash rebuild,
occurring every 5 seconds if host is under fire.
Then there is the problem of sharing this hash table for all netns.
It is time to switch to rhashtables, and allocate one of them per netns
to speedup netns dismantle, since this is a critical metric these days.
Lookup is now using RCU. A followup patch will even remove
the refcount hold/release left from prior implementation and save
a couple of atomic operations.
Before this patch, 16 cpus (16 RX queue NIC) could not handle more
than 1 Mpps frags DDOS.
After the patch, I reach 9 Mpps without any tuning, and can use up to 2GB
of storage for the fragments (exact number depends on frags being evicted
after timeout)
$ grep FRAG /proc/net/sockstat
FRAG: inuse 1966916 memory 2140004608
A followup patch will change the limits for 64bit arches.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Florian Westphal <fw@strlen.de>
Cc: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Alexander Aring <alex.aring@gmail.com>
Cc: Stefan Schmidt <stefan@osg.samsung.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-03-31 19:58:49 +00:00
|
|
|
struct frag_v4_compare_key key = {
|
|
|
|
.saddr = iph->saddr,
|
|
|
|
.daddr = iph->daddr,
|
|
|
|
.user = user,
|
|
|
|
.vif = vif,
|
|
|
|
.id = iph->id,
|
|
|
|
.protocol = iph->protocol,
|
|
|
|
};
|
2007-10-18 02:46:47 +00:00
|
|
|
struct inet_frag_queue *q;
|
2008-06-28 03:06:08 +00:00
|
|
|
|
inet: frags: use rhashtables for reassembly units
Some applications still rely on IP fragmentation, and to be fair linux
reassembly unit is not working under any serious load.
It uses static hash tables of 1024 buckets, and up to 128 items per bucket (!!!)
A work queue is supposed to garbage collect items when host is under memory
pressure, and doing a hash rebuild, changing seed used in hash computations.
This work queue blocks softirqs for up to 25 ms when doing a hash rebuild,
occurring every 5 seconds if host is under fire.
Then there is the problem of sharing this hash table for all netns.
It is time to switch to rhashtables, and allocate one of them per netns
to speedup netns dismantle, since this is a critical metric these days.
Lookup is now using RCU. A followup patch will even remove
the refcount hold/release left from prior implementation and save
a couple of atomic operations.
Before this patch, 16 cpus (16 RX queue NIC) could not handle more
than 1 Mpps frags DDOS.
After the patch, I reach 9 Mpps without any tuning, and can use up to 2GB
of storage for the fragments (exact number depends on frags being evicted
after timeout)
$ grep FRAG /proc/net/sockstat
FRAG: inuse 1966916 memory 2140004608
A followup patch will change the limits for 64bit arches.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Florian Westphal <fw@strlen.de>
Cc: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Alexander Aring <alex.aring@gmail.com>
Cc: Stefan Schmidt <stefan@osg.samsung.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-03-31 19:58:49 +00:00
|
|
|
q = inet_frag_find(&net->ipv4.frags, &key);
|
2018-03-31 19:58:52 +00:00
|
|
|
if (!q)
|
2013-03-15 11:32:30 +00:00
|
|
|
return NULL;
|
2018-03-31 19:58:52 +00:00
|
|
|
|
2007-10-18 02:46:47 +00:00
|
|
|
return container_of(q, struct ipq, q);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2005-12-14 07:14:27 +00:00
|
|
|
/* Is the fragment too far ahead to be part of ipq? */
|
2014-11-04 19:44:04 +00:00
|
|
|
static int ip_frag_too_far(struct ipq *qp)
|
2005-12-14 07:14:27 +00:00
|
|
|
{
|
|
|
|
struct inet_peer *peer = qp->peer;
|
2016-02-15 10:11:31 +00:00
|
|
|
unsigned int max = qp->q.net->max_dist;
|
2005-12-14 07:14:27 +00:00
|
|
|
unsigned int start, end;
|
|
|
|
|
|
|
|
int rc;
|
|
|
|
|
|
|
|
if (!peer || !max)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
start = qp->rid;
|
|
|
|
end = atomic_inc_return(&peer->rid);
|
|
|
|
qp->rid = end;
|
|
|
|
|
2018-08-02 23:34:39 +00:00
|
|
|
rc = qp->q.fragments_tail && (end - start) > max;
|
2005-12-14 07:14:27 +00:00
|
|
|
|
|
|
|
if (rc) {
|
2008-07-17 03:20:11 +00:00
|
|
|
struct net *net;
|
|
|
|
|
|
|
|
net = container_of(qp->q.net, struct net, ipv4.frags);
|
2016-04-27 23:44:35 +00:00
|
|
|
__IP_INC_STATS(net, IPSTATS_MIB_REASMFAILS);
|
2005-12-14 07:14:27 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int ip_frag_reinit(struct ipq *qp)
|
|
|
|
{
|
2013-01-28 23:45:12 +00:00
|
|
|
unsigned int sum_truesize = 0;
|
2005-12-14 07:14:27 +00:00
|
|
|
|
2008-01-22 14:09:37 +00:00
|
|
|
if (!mod_timer(&qp->q.timer, jiffies + qp->q.net->timeout)) {
|
2017-06-30 10:08:07 +00:00
|
|
|
refcount_inc(&qp->q.refcnt);
|
2005-12-14 07:14:27 +00:00
|
|
|
return -ETIMEDOUT;
|
|
|
|
}
|
|
|
|
|
2018-08-11 20:27:25 +00:00
|
|
|
sum_truesize = inet_frag_rbtree_purge(&qp->q.rb_fragments);
|
2015-07-23 10:05:38 +00:00
|
|
|
sub_frag_mem_limit(qp->q.net, sum_truesize);
|
2005-12-14 07:14:27 +00:00
|
|
|
|
2014-08-01 10:29:44 +00:00
|
|
|
qp->q.flags = 0;
|
2007-10-15 09:24:19 +00:00
|
|
|
qp->q.len = 0;
|
|
|
|
qp->q.meat = 0;
|
|
|
|
qp->q.fragments = NULL;
|
2018-08-02 23:34:39 +00:00
|
|
|
qp->q.rb_fragments = RB_ROOT;
|
2010-06-29 04:39:37 +00:00
|
|
|
qp->q.fragments_tail = NULL;
|
2018-08-11 20:27:25 +00:00
|
|
|
qp->q.last_run_head = NULL;
|
2005-12-14 07:14:27 +00:00
|
|
|
qp->iif = 0;
|
2011-01-05 07:52:55 +00:00
|
|
|
qp->ecn = 0;
|
2005-12-14 07:14:27 +00:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Add new segment to existing queue. */
|
2007-10-14 07:38:15 +00:00
|
|
|
static int ip_frag_queue(struct ipq *qp, struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2018-08-02 23:34:37 +00:00
|
|
|
struct net *net = container_of(qp->q.net, struct net, ipv4.frags);
|
2018-08-02 23:34:39 +00:00
|
|
|
struct rb_node **rbn, *parent;
|
2018-08-11 20:27:25 +00:00
|
|
|
struct sk_buff *skb1, *prev_tail;
|
2007-10-14 07:38:15 +00:00
|
|
|
struct net_device *dev;
|
2015-05-22 14:32:51 +00:00
|
|
|
unsigned int fragsize;
|
2005-04-16 22:20:36 +00:00
|
|
|
int flags, offset;
|
|
|
|
int ihl, end;
|
2007-10-14 07:38:15 +00:00
|
|
|
int err = -ENOENT;
|
2011-01-05 07:52:55 +00:00
|
|
|
u8 ecn;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2014-08-01 10:29:44 +00:00
|
|
|
if (qp->q.flags & INET_FRAG_COMPLETE)
|
2005-04-16 22:20:36 +00:00
|
|
|
goto err;
|
|
|
|
|
2005-12-14 07:14:27 +00:00
|
|
|
if (!(IPCB(skb)->flags & IPSKB_FRAG_COMPLETE) &&
|
2007-10-14 07:38:15 +00:00
|
|
|
unlikely(ip_frag_too_far(qp)) &&
|
|
|
|
unlikely(err = ip_frag_reinit(qp))) {
|
2005-12-14 07:14:27 +00:00
|
|
|
ipq_kill(qp);
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
2011-01-05 07:52:55 +00:00
|
|
|
ecn = ip4_frag_ecn(ip_hdr(skb)->tos);
|
2007-04-21 05:47:35 +00:00
|
|
|
offset = ntohs(ip_hdr(skb)->frag_off);
|
2005-04-16 22:20:36 +00:00
|
|
|
flags = offset & ~IP_OFFSET;
|
|
|
|
offset &= IP_OFFSET;
|
|
|
|
offset <<= 3; /* offset is in 8-byte chunks */
|
2007-03-12 23:09:15 +00:00
|
|
|
ihl = ip_hdrlen(skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Determine the position of this fragment. */
|
2015-07-21 07:43:59 +00:00
|
|
|
end = offset + skb->len - skb_network_offset(skb) - ihl;
|
2007-10-14 07:38:15 +00:00
|
|
|
err = -EINVAL;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Is this the final fragment? */
|
|
|
|
if ((flags & IP_MF) == 0) {
|
|
|
|
/* If we already have some bits beyond end
|
2011-11-29 04:31:00 +00:00
|
|
|
* or have different end, the segment is corrupted.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2007-10-15 09:24:19 +00:00
|
|
|
if (end < qp->q.len ||
|
2014-08-01 10:29:44 +00:00
|
|
|
((qp->q.flags & INET_FRAG_LAST_IN) && end != qp->q.len))
|
2018-08-28 18:36:19 +00:00
|
|
|
goto discard_qp;
|
2014-08-01 10:29:44 +00:00
|
|
|
qp->q.flags |= INET_FRAG_LAST_IN;
|
2007-10-15 09:24:19 +00:00
|
|
|
qp->q.len = end;
|
2005-04-16 22:20:36 +00:00
|
|
|
} else {
|
|
|
|
if (end&7) {
|
|
|
|
end &= ~7;
|
|
|
|
if (skb->ip_summed != CHECKSUM_UNNECESSARY)
|
|
|
|
skb->ip_summed = CHECKSUM_NONE;
|
|
|
|
}
|
2007-10-15 09:24:19 +00:00
|
|
|
if (end > qp->q.len) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Some bits beyond end -> corruption. */
|
2014-08-01 10:29:44 +00:00
|
|
|
if (qp->q.flags & INET_FRAG_LAST_IN)
|
2018-08-28 18:36:19 +00:00
|
|
|
goto discard_qp;
|
2007-10-15 09:24:19 +00:00
|
|
|
qp->q.len = end;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
if (end == offset)
|
2018-08-28 18:36:19 +00:00
|
|
|
goto discard_qp;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-10-14 07:38:15 +00:00
|
|
|
err = -ENOMEM;
|
2015-07-21 07:43:59 +00:00
|
|
|
if (!pskb_pull(skb, skb_network_offset(skb) + ihl))
|
2018-08-28 18:36:19 +00:00
|
|
|
goto discard_qp;
|
2007-10-14 07:38:15 +00:00
|
|
|
|
|
|
|
err = pskb_trim_rcsum(skb, end - offset);
|
|
|
|
if (err)
|
2018-08-28 18:36:19 +00:00
|
|
|
goto discard_qp;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2018-08-02 23:34:39 +00:00
|
|
|
/* Note : skb->rbnode and skb->dev share the same location. */
|
|
|
|
dev = skb->dev;
|
|
|
|
/* Makes sure compiler wont do silly aliasing games */
|
|
|
|
barrier();
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2018-08-02 23:34:37 +00:00
|
|
|
/* RFC5722, Section 4, amended by Errata ID : 3089
|
|
|
|
* When reassembling an IPv6 datagram, if
|
|
|
|
* one or more its constituent fragments is determined to be an
|
|
|
|
* overlapping fragment, the entire datagram (and any constituent
|
|
|
|
* fragments) MUST be silently discarded.
|
|
|
|
*
|
2018-08-02 23:34:39 +00:00
|
|
|
* We do the same here for IPv4 (and increment an snmp counter).
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
|
2018-08-28 18:36:19 +00:00
|
|
|
err = -EINVAL;
|
2018-08-02 23:34:39 +00:00
|
|
|
/* Find out where to put this fragment. */
|
2018-08-11 20:27:25 +00:00
|
|
|
prev_tail = qp->q.fragments_tail;
|
|
|
|
if (!prev_tail)
|
|
|
|
ip4_frag_create_run(&qp->q, skb); /* First fragment. */
|
|
|
|
else if (prev_tail->ip_defrag_offset + prev_tail->len < end) {
|
|
|
|
/* This is the common case: skb goes to the end. */
|
2018-08-02 23:34:39 +00:00
|
|
|
/* Detect and discard overlaps. */
|
2018-08-11 20:27:25 +00:00
|
|
|
if (offset < prev_tail->ip_defrag_offset + prev_tail->len)
|
2018-08-28 18:36:19 +00:00
|
|
|
goto overlap;
|
2018-08-11 20:27:25 +00:00
|
|
|
if (offset == prev_tail->ip_defrag_offset + prev_tail->len)
|
|
|
|
ip4_frag_append_to_last_run(&qp->q, skb);
|
|
|
|
else
|
|
|
|
ip4_frag_create_run(&qp->q, skb);
|
2018-08-02 23:34:39 +00:00
|
|
|
} else {
|
2018-08-11 20:27:25 +00:00
|
|
|
/* Binary search. Note that skb can become the first fragment,
|
|
|
|
* but not the last (covered above).
|
|
|
|
*/
|
2018-08-02 23:34:39 +00:00
|
|
|
rbn = &qp->q.rb_fragments.rb_node;
|
|
|
|
do {
|
|
|
|
parent = *rbn;
|
|
|
|
skb1 = rb_to_skb(parent);
|
|
|
|
if (end <= skb1->ip_defrag_offset)
|
|
|
|
rbn = &parent->rb_left;
|
2018-08-11 20:27:25 +00:00
|
|
|
else if (offset >= skb1->ip_defrag_offset +
|
|
|
|
FRAG_CB(skb1)->frag_run_len)
|
2018-08-02 23:34:39 +00:00
|
|
|
rbn = &parent->rb_right;
|
|
|
|
else /* Found an overlap with skb1. */
|
2018-08-28 18:36:19 +00:00
|
|
|
goto overlap;
|
2018-08-02 23:34:39 +00:00
|
|
|
} while (*rbn);
|
|
|
|
/* Here we have parent properly set, and rbn pointing to
|
2018-08-11 20:27:25 +00:00
|
|
|
* one of its NULL left/right children. Insert skb.
|
|
|
|
*/
|
|
|
|
ip4_frag_init_run(skb);
|
2018-08-02 23:34:39 +00:00
|
|
|
rb_link_node(&skb->rbnode, parent, rbn);
|
2018-08-11 20:27:25 +00:00
|
|
|
rb_insert_color(&skb->rbnode, &qp->q.rb_fragments);
|
2018-08-02 23:34:39 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2018-03-31 19:58:58 +00:00
|
|
|
if (dev)
|
|
|
|
qp->iif = dev->ifindex;
|
|
|
|
skb->ip_defrag_offset = offset;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-10-15 09:24:19 +00:00
|
|
|
qp->q.stamp = skb->tstamp;
|
|
|
|
qp->q.meat += skb->len;
|
2011-01-05 07:52:55 +00:00
|
|
|
qp->ecn |= ecn;
|
2015-07-23 10:05:38 +00:00
|
|
|
add_frag_mem_limit(qp->q.net, skb->truesize);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (offset == 0)
|
2014-08-01 10:29:44 +00:00
|
|
|
qp->q.flags |= INET_FRAG_FIRST_IN;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2015-05-22 14:32:51 +00:00
|
|
|
fragsize = skb->len + ihl;
|
|
|
|
|
|
|
|
if (fragsize > qp->q.max_size)
|
|
|
|
qp->q.max_size = fragsize;
|
|
|
|
|
2012-08-26 17:13:55 +00:00
|
|
|
if (ip_hdr(skb)->frag_off & htons(IP_DF) &&
|
2015-05-22 14:32:51 +00:00
|
|
|
fragsize > qp->max_df_size)
|
|
|
|
qp->max_df_size = fragsize;
|
2012-08-26 17:13:55 +00:00
|
|
|
|
2014-08-01 10:29:44 +00:00
|
|
|
if (qp->q.flags == (INET_FRAG_FIRST_IN | INET_FRAG_LAST_IN) &&
|
2013-04-16 12:55:41 +00:00
|
|
|
qp->q.meat == qp->q.len) {
|
|
|
|
unsigned long orefdst = skb->_skb_refdst;
|
2007-10-14 07:38:15 +00:00
|
|
|
|
2013-04-16 12:55:41 +00:00
|
|
|
skb->_skb_refdst = 0UL;
|
2018-08-11 20:27:25 +00:00
|
|
|
err = ip_frag_reasm(qp, skb, prev_tail, dev);
|
2013-04-16 12:55:41 +00:00
|
|
|
skb->_skb_refdst = orefdst;
|
2018-08-28 18:36:19 +00:00
|
|
|
if (err)
|
|
|
|
inet_frag_kill(&qp->q);
|
2013-04-16 12:55:41 +00:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
skb_dst_drop(skb);
|
2007-10-14 07:38:15 +00:00
|
|
|
return -EINPROGRESS;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2018-08-28 18:36:19 +00:00
|
|
|
overlap:
|
|
|
|
__IP_INC_STATS(net, IPSTATS_MIB_REASM_OVERLAPS);
|
2018-08-02 23:34:37 +00:00
|
|
|
discard_qp:
|
|
|
|
inet_frag_kill(&qp->q);
|
2005-04-16 22:20:36 +00:00
|
|
|
err:
|
|
|
|
kfree_skb(skb);
|
2007-10-14 07:38:15 +00:00
|
|
|
return err;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Build a new IP datagram from all its fragments. */
|
2018-08-02 23:34:39 +00:00
|
|
|
static int ip_frag_reasm(struct ipq *qp, struct sk_buff *skb,
|
2018-08-11 20:27:25 +00:00
|
|
|
struct sk_buff *prev_tail, struct net_device *dev)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2009-03-19 06:26:11 +00:00
|
|
|
struct net *net = container_of(qp->q.net, struct net, ipv4.frags);
|
2005-04-16 22:20:36 +00:00
|
|
|
struct iphdr *iph;
|
2018-08-02 23:34:39 +00:00
|
|
|
struct sk_buff *fp, *head = skb_rb_first(&qp->q.rb_fragments);
|
|
|
|
struct sk_buff **nextp; /* To build frag_list. */
|
|
|
|
struct rb_node *rbn;
|
2005-04-16 22:20:36 +00:00
|
|
|
int len;
|
|
|
|
int ihlen;
|
2007-10-14 07:38:15 +00:00
|
|
|
int err;
|
2011-05-16 08:37:37 +00:00
|
|
|
u8 ecn;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
ipq_kill(qp);
|
|
|
|
|
2013-03-22 08:24:37 +00:00
|
|
|
ecn = ip_frag_ecn_table[qp->ecn];
|
2011-05-16 08:37:37 +00:00
|
|
|
if (unlikely(ecn == 0xff)) {
|
|
|
|
err = -EINVAL;
|
|
|
|
goto out_fail;
|
|
|
|
}
|
2007-10-14 07:38:15 +00:00
|
|
|
/* Make the one we just received the head. */
|
2018-08-02 23:34:39 +00:00
|
|
|
if (head != skb) {
|
|
|
|
fp = skb_clone(skb, GFP_ATOMIC);
|
2007-10-14 07:38:15 +00:00
|
|
|
if (!fp)
|
|
|
|
goto out_nomem;
|
2018-08-11 20:27:25 +00:00
|
|
|
FRAG_CB(fp)->next_frag = FRAG_CB(skb)->next_frag;
|
|
|
|
if (RB_EMPTY_NODE(&skb->rbnode))
|
|
|
|
FRAG_CB(prev_tail)->next_frag = fp;
|
|
|
|
else
|
|
|
|
rb_replace_node(&skb->rbnode, &fp->rbnode,
|
|
|
|
&qp->q.rb_fragments);
|
2018-08-02 23:34:39 +00:00
|
|
|
if (qp->q.fragments_tail == skb)
|
2010-06-29 04:39:37 +00:00
|
|
|
qp->q.fragments_tail = fp;
|
2018-08-02 23:34:39 +00:00
|
|
|
skb_morph(skb, head);
|
2018-08-11 20:27:25 +00:00
|
|
|
FRAG_CB(skb)->next_frag = FRAG_CB(head)->next_frag;
|
2018-08-02 23:34:39 +00:00
|
|
|
rb_replace_node(&head->rbnode, &skb->rbnode,
|
|
|
|
&qp->q.rb_fragments);
|
|
|
|
consume_skb(head);
|
|
|
|
head = skb;
|
2007-10-14 07:38:15 +00:00
|
|
|
}
|
|
|
|
|
2018-03-31 19:58:58 +00:00
|
|
|
WARN_ON(head->ip_defrag_offset != 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Allocate a new buffer for the datagram. */
|
2007-03-12 23:09:15 +00:00
|
|
|
ihlen = ip_hdrlen(head);
|
2007-10-15 09:24:19 +00:00
|
|
|
len = ihlen + qp->q.len;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-10-14 07:38:15 +00:00
|
|
|
err = -E2BIG;
|
2007-03-09 04:44:43 +00:00
|
|
|
if (len > 65535)
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out_oversize;
|
|
|
|
|
|
|
|
/* Head of list must not be cloned. */
|
2013-02-14 09:44:49 +00:00
|
|
|
if (skb_unclone(head, GFP_ATOMIC))
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out_nomem;
|
|
|
|
|
|
|
|
/* If the first fragment is fragmented itself, we split
|
|
|
|
* it to two chunks: the first with data and paged part
|
|
|
|
* and the second, holding only fragments. */
|
2010-08-23 07:13:46 +00:00
|
|
|
if (skb_has_frag_list(head)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
struct sk_buff *clone;
|
|
|
|
int i, plen = 0;
|
|
|
|
|
2015-04-03 08:17:26 +00:00
|
|
|
clone = alloc_skb(0, GFP_ATOMIC);
|
|
|
|
if (!clone)
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out_nomem;
|
|
|
|
skb_shinfo(clone)->frag_list = skb_shinfo(head)->frag_list;
|
2009-06-09 07:19:37 +00:00
|
|
|
skb_frag_list_init(head);
|
2011-10-18 21:00:24 +00:00
|
|
|
for (i = 0; i < skb_shinfo(head)->nr_frags; i++)
|
|
|
|
plen += skb_frag_size(&skb_shinfo(head)->frags[i]);
|
2005-04-16 22:20:36 +00:00
|
|
|
clone->len = clone->data_len = head->data_len - plen;
|
2018-08-11 20:27:25 +00:00
|
|
|
head->truesize += clone->truesize;
|
2005-04-16 22:20:36 +00:00
|
|
|
clone->csum = 0;
|
|
|
|
clone->ip_summed = head->ip_summed;
|
2015-07-23 10:05:38 +00:00
|
|
|
add_frag_mem_limit(qp->q.net, clone->truesize);
|
2018-08-02 23:34:39 +00:00
|
|
|
skb_shinfo(head)->frag_list = clone;
|
|
|
|
nextp = &clone->next;
|
|
|
|
} else {
|
|
|
|
nextp = &skb_shinfo(head)->frag_list;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-04-11 03:50:43 +00:00
|
|
|
skb_push(head, head->data - skb_network_header(head));
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2018-08-02 23:34:39 +00:00
|
|
|
/* Traverse the tree in order, to build frag_list. */
|
2018-08-11 20:27:25 +00:00
|
|
|
fp = FRAG_CB(head)->next_frag;
|
2018-08-02 23:34:39 +00:00
|
|
|
rbn = rb_next(&head->rbnode);
|
|
|
|
rb_erase(&head->rbnode, &qp->q.rb_fragments);
|
2018-08-11 20:27:25 +00:00
|
|
|
while (rbn || fp) {
|
|
|
|
/* fp points to the next sk_buff in the current run;
|
|
|
|
* rbn points to the next run.
|
|
|
|
*/
|
|
|
|
/* Go through the current run. */
|
|
|
|
while (fp) {
|
|
|
|
*nextp = fp;
|
|
|
|
nextp = &fp->next;
|
|
|
|
fp->prev = NULL;
|
|
|
|
memset(&fp->rbnode, 0, sizeof(fp->rbnode));
|
|
|
|
head->data_len += fp->len;
|
|
|
|
head->len += fp->len;
|
|
|
|
if (head->ip_summed != fp->ip_summed)
|
|
|
|
head->ip_summed = CHECKSUM_NONE;
|
|
|
|
else if (head->ip_summed == CHECKSUM_COMPLETE)
|
|
|
|
head->csum = csum_add(head->csum, fp->csum);
|
|
|
|
head->truesize += fp->truesize;
|
|
|
|
fp = FRAG_CB(fp)->next_frag;
|
|
|
|
}
|
|
|
|
/* Move to the next run. */
|
|
|
|
if (rbn) {
|
|
|
|
struct rb_node *rbnext = rb_next(rbn);
|
|
|
|
|
|
|
|
fp = rb_to_skb(rbn);
|
|
|
|
rb_erase(rbn, &qp->q.rb_fragments);
|
|
|
|
rbn = rbnext;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2015-08-01 06:52:20 +00:00
|
|
|
sub_frag_mem_limit(qp->q.net, head->truesize);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2018-08-02 23:34:39 +00:00
|
|
|
*nextp = NULL;
|
2005-04-16 22:20:36 +00:00
|
|
|
head->next = NULL;
|
2018-08-02 23:34:39 +00:00
|
|
|
head->prev = NULL;
|
2005-04-16 22:20:36 +00:00
|
|
|
head->dev = dev;
|
2007-10-15 09:24:19 +00:00
|
|
|
head->tstamp = qp->q.stamp;
|
2015-05-22 14:32:51 +00:00
|
|
|
IPCB(head)->frag_max_size = max(qp->max_df_size, qp->q.max_size);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-04-21 05:47:35 +00:00
|
|
|
iph = ip_hdr(head);
|
2005-04-16 22:20:36 +00:00
|
|
|
iph->tot_len = htons(len);
|
2011-05-16 08:37:37 +00:00
|
|
|
iph->tos |= ecn;
|
2015-05-22 14:32:51 +00:00
|
|
|
|
|
|
|
/* When we set IP_DF on a refragmented skb we must also force a
|
|
|
|
* call to ip_fragment to avoid forwarding a DF-skb of size s while
|
|
|
|
* original sender only sent fragments of size f (where f < s).
|
|
|
|
*
|
|
|
|
* We only set DF/IPSKB_FRAG_PMTU if such DF fragment was the largest
|
|
|
|
* frag seen to avoid sending tiny DF-fragments in case skb was built
|
|
|
|
* from one very small df-fragment and one large non-df frag.
|
|
|
|
*/
|
|
|
|
if (qp->max_df_size == qp->q.max_size) {
|
|
|
|
IPCB(head)->flags |= IPSKB_FRAG_PMTU;
|
|
|
|
iph->frag_off = htons(IP_DF);
|
|
|
|
} else {
|
|
|
|
iph->frag_off = 0;
|
|
|
|
}
|
|
|
|
|
2015-07-21 07:43:59 +00:00
|
|
|
ip_send_check(iph);
|
|
|
|
|
2016-04-27 23:44:35 +00:00
|
|
|
__IP_INC_STATS(net, IPSTATS_MIB_REASMOKS);
|
2007-10-15 09:24:19 +00:00
|
|
|
qp->q.fragments = NULL;
|
2018-08-02 23:34:39 +00:00
|
|
|
qp->q.rb_fragments = RB_ROOT;
|
2010-06-29 04:39:37 +00:00
|
|
|
qp->q.fragments_tail = NULL;
|
2018-08-11 20:27:25 +00:00
|
|
|
qp->q.last_run_head = NULL;
|
2007-10-14 07:38:15 +00:00
|
|
|
return 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
out_nomem:
|
2014-11-11 18:59:17 +00:00
|
|
|
net_dbg_ratelimited("queue_glue: no memory for gluing queue %p\n", qp);
|
2007-10-18 04:37:22 +00:00
|
|
|
err = -ENOMEM;
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out_fail;
|
|
|
|
out_oversize:
|
inet: frags: use rhashtables for reassembly units
Some applications still rely on IP fragmentation, and to be fair linux
reassembly unit is not working under any serious load.
It uses static hash tables of 1024 buckets, and up to 128 items per bucket (!!!)
A work queue is supposed to garbage collect items when host is under memory
pressure, and doing a hash rebuild, changing seed used in hash computations.
This work queue blocks softirqs for up to 25 ms when doing a hash rebuild,
occurring every 5 seconds if host is under fire.
Then there is the problem of sharing this hash table for all netns.
It is time to switch to rhashtables, and allocate one of them per netns
to speedup netns dismantle, since this is a critical metric these days.
Lookup is now using RCU. A followup patch will even remove
the refcount hold/release left from prior implementation and save
a couple of atomic operations.
Before this patch, 16 cpus (16 RX queue NIC) could not handle more
than 1 Mpps frags DDOS.
After the patch, I reach 9 Mpps without any tuning, and can use up to 2GB
of storage for the fragments (exact number depends on frags being evicted
after timeout)
$ grep FRAG /proc/net/sockstat
FRAG: inuse 1966916 memory 2140004608
A followup patch will change the limits for 64bit arches.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Florian Westphal <fw@strlen.de>
Cc: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Alexander Aring <alex.aring@gmail.com>
Cc: Stefan Schmidt <stefan@osg.samsung.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-03-31 19:58:49 +00:00
|
|
|
net_info_ratelimited("Oversized IP packet from %pI4\n", &qp->q.key.v4.saddr);
|
2005-04-16 22:20:36 +00:00
|
|
|
out_fail:
|
2016-04-27 23:44:35 +00:00
|
|
|
__IP_INC_STATS(net, IPSTATS_MIB_REASMFAILS);
|
2007-10-14 07:38:15 +00:00
|
|
|
return err;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Process an incoming IP datagram fragment. */
|
2015-10-09 18:44:54 +00:00
|
|
|
int ip_defrag(struct net *net, struct sk_buff *skb, u32 user)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2015-08-13 20:59:09 +00:00
|
|
|
struct net_device *dev = skb->dev ? : skb_dst(skb)->dev;
|
2015-09-30 03:07:13 +00:00
|
|
|
int vif = l3mdev_master_ifindex_rcu(dev);
|
2005-04-16 22:20:36 +00:00
|
|
|
struct ipq *qp;
|
2007-02-09 14:24:47 +00:00
|
|
|
|
2016-04-27 23:44:35 +00:00
|
|
|
__IP_INC_STATS(net, IPSTATS_MIB_REASMREQDS);
|
2016-01-22 23:49:12 +00:00
|
|
|
skb_orphan(skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Lookup (or create) queue header */
|
2015-08-13 20:59:09 +00:00
|
|
|
qp = ip_find(net, ip_hdr(skb), user, vif);
|
2015-04-03 08:17:27 +00:00
|
|
|
if (qp) {
|
2007-10-14 07:38:15 +00:00
|
|
|
int ret;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-10-15 09:24:19 +00:00
|
|
|
spin_lock(&qp->q.lock);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-10-14 07:38:15 +00:00
|
|
|
ret = ip_frag_queue(qp, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-10-15 09:24:19 +00:00
|
|
|
spin_unlock(&qp->q.lock);
|
2007-10-15 09:41:09 +00:00
|
|
|
ipq_put(qp);
|
2007-10-14 07:38:32 +00:00
|
|
|
return ret;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2016-04-27 23:44:35 +00:00
|
|
|
__IP_INC_STATS(net, IPSTATS_MIB_REASMFAILS);
|
2005-04-16 22:20:36 +00:00
|
|
|
kfree_skb(skb);
|
2007-10-14 07:38:32 +00:00
|
|
|
return -ENOMEM;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2010-07-09 21:22:10 +00:00
|
|
|
EXPORT_SYMBOL(ip_defrag);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2015-10-09 18:44:54 +00:00
|
|
|
struct sk_buff *ip_check_defrag(struct net *net, struct sk_buff *skb, u32 user)
|
2011-10-06 10:28:31 +00:00
|
|
|
{
|
2012-12-09 23:41:06 +00:00
|
|
|
struct iphdr iph;
|
2015-03-05 07:29:39 +00:00
|
|
|
int netoff;
|
2011-10-06 10:28:31 +00:00
|
|
|
u32 len;
|
|
|
|
|
|
|
|
if (skb->protocol != htons(ETH_P_IP))
|
|
|
|
return skb;
|
|
|
|
|
2015-03-05 07:29:39 +00:00
|
|
|
netoff = skb_network_offset(skb);
|
|
|
|
|
|
|
|
if (skb_copy_bits(skb, netoff, &iph, sizeof(iph)) < 0)
|
2011-10-06 10:28:31 +00:00
|
|
|
return skb;
|
|
|
|
|
2012-12-09 23:41:06 +00:00
|
|
|
if (iph.ihl < 5 || iph.version != 4)
|
2011-10-06 10:28:31 +00:00
|
|
|
return skb;
|
2012-12-09 23:41:06 +00:00
|
|
|
|
|
|
|
len = ntohs(iph.tot_len);
|
2015-03-05 07:29:39 +00:00
|
|
|
if (skb->len < netoff + len || len < (iph.ihl * 4))
|
2011-10-06 10:28:31 +00:00
|
|
|
return skb;
|
|
|
|
|
2012-12-09 23:41:06 +00:00
|
|
|
if (ip_is_fragment(&iph)) {
|
2011-10-06 10:28:31 +00:00
|
|
|
skb = skb_share_check(skb, GFP_ATOMIC);
|
|
|
|
if (skb) {
|
2015-03-05 07:29:39 +00:00
|
|
|
if (!pskb_may_pull(skb, netoff + iph.ihl * 4))
|
2012-12-09 23:41:06 +00:00
|
|
|
return skb;
|
2015-03-05 07:29:39 +00:00
|
|
|
if (pskb_trim_rcsum(skb, netoff + len))
|
2011-10-06 10:28:31 +00:00
|
|
|
return skb;
|
|
|
|
memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
|
2015-10-09 18:44:54 +00:00
|
|
|
if (ip_defrag(net, skb, user))
|
2011-10-06 10:28:31 +00:00
|
|
|
return NULL;
|
2013-12-16 06:12:18 +00:00
|
|
|
skb_clear_hash(skb);
|
2011-10-06 10:28:31 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(ip_check_defrag);
|
|
|
|
|
2018-08-11 20:27:24 +00:00
|
|
|
unsigned int inet_frag_rbtree_purge(struct rb_root *root)
|
|
|
|
{
|
|
|
|
struct rb_node *p = rb_first(root);
|
|
|
|
unsigned int sum = 0;
|
|
|
|
|
|
|
|
while (p) {
|
|
|
|
struct sk_buff *skb = rb_entry(p, struct sk_buff, rbnode);
|
|
|
|
|
|
|
|
p = rb_next(p);
|
|
|
|
rb_erase(&skb->rbnode, root);
|
|
|
|
while (skb) {
|
|
|
|
struct sk_buff *next = FRAG_CB(skb)->next_frag;
|
|
|
|
|
|
|
|
sum += skb->truesize;
|
|
|
|
kfree_skb(skb);
|
|
|
|
skb = next;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return sum;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(inet_frag_rbtree_purge);
|
|
|
|
|
2008-01-22 13:58:31 +00:00
|
|
|
#ifdef CONFIG_SYSCTL
|
2018-04-04 15:35:10 +00:00
|
|
|
static int dist_min;
|
2008-01-22 13:58:31 +00:00
|
|
|
|
2008-05-19 20:51:29 +00:00
|
|
|
static struct ctl_table ip4_frags_ns_ctl_table[] = {
|
2008-01-22 13:58:31 +00:00
|
|
|
{
|
|
|
|
.procname = "ipfrag_high_thresh",
|
2008-01-22 14:10:13 +00:00
|
|
|
.data = &init_net.ipv4.frags.high_thresh,
|
2018-03-31 19:58:53 +00:00
|
|
|
.maxlen = sizeof(unsigned long),
|
2008-01-22 13:58:31 +00:00
|
|
|
.mode = 0644,
|
2018-03-31 19:58:53 +00:00
|
|
|
.proc_handler = proc_doulongvec_minmax,
|
2014-07-24 14:50:37 +00:00
|
|
|
.extra1 = &init_net.ipv4.frags.low_thresh
|
2008-01-22 13:58:31 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
.procname = "ipfrag_low_thresh",
|
2008-01-22 14:10:13 +00:00
|
|
|
.data = &init_net.ipv4.frags.low_thresh,
|
2018-03-31 19:58:53 +00:00
|
|
|
.maxlen = sizeof(unsigned long),
|
2008-01-22 13:58:31 +00:00
|
|
|
.mode = 0644,
|
2018-03-31 19:58:53 +00:00
|
|
|
.proc_handler = proc_doulongvec_minmax,
|
2014-07-24 14:50:37 +00:00
|
|
|
.extra2 = &init_net.ipv4.frags.high_thresh
|
2008-01-22 13:58:31 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
.procname = "ipfrag_time",
|
2008-01-22 14:09:37 +00:00
|
|
|
.data = &init_net.ipv4.frags.timeout,
|
2008-01-22 13:58:31 +00:00
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
2008-11-04 02:21:05 +00:00
|
|
|
.proc_handler = proc_dointvec_jiffies,
|
2008-01-22 13:58:31 +00:00
|
|
|
},
|
2016-02-15 10:11:31 +00:00
|
|
|
{
|
|
|
|
.procname = "ipfrag_max_dist",
|
|
|
|
.data = &init_net.ipv4.frags.max_dist,
|
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
|
|
|
.proc_handler = proc_dointvec_minmax,
|
2018-04-04 15:35:10 +00:00
|
|
|
.extra1 = &dist_min,
|
2016-02-15 10:11:31 +00:00
|
|
|
},
|
2008-05-19 20:53:02 +00:00
|
|
|
{ }
|
|
|
|
};
|
|
|
|
|
2014-07-24 14:50:35 +00:00
|
|
|
/* secret interval has been deprecated */
|
|
|
|
static int ip4_frags_secret_interval_unused;
|
2008-05-19 20:53:02 +00:00
|
|
|
static struct ctl_table ip4_frags_ctl_table[] = {
|
2008-01-22 13:58:31 +00:00
|
|
|
{
|
|
|
|
.procname = "ipfrag_secret_interval",
|
2014-07-24 14:50:35 +00:00
|
|
|
.data = &ip4_frags_secret_interval_unused,
|
2008-01-22 13:58:31 +00:00
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
2008-11-04 02:21:05 +00:00
|
|
|
.proc_handler = proc_dointvec_jiffies,
|
2008-01-22 13:58:31 +00:00
|
|
|
},
|
|
|
|
{ }
|
|
|
|
};
|
|
|
|
|
2010-01-17 03:35:32 +00:00
|
|
|
static int __net_init ip4_frags_ns_ctl_register(struct net *net)
|
2008-01-22 13:58:31 +00:00
|
|
|
{
|
2008-01-22 14:08:36 +00:00
|
|
|
struct ctl_table *table;
|
2008-01-22 13:58:31 +00:00
|
|
|
struct ctl_table_header *hdr;
|
|
|
|
|
2008-05-19 20:51:29 +00:00
|
|
|
table = ip4_frags_ns_ctl_table;
|
2009-11-25 23:14:13 +00:00
|
|
|
if (!net_eq(net, &init_net)) {
|
2008-05-19 20:51:29 +00:00
|
|
|
table = kmemdup(table, sizeof(ip4_frags_ns_ctl_table), GFP_KERNEL);
|
2015-04-03 08:17:26 +00:00
|
|
|
if (!table)
|
2008-01-22 14:08:36 +00:00
|
|
|
goto err_alloc;
|
|
|
|
|
2008-01-22 14:10:13 +00:00
|
|
|
table[0].data = &net->ipv4.frags.high_thresh;
|
2014-07-24 14:50:37 +00:00
|
|
|
table[0].extra1 = &net->ipv4.frags.low_thresh;
|
|
|
|
table[0].extra2 = &init_net.ipv4.frags.high_thresh;
|
2008-01-22 14:10:13 +00:00
|
|
|
table[1].data = &net->ipv4.frags.low_thresh;
|
2014-07-24 14:50:37 +00:00
|
|
|
table[1].extra2 = &net->ipv4.frags.high_thresh;
|
2008-01-22 14:09:37 +00:00
|
|
|
table[2].data = &net->ipv4.frags.timeout;
|
2016-02-15 10:11:31 +00:00
|
|
|
table[3].data = &net->ipv4.frags.max_dist;
|
2008-01-22 14:08:36 +00:00
|
|
|
}
|
|
|
|
|
2012-04-19 13:44:49 +00:00
|
|
|
hdr = register_net_sysctl(net, "net/ipv4", table);
|
2015-04-03 08:17:26 +00:00
|
|
|
if (!hdr)
|
2008-01-22 14:08:36 +00:00
|
|
|
goto err_reg;
|
|
|
|
|
|
|
|
net->ipv4.frags_hdr = hdr;
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
err_reg:
|
2009-11-25 23:14:13 +00:00
|
|
|
if (!net_eq(net, &init_net))
|
2008-01-22 14:08:36 +00:00
|
|
|
kfree(table);
|
|
|
|
err_alloc:
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
2010-01-17 03:35:32 +00:00
|
|
|
static void __net_exit ip4_frags_ns_ctl_unregister(struct net *net)
|
2008-01-22 14:08:36 +00:00
|
|
|
{
|
|
|
|
struct ctl_table *table;
|
|
|
|
|
|
|
|
table = net->ipv4.frags_hdr->ctl_table_arg;
|
|
|
|
unregister_net_sysctl_table(net->ipv4.frags_hdr);
|
|
|
|
kfree(table);
|
2008-01-22 13:58:31 +00:00
|
|
|
}
|
2008-05-19 20:53:02 +00:00
|
|
|
|
2014-10-01 17:18:57 +00:00
|
|
|
static void __init ip4_frags_ctl_register(void)
|
2008-05-19 20:53:02 +00:00
|
|
|
{
|
2012-04-19 13:22:55 +00:00
|
|
|
register_net_sysctl(&init_net, "net/ipv4", ip4_frags_ctl_table);
|
2008-05-19 20:53:02 +00:00
|
|
|
}
|
2008-01-22 13:58:31 +00:00
|
|
|
#else
|
2014-11-04 19:44:04 +00:00
|
|
|
static int ip4_frags_ns_ctl_register(struct net *net)
|
2008-01-22 13:58:31 +00:00
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
2008-01-22 14:08:36 +00:00
|
|
|
|
2014-11-04 19:44:04 +00:00
|
|
|
static void ip4_frags_ns_ctl_unregister(struct net *net)
|
2008-01-22 14:08:36 +00:00
|
|
|
{
|
|
|
|
}
|
2008-05-19 20:53:02 +00:00
|
|
|
|
2014-11-04 19:44:04 +00:00
|
|
|
static void __init ip4_frags_ctl_register(void)
|
2008-05-19 20:53:02 +00:00
|
|
|
{
|
|
|
|
}
|
2008-01-22 13:58:31 +00:00
|
|
|
#endif
|
|
|
|
|
2010-01-17 03:35:32 +00:00
|
|
|
static int __net_init ipv4_frags_init_net(struct net *net)
|
2008-01-22 13:58:31 +00:00
|
|
|
{
|
2018-03-31 19:58:43 +00:00
|
|
|
int res;
|
|
|
|
|
net: increase fragment memory usage limits
Increase the amount of memory usage limits for incomplete
IP fragments.
Arguing for new thresh high/low values:
High threshold = 4 MBytes
Low threshold = 3 MBytes
The fragmentation memory accounting code, tries to account for the
real memory usage, by measuring both the size of frag queue struct
(inet_frag_queue (ipv4:ipq/ipv6:frag_queue)) and the SKB's truesize.
We want to be able to handle/hold-on-to enough fragments, to ensure
good performance, without causing incomplete fragments to hurt
scalability, by causing the number of inet_frag_queue to grow too much
(resulting longer searches for frag queues).
For IPv4, how much memory does the largest frag consume.
Maximum size fragment is 64K, which is approx 44 fragments with
MTU(1500) sized packets. Sizeof(struct ipq) is 200. A 1500 byte
packet results in a truesize of 2944 (not 2048 as I first assumed)
(44*2944)+200 = 129736 bytes
The current default high thresh of 262144 bytes, is obviously
problematic, as only two 64K fragments can fit in the queue at the
same time.
How many 64K fragment can we fit into 4 MBytes:
4*2^20/((44*2944)+200) = 32.34 fragment in queues
An attacker could send a separate/distinct fake fragment packets per
queue, causing us to allocate one inet_frag_queue per packet, and thus
attacking the hash table and its lists.
How many frag queue do we need to store, and given a current hash size
of 64, what is the average list length.
Using one MTU sized fragment per inet_frag_queue, each consuming
(2944+200) 3144 bytes.
4*2^20/(2944+200) = 1334 frag queues -> 21 avg list length
An attack could send small fragments, the smallest packet I could send
resulted in a truesize of 896 bytes (I'm a little surprised by this).
4*2^20/(896+200) = 3827 frag queues -> 59 avg list length
When increasing these number, we also need to followup with
improvements, that is going to help scalability. Simply increasing
the hash size, is not enough as the current implementation does not
have a per hash bucket locking.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-01-15 07:16:35 +00:00
|
|
|
/* Fragment cache limits.
|
|
|
|
*
|
|
|
|
* The fragment memory accounting code, (tries to) account for
|
|
|
|
* the real memory usage, by measuring both the size of frag
|
|
|
|
* queue struct (inet_frag_queue (ipv4:ipq/ipv6:frag_queue))
|
|
|
|
* and the SKB's truesize.
|
|
|
|
*
|
|
|
|
* A 64K fragment consumes 129736 bytes (44*2944)+200
|
|
|
|
* (1500 truesize == 2944, sizeof(struct ipq) == 200)
|
|
|
|
*
|
|
|
|
* We will commit 4MB at one time. Should we cross that limit
|
|
|
|
* we will prune down to 3MB, making room for approx 8 big 64K
|
|
|
|
* fragments 8x128k.
|
2008-01-22 14:10:13 +00:00
|
|
|
*/
|
net: increase fragment memory usage limits
Increase the amount of memory usage limits for incomplete
IP fragments.
Arguing for new thresh high/low values:
High threshold = 4 MBytes
Low threshold = 3 MBytes
The fragmentation memory accounting code, tries to account for the
real memory usage, by measuring both the size of frag queue struct
(inet_frag_queue (ipv4:ipq/ipv6:frag_queue)) and the SKB's truesize.
We want to be able to handle/hold-on-to enough fragments, to ensure
good performance, without causing incomplete fragments to hurt
scalability, by causing the number of inet_frag_queue to grow too much
(resulting longer searches for frag queues).
For IPv4, how much memory does the largest frag consume.
Maximum size fragment is 64K, which is approx 44 fragments with
MTU(1500) sized packets. Sizeof(struct ipq) is 200. A 1500 byte
packet results in a truesize of 2944 (not 2048 as I first assumed)
(44*2944)+200 = 129736 bytes
The current default high thresh of 262144 bytes, is obviously
problematic, as only two 64K fragments can fit in the queue at the
same time.
How many 64K fragment can we fit into 4 MBytes:
4*2^20/((44*2944)+200) = 32.34 fragment in queues
An attacker could send a separate/distinct fake fragment packets per
queue, causing us to allocate one inet_frag_queue per packet, and thus
attacking the hash table and its lists.
How many frag queue do we need to store, and given a current hash size
of 64, what is the average list length.
Using one MTU sized fragment per inet_frag_queue, each consuming
(2944+200) 3144 bytes.
4*2^20/(2944+200) = 1334 frag queues -> 21 avg list length
An attack could send small fragments, the smallest packet I could send
resulted in a truesize of 896 bytes (I'm a little surprised by this).
4*2^20/(896+200) = 3827 frag queues -> 59 avg list length
When increasing these number, we also need to followup with
improvements, that is going to help scalability. Simply increasing
the hash size, is not enough as the current implementation does not
have a per hash bucket locking.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-01-15 07:16:35 +00:00
|
|
|
net->ipv4.frags.high_thresh = 4 * 1024 * 1024;
|
|
|
|
net->ipv4.frags.low_thresh = 3 * 1024 * 1024;
|
2008-01-22 14:09:37 +00:00
|
|
|
/*
|
|
|
|
* Important NOTE! Fragment queue must be destroyed before MSL expires.
|
|
|
|
* RFC791 is wrong proposing to prolongate timer each fragment arrival
|
|
|
|
* by TTL.
|
|
|
|
*/
|
|
|
|
net->ipv4.frags.timeout = IP_FRAG_TIME;
|
|
|
|
|
2016-02-15 10:11:31 +00:00
|
|
|
net->ipv4.frags.max_dist = 64;
|
2018-03-31 19:58:44 +00:00
|
|
|
net->ipv4.frags.f = &ip4_frags;
|
2016-02-15 10:11:31 +00:00
|
|
|
|
2018-03-31 19:58:43 +00:00
|
|
|
res = inet_frags_init_net(&net->ipv4.frags);
|
|
|
|
if (res < 0)
|
|
|
|
return res;
|
|
|
|
res = ip4_frags_ns_ctl_register(net);
|
|
|
|
if (res < 0)
|
2018-03-31 19:58:44 +00:00
|
|
|
inet_frags_exit_net(&net->ipv4.frags);
|
2018-03-31 19:58:43 +00:00
|
|
|
return res;
|
2008-01-22 13:58:31 +00:00
|
|
|
}
|
|
|
|
|
2010-01-17 03:35:32 +00:00
|
|
|
static void __net_exit ipv4_frags_exit_net(struct net *net)
|
2008-01-22 14:12:39 +00:00
|
|
|
{
|
2008-05-19 20:51:29 +00:00
|
|
|
ip4_frags_ns_ctl_unregister(net);
|
2018-03-31 19:58:44 +00:00
|
|
|
inet_frags_exit_net(&net->ipv4.frags);
|
2008-01-22 14:12:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static struct pernet_operations ip4_frags_ops = {
|
|
|
|
.init = ipv4_frags_init_net,
|
|
|
|
.exit = ipv4_frags_exit_net,
|
|
|
|
};
|
|
|
|
|
inet: frags: use rhashtables for reassembly units
Some applications still rely on IP fragmentation, and to be fair linux
reassembly unit is not working under any serious load.
It uses static hash tables of 1024 buckets, and up to 128 items per bucket (!!!)
A work queue is supposed to garbage collect items when host is under memory
pressure, and doing a hash rebuild, changing seed used in hash computations.
This work queue blocks softirqs for up to 25 ms when doing a hash rebuild,
occurring every 5 seconds if host is under fire.
Then there is the problem of sharing this hash table for all netns.
It is time to switch to rhashtables, and allocate one of them per netns
to speedup netns dismantle, since this is a critical metric these days.
Lookup is now using RCU. A followup patch will even remove
the refcount hold/release left from prior implementation and save
a couple of atomic operations.
Before this patch, 16 cpus (16 RX queue NIC) could not handle more
than 1 Mpps frags DDOS.
After the patch, I reach 9 Mpps without any tuning, and can use up to 2GB
of storage for the fragments (exact number depends on frags being evicted
after timeout)
$ grep FRAG /proc/net/sockstat
FRAG: inuse 1966916 memory 2140004608
A followup patch will change the limits for 64bit arches.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Florian Westphal <fw@strlen.de>
Cc: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Alexander Aring <alex.aring@gmail.com>
Cc: Stefan Schmidt <stefan@osg.samsung.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-03-31 19:58:49 +00:00
|
|
|
|
|
|
|
static u32 ip4_key_hashfn(const void *data, u32 len, u32 seed)
|
|
|
|
{
|
|
|
|
return jhash2(data,
|
|
|
|
sizeof(struct frag_v4_compare_key) / sizeof(u32), seed);
|
|
|
|
}
|
|
|
|
|
|
|
|
static u32 ip4_obj_hashfn(const void *data, u32 len, u32 seed)
|
|
|
|
{
|
|
|
|
const struct inet_frag_queue *fq = data;
|
|
|
|
|
|
|
|
return jhash2((const u32 *)&fq->key.v4,
|
|
|
|
sizeof(struct frag_v4_compare_key) / sizeof(u32), seed);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int ip4_obj_cmpfn(struct rhashtable_compare_arg *arg, const void *ptr)
|
|
|
|
{
|
|
|
|
const struct frag_v4_compare_key *key = arg->key;
|
|
|
|
const struct inet_frag_queue *fq = ptr;
|
|
|
|
|
|
|
|
return !!memcmp(&fq->key, key, sizeof(*key));
|
|
|
|
}
|
|
|
|
|
|
|
|
static const struct rhashtable_params ip4_rhash_params = {
|
|
|
|
.head_offset = offsetof(struct inet_frag_queue, node),
|
|
|
|
.key_offset = offsetof(struct inet_frag_queue, key),
|
|
|
|
.key_len = sizeof(struct frag_v4_compare_key),
|
|
|
|
.hashfn = ip4_key_hashfn,
|
|
|
|
.obj_hashfn = ip4_obj_hashfn,
|
|
|
|
.obj_cmpfn = ip4_obj_cmpfn,
|
|
|
|
.automatic_shrinking = true,
|
|
|
|
};
|
|
|
|
|
2007-04-19 23:16:32 +00:00
|
|
|
void __init ipfrag_init(void)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2007-10-18 02:46:47 +00:00
|
|
|
ip4_frags.constructor = ip4_frag_init;
|
2007-10-15 09:39:14 +00:00
|
|
|
ip4_frags.destructor = ip4_frag_free;
|
|
|
|
ip4_frags.qsize = sizeof(struct ipq);
|
2007-10-18 02:45:23 +00:00
|
|
|
ip4_frags.frag_expire = ip_expire;
|
2014-08-01 10:29:48 +00:00
|
|
|
ip4_frags.frags_cache_name = ip_frag_cache_name;
|
inet: frags: use rhashtables for reassembly units
Some applications still rely on IP fragmentation, and to be fair linux
reassembly unit is not working under any serious load.
It uses static hash tables of 1024 buckets, and up to 128 items per bucket (!!!)
A work queue is supposed to garbage collect items when host is under memory
pressure, and doing a hash rebuild, changing seed used in hash computations.
This work queue blocks softirqs for up to 25 ms when doing a hash rebuild,
occurring every 5 seconds if host is under fire.
Then there is the problem of sharing this hash table for all netns.
It is time to switch to rhashtables, and allocate one of them per netns
to speedup netns dismantle, since this is a critical metric these days.
Lookup is now using RCU. A followup patch will even remove
the refcount hold/release left from prior implementation and save
a couple of atomic operations.
Before this patch, 16 cpus (16 RX queue NIC) could not handle more
than 1 Mpps frags DDOS.
After the patch, I reach 9 Mpps without any tuning, and can use up to 2GB
of storage for the fragments (exact number depends on frags being evicted
after timeout)
$ grep FRAG /proc/net/sockstat
FRAG: inuse 1966916 memory 2140004608
A followup patch will change the limits for 64bit arches.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Florian Westphal <fw@strlen.de>
Cc: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Alexander Aring <alex.aring@gmail.com>
Cc: Stefan Schmidt <stefan@osg.samsung.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-03-31 19:58:49 +00:00
|
|
|
ip4_frags.rhash_params = ip4_rhash_params;
|
2014-08-01 10:29:48 +00:00
|
|
|
if (inet_frags_init(&ip4_frags))
|
|
|
|
panic("IP: failed to allocate ip4_frags cache\n");
|
2018-03-31 19:58:47 +00:00
|
|
|
ip4_frags_ctl_register();
|
|
|
|
register_pernet_subsys(&ip4_frags_ops);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|