2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* Linux INET6 implementation
|
|
|
|
* FIB front-end.
|
|
|
|
*
|
|
|
|
* Authors:
|
2007-02-09 14:24:49 +00:00
|
|
|
* Pedro Roque <roque@di.fc.ul.pt>
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
|
|
|
* This program is free software; you can redistribute it and/or
|
|
|
|
* modify it under the terms of the GNU General Public License
|
|
|
|
* as published by the Free Software Foundation; either version
|
|
|
|
* 2 of the License, or (at your option) any later version.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* Changes:
|
|
|
|
*
|
|
|
|
* YOSHIFUJI Hideaki @USAGI
|
|
|
|
* reworked default router selection.
|
|
|
|
* - respect outgoing interface
|
|
|
|
* - select from (probably) reachable routers (i.e.
|
|
|
|
* routers in REACHABLE, STALE, DELAY or PROBE states).
|
|
|
|
* - always select the same router if it is (probably)
|
|
|
|
* reachable. otherwise, round-robin the list.
|
2006-08-24 00:23:25 +00:00
|
|
|
* Ville Nuorvala
|
|
|
|
* Fixed routing subtrees.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
|
2012-05-15 14:11:53 +00:00
|
|
|
#define pr_fmt(fmt) "IPv6: " fmt
|
|
|
|
|
2006-01-11 20:17:47 +00:00
|
|
|
#include <linux/capability.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <linux/errno.h>
|
2011-07-15 15:47:34 +00:00
|
|
|
#include <linux/export.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <linux/types.h>
|
|
|
|
#include <linux/times.h>
|
|
|
|
#include <linux/socket.h>
|
|
|
|
#include <linux/sockios.h>
|
|
|
|
#include <linux/net.h>
|
|
|
|
#include <linux/route.h>
|
|
|
|
#include <linux/netdevice.h>
|
|
|
|
#include <linux/in6.h>
|
2008-04-03 00:22:53 +00:00
|
|
|
#include <linux/mroute6.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <linux/init.h>
|
|
|
|
#include <linux/if_arp.h>
|
|
|
|
#include <linux/proc_fs.h>
|
|
|
|
#include <linux/seq_file.h>
|
2008-03-04 07:28:58 +00:00
|
|
|
#include <linux/nsproxy.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 08:04:11 +00:00
|
|
|
#include <linux/slab.h>
|
2017-10-06 19:05:57 +00:00
|
|
|
#include <linux/jhash.h>
|
2007-09-12 10:01:34 +00:00
|
|
|
#include <net/net_namespace.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <net/snmp.h>
|
|
|
|
#include <net/ipv6.h>
|
|
|
|
#include <net/ip6_fib.h>
|
|
|
|
#include <net/ip6_route.h>
|
|
|
|
#include <net/ndisc.h>
|
|
|
|
#include <net/addrconf.h>
|
|
|
|
#include <net/tcp.h>
|
|
|
|
#include <linux/rtnetlink.h>
|
|
|
|
#include <net/dst.h>
|
2015-08-20 11:56:31 +00:00
|
|
|
#include <net/dst_metadata.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <net/xfrm.h>
|
2006-07-31 03:43:36 +00:00
|
|
|
#include <net/netevent.h>
|
2006-08-15 07:35:24 +00:00
|
|
|
#include <net/netlink.h>
|
2012-10-22 03:42:09 +00:00
|
|
|
#include <net/nexthop.h>
|
2015-07-21 08:43:48 +00:00
|
|
|
#include <net/lwtunnel.h>
|
2015-08-20 11:56:31 +00:00
|
|
|
#include <net/ip_tunnels.h>
|
2015-10-12 18:47:10 +00:00
|
|
|
#include <net/l3mdev.h>
|
2015-11-19 20:24:22 +00:00
|
|
|
#include <trace/events/fib6.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2016-12-24 19:46:01 +00:00
|
|
|
#include <linux/uaccess.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
#ifdef CONFIG_SYSCTL
|
|
|
|
#include <linux/sysctl.h>
|
|
|
|
#endif
|
|
|
|
|
2013-07-11 10:43:42 +00:00
|
|
|
enum rt6_nud_state {
|
2013-12-11 12:48:20 +00:00
|
|
|
RT6_NUD_FAIL_HARD = -3,
|
|
|
|
RT6_NUD_FAIL_PROBE = -2,
|
|
|
|
RT6_NUD_FAIL_DO_RR = -1,
|
2013-07-11 10:43:42 +00:00
|
|
|
RT6_NUD_SUCCEED = 1
|
|
|
|
};
|
|
|
|
|
2015-05-23 03:56:05 +00:00
|
|
|
static void ip6_rt_copy_init(struct rt6_info *rt, struct rt6_info *ort);
|
2005-04-16 22:20:36 +00:00
|
|
|
static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie);
|
2010-12-13 20:52:14 +00:00
|
|
|
static unsigned int ip6_default_advmss(const struct dst_entry *dst);
|
2011-11-23 02:12:51 +00:00
|
|
|
static unsigned int ip6_mtu(const struct dst_entry *dst);
|
2005-04-16 22:20:36 +00:00
|
|
|
static struct dst_entry *ip6_negative_advice(struct dst_entry *);
|
|
|
|
static void ip6_dst_destroy(struct dst_entry *);
|
|
|
|
static void ip6_dst_ifdown(struct dst_entry *,
|
|
|
|
struct net_device *dev, int how);
|
2008-01-18 11:56:57 +00:00
|
|
|
static int ip6_dst_gc(struct dst_ops *ops);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
static int ip6_pkt_discard(struct sk_buff *skb);
|
2015-10-07 21:48:47 +00:00
|
|
|
static int ip6_pkt_discard_out(struct net *net, struct sock *sk, struct sk_buff *skb);
|
2013-12-02 14:25:21 +00:00
|
|
|
static int ip6_pkt_prohibit(struct sk_buff *skb);
|
2015-10-07 21:48:47 +00:00
|
|
|
static int ip6_pkt_prohibit_out(struct net *net, struct sock *sk, struct sk_buff *skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
static void ip6_link_failure(struct sk_buff *skb);
|
2012-07-17 10:29:28 +00:00
|
|
|
static void ip6_rt_update_pmtu(struct dst_entry *dst, struct sock *sk,
|
|
|
|
struct sk_buff *skb, u32 mtu);
|
|
|
|
static void rt6_do_redirect(struct dst_entry *dst, struct sock *sk,
|
|
|
|
struct sk_buff *skb);
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
static void rt6_dst_from_metrics_check(struct rt6_info *rt);
|
2013-06-28 15:35:48 +00:00
|
|
|
static int rt6_score_route(struct rt6_info *rt, int oif, int strict);
|
2017-02-02 20:37:11 +00:00
|
|
|
static size_t rt6_nlmsg_size(struct rt6_info *rt);
|
|
|
|
static int rt6_fill_node(struct net *net,
|
|
|
|
struct sk_buff *skb, struct rt6_info *rt,
|
|
|
|
struct in6_addr *dst, struct in6_addr *src,
|
|
|
|
int iif, int type, u32 portid, u32 seq,
|
|
|
|
unsigned int flags);
|
2017-10-06 19:05:57 +00:00
|
|
|
static struct rt6_info *rt6_find_cached_rt(struct rt6_info *rt,
|
|
|
|
struct in6_addr *daddr,
|
|
|
|
struct in6_addr *saddr);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-03-21 01:06:24 +00:00
|
|
|
#ifdef CONFIG_IPV6_ROUTE_INFO
|
2008-03-04 21:46:48 +00:00
|
|
|
static struct rt6_info *rt6_add_route_info(struct net *net,
|
2011-04-22 04:53:02 +00:00
|
|
|
const struct in6_addr *prefix, int prefixlen,
|
2016-10-24 17:52:35 +00:00
|
|
|
const struct in6_addr *gwaddr,
|
|
|
|
struct net_device *dev,
|
2012-04-15 05:58:06 +00:00
|
|
|
unsigned int pref);
|
2008-03-04 21:46:48 +00:00
|
|
|
static struct rt6_info *rt6_get_route_info(struct net *net,
|
2011-04-22 04:53:02 +00:00
|
|
|
const struct in6_addr *prefix, int prefixlen,
|
2016-10-24 17:52:35 +00:00
|
|
|
const struct in6_addr *gwaddr,
|
|
|
|
struct net_device *dev);
|
2006-03-21 01:06:24 +00:00
|
|
|
#endif
|
|
|
|
|
2015-05-23 03:56:04 +00:00
|
|
|
struct uncached_list {
|
|
|
|
spinlock_t lock;
|
|
|
|
struct list_head head;
|
|
|
|
};
|
|
|
|
|
|
|
|
static DEFINE_PER_CPU_ALIGNED(struct uncached_list, rt6_uncached_list);
|
|
|
|
|
|
|
|
static void rt6_uncached_list_add(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
struct uncached_list *ul = raw_cpu_ptr(&rt6_uncached_list);
|
|
|
|
|
|
|
|
rt->rt6i_uncached_list = ul;
|
|
|
|
|
|
|
|
spin_lock_bh(&ul->lock);
|
|
|
|
list_add_tail(&rt->rt6i_uncached, &ul->head);
|
|
|
|
spin_unlock_bh(&ul->lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void rt6_uncached_list_del(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
if (!list_empty(&rt->rt6i_uncached)) {
|
|
|
|
struct uncached_list *ul = rt->rt6i_uncached_list;
|
2017-10-06 19:06:11 +00:00
|
|
|
struct net *net = dev_net(rt->dst.dev);
|
2015-05-23 03:56:04 +00:00
|
|
|
|
|
|
|
spin_lock_bh(&ul->lock);
|
|
|
|
list_del(&rt->rt6i_uncached);
|
2017-10-06 19:06:11 +00:00
|
|
|
atomic_dec(&net->ipv6.rt6_stats->fib_rt_uncache);
|
2015-05-23 03:56:04 +00:00
|
|
|
spin_unlock_bh(&ul->lock);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void rt6_uncached_list_flush_dev(struct net *net, struct net_device *dev)
|
|
|
|
{
|
|
|
|
struct net_device *loopback_dev = net->loopback_dev;
|
|
|
|
int cpu;
|
|
|
|
|
2015-10-12 16:02:08 +00:00
|
|
|
if (dev == loopback_dev)
|
|
|
|
return;
|
|
|
|
|
2015-05-23 03:56:04 +00:00
|
|
|
for_each_possible_cpu(cpu) {
|
|
|
|
struct uncached_list *ul = per_cpu_ptr(&rt6_uncached_list, cpu);
|
|
|
|
struct rt6_info *rt;
|
|
|
|
|
|
|
|
spin_lock_bh(&ul->lock);
|
|
|
|
list_for_each_entry(rt, &ul->head, rt6i_uncached) {
|
|
|
|
struct inet6_dev *rt_idev = rt->rt6i_idev;
|
|
|
|
struct net_device *rt_dev = rt->dst.dev;
|
|
|
|
|
2015-10-12 16:02:08 +00:00
|
|
|
if (rt_idev->dev == dev) {
|
2015-05-23 03:56:04 +00:00
|
|
|
rt->rt6i_idev = in6_dev_get(loopback_dev);
|
|
|
|
in6_dev_put(rt_idev);
|
|
|
|
}
|
|
|
|
|
2015-10-12 16:02:08 +00:00
|
|
|
if (rt_dev == dev) {
|
2015-05-23 03:56:04 +00:00
|
|
|
rt->dst.dev = loopback_dev;
|
|
|
|
dev_hold(rt->dst.dev);
|
|
|
|
dev_put(rt_dev);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
spin_unlock_bh(&ul->lock);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-05-23 03:56:06 +00:00
|
|
|
static u32 *rt6_pcpu_cow_metrics(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
return dst_metrics_write_ptr(rt->dst.from);
|
|
|
|
}
|
|
|
|
|
2011-01-27 22:58:42 +00:00
|
|
|
static u32 *ipv6_cow_metrics(struct dst_entry *dst, unsigned long old)
|
|
|
|
{
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
struct rt6_info *rt = (struct rt6_info *)dst;
|
2011-01-27 22:58:42 +00:00
|
|
|
|
2015-05-23 03:56:06 +00:00
|
|
|
if (rt->rt6i_flags & RTF_PCPU)
|
|
|
|
return rt6_pcpu_cow_metrics(rt);
|
|
|
|
else if (rt->rt6i_flags & RTF_CACHE)
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
return NULL;
|
|
|
|
else
|
2015-02-13 00:14:08 +00:00
|
|
|
return dst_cow_metrics_generic(dst, old);
|
2011-01-27 22:58:42 +00:00
|
|
|
}
|
|
|
|
|
2012-07-03 04:52:24 +00:00
|
|
|
static inline const void *choose_neigh_daddr(struct rt6_info *rt,
|
|
|
|
struct sk_buff *skb,
|
|
|
|
const void *daddr)
|
2012-01-26 20:22:32 +00:00
|
|
|
{
|
|
|
|
struct in6_addr *p = &rt->rt6i_gateway;
|
|
|
|
|
2012-01-26 21:29:16 +00:00
|
|
|
if (!ipv6_addr_any(p))
|
2012-01-26 20:22:32 +00:00
|
|
|
return (const void *) p;
|
2012-07-03 04:52:24 +00:00
|
|
|
else if (skb)
|
|
|
|
return &ipv6_hdr(skb)->daddr;
|
2012-01-26 20:22:32 +00:00
|
|
|
return daddr;
|
|
|
|
}
|
|
|
|
|
2012-07-03 04:52:24 +00:00
|
|
|
static struct neighbour *ip6_neigh_lookup(const struct dst_entry *dst,
|
|
|
|
struct sk_buff *skb,
|
|
|
|
const void *daddr)
|
2011-07-18 07:40:17 +00:00
|
|
|
{
|
2012-01-26 20:22:32 +00:00
|
|
|
struct rt6_info *rt = (struct rt6_info *) dst;
|
|
|
|
struct neighbour *n;
|
|
|
|
|
2012-07-03 04:52:24 +00:00
|
|
|
daddr = choose_neigh_daddr(rt, skb, daddr);
|
2013-01-17 12:53:09 +00:00
|
|
|
n = __ipv6_neigh_lookup(dst->dev, daddr);
|
2011-12-28 20:41:23 +00:00
|
|
|
if (n)
|
|
|
|
return n;
|
|
|
|
return neigh_create(&nd_tbl, daddr, dst->dev);
|
|
|
|
}
|
|
|
|
|
2017-02-06 21:14:15 +00:00
|
|
|
static void ip6_confirm_neigh(const struct dst_entry *dst, const void *daddr)
|
|
|
|
{
|
|
|
|
struct net_device *dev = dst->dev;
|
|
|
|
struct rt6_info *rt = (struct rt6_info *)dst;
|
|
|
|
|
|
|
|
daddr = choose_neigh_daddr(rt, NULL, daddr);
|
|
|
|
if (!daddr)
|
|
|
|
return;
|
|
|
|
if (dev->flags & (IFF_NOARP | IFF_LOOPBACK))
|
|
|
|
return;
|
|
|
|
if (ipv6_addr_is_multicast((const struct in6_addr *)daddr))
|
|
|
|
return;
|
|
|
|
__ipv6_confirm_neigh(dev, daddr);
|
|
|
|
}
|
|
|
|
|
2008-03-04 21:48:53 +00:00
|
|
|
static struct dst_ops ip6_dst_ops_template = {
|
2005-04-16 22:20:36 +00:00
|
|
|
.family = AF_INET6,
|
|
|
|
.gc = ip6_dst_gc,
|
|
|
|
.gc_thresh = 1024,
|
|
|
|
.check = ip6_dst_check,
|
2010-12-13 20:52:14 +00:00
|
|
|
.default_advmss = ip6_default_advmss,
|
2011-11-23 02:12:51 +00:00
|
|
|
.mtu = ip6_mtu,
|
2011-01-27 22:58:42 +00:00
|
|
|
.cow_metrics = ipv6_cow_metrics,
|
2005-04-16 22:20:36 +00:00
|
|
|
.destroy = ip6_dst_destroy,
|
|
|
|
.ifdown = ip6_dst_ifdown,
|
|
|
|
.negative_advice = ip6_negative_advice,
|
|
|
|
.link_failure = ip6_link_failure,
|
|
|
|
.update_pmtu = ip6_rt_update_pmtu,
|
2012-07-12 07:05:02 +00:00
|
|
|
.redirect = rt6_do_redirect,
|
2015-10-07 21:48:39 +00:00
|
|
|
.local_out = __ip6_local_out,
|
2011-07-18 07:40:17 +00:00
|
|
|
.neigh_lookup = ip6_neigh_lookup,
|
2017-02-06 21:14:15 +00:00
|
|
|
.confirm_neigh = ip6_confirm_neigh,
|
2005-04-16 22:20:36 +00:00
|
|
|
};
|
|
|
|
|
2011-11-23 02:12:51 +00:00
|
|
|
static unsigned int ip6_blackhole_mtu(const struct dst_entry *dst)
|
2011-01-31 21:16:00 +00:00
|
|
|
{
|
2011-11-23 02:13:31 +00:00
|
|
|
unsigned int mtu = dst_metric_raw(dst, RTAX_MTU);
|
|
|
|
|
|
|
|
return mtu ? : dst->dev->mtu;
|
2011-01-31 21:16:00 +00:00
|
|
|
}
|
|
|
|
|
2012-07-17 10:29:28 +00:00
|
|
|
static void ip6_rt_blackhole_update_pmtu(struct dst_entry *dst, struct sock *sk,
|
|
|
|
struct sk_buff *skb, u32 mtu)
|
2007-05-25 01:17:54 +00:00
|
|
|
{
|
|
|
|
}
|
|
|
|
|
2012-07-17 10:29:28 +00:00
|
|
|
static void ip6_rt_blackhole_redirect(struct dst_entry *dst, struct sock *sk,
|
|
|
|
struct sk_buff *skb)
|
2012-07-12 07:39:24 +00:00
|
|
|
{
|
|
|
|
}
|
|
|
|
|
2007-05-25 01:17:54 +00:00
|
|
|
static struct dst_ops ip6_dst_blackhole_ops = {
|
|
|
|
.family = AF_INET6,
|
|
|
|
.destroy = ip6_dst_destroy,
|
|
|
|
.check = ip6_dst_check,
|
2011-11-23 02:12:51 +00:00
|
|
|
.mtu = ip6_blackhole_mtu,
|
2011-02-18 19:39:01 +00:00
|
|
|
.default_advmss = ip6_default_advmss,
|
2007-05-25 01:17:54 +00:00
|
|
|
.update_pmtu = ip6_rt_blackhole_update_pmtu,
|
2012-07-12 07:39:24 +00:00
|
|
|
.redirect = ip6_rt_blackhole_redirect,
|
2015-10-15 23:39:58 +00:00
|
|
|
.cow_metrics = dst_cow_metrics_generic,
|
2011-07-18 07:40:17 +00:00
|
|
|
.neigh_lookup = ip6_neigh_lookup,
|
2007-05-25 01:17:54 +00:00
|
|
|
};
|
|
|
|
|
net: Implement read-only protection and COW'ing of metrics.
Routing metrics are now copy-on-write.
Initially a route entry points it's metrics at a read-only location.
If a routing table entry exists, it will point there. Else it will
point at the all zero metric place-holder called 'dst_default_metrics'.
The writeability state of the metrics is stored in the low bits of the
metrics pointer, we have two bits left to spare if we want to store
more states.
For the initial implementation, COW is implemented simply via kmalloc.
However future enhancements will change this to place the writable
metrics somewhere else, in order to increase sharing. Very likely
this "somewhere else" will be the inetpeer cache.
Note also that this means that metrics updates may transiently fail
if we cannot COW the metrics successfully.
But even by itself, this patch should decrease memory usage and
increase cache locality especially for routing workloads. In those
cases the read-only metric copies stay in place and never get written
to.
TCP workloads where metrics get updated, and those rare cases where
PMTU triggers occur, will take a very slight performance hit. But
that hit will be alleviated when the long-term writable metrics
move to a more sharable location.
Since the metrics storage went from a u32 array of RTAX_MAX entries to
what is essentially a pointer, some retooling of the dst_entry layout
was necessary.
Most importantly, we need to preserve the alignment of the reference
count so that it doesn't share cache lines with the read-mostly state,
as per Eric Dumazet's alignment assertion checks.
The only non-trivial bit here is the move of the 'flags' member into
the writeable cacheline. This is OK since we are always accessing the
flags around the same moment when we made a modification to the
reference count.
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-01-27 04:51:05 +00:00
|
|
|
static const u32 ip6_template_metrics[RTAX_MAX] = {
|
2012-10-24 06:01:18 +00:00
|
|
|
[RTAX_HOPLIMIT - 1] = 0,
|
net: Implement read-only protection and COW'ing of metrics.
Routing metrics are now copy-on-write.
Initially a route entry points it's metrics at a read-only location.
If a routing table entry exists, it will point there. Else it will
point at the all zero metric place-holder called 'dst_default_metrics'.
The writeability state of the metrics is stored in the low bits of the
metrics pointer, we have two bits left to spare if we want to store
more states.
For the initial implementation, COW is implemented simply via kmalloc.
However future enhancements will change this to place the writable
metrics somewhere else, in order to increase sharing. Very likely
this "somewhere else" will be the inetpeer cache.
Note also that this means that metrics updates may transiently fail
if we cannot COW the metrics successfully.
But even by itself, this patch should decrease memory usage and
increase cache locality especially for routing workloads. In those
cases the read-only metric copies stay in place and never get written
to.
TCP workloads where metrics get updated, and those rare cases where
PMTU triggers occur, will take a very slight performance hit. But
that hit will be alleviated when the long-term writable metrics
move to a more sharable location.
Since the metrics storage went from a u32 array of RTAX_MAX entries to
what is essentially a pointer, some retooling of the dst_entry layout
was necessary.
Most importantly, we need to preserve the alignment of the reference
count so that it doesn't share cache lines with the read-mostly state,
as per Eric Dumazet's alignment assertion checks.
The only non-trivial bit here is the move of the 'flags' member into
the writeable cacheline. This is OK since we are always accessing the
flags around the same moment when we made a modification to the
reference count.
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-01-27 04:51:05 +00:00
|
|
|
};
|
|
|
|
|
2012-09-11 21:47:51 +00:00
|
|
|
static const struct rt6_info ip6_null_entry_template = {
|
2010-06-11 06:31:35 +00:00
|
|
|
.dst = {
|
|
|
|
.__refcnt = ATOMIC_INIT(1),
|
|
|
|
.__use = 1,
|
2012-09-10 22:09:47 +00:00
|
|
|
.obsolete = DST_OBSOLETE_FORCE_CHK,
|
2010-06-11 06:31:35 +00:00
|
|
|
.error = -ENETUNREACH,
|
|
|
|
.input = ip6_pkt_discard,
|
|
|
|
.output = ip6_pkt_discard_out,
|
2005-04-16 22:20:36 +00:00
|
|
|
},
|
|
|
|
.rt6i_flags = (RTF_REJECT | RTF_NONEXTHOP),
|
2009-05-21 00:38:59 +00:00
|
|
|
.rt6i_protocol = RTPROT_KERNEL,
|
2005-04-16 22:20:36 +00:00
|
|
|
.rt6i_metric = ~(u32) 0,
|
|
|
|
.rt6i_ref = ATOMIC_INIT(1),
|
|
|
|
};
|
|
|
|
|
2006-08-04 10:39:02 +00:00
|
|
|
#ifdef CONFIG_IPV6_MULTIPLE_TABLES
|
|
|
|
|
2012-09-11 21:47:51 +00:00
|
|
|
static const struct rt6_info ip6_prohibit_entry_template = {
|
2010-06-11 06:31:35 +00:00
|
|
|
.dst = {
|
|
|
|
.__refcnt = ATOMIC_INIT(1),
|
|
|
|
.__use = 1,
|
2012-09-10 22:09:47 +00:00
|
|
|
.obsolete = DST_OBSOLETE_FORCE_CHK,
|
2010-06-11 06:31:35 +00:00
|
|
|
.error = -EACCES,
|
|
|
|
.input = ip6_pkt_prohibit,
|
|
|
|
.output = ip6_pkt_prohibit_out,
|
2006-08-04 10:39:02 +00:00
|
|
|
},
|
|
|
|
.rt6i_flags = (RTF_REJECT | RTF_NONEXTHOP),
|
2009-05-21 00:38:59 +00:00
|
|
|
.rt6i_protocol = RTPROT_KERNEL,
|
2006-08-04 10:39:02 +00:00
|
|
|
.rt6i_metric = ~(u32) 0,
|
|
|
|
.rt6i_ref = ATOMIC_INIT(1),
|
|
|
|
};
|
|
|
|
|
2012-09-11 21:47:51 +00:00
|
|
|
static const struct rt6_info ip6_blk_hole_entry_template = {
|
2010-06-11 06:31:35 +00:00
|
|
|
.dst = {
|
|
|
|
.__refcnt = ATOMIC_INIT(1),
|
|
|
|
.__use = 1,
|
2012-09-10 22:09:47 +00:00
|
|
|
.obsolete = DST_OBSOLETE_FORCE_CHK,
|
2010-06-11 06:31:35 +00:00
|
|
|
.error = -EINVAL,
|
|
|
|
.input = dst_discard,
|
2015-10-07 21:48:47 +00:00
|
|
|
.output = dst_discard_out,
|
2006-08-04 10:39:02 +00:00
|
|
|
},
|
|
|
|
.rt6i_flags = (RTF_REJECT | RTF_NONEXTHOP),
|
2009-05-21 00:38:59 +00:00
|
|
|
.rt6i_protocol = RTPROT_KERNEL,
|
2006-08-04 10:39:02 +00:00
|
|
|
.rt6i_metric = ~(u32) 0,
|
|
|
|
.rt6i_ref = ATOMIC_INIT(1),
|
|
|
|
};
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
2015-10-15 23:39:57 +00:00
|
|
|
static void rt6_info_init(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
struct dst_entry *dst = &rt->dst;
|
|
|
|
|
|
|
|
memset(dst + 1, 0, sizeof(*rt) - sizeof(*dst));
|
|
|
|
INIT_LIST_HEAD(&rt->rt6i_siblings);
|
|
|
|
INIT_LIST_HEAD(&rt->rt6i_uncached);
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* allocate dst with ip6_dst_ops */
|
2015-05-23 03:56:06 +00:00
|
|
|
static struct rt6_info *__ip6_dst_alloc(struct net *net,
|
|
|
|
struct net_device *dev,
|
2015-08-14 18:05:52 +00:00
|
|
|
int flags)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2012-06-10 05:36:36 +00:00
|
|
|
struct rt6_info *rt = dst_alloc(&net->ipv6.ip6_dst_ops, dev,
|
2017-06-17 17:42:41 +00:00
|
|
|
1, DST_OBSOLETE_FORCE_CHK, flags);
|
2011-04-28 21:31:47 +00:00
|
|
|
|
2017-10-06 19:06:11 +00:00
|
|
|
if (rt) {
|
2015-10-15 23:39:57 +00:00
|
|
|
rt6_info_init(rt);
|
2017-10-06 19:06:11 +00:00
|
|
|
atomic_inc(&net->ipv6.rt6_stats->fib_rt_alloc);
|
|
|
|
}
|
2012-07-05 23:37:09 +00:00
|
|
|
|
2011-04-28 21:31:47 +00:00
|
|
|
return rt;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2016-04-07 18:10:06 +00:00
|
|
|
struct rt6_info *ip6_dst_alloc(struct net *net,
|
|
|
|
struct net_device *dev,
|
|
|
|
int flags)
|
2015-05-23 03:56:06 +00:00
|
|
|
{
|
2015-08-14 18:05:52 +00:00
|
|
|
struct rt6_info *rt = __ip6_dst_alloc(net, dev, flags);
|
2015-05-23 03:56:06 +00:00
|
|
|
|
|
|
|
if (rt) {
|
|
|
|
rt->rt6i_pcpu = alloc_percpu_gfp(struct rt6_info *, GFP_ATOMIC);
|
2017-10-09 13:01:37 +00:00
|
|
|
if (!rt->rt6i_pcpu) {
|
2017-06-17 17:42:36 +00:00
|
|
|
dst_release_immediate(&rt->dst);
|
2015-05-23 03:56:06 +00:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return rt;
|
|
|
|
}
|
2016-04-07 18:10:06 +00:00
|
|
|
EXPORT_SYMBOL(ip6_dst_alloc);
|
2015-05-23 03:56:06 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
static void ip6_dst_destroy(struct dst_entry *dst)
|
|
|
|
{
|
|
|
|
struct rt6_info *rt = (struct rt6_info *)dst;
|
2017-10-06 19:05:57 +00:00
|
|
|
struct rt6_exception_bucket *bucket;
|
ipv6: fix race condition regarding dst->expires and dst->from.
Eric Dumazet wrote:
| Some strange crashes happen in rt6_check_expired(), with access
| to random addresses.
|
| At first glance, it looks like the RTF_EXPIRES and
| stuff added in commit 1716a96101c49186b
| (ipv6: fix problem with expired dst cache)
| are racy : same dst could be manipulated at the same time
| on different cpus.
|
| At some point, our stack believes rt->dst.from contains a dst pointer,
| while its really a jiffie value (as rt->dst.expires shares the same area
| of memory)
|
| rt6_update_expires() should be fixed, or am I missing something ?
|
| CC Neil because of https://bugzilla.redhat.com/show_bug.cgi?id=892060
Because we do not have any locks for dst_entry, we cannot change
essential structure in the entry; e.g., we cannot change reference
to other entity.
To fix this issue, split 'from' and 'expires' field in dst_entry
out of union. Once it is 'from' is assigned in the constructor,
keep the reference until the very last stage of the life time of
the object.
Of course, it is unsafe to change 'from', so make rt6_set_from simple
just for fresh entries.
Reported-by: Eric Dumazet <eric.dumazet@gmail.com>
Reported-by: Neil Horman <nhorman@tuxdriver.com>
CC: Gao Feng <gaofeng@cn.fujitsu.com>
Signed-off-by: YOSHIFUJI Hideaki <yoshfuji@linux-ipv6.org>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Reported-by: Steinar H. Gunderson <sesse@google.com>
Reviewed-by: Neil Horman <nhorman@tuxdriver.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-02-20 00:29:08 +00:00
|
|
|
struct dst_entry *from = dst->from;
|
2015-05-23 03:56:04 +00:00
|
|
|
struct inet6_dev *idev;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
dst_destroy_metrics_generic(dst);
|
2015-07-02 14:30:24 +00:00
|
|
|
free_percpu(rt->rt6i_pcpu);
|
2015-05-23 03:56:04 +00:00
|
|
|
rt6_uncached_list_del(rt);
|
|
|
|
|
|
|
|
idev = rt->rt6i_idev;
|
2011-12-03 23:02:47 +00:00
|
|
|
if (idev) {
|
2005-04-16 22:20:36 +00:00
|
|
|
rt->rt6i_idev = NULL;
|
|
|
|
in6_dev_put(idev);
|
2007-02-09 14:24:49 +00:00
|
|
|
}
|
2017-10-06 19:05:57 +00:00
|
|
|
bucket = rcu_dereference_protected(rt->rt6i_exception_bucket, 1);
|
|
|
|
if (bucket) {
|
|
|
|
rt->rt6i_exception_bucket = NULL;
|
|
|
|
kfree(bucket);
|
|
|
|
}
|
2012-04-06 00:13:10 +00:00
|
|
|
|
ipv6: fix race condition regarding dst->expires and dst->from.
Eric Dumazet wrote:
| Some strange crashes happen in rt6_check_expired(), with access
| to random addresses.
|
| At first glance, it looks like the RTF_EXPIRES and
| stuff added in commit 1716a96101c49186b
| (ipv6: fix problem with expired dst cache)
| are racy : same dst could be manipulated at the same time
| on different cpus.
|
| At some point, our stack believes rt->dst.from contains a dst pointer,
| while its really a jiffie value (as rt->dst.expires shares the same area
| of memory)
|
| rt6_update_expires() should be fixed, or am I missing something ?
|
| CC Neil because of https://bugzilla.redhat.com/show_bug.cgi?id=892060
Because we do not have any locks for dst_entry, we cannot change
essential structure in the entry; e.g., we cannot change reference
to other entity.
To fix this issue, split 'from' and 'expires' field in dst_entry
out of union. Once it is 'from' is assigned in the constructor,
keep the reference until the very last stage of the life time of
the object.
Of course, it is unsafe to change 'from', so make rt6_set_from simple
just for fresh entries.
Reported-by: Eric Dumazet <eric.dumazet@gmail.com>
Reported-by: Neil Horman <nhorman@tuxdriver.com>
CC: Gao Feng <gaofeng@cn.fujitsu.com>
Signed-off-by: YOSHIFUJI Hideaki <yoshfuji@linux-ipv6.org>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Reported-by: Steinar H. Gunderson <sesse@google.com>
Reviewed-by: Neil Horman <nhorman@tuxdriver.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-02-20 00:29:08 +00:00
|
|
|
dst->from = NULL;
|
|
|
|
dst_release(from);
|
2010-11-30 20:27:11 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
static void ip6_dst_ifdown(struct dst_entry *dst, struct net_device *dev,
|
|
|
|
int how)
|
|
|
|
{
|
|
|
|
struct rt6_info *rt = (struct rt6_info *)dst;
|
|
|
|
struct inet6_dev *idev = rt->rt6i_idev;
|
2007-12-07 08:38:10 +00:00
|
|
|
struct net_device *loopback_dev =
|
2008-03-25 12:47:49 +00:00
|
|
|
dev_net(dev)->loopback_dev;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-08-14 17:44:59 +00:00
|
|
|
if (idev && idev->dev != loopback_dev) {
|
|
|
|
struct inet6_dev *loopback_idev = in6_dev_get(loopback_dev);
|
|
|
|
if (loopback_idev) {
|
|
|
|
rt->rt6i_idev = loopback_idev;
|
|
|
|
in6_dev_put(idev);
|
2012-07-03 05:43:47 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-11-11 19:51:07 +00:00
|
|
|
static bool __rt6_check_expired(const struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
if (rt->rt6i_flags & RTF_EXPIRES)
|
|
|
|
return time_after(jiffies, rt->dst.expires);
|
|
|
|
else
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2012-05-18 18:57:34 +00:00
|
|
|
static bool rt6_check_expired(const struct rt6_info *rt)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2012-04-06 00:13:10 +00:00
|
|
|
if (rt->rt6i_flags & RTF_EXPIRES) {
|
|
|
|
if (time_after(jiffies, rt->dst.expires))
|
2012-05-18 18:57:34 +00:00
|
|
|
return true;
|
2012-04-06 00:13:10 +00:00
|
|
|
} else if (rt->dst.from) {
|
2017-08-26 12:10:10 +00:00
|
|
|
return rt->dst.obsolete != DST_OBSOLETE_FORCE_CHK ||
|
|
|
|
rt6_check_expired((struct rt6_info *)rt->dst.from);
|
2012-04-06 00:13:10 +00:00
|
|
|
}
|
2012-05-18 18:57:34 +00:00
|
|
|
return false;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2012-10-22 03:42:09 +00:00
|
|
|
static struct rt6_info *rt6_multipath_select(struct rt6_info *match,
|
2013-06-28 15:35:48 +00:00
|
|
|
struct flowi6 *fl6, int oif,
|
|
|
|
int strict)
|
2012-10-22 03:42:09 +00:00
|
|
|
{
|
|
|
|
struct rt6_info *sibling, *next_sibling;
|
|
|
|
int route_choosen;
|
|
|
|
|
2017-08-23 07:58:31 +00:00
|
|
|
/* We might have already computed the hash for ICMPv6 errors. In such
|
|
|
|
* case it will always be non-zero. Otherwise now is the time to do it.
|
|
|
|
*/
|
|
|
|
if (!fl6->mp_hash)
|
|
|
|
fl6->mp_hash = rt6_multipath_hash(fl6, NULL);
|
|
|
|
|
|
|
|
route_choosen = fl6->mp_hash % (match->rt6i_nsiblings + 1);
|
2012-10-22 03:42:09 +00:00
|
|
|
/* Don't change the route, if route_choosen == 0
|
|
|
|
* (siblings does not include ourself)
|
|
|
|
*/
|
|
|
|
if (route_choosen)
|
|
|
|
list_for_each_entry_safe(sibling, next_sibling,
|
|
|
|
&match->rt6i_siblings, rt6i_siblings) {
|
|
|
|
route_choosen--;
|
|
|
|
if (route_choosen == 0) {
|
2013-06-28 15:35:48 +00:00
|
|
|
if (rt6_score_route(sibling, oif, strict) < 0)
|
|
|
|
break;
|
2012-10-22 03:42:09 +00:00
|
|
|
match = sibling;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return match;
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
* Route lookup. rcu_read_lock() should be held.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
|
2008-03-04 21:48:30 +00:00
|
|
|
static inline struct rt6_info *rt6_device_match(struct net *net,
|
|
|
|
struct rt6_info *rt,
|
2011-04-22 04:53:02 +00:00
|
|
|
const struct in6_addr *saddr,
|
2005-04-16 22:20:36 +00:00
|
|
|
int oif,
|
2008-06-28 03:14:54 +00:00
|
|
|
int flags)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct rt6_info *local = NULL;
|
|
|
|
struct rt6_info *sprt;
|
|
|
|
|
2008-07-02 09:30:18 +00:00
|
|
|
if (!oif && ipv6_addr_any(saddr))
|
|
|
|
goto out;
|
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
for (sprt = rt; sprt; sprt = rcu_dereference(sprt->dst.rt6_next)) {
|
2011-12-29 01:19:20 +00:00
|
|
|
struct net_device *dev = sprt->dst.dev;
|
2008-07-02 09:30:18 +00:00
|
|
|
|
|
|
|
if (oif) {
|
2005-04-16 22:20:36 +00:00
|
|
|
if (dev->ifindex == oif)
|
|
|
|
return sprt;
|
|
|
|
if (dev->flags & IFF_LOOPBACK) {
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!sprt->rt6i_idev ||
|
2005-04-16 22:20:36 +00:00
|
|
|
sprt->rt6i_idev->dev->ifindex != oif) {
|
2015-09-25 21:22:54 +00:00
|
|
|
if (flags & RT6_LOOKUP_F_IFACE)
|
2005-04-16 22:20:36 +00:00
|
|
|
continue;
|
2015-09-25 21:22:54 +00:00
|
|
|
if (local &&
|
|
|
|
local->rt6i_idev->dev->ifindex == oif)
|
2005-04-16 22:20:36 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
local = sprt;
|
|
|
|
}
|
2008-07-02 09:30:18 +00:00
|
|
|
} else {
|
|
|
|
if (ipv6_chk_addr(net, saddr, dev,
|
|
|
|
flags & RT6_LOOKUP_F_IFACE))
|
|
|
|
return sprt;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2008-07-02 09:30:18 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2008-07-02 09:30:18 +00:00
|
|
|
if (oif) {
|
2005-04-16 22:20:36 +00:00
|
|
|
if (local)
|
|
|
|
return local;
|
|
|
|
|
2008-06-28 03:14:54 +00:00
|
|
|
if (flags & RT6_LOOKUP_F_IFACE)
|
2008-03-04 21:48:30 +00:00
|
|
|
return net->ipv6.ip6_null_entry;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2008-07-02 09:30:18 +00:00
|
|
|
out:
|
2005-04-16 22:20:36 +00:00
|
|
|
return rt;
|
|
|
|
}
|
|
|
|
|
2006-03-21 01:05:13 +00:00
|
|
|
#ifdef CONFIG_IPV6_ROUTER_PREF
|
2013-10-21 04:17:15 +00:00
|
|
|
struct __rt6_probe_work {
|
|
|
|
struct work_struct work;
|
|
|
|
struct in6_addr target;
|
|
|
|
struct net_device *dev;
|
|
|
|
};
|
|
|
|
|
|
|
|
static void rt6_probe_deferred(struct work_struct *w)
|
|
|
|
{
|
|
|
|
struct in6_addr mcaddr;
|
|
|
|
struct __rt6_probe_work *work =
|
|
|
|
container_of(w, struct __rt6_probe_work, work);
|
|
|
|
|
|
|
|
addrconf_addr_solict_mult(&work->target, &mcaddr);
|
2016-12-02 22:00:08 +00:00
|
|
|
ndisc_send_ns(work->dev, &work->target, &mcaddr, NULL, 0);
|
2013-10-21 04:17:15 +00:00
|
|
|
dev_put(work->dev);
|
2015-02-08 09:14:07 +00:00
|
|
|
kfree(work);
|
2013-10-21 04:17:15 +00:00
|
|
|
}
|
|
|
|
|
2006-03-21 01:05:13 +00:00
|
|
|
static void rt6_probe(struct rt6_info *rt)
|
|
|
|
{
|
2015-07-24 16:57:42 +00:00
|
|
|
struct __rt6_probe_work *work;
|
2011-07-29 19:00:53 +00:00
|
|
|
struct neighbour *neigh;
|
2006-03-21 01:05:13 +00:00
|
|
|
/*
|
|
|
|
* Okay, this does not seem to be appropriate
|
|
|
|
* for now, however, we need to check if it
|
|
|
|
* is really so; aka Router Reachability Probing.
|
|
|
|
*
|
|
|
|
* Router Reachability Probe MUST be rate-limited
|
|
|
|
* to no more than one per minute.
|
|
|
|
*/
|
2013-01-17 12:53:43 +00:00
|
|
|
if (!rt || !(rt->rt6i_flags & RTF_GATEWAY))
|
2013-01-17 12:53:02 +00:00
|
|
|
return;
|
2013-01-17 12:53:43 +00:00
|
|
|
rcu_read_lock_bh();
|
|
|
|
neigh = __ipv6_neigh_lookup_noref(rt->dst.dev, &rt->rt6i_gateway);
|
|
|
|
if (neigh) {
|
2015-07-24 16:57:43 +00:00
|
|
|
if (neigh->nud_state & NUD_VALID)
|
|
|
|
goto out;
|
|
|
|
|
2015-07-24 16:57:42 +00:00
|
|
|
work = NULL;
|
2013-01-17 12:53:43 +00:00
|
|
|
write_lock(&neigh->lock);
|
2015-07-24 16:57:42 +00:00
|
|
|
if (!(neigh->nud_state & NUD_VALID) &&
|
|
|
|
time_after(jiffies,
|
|
|
|
neigh->updated +
|
|
|
|
rt->rt6i_idev->cnf.rtr_probe_interval)) {
|
|
|
|
work = kmalloc(sizeof(*work), GFP_ATOMIC);
|
|
|
|
if (work)
|
|
|
|
__neigh_set_probe_once(neigh);
|
2013-10-21 04:17:15 +00:00
|
|
|
}
|
2013-01-17 12:53:43 +00:00
|
|
|
write_unlock(&neigh->lock);
|
2015-07-24 16:57:42 +00:00
|
|
|
} else {
|
|
|
|
work = kmalloc(sizeof(*work), GFP_ATOMIC);
|
2011-07-29 19:00:53 +00:00
|
|
|
}
|
2015-07-24 16:57:42 +00:00
|
|
|
|
|
|
|
if (work) {
|
|
|
|
INIT_WORK(&work->work, rt6_probe_deferred);
|
|
|
|
work->target = rt->rt6i_gateway;
|
|
|
|
dev_hold(rt->dst.dev);
|
|
|
|
work->dev = rt->dst.dev;
|
|
|
|
schedule_work(&work->work);
|
|
|
|
}
|
|
|
|
|
2015-07-24 16:57:43 +00:00
|
|
|
out:
|
2013-01-17 12:53:43 +00:00
|
|
|
rcu_read_unlock_bh();
|
2006-03-21 01:05:13 +00:00
|
|
|
}
|
|
|
|
#else
|
|
|
|
static inline void rt6_probe(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
2006-03-21 01:00:26 +00:00
|
|
|
* Default Router Selection (RFC 2461 6.3.6)
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2007-03-22 19:27:49 +00:00
|
|
|
static inline int rt6_check_dev(struct rt6_info *rt, int oif)
|
2006-03-21 01:00:26 +00:00
|
|
|
{
|
2011-12-29 01:19:20 +00:00
|
|
|
struct net_device *dev = rt->dst.dev;
|
2007-04-06 18:42:27 +00:00
|
|
|
if (!oif || dev->ifindex == oif)
|
2006-03-21 01:00:26 +00:00
|
|
|
return 2;
|
2007-04-06 18:42:27 +00:00
|
|
|
if ((dev->flags & IFF_LOOPBACK) &&
|
|
|
|
rt->rt6i_idev && rt->rt6i_idev->dev->ifindex == oif)
|
|
|
|
return 1;
|
|
|
|
return 0;
|
2006-03-21 01:00:26 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-07-11 10:43:42 +00:00
|
|
|
static inline enum rt6_nud_state rt6_check_neigh(struct rt6_info *rt)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2011-07-29 19:00:53 +00:00
|
|
|
struct neighbour *neigh;
|
2013-07-11 10:43:42 +00:00
|
|
|
enum rt6_nud_state ret = RT6_NUD_FAIL_HARD;
|
2011-07-29 19:00:53 +00:00
|
|
|
|
2006-05-26 20:23:41 +00:00
|
|
|
if (rt->rt6i_flags & RTF_NONEXTHOP ||
|
|
|
|
!(rt->rt6i_flags & RTF_GATEWAY))
|
2013-07-11 10:43:42 +00:00
|
|
|
return RT6_NUD_SUCCEED;
|
2013-01-17 12:53:38 +00:00
|
|
|
|
|
|
|
rcu_read_lock_bh();
|
|
|
|
neigh = __ipv6_neigh_lookup_noref(rt->dst.dev, &rt->rt6i_gateway);
|
|
|
|
if (neigh) {
|
|
|
|
read_lock(&neigh->lock);
|
2006-03-21 01:00:26 +00:00
|
|
|
if (neigh->nud_state & NUD_VALID)
|
2013-07-11 10:43:42 +00:00
|
|
|
ret = RT6_NUD_SUCCEED;
|
2008-01-19 08:35:16 +00:00
|
|
|
#ifdef CONFIG_IPV6_ROUTER_PREF
|
2012-12-03 10:26:54 +00:00
|
|
|
else if (!(neigh->nud_state & NUD_FAILED))
|
2013-07-11 10:43:42 +00:00
|
|
|
ret = RT6_NUD_SUCCEED;
|
2013-12-11 12:48:20 +00:00
|
|
|
else
|
|
|
|
ret = RT6_NUD_FAIL_PROBE;
|
2008-01-19 08:35:16 +00:00
|
|
|
#endif
|
2013-01-17 12:53:38 +00:00
|
|
|
read_unlock(&neigh->lock);
|
2013-07-11 10:43:42 +00:00
|
|
|
} else {
|
|
|
|
ret = IS_ENABLED(CONFIG_IPV6_ROUTER_PREF) ?
|
2013-12-11 12:48:20 +00:00
|
|
|
RT6_NUD_SUCCEED : RT6_NUD_FAIL_DO_RR;
|
2012-12-03 10:26:54 +00:00
|
|
|
}
|
2013-01-17 12:53:38 +00:00
|
|
|
rcu_read_unlock_bh();
|
|
|
|
|
2012-12-03 10:26:54 +00:00
|
|
|
return ret;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2006-03-21 01:00:26 +00:00
|
|
|
static int rt6_score_route(struct rt6_info *rt, int oif,
|
|
|
|
int strict)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2012-12-03 10:26:54 +00:00
|
|
|
int m;
|
2007-02-09 14:24:49 +00:00
|
|
|
|
2006-05-26 20:23:41 +00:00
|
|
|
m = rt6_check_dev(rt, oif);
|
2006-08-24 00:25:05 +00:00
|
|
|
if (!m && (strict & RT6_LOOKUP_F_IFACE))
|
2013-07-11 10:43:42 +00:00
|
|
|
return RT6_NUD_FAIL_HARD;
|
2006-03-21 01:04:53 +00:00
|
|
|
#ifdef CONFIG_IPV6_ROUTER_PREF
|
|
|
|
m |= IPV6_DECODE_PREF(IPV6_EXTRACT_PREF(rt->rt6i_flags)) << 2;
|
|
|
|
#endif
|
2013-07-11 10:43:42 +00:00
|
|
|
if (strict & RT6_LOOKUP_F_REACHABLE) {
|
|
|
|
int n = rt6_check_neigh(rt);
|
|
|
|
if (n < 0)
|
|
|
|
return n;
|
|
|
|
}
|
2006-03-21 01:00:26 +00:00
|
|
|
return m;
|
|
|
|
}
|
|
|
|
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
static struct rt6_info *find_match(struct rt6_info *rt, int oif, int strict,
|
2013-07-11 10:43:42 +00:00
|
|
|
int *mpri, struct rt6_info *match,
|
|
|
|
bool *do_rr)
|
2006-03-21 01:00:26 +00:00
|
|
|
{
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
int m;
|
2013-07-11 10:43:42 +00:00
|
|
|
bool match_do_rr = false;
|
2015-08-13 14:39:01 +00:00
|
|
|
struct inet6_dev *idev = rt->rt6i_idev;
|
|
|
|
struct net_device *dev = rt->dst.dev;
|
|
|
|
|
|
|
|
if (dev && !netif_carrier_ok(dev) &&
|
2016-10-24 19:27:23 +00:00
|
|
|
idev->cnf.ignore_routes_with_linkdown &&
|
|
|
|
!(strict & RT6_LOOKUP_F_IGNORE_LINKSTATE))
|
2015-08-13 14:39:01 +00:00
|
|
|
goto out;
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
|
|
|
|
if (rt6_check_expired(rt))
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
m = rt6_score_route(rt, oif, strict);
|
2013-12-11 12:48:20 +00:00
|
|
|
if (m == RT6_NUD_FAIL_DO_RR) {
|
2013-07-11 10:43:42 +00:00
|
|
|
match_do_rr = true;
|
|
|
|
m = 0; /* lowest valid score */
|
2013-12-11 12:48:20 +00:00
|
|
|
} else if (m == RT6_NUD_FAIL_HARD) {
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
goto out;
|
2013-07-11 10:43:42 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (strict & RT6_LOOKUP_F_REACHABLE)
|
|
|
|
rt6_probe(rt);
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
|
2013-12-11 12:48:20 +00:00
|
|
|
/* note that m can be RT6_NUD_FAIL_PROBE at this point */
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
if (m > *mpri) {
|
2013-07-11 10:43:42 +00:00
|
|
|
*do_rr = match_do_rr;
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
*mpri = m;
|
|
|
|
match = rt;
|
|
|
|
}
|
|
|
|
out:
|
|
|
|
return match;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct rt6_info *find_rr_leaf(struct fib6_node *fn,
|
2017-10-06 19:06:08 +00:00
|
|
|
struct rt6_info *leaf,
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
struct rt6_info *rr_head,
|
2013-07-11 10:43:42 +00:00
|
|
|
u32 metric, int oif, int strict,
|
|
|
|
bool *do_rr)
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
{
|
2015-04-28 20:03:04 +00:00
|
|
|
struct rt6_info *rt, *match, *cont;
|
2006-03-21 01:00:26 +00:00
|
|
|
int mpri = -1;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
match = NULL;
|
2015-04-28 20:03:04 +00:00
|
|
|
cont = NULL;
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
for (rt = rr_head; rt; rt = rcu_dereference(rt->dst.rt6_next)) {
|
2015-04-28 20:03:04 +00:00
|
|
|
if (rt->rt6i_metric != metric) {
|
|
|
|
cont = rt;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
match = find_match(rt, oif, strict, &mpri, match, do_rr);
|
|
|
|
}
|
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
for (rt = leaf; rt && rt != rr_head;
|
|
|
|
rt = rcu_dereference(rt->dst.rt6_next)) {
|
2015-04-28 20:03:04 +00:00
|
|
|
if (rt->rt6i_metric != metric) {
|
|
|
|
cont = rt;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2013-07-11 10:43:42 +00:00
|
|
|
match = find_match(rt, oif, strict, &mpri, match, do_rr);
|
2015-04-28 20:03:04 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (match || !cont)
|
|
|
|
return match;
|
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
for (rt = cont; rt; rt = rcu_dereference(rt->dst.rt6_next))
|
2013-07-11 10:43:42 +00:00
|
|
|
match = find_match(rt, oif, strict, &mpri, match, do_rr);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
return match;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-10-06 19:06:08 +00:00
|
|
|
static struct rt6_info *rt6_select(struct net *net, struct fib6_node *fn,
|
|
|
|
int oif, int strict)
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
{
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
struct rt6_info *leaf = rcu_dereference(fn->leaf);
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
struct rt6_info *match, *rt0;
|
2013-07-11 10:43:42 +00:00
|
|
|
bool do_rr = false;
|
2017-10-06 19:06:09 +00:00
|
|
|
int key_plen;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-10-23 21:59:35 +00:00
|
|
|
if (!leaf || leaf == net->ipv6.ip6_null_entry)
|
2017-10-06 19:06:08 +00:00
|
|
|
return net->ipv6.ip6_null_entry;
|
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rt0 = rcu_dereference(fn->rr_ptr);
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
if (!rt0)
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rt0 = leaf;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-10-06 19:06:09 +00:00
|
|
|
/* Double check to make sure fn is not an intermediate node
|
|
|
|
* and fn->leaf does not points to its child's leaf
|
|
|
|
* (This might happen if all routes under fn are deleted from
|
|
|
|
* the tree and fib6_repair_tree() is called on the node.)
|
|
|
|
*/
|
|
|
|
key_plen = rt0->rt6i_dst.plen;
|
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
|
|
|
if (rt0->rt6i_src.plen)
|
|
|
|
key_plen = rt0->rt6i_src.plen;
|
|
|
|
#endif
|
|
|
|
if (fn->fn_bit != key_plen)
|
|
|
|
return net->ipv6.ip6_null_entry;
|
|
|
|
|
2017-10-06 19:06:08 +00:00
|
|
|
match = find_rr_leaf(fn, leaf, rt0, rt0->rt6i_metric, oif, strict,
|
2013-07-11 10:43:42 +00:00
|
|
|
&do_rr);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-07-11 10:43:42 +00:00
|
|
|
if (do_rr) {
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
struct rt6_info *next = rcu_dereference(rt0->dst.rt6_next);
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
|
2006-03-21 01:00:26 +00:00
|
|
|
/* no entries matched; do round-robin */
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
if (!next || next->rt6i_metric != rt0->rt6i_metric)
|
2017-10-06 19:06:08 +00:00
|
|
|
next = leaf;
|
[IPV6]: Fix routing round-robin locking.
As per RFC2461, section 6.3.6, item #2, when no routers on the
matching list are known to be reachable or probably reachable we
do round robin on those available routes so that we make sure
to probe as many of them as possible to detect when one becomes
reachable faster.
Each routing table has a rwlock protecting the tree and the linked
list of routes at each leaf. The round robin code executes during
lookup and thus with the rwlock taken as a reader. A small local
spinlock tries to provide protection but this does not work at all
for two reasons:
1) The round-robin list manipulation, as coded, goes like this (with
read lock held):
walk routes finding head and tail
spin_lock();
rotate list using head and tail
spin_unlock();
While one thread is rotating the list, another thread can
end up with stale values of head and tail and then proceed
to corrupt the list when it gets the lock. This ends up causing
the OOPS in fib6_add() later onthat many people have been hitting.
2) All the other code paths that run with the rwlock held as
a reader do not expect the list to change on them, they
expect it to remain completely fixed while they hold the
lock in that way.
So, simply stated, it is impossible to implement this correctly using
a manipulation of the list without violating the rwlock locking
semantics.
Reimplement using a per-fib6_node round-robin pointer. This way we
don't need to manipulate the list at all, and since the round-robin
pointer can only ever point to real existing entries we don't need
to perform any locking on the changing of the round-robin pointer
itself. We only need to reset the round-robin pointer to NULL when
the entry it is pointing to is removed.
The idea is from Thomas Graf and it is very similar to how this
was implemented before the advanced router selection code when in.
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-25 03:36:25 +00:00
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
if (next != rt0) {
|
|
|
|
spin_lock_bh(&leaf->rt6i_table->tb6_lock);
|
|
|
|
/* make sure next is not being deleted from the tree */
|
|
|
|
if (next->rt6i_node)
|
|
|
|
rcu_assign_pointer(fn->rr_ptr, next);
|
|
|
|
spin_unlock_bh(&leaf->rt6i_table->tb6_lock);
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2010-09-22 20:43:57 +00:00
|
|
|
return match ? match : net->ipv6.ip6_null_entry;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2015-05-23 03:55:59 +00:00
|
|
|
static bool rt6_is_gw_or_nonexthop(const struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
return (rt->rt6i_flags & (RTF_NONEXTHOP | RTF_GATEWAY));
|
|
|
|
}
|
|
|
|
|
2006-03-21 01:06:24 +00:00
|
|
|
#ifdef CONFIG_IPV6_ROUTE_INFO
|
|
|
|
int rt6_route_rcv(struct net_device *dev, u8 *opt, int len,
|
2011-04-22 04:53:02 +00:00
|
|
|
const struct in6_addr *gwaddr)
|
2006-03-21 01:06:24 +00:00
|
|
|
{
|
2008-03-25 12:47:49 +00:00
|
|
|
struct net *net = dev_net(dev);
|
2006-03-21 01:06:24 +00:00
|
|
|
struct route_info *rinfo = (struct route_info *) opt;
|
|
|
|
struct in6_addr prefix_buf, *prefix;
|
|
|
|
unsigned int pref;
|
2008-05-27 08:37:49 +00:00
|
|
|
unsigned long lifetime;
|
2006-03-21 01:06:24 +00:00
|
|
|
struct rt6_info *rt;
|
|
|
|
|
|
|
|
if (len < sizeof(struct route_info)) {
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Sanity check for prefix_len and length */
|
|
|
|
if (rinfo->length > 3) {
|
|
|
|
return -EINVAL;
|
|
|
|
} else if (rinfo->prefix_len > 128) {
|
|
|
|
return -EINVAL;
|
|
|
|
} else if (rinfo->prefix_len > 64) {
|
|
|
|
if (rinfo->length < 2) {
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
} else if (rinfo->prefix_len > 0) {
|
|
|
|
if (rinfo->length < 1) {
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
pref = rinfo->route_pref;
|
|
|
|
if (pref == ICMPV6_ROUTER_PREF_INVALID)
|
2009-09-10 06:25:11 +00:00
|
|
|
return -EINVAL;
|
2006-03-21 01:06:24 +00:00
|
|
|
|
2008-05-27 08:37:49 +00:00
|
|
|
lifetime = addrconf_timeout_fixup(ntohl(rinfo->lifetime), HZ);
|
2006-03-21 01:06:24 +00:00
|
|
|
|
|
|
|
if (rinfo->length == 3)
|
|
|
|
prefix = (struct in6_addr *)rinfo->prefix;
|
|
|
|
else {
|
|
|
|
/* this function is safe */
|
|
|
|
ipv6_addr_prefix(&prefix_buf,
|
|
|
|
(struct in6_addr *)rinfo->prefix,
|
|
|
|
rinfo->prefix_len);
|
|
|
|
prefix = &prefix_buf;
|
|
|
|
}
|
|
|
|
|
2013-11-08 01:56:53 +00:00
|
|
|
if (rinfo->prefix_len == 0)
|
|
|
|
rt = rt6_get_dflt_router(gwaddr, dev);
|
|
|
|
else
|
|
|
|
rt = rt6_get_route_info(net, prefix, rinfo->prefix_len,
|
2016-10-24 17:52:35 +00:00
|
|
|
gwaddr, dev);
|
2006-03-21 01:06:24 +00:00
|
|
|
|
|
|
|
if (rt && !lifetime) {
|
2006-08-22 07:00:21 +00:00
|
|
|
ip6_del_rt(rt);
|
2006-03-21 01:06:24 +00:00
|
|
|
rt = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!rt && lifetime)
|
2016-10-24 17:52:35 +00:00
|
|
|
rt = rt6_add_route_info(net, prefix, rinfo->prefix_len, gwaddr,
|
|
|
|
dev, pref);
|
2006-03-21 01:06:24 +00:00
|
|
|
else if (rt)
|
|
|
|
rt->rt6i_flags = RTF_ROUTEINFO |
|
|
|
|
(rt->rt6i_flags & ~RTF_PREF_MASK) | RTF_PREF(pref);
|
|
|
|
|
|
|
|
if (rt) {
|
2012-04-06 00:13:10 +00:00
|
|
|
if (!addrconf_finite_timeout(lifetime))
|
|
|
|
rt6_clean_expires(rt);
|
|
|
|
else
|
|
|
|
rt6_set_expires(rt, jiffies + HZ * lifetime);
|
|
|
|
|
2012-10-29 00:13:19 +00:00
|
|
|
ip6_rt_put(rt);
|
2006-03-21 01:06:24 +00:00
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2014-10-20 20:42:43 +00:00
|
|
|
static struct fib6_node* fib6_backtrack(struct fib6_node *fn,
|
|
|
|
struct in6_addr *saddr)
|
|
|
|
{
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
struct fib6_node *pn, *sn;
|
2014-10-20 20:42:43 +00:00
|
|
|
while (1) {
|
|
|
|
if (fn->fn_flags & RTN_TL_ROOT)
|
|
|
|
return NULL;
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
pn = rcu_dereference(fn->parent);
|
|
|
|
sn = FIB6_SUBTREE(pn);
|
|
|
|
if (sn && sn != fn)
|
|
|
|
fn = fib6_lookup(sn, NULL, saddr);
|
2014-10-20 20:42:43 +00:00
|
|
|
else
|
|
|
|
fn = pn;
|
|
|
|
if (fn->fn_flags & RTN_RTINFO)
|
|
|
|
return fn;
|
|
|
|
}
|
|
|
|
}
|
2006-08-05 06:20:06 +00:00
|
|
|
|
2017-10-06 19:06:06 +00:00
|
|
|
static bool ip6_hold_safe(struct net *net, struct rt6_info **prt,
|
|
|
|
bool null_fallback)
|
|
|
|
{
|
|
|
|
struct rt6_info *rt = *prt;
|
|
|
|
|
|
|
|
if (dst_hold_safe(&rt->dst))
|
|
|
|
return true;
|
|
|
|
if (null_fallback) {
|
|
|
|
rt = net->ipv6.ip6_null_entry;
|
|
|
|
dst_hold(&rt->dst);
|
|
|
|
} else {
|
|
|
|
rt = NULL;
|
|
|
|
}
|
|
|
|
*prt = rt;
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2008-03-04 21:48:30 +00:00
|
|
|
static struct rt6_info *ip6_pol_route_lookup(struct net *net,
|
|
|
|
struct fib6_table *table,
|
2011-03-12 21:22:43 +00:00
|
|
|
struct flowi6 *fl6, int flags)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2017-10-06 19:06:03 +00:00
|
|
|
struct rt6_info *rt, *rt_cache;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct fib6_node *fn;
|
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_lock();
|
2011-03-12 21:22:43 +00:00
|
|
|
fn = fib6_lookup(&table->tb6_root, &fl6->daddr, &fl6->saddr);
|
2006-08-05 06:20:06 +00:00
|
|
|
restart:
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rt = rcu_dereference(fn->leaf);
|
|
|
|
if (!rt) {
|
|
|
|
rt = net->ipv6.ip6_null_entry;
|
|
|
|
} else {
|
|
|
|
rt = rt6_device_match(net, rt, &fl6->saddr,
|
|
|
|
fl6->flowi6_oif, flags);
|
|
|
|
if (rt->rt6i_nsiblings && fl6->flowi6_oif == 0)
|
|
|
|
rt = rt6_multipath_select(rt, fl6,
|
|
|
|
fl6->flowi6_oif, flags);
|
|
|
|
}
|
2014-10-20 20:42:43 +00:00
|
|
|
if (rt == net->ipv6.ip6_null_entry) {
|
|
|
|
fn = fib6_backtrack(fn, &fl6->saddr);
|
|
|
|
if (fn)
|
|
|
|
goto restart;
|
|
|
|
}
|
2017-10-06 19:06:03 +00:00
|
|
|
/* Search through exception table */
|
|
|
|
rt_cache = rt6_find_cached_rt(rt, &fl6->daddr, &fl6->saddr);
|
|
|
|
if (rt_cache)
|
|
|
|
rt = rt_cache;
|
|
|
|
|
2017-10-06 19:06:06 +00:00
|
|
|
if (ip6_hold_safe(net, &rt, true))
|
|
|
|
dst_use_noref(&rt->dst, jiffies);
|
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2015-11-19 20:24:22 +00:00
|
|
|
|
2017-10-19 07:31:43 +00:00
|
|
|
trace_fib6_table_lookup(net, rt, table, fl6);
|
2015-11-19 20:24:22 +00:00
|
|
|
|
2006-08-05 06:20:06 +00:00
|
|
|
return rt;
|
|
|
|
|
|
|
|
}
|
|
|
|
|
2014-08-24 20:53:10 +00:00
|
|
|
struct dst_entry *ip6_route_lookup(struct net *net, struct flowi6 *fl6,
|
2011-09-05 14:05:44 +00:00
|
|
|
int flags)
|
|
|
|
{
|
|
|
|
return fib6_rule_lookup(net, fl6, flags, ip6_pol_route_lookup);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(ip6_route_lookup);
|
|
|
|
|
[IPV6]: Make address arguments const.
- net/ipv6/addrconf.c:
ipv6_get_ifaddr(), ipv6_dev_get_saddr()
- net/ipv6/mcast.c:
ipv6_sock_mc_join(), ipv6_sock_mc_drop(),
inet6_mc_check(),
ipv6_dev_mc_inc(), __ipv6_dev_mc_dec(), ipv6_dev_mc_dec(),
ipv6_chk_mcast_addr()
- net/ipv6/route.c:
rt6_lookup(), icmp6_dst_alloc()
- net/ipv6/ip6_output.c:
ip6_nd_hdr()
- net/ipv6/ndisc.c:
ndisc_send_ns(), ndisc_send_rs(), ndisc_send_redirect(),
ndisc_get_neigh(), __ndisc_send()
Signed-off-by: YOSHIFUJI Hideaki <yoshfuji@linux-ipv6.org>
2008-04-10 06:42:10 +00:00
|
|
|
struct rt6_info *rt6_lookup(struct net *net, const struct in6_addr *daddr,
|
|
|
|
const struct in6_addr *saddr, int oif, int strict)
|
2006-08-05 06:20:06 +00:00
|
|
|
{
|
2011-03-12 21:22:43 +00:00
|
|
|
struct flowi6 fl6 = {
|
|
|
|
.flowi6_oif = oif,
|
|
|
|
.daddr = *daddr,
|
2006-08-05 06:20:06 +00:00
|
|
|
};
|
|
|
|
struct dst_entry *dst;
|
2006-08-24 00:25:05 +00:00
|
|
|
int flags = strict ? RT6_LOOKUP_F_IFACE : 0;
|
2006-08-05 06:20:06 +00:00
|
|
|
|
2006-10-13 22:01:03 +00:00
|
|
|
if (saddr) {
|
2011-03-12 21:22:43 +00:00
|
|
|
memcpy(&fl6.saddr, saddr, sizeof(*saddr));
|
2006-10-13 22:01:03 +00:00
|
|
|
flags |= RT6_LOOKUP_F_HAS_SADDR;
|
|
|
|
}
|
|
|
|
|
2011-03-12 21:22:43 +00:00
|
|
|
dst = fib6_rule_lookup(net, &fl6, flags, ip6_pol_route_lookup);
|
2006-08-05 06:20:06 +00:00
|
|
|
if (dst->error == 0)
|
|
|
|
return (struct rt6_info *) dst;
|
|
|
|
|
|
|
|
dst_release(dst);
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
return NULL;
|
|
|
|
}
|
2007-02-22 13:05:40 +00:00
|
|
|
EXPORT_SYMBOL(rt6_lookup);
|
|
|
|
|
2006-08-05 06:20:06 +00:00
|
|
|
/* ip6_ins_rt is called with FREE table->tb6_lock.
|
2017-06-17 17:42:33 +00:00
|
|
|
* It takes new route entry, the addition fails by any reason the
|
|
|
|
* route is released.
|
|
|
|
* Caller must hold dst before calling it.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
|
2014-03-27 12:04:08 +00:00
|
|
|
static int __ip6_ins_rt(struct rt6_info *rt, struct nl_info *info,
|
2017-05-21 16:12:04 +00:00
|
|
|
struct mx6_config *mxc,
|
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
int err;
|
2006-08-05 06:20:06 +00:00
|
|
|
struct fib6_table *table;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-08-05 06:20:06 +00:00
|
|
|
table = rt->rt6i_table;
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
spin_lock_bh(&table->tb6_lock);
|
2017-05-21 16:12:04 +00:00
|
|
|
err = fib6_add(&table->tb6_root, rt, info, mxc, extack);
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
spin_unlock_bh(&table->tb6_lock);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2006-08-22 07:00:45 +00:00
|
|
|
int ip6_ins_rt(struct rt6_info *rt)
|
|
|
|
{
|
2015-01-05 22:57:44 +00:00
|
|
|
struct nl_info info = { .nl_net = dev_net(rt->dst.dev), };
|
|
|
|
struct mx6_config mxc = { .mx = NULL, };
|
|
|
|
|
2017-06-17 17:42:33 +00:00
|
|
|
/* Hold dst to account for the reference from the fib6 tree */
|
|
|
|
dst_hold(&rt->dst);
|
2017-05-21 16:12:04 +00:00
|
|
|
return __ip6_ins_rt(rt, &info, &mxc, NULL);
|
2006-08-22 07:00:45 +00:00
|
|
|
}
|
|
|
|
|
2017-08-17 19:17:20 +00:00
|
|
|
/* called with rcu_lock held */
|
|
|
|
static struct net_device *ip6_rt_get_dev_rcu(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
struct net_device *dev = rt->dst.dev;
|
|
|
|
|
|
|
|
if (rt->rt6i_flags & RTF_LOCAL) {
|
|
|
|
/* for copies of local routes, dst->dev needs to be the
|
|
|
|
* device if it is a master device, the master device if
|
|
|
|
* device is enslaved, and the loopback as the default
|
|
|
|
*/
|
|
|
|
if (netif_is_l3_slave(dev) &&
|
|
|
|
!rt6_need_strict(&rt->rt6i_dst.addr))
|
|
|
|
dev = l3mdev_master_dev_rcu(dev);
|
|
|
|
else if (!netif_is_l3_master(dev))
|
|
|
|
dev = dev_net(dev)->loopback_dev;
|
|
|
|
/* last case is netif_is_l3_master(dev) is true in which
|
|
|
|
* case we want dev returned to be dev
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
|
|
|
|
return dev;
|
|
|
|
}
|
|
|
|
|
2015-05-23 03:55:59 +00:00
|
|
|
static struct rt6_info *ip6_rt_cache_alloc(struct rt6_info *ort,
|
|
|
|
const struct in6_addr *daddr,
|
|
|
|
const struct in6_addr *saddr)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2017-08-17 19:17:20 +00:00
|
|
|
struct net_device *dev;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct rt6_info *rt;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Clone the route.
|
|
|
|
*/
|
|
|
|
|
2015-05-23 03:56:06 +00:00
|
|
|
if (ort->rt6i_flags & (RTF_CACHE | RTF_PCPU))
|
2015-05-23 03:56:05 +00:00
|
|
|
ort = (struct rt6_info *)ort->dst.from;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-08-17 19:17:20 +00:00
|
|
|
rcu_read_lock();
|
|
|
|
dev = ip6_rt_get_dev_rcu(ort);
|
|
|
|
rt = __ip6_dst_alloc(dev_net(dev), dev, 0);
|
|
|
|
rcu_read_unlock();
|
2015-05-23 03:56:05 +00:00
|
|
|
if (!rt)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
ip6_rt_copy_init(rt, ort);
|
|
|
|
rt->rt6i_flags |= RTF_CACHE;
|
|
|
|
rt->rt6i_metric = 0;
|
|
|
|
rt->dst.flags |= DST_HOST;
|
|
|
|
rt->rt6i_dst.addr = *daddr;
|
|
|
|
rt->rt6i_dst.plen = 128;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2015-05-23 03:56:05 +00:00
|
|
|
if (!rt6_is_gw_or_nonexthop(ort)) {
|
|
|
|
if (ort->rt6i_dst.plen != 128 &&
|
|
|
|
ipv6_addr_equal(&ort->rt6i_dst.addr, daddr))
|
|
|
|
rt->rt6i_flags |= RTF_ANYCAST;
|
2005-04-16 22:20:36 +00:00
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
2015-05-23 03:56:05 +00:00
|
|
|
if (rt->rt6i_src.plen && saddr) {
|
|
|
|
rt->rt6i_src.addr = *saddr;
|
|
|
|
rt->rt6i_src.plen = 128;
|
2015-05-23 03:55:59 +00:00
|
|
|
}
|
2015-05-23 03:56:05 +00:00
|
|
|
#endif
|
2006-03-21 00:55:51 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-03-21 00:55:51 +00:00
|
|
|
return rt;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2015-05-23 03:56:06 +00:00
|
|
|
static struct rt6_info *ip6_rt_pcpu_alloc(struct rt6_info *rt)
|
|
|
|
{
|
2017-08-17 19:17:20 +00:00
|
|
|
struct net_device *dev;
|
2015-05-23 03:56:06 +00:00
|
|
|
struct rt6_info *pcpu_rt;
|
|
|
|
|
2017-08-17 19:17:20 +00:00
|
|
|
rcu_read_lock();
|
|
|
|
dev = ip6_rt_get_dev_rcu(rt);
|
|
|
|
pcpu_rt = __ip6_dst_alloc(dev_net(dev), dev, rt->dst.flags);
|
|
|
|
rcu_read_unlock();
|
2015-05-23 03:56:06 +00:00
|
|
|
if (!pcpu_rt)
|
|
|
|
return NULL;
|
|
|
|
ip6_rt_copy_init(pcpu_rt, rt);
|
|
|
|
pcpu_rt->rt6i_protocol = rt->rt6i_protocol;
|
|
|
|
pcpu_rt->rt6i_flags |= RTF_PCPU;
|
|
|
|
return pcpu_rt;
|
|
|
|
}
|
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
/* It should be called with rcu_read_lock() acquired */
|
2015-05-23 03:56:06 +00:00
|
|
|
static struct rt6_info *rt6_get_pcpu_route(struct rt6_info *rt)
|
|
|
|
{
|
2015-08-14 18:05:53 +00:00
|
|
|
struct rt6_info *pcpu_rt, **p;
|
2015-05-23 03:56:06 +00:00
|
|
|
|
|
|
|
p = this_cpu_ptr(rt->rt6i_pcpu);
|
|
|
|
pcpu_rt = *p;
|
|
|
|
|
2017-10-06 19:06:06 +00:00
|
|
|
if (pcpu_rt && ip6_hold_safe(NULL, &pcpu_rt, false))
|
2015-08-14 18:05:53 +00:00
|
|
|
rt6_dst_from_metrics_check(pcpu_rt);
|
2017-10-06 19:06:06 +00:00
|
|
|
|
2015-08-14 18:05:53 +00:00
|
|
|
return pcpu_rt;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct rt6_info *rt6_make_pcpu_route(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
struct rt6_info *pcpu_rt, *prev, **p;
|
2015-05-23 03:56:06 +00:00
|
|
|
|
|
|
|
pcpu_rt = ip6_rt_pcpu_alloc(rt);
|
|
|
|
if (!pcpu_rt) {
|
|
|
|
struct net *net = dev_net(rt->dst.dev);
|
|
|
|
|
2015-08-14 18:05:54 +00:00
|
|
|
dst_hold(&net->ipv6.ip6_null_entry->dst);
|
|
|
|
return net->ipv6.ip6_null_entry;
|
2015-05-23 03:56:06 +00:00
|
|
|
}
|
|
|
|
|
2017-10-06 19:06:04 +00:00
|
|
|
dst_hold(&pcpu_rt->dst);
|
|
|
|
p = this_cpu_ptr(rt->rt6i_pcpu);
|
|
|
|
prev = cmpxchg(p, NULL, pcpu_rt);
|
2017-10-09 04:07:18 +00:00
|
|
|
BUG_ON(prev);
|
2017-10-06 19:06:04 +00:00
|
|
|
|
2015-05-23 03:56:06 +00:00
|
|
|
rt6_dst_from_metrics_check(pcpu_rt);
|
|
|
|
return pcpu_rt;
|
|
|
|
}
|
|
|
|
|
2017-10-06 19:05:57 +00:00
|
|
|
/* exception hash table implementation
|
|
|
|
*/
|
|
|
|
static DEFINE_SPINLOCK(rt6_exception_lock);
|
|
|
|
|
|
|
|
/* Remove rt6_ex from hash table and free the memory
|
|
|
|
* Caller must hold rt6_exception_lock
|
|
|
|
*/
|
|
|
|
static void rt6_remove_exception(struct rt6_exception_bucket *bucket,
|
|
|
|
struct rt6_exception *rt6_ex)
|
|
|
|
{
|
2017-10-10 17:01:16 +00:00
|
|
|
struct net *net;
|
2017-10-06 19:06:11 +00:00
|
|
|
|
2017-10-06 19:05:57 +00:00
|
|
|
if (!bucket || !rt6_ex)
|
|
|
|
return;
|
2017-10-10 17:01:16 +00:00
|
|
|
|
|
|
|
net = dev_net(rt6_ex->rt6i->dst.dev);
|
2017-10-06 19:05:57 +00:00
|
|
|
rt6_ex->rt6i->rt6i_node = NULL;
|
|
|
|
hlist_del_rcu(&rt6_ex->hlist);
|
|
|
|
rt6_release(rt6_ex->rt6i);
|
|
|
|
kfree_rcu(rt6_ex, rcu);
|
|
|
|
WARN_ON_ONCE(!bucket->depth);
|
|
|
|
bucket->depth--;
|
2017-10-06 19:06:11 +00:00
|
|
|
net->ipv6.rt6_stats->fib_rt_cache--;
|
2017-10-06 19:05:57 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Remove oldest rt6_ex in bucket and free the memory
|
|
|
|
* Caller must hold rt6_exception_lock
|
|
|
|
*/
|
|
|
|
static void rt6_exception_remove_oldest(struct rt6_exception_bucket *bucket)
|
|
|
|
{
|
|
|
|
struct rt6_exception *rt6_ex, *oldest = NULL;
|
|
|
|
|
|
|
|
if (!bucket)
|
|
|
|
return;
|
|
|
|
|
|
|
|
hlist_for_each_entry(rt6_ex, &bucket->chain, hlist) {
|
|
|
|
if (!oldest || time_before(rt6_ex->stamp, oldest->stamp))
|
|
|
|
oldest = rt6_ex;
|
|
|
|
}
|
|
|
|
rt6_remove_exception(bucket, oldest);
|
|
|
|
}
|
|
|
|
|
|
|
|
static u32 rt6_exception_hash(const struct in6_addr *dst,
|
|
|
|
const struct in6_addr *src)
|
|
|
|
{
|
|
|
|
static u32 seed __read_mostly;
|
|
|
|
u32 val;
|
|
|
|
|
|
|
|
net_get_random_once(&seed, sizeof(seed));
|
|
|
|
val = jhash(dst, sizeof(*dst), seed);
|
|
|
|
|
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
|
|
|
if (src)
|
|
|
|
val = jhash(src, sizeof(*src), val);
|
|
|
|
#endif
|
|
|
|
return hash_32(val, FIB6_EXCEPTION_BUCKET_SIZE_SHIFT);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Helper function to find the cached rt in the hash table
|
|
|
|
* and update bucket pointer to point to the bucket for this
|
|
|
|
* (daddr, saddr) pair
|
|
|
|
* Caller must hold rt6_exception_lock
|
|
|
|
*/
|
|
|
|
static struct rt6_exception *
|
|
|
|
__rt6_find_exception_spinlock(struct rt6_exception_bucket **bucket,
|
|
|
|
const struct in6_addr *daddr,
|
|
|
|
const struct in6_addr *saddr)
|
|
|
|
{
|
|
|
|
struct rt6_exception *rt6_ex;
|
|
|
|
u32 hval;
|
|
|
|
|
|
|
|
if (!(*bucket) || !daddr)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
hval = rt6_exception_hash(daddr, saddr);
|
|
|
|
*bucket += hval;
|
|
|
|
|
|
|
|
hlist_for_each_entry(rt6_ex, &(*bucket)->chain, hlist) {
|
|
|
|
struct rt6_info *rt6 = rt6_ex->rt6i;
|
|
|
|
bool matched = ipv6_addr_equal(daddr, &rt6->rt6i_dst.addr);
|
|
|
|
|
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
|
|
|
if (matched && saddr)
|
|
|
|
matched = ipv6_addr_equal(saddr, &rt6->rt6i_src.addr);
|
|
|
|
#endif
|
|
|
|
if (matched)
|
|
|
|
return rt6_ex;
|
|
|
|
}
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Helper function to find the cached rt in the hash table
|
|
|
|
* and update bucket pointer to point to the bucket for this
|
|
|
|
* (daddr, saddr) pair
|
|
|
|
* Caller must hold rcu_read_lock()
|
|
|
|
*/
|
|
|
|
static struct rt6_exception *
|
|
|
|
__rt6_find_exception_rcu(struct rt6_exception_bucket **bucket,
|
|
|
|
const struct in6_addr *daddr,
|
|
|
|
const struct in6_addr *saddr)
|
|
|
|
{
|
|
|
|
struct rt6_exception *rt6_ex;
|
|
|
|
u32 hval;
|
|
|
|
|
|
|
|
WARN_ON_ONCE(!rcu_read_lock_held());
|
|
|
|
|
|
|
|
if (!(*bucket) || !daddr)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
hval = rt6_exception_hash(daddr, saddr);
|
|
|
|
*bucket += hval;
|
|
|
|
|
|
|
|
hlist_for_each_entry_rcu(rt6_ex, &(*bucket)->chain, hlist) {
|
|
|
|
struct rt6_info *rt6 = rt6_ex->rt6i;
|
|
|
|
bool matched = ipv6_addr_equal(daddr, &rt6->rt6i_dst.addr);
|
|
|
|
|
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
|
|
|
if (matched && saddr)
|
|
|
|
matched = ipv6_addr_equal(saddr, &rt6->rt6i_src.addr);
|
|
|
|
#endif
|
|
|
|
if (matched)
|
|
|
|
return rt6_ex;
|
|
|
|
}
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int rt6_insert_exception(struct rt6_info *nrt,
|
|
|
|
struct rt6_info *ort)
|
|
|
|
{
|
2017-10-06 19:06:11 +00:00
|
|
|
struct net *net = dev_net(ort->dst.dev);
|
2017-10-06 19:05:57 +00:00
|
|
|
struct rt6_exception_bucket *bucket;
|
|
|
|
struct in6_addr *src_key = NULL;
|
|
|
|
struct rt6_exception *rt6_ex;
|
|
|
|
int err = 0;
|
|
|
|
|
|
|
|
/* ort can't be a cache or pcpu route */
|
|
|
|
if (ort->rt6i_flags & (RTF_CACHE | RTF_PCPU))
|
|
|
|
ort = (struct rt6_info *)ort->dst.from;
|
|
|
|
WARN_ON_ONCE(ort->rt6i_flags & (RTF_CACHE | RTF_PCPU));
|
|
|
|
|
|
|
|
spin_lock_bh(&rt6_exception_lock);
|
|
|
|
|
|
|
|
if (ort->exception_bucket_flushed) {
|
|
|
|
err = -EINVAL;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
bucket = rcu_dereference_protected(ort->rt6i_exception_bucket,
|
|
|
|
lockdep_is_held(&rt6_exception_lock));
|
|
|
|
if (!bucket) {
|
|
|
|
bucket = kcalloc(FIB6_EXCEPTION_BUCKET_SIZE, sizeof(*bucket),
|
|
|
|
GFP_ATOMIC);
|
|
|
|
if (!bucket) {
|
|
|
|
err = -ENOMEM;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
rcu_assign_pointer(ort->rt6i_exception_bucket, bucket);
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
|
|
|
/* rt6i_src.plen != 0 indicates ort is in subtree
|
|
|
|
* and exception table is indexed by a hash of
|
|
|
|
* both rt6i_dst and rt6i_src.
|
|
|
|
* Otherwise, the exception table is indexed by
|
|
|
|
* a hash of only rt6i_dst.
|
|
|
|
*/
|
|
|
|
if (ort->rt6i_src.plen)
|
|
|
|
src_key = &nrt->rt6i_src.addr;
|
|
|
|
#endif
|
2017-10-06 19:05:58 +00:00
|
|
|
|
|
|
|
/* Update rt6i_prefsrc as it could be changed
|
|
|
|
* in rt6_remove_prefsrc()
|
|
|
|
*/
|
|
|
|
nrt->rt6i_prefsrc = ort->rt6i_prefsrc;
|
2017-10-06 19:05:59 +00:00
|
|
|
/* rt6_mtu_change() might lower mtu on ort.
|
|
|
|
* Only insert this exception route if its mtu
|
|
|
|
* is less than ort's mtu value.
|
|
|
|
*/
|
|
|
|
if (nrt->rt6i_pmtu >= dst_mtu(&ort->dst)) {
|
|
|
|
err = -EINVAL;
|
|
|
|
goto out;
|
|
|
|
}
|
2017-10-06 19:05:58 +00:00
|
|
|
|
2017-10-06 19:05:57 +00:00
|
|
|
rt6_ex = __rt6_find_exception_spinlock(&bucket, &nrt->rt6i_dst.addr,
|
|
|
|
src_key);
|
|
|
|
if (rt6_ex)
|
|
|
|
rt6_remove_exception(bucket, rt6_ex);
|
|
|
|
|
|
|
|
rt6_ex = kzalloc(sizeof(*rt6_ex), GFP_ATOMIC);
|
|
|
|
if (!rt6_ex) {
|
|
|
|
err = -ENOMEM;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
rt6_ex->rt6i = nrt;
|
|
|
|
rt6_ex->stamp = jiffies;
|
|
|
|
atomic_inc(&nrt->rt6i_ref);
|
|
|
|
nrt->rt6i_node = ort->rt6i_node;
|
|
|
|
hlist_add_head_rcu(&rt6_ex->hlist, &bucket->chain);
|
|
|
|
bucket->depth++;
|
2017-10-06 19:06:11 +00:00
|
|
|
net->ipv6.rt6_stats->fib_rt_cache++;
|
2017-10-06 19:05:57 +00:00
|
|
|
|
|
|
|
if (bucket->depth > FIB6_MAX_DEPTH)
|
|
|
|
rt6_exception_remove_oldest(bucket);
|
|
|
|
|
|
|
|
out:
|
|
|
|
spin_unlock_bh(&rt6_exception_lock);
|
|
|
|
|
|
|
|
/* Update fn->fn_sernum to invalidate all cached dst */
|
2017-10-19 14:07:10 +00:00
|
|
|
if (!err) {
|
2017-10-06 19:05:57 +00:00
|
|
|
fib6_update_sernum(ort);
|
2017-10-19 14:07:10 +00:00
|
|
|
fib6_force_start_gc(net);
|
|
|
|
}
|
2017-10-06 19:05:57 +00:00
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
void rt6_flush_exceptions(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
struct rt6_exception_bucket *bucket;
|
|
|
|
struct rt6_exception *rt6_ex;
|
|
|
|
struct hlist_node *tmp;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
spin_lock_bh(&rt6_exception_lock);
|
|
|
|
/* Prevent rt6_insert_exception() to recreate the bucket list */
|
|
|
|
rt->exception_bucket_flushed = 1;
|
|
|
|
|
|
|
|
bucket = rcu_dereference_protected(rt->rt6i_exception_bucket,
|
|
|
|
lockdep_is_held(&rt6_exception_lock));
|
|
|
|
if (!bucket)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
for (i = 0; i < FIB6_EXCEPTION_BUCKET_SIZE; i++) {
|
|
|
|
hlist_for_each_entry_safe(rt6_ex, tmp, &bucket->chain, hlist)
|
|
|
|
rt6_remove_exception(bucket, rt6_ex);
|
|
|
|
WARN_ON_ONCE(bucket->depth);
|
|
|
|
bucket++;
|
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
|
|
|
spin_unlock_bh(&rt6_exception_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Find cached rt in the hash table inside passed in rt
|
|
|
|
* Caller has to hold rcu_read_lock()
|
|
|
|
*/
|
|
|
|
static struct rt6_info *rt6_find_cached_rt(struct rt6_info *rt,
|
|
|
|
struct in6_addr *daddr,
|
|
|
|
struct in6_addr *saddr)
|
|
|
|
{
|
|
|
|
struct rt6_exception_bucket *bucket;
|
|
|
|
struct in6_addr *src_key = NULL;
|
|
|
|
struct rt6_exception *rt6_ex;
|
|
|
|
struct rt6_info *res = NULL;
|
|
|
|
|
|
|
|
bucket = rcu_dereference(rt->rt6i_exception_bucket);
|
|
|
|
|
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
|
|
|
/* rt6i_src.plen != 0 indicates rt is in subtree
|
|
|
|
* and exception table is indexed by a hash of
|
|
|
|
* both rt6i_dst and rt6i_src.
|
|
|
|
* Otherwise, the exception table is indexed by
|
|
|
|
* a hash of only rt6i_dst.
|
|
|
|
*/
|
|
|
|
if (rt->rt6i_src.plen)
|
|
|
|
src_key = saddr;
|
|
|
|
#endif
|
|
|
|
rt6_ex = __rt6_find_exception_rcu(&bucket, daddr, src_key);
|
|
|
|
|
|
|
|
if (rt6_ex && !rt6_check_expired(rt6_ex->rt6i))
|
|
|
|
res = rt6_ex->rt6i;
|
|
|
|
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Remove the passed in cached rt from the hash table that contains it */
|
|
|
|
int rt6_remove_exception_rt(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
struct rt6_info *from = (struct rt6_info *)rt->dst.from;
|
|
|
|
struct rt6_exception_bucket *bucket;
|
|
|
|
struct in6_addr *src_key = NULL;
|
|
|
|
struct rt6_exception *rt6_ex;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
if (!from ||
|
2017-10-10 18:10:30 +00:00
|
|
|
!(rt->rt6i_flags & RTF_CACHE))
|
2017-10-06 19:05:57 +00:00
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
if (!rcu_access_pointer(from->rt6i_exception_bucket))
|
|
|
|
return -ENOENT;
|
|
|
|
|
|
|
|
spin_lock_bh(&rt6_exception_lock);
|
|
|
|
bucket = rcu_dereference_protected(from->rt6i_exception_bucket,
|
|
|
|
lockdep_is_held(&rt6_exception_lock));
|
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
|
|
|
/* rt6i_src.plen != 0 indicates 'from' is in subtree
|
|
|
|
* and exception table is indexed by a hash of
|
|
|
|
* both rt6i_dst and rt6i_src.
|
|
|
|
* Otherwise, the exception table is indexed by
|
|
|
|
* a hash of only rt6i_dst.
|
|
|
|
*/
|
|
|
|
if (from->rt6i_src.plen)
|
|
|
|
src_key = &rt->rt6i_src.addr;
|
|
|
|
#endif
|
|
|
|
rt6_ex = __rt6_find_exception_spinlock(&bucket,
|
|
|
|
&rt->rt6i_dst.addr,
|
|
|
|
src_key);
|
|
|
|
if (rt6_ex) {
|
|
|
|
rt6_remove_exception(bucket, rt6_ex);
|
|
|
|
err = 0;
|
|
|
|
} else {
|
|
|
|
err = -ENOENT;
|
|
|
|
}
|
|
|
|
|
|
|
|
spin_unlock_bh(&rt6_exception_lock);
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Find rt6_ex which contains the passed in rt cache and
|
|
|
|
* refresh its stamp
|
|
|
|
*/
|
|
|
|
static void rt6_update_exception_stamp_rt(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
struct rt6_info *from = (struct rt6_info *)rt->dst.from;
|
|
|
|
struct rt6_exception_bucket *bucket;
|
|
|
|
struct in6_addr *src_key = NULL;
|
|
|
|
struct rt6_exception *rt6_ex;
|
|
|
|
|
|
|
|
if (!from ||
|
2017-10-10 18:10:30 +00:00
|
|
|
!(rt->rt6i_flags & RTF_CACHE))
|
2017-10-06 19:05:57 +00:00
|
|
|
return;
|
|
|
|
|
|
|
|
rcu_read_lock();
|
|
|
|
bucket = rcu_dereference(from->rt6i_exception_bucket);
|
|
|
|
|
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
|
|
|
/* rt6i_src.plen != 0 indicates 'from' is in subtree
|
|
|
|
* and exception table is indexed by a hash of
|
|
|
|
* both rt6i_dst and rt6i_src.
|
|
|
|
* Otherwise, the exception table is indexed by
|
|
|
|
* a hash of only rt6i_dst.
|
|
|
|
*/
|
|
|
|
if (from->rt6i_src.plen)
|
|
|
|
src_key = &rt->rt6i_src.addr;
|
|
|
|
#endif
|
|
|
|
rt6_ex = __rt6_find_exception_rcu(&bucket,
|
|
|
|
&rt->rt6i_dst.addr,
|
|
|
|
src_key);
|
|
|
|
if (rt6_ex)
|
|
|
|
rt6_ex->stamp = jiffies;
|
|
|
|
|
|
|
|
rcu_read_unlock();
|
|
|
|
}
|
|
|
|
|
2017-10-06 19:05:58 +00:00
|
|
|
static void rt6_exceptions_remove_prefsrc(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
struct rt6_exception_bucket *bucket;
|
|
|
|
struct rt6_exception *rt6_ex;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
bucket = rcu_dereference_protected(rt->rt6i_exception_bucket,
|
|
|
|
lockdep_is_held(&rt6_exception_lock));
|
|
|
|
|
|
|
|
if (bucket) {
|
|
|
|
for (i = 0; i < FIB6_EXCEPTION_BUCKET_SIZE; i++) {
|
|
|
|
hlist_for_each_entry(rt6_ex, &bucket->chain, hlist) {
|
|
|
|
rt6_ex->rt6i->rt6i_prefsrc.plen = 0;
|
|
|
|
}
|
|
|
|
bucket++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-10-06 19:05:59 +00:00
|
|
|
static void rt6_exceptions_update_pmtu(struct rt6_info *rt, int mtu)
|
|
|
|
{
|
|
|
|
struct rt6_exception_bucket *bucket;
|
|
|
|
struct rt6_exception *rt6_ex;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
bucket = rcu_dereference_protected(rt->rt6i_exception_bucket,
|
|
|
|
lockdep_is_held(&rt6_exception_lock));
|
|
|
|
|
|
|
|
if (bucket) {
|
|
|
|
for (i = 0; i < FIB6_EXCEPTION_BUCKET_SIZE; i++) {
|
|
|
|
hlist_for_each_entry(rt6_ex, &bucket->chain, hlist) {
|
|
|
|
struct rt6_info *entry = rt6_ex->rt6i;
|
|
|
|
/* For RTF_CACHE with rt6i_pmtu == 0
|
|
|
|
* (i.e. a redirected route),
|
|
|
|
* the metrics of its rt->dst.from has already
|
|
|
|
* been updated.
|
|
|
|
*/
|
|
|
|
if (entry->rt6i_pmtu && entry->rt6i_pmtu > mtu)
|
|
|
|
entry->rt6i_pmtu = mtu;
|
|
|
|
}
|
|
|
|
bucket++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-10-06 19:06:00 +00:00
|
|
|
#define RTF_CACHE_GATEWAY (RTF_GATEWAY | RTF_CACHE)
|
|
|
|
|
|
|
|
static void rt6_exceptions_clean_tohost(struct rt6_info *rt,
|
|
|
|
struct in6_addr *gateway)
|
|
|
|
{
|
|
|
|
struct rt6_exception_bucket *bucket;
|
|
|
|
struct rt6_exception *rt6_ex;
|
|
|
|
struct hlist_node *tmp;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
if (!rcu_access_pointer(rt->rt6i_exception_bucket))
|
|
|
|
return;
|
|
|
|
|
|
|
|
spin_lock_bh(&rt6_exception_lock);
|
|
|
|
bucket = rcu_dereference_protected(rt->rt6i_exception_bucket,
|
|
|
|
lockdep_is_held(&rt6_exception_lock));
|
|
|
|
|
|
|
|
if (bucket) {
|
|
|
|
for (i = 0; i < FIB6_EXCEPTION_BUCKET_SIZE; i++) {
|
|
|
|
hlist_for_each_entry_safe(rt6_ex, tmp,
|
|
|
|
&bucket->chain, hlist) {
|
|
|
|
struct rt6_info *entry = rt6_ex->rt6i;
|
|
|
|
|
|
|
|
if ((entry->rt6i_flags & RTF_CACHE_GATEWAY) ==
|
|
|
|
RTF_CACHE_GATEWAY &&
|
|
|
|
ipv6_addr_equal(gateway,
|
|
|
|
&entry->rt6i_gateway)) {
|
|
|
|
rt6_remove_exception(bucket, rt6_ex);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
bucket++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
spin_unlock_bh(&rt6_exception_lock);
|
|
|
|
}
|
|
|
|
|
2017-10-06 19:06:01 +00:00
|
|
|
static void rt6_age_examine_exception(struct rt6_exception_bucket *bucket,
|
|
|
|
struct rt6_exception *rt6_ex,
|
|
|
|
struct fib6_gc_args *gc_args,
|
|
|
|
unsigned long now)
|
|
|
|
{
|
|
|
|
struct rt6_info *rt = rt6_ex->rt6i;
|
|
|
|
|
2017-10-19 14:07:11 +00:00
|
|
|
/* we are pruning and obsoleting aged-out and non gateway exceptions
|
|
|
|
* even if others have still references to them, so that on next
|
|
|
|
* dst_check() such references can be dropped.
|
|
|
|
* EXPIRES exceptions - e.g. pmtu-generated ones are pruned when
|
|
|
|
* expired, independently from their aging, as per RFC 8201 section 4
|
|
|
|
*/
|
|
|
|
if (!(rt->rt6i_flags & RTF_EXPIRES) &&
|
2017-10-06 19:06:01 +00:00
|
|
|
time_after_eq(now, rt->dst.lastuse + gc_args->timeout)) {
|
|
|
|
RT6_TRACE("aging clone %p\n", rt);
|
|
|
|
rt6_remove_exception(bucket, rt6_ex);
|
|
|
|
return;
|
|
|
|
} else if (rt->rt6i_flags & RTF_GATEWAY) {
|
|
|
|
struct neighbour *neigh;
|
|
|
|
__u8 neigh_flags = 0;
|
|
|
|
|
|
|
|
neigh = dst_neigh_lookup(&rt->dst, &rt->rt6i_gateway);
|
|
|
|
if (neigh) {
|
|
|
|
neigh_flags = neigh->flags;
|
|
|
|
neigh_release(neigh);
|
|
|
|
}
|
|
|
|
if (!(neigh_flags & NTF_ROUTER)) {
|
|
|
|
RT6_TRACE("purging route %p via non-router but gateway\n",
|
|
|
|
rt);
|
|
|
|
rt6_remove_exception(bucket, rt6_ex);
|
|
|
|
return;
|
|
|
|
}
|
2017-10-19 14:07:11 +00:00
|
|
|
} else if (__rt6_check_expired(rt)) {
|
|
|
|
RT6_TRACE("purging expired route %p\n", rt);
|
|
|
|
rt6_remove_exception(bucket, rt6_ex);
|
|
|
|
return;
|
2017-10-06 19:06:01 +00:00
|
|
|
}
|
|
|
|
gc_args->more++;
|
|
|
|
}
|
|
|
|
|
|
|
|
void rt6_age_exceptions(struct rt6_info *rt,
|
|
|
|
struct fib6_gc_args *gc_args,
|
|
|
|
unsigned long now)
|
|
|
|
{
|
|
|
|
struct rt6_exception_bucket *bucket;
|
|
|
|
struct rt6_exception *rt6_ex;
|
|
|
|
struct hlist_node *tmp;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
if (!rcu_access_pointer(rt->rt6i_exception_bucket))
|
|
|
|
return;
|
|
|
|
|
|
|
|
spin_lock_bh(&rt6_exception_lock);
|
|
|
|
bucket = rcu_dereference_protected(rt->rt6i_exception_bucket,
|
|
|
|
lockdep_is_held(&rt6_exception_lock));
|
|
|
|
|
|
|
|
if (bucket) {
|
|
|
|
for (i = 0; i < FIB6_EXCEPTION_BUCKET_SIZE; i++) {
|
|
|
|
hlist_for_each_entry_safe(rt6_ex, tmp,
|
|
|
|
&bucket->chain, hlist) {
|
|
|
|
rt6_age_examine_exception(bucket, rt6_ex,
|
|
|
|
gc_args, now);
|
|
|
|
}
|
|
|
|
bucket++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
spin_unlock_bh(&rt6_exception_lock);
|
|
|
|
}
|
|
|
|
|
2016-06-13 20:44:19 +00:00
|
|
|
struct rt6_info *ip6_pol_route(struct net *net, struct fib6_table *table,
|
|
|
|
int oif, struct flowi6 *fl6, int flags)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2014-10-20 20:42:45 +00:00
|
|
|
struct fib6_node *fn, *saved_fn;
|
2017-10-06 19:06:03 +00:00
|
|
|
struct rt6_info *rt, *rt_cache;
|
2006-08-05 06:20:06 +00:00
|
|
|
int strict = 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-08-24 00:25:05 +00:00
|
|
|
strict |= flags & RT6_LOOKUP_F_IFACE;
|
2016-10-24 19:27:23 +00:00
|
|
|
strict |= flags & RT6_LOOKUP_F_IGNORE_LINKSTATE;
|
2014-10-20 20:42:45 +00:00
|
|
|
if (net->ipv6.devconf_all->forwarding == 0)
|
|
|
|
strict |= RT6_LOOKUP_F_REACHABLE;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_lock();
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-03-12 21:22:43 +00:00
|
|
|
fn = fib6_lookup(&table->tb6_root, &fl6->daddr, &fl6->saddr);
|
2014-10-20 20:42:45 +00:00
|
|
|
saved_fn = fn;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2015-10-12 18:47:10 +00:00
|
|
|
if (fl6->flowi6_flags & FLOWI_FLAG_SKIP_NH_OIF)
|
|
|
|
oif = 0;
|
|
|
|
|
2014-10-20 20:42:43 +00:00
|
|
|
redo_rt6_select:
|
2017-10-06 19:06:08 +00:00
|
|
|
rt = rt6_select(net, fn, oif, strict);
|
2013-06-28 15:35:48 +00:00
|
|
|
if (rt->rt6i_nsiblings)
|
2014-10-20 20:42:45 +00:00
|
|
|
rt = rt6_multipath_select(rt, fl6, oif, strict);
|
2014-10-20 20:42:43 +00:00
|
|
|
if (rt == net->ipv6.ip6_null_entry) {
|
|
|
|
fn = fib6_backtrack(fn, &fl6->saddr);
|
|
|
|
if (fn)
|
|
|
|
goto redo_rt6_select;
|
2014-10-20 20:42:45 +00:00
|
|
|
else if (strict & RT6_LOOKUP_F_REACHABLE) {
|
|
|
|
/* also consider unreachable route */
|
|
|
|
strict &= ~RT6_LOOKUP_F_REACHABLE;
|
|
|
|
fn = saved_fn;
|
|
|
|
goto redo_rt6_select;
|
|
|
|
}
|
2014-10-20 20:42:43 +00:00
|
|
|
}
|
|
|
|
|
2017-10-06 19:06:03 +00:00
|
|
|
/*Search through exception table */
|
|
|
|
rt_cache = rt6_find_cached_rt(rt, &fl6->daddr, &fl6->saddr);
|
|
|
|
if (rt_cache)
|
|
|
|
rt = rt_cache;
|
2006-03-21 00:59:08 +00:00
|
|
|
|
2017-10-06 19:06:06 +00:00
|
|
|
if (rt == net->ipv6.ip6_null_entry) {
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2017-10-06 19:06:06 +00:00
|
|
|
dst_hold(&rt->dst);
|
2017-10-19 07:31:43 +00:00
|
|
|
trace_fib6_table_lookup(net, rt, table, fl6);
|
2017-10-06 19:06:06 +00:00
|
|
|
return rt;
|
|
|
|
} else if (rt->rt6i_flags & RTF_CACHE) {
|
|
|
|
if (ip6_hold_safe(net, &rt, true)) {
|
|
|
|
dst_use_noref(&rt->dst, jiffies);
|
|
|
|
rt6_dst_from_metrics_check(rt);
|
|
|
|
}
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2017-10-19 07:31:43 +00:00
|
|
|
trace_fib6_table_lookup(net, rt, table, fl6);
|
2015-05-23 03:56:06 +00:00
|
|
|
return rt;
|
2015-05-23 03:56:03 +00:00
|
|
|
} else if (unlikely((fl6->flowi6_flags & FLOWI_FLAG_KNOWN_NH) &&
|
|
|
|
!(rt->rt6i_flags & RTF_GATEWAY))) {
|
|
|
|
/* Create a RTF_CACHE clone which will not be
|
|
|
|
* owned by the fib6 tree. It is for the special case where
|
|
|
|
* the daddr in the skb during the neighbor look-up is different
|
|
|
|
* from the fl6->daddr used to look-up route here.
|
|
|
|
*/
|
|
|
|
|
|
|
|
struct rt6_info *uncached_rt;
|
|
|
|
|
2017-10-06 19:06:06 +00:00
|
|
|
if (ip6_hold_safe(net, &rt, true)) {
|
|
|
|
dst_use_noref(&rt->dst, jiffies);
|
|
|
|
} else {
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2017-10-06 19:06:06 +00:00
|
|
|
uncached_rt = rt;
|
|
|
|
goto uncached_rt_out;
|
|
|
|
}
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2015-05-23 03:56:06 +00:00
|
|
|
|
2015-05-23 03:56:03 +00:00
|
|
|
uncached_rt = ip6_rt_cache_alloc(rt, &fl6->daddr, NULL);
|
|
|
|
dst_release(&rt->dst);
|
2006-08-05 06:20:06 +00:00
|
|
|
|
2017-06-17 17:42:33 +00:00
|
|
|
if (uncached_rt) {
|
|
|
|
/* Uncached_rt's refcnt is taken during ip6_rt_cache_alloc()
|
|
|
|
* No need for another dst_hold()
|
|
|
|
*/
|
2015-05-23 03:56:04 +00:00
|
|
|
rt6_uncached_list_add(uncached_rt);
|
2017-10-06 19:06:11 +00:00
|
|
|
atomic_inc(&net->ipv6.rt6_stats->fib_rt_uncache);
|
2017-06-17 17:42:33 +00:00
|
|
|
} else {
|
2015-05-23 03:56:03 +00:00
|
|
|
uncached_rt = net->ipv6.ip6_null_entry;
|
2017-06-17 17:42:33 +00:00
|
|
|
dst_hold(&uncached_rt->dst);
|
|
|
|
}
|
2015-11-19 20:24:22 +00:00
|
|
|
|
2017-10-06 19:06:06 +00:00
|
|
|
uncached_rt_out:
|
2017-10-19 07:31:43 +00:00
|
|
|
trace_fib6_table_lookup(net, uncached_rt, table, fl6);
|
2015-05-23 03:56:03 +00:00
|
|
|
return uncached_rt;
|
|
|
|
|
2015-05-23 03:56:06 +00:00
|
|
|
} else {
|
|
|
|
/* Get a percpu copy */
|
|
|
|
|
|
|
|
struct rt6_info *pcpu_rt;
|
|
|
|
|
2017-10-06 19:06:06 +00:00
|
|
|
dst_use_noref(&rt->dst, jiffies);
|
2017-10-09 04:07:18 +00:00
|
|
|
local_bh_disable();
|
2015-05-23 03:56:06 +00:00
|
|
|
pcpu_rt = rt6_get_pcpu_route(rt);
|
|
|
|
|
2017-10-09 04:07:18 +00:00
|
|
|
if (!pcpu_rt) {
|
2017-10-06 19:06:04 +00:00
|
|
|
/* atomic_inc_not_zero() is needed when using rcu */
|
|
|
|
if (atomic_inc_not_zero(&rt->rt6i_ref)) {
|
2017-10-09 04:07:18 +00:00
|
|
|
/* No dst_hold() on rt is needed because grabbing
|
2017-10-06 19:06:04 +00:00
|
|
|
* rt->rt6i_ref makes sure rt can't be released.
|
|
|
|
*/
|
|
|
|
pcpu_rt = rt6_make_pcpu_route(rt);
|
|
|
|
rt6_release(rt);
|
|
|
|
} else {
|
|
|
|
/* rt is already removed from tree */
|
|
|
|
pcpu_rt = net->ipv6.ip6_null_entry;
|
|
|
|
dst_hold(&pcpu_rt->dst);
|
|
|
|
}
|
2015-08-14 18:05:54 +00:00
|
|
|
}
|
2017-10-09 04:07:18 +00:00
|
|
|
local_bh_enable();
|
|
|
|
rcu_read_unlock();
|
2017-10-19 07:31:43 +00:00
|
|
|
trace_fib6_table_lookup(net, pcpu_rt, table, fl6);
|
2015-05-23 03:56:06 +00:00
|
|
|
return pcpu_rt;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2016-06-13 20:44:19 +00:00
|
|
|
EXPORT_SYMBOL_GPL(ip6_pol_route);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2008-03-04 21:48:30 +00:00
|
|
|
static struct rt6_info *ip6_pol_route_input(struct net *net, struct fib6_table *table,
|
2011-03-12 21:22:43 +00:00
|
|
|
struct flowi6 *fl6, int flags)
|
2007-10-15 20:02:51 +00:00
|
|
|
{
|
2011-03-12 21:22:43 +00:00
|
|
|
return ip6_pol_route(net, table, fl6->flowi6_iif, fl6, flags);
|
2007-10-15 20:02:51 +00:00
|
|
|
}
|
|
|
|
|
2016-09-16 19:59:08 +00:00
|
|
|
struct dst_entry *ip6_route_input_lookup(struct net *net,
|
|
|
|
struct net_device *dev,
|
|
|
|
struct flowi6 *fl6, int flags)
|
2012-04-01 04:03:45 +00:00
|
|
|
{
|
|
|
|
if (rt6_need_strict(&fl6->daddr) && dev->type != ARPHRD_PIMREG)
|
|
|
|
flags |= RT6_LOOKUP_F_IFACE;
|
|
|
|
|
|
|
|
return fib6_rule_lookup(net, fl6, flags, ip6_pol_route_input);
|
|
|
|
}
|
2016-09-16 19:59:08 +00:00
|
|
|
EXPORT_SYMBOL_GPL(ip6_route_input_lookup);
|
2012-04-01 04:03:45 +00:00
|
|
|
|
2017-08-23 07:58:29 +00:00
|
|
|
static void ip6_multipath_l3_keys(const struct sk_buff *skb,
|
|
|
|
struct flow_keys *keys)
|
|
|
|
{
|
|
|
|
const struct ipv6hdr *outer_iph = ipv6_hdr(skb);
|
|
|
|
const struct ipv6hdr *key_iph = outer_iph;
|
|
|
|
const struct ipv6hdr *inner_iph;
|
|
|
|
const struct icmp6hdr *icmph;
|
|
|
|
struct ipv6hdr _inner_iph;
|
|
|
|
|
|
|
|
if (likely(outer_iph->nexthdr != IPPROTO_ICMPV6))
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
icmph = icmp6_hdr(skb);
|
|
|
|
if (icmph->icmp6_type != ICMPV6_DEST_UNREACH &&
|
|
|
|
icmph->icmp6_type != ICMPV6_PKT_TOOBIG &&
|
|
|
|
icmph->icmp6_type != ICMPV6_TIME_EXCEED &&
|
|
|
|
icmph->icmp6_type != ICMPV6_PARAMPROB)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
inner_iph = skb_header_pointer(skb,
|
|
|
|
skb_transport_offset(skb) + sizeof(*icmph),
|
|
|
|
sizeof(_inner_iph), &_inner_iph);
|
|
|
|
if (!inner_iph)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
key_iph = inner_iph;
|
|
|
|
out:
|
|
|
|
memset(keys, 0, sizeof(*keys));
|
|
|
|
keys->control.addr_type = FLOW_DISSECTOR_KEY_IPV6_ADDRS;
|
|
|
|
keys->addrs.v6addrs.src = key_iph->saddr;
|
|
|
|
keys->addrs.v6addrs.dst = key_iph->daddr;
|
|
|
|
keys->tags.flow_label = ip6_flowinfo(key_iph);
|
|
|
|
keys->basic.ip_proto = key_iph->nexthdr;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* if skb is set it will be used and fl6 can be NULL */
|
|
|
|
u32 rt6_multipath_hash(const struct flowi6 *fl6, const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct flow_keys hash_keys;
|
|
|
|
|
|
|
|
if (skb) {
|
|
|
|
ip6_multipath_l3_keys(skb, &hash_keys);
|
|
|
|
return flow_hash_from_keys(&hash_keys);
|
|
|
|
}
|
|
|
|
|
|
|
|
return get_hash_from_flowi6(fl6);
|
|
|
|
}
|
|
|
|
|
2006-08-05 06:20:06 +00:00
|
|
|
void ip6_route_input(struct sk_buff *skb)
|
|
|
|
{
|
2011-04-22 04:53:02 +00:00
|
|
|
const struct ipv6hdr *iph = ipv6_hdr(skb);
|
2008-03-25 12:47:49 +00:00
|
|
|
struct net *net = dev_net(skb->dev);
|
2006-10-13 22:01:03 +00:00
|
|
|
int flags = RT6_LOOKUP_F_HAS_SADDR;
|
2015-08-20 11:56:31 +00:00
|
|
|
struct ip_tunnel_info *tun_info;
|
2011-03-12 21:22:43 +00:00
|
|
|
struct flowi6 fl6 = {
|
2016-09-10 19:09:57 +00:00
|
|
|
.flowi6_iif = skb->dev->ifindex,
|
2011-03-12 21:22:43 +00:00
|
|
|
.daddr = iph->daddr,
|
|
|
|
.saddr = iph->saddr,
|
2013-01-13 05:01:51 +00:00
|
|
|
.flowlabel = ip6_flowinfo(iph),
|
2011-03-12 21:22:43 +00:00
|
|
|
.flowi6_mark = skb->mark,
|
|
|
|
.flowi6_proto = iph->nexthdr,
|
2006-08-05 06:20:06 +00:00
|
|
|
};
|
2006-10-13 22:01:03 +00:00
|
|
|
|
2015-08-20 11:56:31 +00:00
|
|
|
tun_info = skb_tunnel_info(skb);
|
2015-08-28 18:48:19 +00:00
|
|
|
if (tun_info && !(tun_info->mode & IP_TUNNEL_INFO_TX))
|
2015-08-20 11:56:31 +00:00
|
|
|
fl6.flowi6_tun_key.tun_id = tun_info->key.tun_id;
|
2017-08-23 07:58:29 +00:00
|
|
|
if (unlikely(fl6.flowi6_proto == IPPROTO_ICMPV6))
|
|
|
|
fl6.mp_hash = rt6_multipath_hash(&fl6, skb);
|
2015-08-20 11:56:26 +00:00
|
|
|
skb_dst_drop(skb);
|
2012-04-01 04:03:45 +00:00
|
|
|
skb_dst_set(skb, ip6_route_input_lookup(net, skb->dev, &fl6, flags));
|
2006-08-05 06:20:06 +00:00
|
|
|
}
|
|
|
|
|
2008-03-04 21:48:30 +00:00
|
|
|
static struct rt6_info *ip6_pol_route_output(struct net *net, struct fib6_table *table,
|
2011-03-12 21:22:43 +00:00
|
|
|
struct flowi6 *fl6, int flags)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2011-03-12 21:22:43 +00:00
|
|
|
return ip6_pol_route(net, table, fl6->flowi6_oif, fl6, flags);
|
2006-08-05 06:20:06 +00:00
|
|
|
}
|
|
|
|
|
2016-01-29 11:30:19 +00:00
|
|
|
struct dst_entry *ip6_route_output_flags(struct net *net, const struct sock *sk,
|
|
|
|
struct flowi6 *fl6, int flags)
|
2006-08-05 06:20:06 +00:00
|
|
|
{
|
2015-10-21 15:42:22 +00:00
|
|
|
bool any_src;
|
2006-08-05 06:20:06 +00:00
|
|
|
|
2016-09-10 19:09:56 +00:00
|
|
|
if (rt6_need_strict(&fl6->daddr)) {
|
|
|
|
struct dst_entry *dst;
|
|
|
|
|
|
|
|
dst = l3mdev_link_scope_lookup(net, fl6);
|
|
|
|
if (dst)
|
|
|
|
return dst;
|
|
|
|
}
|
2015-10-12 18:47:10 +00:00
|
|
|
|
2012-08-08 21:53:36 +00:00
|
|
|
fl6->flowi6_iif = LOOPBACK_IFINDEX;
|
2012-06-25 15:42:26 +00:00
|
|
|
|
2015-10-21 15:42:22 +00:00
|
|
|
any_src = ipv6_addr_any(&fl6->saddr);
|
2015-09-28 17:12:13 +00:00
|
|
|
if ((sk && sk->sk_bound_dev_if) || rt6_need_strict(&fl6->daddr) ||
|
2015-10-21 15:42:22 +00:00
|
|
|
(fl6->flowi6_oif && any_src))
|
2006-08-24 00:25:05 +00:00
|
|
|
flags |= RT6_LOOKUP_F_IFACE;
|
2006-08-05 06:20:06 +00:00
|
|
|
|
2015-10-21 15:42:22 +00:00
|
|
|
if (!any_src)
|
2006-10-13 22:01:03 +00:00
|
|
|
flags |= RT6_LOOKUP_F_HAS_SADDR;
|
2010-03-07 00:14:44 +00:00
|
|
|
else if (sk)
|
|
|
|
flags |= rt6_srcprefs2flags(inet6_sk(sk)->srcprefs);
|
2006-10-13 22:01:03 +00:00
|
|
|
|
2011-03-12 21:22:43 +00:00
|
|
|
return fib6_rule_lookup(net, fl6, flags, ip6_pol_route_output);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2016-01-29 11:30:19 +00:00
|
|
|
EXPORT_SYMBOL_GPL(ip6_route_output_flags);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-03-01 22:59:04 +00:00
|
|
|
struct dst_entry *ip6_blackhole_route(struct net *net, struct dst_entry *dst_orig)
|
2007-05-25 01:17:54 +00:00
|
|
|
{
|
2011-04-28 21:13:38 +00:00
|
|
|
struct rt6_info *rt, *ort = (struct rt6_info *) dst_orig;
|
2017-06-17 17:42:26 +00:00
|
|
|
struct net_device *loopback_dev = net->loopback_dev;
|
2007-05-25 01:17:54 +00:00
|
|
|
struct dst_entry *new = NULL;
|
|
|
|
|
2017-06-17 17:42:26 +00:00
|
|
|
rt = dst_alloc(&ip6_dst_blackhole_ops, loopback_dev, 1,
|
2017-10-09 06:39:43 +00:00
|
|
|
DST_OBSOLETE_DEAD, 0);
|
2007-05-25 01:17:54 +00:00
|
|
|
if (rt) {
|
2015-10-15 23:39:58 +00:00
|
|
|
rt6_info_init(rt);
|
2017-10-06 19:06:11 +00:00
|
|
|
atomic_inc(&net->ipv6.rt6_stats->fib_rt_alloc);
|
2012-07-05 23:37:09 +00:00
|
|
|
|
2015-10-15 23:39:58 +00:00
|
|
|
new = &rt->dst;
|
2007-05-25 01:17:54 +00:00
|
|
|
new->__use = 1;
|
2007-11-14 05:34:06 +00:00
|
|
|
new->input = dst_discard;
|
2015-10-07 21:48:47 +00:00
|
|
|
new->output = dst_discard_out;
|
2007-05-25 01:17:54 +00:00
|
|
|
|
2015-10-15 23:39:58 +00:00
|
|
|
dst_copy_metrics(new, &ort->dst);
|
2007-05-25 01:17:54 +00:00
|
|
|
|
2017-06-17 17:42:26 +00:00
|
|
|
rt->rt6i_idev = in6_dev_get(loopback_dev);
|
2011-11-21 03:39:03 +00:00
|
|
|
rt->rt6i_gateway = ort->rt6i_gateway;
|
2015-10-15 23:39:58 +00:00
|
|
|
rt->rt6i_flags = ort->rt6i_flags & ~RTF_PCPU;
|
2007-05-25 01:17:54 +00:00
|
|
|
rt->rt6i_metric = 0;
|
|
|
|
|
|
|
|
memcpy(&rt->rt6i_dst, &ort->rt6i_dst, sizeof(struct rt6key));
|
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
|
|
|
memcpy(&rt->rt6i_src, &ort->rt6i_src, sizeof(struct rt6key));
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2011-03-01 22:45:33 +00:00
|
|
|
dst_release(dst_orig);
|
|
|
|
return new ? new : ERR_PTR(-ENOMEM);
|
2007-05-25 01:17:54 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* Destination cache support functions
|
|
|
|
*/
|
|
|
|
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
static void rt6_dst_from_metrics_check(struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
if (rt->dst.from &&
|
|
|
|
dst_metrics_ptr(&rt->dst) != dst_metrics_ptr(rt->dst.from))
|
|
|
|
dst_init_metrics(&rt->dst, dst_metrics_ptr(rt->dst.from), true);
|
|
|
|
}
|
|
|
|
|
2015-05-23 03:56:03 +00:00
|
|
|
static struct dst_entry *rt6_check(struct rt6_info *rt, u32 cookie)
|
|
|
|
{
|
2017-08-25 07:05:42 +00:00
|
|
|
u32 rt_cookie = 0;
|
2017-08-21 16:47:10 +00:00
|
|
|
|
|
|
|
if (!rt6_get_cookie_safe(rt, &rt_cookie) || rt_cookie != cookie)
|
2015-05-23 03:56:03 +00:00
|
|
|
return NULL;
|
|
|
|
|
|
|
|
if (rt6_check_expired(rt))
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
return &rt->dst;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct dst_entry *rt6_dst_from_check(struct rt6_info *rt, u32 cookie)
|
|
|
|
{
|
2015-11-11 19:51:07 +00:00
|
|
|
if (!__rt6_check_expired(rt) &&
|
|
|
|
rt->dst.obsolete == DST_OBSOLETE_FORCE_CHK &&
|
2015-05-23 03:56:03 +00:00
|
|
|
rt6_check((struct rt6_info *)(rt->dst.from), cookie))
|
|
|
|
return &rt->dst;
|
|
|
|
else
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie)
|
|
|
|
{
|
|
|
|
struct rt6_info *rt;
|
|
|
|
|
|
|
|
rt = (struct rt6_info *) dst;
|
|
|
|
|
2012-09-10 22:09:46 +00:00
|
|
|
/* All IPV6 dsts are created with ->obsolete set to the value
|
|
|
|
* DST_OBSOLETE_FORCE_CHK which forces validation calls down
|
|
|
|
* into this function always.
|
|
|
|
*/
|
2013-10-24 05:48:24 +00:00
|
|
|
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
rt6_dst_from_metrics_check(rt);
|
|
|
|
|
2015-11-11 19:51:08 +00:00
|
|
|
if (rt->rt6i_flags & RTF_PCPU ||
|
2017-06-17 17:42:42 +00:00
|
|
|
(unlikely(!list_empty(&rt->rt6i_uncached)) && rt->dst.from))
|
2015-05-23 03:56:03 +00:00
|
|
|
return rt6_dst_from_check(rt, cookie);
|
|
|
|
else
|
|
|
|
return rt6_check(rt, cookie);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static struct dst_entry *ip6_negative_advice(struct dst_entry *dst)
|
|
|
|
{
|
|
|
|
struct rt6_info *rt = (struct rt6_info *) dst;
|
|
|
|
|
|
|
|
if (rt) {
|
2010-03-28 07:15:45 +00:00
|
|
|
if (rt->rt6i_flags & RTF_CACHE) {
|
|
|
|
if (rt6_check_expired(rt)) {
|
|
|
|
ip6_del_rt(rt);
|
|
|
|
dst = NULL;
|
|
|
|
}
|
|
|
|
} else {
|
2005-04-16 22:20:36 +00:00
|
|
|
dst_release(dst);
|
2010-03-28 07:15:45 +00:00
|
|
|
dst = NULL;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2010-03-28 07:15:45 +00:00
|
|
|
return dst;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static void ip6_link_failure(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct rt6_info *rt;
|
|
|
|
|
2010-02-18 08:25:24 +00:00
|
|
|
icmpv6_send(skb, ICMPV6_DEST_UNREACH, ICMPV6_ADDR_UNREACH, 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2009-06-02 05:19:30 +00:00
|
|
|
rt = (struct rt6_info *) skb_dst(skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (rt) {
|
2013-07-10 21:00:57 +00:00
|
|
|
if (rt->rt6i_flags & RTF_CACHE) {
|
2017-06-17 17:42:35 +00:00
|
|
|
if (dst_hold_safe(&rt->dst))
|
|
|
|
ip6_del_rt(rt);
|
2017-08-21 16:47:10 +00:00
|
|
|
} else {
|
|
|
|
struct fib6_node *fn;
|
|
|
|
|
|
|
|
rcu_read_lock();
|
|
|
|
fn = rcu_dereference(rt->rt6i_node);
|
|
|
|
if (fn && (rt->rt6i_flags & RTF_DEFAULT))
|
|
|
|
fn->fn_sernum = -1;
|
|
|
|
rcu_read_unlock();
|
2013-07-10 21:00:57 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-05-23 03:56:00 +00:00
|
|
|
static void rt6_do_update_pmtu(struct rt6_info *rt, u32 mtu)
|
|
|
|
{
|
|
|
|
struct net *net = dev_net(rt->dst.dev);
|
|
|
|
|
|
|
|
rt->rt6i_flags |= RTF_MODIFIED;
|
|
|
|
rt->rt6i_pmtu = mtu;
|
|
|
|
rt6_update_expires(rt, net->ipv6.sysctl.ip6_rt_mtu_expires);
|
|
|
|
}
|
|
|
|
|
2015-11-11 19:51:06 +00:00
|
|
|
static bool rt6_cache_allowed_for_pmtu(const struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
return !(rt->rt6i_flags & RTF_CACHE) &&
|
2017-08-25 22:03:10 +00:00
|
|
|
(rt->rt6i_flags & RTF_PCPU ||
|
|
|
|
rcu_access_pointer(rt->rt6i_node));
|
2015-11-11 19:51:06 +00:00
|
|
|
}
|
|
|
|
|
2015-05-23 03:56:00 +00:00
|
|
|
static void __ip6_rt_update_pmtu(struct dst_entry *dst, const struct sock *sk,
|
|
|
|
const struct ipv6hdr *iph, u32 mtu)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2017-02-06 21:14:16 +00:00
|
|
|
const struct in6_addr *daddr, *saddr;
|
2014-08-24 20:53:10 +00:00
|
|
|
struct rt6_info *rt6 = (struct rt6_info *)dst;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2015-05-23 03:56:00 +00:00
|
|
|
if (rt6->rt6i_flags & RTF_LOCAL)
|
|
|
|
return;
|
2012-06-15 21:54:11 +00:00
|
|
|
|
2016-10-28 10:18:01 +00:00
|
|
|
if (dst_metric_locked(dst, RTAX_MTU))
|
|
|
|
return;
|
|
|
|
|
2017-02-06 21:14:16 +00:00
|
|
|
if (iph) {
|
|
|
|
daddr = &iph->daddr;
|
|
|
|
saddr = &iph->saddr;
|
|
|
|
} else if (sk) {
|
|
|
|
daddr = &sk->sk_v6_daddr;
|
|
|
|
saddr = &inet6_sk(sk)->saddr;
|
|
|
|
} else {
|
|
|
|
daddr = NULL;
|
|
|
|
saddr = NULL;
|
|
|
|
}
|
|
|
|
dst_confirm_neigh(dst, daddr);
|
2015-05-23 03:56:00 +00:00
|
|
|
mtu = max_t(u32, mtu, IPV6_MIN_MTU);
|
|
|
|
if (mtu >= dst_mtu(dst))
|
|
|
|
return;
|
2015-01-15 21:34:25 +00:00
|
|
|
|
2015-11-11 19:51:06 +00:00
|
|
|
if (!rt6_cache_allowed_for_pmtu(rt6)) {
|
2015-05-23 03:56:00 +00:00
|
|
|
rt6_do_update_pmtu(rt6, mtu);
|
2017-10-06 19:06:03 +00:00
|
|
|
/* update rt6_ex->stamp for cache */
|
|
|
|
if (rt6->rt6i_flags & RTF_CACHE)
|
|
|
|
rt6_update_exception_stamp_rt(rt6);
|
2017-02-06 21:14:16 +00:00
|
|
|
} else if (daddr) {
|
2015-05-23 03:56:00 +00:00
|
|
|
struct rt6_info *nrt6;
|
|
|
|
|
|
|
|
nrt6 = ip6_rt_cache_alloc(rt6, daddr, saddr);
|
|
|
|
if (nrt6) {
|
|
|
|
rt6_do_update_pmtu(nrt6, mtu);
|
2017-10-06 19:06:03 +00:00
|
|
|
if (rt6_insert_exception(nrt6, rt6))
|
|
|
|
dst_release_immediate(&nrt6->dst);
|
2015-05-23 03:56:00 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-05-23 03:56:00 +00:00
|
|
|
static void ip6_rt_update_pmtu(struct dst_entry *dst, struct sock *sk,
|
|
|
|
struct sk_buff *skb, u32 mtu)
|
|
|
|
{
|
|
|
|
__ip6_rt_update_pmtu(dst, sk, skb ? ipv6_hdr(skb) : NULL, mtu);
|
|
|
|
}
|
|
|
|
|
2012-06-16 03:01:57 +00:00
|
|
|
void ip6_update_pmtu(struct sk_buff *skb, struct net *net, __be32 mtu,
|
2016-11-03 17:23:43 +00:00
|
|
|
int oif, u32 mark, kuid_t uid)
|
2012-06-15 21:54:11 +00:00
|
|
|
{
|
|
|
|
const struct ipv6hdr *iph = (struct ipv6hdr *) skb->data;
|
|
|
|
struct dst_entry *dst;
|
|
|
|
struct flowi6 fl6;
|
|
|
|
|
|
|
|
memset(&fl6, 0, sizeof(fl6));
|
|
|
|
fl6.flowi6_oif = oif;
|
2014-05-13 17:17:34 +00:00
|
|
|
fl6.flowi6_mark = mark ? mark : IP6_REPLY_MARK(net, skb->mark);
|
2012-06-15 21:54:11 +00:00
|
|
|
fl6.daddr = iph->daddr;
|
|
|
|
fl6.saddr = iph->saddr;
|
2013-01-13 05:01:51 +00:00
|
|
|
fl6.flowlabel = ip6_flowinfo(iph);
|
2016-11-03 17:23:43 +00:00
|
|
|
fl6.flowi6_uid = uid;
|
2012-06-15 21:54:11 +00:00
|
|
|
|
|
|
|
dst = ip6_route_output(net, NULL, &fl6);
|
|
|
|
if (!dst->error)
|
2015-05-23 03:56:00 +00:00
|
|
|
__ip6_rt_update_pmtu(dst, NULL, iph, ntohl(mtu));
|
2012-06-15 21:54:11 +00:00
|
|
|
dst_release(dst);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(ip6_update_pmtu);
|
|
|
|
|
|
|
|
void ip6_sk_update_pmtu(struct sk_buff *skb, struct sock *sk, __be32 mtu)
|
|
|
|
{
|
ipv6: datagram: Update dst cache of a connected datagram sk during pmtu update
There is a case in connected UDP socket such that
getsockopt(IPV6_MTU) will return a stale MTU value. The reproducible
sequence could be the following:
1. Create a connected UDP socket
2. Send some datagrams out
3. Receive a ICMPV6_PKT_TOOBIG
4. No new outgoing datagrams to trigger the sk_dst_check()
logic to update the sk->sk_dst_cache.
5. getsockopt(IPV6_MTU) returns the mtu from the invalid
sk->sk_dst_cache instead of the newly created RTF_CACHE clone.
This patch updates the sk->sk_dst_cache for a connected datagram sk
during pmtu-update code path.
Note that the sk->sk_v6_daddr is used to do the route lookup
instead of skb->data (i.e. iph). It is because a UDP socket can become
connected after sending out some datagrams in un-connected state. or
It can be connected multiple times to different destinations. Hence,
iph may not be related to where sk is currently connected to.
It is done under '!sock_owned_by_user(sk)' condition because
the user may make another ip6_datagram_connect() (i.e changing
the sk->sk_v6_daddr) while dst lookup is happening in the pmtu-update
code path.
For the sock_owned_by_user(sk) == true case, the next patch will
introduce a release_cb() which will update the sk->sk_dst_cache.
Test:
Server (Connected UDP Socket):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Route Details:
[root@arch-fb-vm1 ~]# ip -6 r show | egrep '2fac'
2fac::/64 dev eth0 proto kernel metric 256 pref medium
2fac:face::/64 via 2fac::face dev eth0 metric 1024 pref medium
A simple python code to create a connected UDP socket:
import socket
import errno
HOST = '2fac::1'
PORT = 8080
s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
s.bind((HOST, PORT))
s.connect(('2fac:face::face', 53))
print("connected")
while True:
try:
data = s.recv(1024)
except socket.error as se:
if se.errno == errno.EMSGSIZE:
pmtu = s.getsockopt(41, 24)
print("PMTU:%d" % pmtu)
break
s.close()
Python program output after getting a ICMPV6_PKT_TOOBIG:
[root@arch-fb-vm1 ~]# python2 ~/devshare/kernel/tasks/fib6/udp-connect-53-8080.py
connected
PMTU:1300
Cache routes after recieving TOOBIG:
[root@arch-fb-vm1 ~]# ip -6 r show table cache
2fac:face::face via 2fac::face dev eth0 metric 0
cache expires 463sec mtu 1300 pref medium
Client (Send the ICMPV6_PKT_TOOBIG):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
scapy is used to generate the TOOBIG message. Here is the scapy script I have
used:
>>> p=Ether(src='da:75:4d:36:ac:32', dst='52:54:00:12:34:66', type=0x86dd)/IPv6(src='2fac::face', dst='2fac::1')/ICMPv6PacketTooBig(mtu=1300)/IPv6(src='2fac::
1',dst='2fac:face::face', nh='UDP')/UDP(sport=8080,dport=53)
>>> sendp(p, iface='qemubr0')
Fixes: 45e4fd26683c ("ipv6: Only create RTF_CACHE routes after encountering pmtu exception")
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reported-by: Wei Wang <weiwan@google.com>
Cc: Cong Wang <xiyou.wangcong@gmail.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Wei Wang <weiwan@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-11 22:29:36 +00:00
|
|
|
struct dst_entry *dst;
|
|
|
|
|
2012-06-15 21:54:11 +00:00
|
|
|
ip6_update_pmtu(skb, sock_net(sk), mtu,
|
2016-11-03 17:23:43 +00:00
|
|
|
sk->sk_bound_dev_if, sk->sk_mark, sk->sk_uid);
|
ipv6: datagram: Update dst cache of a connected datagram sk during pmtu update
There is a case in connected UDP socket such that
getsockopt(IPV6_MTU) will return a stale MTU value. The reproducible
sequence could be the following:
1. Create a connected UDP socket
2. Send some datagrams out
3. Receive a ICMPV6_PKT_TOOBIG
4. No new outgoing datagrams to trigger the sk_dst_check()
logic to update the sk->sk_dst_cache.
5. getsockopt(IPV6_MTU) returns the mtu from the invalid
sk->sk_dst_cache instead of the newly created RTF_CACHE clone.
This patch updates the sk->sk_dst_cache for a connected datagram sk
during pmtu-update code path.
Note that the sk->sk_v6_daddr is used to do the route lookup
instead of skb->data (i.e. iph). It is because a UDP socket can become
connected after sending out some datagrams in un-connected state. or
It can be connected multiple times to different destinations. Hence,
iph may not be related to where sk is currently connected to.
It is done under '!sock_owned_by_user(sk)' condition because
the user may make another ip6_datagram_connect() (i.e changing
the sk->sk_v6_daddr) while dst lookup is happening in the pmtu-update
code path.
For the sock_owned_by_user(sk) == true case, the next patch will
introduce a release_cb() which will update the sk->sk_dst_cache.
Test:
Server (Connected UDP Socket):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Route Details:
[root@arch-fb-vm1 ~]# ip -6 r show | egrep '2fac'
2fac::/64 dev eth0 proto kernel metric 256 pref medium
2fac:face::/64 via 2fac::face dev eth0 metric 1024 pref medium
A simple python code to create a connected UDP socket:
import socket
import errno
HOST = '2fac::1'
PORT = 8080
s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
s.bind((HOST, PORT))
s.connect(('2fac:face::face', 53))
print("connected")
while True:
try:
data = s.recv(1024)
except socket.error as se:
if se.errno == errno.EMSGSIZE:
pmtu = s.getsockopt(41, 24)
print("PMTU:%d" % pmtu)
break
s.close()
Python program output after getting a ICMPV6_PKT_TOOBIG:
[root@arch-fb-vm1 ~]# python2 ~/devshare/kernel/tasks/fib6/udp-connect-53-8080.py
connected
PMTU:1300
Cache routes after recieving TOOBIG:
[root@arch-fb-vm1 ~]# ip -6 r show table cache
2fac:face::face via 2fac::face dev eth0 metric 0
cache expires 463sec mtu 1300 pref medium
Client (Send the ICMPV6_PKT_TOOBIG):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
scapy is used to generate the TOOBIG message. Here is the scapy script I have
used:
>>> p=Ether(src='da:75:4d:36:ac:32', dst='52:54:00:12:34:66', type=0x86dd)/IPv6(src='2fac::face', dst='2fac::1')/ICMPv6PacketTooBig(mtu=1300)/IPv6(src='2fac::
1',dst='2fac:face::face', nh='UDP')/UDP(sport=8080,dport=53)
>>> sendp(p, iface='qemubr0')
Fixes: 45e4fd26683c ("ipv6: Only create RTF_CACHE routes after encountering pmtu exception")
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reported-by: Wei Wang <weiwan@google.com>
Cc: Cong Wang <xiyou.wangcong@gmail.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Wei Wang <weiwan@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-11 22:29:36 +00:00
|
|
|
|
|
|
|
dst = __sk_dst_get(sk);
|
|
|
|
if (!dst || !dst->obsolete ||
|
|
|
|
dst->ops->check(dst, inet6_sk(sk)->dst_cookie))
|
|
|
|
return;
|
|
|
|
|
|
|
|
bh_lock_sock(sk);
|
|
|
|
if (!sock_owned_by_user(sk) && !ipv6_addr_v4mapped(&sk->sk_v6_daddr))
|
|
|
|
ip6_datagram_dst_update(sk, false);
|
|
|
|
bh_unlock_sock(sk);
|
2012-06-15 21:54:11 +00:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(ip6_sk_update_pmtu);
|
|
|
|
|
2013-09-04 11:44:21 +00:00
|
|
|
/* Handle redirects */
|
|
|
|
struct ip6rd_flowi {
|
|
|
|
struct flowi6 fl6;
|
|
|
|
struct in6_addr gateway;
|
|
|
|
};
|
|
|
|
|
|
|
|
static struct rt6_info *__ip6_route_redirect(struct net *net,
|
|
|
|
struct fib6_table *table,
|
|
|
|
struct flowi6 *fl6,
|
|
|
|
int flags)
|
|
|
|
{
|
|
|
|
struct ip6rd_flowi *rdfl = (struct ip6rd_flowi *)fl6;
|
2017-10-06 19:06:03 +00:00
|
|
|
struct rt6_info *rt, *rt_cache;
|
2013-09-04 11:44:21 +00:00
|
|
|
struct fib6_node *fn;
|
|
|
|
|
|
|
|
/* Get the "current" route for this destination and
|
2017-01-07 22:53:00 +00:00
|
|
|
* check if the redirect has come from appropriate router.
|
2013-09-04 11:44:21 +00:00
|
|
|
*
|
|
|
|
* RFC 4861 specifies that redirects should only be
|
|
|
|
* accepted if they come from the nexthop to the target.
|
|
|
|
* Due to the way the routes are chosen, this notion
|
|
|
|
* is a bit fuzzy and one might need to check all possible
|
|
|
|
* routes.
|
|
|
|
*/
|
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_lock();
|
2013-09-04 11:44:21 +00:00
|
|
|
fn = fib6_lookup(&table->tb6_root, &fl6->daddr, &fl6->saddr);
|
|
|
|
restart:
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
for_each_fib6_node_rt_rcu(fn) {
|
2013-09-04 11:44:21 +00:00
|
|
|
if (rt6_check_expired(rt))
|
|
|
|
continue;
|
|
|
|
if (rt->dst.error)
|
|
|
|
break;
|
|
|
|
if (!(rt->rt6i_flags & RTF_GATEWAY))
|
|
|
|
continue;
|
|
|
|
if (fl6->flowi6_oif != rt->dst.dev->ifindex)
|
|
|
|
continue;
|
2017-10-06 19:06:03 +00:00
|
|
|
/* rt_cache's gateway might be different from its 'parent'
|
|
|
|
* in the case of an ip redirect.
|
|
|
|
* So we keep searching in the exception table if the gateway
|
|
|
|
* is different.
|
|
|
|
*/
|
|
|
|
if (!ipv6_addr_equal(&rdfl->gateway, &rt->rt6i_gateway)) {
|
|
|
|
rt_cache = rt6_find_cached_rt(rt,
|
|
|
|
&fl6->daddr,
|
|
|
|
&fl6->saddr);
|
|
|
|
if (rt_cache &&
|
|
|
|
ipv6_addr_equal(&rdfl->gateway,
|
|
|
|
&rt_cache->rt6i_gateway)) {
|
|
|
|
rt = rt_cache;
|
|
|
|
break;
|
|
|
|
}
|
2013-09-04 11:44:21 +00:00
|
|
|
continue;
|
2017-10-06 19:06:03 +00:00
|
|
|
}
|
2013-09-04 11:44:21 +00:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!rt)
|
|
|
|
rt = net->ipv6.ip6_null_entry;
|
|
|
|
else if (rt->dst.error) {
|
|
|
|
rt = net->ipv6.ip6_null_entry;
|
2015-01-21 03:16:02 +00:00
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (rt == net->ipv6.ip6_null_entry) {
|
2014-10-20 20:42:43 +00:00
|
|
|
fn = fib6_backtrack(fn, &fl6->saddr);
|
|
|
|
if (fn)
|
|
|
|
goto restart;
|
2013-09-04 11:44:21 +00:00
|
|
|
}
|
2014-10-20 20:42:43 +00:00
|
|
|
|
2015-01-21 03:16:02 +00:00
|
|
|
out:
|
2017-10-06 19:06:06 +00:00
|
|
|
ip6_hold_safe(net, &rt, true);
|
2013-09-04 11:44:21 +00:00
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2013-09-04 11:44:21 +00:00
|
|
|
|
2017-10-19 07:31:43 +00:00
|
|
|
trace_fib6_table_lookup(net, rt, table, fl6);
|
2013-09-04 11:44:21 +00:00
|
|
|
return rt;
|
|
|
|
};
|
|
|
|
|
|
|
|
static struct dst_entry *ip6_route_redirect(struct net *net,
|
|
|
|
const struct flowi6 *fl6,
|
|
|
|
const struct in6_addr *gateway)
|
|
|
|
{
|
|
|
|
int flags = RT6_LOOKUP_F_HAS_SADDR;
|
|
|
|
struct ip6rd_flowi rdfl;
|
|
|
|
|
|
|
|
rdfl.fl6 = *fl6;
|
|
|
|
rdfl.gateway = *gateway;
|
|
|
|
|
|
|
|
return fib6_rule_lookup(net, &rdfl.fl6,
|
|
|
|
flags, __ip6_route_redirect);
|
|
|
|
}
|
|
|
|
|
2016-11-03 17:23:43 +00:00
|
|
|
void ip6_redirect(struct sk_buff *skb, struct net *net, int oif, u32 mark,
|
|
|
|
kuid_t uid)
|
2012-07-12 07:08:07 +00:00
|
|
|
{
|
|
|
|
const struct ipv6hdr *iph = (struct ipv6hdr *) skb->data;
|
|
|
|
struct dst_entry *dst;
|
|
|
|
struct flowi6 fl6;
|
|
|
|
|
|
|
|
memset(&fl6, 0, sizeof(fl6));
|
2014-04-28 07:51:56 +00:00
|
|
|
fl6.flowi6_iif = LOOPBACK_IFINDEX;
|
2012-07-12 07:08:07 +00:00
|
|
|
fl6.flowi6_oif = oif;
|
|
|
|
fl6.flowi6_mark = mark;
|
|
|
|
fl6.daddr = iph->daddr;
|
|
|
|
fl6.saddr = iph->saddr;
|
2013-01-13 05:01:51 +00:00
|
|
|
fl6.flowlabel = ip6_flowinfo(iph);
|
2016-11-03 17:23:43 +00:00
|
|
|
fl6.flowi6_uid = uid;
|
2012-07-12 07:08:07 +00:00
|
|
|
|
2013-09-04 11:44:21 +00:00
|
|
|
dst = ip6_route_redirect(net, &fl6, &ipv6_hdr(skb)->saddr);
|
|
|
|
rt6_do_redirect(dst, NULL, skb);
|
2012-07-12 07:08:07 +00:00
|
|
|
dst_release(dst);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(ip6_redirect);
|
|
|
|
|
2013-08-22 04:07:35 +00:00
|
|
|
void ip6_redirect_no_header(struct sk_buff *skb, struct net *net, int oif,
|
|
|
|
u32 mark)
|
|
|
|
{
|
|
|
|
const struct ipv6hdr *iph = ipv6_hdr(skb);
|
|
|
|
const struct rd_msg *msg = (struct rd_msg *)icmp6_hdr(skb);
|
|
|
|
struct dst_entry *dst;
|
|
|
|
struct flowi6 fl6;
|
|
|
|
|
|
|
|
memset(&fl6, 0, sizeof(fl6));
|
2014-04-28 07:51:56 +00:00
|
|
|
fl6.flowi6_iif = LOOPBACK_IFINDEX;
|
2013-08-22 04:07:35 +00:00
|
|
|
fl6.flowi6_oif = oif;
|
|
|
|
fl6.flowi6_mark = mark;
|
|
|
|
fl6.daddr = msg->dest;
|
|
|
|
fl6.saddr = iph->daddr;
|
2016-11-03 17:23:43 +00:00
|
|
|
fl6.flowi6_uid = sock_net_uid(net, NULL);
|
2013-08-22 04:07:35 +00:00
|
|
|
|
2013-09-04 11:44:21 +00:00
|
|
|
dst = ip6_route_redirect(net, &fl6, &iph->saddr);
|
|
|
|
rt6_do_redirect(dst, NULL, skb);
|
2013-08-22 04:07:35 +00:00
|
|
|
dst_release(dst);
|
|
|
|
}
|
|
|
|
|
2012-07-12 07:08:07 +00:00
|
|
|
void ip6_sk_redirect(struct sk_buff *skb, struct sock *sk)
|
|
|
|
{
|
2016-11-03 17:23:43 +00:00
|
|
|
ip6_redirect(skb, sock_net(sk), sk->sk_bound_dev_if, sk->sk_mark,
|
|
|
|
sk->sk_uid);
|
2012-07-12 07:08:07 +00:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(ip6_sk_redirect);
|
|
|
|
|
2010-12-13 20:52:14 +00:00
|
|
|
static unsigned int ip6_default_advmss(const struct dst_entry *dst)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2010-12-13 20:52:14 +00:00
|
|
|
struct net_device *dev = dst->dev;
|
|
|
|
unsigned int mtu = dst_mtu(dst);
|
|
|
|
struct net *net = dev_net(dev);
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
mtu -= sizeof(struct ipv6hdr) + sizeof(struct tcphdr);
|
|
|
|
|
2008-03-04 21:47:47 +00:00
|
|
|
if (mtu < net->ipv6.sysctl.ip6_rt_min_advmss)
|
|
|
|
mtu = net->ipv6.sysctl.ip6_rt_min_advmss;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/*
|
2007-02-09 14:24:49 +00:00
|
|
|
* Maximal non-jumbo IPv6 payload is IPV6_MAXPLEN and
|
|
|
|
* corresponding MSS is IPV6_MAXPLEN - tcp_header_size.
|
|
|
|
* IPV6_MAXPLEN is also valid and means: "any MSS,
|
2005-04-16 22:20:36 +00:00
|
|
|
* rely only on pmtu discovery"
|
|
|
|
*/
|
|
|
|
if (mtu > IPV6_MAXPLEN - sizeof(struct tcphdr))
|
|
|
|
mtu = IPV6_MAXPLEN;
|
|
|
|
return mtu;
|
|
|
|
}
|
|
|
|
|
2011-11-23 02:12:51 +00:00
|
|
|
static unsigned int ip6_mtu(const struct dst_entry *dst)
|
2010-12-14 21:01:14 +00:00
|
|
|
{
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
const struct rt6_info *rt = (const struct rt6_info *)dst;
|
|
|
|
unsigned int mtu = rt->rt6i_pmtu;
|
2010-12-14 21:01:14 +00:00
|
|
|
struct inet6_dev *idev;
|
2011-11-23 02:13:31 +00:00
|
|
|
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
if (mtu)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
mtu = dst_metric_raw(dst, RTAX_MTU);
|
2011-11-23 02:13:31 +00:00
|
|
|
if (mtu)
|
2014-04-11 04:23:36 +00:00
|
|
|
goto out;
|
2011-11-23 02:13:31 +00:00
|
|
|
|
|
|
|
mtu = IPV6_MIN_MTU;
|
2010-12-14 21:01:14 +00:00
|
|
|
|
|
|
|
rcu_read_lock();
|
|
|
|
idev = __in6_dev_get(dst->dev);
|
|
|
|
if (idev)
|
|
|
|
mtu = idev->cnf.mtu6;
|
|
|
|
rcu_read_unlock();
|
|
|
|
|
2014-04-11 04:23:36 +00:00
|
|
|
out:
|
2016-08-25 03:10:43 +00:00
|
|
|
mtu = min_t(unsigned int, mtu, IP6_MAX_MTU);
|
|
|
|
|
|
|
|
return mtu - lwtunnel_headroom(dst->lwtstate, mtu);
|
2010-12-14 21:01:14 +00:00
|
|
|
}
|
|
|
|
|
2007-12-07 00:11:48 +00:00
|
|
|
struct dst_entry *icmp6_dst_alloc(struct net_device *dev,
|
2011-12-06 22:04:13 +00:00
|
|
|
struct flowi6 *fl6)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2011-12-06 22:04:13 +00:00
|
|
|
struct dst_entry *dst;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct rt6_info *rt;
|
|
|
|
struct inet6_dev *idev = in6_dev_get(dev);
|
2008-03-25 12:47:49 +00:00
|
|
|
struct net *net = dev_net(dev);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-12-03 23:02:47 +00:00
|
|
|
if (unlikely(!idev))
|
2012-03-14 21:13:11 +00:00
|
|
|
return ERR_PTR(-ENODEV);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2015-08-14 18:05:52 +00:00
|
|
|
rt = ip6_dst_alloc(net, dev, 0);
|
2011-12-03 23:02:47 +00:00
|
|
|
if (unlikely(!rt)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
in6_dev_put(idev);
|
2011-12-06 22:04:13 +00:00
|
|
|
dst = ERR_PTR(-ENOMEM);
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2011-09-05 21:34:30 +00:00
|
|
|
rt->dst.flags |= DST_HOST;
|
|
|
|
rt->dst.output = ip6_output;
|
2013-10-20 12:43:04 +00:00
|
|
|
rt->rt6i_gateway = fl6->daddr;
|
2011-12-06 22:04:13 +00:00
|
|
|
rt->rt6i_dst.addr = fl6->daddr;
|
2011-09-05 21:34:30 +00:00
|
|
|
rt->rt6i_dst.plen = 128;
|
|
|
|
rt->rt6i_idev = idev;
|
2012-10-24 06:01:18 +00:00
|
|
|
dst_metric_set(&rt->dst, RTAX_HOPLIMIT, 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-06-17 17:42:36 +00:00
|
|
|
/* Add this dst into uncached_list so that rt6_ifdown() can
|
|
|
|
* do proper release of the net_device
|
|
|
|
*/
|
|
|
|
rt6_uncached_list_add(rt);
|
2017-10-06 19:06:11 +00:00
|
|
|
atomic_inc(&net->ipv6.rt6_stats->fib_rt_uncache);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-12-06 22:04:13 +00:00
|
|
|
dst = xfrm_lookup(net, &rt->dst, flowi6_to_flowi(fl6), NULL, 0);
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
out:
|
2011-12-06 22:04:13 +00:00
|
|
|
return dst;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2008-01-18 11:56:57 +00:00
|
|
|
static int ip6_dst_gc(struct dst_ops *ops)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2009-08-29 01:34:49 +00:00
|
|
|
struct net *net = container_of(ops, struct net, ipv6.ip6_dst_ops);
|
2008-03-04 21:50:14 +00:00
|
|
|
int rt_min_interval = net->ipv6.sysctl.ip6_rt_gc_min_interval;
|
|
|
|
int rt_max_size = net->ipv6.sysctl.ip6_rt_max_size;
|
|
|
|
int rt_elasticity = net->ipv6.sysctl.ip6_rt_gc_elasticity;
|
|
|
|
int rt_gc_timeout = net->ipv6.sysctl.ip6_rt_gc_timeout;
|
|
|
|
unsigned long rt_last_gc = net->ipv6.ip6_rt_last_gc;
|
2010-10-08 06:37:34 +00:00
|
|
|
int entries;
|
2008-03-04 21:50:14 +00:00
|
|
|
|
2010-10-08 06:37:34 +00:00
|
|
|
entries = dst_entries_get_fast(ops);
|
2013-08-01 08:04:24 +00:00
|
|
|
if (time_after(rt_last_gc + rt_min_interval, jiffies) &&
|
2010-10-08 06:37:34 +00:00
|
|
|
entries <= rt_max_size)
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out;
|
|
|
|
|
2008-03-04 21:49:47 +00:00
|
|
|
net->ipv6.ip6_rt_gc_expire++;
|
2014-05-19 09:30:28 +00:00
|
|
|
fib6_run_gc(net->ipv6.ip6_rt_gc_expire, net, true);
|
2010-10-08 06:37:34 +00:00
|
|
|
entries = dst_entries_get_slow(ops);
|
|
|
|
if (entries < ops->gc_thresh)
|
2008-03-04 21:50:14 +00:00
|
|
|
net->ipv6.ip6_rt_gc_expire = rt_gc_timeout>>1;
|
2005-04-16 22:20:36 +00:00
|
|
|
out:
|
2008-03-04 21:50:14 +00:00
|
|
|
net->ipv6.ip6_rt_gc_expire -= net->ipv6.ip6_rt_gc_expire>>rt_elasticity;
|
2010-10-08 06:37:34 +00:00
|
|
|
return entries > rt_max_size;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2015-01-05 22:57:44 +00:00
|
|
|
static int ip6_convert_metrics(struct mx6_config *mxc,
|
|
|
|
const struct fib6_config *cfg)
|
|
|
|
{
|
tcp: use dctcp if enabled on the route to the initiator
Currently, the following case doesn't use DCTCP, even if it should:
A responder has f.e. Cubic as system wide default, but for a specific
route to the initiating host, DCTCP is being set in RTAX_CC_ALGO. The
initiating host then uses DCTCP as congestion control, but since the
initiator sets ECT(0), tcp_ecn_create_request() doesn't set ecn_ok,
and we have to fall back to Reno after 3WHS completes.
We were thinking on how to solve this in a minimal, non-intrusive
way without bloating tcp_ecn_create_request() needlessly: lets cache
the CA ecn option flag in RTAX_FEATURES. In other words, when ECT(0)
is set on the SYN packet, set ecn_ok=1 iff route RTAX_FEATURES
contains the unexposed (internal-only) DST_FEATURE_ECN_CA. This allows
to only do a single metric feature lookup inside tcp_ecn_create_request().
Joint work with Florian Westphal.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-08-31 13:58:47 +00:00
|
|
|
bool ecn_ca = false;
|
2015-01-05 22:57:44 +00:00
|
|
|
struct nlattr *nla;
|
|
|
|
int remaining;
|
|
|
|
u32 *mp;
|
|
|
|
|
2015-03-29 13:00:04 +00:00
|
|
|
if (!cfg->fc_mx)
|
2015-01-05 22:57:44 +00:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
mp = kzalloc(sizeof(u32) * RTAX_MAX, GFP_KERNEL);
|
|
|
|
if (unlikely(!mp))
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
nla_for_each_attr(nla, cfg->fc_mx, cfg->fc_mx_len, remaining) {
|
|
|
|
int type = nla_type(nla);
|
2015-08-31 13:58:45 +00:00
|
|
|
u32 val;
|
2015-01-05 22:57:44 +00:00
|
|
|
|
2015-08-31 13:58:45 +00:00
|
|
|
if (!type)
|
|
|
|
continue;
|
|
|
|
if (unlikely(type > RTAX_MAX))
|
|
|
|
goto err;
|
2015-01-05 22:57:47 +00:00
|
|
|
|
2015-08-31 13:58:45 +00:00
|
|
|
if (type == RTAX_CC_ALGO) {
|
|
|
|
char tmp[TCP_CA_NAME_MAX];
|
2015-01-05 22:57:44 +00:00
|
|
|
|
2015-08-31 13:58:45 +00:00
|
|
|
nla_strlcpy(tmp, nla, sizeof(tmp));
|
tcp: use dctcp if enabled on the route to the initiator
Currently, the following case doesn't use DCTCP, even if it should:
A responder has f.e. Cubic as system wide default, but for a specific
route to the initiating host, DCTCP is being set in RTAX_CC_ALGO. The
initiating host then uses DCTCP as congestion control, but since the
initiator sets ECT(0), tcp_ecn_create_request() doesn't set ecn_ok,
and we have to fall back to Reno after 3WHS completes.
We were thinking on how to solve this in a minimal, non-intrusive
way without bloating tcp_ecn_create_request() needlessly: lets cache
the CA ecn option flag in RTAX_FEATURES. In other words, when ECT(0)
is set on the SYN packet, set ecn_ok=1 iff route RTAX_FEATURES
contains the unexposed (internal-only) DST_FEATURE_ECN_CA. This allows
to only do a single metric feature lookup inside tcp_ecn_create_request().
Joint work with Florian Westphal.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-08-31 13:58:47 +00:00
|
|
|
val = tcp_ca_get_key_by_name(tmp, &ecn_ca);
|
2015-08-31 13:58:45 +00:00
|
|
|
if (val == TCP_CA_UNSPEC)
|
|
|
|
goto err;
|
|
|
|
} else {
|
|
|
|
val = nla_get_u32(nla);
|
2015-01-05 22:57:44 +00:00
|
|
|
}
|
2016-05-13 16:33:41 +00:00
|
|
|
if (type == RTAX_HOPLIMIT && val > 255)
|
|
|
|
val = 255;
|
2015-08-31 13:58:46 +00:00
|
|
|
if (type == RTAX_FEATURES && (val & ~RTAX_FEATURE_MASK))
|
|
|
|
goto err;
|
2015-08-31 13:58:45 +00:00
|
|
|
|
|
|
|
mp[type - 1] = val;
|
|
|
|
__set_bit(type - 1, mxc->mx_valid);
|
2015-01-05 22:57:44 +00:00
|
|
|
}
|
|
|
|
|
tcp: use dctcp if enabled on the route to the initiator
Currently, the following case doesn't use DCTCP, even if it should:
A responder has f.e. Cubic as system wide default, but for a specific
route to the initiating host, DCTCP is being set in RTAX_CC_ALGO. The
initiating host then uses DCTCP as congestion control, but since the
initiator sets ECT(0), tcp_ecn_create_request() doesn't set ecn_ok,
and we have to fall back to Reno after 3WHS completes.
We were thinking on how to solve this in a minimal, non-intrusive
way without bloating tcp_ecn_create_request() needlessly: lets cache
the CA ecn option flag in RTAX_FEATURES. In other words, when ECT(0)
is set on the SYN packet, set ecn_ok=1 iff route RTAX_FEATURES
contains the unexposed (internal-only) DST_FEATURE_ECN_CA. This allows
to only do a single metric feature lookup inside tcp_ecn_create_request().
Joint work with Florian Westphal.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-08-31 13:58:47 +00:00
|
|
|
if (ecn_ca) {
|
|
|
|
__set_bit(RTAX_FEATURES - 1, mxc->mx_valid);
|
|
|
|
mp[RTAX_FEATURES - 1] |= DST_FEATURE_ECN_CA;
|
|
|
|
}
|
2015-01-05 22:57:44 +00:00
|
|
|
|
tcp: use dctcp if enabled on the route to the initiator
Currently, the following case doesn't use DCTCP, even if it should:
A responder has f.e. Cubic as system wide default, but for a specific
route to the initiating host, DCTCP is being set in RTAX_CC_ALGO. The
initiating host then uses DCTCP as congestion control, but since the
initiator sets ECT(0), tcp_ecn_create_request() doesn't set ecn_ok,
and we have to fall back to Reno after 3WHS completes.
We were thinking on how to solve this in a minimal, non-intrusive
way without bloating tcp_ecn_create_request() needlessly: lets cache
the CA ecn option flag in RTAX_FEATURES. In other words, when ECT(0)
is set on the SYN packet, set ecn_ok=1 iff route RTAX_FEATURES
contains the unexposed (internal-only) DST_FEATURE_ECN_CA. This allows
to only do a single metric feature lookup inside tcp_ecn_create_request().
Joint work with Florian Westphal.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-08-31 13:58:47 +00:00
|
|
|
mxc->mx = mp;
|
2015-01-05 22:57:44 +00:00
|
|
|
return 0;
|
|
|
|
err:
|
|
|
|
kfree(mp);
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2016-04-25 04:26:04 +00:00
|
|
|
static struct rt6_info *ip6_nh_lookup_table(struct net *net,
|
|
|
|
struct fib6_config *cfg,
|
|
|
|
const struct in6_addr *gw_addr)
|
|
|
|
{
|
|
|
|
struct flowi6 fl6 = {
|
|
|
|
.flowi6_oif = cfg->fc_ifindex,
|
|
|
|
.daddr = *gw_addr,
|
|
|
|
.saddr = cfg->fc_prefsrc,
|
|
|
|
};
|
|
|
|
struct fib6_table *table;
|
|
|
|
struct rt6_info *rt;
|
2016-10-24 19:27:23 +00:00
|
|
|
int flags = RT6_LOOKUP_F_IFACE | RT6_LOOKUP_F_IGNORE_LINKSTATE;
|
2016-04-25 04:26:04 +00:00
|
|
|
|
|
|
|
table = fib6_get_table(net, cfg->fc_table);
|
|
|
|
if (!table)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
if (!ipv6_addr_any(&cfg->fc_prefsrc))
|
|
|
|
flags |= RT6_LOOKUP_F_HAS_SADDR;
|
|
|
|
|
|
|
|
rt = ip6_pol_route(net, table, cfg->fc_ifindex, &fl6, flags);
|
|
|
|
|
|
|
|
/* if table lookup failed, fall back to full lookup */
|
|
|
|
if (rt == net->ipv6.ip6_null_entry) {
|
|
|
|
ip6_rt_put(rt);
|
|
|
|
rt = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
return rt;
|
|
|
|
}
|
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
static struct rt6_info *ip6_route_info_create(struct fib6_config *cfg,
|
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2008-03-04 21:47:47 +00:00
|
|
|
struct net *net = cfg->fc_nlinfo.nl_net;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct rt6_info *rt = NULL;
|
|
|
|
struct net_device *dev = NULL;
|
|
|
|
struct inet6_dev *idev = NULL;
|
2006-08-05 06:20:06 +00:00
|
|
|
struct fib6_table *table;
|
2005-04-16 22:20:36 +00:00
|
|
|
int addr_type;
|
2015-10-10 15:26:36 +00:00
|
|
|
int err = -EINVAL;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-04-19 21:19:43 +00:00
|
|
|
/* RTF_PCPU is an internal flag; can not be set by userspace */
|
2017-05-21 16:12:05 +00:00
|
|
|
if (cfg->fc_flags & RTF_PCPU) {
|
|
|
|
NL_SET_ERR_MSG(extack, "Userspace can not set RTF_PCPU");
|
2017-04-19 21:19:43 +00:00
|
|
|
goto out;
|
2017-05-21 16:12:05 +00:00
|
|
|
}
|
2017-04-19 21:19:43 +00:00
|
|
|
|
2017-05-21 16:12:05 +00:00
|
|
|
if (cfg->fc_dst_len > 128) {
|
|
|
|
NL_SET_ERR_MSG(extack, "Invalid prefix length");
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
if (cfg->fc_src_len > 128) {
|
|
|
|
NL_SET_ERR_MSG(extack, "Invalid source address length");
|
2015-10-10 15:26:36 +00:00
|
|
|
goto out;
|
2017-05-21 16:12:05 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
#ifndef CONFIG_IPV6_SUBTREES
|
2017-05-21 16:12:05 +00:00
|
|
|
if (cfg->fc_src_len) {
|
|
|
|
NL_SET_ERR_MSG(extack,
|
|
|
|
"Specifying source address requires IPV6_SUBTREES to be enabled");
|
2015-10-10 15:26:36 +00:00
|
|
|
goto out;
|
2017-05-21 16:12:05 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif
|
2006-08-22 07:01:08 +00:00
|
|
|
if (cfg->fc_ifindex) {
|
2005-04-16 22:20:36 +00:00
|
|
|
err = -ENODEV;
|
2008-03-04 21:47:47 +00:00
|
|
|
dev = dev_get_by_index(net, cfg->fc_ifindex);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (!dev)
|
|
|
|
goto out;
|
|
|
|
idev = in6_dev_get(dev);
|
|
|
|
if (!idev)
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2006-08-22 07:01:08 +00:00
|
|
|
if (cfg->fc_metric == 0)
|
|
|
|
cfg->fc_metric = IP6_RT_PRIO_USER;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-11-14 00:14:49 +00:00
|
|
|
err = -ENOBUFS;
|
2011-12-03 23:02:47 +00:00
|
|
|
if (cfg->fc_nlinfo.nlh &&
|
|
|
|
!(cfg->fc_nlinfo.nlh->nlmsg_flags & NLM_F_CREATE)) {
|
2011-11-14 00:14:49 +00:00
|
|
|
table = fib6_get_table(net, cfg->fc_table);
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!table) {
|
2012-05-15 14:11:53 +00:00
|
|
|
pr_warn("NLM_F_CREATE should be specified when creating new route\n");
|
2011-11-14 00:14:49 +00:00
|
|
|
table = fib6_new_table(net, cfg->fc_table);
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
table = fib6_new_table(net, cfg->fc_table);
|
|
|
|
}
|
2011-12-03 23:02:47 +00:00
|
|
|
|
|
|
|
if (!table)
|
2006-08-05 06:20:06 +00:00
|
|
|
goto out;
|
|
|
|
|
2015-08-14 18:05:52 +00:00
|
|
|
rt = ip6_dst_alloc(net, NULL,
|
|
|
|
(cfg->fc_flags & RTF_ADDRCONF) ? 0 : DST_NOCOUNT);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!rt) {
|
2005-04-16 22:20:36 +00:00
|
|
|
err = -ENOMEM;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2012-04-06 00:13:10 +00:00
|
|
|
if (cfg->fc_flags & RTF_EXPIRES)
|
|
|
|
rt6_set_expires(rt, jiffies +
|
|
|
|
clock_t_to_jiffies(cfg->fc_expires));
|
|
|
|
else
|
|
|
|
rt6_clean_expires(rt);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-08-22 07:01:08 +00:00
|
|
|
if (cfg->fc_protocol == RTPROT_UNSPEC)
|
|
|
|
cfg->fc_protocol = RTPROT_BOOT;
|
|
|
|
rt->rt6i_protocol = cfg->fc_protocol;
|
|
|
|
|
|
|
|
addr_type = ipv6_addr_type(&cfg->fc_dst);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (addr_type & IPV6_ADDR_MULTICAST)
|
2010-06-11 06:31:35 +00:00
|
|
|
rt->dst.input = ip6_mc_input;
|
2010-09-27 00:07:02 +00:00
|
|
|
else if (cfg->fc_flags & RTF_LOCAL)
|
|
|
|
rt->dst.input = ip6_input;
|
2005-04-16 22:20:36 +00:00
|
|
|
else
|
2010-06-11 06:31:35 +00:00
|
|
|
rt->dst.input = ip6_forward;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2010-06-11 06:31:35 +00:00
|
|
|
rt->dst.output = ip6_output;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2015-07-21 08:43:48 +00:00
|
|
|
if (cfg->fc_encap) {
|
|
|
|
struct lwtunnel_state *lwtstate;
|
|
|
|
|
2017-01-30 20:07:37 +00:00
|
|
|
err = lwtunnel_build_state(cfg->fc_encap_type,
|
2015-08-24 16:45:41 +00:00
|
|
|
cfg->fc_encap, AF_INET6, cfg,
|
2017-05-27 22:19:28 +00:00
|
|
|
&lwtstate, extack);
|
2015-07-21 08:43:48 +00:00
|
|
|
if (err)
|
|
|
|
goto out;
|
2015-08-20 11:56:25 +00:00
|
|
|
rt->dst.lwtstate = lwtstate_get(lwtstate);
|
|
|
|
if (lwtunnel_output_redirect(rt->dst.lwtstate)) {
|
|
|
|
rt->dst.lwtstate->orig_output = rt->dst.output;
|
|
|
|
rt->dst.output = lwtunnel_output;
|
2015-08-17 20:42:24 +00:00
|
|
|
}
|
2015-08-20 11:56:25 +00:00
|
|
|
if (lwtunnel_input_redirect(rt->dst.lwtstate)) {
|
|
|
|
rt->dst.lwtstate->orig_input = rt->dst.input;
|
|
|
|
rt->dst.input = lwtunnel_input;
|
2015-08-17 20:42:24 +00:00
|
|
|
}
|
2015-07-21 08:43:48 +00:00
|
|
|
}
|
|
|
|
|
2006-08-22 07:01:08 +00:00
|
|
|
ipv6_addr_prefix(&rt->rt6i_dst.addr, &cfg->fc_dst, cfg->fc_dst_len);
|
|
|
|
rt->rt6i_dst.plen = cfg->fc_dst_len;
|
2015-04-28 20:03:07 +00:00
|
|
|
if (rt->rt6i_dst.plen == 128)
|
2014-03-27 12:04:08 +00:00
|
|
|
rt->dst.flags |= DST_HOST;
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
2006-08-22 07:01:08 +00:00
|
|
|
ipv6_addr_prefix(&rt->rt6i_src.addr, &cfg->fc_src, cfg->fc_src_len);
|
|
|
|
rt->rt6i_src.plen = cfg->fc_src_len;
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif
|
|
|
|
|
2006-08-22 07:01:08 +00:00
|
|
|
rt->rt6i_metric = cfg->fc_metric;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* We cannot add true routes via loopback here,
|
|
|
|
they would result in kernel looping; promote them to reject routes
|
|
|
|
*/
|
2006-08-22 07:01:08 +00:00
|
|
|
if ((cfg->fc_flags & RTF_REJECT) ||
|
2011-12-03 23:02:47 +00:00
|
|
|
(dev && (dev->flags & IFF_LOOPBACK) &&
|
|
|
|
!(addr_type & IPV6_ADDR_LOOPBACK) &&
|
|
|
|
!(cfg->fc_flags & RTF_LOCAL))) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* hold loopback dev/idev if we haven't done so. */
|
2008-03-04 21:47:47 +00:00
|
|
|
if (dev != net->loopback_dev) {
|
2005-04-16 22:20:36 +00:00
|
|
|
if (dev) {
|
|
|
|
dev_put(dev);
|
|
|
|
in6_dev_put(idev);
|
|
|
|
}
|
2008-03-04 21:47:47 +00:00
|
|
|
dev = net->loopback_dev;
|
2005-04-16 22:20:36 +00:00
|
|
|
dev_hold(dev);
|
|
|
|
idev = in6_dev_get(dev);
|
|
|
|
if (!idev) {
|
|
|
|
err = -ENODEV;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
rt->rt6i_flags = RTF_REJECT|RTF_NONEXTHOP;
|
2012-09-05 02:12:42 +00:00
|
|
|
switch (cfg->fc_type) {
|
|
|
|
case RTN_BLACKHOLE:
|
|
|
|
rt->dst.error = -EINVAL;
|
2015-10-07 21:48:47 +00:00
|
|
|
rt->dst.output = dst_discard_out;
|
2013-12-02 14:25:21 +00:00
|
|
|
rt->dst.input = dst_discard;
|
2012-09-05 02:12:42 +00:00
|
|
|
break;
|
|
|
|
case RTN_PROHIBIT:
|
|
|
|
rt->dst.error = -EACCES;
|
2013-12-02 14:25:21 +00:00
|
|
|
rt->dst.output = ip6_pkt_prohibit_out;
|
|
|
|
rt->dst.input = ip6_pkt_prohibit;
|
2012-09-05 02:12:42 +00:00
|
|
|
break;
|
2012-09-06 05:53:35 +00:00
|
|
|
case RTN_THROW:
|
2015-09-17 14:01:32 +00:00
|
|
|
case RTN_UNREACHABLE:
|
2012-09-05 02:12:42 +00:00
|
|
|
default:
|
2013-12-02 14:25:21 +00:00
|
|
|
rt->dst.error = (cfg->fc_type == RTN_THROW) ? -EAGAIN
|
2015-09-17 14:01:32 +00:00
|
|
|
: (cfg->fc_type == RTN_UNREACHABLE)
|
|
|
|
? -EHOSTUNREACH : -ENETUNREACH;
|
2013-12-02 14:25:21 +00:00
|
|
|
rt->dst.output = ip6_pkt_discard_out;
|
|
|
|
rt->dst.input = ip6_pkt_discard;
|
2012-09-05 02:12:42 +00:00
|
|
|
break;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
goto install_route;
|
|
|
|
}
|
|
|
|
|
2006-08-22 07:01:08 +00:00
|
|
|
if (cfg->fc_flags & RTF_GATEWAY) {
|
2011-04-22 04:53:02 +00:00
|
|
|
const struct in6_addr *gw_addr;
|
2005-04-16 22:20:36 +00:00
|
|
|
int gwa_type;
|
|
|
|
|
2006-08-22 07:01:08 +00:00
|
|
|
gw_addr = &cfg->fc_gateway;
|
2015-08-07 08:54:28 +00:00
|
|
|
gwa_type = ipv6_addr_type(gw_addr);
|
2015-05-20 22:25:41 +00:00
|
|
|
|
|
|
|
/* if gw_addr is local we will fail to detect this in case
|
|
|
|
* address is still TENTATIVE (DAD in progress). rt6_lookup()
|
|
|
|
* will return already-added prefix route via interface that
|
|
|
|
* prefix route was assigned to, which might be non-loopback.
|
|
|
|
*/
|
|
|
|
err = -EINVAL;
|
2015-08-07 08:54:28 +00:00
|
|
|
if (ipv6_chk_addr_and_flags(net, gw_addr,
|
|
|
|
gwa_type & IPV6_ADDR_LINKLOCAL ?
|
2017-05-21 16:12:05 +00:00
|
|
|
dev : NULL, 0, 0)) {
|
|
|
|
NL_SET_ERR_MSG(extack, "Invalid gateway address");
|
2015-05-20 22:25:41 +00:00
|
|
|
goto out;
|
2017-05-21 16:12:05 +00:00
|
|
|
}
|
2011-11-21 03:39:03 +00:00
|
|
|
rt->rt6i_gateway = *gw_addr;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (gwa_type != (IPV6_ADDR_LINKLOCAL|IPV6_ADDR_UNICAST)) {
|
2016-04-25 04:26:04 +00:00
|
|
|
struct rt6_info *grt = NULL;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* IPv6 strictly inhibits using not link-local
|
|
|
|
addresses as nexthop address.
|
|
|
|
Otherwise, router will not able to send redirects.
|
|
|
|
It is very good, but in some (rare!) circumstances
|
|
|
|
(SIT, PtP, NBMA NOARP links) it is handy to allow
|
|
|
|
some exceptions. --ANK
|
2016-12-04 04:57:09 +00:00
|
|
|
We allow IPv4-mapped nexthops to support RFC4798-type
|
|
|
|
addressing
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2016-12-04 04:57:09 +00:00
|
|
|
if (!(gwa_type & (IPV6_ADDR_UNICAST |
|
2017-05-21 16:12:05 +00:00
|
|
|
IPV6_ADDR_MAPPED))) {
|
|
|
|
NL_SET_ERR_MSG(extack,
|
|
|
|
"Invalid gateway address");
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out;
|
2017-05-21 16:12:05 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2016-09-18 15:46:07 +00:00
|
|
|
if (cfg->fc_table) {
|
2016-04-25 04:26:04 +00:00
|
|
|
grt = ip6_nh_lookup_table(net, cfg, gw_addr);
|
|
|
|
|
2016-09-18 15:46:07 +00:00
|
|
|
if (grt) {
|
|
|
|
if (grt->rt6i_flags & RTF_GATEWAY ||
|
|
|
|
(dev && dev != grt->dst.dev)) {
|
|
|
|
ip6_rt_put(grt);
|
|
|
|
grt = NULL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-04-25 04:26:04 +00:00
|
|
|
if (!grt)
|
|
|
|
grt = rt6_lookup(net, gw_addr, NULL,
|
|
|
|
cfg->fc_ifindex, 1);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
err = -EHOSTUNREACH;
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!grt)
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out;
|
|
|
|
if (dev) {
|
2011-12-29 01:19:20 +00:00
|
|
|
if (dev != grt->dst.dev) {
|
2012-10-29 00:13:19 +00:00
|
|
|
ip6_rt_put(grt);
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
} else {
|
2011-12-29 01:19:20 +00:00
|
|
|
dev = grt->dst.dev;
|
2005-04-16 22:20:36 +00:00
|
|
|
idev = grt->rt6i_idev;
|
|
|
|
dev_hold(dev);
|
|
|
|
in6_dev_hold(grt->rt6i_idev);
|
|
|
|
}
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!(grt->rt6i_flags & RTF_GATEWAY))
|
2005-04-16 22:20:36 +00:00
|
|
|
err = 0;
|
2012-10-29 00:13:19 +00:00
|
|
|
ip6_rt_put(grt);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (err)
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
err = -EINVAL;
|
2017-05-21 16:12:05 +00:00
|
|
|
if (!dev) {
|
|
|
|
NL_SET_ERR_MSG(extack, "Egress device not specified");
|
|
|
|
goto out;
|
|
|
|
} else if (dev->flags & IFF_LOOPBACK) {
|
|
|
|
NL_SET_ERR_MSG(extack,
|
|
|
|
"Egress device can not be loopback device for this route");
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out;
|
2017-05-21 16:12:05 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
err = -ENODEV;
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!dev)
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out;
|
|
|
|
|
2011-04-13 21:10:57 +00:00
|
|
|
if (!ipv6_addr_any(&cfg->fc_prefsrc)) {
|
|
|
|
if (!ipv6_chk_addr(net, &cfg->fc_prefsrc, dev, 0)) {
|
2017-05-21 16:12:05 +00:00
|
|
|
NL_SET_ERR_MSG(extack, "Invalid source address");
|
2011-04-13 21:10:57 +00:00
|
|
|
err = -EINVAL;
|
|
|
|
goto out;
|
|
|
|
}
|
2011-11-21 03:39:03 +00:00
|
|
|
rt->rt6i_prefsrc.addr = cfg->fc_prefsrc;
|
2011-04-13 21:10:57 +00:00
|
|
|
rt->rt6i_prefsrc.plen = 128;
|
|
|
|
} else
|
|
|
|
rt->rt6i_prefsrc.plen = 0;
|
|
|
|
|
2006-08-22 07:01:08 +00:00
|
|
|
rt->rt6i_flags = cfg->fc_flags;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
install_route:
|
2010-06-11 06:31:35 +00:00
|
|
|
rt->dst.dev = dev;
|
2005-04-16 22:20:36 +00:00
|
|
|
rt->rt6i_idev = idev;
|
2006-08-05 06:20:06 +00:00
|
|
|
rt->rt6i_table = table;
|
2008-03-04 07:31:11 +00:00
|
|
|
|
2008-03-25 12:47:49 +00:00
|
|
|
cfg->fc_nlinfo.nl_net = dev_net(dev);
|
2008-03-04 07:31:11 +00:00
|
|
|
|
2015-10-10 15:26:36 +00:00
|
|
|
return rt;
|
2015-09-08 17:53:04 +00:00
|
|
|
out:
|
|
|
|
if (dev)
|
|
|
|
dev_put(dev);
|
|
|
|
if (idev)
|
|
|
|
in6_dev_put(idev);
|
2017-06-17 17:42:36 +00:00
|
|
|
if (rt)
|
|
|
|
dst_release_immediate(&rt->dst);
|
2015-09-08 17:53:04 +00:00
|
|
|
|
2015-10-10 15:26:36 +00:00
|
|
|
return ERR_PTR(err);
|
2015-09-08 17:53:04 +00:00
|
|
|
}
|
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
int ip6_route_add(struct fib6_config *cfg,
|
|
|
|
struct netlink_ext_ack *extack)
|
2015-09-08 17:53:04 +00:00
|
|
|
{
|
|
|
|
struct mx6_config mxc = { .mx = NULL, };
|
2015-10-10 15:26:36 +00:00
|
|
|
struct rt6_info *rt;
|
2015-09-08 17:53:04 +00:00
|
|
|
int err;
|
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
rt = ip6_route_info_create(cfg, extack);
|
2015-10-10 15:26:36 +00:00
|
|
|
if (IS_ERR(rt)) {
|
|
|
|
err = PTR_ERR(rt);
|
|
|
|
rt = NULL;
|
2015-09-08 17:53:04 +00:00
|
|
|
goto out;
|
2015-10-10 15:26:36 +00:00
|
|
|
}
|
2015-09-08 17:53:04 +00:00
|
|
|
|
2015-01-05 22:57:44 +00:00
|
|
|
err = ip6_convert_metrics(&mxc, cfg);
|
|
|
|
if (err)
|
|
|
|
goto out;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
err = __ip6_ins_rt(rt, &cfg->fc_nlinfo, &mxc, extack);
|
2015-01-05 22:57:44 +00:00
|
|
|
|
|
|
|
kfree(mxc.mx);
|
2015-09-08 17:53:04 +00:00
|
|
|
|
2015-01-05 22:57:44 +00:00
|
|
|
return err;
|
2005-04-16 22:20:36 +00:00
|
|
|
out:
|
2017-06-17 17:42:36 +00:00
|
|
|
if (rt)
|
|
|
|
dst_release_immediate(&rt->dst);
|
2015-09-08 17:53:04 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2006-08-22 07:01:08 +00:00
|
|
|
static int __ip6_del_rt(struct rt6_info *rt, struct nl_info *info)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
int err;
|
2006-08-05 06:20:06 +00:00
|
|
|
struct fib6_table *table;
|
2011-12-29 01:19:20 +00:00
|
|
|
struct net *net = dev_net(rt->dst.dev);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-06-17 17:42:42 +00:00
|
|
|
if (rt == net->ipv6.ip6_null_entry) {
|
2012-09-19 19:25:34 +00:00
|
|
|
err = -ENOENT;
|
|
|
|
goto out;
|
|
|
|
}
|
2006-08-07 05:22:47 +00:00
|
|
|
|
2006-08-05 06:20:06 +00:00
|
|
|
table = rt->rt6i_table;
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
spin_lock_bh(&table->tb6_lock);
|
2006-08-22 07:01:08 +00:00
|
|
|
err = fib6_del(rt, info);
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
spin_unlock_bh(&table->tb6_lock);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2012-09-19 19:25:34 +00:00
|
|
|
out:
|
2012-10-29 00:13:19 +00:00
|
|
|
ip6_rt_put(rt);
|
2005-04-16 22:20:36 +00:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2006-08-22 07:00:21 +00:00
|
|
|
int ip6_del_rt(struct rt6_info *rt)
|
|
|
|
{
|
2008-01-10 11:26:13 +00:00
|
|
|
struct nl_info info = {
|
2011-12-29 01:19:20 +00:00
|
|
|
.nl_net = dev_net(rt->dst.dev),
|
2008-01-10 11:26:13 +00:00
|
|
|
};
|
2007-12-13 17:45:12 +00:00
|
|
|
return __ip6_del_rt(rt, &info);
|
2006-08-22 07:00:21 +00:00
|
|
|
}
|
|
|
|
|
2017-02-02 20:37:08 +00:00
|
|
|
static int __ip6_del_rt_siblings(struct rt6_info *rt, struct fib6_config *cfg)
|
|
|
|
{
|
|
|
|
struct nl_info *info = &cfg->fc_nlinfo;
|
2017-02-28 00:07:43 +00:00
|
|
|
struct net *net = info->nl_net;
|
2017-02-02 20:37:11 +00:00
|
|
|
struct sk_buff *skb = NULL;
|
2017-02-02 20:37:08 +00:00
|
|
|
struct fib6_table *table;
|
2017-02-28 00:07:43 +00:00
|
|
|
int err = -ENOENT;
|
2017-02-02 20:37:08 +00:00
|
|
|
|
2017-02-28 00:07:43 +00:00
|
|
|
if (rt == net->ipv6.ip6_null_entry)
|
|
|
|
goto out_put;
|
2017-02-02 20:37:08 +00:00
|
|
|
table = rt->rt6i_table;
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
spin_lock_bh(&table->tb6_lock);
|
2017-02-02 20:37:08 +00:00
|
|
|
|
|
|
|
if (rt->rt6i_nsiblings && cfg->fc_delete_all_nh) {
|
|
|
|
struct rt6_info *sibling, *next_sibling;
|
|
|
|
|
2017-02-02 20:37:11 +00:00
|
|
|
/* prefer to send a single notification with all hops */
|
|
|
|
skb = nlmsg_new(rt6_nlmsg_size(rt), gfp_any());
|
|
|
|
if (skb) {
|
|
|
|
u32 seq = info->nlh ? info->nlh->nlmsg_seq : 0;
|
|
|
|
|
2017-02-28 00:07:43 +00:00
|
|
|
if (rt6_fill_node(net, skb, rt,
|
2017-02-02 20:37:11 +00:00
|
|
|
NULL, NULL, 0, RTM_DELROUTE,
|
|
|
|
info->portid, seq, 0) < 0) {
|
|
|
|
kfree_skb(skb);
|
|
|
|
skb = NULL;
|
|
|
|
} else
|
|
|
|
info->skip_notify = 1;
|
|
|
|
}
|
|
|
|
|
2017-02-02 20:37:08 +00:00
|
|
|
list_for_each_entry_safe(sibling, next_sibling,
|
|
|
|
&rt->rt6i_siblings,
|
|
|
|
rt6i_siblings) {
|
|
|
|
err = fib6_del(sibling, info);
|
|
|
|
if (err)
|
2017-02-28 00:07:43 +00:00
|
|
|
goto out_unlock;
|
2017-02-02 20:37:08 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
err = fib6_del(rt, info);
|
2017-02-28 00:07:43 +00:00
|
|
|
out_unlock:
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
spin_unlock_bh(&table->tb6_lock);
|
2017-02-28 00:07:43 +00:00
|
|
|
out_put:
|
2017-02-02 20:37:08 +00:00
|
|
|
ip6_rt_put(rt);
|
2017-02-02 20:37:11 +00:00
|
|
|
|
|
|
|
if (skb) {
|
2017-02-28 00:07:43 +00:00
|
|
|
rtnl_notify(skb, net, info->portid, RTNLGRP_IPV6_ROUTE,
|
2017-02-02 20:37:11 +00:00
|
|
|
info->nlh, gfp_any());
|
|
|
|
}
|
2017-02-02 20:37:08 +00:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
static int ip6_route_del(struct fib6_config *cfg,
|
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2017-10-06 19:06:03 +00:00
|
|
|
struct rt6_info *rt, *rt_cache;
|
2006-08-05 06:20:06 +00:00
|
|
|
struct fib6_table *table;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct fib6_node *fn;
|
|
|
|
int err = -ESRCH;
|
|
|
|
|
2008-03-04 21:47:47 +00:00
|
|
|
table = fib6_get_table(cfg->fc_nlinfo.nl_net, cfg->fc_table);
|
2017-05-21 16:12:05 +00:00
|
|
|
if (!table) {
|
|
|
|
NL_SET_ERR_MSG(extack, "FIB table does not exist");
|
2006-08-05 06:20:06 +00:00
|
|
|
return err;
|
2017-05-21 16:12:05 +00:00
|
|
|
}
|
2006-08-05 06:20:06 +00:00
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_lock();
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-08-05 06:20:06 +00:00
|
|
|
fn = fib6_locate(&table->tb6_root,
|
2006-08-22 07:01:08 +00:00
|
|
|
&cfg->fc_dst, cfg->fc_dst_len,
|
2017-10-06 19:06:02 +00:00
|
|
|
&cfg->fc_src, cfg->fc_src_len,
|
2017-10-06 19:06:03 +00:00
|
|
|
!(cfg->fc_flags & RTF_CACHE));
|
2007-02-09 14:24:49 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
if (fn) {
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
for_each_fib6_node_rt_rcu(fn) {
|
2017-10-06 19:06:03 +00:00
|
|
|
if (cfg->fc_flags & RTF_CACHE) {
|
|
|
|
rt_cache = rt6_find_cached_rt(rt, &cfg->fc_dst,
|
|
|
|
&cfg->fc_src);
|
|
|
|
if (!rt_cache)
|
|
|
|
continue;
|
|
|
|
rt = rt_cache;
|
|
|
|
}
|
2006-08-22 07:01:08 +00:00
|
|
|
if (cfg->fc_ifindex &&
|
2011-12-29 01:19:20 +00:00
|
|
|
(!rt->dst.dev ||
|
|
|
|
rt->dst.dev->ifindex != cfg->fc_ifindex))
|
2005-04-16 22:20:36 +00:00
|
|
|
continue;
|
2006-08-22 07:01:08 +00:00
|
|
|
if (cfg->fc_flags & RTF_GATEWAY &&
|
|
|
|
!ipv6_addr_equal(&cfg->fc_gateway, &rt->rt6i_gateway))
|
2005-04-16 22:20:36 +00:00
|
|
|
continue;
|
2006-08-22 07:01:08 +00:00
|
|
|
if (cfg->fc_metric && cfg->fc_metric != rt->rt6i_metric)
|
2005-04-16 22:20:36 +00:00
|
|
|
continue;
|
2016-12-16 08:30:59 +00:00
|
|
|
if (cfg->fc_protocol && cfg->fc_protocol != rt->rt6i_protocol)
|
|
|
|
continue;
|
2017-10-06 19:06:06 +00:00
|
|
|
if (!dst_hold_safe(&rt->dst))
|
|
|
|
break;
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-02-02 20:37:08 +00:00
|
|
|
/* if gateway was specified only delete the one hop */
|
|
|
|
if (cfg->fc_flags & RTF_GATEWAY)
|
|
|
|
return __ip6_del_rt(rt, &cfg->fc_nlinfo);
|
|
|
|
|
|
|
|
return __ip6_del_rt_siblings(rt, cfg);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2012-07-17 10:29:28 +00:00
|
|
|
static void rt6_do_redirect(struct dst_entry *dst, struct sock *sk, struct sk_buff *skb)
|
2006-08-24 00:18:26 +00:00
|
|
|
{
|
|
|
|
struct netevent_redirect netevent;
|
2012-07-12 06:43:53 +00:00
|
|
|
struct rt6_info *rt, *nrt = NULL;
|
|
|
|
struct ndisc_options ndopts;
|
|
|
|
struct inet6_dev *in6_dev;
|
|
|
|
struct neighbour *neigh;
|
2013-01-05 16:34:51 +00:00
|
|
|
struct rd_msg *msg;
|
2012-07-12 07:05:02 +00:00
|
|
|
int optlen, on_link;
|
|
|
|
u8 *lladdr;
|
2012-07-12 06:43:53 +00:00
|
|
|
|
2013-05-28 20:34:26 +00:00
|
|
|
optlen = skb_tail_pointer(skb) - skb_transport_header(skb);
|
2013-01-05 16:34:51 +00:00
|
|
|
optlen -= sizeof(*msg);
|
2012-07-12 06:43:53 +00:00
|
|
|
|
|
|
|
if (optlen < 0) {
|
2012-07-12 07:05:02 +00:00
|
|
|
net_dbg_ratelimited("rt6_do_redirect: packet too short\n");
|
2012-07-12 06:43:53 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2013-01-05 16:34:51 +00:00
|
|
|
msg = (struct rd_msg *)icmp6_hdr(skb);
|
2012-07-12 06:43:53 +00:00
|
|
|
|
2013-01-05 16:34:51 +00:00
|
|
|
if (ipv6_addr_is_multicast(&msg->dest)) {
|
2012-07-12 07:05:02 +00:00
|
|
|
net_dbg_ratelimited("rt6_do_redirect: destination address is multicast\n");
|
2012-07-12 06:43:53 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2012-07-12 07:05:02 +00:00
|
|
|
on_link = 0;
|
2013-01-05 16:34:51 +00:00
|
|
|
if (ipv6_addr_equal(&msg->dest, &msg->target)) {
|
2012-07-12 06:43:53 +00:00
|
|
|
on_link = 1;
|
2013-01-05 16:34:51 +00:00
|
|
|
} else if (ipv6_addr_type(&msg->target) !=
|
2012-07-12 06:43:53 +00:00
|
|
|
(IPV6_ADDR_UNICAST|IPV6_ADDR_LINKLOCAL)) {
|
2012-07-12 07:05:02 +00:00
|
|
|
net_dbg_ratelimited("rt6_do_redirect: target address is not link-local unicast\n");
|
2012-07-12 06:43:53 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
in6_dev = __in6_dev_get(skb->dev);
|
|
|
|
if (!in6_dev)
|
|
|
|
return;
|
|
|
|
if (in6_dev->cnf.forwarding || !in6_dev->cnf.accept_redirects)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* RFC2461 8.1:
|
|
|
|
* The IP source address of the Redirect MUST be the same as the current
|
|
|
|
* first-hop router for the specified ICMP Destination Address.
|
|
|
|
*/
|
|
|
|
|
2016-06-15 19:20:23 +00:00
|
|
|
if (!ndisc_parse_options(skb->dev, msg->opt, optlen, &ndopts)) {
|
2012-07-12 06:43:53 +00:00
|
|
|
net_dbg_ratelimited("rt6_redirect: invalid ND options\n");
|
|
|
|
return;
|
|
|
|
}
|
2012-07-12 07:05:02 +00:00
|
|
|
|
|
|
|
lladdr = NULL;
|
2012-07-12 06:43:53 +00:00
|
|
|
if (ndopts.nd_opts_tgt_lladdr) {
|
|
|
|
lladdr = ndisc_opt_addr_data(ndopts.nd_opts_tgt_lladdr,
|
|
|
|
skb->dev);
|
|
|
|
if (!lladdr) {
|
|
|
|
net_dbg_ratelimited("rt6_redirect: invalid link-layer address length\n");
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-07-12 07:05:02 +00:00
|
|
|
rt = (struct rt6_info *) dst;
|
2015-11-02 00:24:38 +00:00
|
|
|
if (rt->rt6i_flags & RTF_REJECT) {
|
2012-07-12 07:05:02 +00:00
|
|
|
net_dbg_ratelimited("rt6_redirect: source isn't a valid nexthop for redirect target\n");
|
2012-07-12 06:43:53 +00:00
|
|
|
return;
|
2012-07-12 07:05:02 +00:00
|
|
|
}
|
2012-07-12 06:43:53 +00:00
|
|
|
|
2012-07-12 07:05:02 +00:00
|
|
|
/* Redirect received -> path was valid.
|
|
|
|
* Look, redirects are sent only in response to data packets,
|
|
|
|
* so that this nexthop apparently is reachable. --ANK
|
|
|
|
*/
|
2017-02-06 21:14:16 +00:00
|
|
|
dst_confirm_neigh(&rt->dst, &ipv6_hdr(skb)->saddr);
|
2006-08-24 00:18:26 +00:00
|
|
|
|
2013-01-05 16:34:51 +00:00
|
|
|
neigh = __neigh_lookup(&nd_tbl, &msg->target, skb->dev, 1);
|
2012-07-12 07:05:02 +00:00
|
|
|
if (!neigh)
|
|
|
|
return;
|
2006-08-24 00:18:26 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* We have finally decided to accept it.
|
|
|
|
*/
|
|
|
|
|
2016-06-15 19:20:23 +00:00
|
|
|
ndisc_update(skb->dev, neigh, lladdr, NUD_STALE,
|
2005-04-16 22:20:36 +00:00
|
|
|
NEIGH_UPDATE_F_WEAK_OVERRIDE|
|
|
|
|
NEIGH_UPDATE_F_OVERRIDE|
|
|
|
|
(on_link ? 0 : (NEIGH_UPDATE_F_OVERRIDE_ISROUTER|
|
2016-06-15 19:20:23 +00:00
|
|
|
NEIGH_UPDATE_F_ISROUTER)),
|
|
|
|
NDISC_REDIRECT, &ndopts);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2015-05-23 03:56:05 +00:00
|
|
|
nrt = ip6_rt_cache_alloc(rt, &msg->dest, NULL);
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!nrt)
|
2005-04-16 22:20:36 +00:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
nrt->rt6i_flags = RTF_GATEWAY|RTF_UP|RTF_DYNAMIC|RTF_CACHE;
|
|
|
|
if (on_link)
|
|
|
|
nrt->rt6i_flags &= ~RTF_GATEWAY;
|
|
|
|
|
2017-08-03 06:13:46 +00:00
|
|
|
nrt->rt6i_protocol = RTPROT_REDIRECT;
|
2011-11-21 03:39:03 +00:00
|
|
|
nrt->rt6i_gateway = *(struct in6_addr *)neigh->primary_key;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-10-06 19:06:03 +00:00
|
|
|
/* No need to remove rt from the exception table if rt is
|
|
|
|
* a cached route because rt6_insert_exception() will
|
|
|
|
* takes care of it
|
|
|
|
*/
|
|
|
|
if (rt6_insert_exception(nrt, rt)) {
|
|
|
|
dst_release_immediate(&nrt->dst);
|
|
|
|
goto out;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2010-06-11 06:31:35 +00:00
|
|
|
netevent.old = &rt->dst;
|
|
|
|
netevent.new = &nrt->dst;
|
2013-01-05 16:34:51 +00:00
|
|
|
netevent.daddr = &msg->dest;
|
2013-01-14 09:28:27 +00:00
|
|
|
netevent.neigh = neigh;
|
2006-07-31 03:43:36 +00:00
|
|
|
call_netevent_notifiers(NETEVENT_REDIRECT, &netevent);
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
out:
|
2012-07-12 06:43:53 +00:00
|
|
|
neigh_release(neigh);
|
2012-07-12 07:05:02 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* Misc support functions
|
|
|
|
*/
|
|
|
|
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
static void rt6_set_from(struct rt6_info *rt, struct rt6_info *from)
|
|
|
|
{
|
|
|
|
BUG_ON(from->dst.from);
|
|
|
|
|
|
|
|
rt->rt6i_flags &= ~RTF_EXPIRES;
|
|
|
|
dst_hold(&from->dst);
|
|
|
|
rt->dst.from = &from->dst;
|
|
|
|
dst_init_metrics(&rt->dst, dst_metrics_ptr(&from->dst), true);
|
|
|
|
}
|
|
|
|
|
2015-05-23 03:56:05 +00:00
|
|
|
static void ip6_rt_copy_init(struct rt6_info *rt, struct rt6_info *ort)
|
|
|
|
{
|
|
|
|
rt->dst.input = ort->dst.input;
|
|
|
|
rt->dst.output = ort->dst.output;
|
|
|
|
rt->rt6i_dst = ort->rt6i_dst;
|
|
|
|
rt->dst.error = ort->dst.error;
|
|
|
|
rt->rt6i_idev = ort->rt6i_idev;
|
|
|
|
if (rt->rt6i_idev)
|
|
|
|
in6_dev_hold(rt->rt6i_idev);
|
|
|
|
rt->dst.lastuse = jiffies;
|
|
|
|
rt->rt6i_gateway = ort->rt6i_gateway;
|
|
|
|
rt->rt6i_flags = ort->rt6i_flags;
|
|
|
|
rt6_set_from(rt, ort);
|
|
|
|
rt->rt6i_metric = ort->rt6i_metric;
|
2005-04-16 22:20:36 +00:00
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
2015-05-23 03:56:05 +00:00
|
|
|
rt->rt6i_src = ort->rt6i_src;
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif
|
2015-05-23 03:56:05 +00:00
|
|
|
rt->rt6i_prefsrc = ort->rt6i_prefsrc;
|
|
|
|
rt->rt6i_table = ort->rt6i_table;
|
2015-08-20 11:56:25 +00:00
|
|
|
rt->dst.lwtstate = lwtstate_get(ort->dst.lwtstate);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2006-03-21 01:06:24 +00:00
|
|
|
#ifdef CONFIG_IPV6_ROUTE_INFO
|
2008-03-04 21:46:48 +00:00
|
|
|
static struct rt6_info *rt6_get_route_info(struct net *net,
|
2011-04-22 04:53:02 +00:00
|
|
|
const struct in6_addr *prefix, int prefixlen,
|
2016-10-24 17:52:35 +00:00
|
|
|
const struct in6_addr *gwaddr,
|
|
|
|
struct net_device *dev)
|
2006-03-21 01:06:24 +00:00
|
|
|
{
|
2016-10-24 17:52:35 +00:00
|
|
|
u32 tb_id = l3mdev_fib_table(dev) ? : RT6_TABLE_INFO;
|
|
|
|
int ifindex = dev->ifindex;
|
2006-03-21 01:06:24 +00:00
|
|
|
struct fib6_node *fn;
|
|
|
|
struct rt6_info *rt = NULL;
|
2006-08-05 06:20:06 +00:00
|
|
|
struct fib6_table *table;
|
|
|
|
|
2016-10-24 17:52:35 +00:00
|
|
|
table = fib6_get_table(net, tb_id);
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!table)
|
2006-08-05 06:20:06 +00:00
|
|
|
return NULL;
|
2006-03-21 01:06:24 +00:00
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_lock();
|
2017-10-06 19:06:02 +00:00
|
|
|
fn = fib6_locate(&table->tb6_root, prefix, prefixlen, NULL, 0, true);
|
2006-03-21 01:06:24 +00:00
|
|
|
if (!fn)
|
|
|
|
goto out;
|
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
for_each_fib6_node_rt_rcu(fn) {
|
2011-12-29 01:19:20 +00:00
|
|
|
if (rt->dst.dev->ifindex != ifindex)
|
2006-03-21 01:06:24 +00:00
|
|
|
continue;
|
|
|
|
if ((rt->rt6i_flags & (RTF_ROUTEINFO|RTF_GATEWAY)) != (RTF_ROUTEINFO|RTF_GATEWAY))
|
|
|
|
continue;
|
|
|
|
if (!ipv6_addr_equal(&rt->rt6i_gateway, gwaddr))
|
|
|
|
continue;
|
2017-10-06 19:06:06 +00:00
|
|
|
ip6_hold_safe(NULL, &rt, false);
|
2006-03-21 01:06:24 +00:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
out:
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2006-03-21 01:06:24 +00:00
|
|
|
return rt;
|
|
|
|
}
|
|
|
|
|
2008-03-04 21:46:48 +00:00
|
|
|
static struct rt6_info *rt6_add_route_info(struct net *net,
|
2011-04-22 04:53:02 +00:00
|
|
|
const struct in6_addr *prefix, int prefixlen,
|
2016-10-24 17:52:35 +00:00
|
|
|
const struct in6_addr *gwaddr,
|
|
|
|
struct net_device *dev,
|
2012-04-15 05:58:06 +00:00
|
|
|
unsigned int pref)
|
2006-03-21 01:06:24 +00:00
|
|
|
{
|
2006-08-22 07:01:08 +00:00
|
|
|
struct fib6_config cfg = {
|
2008-02-10 07:43:11 +00:00
|
|
|
.fc_metric = IP6_RT_PRIO_USER,
|
2016-10-24 17:52:35 +00:00
|
|
|
.fc_ifindex = dev->ifindex,
|
2006-08-22 07:01:08 +00:00
|
|
|
.fc_dst_len = prefixlen,
|
|
|
|
.fc_flags = RTF_GATEWAY | RTF_ADDRCONF | RTF_ROUTEINFO |
|
|
|
|
RTF_UP | RTF_PREF(pref),
|
2017-08-03 06:13:46 +00:00
|
|
|
.fc_protocol = RTPROT_RA,
|
2012-09-07 20:12:54 +00:00
|
|
|
.fc_nlinfo.portid = 0,
|
2008-03-04 21:46:48 +00:00
|
|
|
.fc_nlinfo.nlh = NULL,
|
|
|
|
.fc_nlinfo.nl_net = net,
|
2006-08-22 07:01:08 +00:00
|
|
|
};
|
|
|
|
|
2016-10-24 17:52:35 +00:00
|
|
|
cfg.fc_table = l3mdev_fib_table(dev) ? : RT6_TABLE_INFO,
|
2011-11-21 03:39:03 +00:00
|
|
|
cfg.fc_dst = *prefix;
|
|
|
|
cfg.fc_gateway = *gwaddr;
|
2006-03-21 01:06:24 +00:00
|
|
|
|
2006-03-21 01:06:42 +00:00
|
|
|
/* We should treat it as a default route if prefix length is 0. */
|
|
|
|
if (!prefixlen)
|
2006-08-22 07:01:08 +00:00
|
|
|
cfg.fc_flags |= RTF_DEFAULT;
|
2006-03-21 01:06:24 +00:00
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
ip6_route_add(&cfg, NULL);
|
2006-03-21 01:06:24 +00:00
|
|
|
|
2016-10-24 17:52:35 +00:00
|
|
|
return rt6_get_route_info(net, prefix, prefixlen, gwaddr, dev);
|
2006-03-21 01:06:24 +00:00
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2011-04-22 04:53:02 +00:00
|
|
|
struct rt6_info *rt6_get_dflt_router(const struct in6_addr *addr, struct net_device *dev)
|
2007-02-09 14:24:49 +00:00
|
|
|
{
|
2016-10-24 17:52:35 +00:00
|
|
|
u32 tb_id = l3mdev_fib_table(dev) ? : RT6_TABLE_DFLT;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct rt6_info *rt;
|
2006-08-05 06:20:06 +00:00
|
|
|
struct fib6_table *table;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2016-10-24 17:52:35 +00:00
|
|
|
table = fib6_get_table(dev_net(dev), tb_id);
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!table)
|
2006-08-05 06:20:06 +00:00
|
|
|
return NULL;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_lock();
|
|
|
|
for_each_fib6_node_rt_rcu(&table->tb6_root) {
|
2011-12-29 01:19:20 +00:00
|
|
|
if (dev == rt->dst.dev &&
|
2006-03-21 01:00:48 +00:00
|
|
|
((rt->rt6i_flags & (RTF_ADDRCONF | RTF_DEFAULT)) == (RTF_ADDRCONF | RTF_DEFAULT)) &&
|
2005-04-16 22:20:36 +00:00
|
|
|
ipv6_addr_equal(&rt->rt6i_gateway, addr))
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (rt)
|
2017-10-06 19:06:06 +00:00
|
|
|
ip6_hold_safe(NULL, &rt, false);
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2005-04-16 22:20:36 +00:00
|
|
|
return rt;
|
|
|
|
}
|
|
|
|
|
2011-04-22 04:53:02 +00:00
|
|
|
struct rt6_info *rt6_add_dflt_router(const struct in6_addr *gwaddr,
|
2006-03-21 01:04:53 +00:00
|
|
|
struct net_device *dev,
|
|
|
|
unsigned int pref)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2006-08-22 07:01:08 +00:00
|
|
|
struct fib6_config cfg = {
|
2015-10-12 18:47:10 +00:00
|
|
|
.fc_table = l3mdev_fib_table(dev) ? : RT6_TABLE_DFLT,
|
2008-02-10 07:43:11 +00:00
|
|
|
.fc_metric = IP6_RT_PRIO_USER,
|
2006-08-22 07:01:08 +00:00
|
|
|
.fc_ifindex = dev->ifindex,
|
|
|
|
.fc_flags = RTF_GATEWAY | RTF_ADDRCONF | RTF_DEFAULT |
|
|
|
|
RTF_UP | RTF_EXPIRES | RTF_PREF(pref),
|
2017-08-03 06:13:46 +00:00
|
|
|
.fc_protocol = RTPROT_RA,
|
2012-09-07 20:12:54 +00:00
|
|
|
.fc_nlinfo.portid = 0,
|
2008-03-04 21:47:47 +00:00
|
|
|
.fc_nlinfo.nlh = NULL,
|
2008-03-25 12:47:49 +00:00
|
|
|
.fc_nlinfo.nl_net = dev_net(dev),
|
2006-08-22 07:01:08 +00:00
|
|
|
};
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-11-21 03:39:03 +00:00
|
|
|
cfg.fc_gateway = *gwaddr;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
if (!ip6_route_add(&cfg, NULL)) {
|
2016-10-24 17:52:35 +00:00
|
|
|
struct fib6_table *table;
|
|
|
|
|
|
|
|
table = fib6_get_table(dev_net(dev), cfg.fc_table);
|
|
|
|
if (table)
|
|
|
|
table->flags |= RT6_TABLE_HAS_DFLT_ROUTER;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
return rt6_get_dflt_router(gwaddr, dev);
|
|
|
|
}
|
|
|
|
|
2016-10-24 17:52:35 +00:00
|
|
|
static void __rt6_purge_dflt_routers(struct fib6_table *table)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct rt6_info *rt;
|
|
|
|
|
|
|
|
restart:
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_lock();
|
|
|
|
for_each_fib6_node_rt_rcu(&table->tb6_root) {
|
2013-03-03 20:46:46 +00:00
|
|
|
if (rt->rt6i_flags & (RTF_DEFAULT | RTF_ADDRCONF) &&
|
|
|
|
(!rt->rt6i_idev || rt->rt6i_idev->cnf.accept_ra != 2)) {
|
2017-10-06 19:06:06 +00:00
|
|
|
if (dst_hold_safe(&rt->dst)) {
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2017-10-06 19:06:06 +00:00
|
|
|
ip6_del_rt(rt);
|
|
|
|
} else {
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2017-10-06 19:06:06 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
goto restart;
|
|
|
|
}
|
|
|
|
}
|
ipv6: replace rwlock with rcu and spinlock in fib6_table
With all the preparation work before, we are now ready to replace rwlock
with rcu and spinlock in fib6_table.
That means now all fib6_node in fib6_table are protected by rcu. And
when freeing fib6_node, call_rcu() is used to wait for the rcu grace
period before releasing the memory.
When accessing fib6_node, corresponding rcu APIs need to be used.
And all previous sessions protected by the write lock will now be
protected by the spin lock per table.
All previous sessions protected by read lock will now be protected by
rcu_read_lock().
A couple of things to note here:
1. As part of the work of replacing rwlock with rcu, the linked list of
fn->leaf now has to be rcu protected as well. So both fn->leaf and
rt->dst.rt6_next are now __rcu tagged and corresponding rcu APIs are
used when manipulating them.
2. For fn->rr_ptr, first of all, it also needs to be rcu protected now
and is tagged with __rcu and rcu APIs are used in corresponding places.
Secondly, fn->rr_ptr is changed in rt6_select() which is a reader
thread. This makes the issue a bit complicated. We think a valid
solution for it is to let rt6_select() grab the tb6_lock if it decides
to change it. As it is not in the normal operation and only happens when
there is no valid neighbor cache for the route, we think the performance
impact should be low.
3. fib6_walk_continue() has to be called with tb6_lock held even in the
route dumping related functions, e.g. inet6_dump_fib(),
fib6_tables_dump() and ipv6_route_seq_ops. It is because
fib6_walk_continue() makes modifications to the walker structure, and so
are fib6_repair_tree() and fib6_del_route(). In order to do proper
syncing between them, we need to let fib6_walk_continue() hold the lock.
We may be able to do further improvement on the way we do the tree walk
to get rid of the need for holding the spin lock. But not for now.
4. When fib6_del_route() removes a route from the tree, we no longer
mark rt->dst.rt6_next to NULL to make simultaneous reader be able to
further traverse the list with rcu. However, rt->dst.rt6_next is only
valid within this same rcu period. No one should access it later.
5. All the operation of atomic_inc(rt->rt6i_ref) is changed to be
performed before we publish this route (either by linking it to fn->leaf
or insert it in the list pointed by fn->leaf) just to be safe because as
soon as we publish the route, some read thread will be able to access it.
Signed-off-by: Wei Wang <weiwan@google.com>
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-06 19:06:10 +00:00
|
|
|
rcu_read_unlock();
|
2016-10-24 17:52:35 +00:00
|
|
|
|
|
|
|
table->flags &= ~RT6_TABLE_HAS_DFLT_ROUTER;
|
|
|
|
}
|
|
|
|
|
|
|
|
void rt6_purge_dflt_routers(struct net *net)
|
|
|
|
{
|
|
|
|
struct fib6_table *table;
|
|
|
|
struct hlist_head *head;
|
|
|
|
unsigned int h;
|
|
|
|
|
|
|
|
rcu_read_lock();
|
|
|
|
|
|
|
|
for (h = 0; h < FIB6_TABLE_HASHSZ; h++) {
|
|
|
|
head = &net->ipv6.fib_table_hash[h];
|
|
|
|
hlist_for_each_entry_rcu(table, head, tb6_hlist) {
|
|
|
|
if (table->flags & RT6_TABLE_HAS_DFLT_ROUTER)
|
|
|
|
__rt6_purge_dflt_routers(table);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
rcu_read_unlock();
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2008-03-04 21:47:47 +00:00
|
|
|
static void rtmsg_to_fib6_config(struct net *net,
|
|
|
|
struct in6_rtmsg *rtmsg,
|
2006-08-22 07:01:08 +00:00
|
|
|
struct fib6_config *cfg)
|
|
|
|
{
|
|
|
|
memset(cfg, 0, sizeof(*cfg));
|
|
|
|
|
2015-10-12 18:47:10 +00:00
|
|
|
cfg->fc_table = l3mdev_fib_table_by_index(net, rtmsg->rtmsg_ifindex) ?
|
|
|
|
: RT6_TABLE_MAIN;
|
2006-08-22 07:01:08 +00:00
|
|
|
cfg->fc_ifindex = rtmsg->rtmsg_ifindex;
|
|
|
|
cfg->fc_metric = rtmsg->rtmsg_metric;
|
|
|
|
cfg->fc_expires = rtmsg->rtmsg_info;
|
|
|
|
cfg->fc_dst_len = rtmsg->rtmsg_dst_len;
|
|
|
|
cfg->fc_src_len = rtmsg->rtmsg_src_len;
|
|
|
|
cfg->fc_flags = rtmsg->rtmsg_flags;
|
|
|
|
|
2008-03-04 21:47:47 +00:00
|
|
|
cfg->fc_nlinfo.nl_net = net;
|
2008-02-27 02:10:03 +00:00
|
|
|
|
2011-11-21 03:39:03 +00:00
|
|
|
cfg->fc_dst = rtmsg->rtmsg_dst;
|
|
|
|
cfg->fc_src = rtmsg->rtmsg_src;
|
|
|
|
cfg->fc_gateway = rtmsg->rtmsg_gateway;
|
2006-08-22 07:01:08 +00:00
|
|
|
}
|
|
|
|
|
2008-03-04 21:47:47 +00:00
|
|
|
int ipv6_route_ioctl(struct net *net, unsigned int cmd, void __user *arg)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2006-08-22 07:01:08 +00:00
|
|
|
struct fib6_config cfg;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct in6_rtmsg rtmsg;
|
|
|
|
int err;
|
|
|
|
|
2014-08-24 20:53:10 +00:00
|
|
|
switch (cmd) {
|
2005-04-16 22:20:36 +00:00
|
|
|
case SIOCADDRT: /* Add a route */
|
|
|
|
case SIOCDELRT: /* Delete a route */
|
net: Allow userns root to control ipv6
Allow an unpriviled user who has created a user namespace, and then
created a network namespace to effectively use the new network
namespace, by reducing capable(CAP_NET_ADMIN) and
capable(CAP_NET_RAW) calls to be ns_capable(net->user_ns,
CAP_NET_ADMIN), or capable(net->user_ns, CAP_NET_RAW) calls.
Settings that merely control a single network device are allowed.
Either the network device is a logical network device where
restrictions make no difference or the network device is hardware NIC
that has been explicity moved from the initial network namespace.
In general policy and network stack state changes are allowed while
resource control is left unchanged.
Allow the SIOCSIFADDR ioctl to add ipv6 addresses.
Allow the SIOCDIFADDR ioctl to delete ipv6 addresses.
Allow the SIOCADDRT ioctl to add ipv6 routes.
Allow the SIOCDELRT ioctl to delete ipv6 routes.
Allow creation of ipv6 raw sockets.
Allow setting the IPV6_JOIN_ANYCAST socket option.
Allow setting the IPV6_FL_A_RENEW parameter of the IPV6_FLOWLABEL_MGR
socket option.
Allow setting the IPV6_TRANSPARENT socket option.
Allow setting the IPV6_HOPOPTS socket option.
Allow setting the IPV6_RTHDRDSTOPTS socket option.
Allow setting the IPV6_DSTOPTS socket option.
Allow setting the IPV6_IPSEC_POLICY socket option.
Allow setting the IPV6_XFRM_POLICY socket option.
Allow sending packets with the IPV6_2292HOPOPTS control message.
Allow sending packets with the IPV6_2292DSTOPTS control message.
Allow sending packets with the IPV6_RTHDRDSTOPTS control message.
Allow setting the multicast routing socket options on non multicast
routing sockets.
Allow the SIOCADDTUNNEL, SIOCCHGTUNNEL, and SIOCDELTUNNEL ioctls for
setting up, changing and deleting tunnels over ipv6.
Allow the SIOCADDTUNNEL, SIOCCHGTUNNEL, SIOCDELTUNNEL ioctls for
setting up, changing and deleting ipv6 over ipv4 tunnels.
Allow the SIOCADDPRL, SIOCDELPRL, SIOCCHGPRL ioctls for adding,
deleting, and changing the potential router list for ISATAP tunnels.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-11-16 03:03:06 +00:00
|
|
|
if (!ns_capable(net->user_ns, CAP_NET_ADMIN))
|
2005-04-16 22:20:36 +00:00
|
|
|
return -EPERM;
|
|
|
|
err = copy_from_user(&rtmsg, arg,
|
|
|
|
sizeof(struct in6_rtmsg));
|
|
|
|
if (err)
|
|
|
|
return -EFAULT;
|
2006-08-22 07:01:08 +00:00
|
|
|
|
2008-03-04 21:47:47 +00:00
|
|
|
rtmsg_to_fib6_config(net, &rtmsg, &cfg);
|
2006-08-22 07:01:08 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
rtnl_lock();
|
|
|
|
switch (cmd) {
|
|
|
|
case SIOCADDRT:
|
2017-05-21 16:12:04 +00:00
|
|
|
err = ip6_route_add(&cfg, NULL);
|
2005-04-16 22:20:36 +00:00
|
|
|
break;
|
|
|
|
case SIOCDELRT:
|
2017-05-21 16:12:04 +00:00
|
|
|
err = ip6_route_del(&cfg, NULL);
|
2005-04-16 22:20:36 +00:00
|
|
|
break;
|
|
|
|
default:
|
|
|
|
err = -EINVAL;
|
|
|
|
}
|
|
|
|
rtnl_unlock();
|
|
|
|
|
|
|
|
return err;
|
2007-04-21 00:09:22 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Drop the packet on the floor
|
|
|
|
*/
|
|
|
|
|
2009-06-23 11:31:07 +00:00
|
|
|
static int ip6_pkt_drop(struct sk_buff *skb, u8 code, int ipstats_mib_noroutes)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2007-04-13 23:18:02 +00:00
|
|
|
int type;
|
2009-06-02 05:19:30 +00:00
|
|
|
struct dst_entry *dst = skb_dst(skb);
|
2007-04-13 23:18:02 +00:00
|
|
|
switch (ipstats_mib_noroutes) {
|
|
|
|
case IPSTATS_MIB_INNOROUTES:
|
2007-04-26 00:54:47 +00:00
|
|
|
type = ipv6_addr_type(&ipv6_hdr(skb)->daddr);
|
2010-02-25 23:28:58 +00:00
|
|
|
if (type == IPV6_ADDR_ANY) {
|
2008-10-08 17:54:51 +00:00
|
|
|
IP6_INC_STATS(dev_net(dst->dev), ip6_dst_idev(dst),
|
|
|
|
IPSTATS_MIB_INADDRERRORS);
|
2007-04-13 23:18:02 +00:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
/* FALLTHROUGH */
|
|
|
|
case IPSTATS_MIB_OUTNOROUTES:
|
2008-10-08 17:54:51 +00:00
|
|
|
IP6_INC_STATS(dev_net(dst->dev), ip6_dst_idev(dst),
|
|
|
|
ipstats_mib_noroutes);
|
2007-04-13 23:18:02 +00:00
|
|
|
break;
|
|
|
|
}
|
2010-02-18 08:25:24 +00:00
|
|
|
icmpv6_send(skb, ICMPV6_DEST_UNREACH, code, 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
kfree_skb(skb);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2006-10-19 03:46:54 +00:00
|
|
|
static int ip6_pkt_discard(struct sk_buff *skb)
|
|
|
|
{
|
2007-04-13 23:18:02 +00:00
|
|
|
return ip6_pkt_drop(skb, ICMPV6_NOROUTE, IPSTATS_MIB_INNOROUTES);
|
2006-10-19 03:46:54 +00:00
|
|
|
}
|
|
|
|
|
2015-10-07 21:48:47 +00:00
|
|
|
static int ip6_pkt_discard_out(struct net *net, struct sock *sk, struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2009-06-02 05:19:30 +00:00
|
|
|
skb->dev = skb_dst(skb)->dev;
|
2007-04-13 23:18:02 +00:00
|
|
|
return ip6_pkt_drop(skb, ICMPV6_NOROUTE, IPSTATS_MIB_OUTNOROUTES);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2006-10-19 03:46:54 +00:00
|
|
|
static int ip6_pkt_prohibit(struct sk_buff *skb)
|
|
|
|
{
|
2007-04-13 23:18:02 +00:00
|
|
|
return ip6_pkt_drop(skb, ICMPV6_ADM_PROHIBITED, IPSTATS_MIB_INNOROUTES);
|
2006-10-19 03:46:54 +00:00
|
|
|
}
|
|
|
|
|
2015-10-07 21:48:47 +00:00
|
|
|
static int ip6_pkt_prohibit_out(struct net *net, struct sock *sk, struct sk_buff *skb)
|
2006-10-19 03:46:54 +00:00
|
|
|
{
|
2009-06-02 05:19:30 +00:00
|
|
|
skb->dev = skb_dst(skb)->dev;
|
2007-04-13 23:18:02 +00:00
|
|
|
return ip6_pkt_drop(skb, ICMPV6_ADM_PROHIBITED, IPSTATS_MIB_OUTNOROUTES);
|
2006-10-19 03:46:54 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* Allocate a dst for local (unicast / anycast) address.
|
|
|
|
*/
|
|
|
|
|
|
|
|
struct rt6_info *addrconf_dst_alloc(struct inet6_dev *idev,
|
|
|
|
const struct in6_addr *addr,
|
2011-12-06 21:48:14 +00:00
|
|
|
bool anycast)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2015-10-12 18:47:10 +00:00
|
|
|
u32 tb_id;
|
2008-03-25 12:47:49 +00:00
|
|
|
struct net *net = dev_net(idev->dev);
|
2017-08-17 19:17:20 +00:00
|
|
|
struct net_device *dev = idev->dev;
|
2016-09-10 19:09:54 +00:00
|
|
|
struct rt6_info *rt;
|
|
|
|
|
|
|
|
rt = ip6_dst_alloc(net, dev, DST_NOCOUNT);
|
2013-12-07 02:33:45 +00:00
|
|
|
if (!rt)
|
2005-04-16 22:20:36 +00:00
|
|
|
return ERR_PTR(-ENOMEM);
|
|
|
|
|
|
|
|
in6_dev_hold(idev);
|
|
|
|
|
2011-06-24 22:23:34 +00:00
|
|
|
rt->dst.flags |= DST_HOST;
|
2010-06-11 06:31:35 +00:00
|
|
|
rt->dst.input = ip6_input;
|
|
|
|
rt->dst.output = ip6_output;
|
2005-04-16 22:20:36 +00:00
|
|
|
rt->rt6i_idev = idev;
|
|
|
|
|
2017-02-02 16:52:21 +00:00
|
|
|
rt->rt6i_protocol = RTPROT_KERNEL;
|
2005-04-16 22:20:36 +00:00
|
|
|
rt->rt6i_flags = RTF_UP | RTF_NONEXTHOP;
|
2005-12-21 13:56:42 +00:00
|
|
|
if (anycast)
|
|
|
|
rt->rt6i_flags |= RTF_ANYCAST;
|
|
|
|
else
|
2005-04-16 22:20:36 +00:00
|
|
|
rt->rt6i_flags |= RTF_LOCAL;
|
|
|
|
|
2013-10-20 12:43:04 +00:00
|
|
|
rt->rt6i_gateway = *addr;
|
2011-11-21 03:39:03 +00:00
|
|
|
rt->rt6i_dst.addr = *addr;
|
2005-04-16 22:20:36 +00:00
|
|
|
rt->rt6i_dst.plen = 128;
|
2015-10-12 18:47:10 +00:00
|
|
|
tb_id = l3mdev_fib_table(idev->dev) ? : RT6_TABLE_LOCAL;
|
|
|
|
rt->rt6i_table = fib6_get_table(net, tb_id);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
return rt;
|
|
|
|
}
|
|
|
|
|
2011-04-13 21:10:57 +00:00
|
|
|
/* remove deleted ip from prefsrc entries */
|
|
|
|
struct arg_dev_net_ip {
|
|
|
|
struct net_device *dev;
|
|
|
|
struct net *net;
|
|
|
|
struct in6_addr *addr;
|
|
|
|
};
|
|
|
|
|
|
|
|
static int fib6_remove_prefsrc(struct rt6_info *rt, void *arg)
|
|
|
|
{
|
|
|
|
struct net_device *dev = ((struct arg_dev_net_ip *)arg)->dev;
|
|
|
|
struct net *net = ((struct arg_dev_net_ip *)arg)->net;
|
|
|
|
struct in6_addr *addr = ((struct arg_dev_net_ip *)arg)->addr;
|
|
|
|
|
2011-12-29 01:19:20 +00:00
|
|
|
if (((void *)rt->dst.dev == dev || !dev) &&
|
2011-04-13 21:10:57 +00:00
|
|
|
rt != net->ipv6.ip6_null_entry &&
|
|
|
|
ipv6_addr_equal(addr, &rt->rt6i_prefsrc.addr)) {
|
2017-10-06 19:05:58 +00:00
|
|
|
spin_lock_bh(&rt6_exception_lock);
|
2011-04-13 21:10:57 +00:00
|
|
|
/* remove prefsrc entry */
|
|
|
|
rt->rt6i_prefsrc.plen = 0;
|
2017-10-06 19:05:58 +00:00
|
|
|
/* need to update cache as well */
|
|
|
|
rt6_exceptions_remove_prefsrc(rt);
|
|
|
|
spin_unlock_bh(&rt6_exception_lock);
|
2011-04-13 21:10:57 +00:00
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
void rt6_remove_prefsrc(struct inet6_ifaddr *ifp)
|
|
|
|
{
|
|
|
|
struct net *net = dev_net(ifp->idev->dev);
|
|
|
|
struct arg_dev_net_ip adni = {
|
|
|
|
.dev = ifp->idev->dev,
|
|
|
|
.net = net,
|
|
|
|
.addr = &ifp->addr,
|
|
|
|
};
|
2013-12-27 08:32:38 +00:00
|
|
|
fib6_clean_all(net, fib6_remove_prefsrc, &adni);
|
2011-04-13 21:10:57 +00:00
|
|
|
}
|
|
|
|
|
2014-05-15 07:56:14 +00:00
|
|
|
#define RTF_RA_ROUTER (RTF_ADDRCONF | RTF_DEFAULT | RTF_GATEWAY)
|
|
|
|
|
|
|
|
/* Remove routers and update dst entries when gateway turn into host. */
|
|
|
|
static int fib6_clean_tohost(struct rt6_info *rt, void *arg)
|
|
|
|
{
|
|
|
|
struct in6_addr *gateway = (struct in6_addr *)arg;
|
|
|
|
|
2017-10-06 19:06:03 +00:00
|
|
|
if (((rt->rt6i_flags & RTF_RA_ROUTER) == RTF_RA_ROUTER) &&
|
|
|
|
ipv6_addr_equal(gateway, &rt->rt6i_gateway)) {
|
2014-05-15 07:56:14 +00:00
|
|
|
return -1;
|
|
|
|
}
|
2017-10-06 19:06:00 +00:00
|
|
|
|
|
|
|
/* Further clean up cached routes in exception table.
|
|
|
|
* This is needed because cached route may have a different
|
|
|
|
* gateway than its 'parent' in the case of an ip redirect.
|
|
|
|
*/
|
|
|
|
rt6_exceptions_clean_tohost(rt, gateway);
|
|
|
|
|
2014-05-15 07:56:14 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
void rt6_clean_tohost(struct net *net, struct in6_addr *gateway)
|
|
|
|
{
|
|
|
|
fib6_clean_all(net, fib6_clean_tohost, gateway);
|
|
|
|
}
|
|
|
|
|
2008-03-04 21:48:30 +00:00
|
|
|
struct arg_dev_net {
|
|
|
|
struct net_device *dev;
|
|
|
|
struct net *net;
|
|
|
|
};
|
|
|
|
|
2017-01-18 15:40:36 +00:00
|
|
|
/* called with write lock held for table with rt */
|
2005-04-16 22:20:36 +00:00
|
|
|
static int fib6_ifdown(struct rt6_info *rt, void *arg)
|
|
|
|
{
|
2010-12-16 17:42:40 +00:00
|
|
|
const struct arg_dev_net *adn = arg;
|
|
|
|
const struct net_device *dev = adn->dev;
|
2008-03-04 21:48:30 +00:00
|
|
|
|
2011-12-29 01:19:20 +00:00
|
|
|
if ((rt->dst.dev == dev || !dev) &&
|
2017-01-18 15:40:36 +00:00
|
|
|
rt != adn->net->ipv6.ip6_null_entry &&
|
|
|
|
(rt->rt6i_nsiblings == 0 ||
|
2017-06-07 18:26:23 +00:00
|
|
|
(dev && netdev_unregistering(dev)) ||
|
2017-01-18 15:40:36 +00:00
|
|
|
!rt->rt6i_idev->cnf.ignore_routes_with_linkdown))
|
2005-04-16 22:20:36 +00:00
|
|
|
return -1;
|
2011-12-26 20:24:36 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-03-04 07:27:06 +00:00
|
|
|
void rt6_ifdown(struct net *net, struct net_device *dev)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2008-03-04 21:48:30 +00:00
|
|
|
struct arg_dev_net adn = {
|
|
|
|
.dev = dev,
|
|
|
|
.net = net,
|
|
|
|
};
|
|
|
|
|
2013-12-27 08:32:38 +00:00
|
|
|
fib6_clean_all(net, fib6_ifdown, &adn);
|
2015-10-12 16:02:08 +00:00
|
|
|
if (dev)
|
|
|
|
rt6_uncached_list_flush_dev(net, dev);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2012-04-15 05:58:06 +00:00
|
|
|
struct rt6_mtu_change_arg {
|
2005-04-16 22:20:36 +00:00
|
|
|
struct net_device *dev;
|
2012-04-15 05:58:06 +00:00
|
|
|
unsigned int mtu;
|
2005-04-16 22:20:36 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
static int rt6_mtu_change_route(struct rt6_info *rt, void *p_arg)
|
|
|
|
{
|
|
|
|
struct rt6_mtu_change_arg *arg = (struct rt6_mtu_change_arg *) p_arg;
|
|
|
|
struct inet6_dev *idev;
|
|
|
|
|
|
|
|
/* In IPv6 pmtu discovery is not optional,
|
|
|
|
so that RTAX_MTU lock cannot disable it.
|
|
|
|
We still use this lock to block changes
|
|
|
|
caused by addrconf/ndisc.
|
|
|
|
*/
|
|
|
|
|
|
|
|
idev = __in6_dev_get(arg->dev);
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!idev)
|
2005-04-16 22:20:36 +00:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* For administrative MTU increase, there is no way to discover
|
|
|
|
IPv6 PMTU increase, so PMTU increase should be updated here.
|
|
|
|
Since RFC 1981 doesn't include administrative MTU increase
|
|
|
|
update PMTU increase is a MUST. (i.e. jumbo frame)
|
|
|
|
*/
|
|
|
|
/*
|
|
|
|
If new MTU is less than route PMTU, this new MTU will be the
|
|
|
|
lowest MTU in the path, update the route PMTU to reflect PMTU
|
|
|
|
decreases; if new MTU is greater than route PMTU, and the
|
|
|
|
old MTU is the lowest MTU in the path, update the route PMTU
|
|
|
|
to reflect the increase. In this case if the other nodes' MTU
|
|
|
|
also have the lowest MTU, TOO BIG MESSAGE will be lead to
|
2017-01-07 22:53:00 +00:00
|
|
|
PMTU discovery.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2011-12-29 01:19:20 +00:00
|
|
|
if (rt->dst.dev == arg->dev &&
|
2016-11-04 21:51:54 +00:00
|
|
|
dst_metric_raw(&rt->dst, RTAX_MTU) &&
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
!dst_metric_locked(&rt->dst, RTAX_MTU)) {
|
2017-10-06 19:05:59 +00:00
|
|
|
spin_lock_bh(&rt6_exception_lock);
|
2017-10-06 19:06:03 +00:00
|
|
|
if (dst_mtu(&rt->dst) >= arg->mtu ||
|
|
|
|
(dst_mtu(&rt->dst) < arg->mtu &&
|
|
|
|
dst_mtu(&rt->dst) == idev->cnf.mtu6)) {
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
dst_metric_set(&rt->dst, RTAX_MTU, arg->mtu);
|
|
|
|
}
|
2017-10-06 19:05:59 +00:00
|
|
|
rt6_exceptions_update_pmtu(rt, arg->mtu);
|
|
|
|
spin_unlock_bh(&rt6_exception_lock);
|
2007-07-26 07:09:55 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2012-04-15 05:58:06 +00:00
|
|
|
void rt6_mtu_change(struct net_device *dev, unsigned int mtu)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2006-08-05 06:20:06 +00:00
|
|
|
struct rt6_mtu_change_arg arg = {
|
|
|
|
.dev = dev,
|
|
|
|
.mtu = mtu,
|
|
|
|
};
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2013-12-27 08:32:38 +00:00
|
|
|
fib6_clean_all(dev_net(dev), rt6_mtu_change_route, &arg);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-06-05 19:38:30 +00:00
|
|
|
static const struct nla_policy rtm_ipv6_policy[RTA_MAX+1] = {
|
2006-08-27 03:13:18 +00:00
|
|
|
[RTA_GATEWAY] = { .len = sizeof(struct in6_addr) },
|
2006-08-22 07:01:08 +00:00
|
|
|
[RTA_OIF] = { .type = NLA_U32 },
|
2006-08-22 07:01:47 +00:00
|
|
|
[RTA_IIF] = { .type = NLA_U32 },
|
2006-08-22 07:01:08 +00:00
|
|
|
[RTA_PRIORITY] = { .type = NLA_U32 },
|
|
|
|
[RTA_METRICS] = { .type = NLA_NESTED },
|
2012-10-22 03:42:09 +00:00
|
|
|
[RTA_MULTIPATH] = { .len = sizeof(struct rtnexthop) },
|
2015-03-11 14:39:21 +00:00
|
|
|
[RTA_PREF] = { .type = NLA_U8 },
|
2015-07-21 08:43:48 +00:00
|
|
|
[RTA_ENCAP_TYPE] = { .type = NLA_U16 },
|
|
|
|
[RTA_ENCAP] = { .type = NLA_NESTED },
|
2015-12-16 09:50:11 +00:00
|
|
|
[RTA_EXPIRES] = { .type = NLA_U32 },
|
2016-11-03 17:23:42 +00:00
|
|
|
[RTA_UID] = { .type = NLA_U32 },
|
2017-02-27 12:59:39 +00:00
|
|
|
[RTA_MARK] = { .type = NLA_U32 },
|
2006-08-22 07:01:08 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
static int rtm_to_fib6_config(struct sk_buff *skb, struct nlmsghdr *nlh,
|
2017-05-21 16:12:04 +00:00
|
|
|
struct fib6_config *cfg,
|
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2006-08-22 07:01:08 +00:00
|
|
|
struct rtmsg *rtm;
|
|
|
|
struct nlattr *tb[RTA_MAX+1];
|
2015-03-11 14:39:21 +00:00
|
|
|
unsigned int pref;
|
2006-08-22 07:01:08 +00:00
|
|
|
int err;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-04-12 12:34:07 +00:00
|
|
|
err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_ipv6_policy,
|
|
|
|
NULL);
|
2006-08-22 07:01:08 +00:00
|
|
|
if (err < 0)
|
|
|
|
goto errout;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-08-22 07:01:08 +00:00
|
|
|
err = -EINVAL;
|
|
|
|
rtm = nlmsg_data(nlh);
|
|
|
|
memset(cfg, 0, sizeof(*cfg));
|
|
|
|
|
|
|
|
cfg->fc_table = rtm->rtm_table;
|
|
|
|
cfg->fc_dst_len = rtm->rtm_dst_len;
|
|
|
|
cfg->fc_src_len = rtm->rtm_src_len;
|
|
|
|
cfg->fc_flags = RTF_UP;
|
|
|
|
cfg->fc_protocol = rtm->rtm_protocol;
|
2012-09-05 02:12:42 +00:00
|
|
|
cfg->fc_type = rtm->rtm_type;
|
2006-08-22 07:01:08 +00:00
|
|
|
|
2012-09-05 02:12:42 +00:00
|
|
|
if (rtm->rtm_type == RTN_UNREACHABLE ||
|
|
|
|
rtm->rtm_type == RTN_BLACKHOLE ||
|
2012-09-06 05:53:35 +00:00
|
|
|
rtm->rtm_type == RTN_PROHIBIT ||
|
|
|
|
rtm->rtm_type == RTN_THROW)
|
2006-08-22 07:01:08 +00:00
|
|
|
cfg->fc_flags |= RTF_REJECT;
|
|
|
|
|
2010-09-27 00:07:02 +00:00
|
|
|
if (rtm->rtm_type == RTN_LOCAL)
|
|
|
|
cfg->fc_flags |= RTF_LOCAL;
|
|
|
|
|
2015-04-28 20:03:03 +00:00
|
|
|
if (rtm->rtm_flags & RTM_F_CLONED)
|
|
|
|
cfg->fc_flags |= RTF_CACHE;
|
|
|
|
|
2012-09-07 20:12:54 +00:00
|
|
|
cfg->fc_nlinfo.portid = NETLINK_CB(skb).portid;
|
2006-08-22 07:01:08 +00:00
|
|
|
cfg->fc_nlinfo.nlh = nlh;
|
2008-03-25 17:26:21 +00:00
|
|
|
cfg->fc_nlinfo.nl_net = sock_net(skb->sk);
|
2006-08-22 07:01:08 +00:00
|
|
|
|
|
|
|
if (tb[RTA_GATEWAY]) {
|
2015-03-29 14:59:26 +00:00
|
|
|
cfg->fc_gateway = nla_get_in6_addr(tb[RTA_GATEWAY]);
|
2006-08-22 07:01:08 +00:00
|
|
|
cfg->fc_flags |= RTF_GATEWAY;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2006-08-22 07:01:08 +00:00
|
|
|
|
|
|
|
if (tb[RTA_DST]) {
|
|
|
|
int plen = (rtm->rtm_dst_len + 7) >> 3;
|
|
|
|
|
|
|
|
if (nla_len(tb[RTA_DST]) < plen)
|
|
|
|
goto errout;
|
|
|
|
|
|
|
|
nla_memcpy(&cfg->fc_dst, tb[RTA_DST], plen);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2006-08-22 07:01:08 +00:00
|
|
|
|
|
|
|
if (tb[RTA_SRC]) {
|
|
|
|
int plen = (rtm->rtm_src_len + 7) >> 3;
|
|
|
|
|
|
|
|
if (nla_len(tb[RTA_SRC]) < plen)
|
|
|
|
goto errout;
|
|
|
|
|
|
|
|
nla_memcpy(&cfg->fc_src, tb[RTA_SRC], plen);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2006-08-22 07:01:08 +00:00
|
|
|
|
2011-04-13 21:10:57 +00:00
|
|
|
if (tb[RTA_PREFSRC])
|
2015-03-29 14:59:26 +00:00
|
|
|
cfg->fc_prefsrc = nla_get_in6_addr(tb[RTA_PREFSRC]);
|
2011-04-13 21:10:57 +00:00
|
|
|
|
2006-08-22 07:01:08 +00:00
|
|
|
if (tb[RTA_OIF])
|
|
|
|
cfg->fc_ifindex = nla_get_u32(tb[RTA_OIF]);
|
|
|
|
|
|
|
|
if (tb[RTA_PRIORITY])
|
|
|
|
cfg->fc_metric = nla_get_u32(tb[RTA_PRIORITY]);
|
|
|
|
|
|
|
|
if (tb[RTA_METRICS]) {
|
|
|
|
cfg->fc_mx = nla_data(tb[RTA_METRICS]);
|
|
|
|
cfg->fc_mx_len = nla_len(tb[RTA_METRICS]);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2006-08-22 07:01:08 +00:00
|
|
|
|
|
|
|
if (tb[RTA_TABLE])
|
|
|
|
cfg->fc_table = nla_get_u32(tb[RTA_TABLE]);
|
|
|
|
|
2012-10-22 03:42:09 +00:00
|
|
|
if (tb[RTA_MULTIPATH]) {
|
|
|
|
cfg->fc_mp = nla_data(tb[RTA_MULTIPATH]);
|
|
|
|
cfg->fc_mp_len = nla_len(tb[RTA_MULTIPATH]);
|
2017-01-17 22:57:36 +00:00
|
|
|
|
|
|
|
err = lwtunnel_valid_encap_type_attr(cfg->fc_mp,
|
2017-05-27 22:19:27 +00:00
|
|
|
cfg->fc_mp_len, extack);
|
2017-01-17 22:57:36 +00:00
|
|
|
if (err < 0)
|
|
|
|
goto errout;
|
2012-10-22 03:42:09 +00:00
|
|
|
}
|
|
|
|
|
2015-03-11 14:39:21 +00:00
|
|
|
if (tb[RTA_PREF]) {
|
|
|
|
pref = nla_get_u8(tb[RTA_PREF]);
|
|
|
|
if (pref != ICMPV6_ROUTER_PREF_LOW &&
|
|
|
|
pref != ICMPV6_ROUTER_PREF_HIGH)
|
|
|
|
pref = ICMPV6_ROUTER_PREF_MEDIUM;
|
|
|
|
cfg->fc_flags |= RTF_PREF(pref);
|
|
|
|
}
|
|
|
|
|
2015-07-21 08:43:48 +00:00
|
|
|
if (tb[RTA_ENCAP])
|
|
|
|
cfg->fc_encap = tb[RTA_ENCAP];
|
|
|
|
|
2017-01-17 22:57:36 +00:00
|
|
|
if (tb[RTA_ENCAP_TYPE]) {
|
2015-07-21 08:43:48 +00:00
|
|
|
cfg->fc_encap_type = nla_get_u16(tb[RTA_ENCAP_TYPE]);
|
|
|
|
|
2017-05-27 22:19:27 +00:00
|
|
|
err = lwtunnel_valid_encap_type(cfg->fc_encap_type, extack);
|
2017-01-17 22:57:36 +00:00
|
|
|
if (err < 0)
|
|
|
|
goto errout;
|
|
|
|
}
|
|
|
|
|
2015-12-16 09:50:11 +00:00
|
|
|
if (tb[RTA_EXPIRES]) {
|
|
|
|
unsigned long timeout = addrconf_timeout_fixup(nla_get_u32(tb[RTA_EXPIRES]), HZ);
|
|
|
|
|
|
|
|
if (addrconf_finite_timeout(timeout)) {
|
|
|
|
cfg->fc_expires = jiffies_to_clock_t(timeout * HZ);
|
|
|
|
cfg->fc_flags |= RTF_EXPIRES;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2006-08-22 07:01:08 +00:00
|
|
|
err = 0;
|
|
|
|
errout:
|
|
|
|
return err;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2015-09-08 17:53:04 +00:00
|
|
|
struct rt6_nh {
|
|
|
|
struct rt6_info *rt6_info;
|
|
|
|
struct fib6_config r_cfg;
|
|
|
|
struct mx6_config mxc;
|
|
|
|
struct list_head next;
|
|
|
|
};
|
|
|
|
|
|
|
|
static void ip6_print_replace_route_err(struct list_head *rt6_nh_list)
|
|
|
|
{
|
|
|
|
struct rt6_nh *nh;
|
|
|
|
|
|
|
|
list_for_each_entry(nh, rt6_nh_list, next) {
|
2017-02-02 20:37:12 +00:00
|
|
|
pr_warn("IPV6: multipath route replace failed (check consistency of installed routes): %pI6c nexthop %pI6c ifi %d\n",
|
2015-09-08 17:53:04 +00:00
|
|
|
&nh->r_cfg.fc_dst, &nh->r_cfg.fc_gateway,
|
|
|
|
nh->r_cfg.fc_ifindex);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static int ip6_route_info_append(struct list_head *rt6_nh_list,
|
|
|
|
struct rt6_info *rt, struct fib6_config *r_cfg)
|
|
|
|
{
|
|
|
|
struct rt6_nh *nh;
|
|
|
|
int err = -EEXIST;
|
|
|
|
|
|
|
|
list_for_each_entry(nh, rt6_nh_list, next) {
|
|
|
|
/* check if rt6_info already exists */
|
2017-07-05 20:41:46 +00:00
|
|
|
if (rt6_duplicate_nexthop(nh->rt6_info, rt))
|
2015-09-08 17:53:04 +00:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
nh = kzalloc(sizeof(*nh), GFP_KERNEL);
|
|
|
|
if (!nh)
|
|
|
|
return -ENOMEM;
|
|
|
|
nh->rt6_info = rt;
|
|
|
|
err = ip6_convert_metrics(&nh->mxc, r_cfg);
|
|
|
|
if (err) {
|
|
|
|
kfree(nh);
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
memcpy(&nh->r_cfg, r_cfg, sizeof(*r_cfg));
|
|
|
|
list_add_tail(&nh->next, rt6_nh_list);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2017-02-02 20:37:10 +00:00
|
|
|
static void ip6_route_mpath_notify(struct rt6_info *rt,
|
|
|
|
struct rt6_info *rt_last,
|
|
|
|
struct nl_info *info,
|
|
|
|
__u16 nlflags)
|
|
|
|
{
|
|
|
|
/* if this is an APPEND route, then rt points to the first route
|
|
|
|
* inserted and rt_last points to last route inserted. Userspace
|
|
|
|
* wants a consistent dump of the route which starts at the first
|
|
|
|
* nexthop. Since sibling routes are always added at the end of
|
|
|
|
* the list, find the first sibling of the last route appended
|
|
|
|
*/
|
|
|
|
if ((nlflags & NLM_F_APPEND) && rt_last && rt_last->rt6i_nsiblings) {
|
|
|
|
rt = list_first_entry(&rt_last->rt6i_siblings,
|
|
|
|
struct rt6_info,
|
|
|
|
rt6i_siblings);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (rt)
|
|
|
|
inet6_rt_notify(RTM_NEWROUTE, rt, info, nlflags);
|
|
|
|
}
|
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
static int ip6_route_multipath_add(struct fib6_config *cfg,
|
|
|
|
struct netlink_ext_ack *extack)
|
2012-10-22 03:42:09 +00:00
|
|
|
{
|
2017-02-02 20:37:10 +00:00
|
|
|
struct rt6_info *rt_notif = NULL, *rt_last = NULL;
|
|
|
|
struct nl_info *info = &cfg->fc_nlinfo;
|
2012-10-22 03:42:09 +00:00
|
|
|
struct fib6_config r_cfg;
|
|
|
|
struct rtnexthop *rtnh;
|
2015-09-08 17:53:04 +00:00
|
|
|
struct rt6_info *rt;
|
|
|
|
struct rt6_nh *err_nh;
|
|
|
|
struct rt6_nh *nh, *nh_safe;
|
2017-02-02 20:37:10 +00:00
|
|
|
__u16 nlflags;
|
2012-10-22 03:42:09 +00:00
|
|
|
int remaining;
|
|
|
|
int attrlen;
|
2015-09-08 17:53:04 +00:00
|
|
|
int err = 1;
|
|
|
|
int nhn = 0;
|
|
|
|
int replace = (cfg->fc_nlinfo.nlh &&
|
|
|
|
(cfg->fc_nlinfo.nlh->nlmsg_flags & NLM_F_REPLACE));
|
|
|
|
LIST_HEAD(rt6_nh_list);
|
2012-10-22 03:42:09 +00:00
|
|
|
|
2017-02-02 20:37:10 +00:00
|
|
|
nlflags = replace ? NLM_F_REPLACE : NLM_F_CREATE;
|
|
|
|
if (info->nlh && info->nlh->nlmsg_flags & NLM_F_APPEND)
|
|
|
|
nlflags |= NLM_F_APPEND;
|
|
|
|
|
2015-05-18 18:53:55 +00:00
|
|
|
remaining = cfg->fc_mp_len;
|
2012-10-22 03:42:09 +00:00
|
|
|
rtnh = (struct rtnexthop *)cfg->fc_mp;
|
|
|
|
|
2015-09-08 17:53:04 +00:00
|
|
|
/* Parse a Multipath Entry and build a list (rt6_nh_list) of
|
|
|
|
* rt6_info structs per nexthop
|
|
|
|
*/
|
2012-10-22 03:42:09 +00:00
|
|
|
while (rtnh_ok(rtnh, remaining)) {
|
|
|
|
memcpy(&r_cfg, cfg, sizeof(*cfg));
|
|
|
|
if (rtnh->rtnh_ifindex)
|
|
|
|
r_cfg.fc_ifindex = rtnh->rtnh_ifindex;
|
|
|
|
|
|
|
|
attrlen = rtnh_attrlen(rtnh);
|
|
|
|
if (attrlen > 0) {
|
|
|
|
struct nlattr *nla, *attrs = rtnh_attrs(rtnh);
|
|
|
|
|
|
|
|
nla = nla_find(attrs, attrlen, RTA_GATEWAY);
|
|
|
|
if (nla) {
|
2015-03-29 14:59:26 +00:00
|
|
|
r_cfg.fc_gateway = nla_get_in6_addr(nla);
|
2012-10-22 03:42:09 +00:00
|
|
|
r_cfg.fc_flags |= RTF_GATEWAY;
|
|
|
|
}
|
2015-07-21 08:43:48 +00:00
|
|
|
r_cfg.fc_encap = nla_find(attrs, attrlen, RTA_ENCAP);
|
|
|
|
nla = nla_find(attrs, attrlen, RTA_ENCAP_TYPE);
|
|
|
|
if (nla)
|
|
|
|
r_cfg.fc_encap_type = nla_get_u16(nla);
|
2012-10-22 03:42:09 +00:00
|
|
|
}
|
2015-09-08 17:53:04 +00:00
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
rt = ip6_route_info_create(&r_cfg, extack);
|
2015-10-10 15:26:36 +00:00
|
|
|
if (IS_ERR(rt)) {
|
|
|
|
err = PTR_ERR(rt);
|
|
|
|
rt = NULL;
|
2015-09-08 17:53:04 +00:00
|
|
|
goto cleanup;
|
2015-10-10 15:26:36 +00:00
|
|
|
}
|
2015-09-08 17:53:04 +00:00
|
|
|
|
|
|
|
err = ip6_route_info_append(&rt6_nh_list, rt, &r_cfg);
|
2012-10-22 03:42:09 +00:00
|
|
|
if (err) {
|
2017-06-17 17:42:36 +00:00
|
|
|
dst_release_immediate(&rt->dst);
|
2015-09-08 17:53:04 +00:00
|
|
|
goto cleanup;
|
|
|
|
}
|
|
|
|
|
|
|
|
rtnh = rtnh_next(rtnh, &remaining);
|
|
|
|
}
|
|
|
|
|
2017-02-02 20:37:10 +00:00
|
|
|
/* for add and replace send one notification with all nexthops.
|
|
|
|
* Skip the notification in fib6_add_rt2node and send one with
|
|
|
|
* the full route when done
|
|
|
|
*/
|
|
|
|
info->skip_notify = 1;
|
|
|
|
|
2015-09-08 17:53:04 +00:00
|
|
|
err_nh = NULL;
|
|
|
|
list_for_each_entry(nh, &rt6_nh_list, next) {
|
2017-02-02 20:37:10 +00:00
|
|
|
rt_last = nh->rt6_info;
|
2017-05-21 16:12:04 +00:00
|
|
|
err = __ip6_ins_rt(nh->rt6_info, info, &nh->mxc, extack);
|
2017-02-02 20:37:10 +00:00
|
|
|
/* save reference to first route for notification */
|
|
|
|
if (!rt_notif && !err)
|
|
|
|
rt_notif = nh->rt6_info;
|
|
|
|
|
2015-09-08 17:53:04 +00:00
|
|
|
/* nh->rt6_info is used or freed at this point, reset to NULL*/
|
|
|
|
nh->rt6_info = NULL;
|
|
|
|
if (err) {
|
|
|
|
if (replace && nhn)
|
|
|
|
ip6_print_replace_route_err(&rt6_nh_list);
|
|
|
|
err_nh = nh;
|
|
|
|
goto add_errout;
|
2012-10-22 03:42:09 +00:00
|
|
|
}
|
2015-09-08 17:53:04 +00:00
|
|
|
|
2012-11-01 22:58:22 +00:00
|
|
|
/* Because each route is added like a single route we remove
|
2015-05-18 18:54:00 +00:00
|
|
|
* these flags after the first nexthop: if there is a collision,
|
|
|
|
* we have already failed to add the first nexthop:
|
|
|
|
* fib6_add_rt2node() has rejected it; when replacing, old
|
|
|
|
* nexthops have been replaced by first new, the rest should
|
|
|
|
* be added to it.
|
2012-11-01 22:58:22 +00:00
|
|
|
*/
|
2015-05-18 18:54:00 +00:00
|
|
|
cfg->fc_nlinfo.nlh->nlmsg_flags &= ~(NLM_F_EXCL |
|
|
|
|
NLM_F_REPLACE);
|
2015-09-08 17:53:04 +00:00
|
|
|
nhn++;
|
|
|
|
}
|
|
|
|
|
2017-02-02 20:37:10 +00:00
|
|
|
/* success ... tell user about new route */
|
|
|
|
ip6_route_mpath_notify(rt_notif, rt_last, info, nlflags);
|
2015-09-08 17:53:04 +00:00
|
|
|
goto cleanup;
|
|
|
|
|
|
|
|
add_errout:
|
2017-02-02 20:37:10 +00:00
|
|
|
/* send notification for routes that were added so that
|
|
|
|
* the delete notifications sent by ip6_route_del are
|
|
|
|
* coherent
|
|
|
|
*/
|
|
|
|
if (rt_notif)
|
|
|
|
ip6_route_mpath_notify(rt_notif, rt_last, info, nlflags);
|
|
|
|
|
2015-09-08 17:53:04 +00:00
|
|
|
/* Delete routes that were already added */
|
|
|
|
list_for_each_entry(nh, &rt6_nh_list, next) {
|
|
|
|
if (err_nh == nh)
|
|
|
|
break;
|
2017-05-21 16:12:04 +00:00
|
|
|
ip6_route_del(&nh->r_cfg, extack);
|
2015-09-08 17:53:04 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
cleanup:
|
|
|
|
list_for_each_entry_safe(nh, nh_safe, &rt6_nh_list, next) {
|
2017-06-17 17:42:36 +00:00
|
|
|
if (nh->rt6_info)
|
|
|
|
dst_release_immediate(&nh->rt6_info->dst);
|
2015-09-09 22:57:12 +00:00
|
|
|
kfree(nh->mxc.mx);
|
2015-09-08 17:53:04 +00:00
|
|
|
list_del(&nh->next);
|
|
|
|
kfree(nh);
|
|
|
|
}
|
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
static int ip6_route_multipath_del(struct fib6_config *cfg,
|
|
|
|
struct netlink_ext_ack *extack)
|
2015-09-08 17:53:04 +00:00
|
|
|
{
|
|
|
|
struct fib6_config r_cfg;
|
|
|
|
struct rtnexthop *rtnh;
|
|
|
|
int remaining;
|
|
|
|
int attrlen;
|
|
|
|
int err = 1, last_err = 0;
|
|
|
|
|
|
|
|
remaining = cfg->fc_mp_len;
|
|
|
|
rtnh = (struct rtnexthop *)cfg->fc_mp;
|
|
|
|
|
|
|
|
/* Parse a Multipath Entry */
|
|
|
|
while (rtnh_ok(rtnh, remaining)) {
|
|
|
|
memcpy(&r_cfg, cfg, sizeof(*cfg));
|
|
|
|
if (rtnh->rtnh_ifindex)
|
|
|
|
r_cfg.fc_ifindex = rtnh->rtnh_ifindex;
|
|
|
|
|
|
|
|
attrlen = rtnh_attrlen(rtnh);
|
|
|
|
if (attrlen > 0) {
|
|
|
|
struct nlattr *nla, *attrs = rtnh_attrs(rtnh);
|
|
|
|
|
|
|
|
nla = nla_find(attrs, attrlen, RTA_GATEWAY);
|
|
|
|
if (nla) {
|
|
|
|
nla_memcpy(&r_cfg.fc_gateway, nla, 16);
|
|
|
|
r_cfg.fc_flags |= RTF_GATEWAY;
|
|
|
|
}
|
|
|
|
}
|
2017-05-21 16:12:04 +00:00
|
|
|
err = ip6_route_del(&r_cfg, extack);
|
2015-09-08 17:53:04 +00:00
|
|
|
if (err)
|
|
|
|
last_err = err;
|
|
|
|
|
2012-10-22 03:42:09 +00:00
|
|
|
rtnh = rtnh_next(rtnh, &remaining);
|
|
|
|
}
|
|
|
|
|
|
|
|
return last_err;
|
|
|
|
}
|
|
|
|
|
2017-04-16 16:48:24 +00:00
|
|
|
static int inet6_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh,
|
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2006-08-22 07:01:08 +00:00
|
|
|
struct fib6_config cfg;
|
|
|
|
int err;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
err = rtm_to_fib6_config(skb, nlh, &cfg, extack);
|
2006-08-22 07:01:08 +00:00
|
|
|
if (err < 0)
|
|
|
|
return err;
|
|
|
|
|
2012-10-22 03:42:09 +00:00
|
|
|
if (cfg.fc_mp)
|
2017-05-21 16:12:04 +00:00
|
|
|
return ip6_route_multipath_del(&cfg, extack);
|
2017-02-02 20:37:08 +00:00
|
|
|
else {
|
|
|
|
cfg.fc_delete_all_nh = 1;
|
2017-05-21 16:12:04 +00:00
|
|
|
return ip6_route_del(&cfg, extack);
|
2017-02-02 20:37:08 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2017-04-16 16:48:24 +00:00
|
|
|
static int inet6_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh,
|
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2006-08-22 07:01:08 +00:00
|
|
|
struct fib6_config cfg;
|
|
|
|
int err;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-05-21 16:12:04 +00:00
|
|
|
err = rtm_to_fib6_config(skb, nlh, &cfg, extack);
|
2006-08-22 07:01:08 +00:00
|
|
|
if (err < 0)
|
|
|
|
return err;
|
|
|
|
|
2012-10-22 03:42:09 +00:00
|
|
|
if (cfg.fc_mp)
|
2017-05-21 16:12:04 +00:00
|
|
|
return ip6_route_multipath_add(&cfg, extack);
|
2012-10-22 03:42:09 +00:00
|
|
|
else
|
2017-05-21 16:12:04 +00:00
|
|
|
return ip6_route_add(&cfg, extack);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2017-02-02 20:37:09 +00:00
|
|
|
static size_t rt6_nlmsg_size(struct rt6_info *rt)
|
2006-11-10 22:10:15 +00:00
|
|
|
{
|
2017-02-02 20:37:09 +00:00
|
|
|
int nexthop_len = 0;
|
|
|
|
|
|
|
|
if (rt->rt6i_nsiblings) {
|
|
|
|
nexthop_len = nla_total_size(0) /* RTA_MULTIPATH */
|
|
|
|
+ NLA_ALIGN(sizeof(struct rtnexthop))
|
|
|
|
+ nla_total_size(16) /* RTA_GATEWAY */
|
|
|
|
+ lwtunnel_get_encap_size(rt->dst.lwtstate);
|
|
|
|
|
|
|
|
nexthop_len *= rt->rt6i_nsiblings;
|
|
|
|
}
|
|
|
|
|
2006-11-10 22:10:15 +00:00
|
|
|
return NLMSG_ALIGN(sizeof(struct rtmsg))
|
|
|
|
+ nla_total_size(16) /* RTA_SRC */
|
|
|
|
+ nla_total_size(16) /* RTA_DST */
|
|
|
|
+ nla_total_size(16) /* RTA_GATEWAY */
|
|
|
|
+ nla_total_size(16) /* RTA_PREFSRC */
|
|
|
|
+ nla_total_size(4) /* RTA_TABLE */
|
|
|
|
+ nla_total_size(4) /* RTA_IIF */
|
|
|
|
+ nla_total_size(4) /* RTA_OIF */
|
|
|
|
+ nla_total_size(4) /* RTA_PRIORITY */
|
2007-01-24 06:09:41 +00:00
|
|
|
+ RTAX_MAX * nla_total_size(4) /* RTA_METRICS */
|
2015-01-05 22:57:47 +00:00
|
|
|
+ nla_total_size(sizeof(struct rta_cacheinfo))
|
2015-03-11 14:39:21 +00:00
|
|
|
+ nla_total_size(TCP_CA_NAME_MAX) /* RTAX_CC_ALGO */
|
2015-07-21 08:43:48 +00:00
|
|
|
+ nla_total_size(1) /* RTA_PREF */
|
2017-02-02 20:37:09 +00:00
|
|
|
+ lwtunnel_get_encap_size(rt->dst.lwtstate)
|
|
|
|
+ nexthop_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int rt6_nexthop_info(struct sk_buff *skb, struct rt6_info *rt,
|
2017-03-06 23:57:31 +00:00
|
|
|
unsigned int *flags, bool skip_oif)
|
2017-02-02 20:37:09 +00:00
|
|
|
{
|
|
|
|
if (!netif_running(rt->dst.dev) || !netif_carrier_ok(rt->dst.dev)) {
|
|
|
|
*flags |= RTNH_F_LINKDOWN;
|
|
|
|
if (rt->rt6i_idev->cnf.ignore_routes_with_linkdown)
|
|
|
|
*flags |= RTNH_F_DEAD;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (rt->rt6i_flags & RTF_GATEWAY) {
|
|
|
|
if (nla_put_in6_addr(skb, RTA_GATEWAY, &rt->rt6i_gateway) < 0)
|
|
|
|
goto nla_put_failure;
|
|
|
|
}
|
|
|
|
|
2017-08-15 07:09:49 +00:00
|
|
|
if (rt->rt6i_nh_flags & RTNH_F_OFFLOAD)
|
2017-08-03 11:28:20 +00:00
|
|
|
*flags |= RTNH_F_OFFLOAD;
|
|
|
|
|
2017-03-06 23:57:31 +00:00
|
|
|
/* not needed for multipath encoding b/c it has a rtnexthop struct */
|
|
|
|
if (!skip_oif && rt->dst.dev &&
|
2017-02-02 20:37:09 +00:00
|
|
|
nla_put_u32(skb, RTA_OIF, rt->dst.dev->ifindex))
|
|
|
|
goto nla_put_failure;
|
|
|
|
|
|
|
|
if (rt->dst.lwtstate &&
|
|
|
|
lwtunnel_fill_encap(skb, rt->dst.lwtstate) < 0)
|
|
|
|
goto nla_put_failure;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
nla_put_failure:
|
|
|
|
return -EMSGSIZE;
|
|
|
|
}
|
|
|
|
|
2017-03-06 23:57:31 +00:00
|
|
|
/* add multipath next hop */
|
2017-02-02 20:37:09 +00:00
|
|
|
static int rt6_add_nexthop(struct sk_buff *skb, struct rt6_info *rt)
|
|
|
|
{
|
|
|
|
struct rtnexthop *rtnh;
|
|
|
|
unsigned int flags = 0;
|
|
|
|
|
|
|
|
rtnh = nla_reserve_nohdr(skb, sizeof(*rtnh));
|
|
|
|
if (!rtnh)
|
|
|
|
goto nla_put_failure;
|
|
|
|
|
|
|
|
rtnh->rtnh_hops = 0;
|
|
|
|
rtnh->rtnh_ifindex = rt->dst.dev ? rt->dst.dev->ifindex : 0;
|
|
|
|
|
2017-03-06 23:57:31 +00:00
|
|
|
if (rt6_nexthop_info(skb, rt, &flags, true) < 0)
|
2017-02-02 20:37:09 +00:00
|
|
|
goto nla_put_failure;
|
|
|
|
|
|
|
|
rtnh->rtnh_flags = flags;
|
|
|
|
|
|
|
|
/* length of rtnetlink header + attributes */
|
|
|
|
rtnh->rtnh_len = nlmsg_get_pos(skb) - (void *)rtnh;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
nla_put_failure:
|
|
|
|
return -EMSGSIZE;
|
2006-11-10 22:10:15 +00:00
|
|
|
}
|
|
|
|
|
2008-08-14 22:33:21 +00:00
|
|
|
static int rt6_fill_node(struct net *net,
|
|
|
|
struct sk_buff *skb, struct rt6_info *rt,
|
2005-06-21 20:51:04 +00:00
|
|
|
struct in6_addr *dst, struct in6_addr *src,
|
2012-09-07 20:12:54 +00:00
|
|
|
int iif, int type, u32 portid, u32 seq,
|
2017-01-17 23:51:08 +00:00
|
|
|
unsigned int flags)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
u32 metrics[RTAX_MAX];
|
2005-04-16 22:20:36 +00:00
|
|
|
struct rtmsg *rtm;
|
2006-08-22 07:01:27 +00:00
|
|
|
struct nlmsghdr *nlh;
|
2006-11-27 17:27:07 +00:00
|
|
|
long expires;
|
2006-08-11 06:09:48 +00:00
|
|
|
u32 table;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2012-09-07 20:12:54 +00:00
|
|
|
nlh = nlmsg_put(skb, portid, seq, type, sizeof(*rtm), flags);
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!nlh)
|
2007-02-01 07:16:40 +00:00
|
|
|
return -EMSGSIZE;
|
2006-08-22 07:01:27 +00:00
|
|
|
|
|
|
|
rtm = nlmsg_data(nlh);
|
2005-04-16 22:20:36 +00:00
|
|
|
rtm->rtm_family = AF_INET6;
|
|
|
|
rtm->rtm_dst_len = rt->rt6i_dst.plen;
|
|
|
|
rtm->rtm_src_len = rt->rt6i_src.plen;
|
|
|
|
rtm->rtm_tos = 0;
|
2006-08-05 06:20:06 +00:00
|
|
|
if (rt->rt6i_table)
|
2006-08-11 06:09:48 +00:00
|
|
|
table = rt->rt6i_table->tb6_id;
|
2006-08-05 06:20:06 +00:00
|
|
|
else
|
2006-08-11 06:09:48 +00:00
|
|
|
table = RT6_TABLE_UNSPEC;
|
|
|
|
rtm->rtm_table = table;
|
2012-04-02 00:27:33 +00:00
|
|
|
if (nla_put_u32(skb, RTA_TABLE, table))
|
|
|
|
goto nla_put_failure;
|
2012-09-05 02:12:42 +00:00
|
|
|
if (rt->rt6i_flags & RTF_REJECT) {
|
|
|
|
switch (rt->dst.error) {
|
|
|
|
case -EINVAL:
|
|
|
|
rtm->rtm_type = RTN_BLACKHOLE;
|
|
|
|
break;
|
|
|
|
case -EACCES:
|
|
|
|
rtm->rtm_type = RTN_PROHIBIT;
|
|
|
|
break;
|
2012-09-06 05:53:35 +00:00
|
|
|
case -EAGAIN:
|
|
|
|
rtm->rtm_type = RTN_THROW;
|
|
|
|
break;
|
2012-09-05 02:12:42 +00:00
|
|
|
default:
|
|
|
|
rtm->rtm_type = RTN_UNREACHABLE;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
2011-12-03 23:02:47 +00:00
|
|
|
else if (rt->rt6i_flags & RTF_LOCAL)
|
2010-09-27 00:07:02 +00:00
|
|
|
rtm->rtm_type = RTN_LOCAL;
|
2017-03-16 01:14:33 +00:00
|
|
|
else if (rt->rt6i_flags & RTF_ANYCAST)
|
|
|
|
rtm->rtm_type = RTN_ANYCAST;
|
2011-12-29 01:19:20 +00:00
|
|
|
else if (rt->dst.dev && (rt->dst.dev->flags & IFF_LOOPBACK))
|
2005-04-16 22:20:36 +00:00
|
|
|
rtm->rtm_type = RTN_LOCAL;
|
|
|
|
else
|
|
|
|
rtm->rtm_type = RTN_UNICAST;
|
|
|
|
rtm->rtm_flags = 0;
|
|
|
|
rtm->rtm_scope = RT_SCOPE_UNIVERSE;
|
|
|
|
rtm->rtm_protocol = rt->rt6i_protocol;
|
|
|
|
|
2011-12-03 23:02:47 +00:00
|
|
|
if (rt->rt6i_flags & RTF_CACHE)
|
2005-04-16 22:20:36 +00:00
|
|
|
rtm->rtm_flags |= RTM_F_CLONED;
|
|
|
|
|
|
|
|
if (dst) {
|
2015-03-29 14:59:25 +00:00
|
|
|
if (nla_put_in6_addr(skb, RTA_DST, dst))
|
2012-04-02 00:27:33 +00:00
|
|
|
goto nla_put_failure;
|
2007-02-09 14:24:49 +00:00
|
|
|
rtm->rtm_dst_len = 128;
|
2005-04-16 22:20:36 +00:00
|
|
|
} else if (rtm->rtm_dst_len)
|
2015-03-29 14:59:25 +00:00
|
|
|
if (nla_put_in6_addr(skb, RTA_DST, &rt->rt6i_dst.addr))
|
2012-04-02 00:27:33 +00:00
|
|
|
goto nla_put_failure;
|
2005-04-16 22:20:36 +00:00
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
|
|
|
if (src) {
|
2015-03-29 14:59:25 +00:00
|
|
|
if (nla_put_in6_addr(skb, RTA_SRC, src))
|
2012-04-02 00:27:33 +00:00
|
|
|
goto nla_put_failure;
|
2007-02-09 14:24:49 +00:00
|
|
|
rtm->rtm_src_len = 128;
|
2012-04-02 00:27:33 +00:00
|
|
|
} else if (rtm->rtm_src_len &&
|
2015-03-29 14:59:25 +00:00
|
|
|
nla_put_in6_addr(skb, RTA_SRC, &rt->rt6i_src.addr))
|
2012-04-02 00:27:33 +00:00
|
|
|
goto nla_put_failure;
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif
|
2008-04-03 00:22:53 +00:00
|
|
|
if (iif) {
|
|
|
|
#ifdef CONFIG_IPV6_MROUTE
|
|
|
|
if (ipv6_addr_is_multicast(&rt->rt6i_dst.addr)) {
|
2017-01-17 23:51:07 +00:00
|
|
|
int err = ip6mr_get_route(net, skb, rtm, portid);
|
|
|
|
|
|
|
|
if (err == 0)
|
|
|
|
return 0;
|
|
|
|
if (err < 0)
|
|
|
|
goto nla_put_failure;
|
2008-04-03 00:22:53 +00:00
|
|
|
} else
|
|
|
|
#endif
|
2012-04-02 00:27:33 +00:00
|
|
|
if (nla_put_u32(skb, RTA_IIF, iif))
|
|
|
|
goto nla_put_failure;
|
2008-04-03 00:22:53 +00:00
|
|
|
} else if (dst) {
|
2005-04-16 22:20:36 +00:00
|
|
|
struct in6_addr saddr_buf;
|
2012-04-02 00:27:33 +00:00
|
|
|
if (ip6_route_get_saddr(net, rt, dst, 0, &saddr_buf) == 0 &&
|
2015-03-29 14:59:25 +00:00
|
|
|
nla_put_in6_addr(skb, RTA_PREFSRC, &saddr_buf))
|
2012-04-02 00:27:33 +00:00
|
|
|
goto nla_put_failure;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2006-08-22 07:01:27 +00:00
|
|
|
|
2011-04-13 21:10:57 +00:00
|
|
|
if (rt->rt6i_prefsrc.plen) {
|
|
|
|
struct in6_addr saddr_buf;
|
2011-11-21 03:39:03 +00:00
|
|
|
saddr_buf = rt->rt6i_prefsrc.addr;
|
2015-03-29 14:59:25 +00:00
|
|
|
if (nla_put_in6_addr(skb, RTA_PREFSRC, &saddr_buf))
|
2012-04-02 00:27:33 +00:00
|
|
|
goto nla_put_failure;
|
2011-04-13 21:10:57 +00:00
|
|
|
}
|
|
|
|
|
ipv6: Stop rt6_info from using inet_peer's metrics
inet_peer is indexed by the dst address alone. However, the fib6 tree
could have multiple routing entries (rt6_info) for the same dst. For
example,
1. A /128 dst via multiple gateways.
2. A RTF_CACHE route cloned from a /128 route.
In the above cases, all of them will share the same metrics and
step on each other.
This patch will steer away from inet_peer's metrics and use
dst_cow_metrics_generic() for everything.
Change Highlights:
1. Remove rt6_cow_metrics() which currently acquires metrics from
inet_peer for DST_HOST route (i.e. /128 route).
2. Add rt6i_pmtu to take care of the pmtu update to avoid creating a
full size metrics just to override the RTAX_MTU.
3. After (2), the RTF_CACHE route can also share the metrics with its
dst.from route, by:
dst_init_metrics(&cache_rt->dst, dst_metrics_ptr(cache_rt->dst.from), true);
4. Stop creating RTF_CACHE route by cloning another RTF_CACHE route. Instead,
directly clone from rt->dst.
[ Currently, cloning from another RTF_CACHE is only possible during
rt6_do_redirect(). Also, the old clone is removed from the tree
immediately after the new clone is added. ]
In case of cloning from an older redirect RTF_CACHE, it should work as
before.
In case of cloning from an older pmtu RTF_CACHE, this patch will forget
the pmtu and re-learn it (if there is any) from the redirected route.
The _rt6i_peer and DST_METRICS_FORCE_OVERWRITE will be removed
in the next cleanup patch.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Reviewed-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-28 20:03:06 +00:00
|
|
|
memcpy(metrics, dst_metrics_ptr(&rt->dst), sizeof(metrics));
|
|
|
|
if (rt->rt6i_pmtu)
|
|
|
|
metrics[RTAX_MTU - 1] = rt->rt6i_pmtu;
|
|
|
|
if (rtnetlink_put_metrics(skb, metrics) < 0)
|
2006-08-22 07:01:27 +00:00
|
|
|
goto nla_put_failure;
|
|
|
|
|
2012-04-02 00:27:33 +00:00
|
|
|
if (nla_put_u32(skb, RTA_PRIORITY, rt->rt6i_metric))
|
|
|
|
goto nla_put_failure;
|
2012-07-29 16:01:30 +00:00
|
|
|
|
2017-02-02 20:37:09 +00:00
|
|
|
/* For multipath routes, walk the siblings list and add
|
|
|
|
* each as a nexthop within RTA_MULTIPATH.
|
|
|
|
*/
|
|
|
|
if (rt->rt6i_nsiblings) {
|
|
|
|
struct rt6_info *sibling, *next_sibling;
|
|
|
|
struct nlattr *mp;
|
|
|
|
|
|
|
|
mp = nla_nest_start(skb, RTA_MULTIPATH);
|
|
|
|
if (!mp)
|
|
|
|
goto nla_put_failure;
|
|
|
|
|
|
|
|
if (rt6_add_nexthop(skb, rt) < 0)
|
|
|
|
goto nla_put_failure;
|
|
|
|
|
|
|
|
list_for_each_entry_safe(sibling, next_sibling,
|
|
|
|
&rt->rt6i_siblings, rt6i_siblings) {
|
|
|
|
if (rt6_add_nexthop(skb, sibling) < 0)
|
|
|
|
goto nla_put_failure;
|
|
|
|
}
|
|
|
|
|
|
|
|
nla_nest_end(skb, mp);
|
|
|
|
} else {
|
2017-03-06 23:57:31 +00:00
|
|
|
if (rt6_nexthop_info(skb, rt, &rtm->rtm_flags, false) < 0)
|
2017-02-02 20:37:09 +00:00
|
|
|
goto nla_put_failure;
|
|
|
|
}
|
|
|
|
|
2012-07-29 16:01:30 +00:00
|
|
|
expires = (rt->rt6i_flags & RTF_EXPIRES) ? rt->dst.expires - jiffies : 0;
|
2008-05-19 23:55:13 +00:00
|
|
|
|
2012-07-10 12:06:14 +00:00
|
|
|
if (rtnl_put_cacheinfo(skb, &rt->dst, 0, expires, rt->dst.error) < 0)
|
2006-11-27 17:27:07 +00:00
|
|
|
goto nla_put_failure;
|
2006-08-22 07:01:27 +00:00
|
|
|
|
2015-03-11 14:39:21 +00:00
|
|
|
if (nla_put_u8(skb, RTA_PREF, IPV6_EXTRACT_PREF(rt->rt6i_flags)))
|
|
|
|
goto nla_put_failure;
|
|
|
|
|
2015-07-21 08:43:48 +00:00
|
|
|
|
2015-01-16 21:09:00 +00:00
|
|
|
nlmsg_end(skb, nlh);
|
|
|
|
return 0;
|
2006-08-22 07:01:27 +00:00
|
|
|
|
|
|
|
nla_put_failure:
|
2007-02-01 07:16:40 +00:00
|
|
|
nlmsg_cancel(skb, nlh);
|
|
|
|
return -EMSGSIZE;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2006-08-11 06:11:17 +00:00
|
|
|
int rt6_dump_route(struct rt6_info *rt, void *p_arg)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct rt6_rtnl_dump_arg *arg = (struct rt6_rtnl_dump_arg *) p_arg;
|
2017-01-26 21:54:08 +00:00
|
|
|
struct net *net = arg->net;
|
|
|
|
|
|
|
|
if (rt == net->ipv6.ip6_null_entry)
|
|
|
|
return 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-08-22 07:01:27 +00:00
|
|
|
if (nlmsg_len(arg->cb->nlh) >= sizeof(struct rtmsg)) {
|
|
|
|
struct rtmsg *rtm = nlmsg_data(arg->cb->nlh);
|
2017-01-17 23:51:08 +00:00
|
|
|
|
|
|
|
/* user wants prefix routes only */
|
|
|
|
if (rtm->rtm_flags & RTM_F_PREFIX &&
|
|
|
|
!(rt->rt6i_flags & RTF_PREFIX_RT)) {
|
|
|
|
/* success since this is not a prefix route */
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-01-26 21:54:08 +00:00
|
|
|
return rt6_fill_node(net,
|
2008-08-14 22:33:21 +00:00
|
|
|
arg->skb, rt, NULL, NULL, 0, RTM_NEWROUTE,
|
2012-09-07 20:12:54 +00:00
|
|
|
NETLINK_CB(arg->cb->skb).portid, arg->cb->nlh->nlmsg_seq,
|
2017-01-17 23:51:08 +00:00
|
|
|
NLM_F_MULTI);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2017-04-16 16:48:24 +00:00
|
|
|
static int inet6_rtm_getroute(struct sk_buff *in_skb, struct nlmsghdr *nlh,
|
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2008-03-25 17:26:21 +00:00
|
|
|
struct net *net = sock_net(in_skb->sk);
|
2006-08-22 07:01:47 +00:00
|
|
|
struct nlattr *tb[RTA_MAX+1];
|
2017-05-25 17:42:40 +00:00
|
|
|
int err, iif = 0, oif = 0;
|
|
|
|
struct dst_entry *dst;
|
2006-08-22 07:01:47 +00:00
|
|
|
struct rt6_info *rt;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct sk_buff *skb;
|
2006-08-22 07:01:47 +00:00
|
|
|
struct rtmsg *rtm;
|
2011-03-12 21:22:43 +00:00
|
|
|
struct flowi6 fl6;
|
2017-05-25 17:42:40 +00:00
|
|
|
bool fibmatch;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-04-12 12:34:07 +00:00
|
|
|
err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_ipv6_policy,
|
2017-04-16 16:48:24 +00:00
|
|
|
extack);
|
2006-08-22 07:01:47 +00:00
|
|
|
if (err < 0)
|
|
|
|
goto errout;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-08-22 07:01:47 +00:00
|
|
|
err = -EINVAL;
|
2011-03-12 21:22:43 +00:00
|
|
|
memset(&fl6, 0, sizeof(fl6));
|
2016-06-11 18:08:19 +00:00
|
|
|
rtm = nlmsg_data(nlh);
|
|
|
|
fl6.flowlabel = ip6_make_flowinfo(rtm->rtm_tos, 0);
|
2017-05-25 17:42:40 +00:00
|
|
|
fibmatch = !!(rtm->rtm_flags & RTM_F_FIB_MATCH);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-08-22 07:01:47 +00:00
|
|
|
if (tb[RTA_SRC]) {
|
|
|
|
if (nla_len(tb[RTA_SRC]) < sizeof(struct in6_addr))
|
|
|
|
goto errout;
|
|
|
|
|
2011-11-21 03:39:03 +00:00
|
|
|
fl6.saddr = *(struct in6_addr *)nla_data(tb[RTA_SRC]);
|
2006-08-22 07:01:47 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (tb[RTA_DST]) {
|
|
|
|
if (nla_len(tb[RTA_DST]) < sizeof(struct in6_addr))
|
|
|
|
goto errout;
|
|
|
|
|
2011-11-21 03:39:03 +00:00
|
|
|
fl6.daddr = *(struct in6_addr *)nla_data(tb[RTA_DST]);
|
2006-08-22 07:01:47 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (tb[RTA_IIF])
|
|
|
|
iif = nla_get_u32(tb[RTA_IIF]);
|
|
|
|
|
|
|
|
if (tb[RTA_OIF])
|
2012-04-01 04:03:45 +00:00
|
|
|
oif = nla_get_u32(tb[RTA_OIF]);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2014-05-15 23:38:41 +00:00
|
|
|
if (tb[RTA_MARK])
|
|
|
|
fl6.flowi6_mark = nla_get_u32(tb[RTA_MARK]);
|
|
|
|
|
2016-11-03 17:23:42 +00:00
|
|
|
if (tb[RTA_UID])
|
|
|
|
fl6.flowi6_uid = make_kuid(current_user_ns(),
|
|
|
|
nla_get_u32(tb[RTA_UID]));
|
|
|
|
else
|
|
|
|
fl6.flowi6_uid = iif ? INVALID_UID : current_uid();
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
if (iif) {
|
|
|
|
struct net_device *dev;
|
2012-04-01 04:03:45 +00:00
|
|
|
int flags = 0;
|
|
|
|
|
2017-08-15 14:34:42 +00:00
|
|
|
rcu_read_lock();
|
|
|
|
|
|
|
|
dev = dev_get_by_index_rcu(net, iif);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (!dev) {
|
2017-08-15 14:34:42 +00:00
|
|
|
rcu_read_unlock();
|
2005-04-16 22:20:36 +00:00
|
|
|
err = -ENODEV;
|
2006-08-22 07:01:47 +00:00
|
|
|
goto errout;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2012-04-01 04:03:45 +00:00
|
|
|
|
|
|
|
fl6.flowi6_iif = iif;
|
|
|
|
|
|
|
|
if (!ipv6_addr_any(&fl6.saddr))
|
|
|
|
flags |= RT6_LOOKUP_F_HAS_SADDR;
|
|
|
|
|
2017-05-25 17:42:40 +00:00
|
|
|
if (!fibmatch)
|
|
|
|
dst = ip6_route_input_lookup(net, dev, &fl6, flags);
|
2017-08-18 11:34:22 +00:00
|
|
|
else
|
|
|
|
dst = ip6_route_lookup(net, &fl6, 0);
|
2017-08-15 14:34:42 +00:00
|
|
|
|
|
|
|
rcu_read_unlock();
|
2012-04-01 04:03:45 +00:00
|
|
|
} else {
|
|
|
|
fl6.flowi6_oif = oif;
|
|
|
|
|
2017-05-25 17:42:40 +00:00
|
|
|
if (!fibmatch)
|
|
|
|
dst = ip6_route_output(net, NULL, &fl6);
|
2017-08-18 11:34:22 +00:00
|
|
|
else
|
|
|
|
dst = ip6_route_lookup(net, &fl6, 0);
|
2017-05-25 17:42:40 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
rt = container_of(dst, struct rt6_info, dst);
|
|
|
|
if (rt->dst.error) {
|
|
|
|
err = rt->dst.error;
|
|
|
|
ip6_rt_put(rt);
|
|
|
|
goto errout;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2017-03-02 04:48:39 +00:00
|
|
|
if (rt == net->ipv6.ip6_null_entry) {
|
|
|
|
err = rt->dst.error;
|
|
|
|
ip6_rt_put(rt);
|
|
|
|
goto errout;
|
|
|
|
}
|
|
|
|
|
2006-08-22 07:01:47 +00:00
|
|
|
skb = alloc_skb(NLMSG_GOODSIZE, GFP_KERNEL);
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!skb) {
|
2012-10-29 00:13:19 +00:00
|
|
|
ip6_rt_put(rt);
|
2006-08-22 07:01:47 +00:00
|
|
|
err = -ENOBUFS;
|
|
|
|
goto errout;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2010-06-11 06:31:35 +00:00
|
|
|
skb_dst_set(skb, &rt->dst);
|
2017-05-25 17:42:40 +00:00
|
|
|
if (fibmatch)
|
|
|
|
err = rt6_fill_node(net, skb, rt, NULL, NULL, iif,
|
|
|
|
RTM_NEWROUTE, NETLINK_CB(in_skb).portid,
|
|
|
|
nlh->nlmsg_seq, 0);
|
|
|
|
else
|
|
|
|
err = rt6_fill_node(net, skb, rt, &fl6.daddr, &fl6.saddr, iif,
|
|
|
|
RTM_NEWROUTE, NETLINK_CB(in_skb).portid,
|
|
|
|
nlh->nlmsg_seq, 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (err < 0) {
|
2006-08-22 07:01:47 +00:00
|
|
|
kfree_skb(skb);
|
|
|
|
goto errout;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2012-09-07 20:12:54 +00:00
|
|
|
err = rtnl_unicast(skb, net, NETLINK_CB(in_skb).portid);
|
2006-08-22 07:01:47 +00:00
|
|
|
errout:
|
2005-04-16 22:20:36 +00:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2015-09-13 17:18:33 +00:00
|
|
|
void inet6_rt_notify(int event, struct rt6_info *rt, struct nl_info *info,
|
|
|
|
unsigned int nlm_flags)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct sk_buff *skb;
|
2008-03-04 21:47:47 +00:00
|
|
|
struct net *net = info->nl_net;
|
2007-12-13 17:45:12 +00:00
|
|
|
u32 seq;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
err = -ENOBUFS;
|
2011-12-03 23:02:47 +00:00
|
|
|
seq = info->nlh ? info->nlh->nlmsg_seq : 0;
|
2006-08-22 07:01:08 +00:00
|
|
|
|
2015-07-21 08:43:48 +00:00
|
|
|
skb = nlmsg_new(rt6_nlmsg_size(rt), gfp_any());
|
2011-12-03 23:02:47 +00:00
|
|
|
if (!skb)
|
2006-08-15 07:35:24 +00:00
|
|
|
goto errout;
|
|
|
|
|
2008-08-14 22:33:21 +00:00
|
|
|
err = rt6_fill_node(net, skb, rt, NULL, NULL, 0,
|
2017-01-17 23:51:08 +00:00
|
|
|
event, info->portid, seq, nlm_flags);
|
2007-02-01 07:16:40 +00:00
|
|
|
if (err < 0) {
|
|
|
|
/* -EMSGSIZE implies BUG in rt6_nlmsg_size() */
|
|
|
|
WARN_ON(err == -EMSGSIZE);
|
|
|
|
kfree_skb(skb);
|
|
|
|
goto errout;
|
|
|
|
}
|
2012-09-07 20:12:54 +00:00
|
|
|
rtnl_notify(skb, net, info->portid, RTNLGRP_IPV6_ROUTE,
|
2009-02-25 07:18:28 +00:00
|
|
|
info->nlh, gfp_any());
|
|
|
|
return;
|
2006-08-15 07:35:24 +00:00
|
|
|
errout:
|
|
|
|
if (err < 0)
|
2008-03-04 21:47:47 +00:00
|
|
|
rtnl_set_sk_err(net, RTNLGRP_IPV6_ROUTE, err);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2008-03-04 21:48:30 +00:00
|
|
|
static int ip6_route_dev_notify(struct notifier_block *this,
|
2013-05-28 01:30:21 +00:00
|
|
|
unsigned long event, void *ptr)
|
2008-03-04 21:48:30 +00:00
|
|
|
{
|
2013-05-28 01:30:21 +00:00
|
|
|
struct net_device *dev = netdev_notifier_info_to_dev(ptr);
|
2008-03-25 12:47:49 +00:00
|
|
|
struct net *net = dev_net(dev);
|
2008-03-04 21:48:30 +00:00
|
|
|
|
2017-05-08 17:12:13 +00:00
|
|
|
if (!(dev->flags & IFF_LOOPBACK))
|
|
|
|
return NOTIFY_OK;
|
|
|
|
|
|
|
|
if (event == NETDEV_REGISTER) {
|
2010-06-11 06:31:35 +00:00
|
|
|
net->ipv6.ip6_null_entry->dst.dev = dev;
|
2008-03-04 21:48:30 +00:00
|
|
|
net->ipv6.ip6_null_entry->rt6i_idev = in6_dev_get(dev);
|
|
|
|
#ifdef CONFIG_IPV6_MULTIPLE_TABLES
|
2010-06-11 06:31:35 +00:00
|
|
|
net->ipv6.ip6_prohibit_entry->dst.dev = dev;
|
2008-03-04 21:48:30 +00:00
|
|
|
net->ipv6.ip6_prohibit_entry->rt6i_idev = in6_dev_get(dev);
|
2010-06-11 06:31:35 +00:00
|
|
|
net->ipv6.ip6_blk_hole_entry->dst.dev = dev;
|
2008-03-04 21:48:30 +00:00
|
|
|
net->ipv6.ip6_blk_hole_entry->rt6i_idev = in6_dev_get(dev);
|
2017-05-08 17:12:13 +00:00
|
|
|
#endif
|
2017-06-20 18:42:27 +00:00
|
|
|
} else if (event == NETDEV_UNREGISTER &&
|
|
|
|
dev->reg_state != NETREG_UNREGISTERED) {
|
|
|
|
/* NETDEV_UNREGISTER could be fired for multiple times by
|
|
|
|
* netdev_wait_allrefs(). Make sure we only call this once.
|
|
|
|
*/
|
2017-08-15 11:09:51 +00:00
|
|
|
in6_dev_put_clear(&net->ipv6.ip6_null_entry->rt6i_idev);
|
2017-05-08 17:12:13 +00:00
|
|
|
#ifdef CONFIG_IPV6_MULTIPLE_TABLES
|
2017-08-15 11:09:51 +00:00
|
|
|
in6_dev_put_clear(&net->ipv6.ip6_prohibit_entry->rt6i_idev);
|
|
|
|
in6_dev_put_clear(&net->ipv6.ip6_blk_hole_entry->rt6i_idev);
|
2008-03-04 21:48:30 +00:00
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
return NOTIFY_OK;
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* /proc
|
|
|
|
*/
|
|
|
|
|
|
|
|
#ifdef CONFIG_PROC_FS
|
|
|
|
|
2007-11-06 13:27:11 +00:00
|
|
|
static const struct file_operations ipv6_route_proc_fops = {
|
|
|
|
.owner = THIS_MODULE,
|
|
|
|
.open = ipv6_route_open,
|
|
|
|
.read = seq_read,
|
|
|
|
.llseek = seq_lseek,
|
2013-09-21 14:55:59 +00:00
|
|
|
.release = seq_release_net,
|
2007-11-06 13:27:11 +00:00
|
|
|
};
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
static int rt6_stats_seq_show(struct seq_file *seq, void *v)
|
|
|
|
{
|
2008-03-04 21:46:23 +00:00
|
|
|
struct net *net = (struct net *)seq->private;
|
2005-04-16 22:20:36 +00:00
|
|
|
seq_printf(seq, "%04x %04x %04x %04x %04x %04x %04x\n",
|
2008-03-04 21:46:23 +00:00
|
|
|
net->ipv6.rt6_stats->fib_nodes,
|
|
|
|
net->ipv6.rt6_stats->fib_route_nodes,
|
2017-10-06 19:06:11 +00:00
|
|
|
atomic_read(&net->ipv6.rt6_stats->fib_rt_alloc),
|
2008-03-04 21:46:23 +00:00
|
|
|
net->ipv6.rt6_stats->fib_rt_entries,
|
|
|
|
net->ipv6.rt6_stats->fib_rt_cache,
|
2010-10-08 06:37:34 +00:00
|
|
|
dst_entries_get_slow(&net->ipv6.ip6_dst_ops),
|
2008-03-04 21:46:23 +00:00
|
|
|
net->ipv6.rt6_stats->fib_discarded_routes);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int rt6_stats_seq_open(struct inode *inode, struct file *file)
|
|
|
|
{
|
2008-07-18 11:07:21 +00:00
|
|
|
return single_open_net(inode, file, rt6_stats_seq_show);
|
2008-03-04 21:46:23 +00:00
|
|
|
}
|
|
|
|
|
2007-02-12 08:55:35 +00:00
|
|
|
static const struct file_operations rt6_stats_seq_fops = {
|
2005-04-16 22:20:36 +00:00
|
|
|
.owner = THIS_MODULE,
|
|
|
|
.open = rt6_stats_seq_open,
|
|
|
|
.read = seq_read,
|
|
|
|
.llseek = seq_lseek,
|
2008-07-18 11:07:44 +00:00
|
|
|
.release = single_release_net,
|
2005-04-16 22:20:36 +00:00
|
|
|
};
|
|
|
|
#endif /* CONFIG_PROC_FS */
|
|
|
|
|
|
|
|
#ifdef CONFIG_SYSCTL
|
|
|
|
|
|
|
|
static
|
2013-06-12 06:04:25 +00:00
|
|
|
int ipv6_sysctl_rtcache_flush(struct ctl_table *ctl, int write,
|
2005-04-16 22:20:36 +00:00
|
|
|
void __user *buffer, size_t *lenp, loff_t *ppos)
|
|
|
|
{
|
2011-02-24 19:48:03 +00:00
|
|
|
struct net *net;
|
|
|
|
int delay;
|
|
|
|
if (!write)
|
2005-04-16 22:20:36 +00:00
|
|
|
return -EINVAL;
|
2011-02-24 19:48:03 +00:00
|
|
|
|
|
|
|
net = (struct net *)ctl->extra1;
|
|
|
|
delay = net->ipv6.sysctl.flush_delay;
|
|
|
|
proc_dointvec(ctl, write, buffer, lenp, ppos);
|
2013-08-01 08:04:14 +00:00
|
|
|
fib6_run_gc(delay <= 0 ? 0 : (unsigned long)delay, net, delay > 0);
|
2011-02-24 19:48:03 +00:00
|
|
|
return 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2013-06-12 06:04:25 +00:00
|
|
|
struct ctl_table ipv6_route_table_template[] = {
|
2007-02-09 14:24:49 +00:00
|
|
|
{
|
2005-04-16 22:20:36 +00:00
|
|
|
.procname = "flush",
|
2008-01-10 11:01:01 +00:00
|
|
|
.data = &init_net.ipv6.sysctl.flush_delay,
|
2005-04-16 22:20:36 +00:00
|
|
|
.maxlen = sizeof(int),
|
2005-04-28 19:11:49 +00:00
|
|
|
.mode = 0200,
|
2008-11-04 02:21:05 +00:00
|
|
|
.proc_handler = ipv6_sysctl_rtcache_flush
|
2005-04-16 22:20:36 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
.procname = "gc_thresh",
|
2008-03-04 21:48:53 +00:00
|
|
|
.data = &ip6_dst_ops_template.gc_thresh,
|
2005-04-16 22:20:36 +00:00
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
2008-11-04 02:21:05 +00:00
|
|
|
.proc_handler = proc_dointvec,
|
2005-04-16 22:20:36 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
.procname = "max_size",
|
2008-01-10 11:01:01 +00:00
|
|
|
.data = &init_net.ipv6.sysctl.ip6_rt_max_size,
|
2005-04-16 22:20:36 +00:00
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
2008-11-04 02:21:05 +00:00
|
|
|
.proc_handler = proc_dointvec,
|
2005-04-16 22:20:36 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
.procname = "gc_min_interval",
|
2008-01-10 11:01:01 +00:00
|
|
|
.data = &init_net.ipv6.sysctl.ip6_rt_gc_min_interval,
|
2005-04-16 22:20:36 +00:00
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
2008-11-04 02:21:05 +00:00
|
|
|
.proc_handler = proc_dointvec_jiffies,
|
2005-04-16 22:20:36 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
.procname = "gc_timeout",
|
2008-01-10 11:01:01 +00:00
|
|
|
.data = &init_net.ipv6.sysctl.ip6_rt_gc_timeout,
|
2005-04-16 22:20:36 +00:00
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
2008-11-04 02:21:05 +00:00
|
|
|
.proc_handler = proc_dointvec_jiffies,
|
2005-04-16 22:20:36 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
.procname = "gc_interval",
|
2008-01-10 11:01:01 +00:00
|
|
|
.data = &init_net.ipv6.sysctl.ip6_rt_gc_interval,
|
2005-04-16 22:20:36 +00:00
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
2008-11-04 02:21:05 +00:00
|
|
|
.proc_handler = proc_dointvec_jiffies,
|
2005-04-16 22:20:36 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
.procname = "gc_elasticity",
|
2008-01-10 11:01:01 +00:00
|
|
|
.data = &init_net.ipv6.sysctl.ip6_rt_gc_elasticity,
|
2005-04-16 22:20:36 +00:00
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
2010-08-15 05:42:51 +00:00
|
|
|
.proc_handler = proc_dointvec,
|
2005-04-16 22:20:36 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
.procname = "mtu_expires",
|
2008-01-10 11:01:01 +00:00
|
|
|
.data = &init_net.ipv6.sysctl.ip6_rt_mtu_expires,
|
2005-04-16 22:20:36 +00:00
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
2008-11-04 02:21:05 +00:00
|
|
|
.proc_handler = proc_dointvec_jiffies,
|
2005-04-16 22:20:36 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
.procname = "min_adv_mss",
|
2008-01-10 11:01:01 +00:00
|
|
|
.data = &init_net.ipv6.sysctl.ip6_rt_min_advmss,
|
2005-04-16 22:20:36 +00:00
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
2010-08-15 05:42:51 +00:00
|
|
|
.proc_handler = proc_dointvec,
|
2005-04-16 22:20:36 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
.procname = "gc_min_interval_ms",
|
2008-01-10 11:01:01 +00:00
|
|
|
.data = &init_net.ipv6.sysctl.ip6_rt_gc_min_interval,
|
2005-04-16 22:20:36 +00:00
|
|
|
.maxlen = sizeof(int),
|
|
|
|
.mode = 0644,
|
2008-11-04 02:21:05 +00:00
|
|
|
.proc_handler = proc_dointvec_ms_jiffies,
|
2005-04-16 22:20:36 +00:00
|
|
|
},
|
2009-11-05 21:32:03 +00:00
|
|
|
{ }
|
2005-04-16 22:20:36 +00:00
|
|
|
};
|
|
|
|
|
2010-01-17 03:35:32 +00:00
|
|
|
struct ctl_table * __net_init ipv6_route_sysctl_init(struct net *net)
|
2008-01-10 10:53:43 +00:00
|
|
|
{
|
|
|
|
struct ctl_table *table;
|
|
|
|
|
|
|
|
table = kmemdup(ipv6_route_table_template,
|
|
|
|
sizeof(ipv6_route_table_template),
|
|
|
|
GFP_KERNEL);
|
2008-02-27 15:24:28 +00:00
|
|
|
|
|
|
|
if (table) {
|
|
|
|
table[0].data = &net->ipv6.sysctl.flush_delay;
|
2011-02-24 19:48:03 +00:00
|
|
|
table[0].extra1 = net;
|
2009-08-29 01:34:49 +00:00
|
|
|
table[1].data = &net->ipv6.ip6_dst_ops.gc_thresh;
|
2008-02-27 15:24:28 +00:00
|
|
|
table[2].data = &net->ipv6.sysctl.ip6_rt_max_size;
|
|
|
|
table[3].data = &net->ipv6.sysctl.ip6_rt_gc_min_interval;
|
|
|
|
table[4].data = &net->ipv6.sysctl.ip6_rt_gc_timeout;
|
|
|
|
table[5].data = &net->ipv6.sysctl.ip6_rt_gc_interval;
|
|
|
|
table[6].data = &net->ipv6.sysctl.ip6_rt_gc_elasticity;
|
|
|
|
table[7].data = &net->ipv6.sysctl.ip6_rt_mtu_expires;
|
|
|
|
table[8].data = &net->ipv6.sysctl.ip6_rt_min_advmss;
|
2009-12-19 04:11:03 +00:00
|
|
|
table[9].data = &net->ipv6.sysctl.ip6_rt_gc_min_interval;
|
2012-11-16 03:02:59 +00:00
|
|
|
|
|
|
|
/* Don't export sysctls to unprivileged users */
|
|
|
|
if (net->user_ns != &init_user_ns)
|
|
|
|
table[0].procname = NULL;
|
2008-02-27 15:24:28 +00:00
|
|
|
}
|
|
|
|
|
2008-01-10 10:53:43 +00:00
|
|
|
return table;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif
|
|
|
|
|
2010-01-17 03:35:32 +00:00
|
|
|
static int __net_init ip6_route_net_init(struct net *net)
|
2008-03-04 21:45:33 +00:00
|
|
|
{
|
2008-04-21 21:25:23 +00:00
|
|
|
int ret = -ENOMEM;
|
2008-03-04 21:48:30 +00:00
|
|
|
|
2009-08-29 01:34:49 +00:00
|
|
|
memcpy(&net->ipv6.ip6_dst_ops, &ip6_dst_ops_template,
|
|
|
|
sizeof(net->ipv6.ip6_dst_ops));
|
2008-03-04 21:49:23 +00:00
|
|
|
|
2010-10-08 06:37:34 +00:00
|
|
|
if (dst_entries_init(&net->ipv6.ip6_dst_ops) < 0)
|
|
|
|
goto out_ip6_dst_ops;
|
|
|
|
|
2008-03-04 21:48:30 +00:00
|
|
|
net->ipv6.ip6_null_entry = kmemdup(&ip6_null_entry_template,
|
|
|
|
sizeof(*net->ipv6.ip6_null_entry),
|
|
|
|
GFP_KERNEL);
|
|
|
|
if (!net->ipv6.ip6_null_entry)
|
2010-10-08 06:37:34 +00:00
|
|
|
goto out_ip6_dst_entries;
|
2010-06-11 06:31:35 +00:00
|
|
|
net->ipv6.ip6_null_entry->dst.path =
|
2008-03-04 21:48:30 +00:00
|
|
|
(struct dst_entry *)net->ipv6.ip6_null_entry;
|
2010-06-11 06:31:35 +00:00
|
|
|
net->ipv6.ip6_null_entry->dst.ops = &net->ipv6.ip6_dst_ops;
|
net: Implement read-only protection and COW'ing of metrics.
Routing metrics are now copy-on-write.
Initially a route entry points it's metrics at a read-only location.
If a routing table entry exists, it will point there. Else it will
point at the all zero metric place-holder called 'dst_default_metrics'.
The writeability state of the metrics is stored in the low bits of the
metrics pointer, we have two bits left to spare if we want to store
more states.
For the initial implementation, COW is implemented simply via kmalloc.
However future enhancements will change this to place the writable
metrics somewhere else, in order to increase sharing. Very likely
this "somewhere else" will be the inetpeer cache.
Note also that this means that metrics updates may transiently fail
if we cannot COW the metrics successfully.
But even by itself, this patch should decrease memory usage and
increase cache locality especially for routing workloads. In those
cases the read-only metric copies stay in place and never get written
to.
TCP workloads where metrics get updated, and those rare cases where
PMTU triggers occur, will take a very slight performance hit. But
that hit will be alleviated when the long-term writable metrics
move to a more sharable location.
Since the metrics storage went from a u32 array of RTAX_MAX entries to
what is essentially a pointer, some retooling of the dst_entry layout
was necessary.
Most importantly, we need to preserve the alignment of the reference
count so that it doesn't share cache lines with the read-mostly state,
as per Eric Dumazet's alignment assertion checks.
The only non-trivial bit here is the move of the 'flags' member into
the writeable cacheline. This is OK since we are always accessing the
flags around the same moment when we made a modification to the
reference count.
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-01-27 04:51:05 +00:00
|
|
|
dst_init_metrics(&net->ipv6.ip6_null_entry->dst,
|
|
|
|
ip6_template_metrics, true);
|
2008-03-04 21:48:30 +00:00
|
|
|
|
|
|
|
#ifdef CONFIG_IPV6_MULTIPLE_TABLES
|
2017-08-08 18:23:49 +00:00
|
|
|
net->ipv6.fib6_has_custom_rules = false;
|
2008-03-04 21:48:30 +00:00
|
|
|
net->ipv6.ip6_prohibit_entry = kmemdup(&ip6_prohibit_entry_template,
|
|
|
|
sizeof(*net->ipv6.ip6_prohibit_entry),
|
|
|
|
GFP_KERNEL);
|
2008-10-07 21:12:10 +00:00
|
|
|
if (!net->ipv6.ip6_prohibit_entry)
|
|
|
|
goto out_ip6_null_entry;
|
2010-06-11 06:31:35 +00:00
|
|
|
net->ipv6.ip6_prohibit_entry->dst.path =
|
2008-03-04 21:48:30 +00:00
|
|
|
(struct dst_entry *)net->ipv6.ip6_prohibit_entry;
|
2010-06-11 06:31:35 +00:00
|
|
|
net->ipv6.ip6_prohibit_entry->dst.ops = &net->ipv6.ip6_dst_ops;
|
net: Implement read-only protection and COW'ing of metrics.
Routing metrics are now copy-on-write.
Initially a route entry points it's metrics at a read-only location.
If a routing table entry exists, it will point there. Else it will
point at the all zero metric place-holder called 'dst_default_metrics'.
The writeability state of the metrics is stored in the low bits of the
metrics pointer, we have two bits left to spare if we want to store
more states.
For the initial implementation, COW is implemented simply via kmalloc.
However future enhancements will change this to place the writable
metrics somewhere else, in order to increase sharing. Very likely
this "somewhere else" will be the inetpeer cache.
Note also that this means that metrics updates may transiently fail
if we cannot COW the metrics successfully.
But even by itself, this patch should decrease memory usage and
increase cache locality especially for routing workloads. In those
cases the read-only metric copies stay in place and never get written
to.
TCP workloads where metrics get updated, and those rare cases where
PMTU triggers occur, will take a very slight performance hit. But
that hit will be alleviated when the long-term writable metrics
move to a more sharable location.
Since the metrics storage went from a u32 array of RTAX_MAX entries to
what is essentially a pointer, some retooling of the dst_entry layout
was necessary.
Most importantly, we need to preserve the alignment of the reference
count so that it doesn't share cache lines with the read-mostly state,
as per Eric Dumazet's alignment assertion checks.
The only non-trivial bit here is the move of the 'flags' member into
the writeable cacheline. This is OK since we are always accessing the
flags around the same moment when we made a modification to the
reference count.
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-01-27 04:51:05 +00:00
|
|
|
dst_init_metrics(&net->ipv6.ip6_prohibit_entry->dst,
|
|
|
|
ip6_template_metrics, true);
|
2008-03-04 21:48:30 +00:00
|
|
|
|
|
|
|
net->ipv6.ip6_blk_hole_entry = kmemdup(&ip6_blk_hole_entry_template,
|
|
|
|
sizeof(*net->ipv6.ip6_blk_hole_entry),
|
|
|
|
GFP_KERNEL);
|
2008-10-07 21:12:10 +00:00
|
|
|
if (!net->ipv6.ip6_blk_hole_entry)
|
|
|
|
goto out_ip6_prohibit_entry;
|
2010-06-11 06:31:35 +00:00
|
|
|
net->ipv6.ip6_blk_hole_entry->dst.path =
|
2008-03-04 21:48:30 +00:00
|
|
|
(struct dst_entry *)net->ipv6.ip6_blk_hole_entry;
|
2010-06-11 06:31:35 +00:00
|
|
|
net->ipv6.ip6_blk_hole_entry->dst.ops = &net->ipv6.ip6_dst_ops;
|
net: Implement read-only protection and COW'ing of metrics.
Routing metrics are now copy-on-write.
Initially a route entry points it's metrics at a read-only location.
If a routing table entry exists, it will point there. Else it will
point at the all zero metric place-holder called 'dst_default_metrics'.
The writeability state of the metrics is stored in the low bits of the
metrics pointer, we have two bits left to spare if we want to store
more states.
For the initial implementation, COW is implemented simply via kmalloc.
However future enhancements will change this to place the writable
metrics somewhere else, in order to increase sharing. Very likely
this "somewhere else" will be the inetpeer cache.
Note also that this means that metrics updates may transiently fail
if we cannot COW the metrics successfully.
But even by itself, this patch should decrease memory usage and
increase cache locality especially for routing workloads. In those
cases the read-only metric copies stay in place and never get written
to.
TCP workloads where metrics get updated, and those rare cases where
PMTU triggers occur, will take a very slight performance hit. But
that hit will be alleviated when the long-term writable metrics
move to a more sharable location.
Since the metrics storage went from a u32 array of RTAX_MAX entries to
what is essentially a pointer, some retooling of the dst_entry layout
was necessary.
Most importantly, we need to preserve the alignment of the reference
count so that it doesn't share cache lines with the read-mostly state,
as per Eric Dumazet's alignment assertion checks.
The only non-trivial bit here is the move of the 'flags' member into
the writeable cacheline. This is OK since we are always accessing the
flags around the same moment when we made a modification to the
reference count.
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-01-27 04:51:05 +00:00
|
|
|
dst_init_metrics(&net->ipv6.ip6_blk_hole_entry->dst,
|
|
|
|
ip6_template_metrics, true);
|
2008-03-04 21:48:30 +00:00
|
|
|
#endif
|
|
|
|
|
2008-10-07 21:15:00 +00:00
|
|
|
net->ipv6.sysctl.flush_delay = 0;
|
|
|
|
net->ipv6.sysctl.ip6_rt_max_size = 4096;
|
|
|
|
net->ipv6.sysctl.ip6_rt_gc_min_interval = HZ / 2;
|
|
|
|
net->ipv6.sysctl.ip6_rt_gc_timeout = 60*HZ;
|
|
|
|
net->ipv6.sysctl.ip6_rt_gc_interval = 30*HZ;
|
|
|
|
net->ipv6.sysctl.ip6_rt_gc_elasticity = 9;
|
|
|
|
net->ipv6.sysctl.ip6_rt_mtu_expires = 10*60*HZ;
|
|
|
|
net->ipv6.sysctl.ip6_rt_min_advmss = IPV6_MIN_MTU - 20 - 40;
|
|
|
|
|
2008-03-04 21:49:47 +00:00
|
|
|
net->ipv6.ip6_rt_gc_expire = 30*HZ;
|
|
|
|
|
2008-03-04 21:48:30 +00:00
|
|
|
ret = 0;
|
|
|
|
out:
|
|
|
|
return ret;
|
2008-03-04 21:49:23 +00:00
|
|
|
|
2008-10-07 21:12:10 +00:00
|
|
|
#ifdef CONFIG_IPV6_MULTIPLE_TABLES
|
|
|
|
out_ip6_prohibit_entry:
|
|
|
|
kfree(net->ipv6.ip6_prohibit_entry);
|
|
|
|
out_ip6_null_entry:
|
|
|
|
kfree(net->ipv6.ip6_null_entry);
|
|
|
|
#endif
|
2010-10-08 06:37:34 +00:00
|
|
|
out_ip6_dst_entries:
|
|
|
|
dst_entries_destroy(&net->ipv6.ip6_dst_ops);
|
2008-03-04 21:49:23 +00:00
|
|
|
out_ip6_dst_ops:
|
|
|
|
goto out;
|
2008-03-04 21:45:33 +00:00
|
|
|
}
|
|
|
|
|
2010-01-17 03:35:32 +00:00
|
|
|
static void __net_exit ip6_route_net_exit(struct net *net)
|
2008-03-04 21:45:33 +00:00
|
|
|
{
|
2008-03-04 21:48:30 +00:00
|
|
|
kfree(net->ipv6.ip6_null_entry);
|
|
|
|
#ifdef CONFIG_IPV6_MULTIPLE_TABLES
|
|
|
|
kfree(net->ipv6.ip6_prohibit_entry);
|
|
|
|
kfree(net->ipv6.ip6_blk_hole_entry);
|
|
|
|
#endif
|
net dst: fix percpu_counter list corruption and poison overwritten
There're some percpu_counter list corruption and poison overwritten warnings
in recent kernel, which is resulted by fc66f95c.
commit fc66f95c switches to use percpu_counter, in ip6_route_net_init, kernel
init the percpu_counter for dst entries, but, the percpu_counter is never destroyed
in ip6_route_net_exit. So if the related data is freed by kernel, the freed percpu_counter
is still on the list, then if we insert/remove other percpu_counter, list corruption
resulted. Also, if the insert/remove option modifies the ->prev,->next pointer of
the freed value, the poison overwritten is resulted then.
With the following patch, the percpu_counter list corruption and poison overwritten
warnings disappeared.
Signed-off-by: Xiaotian Feng <dfeng@redhat.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: "Pekka Savola (ipv6)" <pekkas@netcore.fi>
Cc: James Morris <jmorris@namei.org>
Cc: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
Cc: Patrick McHardy <kaber@trash.net>
Acked-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-11-02 16:11:05 +00:00
|
|
|
dst_entries_destroy(&net->ipv6.ip6_dst_ops);
|
2008-03-04 21:45:33 +00:00
|
|
|
}
|
|
|
|
|
2012-06-18 12:08:33 +00:00
|
|
|
static int __net_init ip6_route_net_init_late(struct net *net)
|
|
|
|
{
|
|
|
|
#ifdef CONFIG_PROC_FS
|
2013-02-18 01:34:54 +00:00
|
|
|
proc_create("ipv6_route", 0, net->proc_net, &ipv6_route_proc_fops);
|
|
|
|
proc_create("rt6_stats", S_IRUGO, net->proc_net, &rt6_stats_seq_fops);
|
2012-06-18 12:08:33 +00:00
|
|
|
#endif
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __net_exit ip6_route_net_exit_late(struct net *net)
|
|
|
|
{
|
|
|
|
#ifdef CONFIG_PROC_FS
|
2013-02-18 01:34:56 +00:00
|
|
|
remove_proc_entry("ipv6_route", net->proc_net);
|
|
|
|
remove_proc_entry("rt6_stats", net->proc_net);
|
2012-06-18 12:08:33 +00:00
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2008-03-04 21:45:33 +00:00
|
|
|
static struct pernet_operations ip6_route_net_ops = {
|
|
|
|
.init = ip6_route_net_init,
|
|
|
|
.exit = ip6_route_net_exit,
|
|
|
|
};
|
|
|
|
|
2012-06-09 23:27:05 +00:00
|
|
|
static int __net_init ipv6_inetpeer_init(struct net *net)
|
|
|
|
{
|
|
|
|
struct inet_peer_base *bp = kmalloc(sizeof(*bp), GFP_KERNEL);
|
|
|
|
|
|
|
|
if (!bp)
|
|
|
|
return -ENOMEM;
|
|
|
|
inet_peer_base_init(bp);
|
|
|
|
net->ipv6.peers = bp;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __net_exit ipv6_inetpeer_exit(struct net *net)
|
|
|
|
{
|
|
|
|
struct inet_peer_base *bp = net->ipv6.peers;
|
|
|
|
|
|
|
|
net->ipv6.peers = NULL;
|
2012-06-09 23:32:41 +00:00
|
|
|
inetpeer_invalidate_tree(bp);
|
2012-06-09 23:27:05 +00:00
|
|
|
kfree(bp);
|
|
|
|
}
|
|
|
|
|
2012-06-10 02:00:16 +00:00
|
|
|
static struct pernet_operations ipv6_inetpeer_ops = {
|
2012-06-09 23:27:05 +00:00
|
|
|
.init = ipv6_inetpeer_init,
|
|
|
|
.exit = ipv6_inetpeer_exit,
|
|
|
|
};
|
|
|
|
|
2012-06-18 12:08:33 +00:00
|
|
|
static struct pernet_operations ip6_route_net_late_ops = {
|
|
|
|
.init = ip6_route_net_init_late,
|
|
|
|
.exit = ip6_route_net_exit_late,
|
|
|
|
};
|
|
|
|
|
2008-03-04 21:48:30 +00:00
|
|
|
static struct notifier_block ip6_route_dev_notifier = {
|
|
|
|
.notifier_call = ip6_route_dev_notify,
|
2017-05-08 17:12:13 +00:00
|
|
|
.priority = ADDRCONF_NOTIFY_PRIORITY - 10,
|
2008-03-04 21:48:30 +00:00
|
|
|
};
|
|
|
|
|
2017-05-04 05:07:31 +00:00
|
|
|
void __init ip6_route_init_special_entries(void)
|
|
|
|
{
|
|
|
|
/* Registering of the loopback is done before this portion of code,
|
|
|
|
* the loopback reference in rt6_info will not be taken, do it
|
|
|
|
* manually for init_net */
|
|
|
|
init_net.ipv6.ip6_null_entry->dst.dev = init_net.loopback_dev;
|
|
|
|
init_net.ipv6.ip6_null_entry->rt6i_idev = in6_dev_get(init_net.loopback_dev);
|
|
|
|
#ifdef CONFIG_IPV6_MULTIPLE_TABLES
|
|
|
|
init_net.ipv6.ip6_prohibit_entry->dst.dev = init_net.loopback_dev;
|
|
|
|
init_net.ipv6.ip6_prohibit_entry->rt6i_idev = in6_dev_get(init_net.loopback_dev);
|
|
|
|
init_net.ipv6.ip6_blk_hole_entry->dst.dev = init_net.loopback_dev;
|
|
|
|
init_net.ipv6.ip6_blk_hole_entry->rt6i_idev = in6_dev_get(init_net.loopback_dev);
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2007-12-07 08:43:48 +00:00
|
|
|
int __init ip6_route_init(void)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2007-12-07 08:43:48 +00:00
|
|
|
int ret;
|
2015-05-23 03:56:04 +00:00
|
|
|
int cpu;
|
2007-12-07 08:43:48 +00:00
|
|
|
|
2008-03-04 21:48:53 +00:00
|
|
|
ret = -ENOMEM;
|
|
|
|
ip6_dst_ops_template.kmem_cachep =
|
2006-08-27 02:25:52 +00:00
|
|
|
kmem_cache_create("ip6_dst_cache", sizeof(struct rt6_info), 0,
|
2007-12-07 08:45:16 +00:00
|
|
|
SLAB_HWCACHE_ALIGN, NULL);
|
2008-03-04 21:48:53 +00:00
|
|
|
if (!ip6_dst_ops_template.kmem_cachep)
|
2009-01-08 02:09:08 +00:00
|
|
|
goto out;
|
2007-05-25 01:17:54 +00:00
|
|
|
|
2010-10-08 06:37:34 +00:00
|
|
|
ret = dst_entries_init(&ip6_dst_blackhole_ops);
|
2008-03-04 21:48:30 +00:00
|
|
|
if (ret)
|
2008-03-04 21:48:10 +00:00
|
|
|
goto out_kmem_cache;
|
|
|
|
|
2012-06-09 23:27:05 +00:00
|
|
|
ret = register_pernet_subsys(&ipv6_inetpeer_ops);
|
|
|
|
if (ret)
|
2012-06-16 08:12:19 +00:00
|
|
|
goto out_dst_entries;
|
2012-06-14 23:00:17 +00:00
|
|
|
|
2012-06-15 22:51:55 +00:00
|
|
|
ret = register_pernet_subsys(&ip6_route_net_ops);
|
|
|
|
if (ret)
|
|
|
|
goto out_register_inetpeer;
|
2012-06-09 23:27:05 +00:00
|
|
|
|
2008-10-01 09:37:56 +00:00
|
|
|
ip6_dst_blackhole_ops.kmem_cachep = ip6_dst_ops_template.kmem_cachep;
|
|
|
|
|
2012-06-16 08:12:19 +00:00
|
|
|
ret = fib6_init();
|
2007-12-07 08:43:48 +00:00
|
|
|
if (ret)
|
2008-03-04 21:48:30 +00:00
|
|
|
goto out_register_subsys;
|
2007-12-07 08:43:48 +00:00
|
|
|
|
|
|
|
ret = xfrm6_init();
|
|
|
|
if (ret)
|
2012-06-16 08:12:19 +00:00
|
|
|
goto out_fib6_init;
|
2007-12-08 08:14:11 +00:00
|
|
|
|
2007-12-07 08:43:48 +00:00
|
|
|
ret = fib6_rules_init();
|
|
|
|
if (ret)
|
|
|
|
goto xfrm6_init;
|
2007-12-08 08:14:54 +00:00
|
|
|
|
2012-06-18 12:08:33 +00:00
|
|
|
ret = register_pernet_subsys(&ip6_route_net_late_ops);
|
|
|
|
if (ret)
|
|
|
|
goto fib6_rules_init;
|
|
|
|
|
2007-12-07 08:43:48 +00:00
|
|
|
ret = -ENOBUFS;
|
2017-08-09 18:41:48 +00:00
|
|
|
if (__rtnl_register(PF_INET6, RTM_NEWROUTE, inet6_rtm_newroute, NULL, 0) ||
|
|
|
|
__rtnl_register(PF_INET6, RTM_DELROUTE, inet6_rtm_delroute, NULL, 0) ||
|
2017-08-15 14:34:43 +00:00
|
|
|
__rtnl_register(PF_INET6, RTM_GETROUTE, inet6_rtm_getroute, NULL,
|
|
|
|
RTNL_FLAG_DOIT_UNLOCKED))
|
2012-06-18 12:08:33 +00:00
|
|
|
goto out_register_late_subsys;
|
2007-03-22 18:58:32 +00:00
|
|
|
|
2008-03-04 21:48:30 +00:00
|
|
|
ret = register_netdevice_notifier(&ip6_route_dev_notifier);
|
2008-03-04 21:45:33 +00:00
|
|
|
if (ret)
|
2012-06-18 12:08:33 +00:00
|
|
|
goto out_register_late_subsys;
|
2008-03-04 21:48:30 +00:00
|
|
|
|
2015-05-23 03:56:04 +00:00
|
|
|
for_each_possible_cpu(cpu) {
|
|
|
|
struct uncached_list *ul = per_cpu_ptr(&rt6_uncached_list, cpu);
|
|
|
|
|
|
|
|
INIT_LIST_HEAD(&ul->head);
|
|
|
|
spin_lock_init(&ul->lock);
|
|
|
|
}
|
|
|
|
|
2007-12-07 08:43:48 +00:00
|
|
|
out:
|
|
|
|
return ret;
|
|
|
|
|
2012-06-18 12:08:33 +00:00
|
|
|
out_register_late_subsys:
|
|
|
|
unregister_pernet_subsys(&ip6_route_net_late_ops);
|
2007-12-07 08:43:48 +00:00
|
|
|
fib6_rules_init:
|
|
|
|
fib6_rules_cleanup();
|
|
|
|
xfrm6_init:
|
|
|
|
xfrm6_fini();
|
2012-06-14 23:00:17 +00:00
|
|
|
out_fib6_init:
|
|
|
|
fib6_gc_cleanup();
|
2008-03-04 21:48:30 +00:00
|
|
|
out_register_subsys:
|
|
|
|
unregister_pernet_subsys(&ip6_route_net_ops);
|
2012-06-15 22:51:55 +00:00
|
|
|
out_register_inetpeer:
|
|
|
|
unregister_pernet_subsys(&ipv6_inetpeer_ops);
|
2010-10-08 06:37:34 +00:00
|
|
|
out_dst_entries:
|
|
|
|
dst_entries_destroy(&ip6_dst_blackhole_ops);
|
2007-12-07 08:43:48 +00:00
|
|
|
out_kmem_cache:
|
2008-03-04 21:49:23 +00:00
|
|
|
kmem_cache_destroy(ip6_dst_ops_template.kmem_cachep);
|
2007-12-07 08:43:48 +00:00
|
|
|
goto out;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void ip6_route_cleanup(void)
|
|
|
|
{
|
2008-03-04 21:48:30 +00:00
|
|
|
unregister_netdevice_notifier(&ip6_route_dev_notifier);
|
2012-06-18 12:08:33 +00:00
|
|
|
unregister_pernet_subsys(&ip6_route_net_late_ops);
|
2006-08-04 10:39:02 +00:00
|
|
|
fib6_rules_cleanup();
|
2005-04-16 22:20:36 +00:00
|
|
|
xfrm6_fini();
|
|
|
|
fib6_gc_cleanup();
|
2012-06-09 23:27:05 +00:00
|
|
|
unregister_pernet_subsys(&ipv6_inetpeer_ops);
|
2008-03-04 21:48:30 +00:00
|
|
|
unregister_pernet_subsys(&ip6_route_net_ops);
|
net dst: fix percpu_counter list corruption and poison overwritten
There're some percpu_counter list corruption and poison overwritten warnings
in recent kernel, which is resulted by fc66f95c.
commit fc66f95c switches to use percpu_counter, in ip6_route_net_init, kernel
init the percpu_counter for dst entries, but, the percpu_counter is never destroyed
in ip6_route_net_exit. So if the related data is freed by kernel, the freed percpu_counter
is still on the list, then if we insert/remove other percpu_counter, list corruption
resulted. Also, if the insert/remove option modifies the ->prev,->next pointer of
the freed value, the poison overwritten is resulted then.
With the following patch, the percpu_counter list corruption and poison overwritten
warnings disappeared.
Signed-off-by: Xiaotian Feng <dfeng@redhat.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: "Pekka Savola (ipv6)" <pekkas@netcore.fi>
Cc: James Morris <jmorris@namei.org>
Cc: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
Cc: Patrick McHardy <kaber@trash.net>
Acked-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-11-02 16:11:05 +00:00
|
|
|
dst_entries_destroy(&ip6_dst_blackhole_ops);
|
2008-03-04 21:49:23 +00:00
|
|
|
kmem_cache_destroy(ip6_dst_ops_template.kmem_cachep);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|