2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* INET An implementation of the TCP/IP protocol suite for the LINUX
|
|
|
|
* operating system. INET is implemented using the BSD Socket
|
|
|
|
* interface as the means of communication with the user level.
|
|
|
|
*
|
|
|
|
* Implementation of the Transmission Control Protocol(TCP).
|
|
|
|
*
|
2005-05-05 23:16:16 +00:00
|
|
|
* Authors: Ross Biro
|
2005-04-16 22:20:36 +00:00
|
|
|
* Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG>
|
|
|
|
* Mark Evans, <evansmp@uhura.aston.ac.uk>
|
|
|
|
* Corey Minyard <wf-rch!minyard@relay.EU.net>
|
|
|
|
* Florian La Roche, <flla@stud.uni-sb.de>
|
|
|
|
* Charles Hedrick, <hedrick@klinzhai.rutgers.edu>
|
|
|
|
* Linus Torvalds, <torvalds@cs.helsinki.fi>
|
|
|
|
* Alan Cox, <gw4pts@gw4pts.ampr.org>
|
|
|
|
* Matthew Dillon, <dillon@apollo.west.oic.com>
|
|
|
|
* Arnt Gulbrandsen, <agulbra@nvg.unit.no>
|
|
|
|
* Jorge Cwik, <jorge@laser.satlink.net>
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Changes:
|
|
|
|
* Pedro Roque : Fast Retransmit/Recovery.
|
|
|
|
* Two receive queues.
|
|
|
|
* Retransmit queue handled by TCP.
|
|
|
|
* Better retransmit timer handling.
|
|
|
|
* New congestion avoidance.
|
|
|
|
* Header prediction.
|
|
|
|
* Variable renaming.
|
|
|
|
*
|
|
|
|
* Eric : Fast Retransmit.
|
|
|
|
* Randy Scott : MSS option defines.
|
|
|
|
* Eric Schenk : Fixes to slow start algorithm.
|
|
|
|
* Eric Schenk : Yet another double ACK bug.
|
|
|
|
* Eric Schenk : Delayed ACK bug fixes.
|
|
|
|
* Eric Schenk : Floyd style fast retrans war avoidance.
|
|
|
|
* David S. Miller : Don't allow zero congestion window.
|
|
|
|
* Eric Schenk : Fix retransmitter so that it sends
|
|
|
|
* next packet on ack of previous packet.
|
|
|
|
* Andi Kleen : Moved open_request checking here
|
|
|
|
* and process RSTs for open_requests.
|
|
|
|
* Andi Kleen : Better prune_queue, and other fixes.
|
2005-11-11 01:13:47 +00:00
|
|
|
* Andrey Savochkin: Fix RTT measurements in the presence of
|
2005-04-16 22:20:36 +00:00
|
|
|
* timestamps.
|
|
|
|
* Andrey Savochkin: Check sequence numbers correctly when
|
|
|
|
* removing SACKs due to in sequence incoming
|
|
|
|
* data segments.
|
|
|
|
* Andi Kleen: Make sure we never ack data there is not
|
|
|
|
* enough room for. Also make this condition
|
|
|
|
* a fatal error if it might still happen.
|
2007-02-09 14:24:47 +00:00
|
|
|
* Andi Kleen: Add tcp_measure_rcv_mss to make
|
2005-04-16 22:20:36 +00:00
|
|
|
* connections with MSS<min(MTU,ann. MSS)
|
2007-02-09 14:24:47 +00:00
|
|
|
* work without delayed acks.
|
2005-04-16 22:20:36 +00:00
|
|
|
* Andi Kleen: Process packets with PSH set in the
|
|
|
|
* fast path.
|
|
|
|
* J Hadi Salim: ECN support
|
|
|
|
* Andrei Gurtov,
|
|
|
|
* Pasi Sarolahti,
|
|
|
|
* Panu Kuhlberg: Experimental audit of TCP (re)transmission
|
|
|
|
* engine. Lots of bugs are found.
|
|
|
|
* Pasi Sarolahti: F-RTO for dealing with spurious RTOs
|
|
|
|
*/
|
|
|
|
|
2012-03-12 07:03:32 +00:00
|
|
|
#define pr_fmt(fmt) "TCP: " fmt
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <linux/mm.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 08:04:11 +00:00
|
|
|
#include <linux/slab.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <linux/module.h>
|
|
|
|
#include <linux/sysctl.h>
|
2009-03-21 20:36:17 +00:00
|
|
|
#include <linux/kernel.h>
|
2008-05-05 05:14:42 +00:00
|
|
|
#include <net/dst.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <net/tcp.h>
|
|
|
|
#include <net/inet_common.h>
|
|
|
|
#include <linux/ipsec.h>
|
|
|
|
#include <asm/unaligned.h>
|
2006-05-24 01:05:53 +00:00
|
|
|
#include <net/netdma.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-09-22 21:15:41 +00:00
|
|
|
int sysctl_tcp_timestamps __read_mostly = 1;
|
|
|
|
int sysctl_tcp_window_scaling __read_mostly = 1;
|
|
|
|
int sysctl_tcp_sack __read_mostly = 1;
|
|
|
|
int sysctl_tcp_fack __read_mostly = 1;
|
|
|
|
int sysctl_tcp_reordering __read_mostly = TCP_FASTRETRANS_THRESH;
|
2010-07-09 21:22:10 +00:00
|
|
|
EXPORT_SYMBOL(sysctl_tcp_reordering);
|
2009-05-04 18:07:36 +00:00
|
|
|
int sysctl_tcp_ecn __read_mostly = 2;
|
2010-07-09 21:22:10 +00:00
|
|
|
EXPORT_SYMBOL(sysctl_tcp_ecn);
|
2006-09-22 21:15:41 +00:00
|
|
|
int sysctl_tcp_dsack __read_mostly = 1;
|
|
|
|
int sysctl_tcp_app_win __read_mostly = 31;
|
2012-05-02 02:28:41 +00:00
|
|
|
int sysctl_tcp_adv_win_scale __read_mostly = 1;
|
2010-07-09 21:22:10 +00:00
|
|
|
EXPORT_SYMBOL(sysctl_tcp_adv_win_scale);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-09-22 21:15:41 +00:00
|
|
|
int sysctl_tcp_stdurg __read_mostly;
|
|
|
|
int sysctl_tcp_rfc1337 __read_mostly;
|
|
|
|
int sysctl_tcp_max_orphans __read_mostly = NR_FILE;
|
[TCP]: Enable SACK enhanced FRTO (RFC4138) by default
Most of the description that follows comes from my mail to
netdev (some editing done):
Main obstacle to FRTO use is its deployment as it has to be on
the sender side where as wireless link is often the receiver's
access link. Take initiative on behalf of unlucky receivers and
enable it by default in future Linux TCP senders. Also IETF
seems to interested in advancing FRTO from experimental [1].
How does FRTO help?
===================
FRTO detects spurious RTOs and avoids a number of unnecessary
retransmissions and a couple of other problems that can arise
due to incorrect guess made at RTO (i.e., that segments were
lost when they actually got delayed which is likely to occur
e.g. in wireless environments with link-layer retransmission).
Though FRTO cannot prevent the first (potentially unnecessary)
retransmission at RTO, I suspect that it won't cost that much
even if you have to pay for each bit (won't be that high
percentage out of all packets after all :-)). However, usually
when you have a spurious RTO, not only the first segment
unnecessarily retransmitted but the *whole window*. It goes like
this: all cumulative ACKs got delayed due to in-order delivery,
then TCP will actually send 1.5*original cwnd worth of data in
the RTO's slow-start when the delayed ACKs arrive (basically the
original cwnd worth of it unnecessarily). In case one is
interested in minimizing unnecessary retransmissions e.g. due to
cost, those rexmissions must never see daylight. Besides, in the
worst case the generated burst overloads the bottleneck buffers
which is likely to significantly delay the further progress of
the flow. In case of ll rexmissions, ACK compression often
occurs at the same time making the burst very "sharp edged" (in
that case TCP often loses most of the segments above high_seq
=> very bad performance too). When FRTO is enabled, those
unnecessary retransmissions are fully avoided except for the
first segment and the cwnd behavior after detected spurious RTO
is determined by the response (one can tune that by sysctl).
Basic version (non-SACK enhanced one), FRTO can fail to detect
spurious RTO as spurious and falls back to conservative
behavior. ACK lossage is much less significant than reordering,
usually the FRTO can detect spurious RTO if at least 2
cumulative ACKs from original window are preserved (excluding
the ACK that advances to high_seq). With SACK-enhanced version,
the detection is quite robust.
FRTO should remove the need to set a high lower bound for the
RTO estimator due to delay spikes that occur relatively common
in some environments (esp. in wireless/cellular ones).
[1] http://www1.ietf.org/mail-archive/web/tcpm/current/msg02862.html
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-09-20 18:36:37 +00:00
|
|
|
int sysctl_tcp_frto __read_mostly = 2;
|
2007-02-27 18:09:49 +00:00
|
|
|
int sysctl_tcp_frto_response __read_mostly;
|
2006-09-22 21:15:41 +00:00
|
|
|
int sysctl_tcp_nometrics_save __read_mostly;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2010-02-18 04:48:19 +00:00
|
|
|
int sysctl_tcp_thin_dupack __read_mostly;
|
|
|
|
|
2006-09-22 21:15:41 +00:00
|
|
|
int sysctl_tcp_moderate_rcvbuf __read_mostly = 1;
|
|
|
|
int sysctl_tcp_abc __read_mostly;
|
2012-05-02 13:30:03 +00:00
|
|
|
int sysctl_tcp_early_retrans __read_mostly = 2;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
#define FLAG_DATA 0x01 /* Incoming frame contained data. */
|
|
|
|
#define FLAG_WIN_UPDATE 0x02 /* Incoming ACK was a window update. */
|
|
|
|
#define FLAG_DATA_ACKED 0x04 /* This ACK acknowledged new data. */
|
|
|
|
#define FLAG_RETRANS_DATA_ACKED 0x08 /* "" "" some of which was retransmitted. */
|
|
|
|
#define FLAG_SYN_ACKED 0x10 /* This ACK acknowledged SYN. */
|
|
|
|
#define FLAG_DATA_SACKED 0x20 /* New SACK. */
|
|
|
|
#define FLAG_ECE 0x40 /* ECE in this ACK */
|
|
|
|
#define FLAG_SLOWPATH 0x100 /* Do not skip RFC checks for window update.*/
|
2007-02-22 07:16:11 +00:00
|
|
|
#define FLAG_ONLY_ORIG_SACKED 0x200 /* SACKs only non-rexmit sent before RTO */
|
2007-08-03 02:46:58 +00:00
|
|
|
#define FLAG_SND_UNA_ADVANCED 0x400 /* Snd_una was changed (!= FLAG_DATA_ACKED) */
|
2007-10-26 06:03:52 +00:00
|
|
|
#define FLAG_DSACKING_ACK 0x800 /* SACK blocks contained D-SACK info */
|
2007-09-20 18:34:38 +00:00
|
|
|
#define FLAG_NONHEAD_RETRANS_ACKED 0x1000 /* Non-head rexmitted data was ACKed */
|
2007-12-31 12:49:21 +00:00
|
|
|
#define FLAG_SACK_RENEGING 0x2000 /* snd_una advanced to a sacked seq */
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
#define FLAG_ACKED (FLAG_DATA_ACKED|FLAG_SYN_ACKED)
|
|
|
|
#define FLAG_NOT_DUP (FLAG_DATA|FLAG_WIN_UPDATE|FLAG_ACKED)
|
|
|
|
#define FLAG_CA_ALERT (FLAG_DATA_SACKED|FLAG_ECE)
|
|
|
|
#define FLAG_FORWARD_PROGRESS (FLAG_ACKED|FLAG_DATA_SACKED)
|
2007-08-03 02:46:58 +00:00
|
|
|
#define FLAG_ANY_PROGRESS (FLAG_FORWARD_PROGRESS|FLAG_SND_UNA_ADVANCED)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
#define TCP_REMNANT (TCP_FLAG_FIN|TCP_FLAG_URG|TCP_FLAG_SYN|TCP_FLAG_PSH)
|
2007-05-27 09:04:16 +00:00
|
|
|
#define TCP_HP_BITS (~(TCP_RESERVED_BITS|TCP_FLAG_PSH))
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-02-09 14:24:47 +00:00
|
|
|
/* Adapt the MSS value used to make delayed ack decision to the
|
2005-04-16 22:20:36 +00:00
|
|
|
* real world.
|
2007-02-09 14:24:47 +00:00
|
|
|
*/
|
2007-12-31 22:57:14 +00:00
|
|
|
static void tcp_measure_rcv_mss(struct sock *sk, const struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 03:10:42 +00:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2007-02-09 14:24:47 +00:00
|
|
|
const unsigned int lss = icsk->icsk_ack.last_seg_size;
|
2005-08-10 03:10:42 +00:00
|
|
|
unsigned int len;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-02-09 14:24:47 +00:00
|
|
|
icsk->icsk_ack.last_seg_size = 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* skb->len may jitter because of SACKs, even if peer
|
|
|
|
* sends good full-sized frames.
|
|
|
|
*/
|
2007-12-31 22:57:14 +00:00
|
|
|
len = skb_shinfo(skb)->gso_size ? : skb->len;
|
2005-08-10 03:10:42 +00:00
|
|
|
if (len >= icsk->icsk_ack.rcv_mss) {
|
|
|
|
icsk->icsk_ack.rcv_mss = len;
|
2005-04-16 22:20:36 +00:00
|
|
|
} else {
|
|
|
|
/* Otherwise, we make more careful check taking into account,
|
|
|
|
* that SACKs block is variable.
|
|
|
|
*
|
|
|
|
* "len" is invariant segment length, including TCP header.
|
|
|
|
*/
|
2007-04-26 01:04:18 +00:00
|
|
|
len += skb->data - skb_transport_header(skb);
|
2009-11-10 09:51:18 +00:00
|
|
|
if (len >= TCP_MSS_DEFAULT + sizeof(struct tcphdr) ||
|
2005-04-16 22:20:36 +00:00
|
|
|
/* If PSH is not set, packet should be
|
|
|
|
* full sized, provided peer TCP is not badly broken.
|
|
|
|
* This observation (if it is correct 8)) allows
|
|
|
|
* to handle super-low mtu links fairly.
|
|
|
|
*/
|
|
|
|
(len >= TCP_MIN_MSS + sizeof(struct tcphdr) &&
|
2007-04-11 04:04:22 +00:00
|
|
|
!(tcp_flag_word(tcp_hdr(skb)) & TCP_REMNANT))) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Subtract also invariant (if peer is RFC compliant),
|
|
|
|
* tcp header plus fixed timestamp option length.
|
|
|
|
* Resulting "len" is MSS free of SACK jitter.
|
|
|
|
*/
|
2005-08-10 03:10:42 +00:00
|
|
|
len -= tcp_sk(sk)->tcp_header_len;
|
|
|
|
icsk->icsk_ack.last_seg_size = len;
|
2005-04-16 22:20:36 +00:00
|
|
|
if (len == lss) {
|
2005-08-10 03:10:42 +00:00
|
|
|
icsk->icsk_ack.rcv_mss = len;
|
2005-04-16 22:20:36 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
2006-09-19 19:52:50 +00:00
|
|
|
if (icsk->icsk_ack.pending & ICSK_ACK_PUSHED)
|
|
|
|
icsk->icsk_ack.pending |= ICSK_ACK_PUSHED2;
|
2005-08-10 03:10:42 +00:00
|
|
|
icsk->icsk_ack.pending |= ICSK_ACK_PUSHED;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
static void tcp_incr_quickack(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 03:10:42 +00:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2012-04-15 05:58:06 +00:00
|
|
|
unsigned int quickacks = tcp_sk(sk)->rcv_wnd / (2 * icsk->icsk_ack.rcv_mss);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
if (quickacks == 0)
|
|
|
|
quickacks = 2;
|
2005-08-10 03:10:42 +00:00
|
|
|
if (quickacks > icsk->icsk_ack.quick)
|
|
|
|
icsk->icsk_ack.quick = min(quickacks, TCP_MAX_QUICKACKS);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2010-09-28 19:30:14 +00:00
|
|
|
static void tcp_enter_quickack_mode(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 03:10:42 +00:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
tcp_incr_quickack(sk);
|
|
|
|
icsk->icsk_ack.pingpong = 0;
|
|
|
|
icsk->icsk_ack.ato = TCP_ATO_MIN;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Send ACKs quickly, if "quick" count is not exhausted
|
|
|
|
* and the session is not interactive.
|
|
|
|
*/
|
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
static inline bool tcp_in_quickack_mode(const struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 03:10:42 +00:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2012-05-16 23:15:34 +00:00
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
return icsk->icsk_ack.quick && !icsk->icsk_ack.pingpong;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-05-27 09:04:16 +00:00
|
|
|
static inline void TCP_ECN_queue_cwr(struct tcp_sock *tp)
|
|
|
|
{
|
2007-12-31 22:57:14 +00:00
|
|
|
if (tp->ecn_flags & TCP_ECN_OK)
|
2007-05-27 09:04:16 +00:00
|
|
|
tp->ecn_flags |= TCP_ECN_QUEUE_CWR;
|
|
|
|
}
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static inline void TCP_ECN_accept_cwr(struct tcp_sock *tp, const struct sk_buff *skb)
|
2007-05-27 09:04:16 +00:00
|
|
|
{
|
|
|
|
if (tcp_hdr(skb)->cwr)
|
|
|
|
tp->ecn_flags &= ~TCP_ECN_DEMAND_CWR;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void TCP_ECN_withdraw_cwr(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
tp->ecn_flags &= ~TCP_ECN_DEMAND_CWR;
|
|
|
|
}
|
|
|
|
|
2011-09-22 20:02:19 +00:00
|
|
|
static inline void TCP_ECN_check_ce(struct tcp_sock *tp, const struct sk_buff *skb)
|
2007-05-27 09:04:16 +00:00
|
|
|
{
|
2011-09-22 20:02:19 +00:00
|
|
|
if (!(tp->ecn_flags & TCP_ECN_OK))
|
|
|
|
return;
|
|
|
|
|
2011-09-27 06:20:08 +00:00
|
|
|
switch (TCP_SKB_CB(skb)->ip_dsfield & INET_ECN_MASK) {
|
2011-09-22 20:02:19 +00:00
|
|
|
case INET_ECN_NOT_ECT:
|
2007-05-27 09:04:16 +00:00
|
|
|
/* Funny extension: if ECT is not set on a segment,
|
2011-09-22 20:02:19 +00:00
|
|
|
* and we already seen ECT on a previous segment,
|
|
|
|
* it is probably a retransmit.
|
|
|
|
*/
|
|
|
|
if (tp->ecn_flags & TCP_ECN_SEEN)
|
2007-05-27 09:04:16 +00:00
|
|
|
tcp_enter_quickack_mode((struct sock *)tp);
|
2011-09-22 20:02:19 +00:00
|
|
|
break;
|
|
|
|
case INET_ECN_CE:
|
|
|
|
tp->ecn_flags |= TCP_ECN_DEMAND_CWR;
|
|
|
|
/* fallinto */
|
|
|
|
default:
|
|
|
|
tp->ecn_flags |= TCP_ECN_SEEN;
|
2007-05-27 09:04:16 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static inline void TCP_ECN_rcv_synack(struct tcp_sock *tp, const struct tcphdr *th)
|
2007-05-27 09:04:16 +00:00
|
|
|
{
|
2007-12-31 22:57:14 +00:00
|
|
|
if ((tp->ecn_flags & TCP_ECN_OK) && (!th->ece || th->cwr))
|
2007-05-27 09:04:16 +00:00
|
|
|
tp->ecn_flags &= ~TCP_ECN_OK;
|
|
|
|
}
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static inline void TCP_ECN_rcv_syn(struct tcp_sock *tp, const struct tcphdr *th)
|
2007-05-27 09:04:16 +00:00
|
|
|
{
|
2007-12-31 22:57:14 +00:00
|
|
|
if ((tp->ecn_flags & TCP_ECN_OK) && (!th->ece || !th->cwr))
|
2007-05-27 09:04:16 +00:00
|
|
|
tp->ecn_flags &= ~TCP_ECN_OK;
|
|
|
|
}
|
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool TCP_ECN_rcv_ecn_echo(const struct tcp_sock *tp, const struct tcphdr *th)
|
2007-05-27 09:04:16 +00:00
|
|
|
{
|
2007-12-31 22:57:14 +00:00
|
|
|
if (th->ece && !th->syn && (tp->ecn_flags & TCP_ECN_OK))
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
|
|
|
return false;
|
2007-05-27 09:04:16 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Buffer size and advertised window tuning.
|
|
|
|
*
|
|
|
|
* 1. Tuning sk->sk_sndbuf, when connection enters established state.
|
|
|
|
*/
|
|
|
|
|
|
|
|
static void tcp_fixup_sndbuf(struct sock *sk)
|
|
|
|
{
|
2011-10-13 07:28:54 +00:00
|
|
|
int sndmem = SKB_TRUESIZE(tcp_sk(sk)->rx_opt.mss_clamp + MAX_TCP_HEADER);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-10-13 18:24:42 +00:00
|
|
|
sndmem *= TCP_INIT_CWND;
|
|
|
|
if (sk->sk_sndbuf < sndmem)
|
|
|
|
sk->sk_sndbuf = min(sndmem, sysctl_tcp_wmem[2]);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* 2. Tuning advertised window (window_clamp, rcv_ssthresh)
|
|
|
|
*
|
|
|
|
* All tcp_full_space() is split to two parts: "network" buffer, allocated
|
|
|
|
* forward and advertised in receiver window (tp->rcv_wnd) and
|
|
|
|
* "application buffer", required to isolate scheduling/application
|
|
|
|
* latencies from network.
|
|
|
|
* window_clamp is maximal advertised window. It can be less than
|
|
|
|
* tcp_full_space(), in this case tcp_full_space() - window_clamp
|
|
|
|
* is reserved for "application" buffer. The less window_clamp is
|
|
|
|
* the smoother our behaviour from viewpoint of network, but the lower
|
|
|
|
* throughput and the higher sensitivity of the connection to losses. 8)
|
|
|
|
*
|
|
|
|
* rcv_ssthresh is more strict window_clamp used at "slow start"
|
|
|
|
* phase to predict further behaviour of this connection.
|
|
|
|
* It is used for two goals:
|
|
|
|
* - to enforce header prediction at sender, even when application
|
|
|
|
* requires some significant "application buffer". It is check #1.
|
|
|
|
* - to prevent pruning of receive queue because of misprediction
|
|
|
|
* of receiver window. Check #2.
|
|
|
|
*
|
|
|
|
* The scheme does not work when sender sends good segments opening
|
2005-11-11 01:13:47 +00:00
|
|
|
* window and then starts to feed us spaghetti. But it should work
|
2005-04-16 22:20:36 +00:00
|
|
|
* in common situations. Otherwise, we have to rely on queue collapsing.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* Slow part of check#2. */
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
static int __tcp_grow_window(const struct sock *sk, const struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Optimize this! */
|
2007-12-21 14:07:53 +00:00
|
|
|
int truesize = tcp_win_from_space(skb->truesize) >> 1;
|
|
|
|
int window = tcp_win_from_space(sysctl_tcp_rmem[2]) >> 1;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
while (tp->rcv_ssthresh <= window) {
|
|
|
|
if (truesize <= skb->len)
|
2005-08-10 03:10:42 +00:00
|
|
|
return 2 * inet_csk(sk)->icsk_ack.rcv_mss;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
truesize >>= 1;
|
|
|
|
window >>= 1;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Check #1 */
|
|
|
|
if (tp->rcv_ssthresh < tp->window_clamp &&
|
|
|
|
(int)tp->rcv_ssthresh < tcp_space(sk) &&
|
2011-12-11 21:47:02 +00:00
|
|
|
!sk_under_memory_pressure(sk)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
int incr;
|
|
|
|
|
|
|
|
/* Check #2. Increase window, if skb with such overhead
|
|
|
|
* will fit to rcvbuf in future.
|
|
|
|
*/
|
|
|
|
if (tcp_win_from_space(skb->truesize) <= skb->len)
|
2007-12-31 22:57:14 +00:00
|
|
|
incr = 2 * tp->advmss;
|
2005-04-16 22:20:36 +00:00
|
|
|
else
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
incr = __tcp_grow_window(sk, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (incr) {
|
2012-04-16 23:28:07 +00:00
|
|
|
incr = max_t(int, incr, 2 * skb->len);
|
2007-12-31 22:57:14 +00:00
|
|
|
tp->rcv_ssthresh = min(tp->rcv_ssthresh + incr,
|
|
|
|
tp->window_clamp);
|
2005-08-10 03:10:42 +00:00
|
|
|
inet_csk(sk)->icsk_ack.quick |= 1;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* 3. Tuning rcvbuf, when connection enters established state. */
|
|
|
|
|
|
|
|
static void tcp_fixup_rcvbuf(struct sock *sk)
|
|
|
|
{
|
2011-10-20 20:53:56 +00:00
|
|
|
u32 mss = tcp_sk(sk)->advmss;
|
|
|
|
u32 icwnd = TCP_DEFAULT_INIT_RCVWND;
|
|
|
|
int rcvmem;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-10-20 20:53:56 +00:00
|
|
|
/* Limit to 10 segments if mss <= 1460,
|
|
|
|
* or 14600/mss segments, with a minimum of two segments.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2011-10-20 20:53:56 +00:00
|
|
|
if (mss > 1460)
|
|
|
|
icwnd = max_t(u32, (1460 * TCP_DEFAULT_INIT_RCVWND) / mss, 2);
|
|
|
|
|
|
|
|
rcvmem = SKB_TRUESIZE(mss + MAX_TCP_HEADER);
|
|
|
|
while (tcp_win_from_space(rcvmem) < mss)
|
2005-04-16 22:20:36 +00:00
|
|
|
rcvmem += 128;
|
2011-10-20 20:53:56 +00:00
|
|
|
|
|
|
|
rcvmem *= icwnd;
|
|
|
|
|
|
|
|
if (sk->sk_rcvbuf < rcvmem)
|
|
|
|
sk->sk_rcvbuf = min(rcvmem, sysctl_tcp_rmem[2]);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2005-11-11 01:13:47 +00:00
|
|
|
/* 4. Try to fixup all. It is made immediately after connection enters
|
2005-04-16 22:20:36 +00:00
|
|
|
* established state.
|
|
|
|
*/
|
|
|
|
static void tcp_init_buffer_space(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int maxwin;
|
|
|
|
|
|
|
|
if (!(sk->sk_userlocks & SOCK_RCVBUF_LOCK))
|
|
|
|
tcp_fixup_rcvbuf(sk);
|
|
|
|
if (!(sk->sk_userlocks & SOCK_SNDBUF_LOCK))
|
|
|
|
tcp_fixup_sndbuf(sk);
|
|
|
|
|
|
|
|
tp->rcvq_space.space = tp->rcv_wnd;
|
|
|
|
|
|
|
|
maxwin = tcp_full_space(sk);
|
|
|
|
|
|
|
|
if (tp->window_clamp >= maxwin) {
|
|
|
|
tp->window_clamp = maxwin;
|
|
|
|
|
|
|
|
if (sysctl_tcp_app_win && maxwin > 4 * tp->advmss)
|
|
|
|
tp->window_clamp = max(maxwin -
|
|
|
|
(maxwin >> sysctl_tcp_app_win),
|
|
|
|
4 * tp->advmss);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Force reservation of one segment. */
|
|
|
|
if (sysctl_tcp_app_win &&
|
|
|
|
tp->window_clamp > 2 * tp->advmss &&
|
|
|
|
tp->window_clamp + tp->advmss > maxwin)
|
|
|
|
tp->window_clamp = max(2 * tp->advmss, maxwin - tp->advmss);
|
|
|
|
|
|
|
|
tp->rcv_ssthresh = min(tp->rcv_ssthresh, tp->window_clamp);
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* 5. Recalculate window clamp after socket hit its memory bounds. */
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
static void tcp_clamp_window(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-08-10 07:03:31 +00:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
icsk->icsk_ack.quick = 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2005-11-11 01:11:48 +00:00
|
|
|
if (sk->sk_rcvbuf < sysctl_tcp_rmem[2] &&
|
|
|
|
!(sk->sk_userlocks & SOCK_RCVBUF_LOCK) &&
|
2011-12-11 21:47:02 +00:00
|
|
|
!sk_under_memory_pressure(sk) &&
|
|
|
|
sk_memory_allocated(sk) < sk_prot_mem_limits(sk, 0)) {
|
2005-11-11 01:11:48 +00:00
|
|
|
sk->sk_rcvbuf = min(atomic_read(&sk->sk_rmem_alloc),
|
|
|
|
sysctl_tcp_rmem[2]);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2005-11-11 01:11:48 +00:00
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf)
|
2007-12-31 22:57:14 +00:00
|
|
|
tp->rcv_ssthresh = min(tp->window_clamp, 2U * tp->advmss);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2006-01-04 00:03:49 +00:00
|
|
|
/* Initialize RCV_MSS value.
|
|
|
|
* RCV_MSS is an our guess about MSS used by the peer.
|
|
|
|
* We haven't any direct information about the MSS.
|
|
|
|
* It's better to underestimate the RCV_MSS rather than overestimate.
|
|
|
|
* Overestimations make us ACKing less frequently than needed.
|
|
|
|
* Underestimations are more easy to detect and fix by tcp_measure_rcv_mss().
|
|
|
|
*/
|
|
|
|
void tcp_initialize_rcv_mss(struct sock *sk)
|
|
|
|
{
|
2011-10-21 09:22:42 +00:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2006-01-04 00:03:49 +00:00
|
|
|
unsigned int hint = min_t(unsigned int, tp->advmss, tp->mss_cache);
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
hint = min(hint, tp->rcv_wnd / 2);
|
2009-11-10 09:51:18 +00:00
|
|
|
hint = min(hint, TCP_MSS_DEFAULT);
|
2006-01-04 00:03:49 +00:00
|
|
|
hint = max(hint, TCP_MIN_MSS);
|
|
|
|
|
|
|
|
inet_csk(sk)->icsk_ack.rcv_mss = hint;
|
|
|
|
}
|
2010-07-09 21:22:10 +00:00
|
|
|
EXPORT_SYMBOL(tcp_initialize_rcv_mss);
|
2006-01-04 00:03:49 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Receiver "autotuning" code.
|
|
|
|
*
|
|
|
|
* The algorithm for RTT estimation w/o timestamps is based on
|
|
|
|
* Dynamic Right-Sizing (DRS) by Wu Feng and Mike Fisk of LANL.
|
2010-10-18 09:03:14 +00:00
|
|
|
* <http://public.lanl.gov/radiant/pubs.html#DRS>
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
|
|
|
* More detail on this code can be found at
|
2010-10-18 09:03:14 +00:00
|
|
|
* <http://staff.psc.edu/jheffner/>,
|
2005-04-16 22:20:36 +00:00
|
|
|
* though this reference is out of date. A new paper
|
|
|
|
* is pending.
|
|
|
|
*/
|
|
|
|
static void tcp_rcv_rtt_update(struct tcp_sock *tp, u32 sample, int win_dep)
|
|
|
|
{
|
|
|
|
u32 new_sample = tp->rcv_rtt_est.rtt;
|
|
|
|
long m = sample;
|
|
|
|
|
|
|
|
if (m == 0)
|
|
|
|
m = 1;
|
|
|
|
|
|
|
|
if (new_sample != 0) {
|
|
|
|
/* If we sample in larger samples in the non-timestamp
|
|
|
|
* case, we could grossly overestimate the RTT especially
|
|
|
|
* with chatty applications or bulk transfer apps which
|
|
|
|
* are stalled on filesystem I/O.
|
|
|
|
*
|
|
|
|
* Also, since we are only going for a minimum in the
|
2005-11-15 23:17:10 +00:00
|
|
|
* non-timestamp case, we do not smooth things out
|
2005-11-11 01:13:47 +00:00
|
|
|
* else with timestamps disabled convergence takes too
|
2005-04-16 22:20:36 +00:00
|
|
|
* long.
|
|
|
|
*/
|
|
|
|
if (!win_dep) {
|
|
|
|
m -= (new_sample >> 3);
|
|
|
|
new_sample += m;
|
2012-04-10 07:59:20 +00:00
|
|
|
} else {
|
|
|
|
m <<= 3;
|
|
|
|
if (m < new_sample)
|
|
|
|
new_sample = m;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
} else {
|
2005-11-11 01:13:47 +00:00
|
|
|
/* No previous measure. */
|
2005-04-16 22:20:36 +00:00
|
|
|
new_sample = m << 3;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (tp->rcv_rtt_est.rtt != new_sample)
|
|
|
|
tp->rcv_rtt_est.rtt = new_sample;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void tcp_rcv_rtt_measure(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
if (tp->rcv_rtt_est.time == 0)
|
|
|
|
goto new_measure;
|
|
|
|
if (before(tp->rcv_nxt, tp->rcv_rtt_est.seq))
|
|
|
|
return;
|
2012-04-27 15:29:37 +00:00
|
|
|
tcp_rcv_rtt_update(tp, tcp_time_stamp - tp->rcv_rtt_est.time, 1);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
new_measure:
|
|
|
|
tp->rcv_rtt_est.seq = tp->rcv_nxt + tp->rcv_wnd;
|
|
|
|
tp->rcv_rtt_est.time = tcp_time_stamp;
|
|
|
|
}
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
static inline void tcp_rcv_rtt_measure_ts(struct sock *sk,
|
|
|
|
const struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 03:10:42 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (tp->rx_opt.rcv_tsecr &&
|
|
|
|
(TCP_SKB_CB(skb)->end_seq -
|
2005-08-10 03:10:42 +00:00
|
|
|
TCP_SKB_CB(skb)->seq >= inet_csk(sk)->icsk_ack.rcv_mss))
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_rcv_rtt_update(tp, tcp_time_stamp - tp->rx_opt.rcv_tsecr, 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This function should be called every time data is copied to user space.
|
|
|
|
* It calculates the appropriate TCP receive buffer space.
|
|
|
|
*/
|
|
|
|
void tcp_rcv_space_adjust(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int time;
|
|
|
|
int space;
|
2007-02-09 14:24:47 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
if (tp->rcvq_space.time == 0)
|
|
|
|
goto new_measure;
|
2007-02-09 14:24:47 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
time = tcp_time_stamp - tp->rcvq_space.time;
|
2007-12-31 22:57:14 +00:00
|
|
|
if (time < (tp->rcv_rtt_est.rtt >> 3) || tp->rcv_rtt_est.rtt == 0)
|
2005-04-16 22:20:36 +00:00
|
|
|
return;
|
2007-02-09 14:24:47 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
space = 2 * (tp->copied_seq - tp->rcvq_space.seq);
|
|
|
|
|
|
|
|
space = max(tp->rcvq_space.space, space);
|
|
|
|
|
|
|
|
if (tp->rcvq_space.space != space) {
|
|
|
|
int rcvmem;
|
|
|
|
|
|
|
|
tp->rcvq_space.space = space;
|
|
|
|
|
2006-02-10 01:06:57 +00:00
|
|
|
if (sysctl_tcp_moderate_rcvbuf &&
|
|
|
|
!(sk->sk_userlocks & SOCK_RCVBUF_LOCK)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
int new_clamp = space;
|
|
|
|
|
|
|
|
/* Receive space grows, normalize in order to
|
|
|
|
* take into account packet headers and sk_buff
|
|
|
|
* structure overhead.
|
|
|
|
*/
|
|
|
|
space /= tp->advmss;
|
|
|
|
if (!space)
|
|
|
|
space = 1;
|
2011-10-13 07:28:54 +00:00
|
|
|
rcvmem = SKB_TRUESIZE(tp->advmss + MAX_TCP_HEADER);
|
2005-04-16 22:20:36 +00:00
|
|
|
while (tcp_win_from_space(rcvmem) < tp->advmss)
|
|
|
|
rcvmem += 128;
|
|
|
|
space *= rcvmem;
|
|
|
|
space = min(space, sysctl_tcp_rmem[2]);
|
|
|
|
if (space > sk->sk_rcvbuf) {
|
|
|
|
sk->sk_rcvbuf = space;
|
|
|
|
|
|
|
|
/* Make the window clamp follow along. */
|
|
|
|
tp->window_clamp = new_clamp;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2007-02-09 14:24:47 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
new_measure:
|
|
|
|
tp->rcvq_space.seq = tp->copied_seq;
|
|
|
|
tp->rcvq_space.time = tcp_time_stamp;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* There is something which you must keep in mind when you analyze the
|
|
|
|
* behavior of the tp->ato delayed ack timeout interval. When a
|
|
|
|
* connection starts up, we want to ack as quickly as possible. The
|
|
|
|
* problem is that "good" TCP's do slow start at the beginning of data
|
|
|
|
* transmission. The means that until we send the first few ACK's the
|
|
|
|
* sender will sit on his end and only queue most of his data, because
|
|
|
|
* he can only send snd_cwnd unacked packets at any given time. For
|
|
|
|
* each ACK we send, he increments snd_cwnd and transmits more of his
|
|
|
|
* queue. -DaveM
|
|
|
|
*/
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
static void tcp_event_data_recv(struct sock *sk, struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-08-10 03:10:42 +00:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
u32 now;
|
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
inet_csk_schedule_ack(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_measure_rcv_mss(sk, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
tcp_rcv_rtt_measure(tp);
|
2007-02-09 14:24:47 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
now = tcp_time_stamp;
|
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
if (!icsk->icsk_ack.ato) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* The _first_ data packet received, initialize
|
|
|
|
* delayed ACK engine.
|
|
|
|
*/
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_incr_quickack(sk);
|
|
|
|
icsk->icsk_ack.ato = TCP_ATO_MIN;
|
2005-04-16 22:20:36 +00:00
|
|
|
} else {
|
2005-08-10 03:10:42 +00:00
|
|
|
int m = now - icsk->icsk_ack.lrcvtime;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
if (m <= TCP_ATO_MIN / 2) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* The fastest case is the first. */
|
2005-08-10 03:10:42 +00:00
|
|
|
icsk->icsk_ack.ato = (icsk->icsk_ack.ato >> 1) + TCP_ATO_MIN / 2;
|
|
|
|
} else if (m < icsk->icsk_ack.ato) {
|
|
|
|
icsk->icsk_ack.ato = (icsk->icsk_ack.ato >> 1) + m;
|
|
|
|
if (icsk->icsk_ack.ato > icsk->icsk_rto)
|
|
|
|
icsk->icsk_ack.ato = icsk->icsk_rto;
|
|
|
|
} else if (m > icsk->icsk_rto) {
|
2005-11-11 01:13:47 +00:00
|
|
|
/* Too long gap. Apparently sender failed to
|
2005-04-16 22:20:36 +00:00
|
|
|
* restart window, so that we send ACKs quickly.
|
|
|
|
*/
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_incr_quickack(sk);
|
2007-12-31 08:11:19 +00:00
|
|
|
sk_mem_reclaim(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
2005-08-10 03:10:42 +00:00
|
|
|
icsk->icsk_ack.lrcvtime = now;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
TCP_ECN_check_ce(tp, skb);
|
|
|
|
|
|
|
|
if (skb->len >= 128)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_grow_window(sk, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Called to compute a smoothed rtt estimate. The data fed to this
|
|
|
|
* routine either comes from timestamps, or from segments that were
|
|
|
|
* known _not_ to have been retransmitted [see Karn/Partridge
|
|
|
|
* Proceedings SIGCOMM 87]. The algorithm is from the SIGCOMM 88
|
|
|
|
* piece by Van Jacobson.
|
|
|
|
* NOTE: the next three routines used to be one big routine.
|
|
|
|
* To save cycles in the RFC 1323 implementation it was better to break
|
|
|
|
* it up into three procedures. -- erics
|
|
|
|
*/
|
2005-11-11 00:56:12 +00:00
|
|
|
static void tcp_rtt_estimator(struct sock *sk, const __u32 mrtt)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
long m = mrtt; /* RTT */
|
|
|
|
|
|
|
|
/* The following amusing code comes from Jacobson's
|
|
|
|
* article in SIGCOMM '88. Note that rtt and mdev
|
|
|
|
* are scaled versions of rtt and mean deviation.
|
2007-02-09 14:24:47 +00:00
|
|
|
* This is designed to be as fast as possible
|
2005-04-16 22:20:36 +00:00
|
|
|
* m stands for "measurement".
|
|
|
|
*
|
|
|
|
* On a 1990 paper the rto value is changed to:
|
|
|
|
* RTO = rtt + 4 * mdev
|
|
|
|
*
|
|
|
|
* Funny. This algorithm seems to be very broken.
|
|
|
|
* These formulae increase RTO, when it should be decreased, increase
|
2005-11-15 23:17:10 +00:00
|
|
|
* too slowly, when it should be increased quickly, decrease too quickly
|
2005-04-16 22:20:36 +00:00
|
|
|
* etc. I guess in BSD RTO takes ONE value, so that it is absolutely
|
|
|
|
* does not matter how to _calculate_ it. Seems, it was trap
|
|
|
|
* that VJ failed to avoid. 8)
|
|
|
|
*/
|
2007-03-09 04:45:19 +00:00
|
|
|
if (m == 0)
|
2005-04-16 22:20:36 +00:00
|
|
|
m = 1;
|
|
|
|
if (tp->srtt != 0) {
|
|
|
|
m -= (tp->srtt >> 3); /* m is now error in rtt est */
|
|
|
|
tp->srtt += m; /* rtt = 7/8 rtt + 1/8 new */
|
|
|
|
if (m < 0) {
|
|
|
|
m = -m; /* m is now abs(error) */
|
|
|
|
m -= (tp->mdev >> 2); /* similar update on mdev */
|
|
|
|
/* This is similar to one of Eifel findings.
|
|
|
|
* Eifel blocks mdev updates when rtt decreases.
|
|
|
|
* This solution is a bit different: we use finer gain
|
|
|
|
* for mdev in this case (alpha*beta).
|
|
|
|
* Like Eifel it also prevents growth of rto,
|
|
|
|
* but also it limits too fast rto decreases,
|
|
|
|
* happening in pure Eifel.
|
|
|
|
*/
|
|
|
|
if (m > 0)
|
|
|
|
m >>= 3;
|
|
|
|
} else {
|
|
|
|
m -= (tp->mdev >> 2); /* similar update on mdev */
|
|
|
|
}
|
|
|
|
tp->mdev += m; /* mdev = 3/4 mdev + 1/4 new */
|
|
|
|
if (tp->mdev > tp->mdev_max) {
|
|
|
|
tp->mdev_max = tp->mdev;
|
|
|
|
if (tp->mdev_max > tp->rttvar)
|
|
|
|
tp->rttvar = tp->mdev_max;
|
|
|
|
}
|
|
|
|
if (after(tp->snd_una, tp->rtt_seq)) {
|
|
|
|
if (tp->mdev_max < tp->rttvar)
|
2007-12-31 22:57:14 +00:00
|
|
|
tp->rttvar -= (tp->rttvar - tp->mdev_max) >> 2;
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->rtt_seq = tp->snd_nxt;
|
2007-08-31 05:10:28 +00:00
|
|
|
tp->mdev_max = tcp_rto_min(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
} else {
|
|
|
|
/* no previous measure. */
|
2007-12-31 22:57:14 +00:00
|
|
|
tp->srtt = m << 3; /* take the measured time to be rtt */
|
|
|
|
tp->mdev = m << 1; /* make sure rto = 3*rtt */
|
2007-08-31 05:10:28 +00:00
|
|
|
tp->mdev_max = tp->rttvar = max(tp->mdev, tcp_rto_min(sk));
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->rtt_seq = tp->snd_nxt;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Calculate rto without backoff. This is the second half of Van Jacobson's
|
|
|
|
* routine referred to above.
|
|
|
|
*/
|
2005-08-10 03:10:42 +00:00
|
|
|
static inline void tcp_set_rto(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 03:10:42 +00:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Old crap is replaced with new one. 8)
|
|
|
|
*
|
|
|
|
* More seriously:
|
|
|
|
* 1. If rtt variance happened to be less 50msec, it is hallucination.
|
|
|
|
* It cannot be less due to utterly erratic ACK generation made
|
|
|
|
* at least by solaris and freebsd. "Erratic ACKs" has _nothing_
|
|
|
|
* to do with delayed acks, because at cwnd>2 true delack timeout
|
|
|
|
* is invisible. Actually, Linux-2.4 also generates erratic
|
2005-11-11 01:13:47 +00:00
|
|
|
* ACKs in some circumstances.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
Revert Backoff [v3]: Revert RTO on ICMP destination unreachable
Here, an ICMP host/network unreachable message, whose payload fits to
TCP's SND.UNA, is taken as an indication that the RTO retransmission has
not been lost due to congestion, but because of a route failure
somewhere along the path.
With true congestion, a router won't trigger such a message and the
patched TCP will operate as standard TCP.
This patch reverts one RTO backoff, if an ICMP host/network unreachable
message, whose payload fits to TCP's SND.UNA, arrives.
Based on the new RTO, the retransmission timer is reset to reflect the
remaining time, or - if the revert clocked out the timer - a retransmission
is sent out immediately.
Backoffs are only reverted, if TCP is in RTO loss recovery, i.e. if
there have been retransmissions and reversible backoffs, already.
Changes from v2:
1) Renaming of skb in tcp_v4_err() moved to another patch.
2) Reintroduced tcp_bound_rto() and __tcp_set_rto().
3) Fixed code comments.
Signed-off-by: Damian Lukowski <damian@tvk.rwth-aachen.de>
Acked-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-08-26 00:16:31 +00:00
|
|
|
inet_csk(sk)->icsk_rto = __tcp_set_rto(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* 2. Fixups made earlier cannot be right.
|
|
|
|
* If we do not estimate RTO correctly without them,
|
|
|
|
* all the algo is pure shit and should be replaced
|
2005-11-11 01:13:47 +00:00
|
|
|
* with correct one. It is exactly, which we pretend to do.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
|
2008-12-06 06:43:08 +00:00
|
|
|
/* NOTE: clamping at TCP_RTO_MIN is not required, current algo
|
|
|
|
* guarantees that rto is higher.
|
|
|
|
*/
|
Revert Backoff [v3]: Revert RTO on ICMP destination unreachable
Here, an ICMP host/network unreachable message, whose payload fits to
TCP's SND.UNA, is taken as an indication that the RTO retransmission has
not been lost due to congestion, but because of a route failure
somewhere along the path.
With true congestion, a router won't trigger such a message and the
patched TCP will operate as standard TCP.
This patch reverts one RTO backoff, if an ICMP host/network unreachable
message, whose payload fits to TCP's SND.UNA, arrives.
Based on the new RTO, the retransmission timer is reset to reflect the
remaining time, or - if the revert clocked out the timer - a retransmission
is sent out immediately.
Backoffs are only reverted, if TCP is in RTO loss recovery, i.e. if
there have been retransmissions and reversible backoffs, already.
Changes from v2:
1) Renaming of skb in tcp_v4_err() moved to another patch.
2) Reintroduced tcp_bound_rto() and __tcp_set_rto().
3) Fixed code comments.
Signed-off-by: Damian Lukowski <damian@tvk.rwth-aachen.de>
Acked-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-08-26 00:16:31 +00:00
|
|
|
tcp_bound_rto(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Save metrics learned by this TCP session.
|
|
|
|
This function is called only, when TCP finishes successfully
|
|
|
|
i.e. when it enters TIME-WAIT or goes from LAST-ACK to CLOSE.
|
|
|
|
*/
|
|
|
|
void tcp_update_metrics(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct dst_entry *dst = __sk_dst_get(sk);
|
|
|
|
|
|
|
|
if (sysctl_tcp_nometrics_save)
|
|
|
|
return;
|
|
|
|
|
|
|
|
dst_confirm(dst);
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
if (dst && (dst->flags & DST_HOST)) {
|
2005-08-10 07:03:31 +00:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
int m;
|
2008-07-19 06:02:15 +00:00
|
|
|
unsigned long rtt;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
if (icsk->icsk_backoff || !tp->srtt) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* This session failed to estimate rtt. Why?
|
|
|
|
* Probably, no packets returned in time.
|
|
|
|
* Reset our results.
|
|
|
|
*/
|
|
|
|
if (!(dst_metric_locked(dst, RTAX_RTT)))
|
2010-12-09 05:16:57 +00:00
|
|
|
dst_metric_set(dst, RTAX_RTT, 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2008-07-19 06:02:15 +00:00
|
|
|
rtt = dst_metric_rtt(dst, RTAX_RTT);
|
|
|
|
m = rtt - tp->srtt;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* If newly calculated rtt larger than stored one,
|
|
|
|
* store new one. Otherwise, use EWMA. Remember,
|
|
|
|
* rtt overestimation is always better than underestimation.
|
|
|
|
*/
|
|
|
|
if (!(dst_metric_locked(dst, RTAX_RTT))) {
|
|
|
|
if (m <= 0)
|
2008-07-19 06:02:15 +00:00
|
|
|
set_dst_metric_rtt(dst, RTAX_RTT, tp->srtt);
|
2005-04-16 22:20:36 +00:00
|
|
|
else
|
2008-07-19 06:02:15 +00:00
|
|
|
set_dst_metric_rtt(dst, RTAX_RTT, rtt - (m >> 3));
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (!(dst_metric_locked(dst, RTAX_RTTVAR))) {
|
2008-07-19 06:02:15 +00:00
|
|
|
unsigned long var;
|
2005-04-16 22:20:36 +00:00
|
|
|
if (m < 0)
|
|
|
|
m = -m;
|
|
|
|
|
|
|
|
/* Scale deviation to rttvar fixed point */
|
|
|
|
m >>= 1;
|
|
|
|
if (m < tp->mdev)
|
|
|
|
m = tp->mdev;
|
|
|
|
|
2008-07-19 06:02:15 +00:00
|
|
|
var = dst_metric_rtt(dst, RTAX_RTTVAR);
|
|
|
|
if (m >= var)
|
|
|
|
var = m;
|
2005-04-16 22:20:36 +00:00
|
|
|
else
|
2008-07-19 06:02:15 +00:00
|
|
|
var -= (var - m) >> 2;
|
|
|
|
|
|
|
|
set_dst_metric_rtt(dst, RTAX_RTTVAR, var);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2009-09-15 08:30:10 +00:00
|
|
|
if (tcp_in_initial_slowstart(tp)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Slow start still did not finish. */
|
|
|
|
if (dst_metric(dst, RTAX_SSTHRESH) &&
|
|
|
|
!dst_metric_locked(dst, RTAX_SSTHRESH) &&
|
|
|
|
(tp->snd_cwnd >> 1) > dst_metric(dst, RTAX_SSTHRESH))
|
2010-12-09 05:16:57 +00:00
|
|
|
dst_metric_set(dst, RTAX_SSTHRESH, tp->snd_cwnd >> 1);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (!dst_metric_locked(dst, RTAX_CWND) &&
|
|
|
|
tp->snd_cwnd > dst_metric(dst, RTAX_CWND))
|
2010-12-09 05:16:57 +00:00
|
|
|
dst_metric_set(dst, RTAX_CWND, tp->snd_cwnd);
|
2005-04-16 22:20:36 +00:00
|
|
|
} else if (tp->snd_cwnd > tp->snd_ssthresh &&
|
2005-08-10 07:03:31 +00:00
|
|
|
icsk->icsk_ca_state == TCP_CA_Open) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Cong. avoidance phase, cwnd is reliable. */
|
|
|
|
if (!dst_metric_locked(dst, RTAX_SSTHRESH))
|
2010-12-09 05:16:57 +00:00
|
|
|
dst_metric_set(dst, RTAX_SSTHRESH,
|
|
|
|
max(tp->snd_cwnd >> 1, tp->snd_ssthresh));
|
2005-04-16 22:20:36 +00:00
|
|
|
if (!dst_metric_locked(dst, RTAX_CWND))
|
2010-12-09 05:16:57 +00:00
|
|
|
dst_metric_set(dst, RTAX_CWND,
|
|
|
|
(dst_metric(dst, RTAX_CWND) +
|
|
|
|
tp->snd_cwnd) >> 1);
|
2005-04-16 22:20:36 +00:00
|
|
|
} else {
|
|
|
|
/* Else slow start did not finish, cwnd is non-sense,
|
|
|
|
ssthresh may be also invalid.
|
|
|
|
*/
|
|
|
|
if (!dst_metric_locked(dst, RTAX_CWND))
|
2010-12-09 05:16:57 +00:00
|
|
|
dst_metric_set(dst, RTAX_CWND,
|
|
|
|
(dst_metric(dst, RTAX_CWND) +
|
|
|
|
tp->snd_ssthresh) >> 1);
|
2008-05-05 05:14:42 +00:00
|
|
|
if (dst_metric(dst, RTAX_SSTHRESH) &&
|
2005-04-16 22:20:36 +00:00
|
|
|
!dst_metric_locked(dst, RTAX_SSTHRESH) &&
|
2008-05-05 05:14:42 +00:00
|
|
|
tp->snd_ssthresh > dst_metric(dst, RTAX_SSTHRESH))
|
2010-12-09 05:16:57 +00:00
|
|
|
dst_metric_set(dst, RTAX_SSTHRESH, tp->snd_ssthresh);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (!dst_metric_locked(dst, RTAX_REORDERING)) {
|
2008-05-05 05:14:42 +00:00
|
|
|
if (dst_metric(dst, RTAX_REORDERING) < tp->reordering &&
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->reordering != sysctl_tcp_reordering)
|
2010-12-09 05:16:57 +00:00
|
|
|
dst_metric_set(dst, RTAX_REORDERING, tp->reordering);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
__u32 tcp_init_cwnd(const struct tcp_sock *tp, const struct dst_entry *dst)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
__u32 cwnd = (dst ? dst_metric(dst, RTAX_INITCWND) : 0);
|
|
|
|
|
2010-08-29 19:23:12 +00:00
|
|
|
if (!cwnd)
|
2011-02-03 01:05:11 +00:00
|
|
|
cwnd = TCP_INIT_CWND;
|
2005-04-16 22:20:36 +00:00
|
|
|
return min_t(__u32, cwnd, tp->snd_cwnd_clamp);
|
|
|
|
}
|
|
|
|
|
2006-01-04 00:03:49 +00:00
|
|
|
/* Set slow start threshold and cwnd not falling to slow start */
|
2007-02-27 18:09:49 +00:00
|
|
|
void tcp_enter_cwr(struct sock *sk, const int set_ssthresh)
|
2006-01-04 00:03:49 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2007-02-27 18:09:49 +00:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2006-01-04 00:03:49 +00:00
|
|
|
|
|
|
|
tp->prior_ssthresh = 0;
|
|
|
|
tp->bytes_acked = 0;
|
2007-03-02 21:27:25 +00:00
|
|
|
if (icsk->icsk_ca_state < TCP_CA_CWR) {
|
2006-01-04 00:03:49 +00:00
|
|
|
tp->undo_marker = 0;
|
2007-02-27 18:09:49 +00:00
|
|
|
if (set_ssthresh)
|
|
|
|
tp->snd_ssthresh = icsk->icsk_ca_ops->ssthresh(sk);
|
2006-01-04 00:03:49 +00:00
|
|
|
tp->snd_cwnd = min(tp->snd_cwnd,
|
|
|
|
tcp_packets_in_flight(tp) + 1U);
|
|
|
|
tp->snd_cwnd_cnt = 0;
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
TCP_ECN_queue_cwr(tp);
|
|
|
|
|
|
|
|
tcp_set_ca_state(sk, TCP_CA_CWR);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2007-08-09 12:14:46 +00:00
|
|
|
/*
|
|
|
|
* Packet counting of FACK is based on in-order assumptions, therefore TCP
|
|
|
|
* disables it when reordering is detected
|
|
|
|
*/
|
|
|
|
static void tcp_disable_fack(struct tcp_sock *tp)
|
|
|
|
{
|
2007-11-16 03:39:31 +00:00
|
|
|
/* RFC3517 uses different metric in lost marker => reset on change */
|
|
|
|
if (tcp_is_fack(tp))
|
|
|
|
tp->lost_skb_hint = NULL;
|
2011-12-20 13:23:24 +00:00
|
|
|
tp->rx_opt.sack_ok &= ~TCP_FACK_ENABLED;
|
2007-08-09 12:14:46 +00:00
|
|
|
}
|
|
|
|
|
2007-10-26 06:03:52 +00:00
|
|
|
/* Take a notice that peer is sending D-SACKs */
|
2007-08-09 12:14:46 +00:00
|
|
|
static void tcp_dsack_seen(struct tcp_sock *tp)
|
|
|
|
{
|
2011-12-20 13:23:24 +00:00
|
|
|
tp->rx_opt.sack_ok |= TCP_DSACK_SEEN;
|
2007-08-09 12:14:46 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Initialize metrics on socket. */
|
|
|
|
|
|
|
|
static void tcp_init_metrics(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct dst_entry *dst = __sk_dst_get(sk);
|
|
|
|
|
|
|
|
if (dst == NULL)
|
|
|
|
goto reset;
|
|
|
|
|
|
|
|
dst_confirm(dst);
|
|
|
|
|
|
|
|
if (dst_metric_locked(dst, RTAX_CWND))
|
|
|
|
tp->snd_cwnd_clamp = dst_metric(dst, RTAX_CWND);
|
|
|
|
if (dst_metric(dst, RTAX_SSTHRESH)) {
|
|
|
|
tp->snd_ssthresh = dst_metric(dst, RTAX_SSTHRESH);
|
|
|
|
if (tp->snd_ssthresh > tp->snd_cwnd_clamp)
|
|
|
|
tp->snd_ssthresh = tp->snd_cwnd_clamp;
|
2011-06-08 11:08:38 +00:00
|
|
|
} else {
|
|
|
|
/* ssthresh may have been reduced unnecessarily during.
|
|
|
|
* 3WHS. Restore it back to its initial default.
|
|
|
|
*/
|
|
|
|
tp->snd_ssthresh = TCP_INFINITE_SSTHRESH;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
if (dst_metric(dst, RTAX_REORDERING) &&
|
|
|
|
tp->reordering != dst_metric(dst, RTAX_REORDERING)) {
|
2007-08-09 12:14:46 +00:00
|
|
|
tcp_disable_fack(tp);
|
2012-05-02 13:30:03 +00:00
|
|
|
tcp_disable_early_retrans(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->reordering = dst_metric(dst, RTAX_REORDERING);
|
|
|
|
}
|
|
|
|
|
2011-06-08 11:08:38 +00:00
|
|
|
if (dst_metric(dst, RTAX_RTT) == 0 || tp->srtt == 0)
|
2005-04-16 22:20:36 +00:00
|
|
|
goto reset;
|
|
|
|
|
|
|
|
/* Initial rtt is determined from SYN,SYN-ACK.
|
|
|
|
* The segment is small and rtt may appear much
|
|
|
|
* less than real one. Use per-dst memory
|
|
|
|
* to make it more realistic.
|
|
|
|
*
|
|
|
|
* A bit of theory. RTT is time passed after "normal" sized packet
|
2005-11-11 01:13:47 +00:00
|
|
|
* is sent until it is ACKed. In normal circumstances sending small
|
2005-04-16 22:20:36 +00:00
|
|
|
* packets force peer to delay ACKs and calculation is correct too.
|
|
|
|
* The algorithm is adaptive and, provided we follow specs, it
|
|
|
|
* NEVER underestimate RTT. BUT! If peer tries to make some clever
|
|
|
|
* tricks sort of "quick acks" for time long enough to decrease RTT
|
|
|
|
* to low value, and then abruptly stops to do it and starts to delay
|
|
|
|
* ACKs, wait for troubles.
|
|
|
|
*/
|
2008-07-19 06:02:15 +00:00
|
|
|
if (dst_metric_rtt(dst, RTAX_RTT) > tp->srtt) {
|
|
|
|
tp->srtt = dst_metric_rtt(dst, RTAX_RTT);
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->rtt_seq = tp->snd_nxt;
|
|
|
|
}
|
2008-07-19 06:02:15 +00:00
|
|
|
if (dst_metric_rtt(dst, RTAX_RTTVAR) > tp->mdev) {
|
|
|
|
tp->mdev = dst_metric_rtt(dst, RTAX_RTTVAR);
|
2007-12-16 22:00:19 +00:00
|
|
|
tp->mdev_max = tp->rttvar = max(tp->mdev, tcp_rto_min(sk));
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_set_rto(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
reset:
|
2011-06-08 11:08:38 +00:00
|
|
|
if (tp->srtt == 0) {
|
2012-04-12 19:48:40 +00:00
|
|
|
/* RFC6298: 5.7 We've failed to get a valid RTT sample from
|
2011-06-08 11:08:38 +00:00
|
|
|
* 3WHS. This is most likely due to retransmission,
|
|
|
|
* including spurious one. Reset the RTO back to 3secs
|
|
|
|
* from the more aggressive 1sec to avoid more spurious
|
|
|
|
* retransmission.
|
2010-12-22 23:23:38 +00:00
|
|
|
*/
|
2011-06-08 11:08:38 +00:00
|
|
|
tp->mdev = tp->mdev_max = tp->rttvar = TCP_TIMEOUT_FALLBACK;
|
|
|
|
inet_csk(sk)->icsk_rto = TCP_TIMEOUT_FALLBACK;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2011-06-08 11:08:38 +00:00
|
|
|
/* Cut cwnd down to 1 per RFC5681 if SYN or SYN-ACK has been
|
2012-04-12 19:48:40 +00:00
|
|
|
* retransmitted. In light of RFC6298 more aggressive 1sec
|
2011-06-08 11:08:38 +00:00
|
|
|
* initRTO, we only reset cwnd when more than 1 SYN/SYN-ACK
|
|
|
|
* retransmission has occurred.
|
|
|
|
*/
|
|
|
|
if (tp->total_retrans > 1)
|
|
|
|
tp->snd_cwnd = 1;
|
|
|
|
else
|
|
|
|
tp->snd_cwnd = tcp_init_cwnd(tp, dst);
|
2010-12-22 23:23:38 +00:00
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
static void tcp_update_reordering(struct sock *sk, const int metric,
|
|
|
|
const int ts)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (metric > tp->reordering) {
|
2008-07-03 08:05:41 +00:00
|
|
|
int mib_idx;
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->reordering = min(TCP_MAX_REORDERING, metric);
|
|
|
|
|
|
|
|
/* This exciting event is worth to be remembered. 8) */
|
|
|
|
if (ts)
|
2008-07-03 08:05:41 +00:00
|
|
|
mib_idx = LINUX_MIB_TCPTSREORDER;
|
2007-08-09 12:14:46 +00:00
|
|
|
else if (tcp_is_reno(tp))
|
2008-07-03 08:05:41 +00:00
|
|
|
mib_idx = LINUX_MIB_TCPRENOREORDER;
|
2007-08-09 12:14:46 +00:00
|
|
|
else if (tcp_is_fack(tp))
|
2008-07-03 08:05:41 +00:00
|
|
|
mib_idx = LINUX_MIB_TCPFACKREORDER;
|
2005-04-16 22:20:36 +00:00
|
|
|
else
|
2008-07-03 08:05:41 +00:00
|
|
|
mib_idx = LINUX_MIB_TCPSACKREORDER;
|
|
|
|
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), mib_idx);
|
2005-04-16 22:20:36 +00:00
|
|
|
#if FASTRETRANS_DEBUG > 1
|
2012-05-15 14:11:54 +00:00
|
|
|
pr_debug("Disorder%d %d %u f%u s%u rr%d\n",
|
|
|
|
tp->rx_opt.sack_ok, inet_csk(sk)->icsk_ca_state,
|
|
|
|
tp->reordering,
|
|
|
|
tp->fackets_out,
|
|
|
|
tp->sacked_out,
|
|
|
|
tp->undo_marker ? tp->undo_retrans : 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif
|
2007-08-09 12:14:46 +00:00
|
|
|
tcp_disable_fack(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2012-05-02 13:30:03 +00:00
|
|
|
|
|
|
|
if (metric > 0)
|
|
|
|
tcp_disable_early_retrans(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2008-09-21 04:20:20 +00:00
|
|
|
/* This must be called before lost_out is incremented */
|
2008-09-21 04:18:55 +00:00
|
|
|
static void tcp_verify_retransmit_hint(struct tcp_sock *tp, struct sk_buff *skb)
|
|
|
|
{
|
2008-09-21 04:20:20 +00:00
|
|
|
if ((tp->retransmit_skb_hint == NULL) ||
|
2008-09-21 04:18:55 +00:00
|
|
|
before(TCP_SKB_CB(skb)->seq,
|
|
|
|
TCP_SKB_CB(tp->retransmit_skb_hint)->seq))
|
2008-09-21 04:20:20 +00:00
|
|
|
tp->retransmit_skb_hint = skb;
|
|
|
|
|
|
|
|
if (!tp->lost_out ||
|
|
|
|
after(TCP_SKB_CB(skb)->end_seq, tp->retransmit_high))
|
|
|
|
tp->retransmit_high = TCP_SKB_CB(skb)->end_seq;
|
2008-09-21 04:18:55 +00:00
|
|
|
}
|
|
|
|
|
2008-09-21 04:19:22 +00:00
|
|
|
static void tcp_skb_mark_lost(struct tcp_sock *tp, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
if (!(TCP_SKB_CB(skb)->sacked & (TCPCB_LOST|TCPCB_SACKED_ACKED))) {
|
|
|
|
tcp_verify_retransmit_hint(tp, skb);
|
|
|
|
|
|
|
|
tp->lost_out += tcp_skb_pcount(skb);
|
|
|
|
TCP_SKB_CB(skb)->sacked |= TCPCB_LOST;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-11-25 05:11:55 +00:00
|
|
|
static void tcp_skb_mark_lost_uncond_verify(struct tcp_sock *tp,
|
|
|
|
struct sk_buff *skb)
|
2008-09-21 04:20:20 +00:00
|
|
|
{
|
|
|
|
tcp_verify_retransmit_hint(tp, skb);
|
|
|
|
|
|
|
|
if (!(TCP_SKB_CB(skb)->sacked & (TCPCB_LOST|TCPCB_SACKED_ACKED))) {
|
|
|
|
tp->lost_out += tcp_skb_pcount(skb);
|
|
|
|
TCP_SKB_CB(skb)->sacked |= TCPCB_LOST;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* This procedure tags the retransmission queue when SACKs arrive.
|
|
|
|
*
|
|
|
|
* We have three tag bits: SACKED(S), RETRANS(R) and LOST(L).
|
|
|
|
* Packets in queue with these bits set are counted in variables
|
|
|
|
* sacked_out, retrans_out and lost_out, correspondingly.
|
|
|
|
*
|
|
|
|
* Valid combinations are:
|
|
|
|
* Tag InFlight Description
|
|
|
|
* 0 1 - orig segment is in flight.
|
|
|
|
* S 0 - nothing flies, orig reached receiver.
|
|
|
|
* L 0 - nothing flies, orig lost by net.
|
|
|
|
* R 2 - both orig and retransmit are in flight.
|
|
|
|
* L|R 1 - orig is lost, retransmit is in flight.
|
|
|
|
* S|R 1 - orig reached receiver, retrans is still in flight.
|
|
|
|
* (L|S|R is logically valid, it could occur when L|R is sacked,
|
|
|
|
* but it is equivalent to plain S and code short-curcuits it to S.
|
|
|
|
* L|S is logically invalid, it would mean -1 packet in flight 8))
|
|
|
|
*
|
|
|
|
* These 6 states form finite state machine, controlled by the following events:
|
|
|
|
* 1. New ACK (+SACK) arrives. (tcp_sacktag_write_queue())
|
|
|
|
* 2. Retransmission. (tcp_retransmit_skb(), tcp_xmit_retransmit_queue())
|
2012-01-19 14:42:21 +00:00
|
|
|
* 3. Loss detection event of two flavors:
|
2005-04-16 22:20:36 +00:00
|
|
|
* A. Scoreboard estimator decided the packet is lost.
|
|
|
|
* A'. Reno "three dupacks" marks head of queue lost.
|
2012-01-19 14:42:21 +00:00
|
|
|
* A''. Its FACK modification, head until snd.fack is lost.
|
|
|
|
* B. SACK arrives sacking SND.NXT at the moment, when the
|
2005-04-16 22:20:36 +00:00
|
|
|
* segment was retransmitted.
|
|
|
|
* 4. D-SACK added new rule: D-SACK changes any tag to S.
|
|
|
|
*
|
|
|
|
* It is pleasant to note, that state diagram turns out to be commutative,
|
|
|
|
* so that we are allowed not to be bothered by order of our actions,
|
|
|
|
* when multiple events arrive simultaneously. (see the function below).
|
|
|
|
*
|
|
|
|
* Reordering detection.
|
|
|
|
* --------------------
|
|
|
|
* Reordering metric is maximal distance, which a packet can be displaced
|
|
|
|
* in packet stream. With SACKs we can estimate it:
|
|
|
|
*
|
|
|
|
* 1. SACK fills old hole and the corresponding segment was not
|
|
|
|
* ever retransmitted -> reordering. Alas, we cannot use it
|
|
|
|
* when segment was retransmitted.
|
|
|
|
* 2. The last flaw is solved with D-SACK. D-SACK arrives
|
|
|
|
* for retransmitted and already SACKed segment -> reordering..
|
|
|
|
* Both of these heuristics are not used in Loss state, when we cannot
|
|
|
|
* account for retransmits accurately.
|
2007-08-25 05:54:44 +00:00
|
|
|
*
|
|
|
|
* SACK block validation.
|
|
|
|
* ----------------------
|
|
|
|
*
|
|
|
|
* SACK block range validation checks that the received SACK block fits to
|
|
|
|
* the expected sequence limits, i.e., it is between SND.UNA and SND.NXT.
|
|
|
|
* Note that SND.UNA is not included to the range though being valid because
|
2007-10-01 22:28:17 +00:00
|
|
|
* it means that the receiver is rather inconsistent with itself reporting
|
|
|
|
* SACK reneging when it should advance SND.UNA. Such SACK block this is
|
|
|
|
* perfectly valid, however, in light of RFC2018 which explicitly states
|
|
|
|
* that "SACK block MUST reflect the newest segment. Even if the newest
|
|
|
|
* segment is going to be discarded ...", not that it looks very clever
|
|
|
|
* in case of head skb. Due to potentional receiver driven attacks, we
|
|
|
|
* choose to avoid immediate execution of a walk in write queue due to
|
|
|
|
* reneging and defer head skb's loss recovery to standard loss recovery
|
|
|
|
* procedure that will eventually trigger (nothing forbids us doing this).
|
2007-08-25 05:54:44 +00:00
|
|
|
*
|
|
|
|
* Implements also blockage to start_seq wrap-around. Problem lies in the
|
|
|
|
* fact that though start_seq (s) is before end_seq (i.e., not reversed),
|
|
|
|
* there's no guarantee that it will be before snd_nxt (n). The problem
|
|
|
|
* happens when start_seq resides between end_seq wrap (e_w) and snd_nxt
|
|
|
|
* wrap (s_w):
|
|
|
|
*
|
|
|
|
* <- outs wnd -> <- wrapzone ->
|
|
|
|
* u e n u_w e_w s n_w
|
|
|
|
* | | | | | | |
|
|
|
|
* |<------------+------+----- TCP seqno space --------------+---------->|
|
|
|
|
* ...-- <2^31 ->| |<--------...
|
|
|
|
* ...---- >2^31 ------>| |<--------...
|
|
|
|
*
|
|
|
|
* Current code wouldn't be vulnerable but it's better still to discard such
|
|
|
|
* crazy SACK blocks. Doing this check for start_seq alone closes somewhat
|
|
|
|
* similar case (end_seq after snd_nxt wrap) as earlier reversed check in
|
|
|
|
* snd_nxt wrap -> snd_una region will then become "well defined", i.e.,
|
|
|
|
* equal to the ideal case (infinite seqno space without wrap caused issues).
|
|
|
|
*
|
|
|
|
* With D-SACK the lower bound is extended to cover sequence space below
|
|
|
|
* SND.UNA down to undo_marker, which is the last point of interest. Yet
|
2007-10-26 06:03:52 +00:00
|
|
|
* again, D-SACK block must not to go across snd_una (for the same reason as
|
2007-08-25 05:54:44 +00:00
|
|
|
* for the normal SACK blocks, explained above). But there all simplicity
|
|
|
|
* ends, TCP might receive valid D-SACKs below that. As long as they reside
|
|
|
|
* fully below undo_marker they do not affect behavior in anyway and can
|
|
|
|
* therefore be safely ignored. In rare cases (which are more or less
|
|
|
|
* theoretical ones), the D-SACK will nicely cross that boundary due to skb
|
|
|
|
* fragmentation and packet reordering past skb's retransmission. To consider
|
|
|
|
* them correctly, the acceptable range must be extended even more though
|
|
|
|
* the exact amount is rather hard to quantify. However, tp->max_window can
|
|
|
|
* be used as an exaggerated estimate.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_is_sackblock_valid(struct tcp_sock *tp, bool is_dsack,
|
|
|
|
u32 start_seq, u32 end_seq)
|
2007-08-25 05:54:44 +00:00
|
|
|
{
|
|
|
|
/* Too far in future, or reversed (interpretation is ambiguous) */
|
|
|
|
if (after(end_seq, tp->snd_nxt) || !before(start_seq, end_seq))
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2007-08-25 05:54:44 +00:00
|
|
|
|
|
|
|
/* Nasty start_seq wrap-around check (see comments above) */
|
|
|
|
if (!before(start_seq, tp->snd_nxt))
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2007-08-25 05:54:44 +00:00
|
|
|
|
2007-10-26 06:03:52 +00:00
|
|
|
/* In outstanding window? ...This is valid exit for D-SACKs too.
|
2007-08-25 05:54:44 +00:00
|
|
|
* start_seq == snd_una is non-sensical (see comments above)
|
|
|
|
*/
|
|
|
|
if (after(start_seq, tp->snd_una))
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-08-25 05:54:44 +00:00
|
|
|
|
|
|
|
if (!is_dsack || !tp->undo_marker)
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2007-08-25 05:54:44 +00:00
|
|
|
|
|
|
|
/* ...Then it's D-SACK, and must reside below snd_una completely */
|
2011-09-19 02:37:34 +00:00
|
|
|
if (after(end_seq, tp->snd_una))
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2007-08-25 05:54:44 +00:00
|
|
|
|
|
|
|
if (!before(start_seq, tp->undo_marker))
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-08-25 05:54:44 +00:00
|
|
|
|
|
|
|
/* Too old */
|
|
|
|
if (!after(end_seq, tp->undo_marker))
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2007-08-25 05:54:44 +00:00
|
|
|
|
|
|
|
/* Undo_marker boundary crossing (overestimates a lot). Known already:
|
|
|
|
* start_seq < undo_marker and end_seq >= undo_marker.
|
|
|
|
*/
|
|
|
|
return !before(start_seq, end_seq - tp->max_window);
|
|
|
|
}
|
|
|
|
|
2007-10-10 09:45:32 +00:00
|
|
|
/* Check for lost retransmit. This superb idea is borrowed from "ratehalving".
|
2012-01-19 14:42:21 +00:00
|
|
|
* Event "B". Later note: FACK people cheated me again 8), we have to account
|
2007-10-10 09:45:32 +00:00
|
|
|
* for reordering! Ugly, but should help.
|
2007-10-12 00:35:41 +00:00
|
|
|
*
|
|
|
|
* Search retransmitted skbs from write_queue that were sent when snd_nxt was
|
|
|
|
* less than what is now known to be received by the other end (derived from
|
2007-11-16 03:42:54 +00:00
|
|
|
* highest SACK block). Also calculate the lowest snd_nxt among the remaining
|
|
|
|
* retransmitted skbs to avoid some costly processing per ACKs.
|
2007-10-10 09:45:32 +00:00
|
|
|
*/
|
2007-12-01 22:47:57 +00:00
|
|
|
static void tcp_mark_lost_retrans(struct sock *sk)
|
2007-10-10 09:45:32 +00:00
|
|
|
{
|
2007-11-16 03:42:54 +00:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2007-10-10 09:45:32 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb;
|
2007-10-12 00:35:41 +00:00
|
|
|
int cnt = 0;
|
2007-10-18 12:07:57 +00:00
|
|
|
u32 new_low_seq = tp->snd_nxt;
|
2007-12-01 22:48:06 +00:00
|
|
|
u32 received_upto = tcp_highest_sack_seq(tp);
|
2007-11-16 03:42:54 +00:00
|
|
|
|
|
|
|
if (!tcp_is_fack(tp) || !tp->retrans_out ||
|
|
|
|
!after(received_upto, tp->lost_retrans_low) ||
|
|
|
|
icsk->icsk_ca_state != TCP_CA_Recovery)
|
2007-12-01 22:47:57 +00:00
|
|
|
return;
|
2007-10-10 09:45:32 +00:00
|
|
|
|
|
|
|
tcp_for_write_queue(skb, sk) {
|
|
|
|
u32 ack_seq = TCP_SKB_CB(skb)->ack_seq;
|
|
|
|
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
2007-10-12 00:35:41 +00:00
|
|
|
if (cnt == tp->retrans_out)
|
2007-10-10 09:45:32 +00:00
|
|
|
break;
|
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, tp->snd_una))
|
|
|
|
continue;
|
|
|
|
|
2007-10-12 00:35:41 +00:00
|
|
|
if (!(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_RETRANS))
|
|
|
|
continue;
|
|
|
|
|
2009-02-28 04:44:32 +00:00
|
|
|
/* TODO: We would like to get rid of tcp_is_fack(tp) only
|
|
|
|
* constraint here (see above) but figuring out that at
|
|
|
|
* least tp->reordering SACK blocks reside between ack_seq
|
|
|
|
* and received_upto is not easy task to do cheaply with
|
|
|
|
* the available datastructures.
|
|
|
|
*
|
|
|
|
* Whether FACK should check here for tp->reordering segs
|
|
|
|
* in-between one could argue for either way (it would be
|
|
|
|
* rather simple to implement as we could count fack_count
|
|
|
|
* during the walk and do tp->fackets_out - fack_count).
|
|
|
|
*/
|
|
|
|
if (after(received_upto, ack_seq)) {
|
2007-10-10 09:45:32 +00:00
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_SACKED_RETRANS;
|
|
|
|
tp->retrans_out -= tcp_skb_pcount(skb);
|
|
|
|
|
2008-09-21 04:20:20 +00:00
|
|
|
tcp_skb_mark_lost_uncond_verify(tp, skb);
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPLOSTRETRANSMIT);
|
2007-10-12 00:35:41 +00:00
|
|
|
} else {
|
2007-10-18 12:07:57 +00:00
|
|
|
if (before(ack_seq, new_low_seq))
|
2007-10-12 00:36:13 +00:00
|
|
|
new_low_seq = ack_seq;
|
2007-10-12 00:35:41 +00:00
|
|
|
cnt += tcp_skb_pcount(skb);
|
2007-10-10 09:45:32 +00:00
|
|
|
}
|
|
|
|
}
|
2007-10-12 00:36:13 +00:00
|
|
|
|
|
|
|
if (tp->retrans_out)
|
|
|
|
tp->lost_retrans_low = new_low_seq;
|
2007-10-10 09:45:32 +00:00
|
|
|
}
|
2007-08-25 05:54:44 +00:00
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_check_dsack(struct sock *sk, const struct sk_buff *ack_skb,
|
|
|
|
struct tcp_sack_block_wire *sp, int num_sacks,
|
|
|
|
u32 prior_snd_una)
|
2007-06-19 05:43:06 +00:00
|
|
|
{
|
2008-07-17 03:29:51 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2008-05-02 23:26:16 +00:00
|
|
|
u32 start_seq_0 = get_unaligned_be32(&sp[0].start_seq);
|
|
|
|
u32 end_seq_0 = get_unaligned_be32(&sp[0].end_seq);
|
2012-05-16 23:15:34 +00:00
|
|
|
bool dup_sack = false;
|
2007-06-19 05:43:06 +00:00
|
|
|
|
|
|
|
if (before(start_seq_0, TCP_SKB_CB(ack_skb)->ack_seq)) {
|
2012-05-16 23:15:34 +00:00
|
|
|
dup_sack = true;
|
2007-08-09 12:14:46 +00:00
|
|
|
tcp_dsack_seen(tp);
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPDSACKRECV);
|
2007-06-19 05:43:06 +00:00
|
|
|
} else if (num_sacks > 1) {
|
2008-05-02 23:26:16 +00:00
|
|
|
u32 end_seq_1 = get_unaligned_be32(&sp[1].end_seq);
|
|
|
|
u32 start_seq_1 = get_unaligned_be32(&sp[1].start_seq);
|
2007-06-19 05:43:06 +00:00
|
|
|
|
|
|
|
if (!after(end_seq_0, end_seq_1) &&
|
|
|
|
!before(start_seq_0, start_seq_1)) {
|
2012-05-16 23:15:34 +00:00
|
|
|
dup_sack = true;
|
2007-08-09 12:14:46 +00:00
|
|
|
tcp_dsack_seen(tp);
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk),
|
|
|
|
LINUX_MIB_TCPDSACKOFORECV);
|
2007-06-19 05:43:06 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* D-SACK for already forgotten data... Do dumb counting. */
|
2011-02-07 12:57:04 +00:00
|
|
|
if (dup_sack && tp->undo_marker && tp->undo_retrans &&
|
2007-06-19 05:43:06 +00:00
|
|
|
!after(end_seq_0, prior_snd_una) &&
|
|
|
|
after(end_seq_0, tp->undo_marker))
|
|
|
|
tp->undo_retrans--;
|
|
|
|
|
|
|
|
return dup_sack;
|
|
|
|
}
|
|
|
|
|
2008-12-06 06:42:22 +00:00
|
|
|
struct tcp_sacktag_state {
|
|
|
|
int reord;
|
|
|
|
int fack_count;
|
|
|
|
int flag;
|
|
|
|
};
|
|
|
|
|
2007-10-12 00:34:25 +00:00
|
|
|
/* Check if skb is fully within the SACK block. In presence of GSO skbs,
|
|
|
|
* the incoming SACK may not exactly match but we can find smaller MSS
|
|
|
|
* aligned portion of it that matches. Therefore we might need to fragment
|
|
|
|
* which may fail and creates some hassle (caller must handle error case
|
|
|
|
* returns).
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
*
|
|
|
|
* FIXME: this could be merged to shift decision code
|
2007-10-12 00:34:25 +00:00
|
|
|
*/
|
2007-10-26 10:57:36 +00:00
|
|
|
static int tcp_match_skb_to_sack(struct sock *sk, struct sk_buff *skb,
|
2012-05-16 23:15:34 +00:00
|
|
|
u32 start_seq, u32 end_seq)
|
2007-10-12 00:34:25 +00:00
|
|
|
{
|
2012-05-16 23:15:34 +00:00
|
|
|
int err;
|
|
|
|
bool in_sack;
|
2007-10-12 00:34:25 +00:00
|
|
|
unsigned int pkt_len;
|
2008-11-25 05:13:50 +00:00
|
|
|
unsigned int mss;
|
2007-10-12 00:34:25 +00:00
|
|
|
|
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq) &&
|
|
|
|
!before(end_seq, TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
|
|
|
if (tcp_skb_pcount(skb) > 1 && !in_sack &&
|
|
|
|
after(TCP_SKB_CB(skb)->end_seq, start_seq)) {
|
2008-11-25 05:13:50 +00:00
|
|
|
mss = tcp_skb_mss(skb);
|
2007-10-12 00:34:25 +00:00
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq);
|
|
|
|
|
2008-11-25 05:13:50 +00:00
|
|
|
if (!in_sack) {
|
2007-10-12 00:34:25 +00:00
|
|
|
pkt_len = start_seq - TCP_SKB_CB(skb)->seq;
|
2008-11-25 05:13:50 +00:00
|
|
|
if (pkt_len < mss)
|
|
|
|
pkt_len = mss;
|
|
|
|
} else {
|
2007-10-12 00:34:25 +00:00
|
|
|
pkt_len = end_seq - TCP_SKB_CB(skb)->seq;
|
2008-11-25 05:13:50 +00:00
|
|
|
if (pkt_len < mss)
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Round if necessary so that SACKs cover only full MSSes
|
|
|
|
* and/or the remaining small portion (if present)
|
|
|
|
*/
|
|
|
|
if (pkt_len > mss) {
|
|
|
|
unsigned int new_len = (pkt_len / mss) * mss;
|
|
|
|
if (!in_sack && new_len < pkt_len) {
|
|
|
|
new_len += mss;
|
|
|
|
if (new_len > skb->len)
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
pkt_len = new_len;
|
|
|
|
}
|
|
|
|
err = tcp_fragment(sk, skb, pkt_len, mss);
|
2007-10-12 00:34:25 +00:00
|
|
|
if (err < 0)
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
return in_sack;
|
|
|
|
}
|
|
|
|
|
2012-02-12 18:37:09 +00:00
|
|
|
/* Mark the given newly-SACKed range as such, adjusting counters and hints. */
|
|
|
|
static u8 tcp_sacktag_one(struct sock *sk,
|
|
|
|
struct tcp_sacktag_state *state, u8 sacked,
|
|
|
|
u32 start_seq, u32 end_seq,
|
2012-05-16 23:15:34 +00:00
|
|
|
bool dup_sack, int pcount)
|
2007-11-16 03:44:56 +00:00
|
|
|
{
|
2007-12-01 22:48:06 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2008-12-06 06:42:22 +00:00
|
|
|
int fack_count = state->fack_count;
|
2007-11-16 03:44:56 +00:00
|
|
|
|
|
|
|
/* Account D-SACK for retransmitted packet. */
|
|
|
|
if (dup_sack && (sacked & TCPCB_RETRANS)) {
|
2011-02-07 12:57:04 +00:00
|
|
|
if (tp->undo_marker && tp->undo_retrans &&
|
2012-02-12 18:37:09 +00:00
|
|
|
after(end_seq, tp->undo_marker))
|
2007-11-16 03:44:56 +00:00
|
|
|
tp->undo_retrans--;
|
2007-12-01 22:47:58 +00:00
|
|
|
if (sacked & TCPCB_SACKED_ACKED)
|
2008-12-06 06:42:22 +00:00
|
|
|
state->reord = min(fack_count, state->reord);
|
2007-11-16 03:44:56 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Nothing to do; acked frame is about to be dropped (was ACKed). */
|
2012-02-12 18:37:09 +00:00
|
|
|
if (!after(end_seq, tp->snd_una))
|
2008-12-06 06:42:22 +00:00
|
|
|
return sacked;
|
2007-11-16 03:44:56 +00:00
|
|
|
|
|
|
|
if (!(sacked & TCPCB_SACKED_ACKED)) {
|
|
|
|
if (sacked & TCPCB_SACKED_RETRANS) {
|
|
|
|
/* If the segment is not tagged as lost,
|
|
|
|
* we do not clear RETRANS, believing
|
|
|
|
* that retransmission is still in flight.
|
|
|
|
*/
|
|
|
|
if (sacked & TCPCB_LOST) {
|
2008-12-06 06:42:22 +00:00
|
|
|
sacked &= ~(TCPCB_LOST|TCPCB_SACKED_RETRANS);
|
2008-11-25 05:14:43 +00:00
|
|
|
tp->lost_out -= pcount;
|
|
|
|
tp->retrans_out -= pcount;
|
2007-11-16 03:44:56 +00:00
|
|
|
}
|
|
|
|
} else {
|
|
|
|
if (!(sacked & TCPCB_RETRANS)) {
|
|
|
|
/* New sack for not retransmitted frame,
|
|
|
|
* which was in hole. It is reordering.
|
|
|
|
*/
|
2012-02-12 18:37:09 +00:00
|
|
|
if (before(start_seq,
|
2007-11-16 03:44:56 +00:00
|
|
|
tcp_highest_sack_seq(tp)))
|
2008-12-06 06:42:22 +00:00
|
|
|
state->reord = min(fack_count,
|
|
|
|
state->reord);
|
2007-11-16 03:44:56 +00:00
|
|
|
|
|
|
|
/* SACK enhanced F-RTO (RFC4138; Appendix B) */
|
2012-02-12 18:37:09 +00:00
|
|
|
if (!after(end_seq, tp->frto_highmark))
|
2008-12-06 06:42:22 +00:00
|
|
|
state->flag |= FLAG_ONLY_ORIG_SACKED;
|
2007-11-16 03:44:56 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (sacked & TCPCB_LOST) {
|
2008-12-06 06:42:22 +00:00
|
|
|
sacked &= ~TCPCB_LOST;
|
2008-11-25 05:14:43 +00:00
|
|
|
tp->lost_out -= pcount;
|
2007-11-16 03:44:56 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-12-06 06:42:22 +00:00
|
|
|
sacked |= TCPCB_SACKED_ACKED;
|
|
|
|
state->flag |= FLAG_DATA_SACKED;
|
2008-11-25 05:14:43 +00:00
|
|
|
tp->sacked_out += pcount;
|
2007-11-16 03:44:56 +00:00
|
|
|
|
2008-11-25 05:14:43 +00:00
|
|
|
fack_count += pcount;
|
2007-11-16 03:44:56 +00:00
|
|
|
|
|
|
|
/* Lost marker hint past SACKed? Tweak RFC3517 cnt */
|
|
|
|
if (!tcp_is_fack(tp) && (tp->lost_skb_hint != NULL) &&
|
2012-02-12 18:37:09 +00:00
|
|
|
before(start_seq, TCP_SKB_CB(tp->lost_skb_hint)->seq))
|
2008-11-25 05:14:43 +00:00
|
|
|
tp->lost_cnt_hint += pcount;
|
2007-11-16 03:44:56 +00:00
|
|
|
|
|
|
|
if (fack_count > tp->fackets_out)
|
|
|
|
tp->fackets_out = fack_count;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* D-SACK. We can detect redundant retransmission in S|R and plain R
|
|
|
|
* frames and clear it. undo_retrans is decreased above, L|R frames
|
|
|
|
* are accounted above as well.
|
|
|
|
*/
|
2008-12-06 06:42:22 +00:00
|
|
|
if (dup_sack && (sacked & TCPCB_SACKED_RETRANS)) {
|
|
|
|
sacked &= ~TCPCB_SACKED_RETRANS;
|
2008-11-25 05:14:43 +00:00
|
|
|
tp->retrans_out -= pcount;
|
2007-11-16 03:44:56 +00:00
|
|
|
}
|
|
|
|
|
2008-12-06 06:42:22 +00:00
|
|
|
return sacked;
|
2007-11-16 03:44:56 +00:00
|
|
|
}
|
|
|
|
|
2012-02-12 18:37:10 +00:00
|
|
|
/* Shift newly-SACKed bytes from this skb to the immediately previous
|
|
|
|
* already-SACKed sk_buff. Mark the newly-SACKed bytes as such.
|
|
|
|
*/
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_shifted_skb(struct sock *sk, struct sk_buff *skb,
|
|
|
|
struct tcp_sacktag_state *state,
|
|
|
|
unsigned int pcount, int shifted, int mss,
|
|
|
|
bool dup_sack)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2008-12-06 06:42:41 +00:00
|
|
|
struct sk_buff *prev = tcp_write_queue_prev(sk, skb);
|
2012-02-12 18:37:10 +00:00
|
|
|
u32 start_seq = TCP_SKB_CB(skb)->seq; /* start of newly-SACKed */
|
|
|
|
u32 end_seq = start_seq + shifted; /* end of newly-SACKed */
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
|
|
|
|
BUG_ON(!pcount);
|
|
|
|
|
2012-02-26 10:06:19 +00:00
|
|
|
/* Adjust counters and hints for the newly sacked sequence
|
|
|
|
* range but discard the return value since prev is already
|
|
|
|
* marked. We must tag the range first because the seq
|
|
|
|
* advancement below implicitly advances
|
|
|
|
* tcp_highest_sack_seq() when skb is highest_sack.
|
|
|
|
*/
|
|
|
|
tcp_sacktag_one(sk, state, TCP_SKB_CB(skb)->sacked,
|
|
|
|
start_seq, end_seq, dup_sack, pcount);
|
|
|
|
|
|
|
|
if (skb == tp->lost_skb_hint)
|
2012-02-13 20:22:08 +00:00
|
|
|
tp->lost_cnt_hint += pcount;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
TCP_SKB_CB(prev)->end_seq += shifted;
|
|
|
|
TCP_SKB_CB(skb)->seq += shifted;
|
|
|
|
|
|
|
|
skb_shinfo(prev)->gso_segs += pcount;
|
|
|
|
BUG_ON(skb_shinfo(skb)->gso_segs < pcount);
|
|
|
|
skb_shinfo(skb)->gso_segs -= pcount;
|
|
|
|
|
|
|
|
/* When we're adding to gso_segs == 1, gso_size will be zero,
|
|
|
|
* in theory this shouldn't be necessary but as long as DSACK
|
|
|
|
* code can come after this skb later on it's better to keep
|
|
|
|
* setting gso_size to something.
|
|
|
|
*/
|
|
|
|
if (!skb_shinfo(prev)->gso_size) {
|
|
|
|
skb_shinfo(prev)->gso_size = mss;
|
|
|
|
skb_shinfo(prev)->gso_type = sk->sk_gso_type;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* CHECKME: To clear or not to clear? Mimics normal skb currently */
|
|
|
|
if (skb_shinfo(skb)->gso_segs <= 1) {
|
|
|
|
skb_shinfo(skb)->gso_size = 0;
|
|
|
|
skb_shinfo(skb)->gso_type = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Difference in this won't matter, both ACKed by the same cumul. ACK */
|
|
|
|
TCP_SKB_CB(prev)->sacked |= (TCP_SKB_CB(skb)->sacked & TCPCB_EVER_RETRANS);
|
|
|
|
|
|
|
|
if (skb->len > 0) {
|
|
|
|
BUG_ON(!tcp_skb_pcount(skb));
|
2008-11-25 05:27:22 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_SACKSHIFTED);
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Whole SKB was eaten :-) */
|
|
|
|
|
2008-11-25 05:26:56 +00:00
|
|
|
if (skb == tp->retransmit_skb_hint)
|
|
|
|
tp->retransmit_skb_hint = prev;
|
|
|
|
if (skb == tp->scoreboard_skb_hint)
|
|
|
|
tp->scoreboard_skb_hint = prev;
|
|
|
|
if (skb == tp->lost_skb_hint) {
|
|
|
|
tp->lost_skb_hint = prev;
|
|
|
|
tp->lost_cnt_hint -= tcp_skb_pcount(prev);
|
|
|
|
}
|
|
|
|
|
2011-09-27 17:25:05 +00:00
|
|
|
TCP_SKB_CB(skb)->tcp_flags |= TCP_SKB_CB(prev)->tcp_flags;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
if (skb == tcp_highest_sack(sk))
|
|
|
|
tcp_advance_highest_sack(sk, skb);
|
|
|
|
|
|
|
|
tcp_unlink_write_queue(skb, sk);
|
|
|
|
sk_wmem_free_skb(sk, skb);
|
|
|
|
|
2008-11-25 05:27:22 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_SACKMERGED);
|
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* I wish gso_size would have a bit more sane initialization than
|
|
|
|
* something-or-zero which complicates things
|
|
|
|
*/
|
2011-10-21 09:22:42 +00:00
|
|
|
static int tcp_skb_seglen(const struct sk_buff *skb)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
{
|
2008-12-06 06:41:26 +00:00
|
|
|
return tcp_skb_pcount(skb) == 1 ? skb->len : tcp_skb_mss(skb);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Shifting pages past head area doesn't work */
|
2011-10-21 09:22:42 +00:00
|
|
|
static int skb_can_shift(const struct sk_buff *skb)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
{
|
|
|
|
return !skb_headlen(skb) && skb_is_nonlinear(skb);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Try collapsing SACK blocks spanning across multiple skbs to a single
|
|
|
|
* skb.
|
|
|
|
*/
|
|
|
|
static struct sk_buff *tcp_shift_skb_data(struct sock *sk, struct sk_buff *skb,
|
2008-12-06 06:42:22 +00:00
|
|
|
struct tcp_sacktag_state *state,
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
u32 start_seq, u32 end_seq,
|
2012-05-16 23:15:34 +00:00
|
|
|
bool dup_sack)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *prev;
|
|
|
|
int mss;
|
|
|
|
int pcount = 0;
|
|
|
|
int len;
|
|
|
|
int in_sack;
|
|
|
|
|
|
|
|
if (!sk_can_gso(sk))
|
|
|
|
goto fallback;
|
|
|
|
|
|
|
|
/* Normally R but no L won't result in plain S */
|
|
|
|
if (!dup_sack &&
|
2008-12-06 06:41:06 +00:00
|
|
|
(TCP_SKB_CB(skb)->sacked & (TCPCB_LOST|TCPCB_SACKED_RETRANS)) == TCPCB_SACKED_RETRANS)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
goto fallback;
|
|
|
|
if (!skb_can_shift(skb))
|
|
|
|
goto fallback;
|
|
|
|
/* This frame is about to be dropped (was ACKed). */
|
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, tp->snd_una))
|
|
|
|
goto fallback;
|
|
|
|
|
|
|
|
/* Can only happen with delayed DSACK + discard craziness */
|
|
|
|
if (unlikely(skb == tcp_write_queue_head(sk)))
|
|
|
|
goto fallback;
|
|
|
|
prev = tcp_write_queue_prev(sk, skb);
|
|
|
|
|
|
|
|
if ((TCP_SKB_CB(prev)->sacked & TCPCB_TAGBITS) != TCPCB_SACKED_ACKED)
|
|
|
|
goto fallback;
|
|
|
|
|
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq) &&
|
|
|
|
!before(end_seq, TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
|
|
|
if (in_sack) {
|
|
|
|
len = skb->len;
|
|
|
|
pcount = tcp_skb_pcount(skb);
|
2008-12-06 06:41:26 +00:00
|
|
|
mss = tcp_skb_seglen(skb);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
|
|
|
|
/* TODO: Fix DSACKs to not fragment already SACKed and we can
|
|
|
|
* drop this restriction as unnecessary
|
|
|
|
*/
|
2008-12-06 06:41:26 +00:00
|
|
|
if (mss != tcp_skb_seglen(prev))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
goto fallback;
|
|
|
|
} else {
|
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, start_seq))
|
|
|
|
goto noop;
|
|
|
|
/* CHECKME: This is non-MSS split case only?, this will
|
|
|
|
* cause skipped skbs due to advancing loop btw, original
|
|
|
|
* has that feature too
|
|
|
|
*/
|
|
|
|
if (tcp_skb_pcount(skb) <= 1)
|
|
|
|
goto noop;
|
|
|
|
|
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq);
|
|
|
|
if (!in_sack) {
|
|
|
|
/* TODO: head merge to next could be attempted here
|
|
|
|
* if (!after(TCP_SKB_CB(skb)->end_seq, end_seq)),
|
|
|
|
* though it might not be worth of the additional hassle
|
|
|
|
*
|
|
|
|
* ...we can probably just fallback to what was done
|
|
|
|
* previously. We could try merging non-SACKed ones
|
|
|
|
* as well but it probably isn't going to buy off
|
|
|
|
* because later SACKs might again split them, and
|
|
|
|
* it would make skb timestamp tracking considerably
|
|
|
|
* harder problem.
|
|
|
|
*/
|
|
|
|
goto fallback;
|
|
|
|
}
|
|
|
|
|
|
|
|
len = end_seq - TCP_SKB_CB(skb)->seq;
|
|
|
|
BUG_ON(len < 0);
|
|
|
|
BUG_ON(len > skb->len);
|
|
|
|
|
|
|
|
/* MSS boundaries should be honoured or else pcount will
|
|
|
|
* severely break even though it makes things bit trickier.
|
|
|
|
* Optimize common case to avoid most of the divides
|
|
|
|
*/
|
|
|
|
mss = tcp_skb_mss(skb);
|
|
|
|
|
|
|
|
/* TODO: Fix DSACKs to not fragment already SACKed and we can
|
|
|
|
* drop this restriction as unnecessary
|
|
|
|
*/
|
2008-12-06 06:41:26 +00:00
|
|
|
if (mss != tcp_skb_seglen(prev))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
goto fallback;
|
|
|
|
|
|
|
|
if (len == mss) {
|
|
|
|
pcount = 1;
|
|
|
|
} else if (len < mss) {
|
|
|
|
goto noop;
|
|
|
|
} else {
|
|
|
|
pcount = len / mss;
|
|
|
|
len = pcount * mss;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-03-05 19:35:04 +00:00
|
|
|
/* tcp_sacktag_one() won't SACK-tag ranges below snd_una */
|
|
|
|
if (!after(TCP_SKB_CB(skb)->seq + len, tp->snd_una))
|
|
|
|
goto fallback;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
if (!skb_shift(prev, skb, len))
|
|
|
|
goto fallback;
|
2009-03-01 08:21:36 +00:00
|
|
|
if (!tcp_shifted_skb(sk, skb, state, pcount, len, mss, dup_sack))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
/* Hole filled allows collapsing with the next as well, this is very
|
|
|
|
* useful when hole on every nth skb pattern happens
|
|
|
|
*/
|
|
|
|
if (prev == tcp_write_queue_tail(sk))
|
|
|
|
goto out;
|
|
|
|
skb = tcp_write_queue_next(sk, prev);
|
|
|
|
|
2008-12-06 06:40:47 +00:00
|
|
|
if (!skb_can_shift(skb) ||
|
|
|
|
(skb == tcp_send_head(sk)) ||
|
|
|
|
((TCP_SKB_CB(skb)->sacked & TCPCB_TAGBITS) != TCPCB_SACKED_ACKED) ||
|
2008-12-06 06:41:26 +00:00
|
|
|
(mss != tcp_skb_seglen(skb)))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
len = skb->len;
|
|
|
|
if (skb_shift(prev, skb, len)) {
|
|
|
|
pcount += tcp_skb_pcount(skb);
|
2009-03-01 08:21:36 +00:00
|
|
|
tcp_shifted_skb(sk, skb, state, tcp_skb_pcount(skb), len, mss, 0);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
2008-12-06 06:42:22 +00:00
|
|
|
state->fack_count += pcount;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
return prev;
|
|
|
|
|
|
|
|
noop:
|
|
|
|
return skb;
|
|
|
|
|
|
|
|
fallback:
|
2008-11-25 05:27:22 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_SACKSHIFTFALLBACK);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
static struct sk_buff *tcp_sacktag_walk(struct sk_buff *skb, struct sock *sk,
|
|
|
|
struct tcp_sack_block *next_dup,
|
2008-12-06 06:42:22 +00:00
|
|
|
struct tcp_sacktag_state *state,
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
u32 start_seq, u32 end_seq,
|
2012-05-16 23:15:34 +00:00
|
|
|
bool dup_sack_in)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
{
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *tmp;
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
tcp_for_write_queue_from(skb, sk) {
|
|
|
|
int in_sack = 0;
|
2012-05-16 23:15:34 +00:00
|
|
|
bool dup_sack = dup_sack_in;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
|
|
|
|
|
|
|
/* queue is in-order => we can short-circuit the walk early */
|
|
|
|
if (!before(TCP_SKB_CB(skb)->seq, end_seq))
|
|
|
|
break;
|
|
|
|
|
|
|
|
if ((next_dup != NULL) &&
|
|
|
|
before(TCP_SKB_CB(skb)->seq, next_dup->end_seq)) {
|
|
|
|
in_sack = tcp_match_skb_to_sack(sk, skb,
|
|
|
|
next_dup->start_seq,
|
|
|
|
next_dup->end_seq);
|
|
|
|
if (in_sack > 0)
|
2012-05-16 23:15:34 +00:00
|
|
|
dup_sack = true;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
}
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
/* skb reference here is a bit tricky to get right, since
|
|
|
|
* shifting can eat and free both this skb and the next,
|
|
|
|
* so not even _safe variant of the loop is enough.
|
|
|
|
*/
|
|
|
|
if (in_sack <= 0) {
|
2008-12-06 06:42:22 +00:00
|
|
|
tmp = tcp_shift_skb_data(sk, skb, state,
|
|
|
|
start_seq, end_seq, dup_sack);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
if (tmp != NULL) {
|
|
|
|
if (tmp != skb) {
|
|
|
|
skb = tmp;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
in_sack = 0;
|
|
|
|
} else {
|
|
|
|
in_sack = tcp_match_skb_to_sack(sk, skb,
|
|
|
|
start_seq,
|
|
|
|
end_seq);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
if (unlikely(in_sack < 0))
|
|
|
|
break;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
if (in_sack) {
|
2012-02-12 18:37:09 +00:00
|
|
|
TCP_SKB_CB(skb)->sacked =
|
|
|
|
tcp_sacktag_one(sk,
|
|
|
|
state,
|
|
|
|
TCP_SKB_CB(skb)->sacked,
|
|
|
|
TCP_SKB_CB(skb)->seq,
|
|
|
|
TCP_SKB_CB(skb)->end_seq,
|
|
|
|
dup_sack,
|
|
|
|
tcp_skb_pcount(skb));
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 05:20:15 +00:00
|
|
|
if (!before(TCP_SKB_CB(skb)->seq,
|
|
|
|
tcp_highest_sack_seq(tp)))
|
|
|
|
tcp_advance_highest_sack(sk, skb);
|
|
|
|
}
|
|
|
|
|
2008-12-06 06:42:22 +00:00
|
|
|
state->fack_count += tcp_skb_pcount(skb);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
}
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Avoid all extra work that is being done by sacktag while walking in
|
|
|
|
* a normal way
|
|
|
|
*/
|
|
|
|
static struct sk_buff *tcp_sacktag_skip(struct sk_buff *skb, struct sock *sk,
|
2008-12-06 06:42:22 +00:00
|
|
|
struct tcp_sacktag_state *state,
|
|
|
|
u32 skip_to_seq)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
{
|
|
|
|
tcp_for_write_queue_from(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
|
|
|
|
2008-11-25 05:12:28 +00:00
|
|
|
if (after(TCP_SKB_CB(skb)->end_seq, skip_to_seq))
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
break;
|
2008-03-03 20:10:16 +00:00
|
|
|
|
2008-12-06 06:42:22 +00:00
|
|
|
state->fack_count += tcp_skb_pcount(skb);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
}
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct sk_buff *tcp_maybe_skipping_dsack(struct sk_buff *skb,
|
|
|
|
struct sock *sk,
|
|
|
|
struct tcp_sack_block *next_dup,
|
2008-12-06 06:42:22 +00:00
|
|
|
struct tcp_sacktag_state *state,
|
|
|
|
u32 skip_to_seq)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
{
|
|
|
|
if (next_dup == NULL)
|
|
|
|
return skb;
|
|
|
|
|
|
|
|
if (before(next_dup->start_seq, skip_to_seq)) {
|
2008-12-06 06:42:22 +00:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, state, next_dup->start_seq);
|
|
|
|
skb = tcp_sacktag_walk(skb, sk, NULL, state,
|
|
|
|
next_dup->start_seq, next_dup->end_seq,
|
|
|
|
1);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static int tcp_sack_cache_ok(const struct tcp_sock *tp, const struct tcp_sack_block *cache)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
{
|
|
|
|
return cache < tp->recv_sack_cache + ARRAY_SIZE(tp->recv_sack_cache);
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
static int
|
2011-10-21 09:22:42 +00:00
|
|
|
tcp_sacktag_write_queue(struct sock *sk, const struct sk_buff *ack_skb,
|
2007-12-31 22:57:14 +00:00
|
|
|
u32 prior_snd_una)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2011-10-21 09:22:42 +00:00
|
|
|
const unsigned char *ptr = (skb_transport_header(ack_skb) +
|
|
|
|
TCP_SKB_CB(ack_skb)->sacked);
|
2007-11-16 03:49:47 +00:00
|
|
|
struct tcp_sack_block_wire *sp_wire = (struct tcp_sack_block_wire *)(ptr+2);
|
2008-07-19 07:07:02 +00:00
|
|
|
struct tcp_sack_block sp[TCP_NUM_SACKS];
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
struct tcp_sack_block *cache;
|
2008-12-06 06:42:22 +00:00
|
|
|
struct tcp_sacktag_state state;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
struct sk_buff *skb;
|
2008-07-19 07:07:02 +00:00
|
|
|
int num_sacks = min(TCP_NUM_SACKS, (ptr[1] - TCPOLEN_SACK_BASE) >> 3);
|
2007-11-16 03:49:47 +00:00
|
|
|
int used_sacks;
|
2012-05-16 23:15:34 +00:00
|
|
|
bool found_dup_sack = false;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
int i, j;
|
2007-02-05 07:35:57 +00:00
|
|
|
int first_sack_index;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2008-12-06 06:42:22 +00:00
|
|
|
state.flag = 0;
|
|
|
|
state.reord = tp->packets_out;
|
|
|
|
|
2007-03-25 04:03:23 +00:00
|
|
|
if (!tp->sacked_out) {
|
2007-10-08 06:37:55 +00:00
|
|
|
if (WARN_ON(tp->fackets_out))
|
|
|
|
tp->fackets_out = 0;
|
2007-12-01 22:48:06 +00:00
|
|
|
tcp_highest_sack_reset(sk);
|
2007-03-25 04:03:23 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2008-07-17 03:29:51 +00:00
|
|
|
found_dup_sack = tcp_check_dsack(sk, ack_skb, sp_wire,
|
2007-06-19 05:43:06 +00:00
|
|
|
num_sacks, prior_snd_una);
|
|
|
|
if (found_dup_sack)
|
2008-12-06 06:42:22 +00:00
|
|
|
state.flag |= FLAG_DSACKING_ACK;
|
2007-02-05 07:36:42 +00:00
|
|
|
|
|
|
|
/* Eliminate too old ACKs, but take into
|
|
|
|
* account more or less fresh ones, they can
|
|
|
|
* contain valid SACK info.
|
|
|
|
*/
|
|
|
|
if (before(TCP_SKB_CB(ack_skb)->ack_seq, prior_snd_una - tp->max_window))
|
|
|
|
return 0;
|
|
|
|
|
2007-11-14 23:47:18 +00:00
|
|
|
if (!tp->packets_out)
|
|
|
|
goto out;
|
|
|
|
|
2007-11-16 03:49:47 +00:00
|
|
|
used_sacks = 0;
|
|
|
|
first_sack_index = 0;
|
|
|
|
for (i = 0; i < num_sacks; i++) {
|
2012-05-16 23:15:34 +00:00
|
|
|
bool dup_sack = !i && found_dup_sack;
|
2007-11-16 03:49:47 +00:00
|
|
|
|
2008-05-02 23:26:16 +00:00
|
|
|
sp[used_sacks].start_seq = get_unaligned_be32(&sp_wire[i].start_seq);
|
|
|
|
sp[used_sacks].end_seq = get_unaligned_be32(&sp_wire[i].end_seq);
|
2007-11-16 03:49:47 +00:00
|
|
|
|
|
|
|
if (!tcp_is_sackblock_valid(tp, dup_sack,
|
|
|
|
sp[used_sacks].start_seq,
|
|
|
|
sp[used_sacks].end_seq)) {
|
2008-07-03 08:05:41 +00:00
|
|
|
int mib_idx;
|
|
|
|
|
2007-11-16 03:49:47 +00:00
|
|
|
if (dup_sack) {
|
|
|
|
if (!tp->undo_marker)
|
2008-07-03 08:05:41 +00:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKIGNOREDNOUNDO;
|
2007-11-16 03:49:47 +00:00
|
|
|
else
|
2008-07-03 08:05:41 +00:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKIGNOREDOLD;
|
2007-11-16 03:49:47 +00:00
|
|
|
} else {
|
|
|
|
/* Don't count olds caused by ACK reordering */
|
|
|
|
if ((TCP_SKB_CB(ack_skb)->ack_seq != tp->snd_una) &&
|
|
|
|
!after(sp[used_sacks].end_seq, tp->snd_una))
|
|
|
|
continue;
|
2008-07-03 08:05:41 +00:00
|
|
|
mib_idx = LINUX_MIB_TCPSACKDISCARD;
|
2007-11-16 03:49:47 +00:00
|
|
|
}
|
2008-07-03 08:05:41 +00:00
|
|
|
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), mib_idx);
|
2007-11-16 03:49:47 +00:00
|
|
|
if (i == 0)
|
|
|
|
first_sack_index = -1;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Ignore very old stuff early */
|
|
|
|
if (!after(sp[used_sacks].end_seq, prior_snd_una))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
used_sacks++;
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
/* order SACK blocks to allow in order walk of the retrans queue */
|
|
|
|
for (i = used_sacks - 1; i > 0; i--) {
|
2007-12-31 22:57:14 +00:00
|
|
|
for (j = 0; j < i; j++) {
|
|
|
|
if (after(sp[j].start_seq, sp[j + 1].start_seq)) {
|
2009-03-21 20:36:17 +00:00
|
|
|
swap(sp[j], sp[j + 1]);
|
2005-11-11 01:14:59 +00:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
/* Track where the first SACK block goes to */
|
|
|
|
if (j == first_sack_index)
|
2007-12-31 22:57:14 +00:00
|
|
|
first_sack_index = j + 1;
|
2005-11-11 01:14:59 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
skb = tcp_write_queue_head(sk);
|
2008-12-06 06:42:22 +00:00
|
|
|
state.fack_count = 0;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
i = 0;
|
|
|
|
|
|
|
|
if (!tp->sacked_out) {
|
|
|
|
/* It's already past, so skip checking against it */
|
|
|
|
cache = tp->recv_sack_cache + ARRAY_SIZE(tp->recv_sack_cache);
|
|
|
|
} else {
|
|
|
|
cache = tp->recv_sack_cache;
|
|
|
|
/* Skip empty blocks in at head of the cache */
|
|
|
|
while (tcp_sack_cache_ok(tp, cache) && !cache->start_seq &&
|
|
|
|
!cache->end_seq)
|
|
|
|
cache++;
|
2007-02-05 07:35:57 +00:00
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
while (i < used_sacks) {
|
2007-11-16 03:49:47 +00:00
|
|
|
u32 start_seq = sp[i].start_seq;
|
|
|
|
u32 end_seq = sp[i].end_seq;
|
2012-05-16 23:15:34 +00:00
|
|
|
bool dup_sack = (found_dup_sack && (i == first_sack_index));
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
struct tcp_sack_block *next_dup = NULL;
|
[TCP]: Process DSACKs that reside within a SACK block
DSACK inside another SACK block were missed if start_seq of DSACK
was larger than SACK block's because sorting prioritizes full
processing of the SACK block before DSACK. After SACK block
sorting situation is like this:
SSSSSSSSS
D
SSSSSS
SSSSSSS
Because write_queue is walked in-order, when the first SACK block
has been processed, TCP is already past the skb for which the
DSACK arrived and we haven't taught it to backtrack (nor should
we), so TCP just continues processing by going to the next SACK
block after the DSACK (if any).
Whenever such DSACK is present, do an embedded checking during
the previous SACK block.
If the DSACK is below snd_una, there won't be overlapping SACK
block, and thus no problem in that case. Also if start_seq of
the DSACK is equal to the actual block, it will be processed
first.
Tested this by using netem to duplicate 15% of packets, and
by printing SACK block when found_dup_sack is true and the
selected skb in the dup_sack = 1 branch (if taken):
SACK block 0: 4344-5792 (relative to snd_una 2019137317)
SACK block 1: 4344-5792 (relative to snd_una 2019137317)
equal start seqnos => next_dup = 0, dup_sack = 1 won't occur...
SACK block 0: 5792-7240 (relative to snd_una 2019214061)
SACK block 1: 2896-7240 (relative to snd_una 2019214061)
DSACK skb match 5792-7240 (relative to snd_una)
...and next_dup = 1 case (after the not shown start_seq sort),
went to dup_sack = 1 branch.
Signed-off-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-01 07:09:37 +00:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
if (found_dup_sack && ((i + 1) == first_sack_index))
|
|
|
|
next_dup = &sp[i + 1];
|
2005-04-16 22:20:36 +00:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
/* Skip too early cached blocks */
|
|
|
|
while (tcp_sack_cache_ok(tp, cache) &&
|
|
|
|
!before(start_seq, cache->end_seq))
|
|
|
|
cache++;
|
|
|
|
|
|
|
|
/* Can skip some work by looking recv_sack_cache? */
|
|
|
|
if (tcp_sack_cache_ok(tp, cache) && !dup_sack &&
|
|
|
|
after(end_seq, cache->start_seq)) {
|
|
|
|
|
|
|
|
/* Head todo? */
|
|
|
|
if (before(start_seq, cache->start_seq)) {
|
2008-12-06 06:42:22 +00:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, &state,
|
|
|
|
start_seq);
|
2007-12-31 22:57:14 +00:00
|
|
|
skb = tcp_sacktag_walk(skb, sk, next_dup,
|
2008-12-06 06:42:22 +00:00
|
|
|
&state,
|
2007-12-31 22:57:14 +00:00
|
|
|
start_seq,
|
|
|
|
cache->start_seq,
|
2008-12-06 06:42:22 +00:00
|
|
|
dup_sack);
|
2007-02-05 07:35:57 +00:00
|
|
|
}
|
2005-11-11 01:14:59 +00:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
/* Rest of the block already fully processed? */
|
2007-11-17 00:17:05 +00:00
|
|
|
if (!after(end_seq, cache->end_seq))
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
goto advance_sp;
|
2007-11-17 00:17:05 +00:00
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
skb = tcp_maybe_skipping_dsack(skb, sk, next_dup,
|
2008-12-06 06:42:22 +00:00
|
|
|
&state,
|
|
|
|
cache->end_seq);
|
[TCP]: Process DSACKs that reside within a SACK block
DSACK inside another SACK block were missed if start_seq of DSACK
was larger than SACK block's because sorting prioritizes full
processing of the SACK block before DSACK. After SACK block
sorting situation is like this:
SSSSSSSSS
D
SSSSSS
SSSSSSS
Because write_queue is walked in-order, when the first SACK block
has been processed, TCP is already past the skb for which the
DSACK arrived and we haven't taught it to backtrack (nor should
we), so TCP just continues processing by going to the next SACK
block after the DSACK (if any).
Whenever such DSACK is present, do an embedded checking during
the previous SACK block.
If the DSACK is below snd_una, there won't be overlapping SACK
block, and thus no problem in that case. Also if start_seq of
the DSACK is equal to the actual block, it will be processed
first.
Tested this by using netem to duplicate 15% of packets, and
by printing SACK block when found_dup_sack is true and the
selected skb in the dup_sack = 1 branch (if taken):
SACK block 0: 4344-5792 (relative to snd_una 2019137317)
SACK block 1: 4344-5792 (relative to snd_una 2019137317)
equal start seqnos => next_dup = 0, dup_sack = 1 won't occur...
SACK block 0: 5792-7240 (relative to snd_una 2019214061)
SACK block 1: 2896-7240 (relative to snd_una 2019214061)
DSACK skb match 5792-7240 (relative to snd_una)
...and next_dup = 1 case (after the not shown start_seq sort),
went to dup_sack = 1 branch.
Signed-off-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-01 07:09:37 +00:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
/* ...tail remains todo... */
|
2007-12-01 22:48:06 +00:00
|
|
|
if (tcp_highest_sack_seq(tp) == cache->end_seq) {
|
2007-11-17 00:17:05 +00:00
|
|
|
/* ...but better entrypoint exists! */
|
2007-12-01 22:48:06 +00:00
|
|
|
skb = tcp_highest_sack(sk);
|
|
|
|
if (skb == NULL)
|
|
|
|
break;
|
2008-12-06 06:42:22 +00:00
|
|
|
state.fack_count = tp->fackets_out;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
cache++;
|
|
|
|
goto walk;
|
[TCP]: Process DSACKs that reside within a SACK block
DSACK inside another SACK block were missed if start_seq of DSACK
was larger than SACK block's because sorting prioritizes full
processing of the SACK block before DSACK. After SACK block
sorting situation is like this:
SSSSSSSSS
D
SSSSSS
SSSSSSS
Because write_queue is walked in-order, when the first SACK block
has been processed, TCP is already past the skb for which the
DSACK arrived and we haven't taught it to backtrack (nor should
we), so TCP just continues processing by going to the next SACK
block after the DSACK (if any).
Whenever such DSACK is present, do an embedded checking during
the previous SACK block.
If the DSACK is below snd_una, there won't be overlapping SACK
block, and thus no problem in that case. Also if start_seq of
the DSACK is equal to the actual block, it will be processed
first.
Tested this by using netem to duplicate 15% of packets, and
by printing SACK block when found_dup_sack is true and the
selected skb in the dup_sack = 1 branch (if taken):
SACK block 0: 4344-5792 (relative to snd_una 2019137317)
SACK block 1: 4344-5792 (relative to snd_una 2019137317)
equal start seqnos => next_dup = 0, dup_sack = 1 won't occur...
SACK block 0: 5792-7240 (relative to snd_una 2019214061)
SACK block 1: 2896-7240 (relative to snd_una 2019214061)
DSACK skb match 5792-7240 (relative to snd_una)
...and next_dup = 1 case (after the not shown start_seq sort),
went to dup_sack = 1 branch.
Signed-off-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-01 07:09:37 +00:00
|
|
|
}
|
|
|
|
|
2008-12-06 06:42:22 +00:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, &state, cache->end_seq);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
/* Check overlap against next cached too (past this one already) */
|
|
|
|
cache++;
|
|
|
|
continue;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-12-01 22:48:06 +00:00
|
|
|
if (!before(start_seq, tcp_highest_sack_seq(tp))) {
|
|
|
|
skb = tcp_highest_sack(sk);
|
|
|
|
if (skb == NULL)
|
|
|
|
break;
|
2008-12-06 06:42:22 +00:00
|
|
|
state.fack_count = tp->fackets_out;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2008-12-06 06:42:22 +00:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, &state, start_seq);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
|
|
|
|
walk:
|
2008-12-06 06:42:22 +00:00
|
|
|
skb = tcp_sacktag_walk(skb, sk, next_dup, &state,
|
|
|
|
start_seq, end_seq, dup_sack);
|
2007-11-11 05:24:19 +00:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
advance_sp:
|
2007-11-11 05:24:19 +00:00
|
|
|
/* SACK enhanced FRTO (RFC4138, Appendix B): Clearing correct
|
|
|
|
* due to in-order walk
|
|
|
|
*/
|
|
|
|
if (after(end_seq, tp->frto_highmark))
|
2008-12-06 06:42:22 +00:00
|
|
|
state.flag &= ~FLAG_ONLY_ORIG_SACKED;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
|
|
|
|
i++;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
/* Clear the head of the cache sack blocks so we can skip it next time */
|
|
|
|
for (i = 0; i < ARRAY_SIZE(tp->recv_sack_cache) - used_sacks; i++) {
|
|
|
|
tp->recv_sack_cache[i].start_seq = 0;
|
|
|
|
tp->recv_sack_cache[i].end_seq = 0;
|
|
|
|
}
|
|
|
|
for (j = 0; j < used_sacks; j++)
|
|
|
|
tp->recv_sack_cache[i++] = sp[j];
|
|
|
|
|
2007-12-01 22:47:57 +00:00
|
|
|
tcp_mark_lost_retrans(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-08-09 11:45:17 +00:00
|
|
|
tcp_verify_left_out(tp);
|
|
|
|
|
2008-12-06 06:42:22 +00:00
|
|
|
if ((state.reord < tp->fackets_out) &&
|
2007-11-16 03:35:11 +00:00
|
|
|
((icsk->icsk_ca_state != TCP_CA_Loss) || tp->undo_marker) &&
|
2007-02-24 00:22:06 +00:00
|
|
|
(!tp->frto_highmark || after(tp->snd_una, tp->frto_highmark)))
|
2008-12-06 06:42:22 +00:00
|
|
|
tcp_update_reordering(sk, tp->fackets_out - state.reord, 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-11-14 23:47:18 +00:00
|
|
|
out:
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
#if FASTRETRANS_DEBUG > 0
|
2008-07-26 04:43:18 +00:00
|
|
|
WARN_ON((int)tp->sacked_out < 0);
|
|
|
|
WARN_ON((int)tp->lost_out < 0);
|
|
|
|
WARN_ON((int)tp->retrans_out < 0);
|
|
|
|
WARN_ON((int)tcp_packets_in_flight(tp) < 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif
|
2008-12-06 06:42:22 +00:00
|
|
|
return state.flag;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2008-04-08 05:33:07 +00:00
|
|
|
/* Limits sacked_out so that sum with lost_out isn't ever larger than
|
2012-05-16 23:15:34 +00:00
|
|
|
* packets_out. Returns false if sacked_out adjustement wasn't necessary.
|
2007-02-22 07:01:36 +00:00
|
|
|
*/
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_limit_reno_sacked(struct tcp_sock *tp)
|
2007-05-27 08:52:00 +00:00
|
|
|
{
|
|
|
|
u32 holes;
|
|
|
|
|
|
|
|
holes = max(tp->lost_out, 1U);
|
|
|
|
holes = min(holes, tp->packets_out);
|
|
|
|
|
|
|
|
if ((tp->sacked_out + holes) > tp->packets_out) {
|
|
|
|
tp->sacked_out = tp->packets_out - holes;
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-05-27 08:52:00 +00:00
|
|
|
}
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2008-04-08 05:33:07 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* If we receive more dupacks than we expected counting segments
|
|
|
|
* in assumption of absent reordering, interpret this as reordering.
|
|
|
|
* The only another reason could be bug in receiver TCP.
|
|
|
|
*/
|
|
|
|
static void tcp_check_reno_reordering(struct sock *sk, const int addend)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
if (tcp_limit_reno_sacked(tp))
|
|
|
|
tcp_update_reordering(sk, tp->packets_out + addend, 0);
|
2007-05-27 08:52:00 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Emulate SACKs for SACKless connection: account for a new dupack. */
|
|
|
|
|
|
|
|
static void tcp_add_reno_sack(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
tp->sacked_out++;
|
|
|
|
tcp_check_reno_reordering(sk, 0);
|
2007-08-09 11:44:16 +00:00
|
|
|
tcp_verify_left_out(tp);
|
2007-05-27 08:52:00 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Account for ACK, ACKing some data in Reno Recovery phase. */
|
|
|
|
|
|
|
|
static void tcp_remove_reno_sacks(struct sock *sk, int acked)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
if (acked > 0) {
|
|
|
|
/* One ACK acked hole. The rest eat duplicate ACKs. */
|
2007-12-31 22:57:14 +00:00
|
|
|
if (acked - 1 >= tp->sacked_out)
|
2007-05-27 08:52:00 +00:00
|
|
|
tp->sacked_out = 0;
|
|
|
|
else
|
2007-12-31 22:57:14 +00:00
|
|
|
tp->sacked_out -= acked - 1;
|
2007-05-27 08:52:00 +00:00
|
|
|
}
|
|
|
|
tcp_check_reno_reordering(sk, acked);
|
2007-08-09 11:44:16 +00:00
|
|
|
tcp_verify_left_out(tp);
|
2007-05-27 08:52:00 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline void tcp_reset_reno_sack(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
tp->sacked_out = 0;
|
|
|
|
}
|
|
|
|
|
tcp FRTO: SACK variant is errorneously used with NewReno
Note: there's actually another bug in FRTO's SACK variant, which
is the causing failure in NewReno case because of the error
that's fixed here. I'll fix the SACK case separately (it's
a separate bug really, though related, but in order to fix that
I need to audit tp->snd_nxt usage a bit).
There were two places where SACK variant of FRTO is getting
incorrectly used even if SACK wasn't negotiated by the TCP flow.
This leads to incorrect setting of frto_highmark with NewReno
if a previous recovery was interrupted by another RTO.
An eventual fallback to conventional recovery then incorrectly
considers one or couple of segments as forward transmissions
though they weren't, which then are not LOST marked during
fallback making them "non-retransmittable" until the next RTO.
In a bad case, those segments are really lost and are the only
one left in the window. Thus TCP needs another RTO to continue.
The next FRTO, however, could again repeat the same events
making the progress of the TCP flow extremely slow.
In order for these events to occur at all, FRTO must occur
again in FRTOs step 3 while the key segments must be lost as
well, which is not too likely in practice. It seems to most
frequently with some small devices such as network printers
that *seem* to accept TCP segments only in-order. In cases
were key segments weren't lost, things get automatically
resolved because those wrongly marked segments don't need to be
retransmitted in order to continue.
I found a reproducer after digging up relevant reports (few
reports in total, none at netdev or lkml I know of), some
cases seemed to indicate middlebox issues which seems now
to be a false assumption some people had made. Bugzilla
#10063 _might_ be related. Damon L. Chesser <damon@damtek.com>
had a reproducable case and was kind enough to tcpdump it
for me. With the tcpdump log it was quite trivial to figure
out.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-05-08 08:09:11 +00:00
|
|
|
static int tcp_is_sackfrto(const struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
return (sysctl_tcp_frto == 0x2) && !tcp_is_reno(tp);
|
|
|
|
}
|
|
|
|
|
2007-10-01 22:27:42 +00:00
|
|
|
/* F-RTO can only be used if TCP has never retransmitted anything other than
|
|
|
|
* head (SACK enhanced variant from Appendix B of RFC4138 is more robust here)
|
|
|
|
*/
|
2012-05-16 23:15:34 +00:00
|
|
|
bool tcp_use_frto(struct sock *sk)
|
2007-02-22 06:59:58 +00:00
|
|
|
{
|
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2008-04-08 05:33:57 +00:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
[TCP] FRTO: Entry is allowed only during (New)Reno like recovery
This interpretation comes from RFC4138:
"If the sender implements some loss recovery algorithm other
than Reno or NewReno [FHG04], the F-RTO algorithm SHOULD
NOT be entered when earlier fast recovery is underway."
I think the RFC means to say (especially in the light of
Appendix B) that ...recovery is underway (not just fast recovery)
or was underway when it was interrupted by an earlier (F-)RTO
that hasn't yet been resolved (snd_una has not advanced enough).
Thus, my interpretation is that whenever TCP has ever
retransmitted other than head, basic version cannot be used
because then the order assumptions which are used as FRTO basis
do not hold.
NewReno has only the head segment retransmitted at a time.
Therefore, walk up to the segment that has not been SACKed, if
that segment is not retransmitted nor anything before it, we know
for sure, that nothing after the non-SACKed segment should be
either. This assumption is valid because TCPCB_EVER_RETRANS does
not leave holes but each non-SACKed segment is rexmitted
in-order.
Check for retrans_out > 1 avoids more expensive walk through the
skb list, as we can know the result beforehand: F-RTO will not be
allowed.
SACKed skb can turn into non-SACked only in the extremely rare
case of SACK reneging, in this case we might fail to detect
retransmissions if there were them for any other than head. To
get rid of that feature, whole rexmit queue would have to be
walked (always) or FRTO should be prevented when SACK reneging
happens. Of course RTO should still trigger after reneging which
makes this issue even less likely to show up. And as long as the
response is as conservative as it's now, nothing bad happens even
then.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-02-22 07:10:39 +00:00
|
|
|
struct sk_buff *skb;
|
|
|
|
|
2007-04-30 07:39:55 +00:00
|
|
|
if (!sysctl_tcp_frto)
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2007-02-22 06:59:58 +00:00
|
|
|
|
2008-04-08 05:33:57 +00:00
|
|
|
/* MTU probe and F-RTO won't really play nicely along currently */
|
|
|
|
if (icsk->icsk_mtup.probe_size)
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2008-04-08 05:33:57 +00:00
|
|
|
|
tcp FRTO: SACK variant is errorneously used with NewReno
Note: there's actually another bug in FRTO's SACK variant, which
is the causing failure in NewReno case because of the error
that's fixed here. I'll fix the SACK case separately (it's
a separate bug really, though related, but in order to fix that
I need to audit tp->snd_nxt usage a bit).
There were two places where SACK variant of FRTO is getting
incorrectly used even if SACK wasn't negotiated by the TCP flow.
This leads to incorrect setting of frto_highmark with NewReno
if a previous recovery was interrupted by another RTO.
An eventual fallback to conventional recovery then incorrectly
considers one or couple of segments as forward transmissions
though they weren't, which then are not LOST marked during
fallback making them "non-retransmittable" until the next RTO.
In a bad case, those segments are really lost and are the only
one left in the window. Thus TCP needs another RTO to continue.
The next FRTO, however, could again repeat the same events
making the progress of the TCP flow extremely slow.
In order for these events to occur at all, FRTO must occur
again in FRTOs step 3 while the key segments must be lost as
well, which is not too likely in practice. It seems to most
frequently with some small devices such as network printers
that *seem* to accept TCP segments only in-order. In cases
were key segments weren't lost, things get automatically
resolved because those wrongly marked segments don't need to be
retransmitted in order to continue.
I found a reproducer after digging up relevant reports (few
reports in total, none at netdev or lkml I know of), some
cases seemed to indicate middlebox issues which seems now
to be a false assumption some people had made. Bugzilla
#10063 _might_ be related. Damon L. Chesser <damon@damtek.com>
had a reproducable case and was kind enough to tcpdump it
for me. With the tcpdump log it was quite trivial to figure
out.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-05-08 08:09:11 +00:00
|
|
|
if (tcp_is_sackfrto(tp))
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-02-22 07:16:11 +00:00
|
|
|
|
[TCP] FRTO: Entry is allowed only during (New)Reno like recovery
This interpretation comes from RFC4138:
"If the sender implements some loss recovery algorithm other
than Reno or NewReno [FHG04], the F-RTO algorithm SHOULD
NOT be entered when earlier fast recovery is underway."
I think the RFC means to say (especially in the light of
Appendix B) that ...recovery is underway (not just fast recovery)
or was underway when it was interrupted by an earlier (F-)RTO
that hasn't yet been resolved (snd_una has not advanced enough).
Thus, my interpretation is that whenever TCP has ever
retransmitted other than head, basic version cannot be used
because then the order assumptions which are used as FRTO basis
do not hold.
NewReno has only the head segment retransmitted at a time.
Therefore, walk up to the segment that has not been SACKed, if
that segment is not retransmitted nor anything before it, we know
for sure, that nothing after the non-SACKed segment should be
either. This assumption is valid because TCPCB_EVER_RETRANS does
not leave holes but each non-SACKed segment is rexmitted
in-order.
Check for retrans_out > 1 avoids more expensive walk through the
skb list, as we can know the result beforehand: F-RTO will not be
allowed.
SACKed skb can turn into non-SACked only in the extremely rare
case of SACK reneging, in this case we might fail to detect
retransmissions if there were them for any other than head. To
get rid of that feature, whole rexmit queue would have to be
walked (always) or FRTO should be prevented when SACK reneging
happens. Of course RTO should still trigger after reneging which
makes this issue even less likely to show up. And as long as the
response is as conservative as it's now, nothing bad happens even
then.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-02-22 07:10:39 +00:00
|
|
|
/* Avoid expensive walking of rexmit queue if possible */
|
|
|
|
if (tp->retrans_out > 1)
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
[TCP] FRTO: Entry is allowed only during (New)Reno like recovery
This interpretation comes from RFC4138:
"If the sender implements some loss recovery algorithm other
than Reno or NewReno [FHG04], the F-RTO algorithm SHOULD
NOT be entered when earlier fast recovery is underway."
I think the RFC means to say (especially in the light of
Appendix B) that ...recovery is underway (not just fast recovery)
or was underway when it was interrupted by an earlier (F-)RTO
that hasn't yet been resolved (snd_una has not advanced enough).
Thus, my interpretation is that whenever TCP has ever
retransmitted other than head, basic version cannot be used
because then the order assumptions which are used as FRTO basis
do not hold.
NewReno has only the head segment retransmitted at a time.
Therefore, walk up to the segment that has not been SACKed, if
that segment is not retransmitted nor anything before it, we know
for sure, that nothing after the non-SACKed segment should be
either. This assumption is valid because TCPCB_EVER_RETRANS does
not leave holes but each non-SACKed segment is rexmitted
in-order.
Check for retrans_out > 1 avoids more expensive walk through the
skb list, as we can know the result beforehand: F-RTO will not be
allowed.
SACKed skb can turn into non-SACked only in the extremely rare
case of SACK reneging, in this case we might fail to detect
retransmissions if there were them for any other than head. To
get rid of that feature, whole rexmit queue would have to be
walked (always) or FRTO should be prevented when SACK reneging
happens. Of course RTO should still trigger after reneging which
makes this issue even less likely to show up. And as long as the
response is as conservative as it's now, nothing bad happens even
then.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-02-22 07:10:39 +00:00
|
|
|
|
2007-03-07 20:12:44 +00:00
|
|
|
skb = tcp_write_queue_head(sk);
|
2008-09-23 09:51:41 +00:00
|
|
|
if (tcp_skb_is_last(sk, skb))
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-03-07 20:12:44 +00:00
|
|
|
skb = tcp_write_queue_next(sk, skb); /* Skips head */
|
|
|
|
tcp_for_write_queue_from(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
2007-12-31 22:57:14 +00:00
|
|
|
if (TCP_SKB_CB(skb)->sacked & TCPCB_RETRANS)
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
[TCP] FRTO: Entry is allowed only during (New)Reno like recovery
This interpretation comes from RFC4138:
"If the sender implements some loss recovery algorithm other
than Reno or NewReno [FHG04], the F-RTO algorithm SHOULD
NOT be entered when earlier fast recovery is underway."
I think the RFC means to say (especially in the light of
Appendix B) that ...recovery is underway (not just fast recovery)
or was underway when it was interrupted by an earlier (F-)RTO
that hasn't yet been resolved (snd_una has not advanced enough).
Thus, my interpretation is that whenever TCP has ever
retransmitted other than head, basic version cannot be used
because then the order assumptions which are used as FRTO basis
do not hold.
NewReno has only the head segment retransmitted at a time.
Therefore, walk up to the segment that has not been SACKed, if
that segment is not retransmitted nor anything before it, we know
for sure, that nothing after the non-SACKed segment should be
either. This assumption is valid because TCPCB_EVER_RETRANS does
not leave holes but each non-SACKed segment is rexmitted
in-order.
Check for retrans_out > 1 avoids more expensive walk through the
skb list, as we can know the result beforehand: F-RTO will not be
allowed.
SACKed skb can turn into non-SACked only in the extremely rare
case of SACK reneging, in this case we might fail to detect
retransmissions if there were them for any other than head. To
get rid of that feature, whole rexmit queue would have to be
walked (always) or FRTO should be prevented when SACK reneging
happens. Of course RTO should still trigger after reneging which
makes this issue even less likely to show up. And as long as the
response is as conservative as it's now, nothing bad happens even
then.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-02-22 07:10:39 +00:00
|
|
|
/* Short-circuit when first non-SACKed skb has been checked */
|
2007-12-31 22:57:14 +00:00
|
|
|
if (!(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED))
|
[TCP] FRTO: Entry is allowed only during (New)Reno like recovery
This interpretation comes from RFC4138:
"If the sender implements some loss recovery algorithm other
than Reno or NewReno [FHG04], the F-RTO algorithm SHOULD
NOT be entered when earlier fast recovery is underway."
I think the RFC means to say (especially in the light of
Appendix B) that ...recovery is underway (not just fast recovery)
or was underway when it was interrupted by an earlier (F-)RTO
that hasn't yet been resolved (snd_una has not advanced enough).
Thus, my interpretation is that whenever TCP has ever
retransmitted other than head, basic version cannot be used
because then the order assumptions which are used as FRTO basis
do not hold.
NewReno has only the head segment retransmitted at a time.
Therefore, walk up to the segment that has not been SACKed, if
that segment is not retransmitted nor anything before it, we know
for sure, that nothing after the non-SACKed segment should be
either. This assumption is valid because TCPCB_EVER_RETRANS does
not leave holes but each non-SACKed segment is rexmitted
in-order.
Check for retrans_out > 1 avoids more expensive walk through the
skb list, as we can know the result beforehand: F-RTO will not be
allowed.
SACKed skb can turn into non-SACked only in the extremely rare
case of SACK reneging, in this case we might fail to detect
retransmissions if there were them for any other than head. To
get rid of that feature, whole rexmit queue would have to be
walked (always) or FRTO should be prevented when SACK reneging
happens. Of course RTO should still trigger after reneging which
makes this issue even less likely to show up. And as long as the
response is as conservative as it's now, nothing bad happens even
then.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-02-22 07:10:39 +00:00
|
|
|
break;
|
|
|
|
}
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-02-22 06:59:58 +00:00
|
|
|
}
|
|
|
|
|
2007-02-22 07:01:36 +00:00
|
|
|
/* RTO occurred, but do not yet enter Loss state. Instead, defer RTO
|
|
|
|
* recovery a bit and use heuristics in tcp_process_frto() to detect if
|
2007-02-22 07:11:57 +00:00
|
|
|
* the RTO was spurious. Only clear SACKED_RETRANS of the head here to
|
|
|
|
* keep retrans_out counting accurate (with SACK F-RTO, other than head
|
|
|
|
* may still have that bit set); TCPCB_LOST and remaining SACKED_RETRANS
|
|
|
|
* bits are handled if the Loss state is really to be entered (in
|
|
|
|
* tcp_enter_frto_loss).
|
2007-02-22 07:02:30 +00:00
|
|
|
*
|
|
|
|
* Do like tcp_enter_loss() would; when RTO expires the second time it
|
|
|
|
* does:
|
|
|
|
* "Reduce ssthresh if it has not yet been made inside this window."
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
void tcp_enter_frto(struct sock *sk)
|
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
2007-02-22 07:02:30 +00:00
|
|
|
if ((!tp->frto_counter && icsk->icsk_ca_state <= TCP_CA_Disorder) ||
|
2007-02-09 14:24:47 +00:00
|
|
|
tp->snd_una == tp->high_seq ||
|
2007-02-22 07:02:30 +00:00
|
|
|
((icsk->icsk_ca_state == TCP_CA_Loss || tp->frto_counter) &&
|
|
|
|
!icsk->icsk_retransmits)) {
|
2005-08-10 07:03:31 +00:00
|
|
|
tp->prior_ssthresh = tcp_current_ssthresh(sk);
|
2007-02-22 07:13:47 +00:00
|
|
|
/* Our state is too optimistic in ssthresh() call because cwnd
|
2007-10-26 06:03:52 +00:00
|
|
|
* is not reduced until tcp_enter_frto_loss() when previous F-RTO
|
2007-02-22 07:13:47 +00:00
|
|
|
* recovery has not yet completed. Pattern would be this: RTO,
|
|
|
|
* Cumulative ACK, RTO (2xRTO for the same segment does not end
|
|
|
|
* up here twice).
|
|
|
|
* RFC4138 should be more specific on what to do, even though
|
|
|
|
* RTO is quite unlikely to occur after the first Cumulative ACK
|
|
|
|
* due to back-off and complexity of triggering events ...
|
|
|
|
*/
|
|
|
|
if (tp->frto_counter) {
|
|
|
|
u32 stored_cwnd;
|
|
|
|
stored_cwnd = tp->snd_cwnd;
|
|
|
|
tp->snd_cwnd = 2;
|
|
|
|
tp->snd_ssthresh = icsk->icsk_ca_ops->ssthresh(sk);
|
|
|
|
tp->snd_cwnd = stored_cwnd;
|
|
|
|
} else {
|
|
|
|
tp->snd_ssthresh = icsk->icsk_ca_ops->ssthresh(sk);
|
|
|
|
}
|
|
|
|
/* ... in theory, cong.control module could do "any tricks" in
|
|
|
|
* ssthresh(), which means that ca_state, lost bits and lost_out
|
|
|
|
* counter would have to be faked before the call occurs. We
|
|
|
|
* consider that too expensive, unlikely and hacky, so modules
|
|
|
|
* using these in ssthresh() must deal these incompatibility
|
|
|
|
* issues if they receives CA_EVENT_FRTO and frto_counter != 0
|
|
|
|
*/
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_ca_event(sk, CA_EVENT_FRTO);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
tp->undo_marker = tp->snd_una;
|
|
|
|
tp->undo_retrans = 0;
|
|
|
|
|
2007-03-07 20:12:44 +00:00
|
|
|
skb = tcp_write_queue_head(sk);
|
2007-09-20 18:34:38 +00:00
|
|
|
if (TCP_SKB_CB(skb)->sacked & TCPCB_RETRANS)
|
|
|
|
tp->undo_marker = 0;
|
2007-02-22 07:11:57 +00:00
|
|
|
if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_RETRANS) {
|
2007-02-22 06:54:52 +00:00
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_SACKED_RETRANS;
|
2007-02-22 07:11:57 +00:00
|
|
|
tp->retrans_out -= tcp_skb_pcount(skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2007-08-09 11:44:16 +00:00
|
|
|
tcp_verify_left_out(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-11-14 05:01:23 +00:00
|
|
|
/* Too bad if TCP was application limited */
|
|
|
|
tp->snd_cwnd = min(tp->snd_cwnd, tcp_packets_in_flight(tp) + 1);
|
|
|
|
|
2007-02-22 07:16:11 +00:00
|
|
|
/* Earlier loss recovery underway (see RFC4138; Appendix B).
|
|
|
|
* The last condition is necessary at least in tp->frto_counter case.
|
|
|
|
*/
|
tcp FRTO: SACK variant is errorneously used with NewReno
Note: there's actually another bug in FRTO's SACK variant, which
is the causing failure in NewReno case because of the error
that's fixed here. I'll fix the SACK case separately (it's
a separate bug really, though related, but in order to fix that
I need to audit tp->snd_nxt usage a bit).
There were two places where SACK variant of FRTO is getting
incorrectly used even if SACK wasn't negotiated by the TCP flow.
This leads to incorrect setting of frto_highmark with NewReno
if a previous recovery was interrupted by another RTO.
An eventual fallback to conventional recovery then incorrectly
considers one or couple of segments as forward transmissions
though they weren't, which then are not LOST marked during
fallback making them "non-retransmittable" until the next RTO.
In a bad case, those segments are really lost and are the only
one left in the window. Thus TCP needs another RTO to continue.
The next FRTO, however, could again repeat the same events
making the progress of the TCP flow extremely slow.
In order for these events to occur at all, FRTO must occur
again in FRTOs step 3 while the key segments must be lost as
well, which is not too likely in practice. It seems to most
frequently with some small devices such as network printers
that *seem* to accept TCP segments only in-order. In cases
were key segments weren't lost, things get automatically
resolved because those wrongly marked segments don't need to be
retransmitted in order to continue.
I found a reproducer after digging up relevant reports (few
reports in total, none at netdev or lkml I know of), some
cases seemed to indicate middlebox issues which seems now
to be a false assumption some people had made. Bugzilla
#10063 _might_ be related. Damon L. Chesser <damon@damtek.com>
had a reproducable case and was kind enough to tcpdump it
for me. With the tcpdump log it was quite trivial to figure
out.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-05-08 08:09:11 +00:00
|
|
|
if (tcp_is_sackfrto(tp) && (tp->frto_counter ||
|
2007-02-22 07:16:11 +00:00
|
|
|
((1 << icsk->icsk_ca_state) & (TCPF_CA_Recovery|TCPF_CA_Loss))) &&
|
|
|
|
after(tp->high_seq, tp->snd_una)) {
|
|
|
|
tp->frto_highmark = tp->high_seq;
|
|
|
|
} else {
|
|
|
|
tp->frto_highmark = tp->snd_nxt;
|
|
|
|
}
|
2007-02-22 07:03:35 +00:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Disorder);
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
2007-02-22 07:02:30 +00:00
|
|
|
tp->frto_counter = 1;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Enter Loss state after F-RTO was applied. Dupack arrived after RTO,
|
|
|
|
* which indicates that we should follow the traditional RTO recovery,
|
|
|
|
* i.e. mark everything lost and do go-back-N retransmission.
|
|
|
|
*/
|
2007-02-22 07:11:57 +00:00
|
|
|
static void tcp_enter_frto_loss(struct sock *sk, int allowed_segments, int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
tp->lost_out = 0;
|
2007-02-22 07:11:57 +00:00
|
|
|
tp->retrans_out = 0;
|
2007-08-09 12:14:46 +00:00
|
|
|
if (tcp_is_reno(tp))
|
2007-05-27 08:53:49 +00:00
|
|
|
tcp_reset_reno_sack(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-03-07 20:12:44 +00:00
|
|
|
tcp_for_write_queue(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
2007-11-14 05:03:13 +00:00
|
|
|
|
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_LOST;
|
2007-02-22 07:11:57 +00:00
|
|
|
/*
|
|
|
|
* Count the retransmission made on RTO correctly (only when
|
|
|
|
* waiting for the first ACK and did not get it)...
|
|
|
|
*/
|
2007-12-31 22:57:14 +00:00
|
|
|
if ((tp->frto_counter == 1) && !(flag & FLAG_DATA_ACKED)) {
|
2007-07-15 07:19:29 +00:00
|
|
|
/* For some reason this R-bit might get cleared? */
|
|
|
|
if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_RETRANS)
|
|
|
|
tp->retrans_out += tcp_skb_pcount(skb);
|
2007-02-22 07:11:57 +00:00
|
|
|
/* ...enter this if branch just for the first segment */
|
|
|
|
flag |= FLAG_DATA_ACKED;
|
|
|
|
} else {
|
2007-09-20 18:34:38 +00:00
|
|
|
if (TCP_SKB_CB(skb)->sacked & TCPCB_RETRANS)
|
|
|
|
tp->undo_marker = 0;
|
2007-11-14 05:03:13 +00:00
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_SACKED_RETRANS;
|
2007-02-22 07:11:57 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2008-05-13 09:54:19 +00:00
|
|
|
/* Marking forward transmissions that were made after RTO lost
|
|
|
|
* can cause unnecessary retransmissions in some scenarios,
|
|
|
|
* SACK blocks will mitigate that in some but not in all cases.
|
|
|
|
* We used to not mark them but it was causing break-ups with
|
|
|
|
* receivers that do only in-order receival.
|
|
|
|
*
|
|
|
|
* TODO: we could detect presence of such receiver and select
|
|
|
|
* different behavior per flow.
|
|
|
|
*/
|
|
|
|
if (!(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED)) {
|
2007-05-27 08:53:49 +00:00
|
|
|
TCP_SKB_CB(skb)->sacked |= TCPCB_LOST;
|
|
|
|
tp->lost_out += tcp_skb_pcount(skb);
|
2008-09-21 04:20:20 +00:00
|
|
|
tp->retransmit_high = TCP_SKB_CB(skb)->end_seq;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
2007-08-09 11:44:16 +00:00
|
|
|
tcp_verify_left_out(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-02-22 07:05:18 +00:00
|
|
|
tp->snd_cwnd = tcp_packets_in_flight(tp) + allowed_segments;
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->snd_cwnd_cnt = 0;
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
tp->frto_counter = 0;
|
2007-10-12 00:32:31 +00:00
|
|
|
tp->bytes_acked = 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
tp->reordering = min_t(unsigned int, tp->reordering,
|
2007-12-31 22:57:14 +00:00
|
|
|
sysctl_tcp_reordering);
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Loss);
|
2008-05-13 09:54:19 +00:00
|
|
|
tp->high_seq = tp->snd_nxt;
|
2005-04-16 22:20:36 +00:00
|
|
|
TCP_ECN_queue_cwr(tp);
|
2005-11-11 01:14:59 +00:00
|
|
|
|
2008-09-21 04:18:32 +00:00
|
|
|
tcp_clear_all_retrans_hints(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-10-12 00:34:57 +00:00
|
|
|
static void tcp_clear_retrans_partial(struct tcp_sock *tp)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
tp->retrans_out = 0;
|
|
|
|
tp->lost_out = 0;
|
|
|
|
|
|
|
|
tp->undo_marker = 0;
|
|
|
|
tp->undo_retrans = 0;
|
|
|
|
}
|
|
|
|
|
2007-10-12 00:34:57 +00:00
|
|
|
void tcp_clear_retrans(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
tcp_clear_retrans_partial(tp);
|
|
|
|
|
|
|
|
tp->fackets_out = 0;
|
|
|
|
tp->sacked_out = 0;
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Enter Loss state. If "how" is not zero, forget all SACK information
|
|
|
|
* and reset tags completely, otherwise preserve SACKs. If receiver
|
|
|
|
* dropped its ofo queue, we will know this due to reneging detection.
|
|
|
|
*/
|
|
|
|
void tcp_enter_loss(struct sock *sk, int how)
|
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
/* Reduce ssthresh if it has not yet been made inside this window. */
|
2005-08-10 07:03:31 +00:00
|
|
|
if (icsk->icsk_ca_state <= TCP_CA_Disorder || tp->snd_una == tp->high_seq ||
|
|
|
|
(icsk->icsk_ca_state == TCP_CA_Loss && !icsk->icsk_retransmits)) {
|
|
|
|
tp->prior_ssthresh = tcp_current_ssthresh(sk);
|
|
|
|
tp->snd_ssthresh = icsk->icsk_ca_ops->ssthresh(sk);
|
|
|
|
tcp_ca_event(sk, CA_EVENT_LOSS);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
tp->snd_cwnd = 1;
|
|
|
|
tp->snd_cwnd_cnt = 0;
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
|
2005-11-11 01:09:53 +00:00
|
|
|
tp->bytes_acked = 0;
|
2007-10-12 00:34:57 +00:00
|
|
|
tcp_clear_retrans_partial(tp);
|
|
|
|
|
|
|
|
if (tcp_is_reno(tp))
|
|
|
|
tcp_reset_reno_sack(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-09-20 18:37:19 +00:00
|
|
|
if (!how) {
|
|
|
|
/* Push undo marker, if it was plain RTO and nothing
|
|
|
|
* was retransmitted. */
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->undo_marker = tp->snd_una;
|
2007-09-20 18:37:19 +00:00
|
|
|
} else {
|
2007-10-12 00:34:57 +00:00
|
|
|
tp->sacked_out = 0;
|
|
|
|
tp->fackets_out = 0;
|
2007-09-20 18:37:19 +00:00
|
|
|
}
|
2008-09-21 04:18:32 +00:00
|
|
|
tcp_clear_all_retrans_hints(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-03-07 20:12:44 +00:00
|
|
|
tcp_for_write_queue(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
2007-10-12 00:34:57 +00:00
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
if (TCP_SKB_CB(skb)->sacked & TCPCB_RETRANS)
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->undo_marker = 0;
|
|
|
|
TCP_SKB_CB(skb)->sacked &= (~TCPCB_TAGBITS)|TCPCB_SACKED_ACKED;
|
|
|
|
if (!(TCP_SKB_CB(skb)->sacked&TCPCB_SACKED_ACKED) || how) {
|
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_SACKED_ACKED;
|
|
|
|
TCP_SKB_CB(skb)->sacked |= TCPCB_LOST;
|
|
|
|
tp->lost_out += tcp_skb_pcount(skb);
|
2008-09-21 04:20:20 +00:00
|
|
|
tp->retransmit_high = TCP_SKB_CB(skb)->end_seq;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
2007-08-09 11:44:16 +00:00
|
|
|
tcp_verify_left_out(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
tp->reordering = min_t(unsigned int, tp->reordering,
|
2007-12-31 22:57:14 +00:00
|
|
|
sysctl_tcp_reordering);
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Loss);
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->high_seq = tp->snd_nxt;
|
|
|
|
TCP_ECN_queue_cwr(tp);
|
2007-10-26 06:03:52 +00:00
|
|
|
/* Abort F-RTO algorithm if one is in progress */
|
2007-05-19 20:56:57 +00:00
|
|
|
tp->frto_counter = 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-12-31 12:49:21 +00:00
|
|
|
/* If ACK arrived pointing to a remembered SACK, it means that our
|
|
|
|
* remembered SACKs do not reflect real state of receiver i.e.
|
|
|
|
* receiver _host_ is heavily congested (or buggy).
|
|
|
|
*
|
|
|
|
* Do processing similar to RTO timeout.
|
|
|
|
*/
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_check_sack_reneging(struct sock *sk, int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2007-12-31 12:49:21 +00:00
|
|
|
if (flag & FLAG_SACK_RENEGING) {
|
2005-08-10 07:03:31 +00:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPSACKRENEGING);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
tcp_enter_loss(sk, 1);
|
2005-08-10 07:03:31 +00:00
|
|
|
icsk->icsk_retransmits++;
|
2007-03-07 20:12:44 +00:00
|
|
|
tcp_retransmit_skb(sk, tcp_write_queue_head(sk));
|
2005-08-10 03:10:42 +00:00
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_RETRANS,
|
2005-08-10 07:03:31 +00:00
|
|
|
icsk->icsk_rto, TCP_RTO_MAX);
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static inline int tcp_fackets_out(const struct tcp_sock *tp)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2007-12-31 22:57:14 +00:00
|
|
|
return tcp_is_reno(tp) ? tp->sacked_out + 1 : tp->fackets_out;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-11-16 03:39:31 +00:00
|
|
|
/* Heurestics to calculate number of duplicate ACKs. There's no dupACKs
|
|
|
|
* counter when SACK is enabled (without SACK, sacked_out is used for
|
|
|
|
* that purpose).
|
|
|
|
*
|
|
|
|
* Instead, with FACK TCP uses fackets_out that includes both SACKed
|
|
|
|
* segments up to the highest received SACK block so far and holes in
|
|
|
|
* between them.
|
|
|
|
*
|
|
|
|
* With reordering, holes may still be in flight, so RFC3517 recovery
|
|
|
|
* uses pure sacked_out (total number of SACKed segments) even though
|
|
|
|
* it violates the RFC that uses duplicate ACKs, often these are equal
|
|
|
|
* but when e.g. out-of-window ACKs or packet duplication occurs,
|
|
|
|
* they differ. Since neither occurs due to loss, TCP should really
|
|
|
|
* ignore them.
|
|
|
|
*/
|
2011-10-21 09:22:42 +00:00
|
|
|
static inline int tcp_dupack_heuristics(const struct tcp_sock *tp)
|
2007-11-16 03:39:31 +00:00
|
|
|
{
|
|
|
|
return tcp_is_fack(tp) ? tp->fackets_out : tp->sacked_out + 1;
|
|
|
|
}
|
|
|
|
|
2012-05-02 13:30:04 +00:00
|
|
|
static bool tcp_pause_early_retransmit(struct sock *sk, int flag)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
unsigned long delay;
|
|
|
|
|
|
|
|
/* Delay early retransmit and entering fast recovery for
|
|
|
|
* max(RTT/4, 2msec) unless ack has ECE mark, no RTT samples
|
|
|
|
* available, or RTO is scheduled to fire first.
|
|
|
|
*/
|
|
|
|
if (sysctl_tcp_early_retrans < 2 || (flag & FLAG_ECE) || !tp->srtt)
|
|
|
|
return false;
|
|
|
|
|
|
|
|
delay = max_t(unsigned long, (tp->srtt >> 5), msecs_to_jiffies(2));
|
|
|
|
if (!time_after(inet_csk(sk)->icsk_timeout, (jiffies + delay)))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_RETRANS, delay, TCP_RTO_MAX);
|
|
|
|
tp->early_retrans_delayed = 1;
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static inline int tcp_skb_timedout(const struct sock *sk,
|
|
|
|
const struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2010-09-22 20:43:57 +00:00
|
|
|
return tcp_time_stamp - TCP_SKB_CB(skb)->when > inet_csk(sk)->icsk_rto;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static inline int tcp_head_timedout(const struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2011-10-21 09:22:42 +00:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
return tp->packets_out &&
|
2007-03-07 20:12:44 +00:00
|
|
|
tcp_skb_timedout(sk, tcp_write_queue_head(sk));
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Linux NewReno/SACK/FACK/ECN state machine.
|
|
|
|
* --------------------------------------
|
|
|
|
*
|
|
|
|
* "Open" Normal state, no dubious events, fast path.
|
|
|
|
* "Disorder" In all the respects it is "Open",
|
|
|
|
* but requires a bit more attention. It is entered when
|
|
|
|
* we see some SACKs or dupacks. It is split of "Open"
|
|
|
|
* mainly to move some processing from fast path to slow one.
|
|
|
|
* "CWR" CWND was reduced due to some Congestion Notification event.
|
|
|
|
* It can be ECN, ICMP source quench, local device congestion.
|
|
|
|
* "Recovery" CWND was reduced, we are fast-retransmitting.
|
|
|
|
* "Loss" CWND was reduced due to RTO timeout or SACK reneging.
|
|
|
|
*
|
|
|
|
* tcp_fastretrans_alert() is entered:
|
|
|
|
* - each incoming ACK, if state is not "Open"
|
|
|
|
* - when arrived ACK is unusual, namely:
|
|
|
|
* * SACK
|
|
|
|
* * Duplicate ACK.
|
|
|
|
* * ECN ECE.
|
|
|
|
*
|
|
|
|
* Counting packets in flight is pretty simple.
|
|
|
|
*
|
|
|
|
* in_flight = packets_out - left_out + retrans_out
|
|
|
|
*
|
|
|
|
* packets_out is SND.NXT-SND.UNA counted in packets.
|
|
|
|
*
|
|
|
|
* retrans_out is number of retransmitted segments.
|
|
|
|
*
|
|
|
|
* left_out is number of segments left network, but not ACKed yet.
|
|
|
|
*
|
|
|
|
* left_out = sacked_out + lost_out
|
|
|
|
*
|
|
|
|
* sacked_out: Packets, which arrived to receiver out of order
|
|
|
|
* and hence not ACKed. With SACKs this number is simply
|
|
|
|
* amount of SACKed data. Even without SACKs
|
|
|
|
* it is easy to give pretty reliable estimate of this number,
|
|
|
|
* counting duplicate ACKs.
|
|
|
|
*
|
|
|
|
* lost_out: Packets lost by network. TCP has no explicit
|
|
|
|
* "loss notification" feedback from network (for now).
|
|
|
|
* It means that this number can be only _guessed_.
|
|
|
|
* Actually, it is the heuristics to predict lossage that
|
|
|
|
* distinguishes different algorithms.
|
|
|
|
*
|
|
|
|
* F.e. after RTO, when all the queue is considered as lost,
|
|
|
|
* lost_out = packets_out and in_flight = retrans_out.
|
|
|
|
*
|
|
|
|
* Essentially, we have now two algorithms counting
|
|
|
|
* lost packets.
|
|
|
|
*
|
|
|
|
* FACK: It is the simplest heuristics. As soon as we decided
|
|
|
|
* that something is lost, we decide that _all_ not SACKed
|
|
|
|
* packets until the most forward SACK are lost. I.e.
|
|
|
|
* lost_out = fackets_out - sacked_out and left_out = fackets_out.
|
|
|
|
* It is absolutely correct estimate, if network does not reorder
|
|
|
|
* packets. And it loses any connection to reality when reordering
|
|
|
|
* takes place. We use FACK by default until reordering
|
|
|
|
* is suspected on the path to this destination.
|
|
|
|
*
|
|
|
|
* NewReno: when Recovery is entered, we assume that one segment
|
|
|
|
* is lost (classic Reno). While we are in Recovery and
|
|
|
|
* a partial ACK arrives, we assume that one more packet
|
|
|
|
* is lost (NewReno). This heuristics are the same in NewReno
|
|
|
|
* and SACK.
|
|
|
|
*
|
|
|
|
* Imagine, that's all! Forget about all this shamanism about CWND inflation
|
|
|
|
* deflation etc. CWND is real congestion window, never inflated, changes
|
|
|
|
* only according to classic VJ rules.
|
|
|
|
*
|
|
|
|
* Really tricky (and requiring careful tuning) part of algorithm
|
|
|
|
* is hidden in functions tcp_time_to_recover() and tcp_xmit_retransmit_queue().
|
|
|
|
* The first determines the moment _when_ we should reduce CWND and,
|
|
|
|
* hence, slow down forward transmission. In fact, it determines the moment
|
|
|
|
* when we decide that hole is caused by loss, rather than by a reorder.
|
|
|
|
*
|
|
|
|
* tcp_xmit_retransmit_queue() decides, _what_ we should retransmit to fill
|
|
|
|
* holes, caused by lost packets.
|
|
|
|
*
|
|
|
|
* And the most logically complicated part of algorithm is undo
|
|
|
|
* heuristics. We detect false retransmits due to both too early
|
|
|
|
* fast retransmit (reordering) and underestimated RTO, analyzing
|
|
|
|
* timestamps and D-SACKs. When we detect that some segments were
|
|
|
|
* retransmitted by mistake and CWND reduction was wrong, we undo
|
|
|
|
* window reduction and abort recovery phase. This logic is hidden
|
|
|
|
* inside several functions named tcp_try_undo_<something>.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* This function decides, when we should leave Disordered state
|
|
|
|
* and enter Recovery phase, reducing congestion window.
|
|
|
|
*
|
|
|
|
* Main question: may we further continue forward transmission
|
|
|
|
* with the same cwnd?
|
|
|
|
*/
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_time_to_recover(struct sock *sk, int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
__u32 packets_out;
|
|
|
|
|
2007-10-26 06:03:52 +00:00
|
|
|
/* Do not perform any recovery during F-RTO algorithm */
|
2007-02-22 07:06:52 +00:00
|
|
|
if (tp->frto_counter)
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2007-02-22 07:06:52 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Trick#1: The loss is proven. */
|
|
|
|
if (tp->lost_out)
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Not-A-Trick#2 : Classic rule... */
|
2009-10-27 03:27:21 +00:00
|
|
|
if (tcp_dupack_heuristics(tp) > tp->reordering)
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Trick#3 : when we use RFC2988 timer restart, fast
|
|
|
|
* retransmit can be triggered by timeout of queue head.
|
|
|
|
*/
|
2007-11-16 03:39:31 +00:00
|
|
|
if (tcp_is_fack(tp) && tcp_head_timedout(sk))
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Trick#4: It is still not OK... But will it be useful to delay
|
|
|
|
* recovery more?
|
|
|
|
*/
|
|
|
|
packets_out = tp->packets_out;
|
|
|
|
if (packets_out <= tp->reordering &&
|
|
|
|
tp->sacked_out >= max_t(__u32, packets_out/2, sysctl_tcp_reordering) &&
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
!tcp_may_send_now(sk)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* We have nothing to send. This connection is limited
|
|
|
|
* either by receiver window or by application.
|
|
|
|
*/
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2010-02-18 04:48:19 +00:00
|
|
|
/* If a thin stream is detected, retransmit after first
|
|
|
|
* received dupack. Employ only if SACK is supported in order
|
|
|
|
* to avoid possible corner-case series of spurious retransmissions
|
|
|
|
* Use only if there are no unsent data.
|
|
|
|
*/
|
|
|
|
if ((tp->thin_dupack || sysctl_tcp_thin_dupack) &&
|
|
|
|
tcp_stream_is_thin(tp) && tcp_dupack_heuristics(tp) > 1 &&
|
|
|
|
tcp_is_sack(tp) && !tcp_send_head(sk))
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2010-02-18 04:48:19 +00:00
|
|
|
|
2012-05-02 13:30:03 +00:00
|
|
|
/* Trick#6: TCP early retransmit, per RFC5827. To avoid spurious
|
|
|
|
* retransmissions due to small network reorderings, we implement
|
|
|
|
* Mitigation A.3 in the RFC and delay the retransmission for a short
|
|
|
|
* interval if appropriate.
|
|
|
|
*/
|
|
|
|
if (tp->do_early_retrans && !tp->retrans_out && tp->sacked_out &&
|
|
|
|
(tp->packets_out == (tp->sacked_out + 1) && tp->packets_out < 4) &&
|
|
|
|
!tcp_may_send_now(sk))
|
2012-05-02 13:30:04 +00:00
|
|
|
return !tcp_pause_early_retransmit(sk, flag);
|
2012-05-02 13:30:03 +00:00
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2009-02-28 04:44:33 +00:00
|
|
|
/* New heuristics: it is possible only after we switched to restart timer
|
|
|
|
* each time when something is ACKed. Hence, we can detect timed out packets
|
|
|
|
* during fast retransmit without falling to slow start.
|
|
|
|
*
|
|
|
|
* Usefulness of this as is very questionable, since we should know which of
|
|
|
|
* the segments is the next to timeout which is relatively expensive to find
|
|
|
|
* in general case unless we add some data structure just for that. The
|
|
|
|
* current approach certainly won't find the right one too often and when it
|
|
|
|
* finally does find _something_ it usually marks large part of the window
|
|
|
|
* right away (because a retransmission with a larger timestamp blocks the
|
|
|
|
* loop from advancing). -ij
|
|
|
|
*/
|
|
|
|
static void tcp_timeout_skbs(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
if (!tcp_is_fack(tp) || !tcp_head_timedout(sk))
|
|
|
|
return;
|
|
|
|
|
|
|
|
skb = tp->scoreboard_skb_hint;
|
|
|
|
if (tp->scoreboard_skb_hint == NULL)
|
|
|
|
skb = tcp_write_queue_head(sk);
|
|
|
|
|
|
|
|
tcp_for_write_queue_from(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
|
|
|
if (!tcp_skb_timedout(sk, skb))
|
|
|
|
break;
|
|
|
|
|
|
|
|
tcp_skb_mark_lost(tp, skb);
|
|
|
|
}
|
|
|
|
|
|
|
|
tp->scoreboard_skb_hint = skb;
|
|
|
|
|
|
|
|
tcp_verify_left_out(tp);
|
|
|
|
}
|
|
|
|
|
2012-01-19 14:42:21 +00:00
|
|
|
/* Detect loss in event "A" above by marking head of queue up as lost.
|
|
|
|
* For FACK or non-SACK(Reno) senders, the first "packets" number of segments
|
|
|
|
* are considered lost. For RFC3517 SACK, a segment is considered lost if it
|
|
|
|
* has at least tp->reordering SACKed seqments above it; "packets" refers to
|
|
|
|
* the maximum SACKed segments to pass before reaching this limit.
|
2007-11-16 03:39:31 +00:00
|
|
|
*/
|
2010-10-14 01:42:30 +00:00
|
|
|
static void tcp_mark_head_lost(struct sock *sk, int packets, int mark_head)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
struct sk_buff *skb;
|
2008-04-08 05:32:38 +00:00
|
|
|
int cnt, oldcnt;
|
|
|
|
int err;
|
|
|
|
unsigned int mss;
|
2012-01-19 14:42:21 +00:00
|
|
|
/* Use SACK to deduce losses of new sequences sent during recovery */
|
|
|
|
const u32 loss_high = tcp_is_sack(tp) ? tp->snd_nxt : tp->high_seq;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2008-07-26 04:43:18 +00:00
|
|
|
WARN_ON(packets > tp->packets_out);
|
2005-11-11 01:14:59 +00:00
|
|
|
if (tp->lost_skb_hint) {
|
|
|
|
skb = tp->lost_skb_hint;
|
|
|
|
cnt = tp->lost_cnt_hint;
|
2010-10-14 01:42:30 +00:00
|
|
|
/* Head already handled? */
|
|
|
|
if (mark_head && skb != tcp_write_queue_head(sk))
|
|
|
|
return;
|
2005-11-11 01:14:59 +00:00
|
|
|
} else {
|
2007-03-07 20:12:44 +00:00
|
|
|
skb = tcp_write_queue_head(sk);
|
2005-11-11 01:14:59 +00:00
|
|
|
cnt = 0;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-03-07 20:12:44 +00:00
|
|
|
tcp_for_write_queue_from(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
2005-11-11 01:14:59 +00:00
|
|
|
/* TODO: do this better */
|
|
|
|
/* this is not the most efficient way to do this... */
|
|
|
|
tp->lost_skb_hint = skb;
|
|
|
|
tp->lost_cnt_hint = cnt;
|
2007-11-16 03:39:31 +00:00
|
|
|
|
2012-01-19 14:42:21 +00:00
|
|
|
if (after(TCP_SKB_CB(skb)->end_seq, loss_high))
|
2008-04-08 05:32:38 +00:00
|
|
|
break;
|
|
|
|
|
|
|
|
oldcnt = cnt;
|
2008-01-31 04:06:02 +00:00
|
|
|
if (tcp_is_fack(tp) || tcp_is_reno(tp) ||
|
2007-11-16 03:39:31 +00:00
|
|
|
(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED))
|
|
|
|
cnt += tcp_skb_pcount(skb);
|
|
|
|
|
2008-04-08 05:32:38 +00:00
|
|
|
if (cnt > packets) {
|
2010-09-24 13:22:06 +00:00
|
|
|
if ((tcp_is_sack(tp) && !tcp_is_fack(tp)) ||
|
2012-03-02 21:36:51 +00:00
|
|
|
(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED) ||
|
2010-09-24 13:22:06 +00:00
|
|
|
(oldcnt >= packets))
|
2008-04-08 05:32:38 +00:00
|
|
|
break;
|
|
|
|
|
|
|
|
mss = skb_shinfo(skb)->gso_size;
|
|
|
|
err = tcp_fragment(sk, skb, (packets - oldcnt) * mss, mss);
|
|
|
|
if (err < 0)
|
|
|
|
break;
|
|
|
|
cnt = packets;
|
|
|
|
}
|
|
|
|
|
2008-09-21 04:19:22 +00:00
|
|
|
tcp_skb_mark_lost(tp, skb);
|
2010-10-14 01:42:30 +00:00
|
|
|
|
|
|
|
if (mark_head)
|
|
|
|
break;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2007-08-09 11:44:16 +00:00
|
|
|
tcp_verify_left_out(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Account newly detected lost packet(s) */
|
|
|
|
|
2007-11-16 03:39:31 +00:00
|
|
|
static void tcp_update_scoreboard(struct sock *sk, int fast_rexmit)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2007-11-16 03:39:31 +00:00
|
|
|
if (tcp_is_reno(tp)) {
|
2010-10-14 01:42:30 +00:00
|
|
|
tcp_mark_head_lost(sk, 1, 1);
|
2007-11-16 03:39:31 +00:00
|
|
|
} else if (tcp_is_fack(tp)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
int lost = tp->fackets_out - tp->reordering;
|
|
|
|
if (lost <= 0)
|
|
|
|
lost = 1;
|
2010-10-14 01:42:30 +00:00
|
|
|
tcp_mark_head_lost(sk, lost, 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
} else {
|
2007-11-16 03:39:31 +00:00
|
|
|
int sacked_upto = tp->sacked_out - tp->reordering;
|
2010-10-14 01:42:30 +00:00
|
|
|
if (sacked_upto >= 0)
|
|
|
|
tcp_mark_head_lost(sk, sacked_upto, 0);
|
|
|
|
else if (fast_rexmit)
|
|
|
|
tcp_mark_head_lost(sk, 1, 1);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2009-02-28 04:44:33 +00:00
|
|
|
tcp_timeout_skbs(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* CWND moderation, preventing bursts due to too big ACKs
|
|
|
|
* in dubious situations.
|
|
|
|
*/
|
|
|
|
static inline void tcp_moderate_cwnd(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
tp->snd_cwnd = min(tp->snd_cwnd,
|
2007-12-31 22:57:14 +00:00
|
|
|
tcp_packets_in_flight(tp) + tcp_max_burst(tp));
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
}
|
|
|
|
|
2006-06-06 00:30:08 +00:00
|
|
|
/* Lower bound on congestion window is slow start threshold
|
|
|
|
* unless congestion avoidance choice decides to overide it.
|
|
|
|
*/
|
|
|
|
static inline u32 tcp_cwnd_min(const struct sock *sk)
|
|
|
|
{
|
|
|
|
const struct tcp_congestion_ops *ca_ops = inet_csk(sk)->icsk_ca_ops;
|
|
|
|
|
|
|
|
return ca_ops->min_cwnd ? ca_ops->min_cwnd(sk) : tcp_sk(sk)->snd_ssthresh;
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Decrease cwnd each second ack. */
|
2007-07-31 02:48:37 +00:00
|
|
|
static void tcp_cwnd_down(struct sock *sk, int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
int decr = tp->snd_cwnd_cnt + 1;
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
if ((flag & (FLAG_ANY_PROGRESS | FLAG_DSACKING_ACK)) ||
|
|
|
|
(tcp_is_reno(tp) && !(flag & FLAG_NOT_DUP))) {
|
|
|
|
tp->snd_cwnd_cnt = decr & 1;
|
2007-07-31 02:48:37 +00:00
|
|
|
decr >>= 1;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-07-31 02:48:37 +00:00
|
|
|
if (decr && tp->snd_cwnd > tcp_cwnd_min(sk))
|
|
|
|
tp->snd_cwnd -= decr;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
tp->snd_cwnd = min(tp->snd_cwnd, tcp_packets_in_flight(tp) + 1);
|
2007-07-31 02:48:37 +00:00
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Nothing was retransmitted or returned timestamp is less
|
|
|
|
* than timestamp of the first retransmission.
|
|
|
|
*/
|
2011-10-21 09:22:42 +00:00
|
|
|
static inline int tcp_packet_delayed(const struct tcp_sock *tp)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
return !tp->retrans_stamp ||
|
|
|
|
(tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr &&
|
2008-04-21 21:46:22 +00:00
|
|
|
before(tp->rx_opt.rcv_tsecr, tp->retrans_stamp));
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Undo procedures. */
|
|
|
|
|
|
|
|
#if FASTRETRANS_DEBUG > 1
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
static void DBGUNDO(struct sock *sk, const char *msg)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
struct inet_sock *inet = inet_sk(sk);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
|
2008-04-14 11:09:36 +00:00
|
|
|
if (sk->sk_family == AF_INET) {
|
2012-05-15 14:11:54 +00:00
|
|
|
pr_debug("Undo %s %pI4/%u c%u l%u ss%u/%u p%u\n",
|
|
|
|
msg,
|
|
|
|
&inet->inet_daddr, ntohs(inet->inet_dport),
|
|
|
|
tp->snd_cwnd, tcp_left_out(tp),
|
|
|
|
tp->snd_ssthresh, tp->prior_ssthresh,
|
|
|
|
tp->packets_out);
|
2008-04-14 11:09:36 +00:00
|
|
|
}
|
2011-12-10 09:48:31 +00:00
|
|
|
#if IS_ENABLED(CONFIG_IPV6)
|
2008-04-14 11:09:36 +00:00
|
|
|
else if (sk->sk_family == AF_INET6) {
|
|
|
|
struct ipv6_pinfo *np = inet6_sk(sk);
|
2012-05-15 14:11:54 +00:00
|
|
|
pr_debug("Undo %s %pI6/%u c%u l%u ss%u/%u p%u\n",
|
|
|
|
msg,
|
|
|
|
&np->daddr, ntohs(inet->inet_dport),
|
|
|
|
tp->snd_cwnd, tcp_left_out(tp),
|
|
|
|
tp->snd_ssthresh, tp->prior_ssthresh,
|
|
|
|
tp->packets_out);
|
2008-04-14 11:09:36 +00:00
|
|
|
}
|
|
|
|
#endif
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
#else
|
|
|
|
#define DBGUNDO(x...) do { } while (0)
|
|
|
|
#endif
|
|
|
|
|
2011-03-23 02:37:11 +00:00
|
|
|
static void tcp_undo_cwr(struct sock *sk, const bool undo_ssthresh)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
if (tp->prior_ssthresh) {
|
2005-08-10 07:03:31 +00:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
if (icsk->icsk_ca_ops->undo_cwnd)
|
|
|
|
tp->snd_cwnd = icsk->icsk_ca_ops->undo_cwnd(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
else
|
2007-12-31 22:57:14 +00:00
|
|
|
tp->snd_cwnd = max(tp->snd_cwnd, tp->snd_ssthresh << 1);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-03-14 10:57:03 +00:00
|
|
|
if (undo_ssthresh && tp->prior_ssthresh > tp->snd_ssthresh) {
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->snd_ssthresh = tp->prior_ssthresh;
|
|
|
|
TCP_ECN_withdraw_cwr(tp);
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
tp->snd_cwnd = max(tp->snd_cwnd, tp->snd_ssthresh);
|
|
|
|
}
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
}
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static inline int tcp_may_undo(const struct tcp_sock *tp)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2007-12-31 22:57:14 +00:00
|
|
|
return tp->undo_marker && (!tp->undo_retrans || tcp_packet_delayed(tp));
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* People celebrate: "We love our President!" */
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_try_undo_recovery(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
if (tcp_may_undo(tp)) {
|
2008-07-03 08:05:41 +00:00
|
|
|
int mib_idx;
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Happy end! We did not retransmit anything
|
|
|
|
* or our original transmission succeeded.
|
|
|
|
*/
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
DBGUNDO(sk, inet_csk(sk)->icsk_ca_state == TCP_CA_Loss ? "loss" : "retrans");
|
2011-03-23 02:37:11 +00:00
|
|
|
tcp_undo_cwr(sk, true);
|
2005-08-10 07:03:31 +00:00
|
|
|
if (inet_csk(sk)->icsk_ca_state == TCP_CA_Loss)
|
2008-07-03 08:05:41 +00:00
|
|
|
mib_idx = LINUX_MIB_TCPLOSSUNDO;
|
2005-04-16 22:20:36 +00:00
|
|
|
else
|
2008-07-03 08:05:41 +00:00
|
|
|
mib_idx = LINUX_MIB_TCPFULLUNDO;
|
|
|
|
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), mib_idx);
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->undo_marker = 0;
|
|
|
|
}
|
2007-08-09 12:14:46 +00:00
|
|
|
if (tp->snd_una == tp->high_seq && tcp_is_reno(tp)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Hold old state until something *above* high_seq
|
|
|
|
* is ACKed. For Reno it is MUST to prevent false
|
|
|
|
* fast retransmits (RFC2582). SACK TCP is safe. */
|
|
|
|
tcp_moderate_cwnd(tp);
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Open);
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Try to undo cwnd reduction, because D-SACKs acked all retransmitted data */
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
static void tcp_try_undo_dsack(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
if (tp->undo_marker && !tp->undo_retrans) {
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
DBGUNDO(sk, "D-SACK");
|
2011-03-23 02:37:11 +00:00
|
|
|
tcp_undo_cwr(sk, true);
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->undo_marker = 0;
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPDSACKUNDO);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2009-12-09 04:54:11 +00:00
|
|
|
/* We can clear retrans_stamp when there are no retransmissions in the
|
|
|
|
* window. It would seem that it is trivially available for us in
|
|
|
|
* tp->retrans_out, however, that kind of assumptions doesn't consider
|
|
|
|
* what will happen if errors occur when sending retransmission for the
|
|
|
|
* second time. ...It could the that such segment has only
|
|
|
|
* TCPCB_EVER_RETRANS set at the present time. It seems that checking
|
|
|
|
* the head skb is enough except for some reneging corner cases that
|
|
|
|
* are not worth the effort.
|
|
|
|
*
|
|
|
|
* Main reason for all this complexity is the fact that connection dying
|
|
|
|
* time now depends on the validity of the retrans_stamp, in particular,
|
|
|
|
* that successive retransmissions of a segment must not advance
|
|
|
|
* retrans_stamp under any conditions.
|
|
|
|
*/
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_any_retrans_done(const struct sock *sk)
|
2009-12-09 04:54:11 +00:00
|
|
|
{
|
2011-10-21 09:22:42 +00:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2009-12-09 04:54:11 +00:00
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
if (tp->retrans_out)
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2009-12-09 04:54:11 +00:00
|
|
|
|
|
|
|
skb = tcp_write_queue_head(sk);
|
|
|
|
if (unlikely(skb && TCP_SKB_CB(skb)->sacked & TCPCB_EVER_RETRANS))
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2009-12-09 04:54:11 +00:00
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2009-12-09 04:54:11 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Undo during fast recovery after partial ACK. */
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
static int tcp_try_undo_partial(struct sock *sk, int acked)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Partial ACK arrived. Force Hoe's retransmit. */
|
2007-11-16 03:39:31 +00:00
|
|
|
int failed = tcp_is_reno(tp) || (tcp_fackets_out(tp) > tp->reordering);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (tcp_may_undo(tp)) {
|
|
|
|
/* Plain luck! Hole if filled with delayed
|
|
|
|
* packet, rather than with a retransmit.
|
|
|
|
*/
|
2009-12-09 04:54:11 +00:00
|
|
|
if (!tcp_any_retrans_done(sk))
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_update_reordering(sk, tcp_fackets_out(tp) + acked, 1);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
DBGUNDO(sk, "Hoe");
|
2011-03-23 02:37:11 +00:00
|
|
|
tcp_undo_cwr(sk, false);
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPPARTIALUNDO);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* So... Do not make Hoe's retransmit yet.
|
|
|
|
* If the first packet was delayed, the rest
|
|
|
|
* ones are most probably delayed as well.
|
|
|
|
*/
|
|
|
|
failed = 0;
|
|
|
|
}
|
|
|
|
return failed;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Undo during loss recovery after partial ACK. */
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_try_undo_loss(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
if (tcp_may_undo(tp)) {
|
|
|
|
struct sk_buff *skb;
|
2007-03-07 20:12:44 +00:00
|
|
|
tcp_for_write_queue(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
2005-04-16 22:20:36 +00:00
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_LOST;
|
|
|
|
}
|
2005-11-11 01:14:59 +00:00
|
|
|
|
2007-09-20 18:30:48 +00:00
|
|
|
tcp_clear_all_retrans_hints(tp);
|
2005-11-11 01:14:59 +00:00
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
DBGUNDO(sk, "partial loss");
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->lost_out = 0;
|
2011-03-23 02:37:11 +00:00
|
|
|
tcp_undo_cwr(sk, true);
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPLOSSUNDO);
|
2005-08-10 03:10:42 +00:00
|
|
|
inet_csk(sk)->icsk_retransmits = 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->undo_marker = 0;
|
2007-08-09 12:14:46 +00:00
|
|
|
if (tcp_is_sack(tp))
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Open);
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
static inline void tcp_complete_cwr(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2011-08-21 20:21:57 +00:00
|
|
|
|
|
|
|
/* Do not moderate cwnd if it's already undone in cwr or recovery. */
|
|
|
|
if (tp->undo_marker) {
|
2012-04-30 06:00:18 +00:00
|
|
|
if (inet_csk(sk)->icsk_ca_state == TCP_CA_CWR) {
|
2011-08-21 20:21:57 +00:00
|
|
|
tp->snd_cwnd = min(tp->snd_cwnd, tp->snd_ssthresh);
|
2012-04-30 06:00:18 +00:00
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
} else if (tp->snd_ssthresh < TCP_INFINITE_SSTHRESH) {
|
|
|
|
/* PRR algorithm. */
|
2011-08-21 20:21:57 +00:00
|
|
|
tp->snd_cwnd = tp->snd_ssthresh;
|
2012-04-30 06:00:18 +00:00
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
}
|
2011-03-14 10:57:03 +00:00
|
|
|
}
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_ca_event(sk, CA_EVENT_COMPLETE_CWR);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2008-06-04 18:34:22 +00:00
|
|
|
static void tcp_try_keep_open(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int state = TCP_CA_Open;
|
|
|
|
|
2011-11-16 08:58:04 +00:00
|
|
|
if (tcp_left_out(tp) || tcp_any_retrans_done(sk))
|
2008-06-04 18:34:22 +00:00
|
|
|
state = TCP_CA_Disorder;
|
|
|
|
|
|
|
|
if (inet_csk(sk)->icsk_ca_state != state) {
|
|
|
|
tcp_set_ca_state(sk, state);
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
static void tcp_try_to_open(struct sock *sk, int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2007-08-09 11:45:17 +00:00
|
|
|
tcp_verify_left_out(tp);
|
|
|
|
|
2009-12-09 04:54:11 +00:00
|
|
|
if (!tp->frto_counter && !tcp_any_retrans_done(sk))
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
if (flag & FLAG_ECE)
|
2007-02-27 18:09:49 +00:00
|
|
|
tcp_enter_cwr(sk, 1);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
if (inet_csk(sk)->icsk_ca_state != TCP_CA_CWR) {
|
2008-06-04 18:34:22 +00:00
|
|
|
tcp_try_keep_open(sk);
|
2011-11-16 08:58:05 +00:00
|
|
|
if (inet_csk(sk)->icsk_ca_state != TCP_CA_Open)
|
|
|
|
tcp_moderate_cwnd(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
} else {
|
2007-07-31 02:48:37 +00:00
|
|
|
tcp_cwnd_down(sk, flag);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2006-03-21 01:53:41 +00:00
|
|
|
static void tcp_mtup_probe_failed(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
icsk->icsk_mtup.search_high = icsk->icsk_mtup.probe_size - 1;
|
|
|
|
icsk->icsk_mtup.probe_size = 0;
|
|
|
|
}
|
|
|
|
|
2009-03-14 14:23:04 +00:00
|
|
|
static void tcp_mtup_probe_success(struct sock *sk)
|
2006-03-21 01:53:41 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
/* FIXME: breaks with very large cwnd */
|
|
|
|
tp->prior_ssthresh = tcp_current_ssthresh(sk);
|
|
|
|
tp->snd_cwnd = tp->snd_cwnd *
|
|
|
|
tcp_mss_to_mtu(sk, tp->mss_cache) /
|
|
|
|
icsk->icsk_mtup.probe_size;
|
|
|
|
tp->snd_cwnd_cnt = 0;
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
2010-10-07 04:18:02 +00:00
|
|
|
tp->snd_ssthresh = tcp_current_ssthresh(sk);
|
2006-03-21 01:53:41 +00:00
|
|
|
|
|
|
|
icsk->icsk_mtup.search_low = icsk->icsk_mtup.probe_size;
|
|
|
|
icsk->icsk_mtup.probe_size = 0;
|
|
|
|
tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
|
|
|
|
}
|
|
|
|
|
2008-11-25 05:11:55 +00:00
|
|
|
/* Do a simple retransmit without using the backoff mechanisms in
|
|
|
|
* tcp_timer. This is used for path mtu discovery.
|
|
|
|
* The socket is already locked here.
|
|
|
|
*/
|
|
|
|
void tcp_simple_retransmit(struct sock *sk)
|
|
|
|
{
|
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb;
|
2009-03-14 14:23:05 +00:00
|
|
|
unsigned int mss = tcp_current_mss(sk);
|
2008-11-25 05:11:55 +00:00
|
|
|
u32 prior_lost = tp->lost_out;
|
|
|
|
|
|
|
|
tcp_for_write_queue(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
2008-12-06 06:41:26 +00:00
|
|
|
if (tcp_skb_seglen(skb) > mss &&
|
2008-11-25 05:11:55 +00:00
|
|
|
!(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED)) {
|
|
|
|
if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_RETRANS) {
|
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_SACKED_RETRANS;
|
|
|
|
tp->retrans_out -= tcp_skb_pcount(skb);
|
|
|
|
}
|
|
|
|
tcp_skb_mark_lost_uncond_verify(tp, skb);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
tcp_clear_retrans_hints_partial(tp);
|
|
|
|
|
|
|
|
if (prior_lost == tp->lost_out)
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (tcp_is_reno(tp))
|
|
|
|
tcp_limit_reno_sacked(tp);
|
|
|
|
|
|
|
|
tcp_verify_left_out(tp);
|
|
|
|
|
|
|
|
/* Don't muck with the congestion window here.
|
|
|
|
* Reason is that we do not increase amount of _data_
|
|
|
|
* in network, but units changed and effective
|
|
|
|
* cwnd/ssthresh really reduced now.
|
|
|
|
*/
|
|
|
|
if (icsk->icsk_ca_state != TCP_CA_Loss) {
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
|
|
|
tp->snd_ssthresh = tcp_current_ssthresh(sk);
|
|
|
|
tp->prior_ssthresh = 0;
|
|
|
|
tp->undo_marker = 0;
|
|
|
|
tcp_set_ca_state(sk, TCP_CA_Loss);
|
|
|
|
}
|
|
|
|
tcp_xmit_retransmit_queue(sk);
|
|
|
|
}
|
2010-07-09 21:22:10 +00:00
|
|
|
EXPORT_SYMBOL(tcp_simple_retransmit);
|
2008-11-25 05:11:55 +00:00
|
|
|
|
2011-08-21 20:21:57 +00:00
|
|
|
/* This function implements the PRR algorithm, specifcally the PRR-SSRB
|
|
|
|
* (proportional rate reduction with slow start reduction bound) as described in
|
|
|
|
* http://www.ietf.org/id/draft-mathis-tcpm-proportional-rate-reduction-01.txt.
|
|
|
|
* It computes the number of packets to send (sndcnt) based on packets newly
|
|
|
|
* delivered:
|
|
|
|
* 1) If the packets in flight is larger than ssthresh, PRR spreads the
|
|
|
|
* cwnd reductions across a full RTT.
|
|
|
|
* 2) If packets in flight is lower than ssthresh (such as due to excess
|
|
|
|
* losses and/or application stalls), do not perform any further cwnd
|
|
|
|
* reductions, but instead slow start up to ssthresh.
|
|
|
|
*/
|
|
|
|
static void tcp_update_cwnd_in_recovery(struct sock *sk, int newly_acked_sacked,
|
|
|
|
int fast_rexmit, int flag)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int sndcnt = 0;
|
|
|
|
int delta = tp->snd_ssthresh - tcp_packets_in_flight(tp);
|
|
|
|
|
|
|
|
if (tcp_packets_in_flight(tp) > tp->snd_ssthresh) {
|
|
|
|
u64 dividend = (u64)tp->snd_ssthresh * tp->prr_delivered +
|
|
|
|
tp->prior_cwnd - 1;
|
|
|
|
sndcnt = div_u64(dividend, tp->prior_cwnd) - tp->prr_out;
|
|
|
|
} else {
|
|
|
|
sndcnt = min_t(int, delta,
|
|
|
|
max_t(int, tp->prr_delivered - tp->prr_out,
|
|
|
|
newly_acked_sacked) + 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
sndcnt = max(sndcnt, (fast_rexmit ? 1 : 0));
|
|
|
|
tp->snd_cwnd = tcp_packets_in_flight(tp) + sndcnt;
|
|
|
|
}
|
|
|
|
|
2012-05-02 13:30:02 +00:00
|
|
|
static void tcp_enter_recovery(struct sock *sk, bool ece_ack)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int mib_idx;
|
|
|
|
|
|
|
|
if (tcp_is_reno(tp))
|
|
|
|
mib_idx = LINUX_MIB_TCPRENORECOVERY;
|
|
|
|
else
|
|
|
|
mib_idx = LINUX_MIB_TCPSACKRECOVERY;
|
|
|
|
|
|
|
|
NET_INC_STATS_BH(sock_net(sk), mib_idx);
|
|
|
|
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
|
|
|
tp->prior_ssthresh = 0;
|
|
|
|
tp->undo_marker = tp->snd_una;
|
|
|
|
tp->undo_retrans = tp->retrans_out;
|
|
|
|
|
|
|
|
if (inet_csk(sk)->icsk_ca_state < TCP_CA_CWR) {
|
|
|
|
if (!ece_ack)
|
|
|
|
tp->prior_ssthresh = tcp_current_ssthresh(sk);
|
|
|
|
tp->snd_ssthresh = inet_csk(sk)->icsk_ca_ops->ssthresh(sk);
|
|
|
|
TCP_ECN_queue_cwr(tp);
|
|
|
|
}
|
|
|
|
|
|
|
|
tp->bytes_acked = 0;
|
|
|
|
tp->snd_cwnd_cnt = 0;
|
|
|
|
tp->prior_cwnd = tp->snd_cwnd;
|
|
|
|
tp->prr_delivered = 0;
|
|
|
|
tp->prr_out = 0;
|
|
|
|
tcp_set_ca_state(sk, TCP_CA_Recovery);
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Process an event, which can update packets-in-flight not trivially.
|
|
|
|
* Main goal of this function is to calculate new estimate for left_out,
|
|
|
|
* taking into account both packets sitting in receiver's buffer and
|
|
|
|
* packets lost by network.
|
|
|
|
*
|
|
|
|
* Besides that it does CWND reduction, when packet loss is detected
|
|
|
|
* and changes state of machine.
|
|
|
|
*
|
|
|
|
* It does _not_ decide what to send, it is made in function
|
|
|
|
* tcp_xmit_retransmit_queue().
|
|
|
|
*/
|
2011-08-21 20:21:57 +00:00
|
|
|
static void tcp_fastretrans_alert(struct sock *sk, int pkts_acked,
|
2011-11-16 08:58:01 +00:00
|
|
|
int newly_acked_sacked, bool is_dupack,
|
|
|
|
int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2007-12-31 22:57:14 +00:00
|
|
|
int do_lost = is_dupack || ((flag & FLAG_DATA_SACKED) &&
|
2007-11-16 03:39:31 +00:00
|
|
|
(tcp_fackets_out(tp) > tp->reordering));
|
2012-05-02 13:30:02 +00:00
|
|
|
int fast_rexmit = 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-12-31 12:43:32 +00:00
|
|
|
if (WARN_ON(!tp->packets_out && tp->sacked_out))
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->sacked_out = 0;
|
2007-10-09 08:24:15 +00:00
|
|
|
if (WARN_ON(!tp->sacked_out && tp->fackets_out))
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->fackets_out = 0;
|
|
|
|
|
2007-02-09 14:24:47 +00:00
|
|
|
/* Now state machine starts.
|
2005-04-16 22:20:36 +00:00
|
|
|
* A. ECE, hence prohibit cwnd undoing, the reduction is required. */
|
2007-12-31 22:57:14 +00:00
|
|
|
if (flag & FLAG_ECE)
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->prior_ssthresh = 0;
|
|
|
|
|
|
|
|
/* B. In all the states check for reneging SACKs. */
|
2007-12-31 12:49:21 +00:00
|
|
|
if (tcp_check_sack_reneging(sk, flag))
|
2005-04-16 22:20:36 +00:00
|
|
|
return;
|
|
|
|
|
2012-01-19 14:42:21 +00:00
|
|
|
/* C. Check consistency of the current state. */
|
2007-08-09 11:44:16 +00:00
|
|
|
tcp_verify_left_out(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2012-01-19 14:42:21 +00:00
|
|
|
/* D. Check state exit conditions. State can be terminated
|
2005-04-16 22:20:36 +00:00
|
|
|
* when high_seq is ACKed. */
|
2005-08-10 07:03:31 +00:00
|
|
|
if (icsk->icsk_ca_state == TCP_CA_Open) {
|
2008-07-26 04:43:18 +00:00
|
|
|
WARN_ON(tp->retrans_out != 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
} else if (!before(tp->snd_una, tp->high_seq)) {
|
2005-08-10 07:03:31 +00:00
|
|
|
switch (icsk->icsk_ca_state) {
|
2005-04-16 22:20:36 +00:00
|
|
|
case TCP_CA_Loss:
|
2005-08-10 07:03:31 +00:00
|
|
|
icsk->icsk_retransmits = 0;
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
if (tcp_try_undo_recovery(sk))
|
2005-04-16 22:20:36 +00:00
|
|
|
return;
|
|
|
|
break;
|
|
|
|
|
|
|
|
case TCP_CA_CWR:
|
|
|
|
/* CWR is to be held something *above* high_seq
|
|
|
|
* is ACKed for CWR bit to reach receiver. */
|
|
|
|
if (tp->snd_una != tp->high_seq) {
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_complete_cwr(sk);
|
|
|
|
tcp_set_ca_state(sk, TCP_CA_Open);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
|
|
|
|
case TCP_CA_Recovery:
|
2007-08-09 12:14:46 +00:00
|
|
|
if (tcp_is_reno(tp))
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_reset_reno_sack(tp);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
if (tcp_try_undo_recovery(sk))
|
2005-04-16 22:20:36 +00:00
|
|
|
return;
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_complete_cwr(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-01-19 14:42:21 +00:00
|
|
|
/* E. Process state. */
|
2005-08-10 07:03:31 +00:00
|
|
|
switch (icsk->icsk_ca_state) {
|
2005-04-16 22:20:36 +00:00
|
|
|
case TCP_CA_Recovery:
|
2007-08-03 02:46:58 +00:00
|
|
|
if (!(flag & FLAG_SND_UNA_ADVANCED)) {
|
2007-08-09 12:14:46 +00:00
|
|
|
if (tcp_is_reno(tp) && is_dupack)
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_add_reno_sack(sk);
|
2007-08-09 11:53:36 +00:00
|
|
|
} else
|
|
|
|
do_lost = tcp_try_undo_partial(sk, pkts_acked);
|
2005-04-16 22:20:36 +00:00
|
|
|
break;
|
|
|
|
case TCP_CA_Loss:
|
2007-12-31 22:57:14 +00:00
|
|
|
if (flag & FLAG_DATA_ACKED)
|
2005-08-10 07:03:31 +00:00
|
|
|
icsk->icsk_retransmits = 0;
|
2008-04-08 05:33:07 +00:00
|
|
|
if (tcp_is_reno(tp) && flag & FLAG_SND_UNA_ADVANCED)
|
|
|
|
tcp_reset_reno_sack(tp);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
if (!tcp_try_undo_loss(sk)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_moderate_cwnd(tp);
|
|
|
|
tcp_xmit_retransmit_queue(sk);
|
|
|
|
return;
|
|
|
|
}
|
2005-08-10 07:03:31 +00:00
|
|
|
if (icsk->icsk_ca_state != TCP_CA_Open)
|
2005-04-16 22:20:36 +00:00
|
|
|
return;
|
|
|
|
/* Loss is undone; fall through to processing in Open state. */
|
|
|
|
default:
|
2007-08-09 12:14:46 +00:00
|
|
|
if (tcp_is_reno(tp)) {
|
2007-08-03 02:46:58 +00:00
|
|
|
if (flag & FLAG_SND_UNA_ADVANCED)
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_reset_reno_sack(tp);
|
|
|
|
if (is_dupack)
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_add_reno_sack(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2011-11-16 08:58:04 +00:00
|
|
|
if (icsk->icsk_ca_state <= TCP_CA_Disorder)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_try_undo_dsack(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2012-05-02 13:30:04 +00:00
|
|
|
if (!tcp_time_to_recover(sk, flag)) {
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_try_to_open(sk, flag);
|
2005-04-16 22:20:36 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2006-03-21 01:53:41 +00:00
|
|
|
/* MTU probe failure: don't reduce cwnd */
|
|
|
|
if (icsk->icsk_ca_state < TCP_CA_CWR &&
|
|
|
|
icsk->icsk_mtup.probe_size &&
|
2006-03-21 05:32:58 +00:00
|
|
|
tp->snd_una == tp->mtu_probe.probe_seq_start) {
|
2006-03-21 01:53:41 +00:00
|
|
|
tcp_mtup_probe_failed(sk);
|
|
|
|
/* Restores the reduction we did in tcp_mtup_probe() */
|
|
|
|
tp->snd_cwnd++;
|
|
|
|
tcp_simple_retransmit(sk);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Otherwise enter Recovery state */
|
2012-05-02 13:30:02 +00:00
|
|
|
tcp_enter_recovery(sk, (flag & FLAG_ECE));
|
2007-11-16 03:39:31 +00:00
|
|
|
fast_rexmit = 1;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-11-16 03:39:31 +00:00
|
|
|
if (do_lost || (tcp_is_fack(tp) && tcp_head_timedout(sk)))
|
|
|
|
tcp_update_scoreboard(sk, fast_rexmit);
|
2011-08-21 20:21:57 +00:00
|
|
|
tp->prr_delivered += newly_acked_sacked;
|
|
|
|
tcp_update_cwnd_in_recovery(sk, newly_acked_sacked, fast_rexmit, flag);
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_xmit_retransmit_queue(sk);
|
|
|
|
}
|
|
|
|
|
2011-06-08 11:08:38 +00:00
|
|
|
void tcp_valid_rtt_meas(struct sock *sk, u32 seq_rtt)
|
2008-12-06 06:43:26 +00:00
|
|
|
{
|
|
|
|
tcp_rtt_estimator(sk, seq_rtt);
|
|
|
|
tcp_set_rto(sk);
|
|
|
|
inet_csk(sk)->icsk_backoff = 0;
|
|
|
|
}
|
2011-06-08 11:08:38 +00:00
|
|
|
EXPORT_SYMBOL(tcp_valid_rtt_meas);
|
2008-12-06 06:43:26 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Read draft-ietf-tcplw-high-performance before mucking
|
2005-11-11 01:13:47 +00:00
|
|
|
* with this code. (Supersedes RFC1323)
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2005-11-11 00:56:12 +00:00
|
|
|
static void tcp_ack_saw_tstamp(struct sock *sk, int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
/* RTTM Rule: A TSecr value received in a segment is used to
|
|
|
|
* update the averaged RTT measurement only if the segment
|
|
|
|
* acknowledges some new data, i.e., only if it advances the
|
|
|
|
* left edge of the send window.
|
|
|
|
*
|
|
|
|
* See draft-ietf-tcplw-high-performance-00, section 3.3.
|
|
|
|
* 1998/04/10 Andrey V. Savochkin <saw@msu.ru>
|
|
|
|
*
|
|
|
|
* Changed: reset backoff as soon as we see the first valid sample.
|
2005-11-11 01:13:47 +00:00
|
|
|
* If we do not, we get strongly overestimated rto. With timestamps
|
2005-04-16 22:20:36 +00:00
|
|
|
* samples are accepted even from very old segments: f.e., when rtt=1
|
|
|
|
* increases to 8, we retransmit 5 times and after 8 seconds delayed
|
|
|
|
* answer arrives rto becomes 120 seconds! If at least one of segments
|
|
|
|
* in window is lost... Voila. --ANK (010210)
|
|
|
|
*/
|
2005-08-10 03:10:42 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2008-12-06 06:43:26 +00:00
|
|
|
|
|
|
|
tcp_valid_rtt_meas(sk, tcp_time_stamp - tp->rx_opt.rcv_tsecr);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2005-11-11 00:56:12 +00:00
|
|
|
static void tcp_ack_no_tstamp(struct sock *sk, u32 seq_rtt, int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
/* We don't have a timestamp. Can only use
|
|
|
|
* packets that are not retransmitted to determine
|
|
|
|
* rtt estimates. Also, we must not reset the
|
|
|
|
* backoff for rto until we get a non-retransmitted
|
|
|
|
* packet. This allows us to deal with a situation
|
|
|
|
* where the network delay has increased suddenly.
|
|
|
|
* I.e. Karn's algorithm. (SIGCOMM '87, p5.)
|
|
|
|
*/
|
|
|
|
|
|
|
|
if (flag & FLAG_RETRANS_DATA_ACKED)
|
|
|
|
return;
|
|
|
|
|
2008-12-06 06:43:26 +00:00
|
|
|
tcp_valid_rtt_meas(sk, seq_rtt);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
static inline void tcp_ack_update_rtt(struct sock *sk, const int flag,
|
2005-11-11 00:56:12 +00:00
|
|
|
const s32 seq_rtt)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 03:10:42 +00:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Note that peer MAY send zero echo. In this case it is ignored. (rfc1323) */
|
|
|
|
if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
|
2005-11-11 00:56:12 +00:00
|
|
|
tcp_ack_saw_tstamp(sk, flag);
|
2005-04-16 22:20:36 +00:00
|
|
|
else if (seq_rtt >= 0)
|
2005-11-11 00:56:12 +00:00
|
|
|
tcp_ack_no_tstamp(sk, seq_rtt, flag);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-12-01 22:47:59 +00:00
|
|
|
static void tcp_cong_avoid(struct sock *sk, u32 ack, u32 in_flight)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2007-12-01 22:47:59 +00:00
|
|
|
icsk->icsk_ca_ops->cong_avoid(sk, ack, in_flight);
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_sk(sk)->snd_cwnd_stamp = tcp_time_stamp;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Restart timer after forward progress on connection.
|
|
|
|
* RFC2988 recommends to restart timer to now+rto.
|
|
|
|
*/
|
2012-05-02 13:30:04 +00:00
|
|
|
void tcp_rearm_rto(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2012-05-02 13:30:04 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
if (!tp->packets_out) {
|
2005-08-10 03:10:42 +00:00
|
|
|
inet_csk_clear_xmit_timer(sk, ICSK_TIME_RETRANS);
|
2005-04-16 22:20:36 +00:00
|
|
|
} else {
|
2012-05-02 13:30:04 +00:00
|
|
|
u32 rto = inet_csk(sk)->icsk_rto;
|
|
|
|
/* Offset the time elapsed after installing regular RTO */
|
|
|
|
if (tp->early_retrans_delayed) {
|
|
|
|
struct sk_buff *skb = tcp_write_queue_head(sk);
|
|
|
|
const u32 rto_time_stamp = TCP_SKB_CB(skb)->when + rto;
|
|
|
|
s32 delta = (s32)(rto_time_stamp - tcp_time_stamp);
|
|
|
|
/* delta may not be positive if the socket is locked
|
|
|
|
* when the delayed ER timer fires and is rescheduled.
|
|
|
|
*/
|
|
|
|
if (delta > 0)
|
|
|
|
rto = delta;
|
|
|
|
}
|
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_RETRANS, rto,
|
|
|
|
TCP_RTO_MAX);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2012-05-02 13:30:04 +00:00
|
|
|
tp->early_retrans_delayed = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* This function is called when the delayed ER timer fires. TCP enters
|
|
|
|
* fast recovery and performs fast-retransmit.
|
|
|
|
*/
|
|
|
|
void tcp_resume_early_retransmit(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
tcp_rearm_rto(sk);
|
|
|
|
|
|
|
|
/* Stop if ER is disabled after the delayed ER timer is scheduled */
|
|
|
|
if (!tp->do_early_retrans)
|
|
|
|
return;
|
|
|
|
|
|
|
|
tcp_enter_recovery(sk, false);
|
|
|
|
tcp_update_scoreboard(sk, 1);
|
|
|
|
tcp_xmit_retransmit_queue(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-09-20 18:33:43 +00:00
|
|
|
/* If we get here, the whole TSO packet has not been acked. */
|
2007-10-09 08:28:45 +00:00
|
|
|
static u32 tcp_tso_acked(struct sock *sk, struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2007-09-20 18:33:43 +00:00
|
|
|
u32 packets_acked;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-09-20 18:33:43 +00:00
|
|
|
BUG_ON(!after(TCP_SKB_CB(skb)->end_seq, tp->snd_una));
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
packets_acked = tcp_skb_pcount(skb);
|
2007-09-20 18:33:43 +00:00
|
|
|
if (tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq))
|
2005-04-16 22:20:36 +00:00
|
|
|
return 0;
|
|
|
|
packets_acked -= tcp_skb_pcount(skb);
|
|
|
|
|
|
|
|
if (packets_acked) {
|
|
|
|
BUG_ON(tcp_skb_pcount(skb) == 0);
|
2007-09-20 18:33:43 +00:00
|
|
|
BUG_ON(!before(TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq));
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-10-09 08:28:45 +00:00
|
|
|
return packets_acked;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-09-20 18:33:43 +00:00
|
|
|
/* Remove acknowledged frames from the retransmission queue. If our packet
|
|
|
|
* is before the ack sequence we can discard it as it's confirmed to have
|
|
|
|
* arrived at the other end.
|
|
|
|
*/
|
2008-10-07 21:43:06 +00:00
|
|
|
static int tcp_clean_rtx_queue(struct sock *sk, int prior_fackets,
|
|
|
|
u32 prior_snd_una)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-11-11 00:56:12 +00:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
struct sk_buff *skb;
|
2007-09-20 18:33:43 +00:00
|
|
|
u32 now = tcp_time_stamp;
|
2012-05-16 23:15:34 +00:00
|
|
|
int fully_acked = true;
|
2007-09-20 18:33:43 +00:00
|
|
|
int flag = 0;
|
2007-12-30 12:37:55 +00:00
|
|
|
u32 pkts_acked = 0;
|
2007-11-11 05:22:18 +00:00
|
|
|
u32 reord = tp->packets_out;
|
2008-09-21 04:25:52 +00:00
|
|
|
u32 prior_sacked = tp->sacked_out;
|
2007-09-20 18:33:43 +00:00
|
|
|
s32 seq_rtt = -1;
|
[TCP]: use non-delayed ACK for congestion control RTT
When a delayed ACK representing two packets arrives, there are two RTT
samples available, one for each packet. The first (in order of seq
number) will be artificially long due to the delay waiting for the
second packet, the second will trigger the ACK and so will not itself
be delayed.
According to rfc1323, the SRTT used for RTO calculation should use the
first rtt, so receivers echo the timestamp from the first packet in
the delayed ack. For congestion control however, it seems measuring
delayed ack delay is not desirable as it varies independently of
congestion.
The patch below causes seq_rtt and last_ackt to be updated with any
available later packet rtts which should have less (and hopefully
zero) delack delay. The rtt value then gets passed to
ca_ops->pkts_acked().
Where TCP_CONG_RTT_STAMP was set, effort was made to supress RTTs from
within a TSO chunk (!fully_acked), using only the final ACK (which
includes any TSO delay) to generate RTTs. This patch removes these
checks so RTTs are passed for each ACK to ca_ops->pkts_acked().
For non-delay based congestion control (cubic, h-tcp), rtt is
sometimes used for rtt-scaling. In shortening the RTT, this may make
them a little less aggressive. Delay-based schemes (eg vegas, veno,
illinois) should get a cleaner, more accurate congestion signal,
particularly for small cwnds. The congestion control module can
potentially also filter out bad RTTs due to the delayed ack alarm by
looking at the associated cnt which (where delayed acking is in use)
should probably be 1 if the alarm went off or greater if the ACK was
triggered by a packet.
Signed-off-by: Gavin McCullagh <gavin.mccullagh@nuim.ie>
Acked-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-12-30 03:11:21 +00:00
|
|
|
s32 ca_seq_rtt = -1;
|
2007-06-15 22:08:43 +00:00
|
|
|
ktime_t last_ackt = net_invalid_timestamp();
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-09-20 18:33:43 +00:00
|
|
|
while ((skb = tcp_write_queue_head(sk)) && skb != tcp_send_head(sk)) {
|
2007-02-09 14:24:47 +00:00
|
|
|
struct tcp_skb_cb *scb = TCP_SKB_CB(skb);
|
2007-12-30 12:37:55 +00:00
|
|
|
u32 acked_pcount;
|
2007-09-20 18:33:43 +00:00
|
|
|
u8 sacked = scb->sacked;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
[TCP]: use non-delayed ACK for congestion control RTT
When a delayed ACK representing two packets arrives, there are two RTT
samples available, one for each packet. The first (in order of seq
number) will be artificially long due to the delay waiting for the
second packet, the second will trigger the ACK and so will not itself
be delayed.
According to rfc1323, the SRTT used for RTO calculation should use the
first rtt, so receivers echo the timestamp from the first packet in
the delayed ack. For congestion control however, it seems measuring
delayed ack delay is not desirable as it varies independently of
congestion.
The patch below causes seq_rtt and last_ackt to be updated with any
available later packet rtts which should have less (and hopefully
zero) delack delay. The rtt value then gets passed to
ca_ops->pkts_acked().
Where TCP_CONG_RTT_STAMP was set, effort was made to supress RTTs from
within a TSO chunk (!fully_acked), using only the final ACK (which
includes any TSO delay) to generate RTTs. This patch removes these
checks so RTTs are passed for each ACK to ca_ops->pkts_acked().
For non-delay based congestion control (cubic, h-tcp), rtt is
sometimes used for rtt-scaling. In shortening the RTT, this may make
them a little less aggressive. Delay-based schemes (eg vegas, veno,
illinois) should get a cleaner, more accurate congestion signal,
particularly for small cwnds. The congestion control module can
potentially also filter out bad RTTs due to the delayed ack alarm by
looking at the associated cnt which (where delayed acking is in use)
should probably be 1 if the alarm went off or greater if the ACK was
triggered by a packet.
Signed-off-by: Gavin McCullagh <gavin.mccullagh@nuim.ie>
Acked-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-12-30 03:11:21 +00:00
|
|
|
/* Determine how many packets and what bytes were acked, tso and else */
|
2005-04-16 22:20:36 +00:00
|
|
|
if (after(scb->end_seq, tp->snd_una)) {
|
2007-10-09 08:28:45 +00:00
|
|
|
if (tcp_skb_pcount(skb) == 1 ||
|
|
|
|
!after(tp->snd_una, scb->seq))
|
|
|
|
break;
|
|
|
|
|
2007-12-30 12:37:55 +00:00
|
|
|
acked_pcount = tcp_tso_acked(sk, skb);
|
|
|
|
if (!acked_pcount)
|
2007-10-09 08:28:45 +00:00
|
|
|
break;
|
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
fully_acked = false;
|
2007-10-09 08:28:45 +00:00
|
|
|
} else {
|
2007-12-30 12:37:55 +00:00
|
|
|
acked_pcount = tcp_skb_pcount(skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-12-30 12:35:27 +00:00
|
|
|
if (sacked & TCPCB_RETRANS) {
|
|
|
|
if (sacked & TCPCB_SACKED_RETRANS)
|
2007-12-30 12:37:55 +00:00
|
|
|
tp->retrans_out -= acked_pcount;
|
2007-12-30 12:35:27 +00:00
|
|
|
flag |= FLAG_RETRANS_DATA_ACKED;
|
|
|
|
ca_seq_rtt = -1;
|
|
|
|
seq_rtt = -1;
|
2007-12-31 22:57:14 +00:00
|
|
|
if ((flag & FLAG_DATA_ACKED) || (acked_pcount > 1))
|
2007-12-30 12:35:27 +00:00
|
|
|
flag |= FLAG_NONHEAD_RETRANS_ACKED;
|
2007-11-11 05:22:18 +00:00
|
|
|
} else {
|
[TCP]: use non-delayed ACK for congestion control RTT
When a delayed ACK representing two packets arrives, there are two RTT
samples available, one for each packet. The first (in order of seq
number) will be artificially long due to the delay waiting for the
second packet, the second will trigger the ACK and so will not itself
be delayed.
According to rfc1323, the SRTT used for RTO calculation should use the
first rtt, so receivers echo the timestamp from the first packet in
the delayed ack. For congestion control however, it seems measuring
delayed ack delay is not desirable as it varies independently of
congestion.
The patch below causes seq_rtt and last_ackt to be updated with any
available later packet rtts which should have less (and hopefully
zero) delack delay. The rtt value then gets passed to
ca_ops->pkts_acked().
Where TCP_CONG_RTT_STAMP was set, effort was made to supress RTTs from
within a TSO chunk (!fully_acked), using only the final ACK (which
includes any TSO delay) to generate RTTs. This patch removes these
checks so RTTs are passed for each ACK to ca_ops->pkts_acked().
For non-delay based congestion control (cubic, h-tcp), rtt is
sometimes used for rtt-scaling. In shortening the RTT, this may make
them a little less aggressive. Delay-based schemes (eg vegas, veno,
illinois) should get a cleaner, more accurate congestion signal,
particularly for small cwnds. The congestion control module can
potentially also filter out bad RTTs due to the delayed ack alarm by
looking at the associated cnt which (where delayed acking is in use)
should probably be 1 if the alarm went off or greater if the ACK was
triggered by a packet.
Signed-off-by: Gavin McCullagh <gavin.mccullagh@nuim.ie>
Acked-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-12-30 03:11:21 +00:00
|
|
|
ca_seq_rtt = now - scb->when;
|
|
|
|
last_ackt = skb->tstamp;
|
2007-11-11 05:22:18 +00:00
|
|
|
if (seq_rtt < 0) {
|
[TCP]: use non-delayed ACK for congestion control RTT
When a delayed ACK representing two packets arrives, there are two RTT
samples available, one for each packet. The first (in order of seq
number) will be artificially long due to the delay waiting for the
second packet, the second will trigger the ACK and so will not itself
be delayed.
According to rfc1323, the SRTT used for RTO calculation should use the
first rtt, so receivers echo the timestamp from the first packet in
the delayed ack. For congestion control however, it seems measuring
delayed ack delay is not desirable as it varies independently of
congestion.
The patch below causes seq_rtt and last_ackt to be updated with any
available later packet rtts which should have less (and hopefully
zero) delack delay. The rtt value then gets passed to
ca_ops->pkts_acked().
Where TCP_CONG_RTT_STAMP was set, effort was made to supress RTTs from
within a TSO chunk (!fully_acked), using only the final ACK (which
includes any TSO delay) to generate RTTs. This patch removes these
checks so RTTs are passed for each ACK to ca_ops->pkts_acked().
For non-delay based congestion control (cubic, h-tcp), rtt is
sometimes used for rtt-scaling. In shortening the RTT, this may make
them a little less aggressive. Delay-based schemes (eg vegas, veno,
illinois) should get a cleaner, more accurate congestion signal,
particularly for small cwnds. The congestion control module can
potentially also filter out bad RTTs due to the delayed ack alarm by
looking at the associated cnt which (where delayed acking is in use)
should probably be 1 if the alarm went off or greater if the ACK was
triggered by a packet.
Signed-off-by: Gavin McCullagh <gavin.mccullagh@nuim.ie>
Acked-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-12-30 03:11:21 +00:00
|
|
|
seq_rtt = ca_seq_rtt;
|
2007-11-11 05:22:18 +00:00
|
|
|
}
|
2007-12-30 12:35:27 +00:00
|
|
|
if (!(sacked & TCPCB_SACKED_ACKED))
|
2007-12-30 12:37:55 +00:00
|
|
|
reord = min(pkts_acked, reord);
|
2005-11-11 00:56:12 +00:00
|
|
|
}
|
2007-12-30 12:35:27 +00:00
|
|
|
|
|
|
|
if (sacked & TCPCB_SACKED_ACKED)
|
2007-12-30 12:37:55 +00:00
|
|
|
tp->sacked_out -= acked_pcount;
|
2007-12-30 12:35:27 +00:00
|
|
|
if (sacked & TCPCB_LOST)
|
2007-12-30 12:37:55 +00:00
|
|
|
tp->lost_out -= acked_pcount;
|
2007-12-30 12:35:27 +00:00
|
|
|
|
2007-12-30 12:37:55 +00:00
|
|
|
tp->packets_out -= acked_pcount;
|
|
|
|
pkts_acked += acked_pcount;
|
2007-10-09 08:28:45 +00:00
|
|
|
|
2007-09-20 18:34:38 +00:00
|
|
|
/* Initial outgoing SYN's get put onto the write_queue
|
|
|
|
* just like anything else we transmit. It is not
|
|
|
|
* true data, and if we misinform our callers that
|
|
|
|
* this ACK acks real data, we will erroneously exit
|
|
|
|
* connection startup slow start one packet too
|
|
|
|
* quickly. This is severely frowned upon behavior.
|
|
|
|
*/
|
2011-09-27 17:25:05 +00:00
|
|
|
if (!(scb->tcp_flags & TCPHDR_SYN)) {
|
2007-09-20 18:34:38 +00:00
|
|
|
flag |= FLAG_DATA_ACKED;
|
|
|
|
} else {
|
|
|
|
flag |= FLAG_SYN_ACKED;
|
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
}
|
|
|
|
|
2007-10-09 08:28:45 +00:00
|
|
|
if (!fully_acked)
|
|
|
|
break;
|
|
|
|
|
2007-03-07 20:12:44 +00:00
|
|
|
tcp_unlink_write_queue(skb, sk);
|
2007-12-31 08:11:19 +00:00
|
|
|
sk_wmem_free_skb(sk, skb);
|
2008-09-21 04:25:52 +00:00
|
|
|
tp->scoreboard_skb_hint = NULL;
|
2008-09-21 04:25:15 +00:00
|
|
|
if (skb == tp->retransmit_skb_hint)
|
|
|
|
tp->retransmit_skb_hint = NULL;
|
2008-09-21 04:25:52 +00:00
|
|
|
if (skb == tp->lost_skb_hint)
|
|
|
|
tp->lost_skb_hint = NULL;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2008-10-07 21:43:06 +00:00
|
|
|
if (likely(between(tp->snd_up, prior_snd_una, tp->snd_una)))
|
|
|
|
tp->snd_up = tp->snd_una;
|
|
|
|
|
2007-12-31 12:49:21 +00:00
|
|
|
if (skb && (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED))
|
|
|
|
flag |= FLAG_SACK_RENEGING;
|
|
|
|
|
2007-09-20 18:33:43 +00:00
|
|
|
if (flag & FLAG_ACKED) {
|
2007-04-24 05:26:16 +00:00
|
|
|
const struct tcp_congestion_ops *ca_ops
|
|
|
|
= inet_csk(sk)->icsk_ca_ops;
|
|
|
|
|
2009-03-14 14:23:04 +00:00
|
|
|
if (unlikely(icsk->icsk_mtup.probe_size &&
|
|
|
|
!after(tp->mtu_probe.probe_seq_end, tp->snd_una))) {
|
|
|
|
tcp_mtup_probe_success(sk);
|
|
|
|
}
|
|
|
|
|
2007-09-20 18:33:43 +00:00
|
|
|
tcp_ack_update_rtt(sk, flag, seq_rtt);
|
2007-08-25 05:53:26 +00:00
|
|
|
tcp_rearm_rto(sk);
|
2005-06-23 19:19:55 +00:00
|
|
|
|
2007-11-11 05:22:18 +00:00
|
|
|
if (tcp_is_reno(tp)) {
|
|
|
|
tcp_remove_reno_sacks(sk, pkts_acked);
|
|
|
|
} else {
|
2009-02-28 04:44:28 +00:00
|
|
|
int delta;
|
|
|
|
|
2007-11-11 05:22:18 +00:00
|
|
|
/* Non-retransmitted hole got filled? That's reordering */
|
|
|
|
if (reord < prior_fackets)
|
|
|
|
tcp_update_reordering(sk, tp->fackets_out - reord, 0);
|
2008-09-21 04:25:52 +00:00
|
|
|
|
2009-02-28 04:44:28 +00:00
|
|
|
delta = tcp_is_fack(tp) ? pkts_acked :
|
|
|
|
prior_sacked - tp->sacked_out;
|
|
|
|
tp->lost_cnt_hint -= min(tp->lost_cnt_hint, delta);
|
2007-11-11 05:22:18 +00:00
|
|
|
}
|
|
|
|
|
2007-10-09 08:24:15 +00:00
|
|
|
tp->fackets_out -= min(pkts_acked, tp->fackets_out);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 03:50:37 +00:00
|
|
|
|
2007-07-26 06:49:34 +00:00
|
|
|
if (ca_ops->pkts_acked) {
|
|
|
|
s32 rtt_us = -1;
|
|
|
|
|
|
|
|
/* Is the ACK triggering packet unambiguous? */
|
2007-09-20 18:33:43 +00:00
|
|
|
if (!(flag & FLAG_RETRANS_DATA_ACKED)) {
|
2007-07-26 06:49:34 +00:00
|
|
|
/* High resolution needed and available? */
|
|
|
|
if (ca_ops->flags & TCP_CONG_RTT_STAMP &&
|
|
|
|
!ktime_equal(last_ackt,
|
|
|
|
net_invalid_timestamp()))
|
|
|
|
rtt_us = ktime_us_delta(ktime_get_real(),
|
|
|
|
last_ackt);
|
2011-03-14 07:52:12 +00:00
|
|
|
else if (ca_seq_rtt >= 0)
|
[TCP]: use non-delayed ACK for congestion control RTT
When a delayed ACK representing two packets arrives, there are two RTT
samples available, one for each packet. The first (in order of seq
number) will be artificially long due to the delay waiting for the
second packet, the second will trigger the ACK and so will not itself
be delayed.
According to rfc1323, the SRTT used for RTO calculation should use the
first rtt, so receivers echo the timestamp from the first packet in
the delayed ack. For congestion control however, it seems measuring
delayed ack delay is not desirable as it varies independently of
congestion.
The patch below causes seq_rtt and last_ackt to be updated with any
available later packet rtts which should have less (and hopefully
zero) delack delay. The rtt value then gets passed to
ca_ops->pkts_acked().
Where TCP_CONG_RTT_STAMP was set, effort was made to supress RTTs from
within a TSO chunk (!fully_acked), using only the final ACK (which
includes any TSO delay) to generate RTTs. This patch removes these
checks so RTTs are passed for each ACK to ca_ops->pkts_acked().
For non-delay based congestion control (cubic, h-tcp), rtt is
sometimes used for rtt-scaling. In shortening the RTT, this may make
them a little less aggressive. Delay-based schemes (eg vegas, veno,
illinois) should get a cleaner, more accurate congestion signal,
particularly for small cwnds. The congestion control module can
potentially also filter out bad RTTs due to the delayed ack alarm by
looking at the associated cnt which (where delayed acking is in use)
should probably be 1 if the alarm went off or greater if the ACK was
triggered by a packet.
Signed-off-by: Gavin McCullagh <gavin.mccullagh@nuim.ie>
Acked-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-12-30 03:11:21 +00:00
|
|
|
rtt_us = jiffies_to_usecs(ca_seq_rtt);
|
2007-07-26 06:49:34 +00:00
|
|
|
}
|
2007-06-15 22:08:43 +00:00
|
|
|
|
2007-07-26 06:49:34 +00:00
|
|
|
ca_ops->pkts_acked(sk, pkts_acked, rtt_us);
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
#if FASTRETRANS_DEBUG > 0
|
2008-07-26 04:43:18 +00:00
|
|
|
WARN_ON((int)tp->sacked_out < 0);
|
|
|
|
WARN_ON((int)tp->lost_out < 0);
|
|
|
|
WARN_ON((int)tp->retrans_out < 0);
|
2007-08-09 12:14:46 +00:00
|
|
|
if (!tp->packets_out && tcp_is_sack(tp)) {
|
2007-10-09 08:59:42 +00:00
|
|
|
icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (tp->lost_out) {
|
2012-05-15 14:11:54 +00:00
|
|
|
pr_debug("Leak l=%u %d\n",
|
|
|
|
tp->lost_out, icsk->icsk_ca_state);
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->lost_out = 0;
|
|
|
|
}
|
|
|
|
if (tp->sacked_out) {
|
2012-05-15 14:11:54 +00:00
|
|
|
pr_debug("Leak s=%u %d\n",
|
|
|
|
tp->sacked_out, icsk->icsk_ca_state);
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->sacked_out = 0;
|
|
|
|
}
|
|
|
|
if (tp->retrans_out) {
|
2012-05-15 14:11:54 +00:00
|
|
|
pr_debug("Leak r=%u %d\n",
|
|
|
|
tp->retrans_out, icsk->icsk_ca_state);
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->retrans_out = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#endif
|
2007-09-20 18:33:43 +00:00
|
|
|
return flag;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static void tcp_ack_probe(struct sock *sk)
|
|
|
|
{
|
2005-08-10 03:10:42 +00:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Was it a usable window open? */
|
|
|
|
|
2007-12-31 12:48:41 +00:00
|
|
|
if (!after(TCP_SKB_CB(tcp_send_head(sk))->end_seq, tcp_wnd_end(tp))) {
|
2005-08-10 03:10:42 +00:00
|
|
|
icsk->icsk_backoff = 0;
|
|
|
|
inet_csk_clear_xmit_timer(sk, ICSK_TIME_PROBE0);
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Socket must be waked up by subsequent tcp_data_snd_check().
|
|
|
|
* This function is not for random using!
|
|
|
|
*/
|
|
|
|
} else {
|
2005-08-10 03:10:42 +00:00
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_PROBE0,
|
2005-08-10 03:11:08 +00:00
|
|
|
min(icsk->icsk_rto << icsk->icsk_backoff, TCP_RTO_MAX),
|
|
|
|
TCP_RTO_MAX);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
static inline int tcp_ack_is_dubious(const struct sock *sk, const int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2010-09-22 20:43:57 +00:00
|
|
|
return !(flag & FLAG_NOT_DUP) || (flag & FLAG_CA_ALERT) ||
|
|
|
|
inet_csk(sk)->icsk_ca_state != TCP_CA_Open;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
static inline int tcp_may_raise_cwnd(const struct sock *sk, const int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
return (!(flag & FLAG_ECE) || tp->snd_cwnd < tp->snd_ssthresh) &&
|
2005-08-10 07:03:31 +00:00
|
|
|
!((1 << inet_csk(sk)->icsk_ca_state) & (TCPF_CA_Recovery | TCPF_CA_CWR));
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Check that window update is acceptable.
|
|
|
|
* The function assumes that snd_una<=ack<=snd_next.
|
|
|
|
*/
|
2007-12-31 22:57:14 +00:00
|
|
|
static inline int tcp_may_update_window(const struct tcp_sock *tp,
|
|
|
|
const u32 ack, const u32 ack_seq,
|
|
|
|
const u32 nwin)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2010-09-22 20:43:57 +00:00
|
|
|
return after(ack, tp->snd_una) ||
|
2005-04-16 22:20:36 +00:00
|
|
|
after(ack_seq, tp->snd_wl1) ||
|
2010-09-22 20:43:57 +00:00
|
|
|
(ack_seq == tp->snd_wl1 && nwin > tp->snd_wnd);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Update our send window.
|
|
|
|
*
|
|
|
|
* Window update algorithm, described in RFC793/RFC1122 (used in linux-2.2
|
|
|
|
* and in FreeBSD. NetBSD's one is even worse.) is wrong.
|
|
|
|
*/
|
2011-10-21 09:22:42 +00:00
|
|
|
static int tcp_ack_update_window(struct sock *sk, const struct sk_buff *skb, u32 ack,
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
u32 ack_seq)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
int flag = 0;
|
2007-04-11 04:04:22 +00:00
|
|
|
u32 nwin = ntohs(tcp_hdr(skb)->window);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-04-11 04:04:22 +00:00
|
|
|
if (likely(!tcp_hdr(skb)->syn))
|
2005-04-16 22:20:36 +00:00
|
|
|
nwin <<= tp->rx_opt.snd_wscale;
|
|
|
|
|
|
|
|
if (tcp_may_update_window(tp, ack, ack_seq, nwin)) {
|
|
|
|
flag |= FLAG_WIN_UPDATE;
|
2009-03-03 06:42:02 +00:00
|
|
|
tcp_update_wl(tp, ack_seq);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (tp->snd_wnd != nwin) {
|
|
|
|
tp->snd_wnd = nwin;
|
|
|
|
|
|
|
|
/* Note, it is the only place, where
|
|
|
|
* fast path is recovered for sending TCP.
|
|
|
|
*/
|
2005-10-27 08:47:46 +00:00
|
|
|
tp->pred_flags = 0;
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_fast_path_check(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (nwin > tp->max_window) {
|
|
|
|
tp->max_window = nwin;
|
2005-12-14 07:26:10 +00:00
|
|
|
tcp_sync_mss(sk, inet_csk(sk)->icsk_pmtu_cookie);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
tp->snd_una = ack;
|
|
|
|
|
|
|
|
return flag;
|
|
|
|
}
|
|
|
|
|
2007-02-22 06:56:19 +00:00
|
|
|
/* A very conservative spurious RTO response algorithm: reduce cwnd and
|
|
|
|
* continue in congestion avoidance.
|
|
|
|
*/
|
|
|
|
static void tcp_conservative_spur_to_response(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
tp->snd_cwnd = min(tp->snd_cwnd, tp->snd_ssthresh);
|
2007-02-22 07:06:03 +00:00
|
|
|
tp->snd_cwnd_cnt = 0;
|
2007-10-12 00:32:31 +00:00
|
|
|
tp->bytes_acked = 0;
|
2007-05-19 20:56:23 +00:00
|
|
|
TCP_ECN_queue_cwr(tp);
|
2007-02-22 06:56:19 +00:00
|
|
|
tcp_moderate_cwnd(tp);
|
|
|
|
}
|
|
|
|
|
2007-02-27 18:09:49 +00:00
|
|
|
/* A conservative spurious RTO response algorithm: reduce cwnd using
|
|
|
|
* rate halving and continue in congestion avoidance.
|
|
|
|
*/
|
|
|
|
static void tcp_ratehalving_spur_to_response(struct sock *sk)
|
|
|
|
{
|
|
|
|
tcp_enter_cwr(sk, 0);
|
|
|
|
}
|
|
|
|
|
2007-03-02 21:34:19 +00:00
|
|
|
static void tcp_undo_spur_to_response(struct sock *sk, int flag)
|
2007-02-27 18:09:49 +00:00
|
|
|
{
|
2007-12-31 22:57:14 +00:00
|
|
|
if (flag & FLAG_ECE)
|
2007-03-02 21:34:19 +00:00
|
|
|
tcp_ratehalving_spur_to_response(sk);
|
|
|
|
else
|
2011-03-23 02:37:11 +00:00
|
|
|
tcp_undo_cwr(sk, true);
|
2007-02-27 18:09:49 +00:00
|
|
|
}
|
|
|
|
|
2007-02-22 07:01:36 +00:00
|
|
|
/* F-RTO spurious RTO detection algorithm (RFC4138)
|
|
|
|
*
|
2007-02-22 07:04:11 +00:00
|
|
|
* F-RTO affects during two new ACKs following RTO (well, almost, see inline
|
|
|
|
* comments). State (ACK number) is kept in frto_counter. When ACK advances
|
|
|
|
* window (but not to or beyond highest sequence sent before RTO):
|
2007-02-22 07:01:36 +00:00
|
|
|
* On First ACK, send two new segments out.
|
|
|
|
* On Second ACK, RTO was likely spurious. Do spurious response (response
|
|
|
|
* algorithm is not part of the F-RTO detection algorithm
|
|
|
|
* given in RFC4138 but can be selected separately).
|
|
|
|
* Otherwise (basically on duplicate ACK), RTO was (likely) caused by a loss
|
2007-04-30 07:42:20 +00:00
|
|
|
* and TCP falls back to conventional RTO recovery. F-RTO allows overriding
|
|
|
|
* of Nagle, this is done using frto_counter states 2 and 3, when a new data
|
|
|
|
* segment of any size sent during F-RTO, state 2 is upgraded to 3.
|
2007-02-22 07:01:36 +00:00
|
|
|
*
|
|
|
|
* Rationale: if the RTO was spurious, new ACKs should arrive from the
|
|
|
|
* original window even after we transmit two new data segments.
|
|
|
|
*
|
2007-02-22 07:16:11 +00:00
|
|
|
* SACK version:
|
|
|
|
* on first step, wait until first cumulative ACK arrives, then move to
|
|
|
|
* the second step. In second step, the next ACK decides.
|
|
|
|
*
|
2007-02-22 07:01:36 +00:00
|
|
|
* F-RTO is implemented (mainly) in four functions:
|
|
|
|
* - tcp_use_frto() is used to determine if TCP is can use F-RTO
|
|
|
|
* - tcp_enter_frto() prepares TCP state on RTO if F-RTO is used, it is
|
|
|
|
* called when tcp_use_frto() showed green light
|
|
|
|
* - tcp_process_frto() handles incoming ACKs during F-RTO algorithm
|
|
|
|
* - tcp_enter_frto_loss() is called if there is not enough evidence
|
|
|
|
* to prove that the RTO is indeed spurious. It transfers the control
|
|
|
|
* from F-RTO to the conventional RTO recovery
|
|
|
|
*/
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_process_frto(struct sock *sk, int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2007-02-09 14:24:47 +00:00
|
|
|
|
2007-08-09 11:44:16 +00:00
|
|
|
tcp_verify_left_out(tp);
|
2007-02-09 14:24:47 +00:00
|
|
|
|
2007-02-22 07:02:30 +00:00
|
|
|
/* Duplicate the behavior from Loss state (fastretrans_alert) */
|
2007-12-31 22:57:14 +00:00
|
|
|
if (flag & FLAG_DATA_ACKED)
|
2007-02-22 07:02:30 +00:00
|
|
|
inet_csk(sk)->icsk_retransmits = 0;
|
|
|
|
|
2007-09-20 18:34:38 +00:00
|
|
|
if ((flag & FLAG_NONHEAD_RETRANS_ACKED) ||
|
|
|
|
((tp->frto_counter >= 2) && (flag & FLAG_RETRANS_DATA_ACKED)))
|
|
|
|
tp->undo_marker = 0;
|
|
|
|
|
2007-02-22 07:05:18 +00:00
|
|
|
if (!before(tp->snd_una, tp->frto_highmark)) {
|
2007-04-30 07:42:20 +00:00
|
|
|
tcp_enter_frto_loss(sk, (tp->frto_counter == 1 ? 2 : 3), flag);
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-02-22 07:05:18 +00:00
|
|
|
}
|
|
|
|
|
tcp FRTO: SACK variant is errorneously used with NewReno
Note: there's actually another bug in FRTO's SACK variant, which
is the causing failure in NewReno case because of the error
that's fixed here. I'll fix the SACK case separately (it's
a separate bug really, though related, but in order to fix that
I need to audit tp->snd_nxt usage a bit).
There were two places where SACK variant of FRTO is getting
incorrectly used even if SACK wasn't negotiated by the TCP flow.
This leads to incorrect setting of frto_highmark with NewReno
if a previous recovery was interrupted by another RTO.
An eventual fallback to conventional recovery then incorrectly
considers one or couple of segments as forward transmissions
though they weren't, which then are not LOST marked during
fallback making them "non-retransmittable" until the next RTO.
In a bad case, those segments are really lost and are the only
one left in the window. Thus TCP needs another RTO to continue.
The next FRTO, however, could again repeat the same events
making the progress of the TCP flow extremely slow.
In order for these events to occur at all, FRTO must occur
again in FRTOs step 3 while the key segments must be lost as
well, which is not too likely in practice. It seems to most
frequently with some small devices such as network printers
that *seem* to accept TCP segments only in-order. In cases
were key segments weren't lost, things get automatically
resolved because those wrongly marked segments don't need to be
retransmitted in order to continue.
I found a reproducer after digging up relevant reports (few
reports in total, none at netdev or lkml I know of), some
cases seemed to indicate middlebox issues which seems now
to be a false assumption some people had made. Bugzilla
#10063 _might_ be related. Damon L. Chesser <damon@damtek.com>
had a reproducable case and was kind enough to tcpdump it
for me. With the tcpdump log it was quite trivial to figure
out.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-05-08 08:09:11 +00:00
|
|
|
if (!tcp_is_sackfrto(tp)) {
|
2007-02-22 07:16:11 +00:00
|
|
|
/* RFC4138 shortcoming in step 2; should also have case c):
|
|
|
|
* ACK isn't duplicate nor advances window, e.g., opposite dir
|
|
|
|
* data, winupdate
|
|
|
|
*/
|
2007-12-31 22:57:14 +00:00
|
|
|
if (!(flag & FLAG_ANY_PROGRESS) && (flag & FLAG_NOT_DUP))
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-02-22 07:16:11 +00:00
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
if (!(flag & FLAG_DATA_ACKED)) {
|
2007-02-22 07:16:11 +00:00
|
|
|
tcp_enter_frto_loss(sk, (tp->frto_counter == 1 ? 0 : 3),
|
|
|
|
flag);
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-02-22 07:16:11 +00:00
|
|
|
}
|
|
|
|
} else {
|
2007-12-31 22:57:14 +00:00
|
|
|
if (!(flag & FLAG_DATA_ACKED) && (tp->frto_counter == 1)) {
|
2007-02-22 07:16:11 +00:00
|
|
|
/* Prevent sending of new data. */
|
|
|
|
tp->snd_cwnd = min(tp->snd_cwnd,
|
|
|
|
tcp_packets_in_flight(tp));
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-02-22 07:16:11 +00:00
|
|
|
}
|
2007-02-22 07:04:11 +00:00
|
|
|
|
2007-04-30 07:42:20 +00:00
|
|
|
if ((tp->frto_counter >= 2) &&
|
2007-12-31 22:57:14 +00:00
|
|
|
(!(flag & FLAG_FORWARD_PROGRESS) ||
|
|
|
|
((flag & FLAG_DATA_SACKED) &&
|
|
|
|
!(flag & FLAG_ONLY_ORIG_SACKED)))) {
|
2007-02-22 07:16:11 +00:00
|
|
|
/* RFC4138 shortcoming (see comment above) */
|
2007-12-31 22:57:14 +00:00
|
|
|
if (!(flag & FLAG_FORWARD_PROGRESS) &&
|
|
|
|
(flag & FLAG_NOT_DUP))
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-02-22 07:16:11 +00:00
|
|
|
|
|
|
|
tcp_enter_frto_loss(sk, 3, flag);
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-02-22 07:16:11 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (tp->frto_counter == 1) {
|
2007-12-05 10:20:21 +00:00
|
|
|
/* tcp_may_send_now needs to see updated state */
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->snd_cwnd = tcp_packets_in_flight(tp) + 2;
|
2007-02-22 07:07:27 +00:00
|
|
|
tp->frto_counter = 2;
|
2007-12-05 10:20:21 +00:00
|
|
|
|
|
|
|
if (!tcp_may_send_now(sk))
|
|
|
|
tcp_enter_frto_loss(sk, 2, flag);
|
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2007-04-30 07:42:20 +00:00
|
|
|
} else {
|
2007-02-27 18:09:49 +00:00
|
|
|
switch (sysctl_tcp_frto_response) {
|
|
|
|
case 2:
|
2007-03-02 21:34:19 +00:00
|
|
|
tcp_undo_spur_to_response(sk, flag);
|
2007-02-27 18:09:49 +00:00
|
|
|
break;
|
|
|
|
case 1:
|
|
|
|
tcp_conservative_spur_to_response(tp);
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
tcp_ratehalving_spur_to_response(sk);
|
|
|
|
break;
|
2007-04-21 00:09:22 +00:00
|
|
|
}
|
2007-02-22 07:07:27 +00:00
|
|
|
tp->frto_counter = 0;
|
2007-09-20 18:34:38 +00:00
|
|
|
tp->undo_marker = 0;
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPSPURIOUSRTOS);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* This routine deals with incoming acks, but not outgoing ones. */
|
2011-10-21 09:22:42 +00:00
|
|
|
static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 07:03:31 +00:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u32 prior_snd_una = tp->snd_una;
|
|
|
|
u32 ack_seq = TCP_SKB_CB(skb)->seq;
|
|
|
|
u32 ack = TCP_SKB_CB(skb)->ack_seq;
|
2011-11-16 08:58:01 +00:00
|
|
|
bool is_dupack = false;
|
2005-04-16 22:20:36 +00:00
|
|
|
u32 prior_in_flight;
|
2007-11-11 05:22:18 +00:00
|
|
|
u32 prior_fackets;
|
2005-04-16 22:20:36 +00:00
|
|
|
int prior_packets;
|
2011-08-21 20:21:57 +00:00
|
|
|
int prior_sacked = tp->sacked_out;
|
2011-11-16 08:58:01 +00:00
|
|
|
int pkts_acked = 0;
|
2011-08-21 20:21:57 +00:00
|
|
|
int newly_acked_sacked = 0;
|
2012-05-16 23:15:34 +00:00
|
|
|
bool frto_cwnd = false;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2009-03-23 04:49:57 +00:00
|
|
|
/* If the ack is older than previous acks
|
2005-04-16 22:20:36 +00:00
|
|
|
* then we can probably ignore it.
|
|
|
|
*/
|
|
|
|
if (before(ack, prior_snd_una))
|
|
|
|
goto old_ack;
|
|
|
|
|
2009-03-23 04:49:57 +00:00
|
|
|
/* If the ack includes data we haven't sent yet, discard
|
|
|
|
* this segment (RFC793 Section 3.9).
|
|
|
|
*/
|
|
|
|
if (after(ack, tp->snd_nxt))
|
|
|
|
goto invalid_ack;
|
|
|
|
|
2012-05-02 13:30:04 +00:00
|
|
|
if (tp->early_retrans_delayed)
|
|
|
|
tcp_rearm_rto(sk);
|
|
|
|
|
2007-08-03 02:46:58 +00:00
|
|
|
if (after(ack, prior_snd_una))
|
|
|
|
flag |= FLAG_SND_UNA_ADVANCED;
|
|
|
|
|
2006-08-29 09:01:44 +00:00
|
|
|
if (sysctl_tcp_abc) {
|
|
|
|
if (icsk->icsk_ca_state < TCP_CA_CWR)
|
|
|
|
tp->bytes_acked += ack - prior_snd_una;
|
|
|
|
else if (icsk->icsk_ca_state == TCP_CA_Loss)
|
|
|
|
/* we assume just one segment left network */
|
2007-12-31 22:57:14 +00:00
|
|
|
tp->bytes_acked += min(ack - prior_snd_una,
|
|
|
|
tp->mss_cache);
|
2006-08-29 09:01:44 +00:00
|
|
|
}
|
2005-11-11 01:09:53 +00:00
|
|
|
|
2007-11-11 05:22:18 +00:00
|
|
|
prior_fackets = tp->fackets_out;
|
2007-12-05 10:21:35 +00:00
|
|
|
prior_in_flight = tcp_packets_in_flight(tp);
|
2007-11-11 05:22:18 +00:00
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
if (!(flag & FLAG_SLOWPATH) && after(ack, prior_snd_una)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Window is constant, pure forward advance.
|
|
|
|
* No more checks are required.
|
|
|
|
* Note, we use the fact that SND.UNA>=SND.WL2.
|
|
|
|
*/
|
2009-03-03 06:42:02 +00:00
|
|
|
tcp_update_wl(tp, ack_seq);
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->snd_una = ack;
|
|
|
|
flag |= FLAG_WIN_UPDATE;
|
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_ca_event(sk, CA_EVENT_FAST_ACK);
|
2005-06-23 19:19:55 +00:00
|
|
|
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPHPACKS);
|
2005-04-16 22:20:36 +00:00
|
|
|
} else {
|
|
|
|
if (ack_seq != TCP_SKB_CB(skb)->end_seq)
|
|
|
|
flag |= FLAG_DATA;
|
|
|
|
else
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPPUREACKS);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
flag |= tcp_ack_update_window(sk, skb, ack, ack_seq);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (TCP_SKB_CB(skb)->sacked)
|
|
|
|
flag |= tcp_sacktag_write_queue(sk, skb, prior_snd_una);
|
|
|
|
|
2007-04-11 04:04:22 +00:00
|
|
|
if (TCP_ECN_rcv_ecn_echo(tp, tcp_hdr(skb)))
|
2005-04-16 22:20:36 +00:00
|
|
|
flag |= FLAG_ECE;
|
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_ca_event(sk, CA_EVENT_SLOW_ACK);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* We passed data and got it acked, remove any soft error
|
|
|
|
* log. Something worked...
|
|
|
|
*/
|
|
|
|
sk->sk_err_soft = 0;
|
tcp: Clear probes_out more aggressively in tcp_ack().
This is based upon an excellent bug report from Eric Dumazet.
tcp_ack() should clear ->icsk_probes_out even if there are packets
outstanding. Otherwise if we get a sequence of ACKs while we do have
packets outstanding over and over again, we'll never clear the
probes_out value and eventually think the connection is too sick and
we'll reset it.
This appears to be some "optimization" added to tcp_ack() in the 2.4.x
timeframe. In 2.2.x, probes_out is pretty much always cleared by
tcp_ack().
Here is Eric's original report:
----------------------------------------
Apparently, we can in some situations reset TCP connections in a couple of seconds when some frames are lost.
In order to reproduce the problem, please try the following program on linux-2.6.25.*
Setup some iptables rules to allow two frames per second sent on loopback interface to tcp destination port 12000
iptables -N SLOWLO
iptables -A SLOWLO -m hashlimit --hashlimit 2 --hashlimit-burst 1 --hashlimit-mode dstip --hashlimit-name slow2 -j ACCEPT
iptables -A SLOWLO -j DROP
iptables -A OUTPUT -o lo -p tcp --dport 12000 -j SLOWLO
Then run the attached program and see the output :
# ./loop
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,1)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,3)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,5)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,7)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,9)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,11)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,201ms,13)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,188ms,15)
write(): Connection timed out
wrote 890 bytes but was interrupted after 9 seconds
ESTAB 0 0 127.0.0.1:12000 127.0.0.1:54455
Exiting read() because no data available (4000 ms timeout).
read 860 bytes
While this tcp session makes progress (sending frames with 50 bytes of payload, every 500ms), linux tcp stack decides to reset it, when tcp_retries 2 is reached (default value : 15)
tcpdump :
15:30:28.856695 IP 127.0.0.1.56554 > 127.0.0.1.12000: S 33788768:33788768(0) win 32792 <mss 16396,nop,nop,sackOK,nop,wscale 7>
15:30:28.856711 IP 127.0.0.1.12000 > 127.0.0.1.56554: S 33899253:33899253(0) ack 33788769 win 32792 <mss 16396,nop,nop,sackOK,nop,wscale 7>
15:30:29.356947 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 1:61(60) ack 1 win 257
15:30:29.356966 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 61 win 257
15:30:29.866415 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 61:111(50) ack 1 win 257
15:30:29.866427 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 111 win 257
15:30:30.366516 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 111:161(50) ack 1 win 257
15:30:30.366527 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 161 win 257
15:30:30.876196 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 161:211(50) ack 1 win 257
15:30:30.876207 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 211 win 257
15:30:31.376282 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 211:261(50) ack 1 win 257
15:30:31.376290 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 261 win 257
15:30:31.885619 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 261:311(50) ack 1 win 257
15:30:31.885631 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 311 win 257
15:30:32.385705 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 311:361(50) ack 1 win 257
15:30:32.385715 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 361 win 257
15:30:32.895249 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 361:411(50) ack 1 win 257
15:30:32.895266 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 411 win 257
15:30:33.395341 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 411:461(50) ack 1 win 257
15:30:33.395351 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 461 win 257
15:30:33.918085 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 461:511(50) ack 1 win 257
15:30:33.918096 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 511 win 257
15:30:34.418163 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 511:561(50) ack 1 win 257
15:30:34.418172 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 561 win 257
15:30:34.927685 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 561:611(50) ack 1 win 257
15:30:34.927698 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 611 win 257
15:30:35.427757 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 611:661(50) ack 1 win 257
15:30:35.427766 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 661 win 257
15:30:35.937359 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 661:711(50) ack 1 win 257
15:30:35.937376 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 711 win 257
15:30:36.437451 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 711:761(50) ack 1 win 257
15:30:36.437464 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 761 win 257
15:30:36.947022 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 761:811(50) ack 1 win 257
15:30:36.947039 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 811 win 257
15:30:37.447135 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 811:861(50) ack 1 win 257
15:30:37.447203 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 861 win 257
15:30:41.448171 IP 127.0.0.1.12000 > 127.0.0.1.56554: F 1:1(0) ack 861 win 257
15:30:41.448189 IP 127.0.0.1.56554 > 127.0.0.1.12000: R 33789629:33789629(0) win 0
Source of program :
/*
* small producer/consumer program.
* setup a listener on 127.0.0.1:12000
* Forks a child
* child connect to 127.0.0.1, and sends 10 bytes on this tcp socket every 100 ms
* Father accepts connection, and read all data
*/
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <stdio.h>
#include <time.h>
#include <sys/poll.h>
int port = 12000;
char buffer[4096];
int main(int argc, char *argv[])
{
int lfd = socket(AF_INET, SOCK_STREAM, 0);
struct sockaddr_in socket_address;
time_t t0, t1;
int on = 1, sfd, res;
unsigned long total = 0;
socklen_t alen = sizeof(socket_address);
pid_t pid;
time(&t0);
socket_address.sin_family = AF_INET;
socket_address.sin_port = htons(port);
socket_address.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
if (lfd == -1) {
perror("socket()");
return 1;
}
setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(int));
if (bind(lfd, (struct sockaddr *)&socket_address, sizeof(socket_address)) == -1) {
perror("bind");
close(lfd);
return 1;
}
if (listen(lfd, 1) == -1) {
perror("listen()");
close(lfd);
return 1;
}
pid = fork();
if (pid == 0) {
int i, cfd = socket(AF_INET, SOCK_STREAM, 0);
close(lfd);
if (connect(cfd, (struct sockaddr *)&socket_address, sizeof(socket_address)) == -1) {
perror("connect()");
return 1;
}
for (i = 0 ; ;) {
res = write(cfd, "blablabla\n", 10);
if (res > 0) total += res;
else if (res == -1) {
perror("write()");
break;
} else break;
usleep(100000);
if (++i == 10) {
system("ss -on dst 127.0.0.1:12000");
i = 0;
}
}
time(&t1);
fprintf(stderr, "wrote %lu bytes but was interrupted after %g seconds\n", total, difftime(t1, t0));
system("ss -on | grep 127.0.0.1:12000");
close(cfd);
return 0;
}
sfd = accept(lfd, (struct sockaddr *)&socket_address, &alen);
if (sfd == -1) {
perror("accept");
return 1;
}
close(lfd);
while (1) {
struct pollfd pfd[1];
pfd[0].fd = sfd;
pfd[0].events = POLLIN;
if (poll(pfd, 1, 4000) == 0) {
fprintf(stderr, "Exiting read() because no data available (4000 ms timeout).\n");
break;
}
res = read(sfd, buffer, sizeof(buffer));
if (res > 0) total += res;
else if (res == 0) break;
else perror("read()");
}
fprintf(stderr, "read %lu bytes\n", total);
close(sfd);
return 0;
}
----------------------------------------
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-07-23 23:38:45 +00:00
|
|
|
icsk->icsk_probes_out = 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->rcv_tstamp = tcp_time_stamp;
|
|
|
|
prior_packets = tp->packets_out;
|
|
|
|
if (!prior_packets)
|
|
|
|
goto no_queue;
|
|
|
|
|
|
|
|
/* See if we can take anything off of the retransmit queue. */
|
2008-10-07 21:43:06 +00:00
|
|
|
flag |= tcp_clean_rtx_queue(sk, prior_fackets, prior_snd_una);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-11-16 08:58:01 +00:00
|
|
|
pkts_acked = prior_packets - tp->packets_out;
|
2011-08-21 20:21:57 +00:00
|
|
|
newly_acked_sacked = (prior_packets - prior_sacked) -
|
|
|
|
(tp->packets_out - tp->sacked_out);
|
|
|
|
|
2007-11-14 23:55:09 +00:00
|
|
|
if (tp->frto_counter)
|
|
|
|
frto_cwnd = tcp_process_frto(sk, flag);
|
2007-10-01 22:28:48 +00:00
|
|
|
/* Guarantee sacktag reordering detection against wrap-arounds */
|
|
|
|
if (before(tp->frto_highmark, tp->snd_una))
|
|
|
|
tp->frto_highmark = 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
if (tcp_ack_is_dubious(sk, flag)) {
|
2005-11-11 01:13:47 +00:00
|
|
|
/* Advance CWND, if state allows this. */
|
2007-02-22 07:08:34 +00:00
|
|
|
if ((flag & FLAG_DATA_ACKED) && !frto_cwnd &&
|
|
|
|
tcp_may_raise_cwnd(sk, flag))
|
2007-12-01 22:47:59 +00:00
|
|
|
tcp_cong_avoid(sk, ack, prior_in_flight);
|
2011-11-16 08:58:01 +00:00
|
|
|
is_dupack = !(flag & (FLAG_SND_UNA_ADVANCED | FLAG_NOT_DUP));
|
|
|
|
tcp_fastretrans_alert(sk, pkts_acked, newly_acked_sacked,
|
|
|
|
is_dupack, flag);
|
2005-04-16 22:20:36 +00:00
|
|
|
} else {
|
2007-02-22 07:08:34 +00:00
|
|
|
if ((flag & FLAG_DATA_ACKED) && !frto_cwnd)
|
2007-12-01 22:47:59 +00:00
|
|
|
tcp_cong_avoid(sk, ack, prior_in_flight);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
if ((flag & FLAG_FORWARD_PROGRESS) || !(flag & FLAG_NOT_DUP))
|
2010-04-08 23:03:29 +00:00
|
|
|
dst_confirm(__sk_dst_get(sk));
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
no_queue:
|
2011-11-16 08:58:02 +00:00
|
|
|
/* If data was DSACKed, see if we can undo a cwnd reduction. */
|
|
|
|
if (flag & FLAG_DSACKING_ACK)
|
|
|
|
tcp_fastretrans_alert(sk, pkts_acked, newly_acked_sacked,
|
|
|
|
is_dupack, flag);
|
2005-04-16 22:20:36 +00:00
|
|
|
/* If this ack opens up a zero window, clear backoff. It was
|
|
|
|
* being used to time the probes, and is probably far higher than
|
|
|
|
* it needs to be for normal retransmission.
|
|
|
|
*/
|
2007-03-07 20:12:44 +00:00
|
|
|
if (tcp_send_head(sk))
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_ack_probe(sk);
|
|
|
|
return 1;
|
|
|
|
|
2009-03-23 04:49:57 +00:00
|
|
|
invalid_ack:
|
|
|
|
SOCK_DEBUG(sk, "Ack %u after %u:%u\n", ack, tp->snd_una, tp->snd_nxt);
|
|
|
|
return -1;
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
old_ack:
|
2011-11-16 08:58:03 +00:00
|
|
|
/* If data was SACKed, tag it and see if we should send more data.
|
|
|
|
* If data was DSACKed, see if we can undo a cwnd reduction.
|
|
|
|
*/
|
2008-06-04 18:34:22 +00:00
|
|
|
if (TCP_SKB_CB(skb)->sacked) {
|
2011-11-16 08:58:03 +00:00
|
|
|
flag |= tcp_sacktag_write_queue(sk, skb, prior_snd_una);
|
|
|
|
newly_acked_sacked = tp->sacked_out - prior_sacked;
|
|
|
|
tcp_fastretrans_alert(sk, pkts_acked, newly_acked_sacked,
|
|
|
|
is_dupack, flag);
|
2008-06-04 18:34:22 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2009-03-23 04:49:57 +00:00
|
|
|
SOCK_DEBUG(sk, "Ack %u before %u:%u\n", ack, tp->snd_una, tp->snd_nxt);
|
2005-04-16 22:20:36 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Look for tcp options. Normally only called on SYN and SYNACK packets.
|
|
|
|
* But, this can also be called on packets in the established flow when
|
|
|
|
* the fast version below fails.
|
|
|
|
*/
|
2011-10-21 09:22:42 +00:00
|
|
|
void tcp_parse_options(const struct sk_buff *skb, struct tcp_options_received *opt_rx,
|
|
|
|
const u8 **hvpp, int estab)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2011-10-21 09:22:42 +00:00
|
|
|
const unsigned char *ptr;
|
|
|
|
const struct tcphdr *th = tcp_hdr(skb);
|
2007-12-31 22:57:14 +00:00
|
|
|
int length = (th->doff * 4) - sizeof(struct tcphdr);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
ptr = (const unsigned char *)(th + 1);
|
2005-04-16 22:20:36 +00:00
|
|
|
opt_rx->saw_tstamp = 0;
|
|
|
|
|
2007-03-09 04:45:19 +00:00
|
|
|
while (length > 0) {
|
2007-12-31 22:57:14 +00:00
|
|
|
int opcode = *ptr++;
|
2005-04-16 22:20:36 +00:00
|
|
|
int opsize;
|
|
|
|
|
|
|
|
switch (opcode) {
|
2008-01-04 04:36:55 +00:00
|
|
|
case TCPOPT_EOL:
|
|
|
|
return;
|
|
|
|
case TCPOPT_NOP: /* Ref: RFC 793 section 3.1 */
|
|
|
|
length--;
|
|
|
|
continue;
|
|
|
|
default:
|
|
|
|
opsize = *ptr++;
|
|
|
|
if (opsize < 2) /* "silly options" */
|
2005-04-16 22:20:36 +00:00
|
|
|
return;
|
2008-01-04 04:36:55 +00:00
|
|
|
if (opsize > length)
|
|
|
|
return; /* don't parse partial options */
|
|
|
|
switch (opcode) {
|
|
|
|
case TCPOPT_MSS:
|
|
|
|
if (opsize == TCPOLEN_MSS && th->syn && !estab) {
|
2008-05-02 23:26:16 +00:00
|
|
|
u16 in_mss = get_unaligned_be16(ptr);
|
2008-01-04 04:36:55 +00:00
|
|
|
if (in_mss) {
|
|
|
|
if (opt_rx->user_mss &&
|
|
|
|
opt_rx->user_mss < in_mss)
|
|
|
|
in_mss = opt_rx->user_mss;
|
|
|
|
opt_rx->mss_clamp = in_mss;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2008-01-04 04:36:55 +00:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case TCPOPT_WINDOW:
|
|
|
|
if (opsize == TCPOLEN_WINDOW && th->syn &&
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 04:56:42 +00:00
|
|
|
!estab && sysctl_tcp_window_scaling) {
|
2008-01-04 04:36:55 +00:00
|
|
|
__u8 snd_wscale = *(__u8 *)ptr;
|
|
|
|
opt_rx->wscale_ok = 1;
|
|
|
|
if (snd_wscale > 14) {
|
2012-05-13 21:56:26 +00:00
|
|
|
net_info_ratelimited("%s: Illegal window scaling value %d >14 received\n",
|
|
|
|
__func__,
|
|
|
|
snd_wscale);
|
2008-01-04 04:36:55 +00:00
|
|
|
snd_wscale = 14;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2008-01-04 04:36:55 +00:00
|
|
|
opt_rx->snd_wscale = snd_wscale;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case TCPOPT_TIMESTAMP:
|
|
|
|
if ((opsize == TCPOLEN_TIMESTAMP) &&
|
|
|
|
((estab && opt_rx->tstamp_ok) ||
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 04:56:42 +00:00
|
|
|
(!estab && sysctl_tcp_timestamps))) {
|
2008-01-04 04:36:55 +00:00
|
|
|
opt_rx->saw_tstamp = 1;
|
2008-05-02 23:26:16 +00:00
|
|
|
opt_rx->rcv_tsval = get_unaligned_be32(ptr);
|
|
|
|
opt_rx->rcv_tsecr = get_unaligned_be32(ptr + 4);
|
2008-01-04 04:36:55 +00:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case TCPOPT_SACK_PERM:
|
|
|
|
if (opsize == TCPOLEN_SACK_PERM && th->syn &&
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 04:56:42 +00:00
|
|
|
!estab && sysctl_tcp_sack) {
|
2011-12-20 13:23:24 +00:00
|
|
|
opt_rx->sack_ok = TCP_SACK_SEEN;
|
2008-01-04 04:36:55 +00:00
|
|
|
tcp_sack_reset(opt_rx);
|
|
|
|
}
|
|
|
|
break;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2008-01-04 04:36:55 +00:00
|
|
|
case TCPOPT_SACK:
|
|
|
|
if ((opsize >= (TCPOLEN_SACK_BASE + TCPOLEN_SACK_PERBLOCK)) &&
|
|
|
|
!((opsize - TCPOLEN_SACK_BASE) % TCPOLEN_SACK_PERBLOCK) &&
|
|
|
|
opt_rx->sack_ok) {
|
|
|
|
TCP_SKB_CB(skb)->sacked = (ptr - 2) - (unsigned char *)th;
|
|
|
|
}
|
|
|
|
break;
|
2006-11-15 03:07:45 +00:00
|
|
|
#ifdef CONFIG_TCP_MD5SIG
|
2008-01-04 04:36:55 +00:00
|
|
|
case TCPOPT_MD5SIG:
|
|
|
|
/*
|
|
|
|
* The MD5 Hash has already been
|
|
|
|
* checked (see tcp_v{4,6}_do_rcv()).
|
|
|
|
*/
|
|
|
|
break;
|
2006-11-15 03:07:45 +00:00
|
|
|
#endif
|
2009-12-02 18:25:27 +00:00
|
|
|
case TCPOPT_COOKIE:
|
|
|
|
/* This option is variable length.
|
|
|
|
*/
|
|
|
|
switch (opsize) {
|
|
|
|
case TCPOLEN_COOKIE_BASE:
|
|
|
|
/* not yet implemented */
|
|
|
|
break;
|
|
|
|
case TCPOLEN_COOKIE_PAIR:
|
|
|
|
/* not yet implemented */
|
|
|
|
break;
|
|
|
|
case TCPOLEN_COOKIE_MIN+0:
|
|
|
|
case TCPOLEN_COOKIE_MIN+2:
|
|
|
|
case TCPOLEN_COOKIE_MIN+4:
|
|
|
|
case TCPOLEN_COOKIE_MIN+6:
|
|
|
|
case TCPOLEN_COOKIE_MAX:
|
|
|
|
/* 16-bit multiple */
|
|
|
|
opt_rx->cookie_plus = opsize;
|
|
|
|
*hvpp = ptr;
|
2010-05-14 10:58:26 +00:00
|
|
|
break;
|
2009-12-02 18:25:27 +00:00
|
|
|
default:
|
|
|
|
/* ignore option */
|
|
|
|
break;
|
2010-05-14 10:58:26 +00:00
|
|
|
}
|
2009-12-02 18:25:27 +00:00
|
|
|
break;
|
2010-05-14 10:58:26 +00:00
|
|
|
}
|
2007-04-21 00:09:22 +00:00
|
|
|
|
2008-01-04 04:36:55 +00:00
|
|
|
ptr += opsize-2;
|
|
|
|
length -= opsize;
|
2007-04-21 00:09:22 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
2010-07-09 21:22:10 +00:00
|
|
|
EXPORT_SYMBOL(tcp_parse_options);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_parse_aligned_timestamp(struct tcp_sock *tp, const struct tcphdr *th)
|
2008-08-23 12:12:29 +00:00
|
|
|
{
|
2011-10-21 09:22:42 +00:00
|
|
|
const __be32 *ptr = (const __be32 *)(th + 1);
|
2008-08-23 12:12:29 +00:00
|
|
|
|
|
|
|
if (*ptr == htonl((TCPOPT_NOP << 24) | (TCPOPT_NOP << 16)
|
|
|
|
| (TCPOPT_TIMESTAMP << 8) | TCPOLEN_TIMESTAMP)) {
|
|
|
|
tp->rx_opt.saw_tstamp = 1;
|
|
|
|
++ptr;
|
|
|
|
tp->rx_opt.rcv_tsval = ntohl(*ptr);
|
|
|
|
++ptr;
|
|
|
|
tp->rx_opt.rcv_tsecr = ntohl(*ptr);
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2008-08-23 12:12:29 +00:00
|
|
|
}
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2008-08-23 12:12:29 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Fast parse options. This hopes to only see timestamps.
|
|
|
|
* If it is wrong it falls back on tcp_parse_options().
|
|
|
|
*/
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_fast_parse_options(const struct sk_buff *skb,
|
|
|
|
const struct tcphdr *th,
|
|
|
|
struct tcp_sock *tp, const u8 **hvpp)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2009-12-02 18:25:27 +00:00
|
|
|
/* In the spirit of fast parsing, compare doff directly to constant
|
|
|
|
* values. Because equality is used, short doff can be ignored here.
|
|
|
|
*/
|
|
|
|
if (th->doff == (sizeof(*th) / 4)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->rx_opt.saw_tstamp = 0;
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2005-04-16 22:20:36 +00:00
|
|
|
} else if (tp->rx_opt.tstamp_ok &&
|
2009-12-02 18:25:27 +00:00
|
|
|
th->doff == ((sizeof(*th) + TCPOLEN_TSTAMP_ALIGNED) / 4)) {
|
2008-08-23 12:12:29 +00:00
|
|
|
if (tcp_parse_aligned_timestamp(tp, th))
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 04:56:42 +00:00
|
|
|
tcp_parse_options(skb, &tp->rx_opt, hvpp, 1);
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2008-04-17 03:29:53 +00:00
|
|
|
#ifdef CONFIG_TCP_MD5SIG
|
|
|
|
/*
|
|
|
|
* Parse MD5 Signature option
|
|
|
|
*/
|
2011-10-21 09:22:42 +00:00
|
|
|
const u8 *tcp_parse_md5sig_option(const struct tcphdr *th)
|
2008-04-17 03:29:53 +00:00
|
|
|
{
|
2011-10-21 09:22:42 +00:00
|
|
|
int length = (th->doff << 2) - sizeof(*th);
|
|
|
|
const u8 *ptr = (const u8 *)(th + 1);
|
2008-04-17 03:29:53 +00:00
|
|
|
|
|
|
|
/* If the TCP option is too short, we can short cut */
|
|
|
|
if (length < TCPOLEN_MD5SIG)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
while (length > 0) {
|
|
|
|
int opcode = *ptr++;
|
|
|
|
int opsize;
|
|
|
|
|
|
|
|
switch(opcode) {
|
|
|
|
case TCPOPT_EOL:
|
|
|
|
return NULL;
|
|
|
|
case TCPOPT_NOP:
|
|
|
|
length--;
|
|
|
|
continue;
|
|
|
|
default:
|
|
|
|
opsize = *ptr++;
|
|
|
|
if (opsize < 2 || opsize > length)
|
|
|
|
return NULL;
|
|
|
|
if (opcode == TCPOPT_MD5SIG)
|
2010-08-08 03:24:28 +00:00
|
|
|
return opsize == TCPOLEN_MD5SIG ? ptr : NULL;
|
2008-04-17 03:29:53 +00:00
|
|
|
}
|
|
|
|
ptr += opsize - 2;
|
|
|
|
length -= opsize;
|
|
|
|
}
|
|
|
|
return NULL;
|
|
|
|
}
|
2010-07-09 21:22:10 +00:00
|
|
|
EXPORT_SYMBOL(tcp_parse_md5sig_option);
|
2008-04-17 03:29:53 +00:00
|
|
|
#endif
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
static inline void tcp_store_ts_recent(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
tp->rx_opt.ts_recent = tp->rx_opt.rcv_tsval;
|
2007-03-05 00:12:44 +00:00
|
|
|
tp->rx_opt.ts_recent_stamp = get_seconds();
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline void tcp_replace_ts_recent(struct tcp_sock *tp, u32 seq)
|
|
|
|
{
|
|
|
|
if (tp->rx_opt.saw_tstamp && !after(seq, tp->rcv_wup)) {
|
|
|
|
/* PAWS bug workaround wrt. ACK frames, the PAWS discard
|
|
|
|
* extra check below makes sure this can only happen
|
|
|
|
* for pure ACK frames. -DaveM
|
|
|
|
*
|
|
|
|
* Not only, also it occurs for expired timestamps.
|
|
|
|
*/
|
|
|
|
|
2009-03-14 14:23:03 +00:00
|
|
|
if (tcp_paws_check(&tp->rx_opt, 0))
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Sorry, PAWS as specified is broken wrt. pure-ACKs -DaveM
|
|
|
|
*
|
|
|
|
* It is not fatal. If this ACK does _not_ change critical state (seqs, window)
|
|
|
|
* it can pass through stack. So, the following predicate verifies that
|
|
|
|
* this segment is not used for anything but congestion avoidance or
|
|
|
|
* fast retransmit. Moreover, we even are able to eliminate most of such
|
|
|
|
* second order effects, if we apply some small "replay" window (~RTO)
|
|
|
|
* to timestamp space.
|
|
|
|
*
|
|
|
|
* All these measures still do not guarantee that we reject wrapped ACKs
|
|
|
|
* on networks with high bandwidth, when sequence space is recycled fastly,
|
|
|
|
* but it guarantees that such events will be very rare and do not affect
|
|
|
|
* connection seriously. This doesn't look nice, but alas, PAWS is really
|
|
|
|
* buggy extension.
|
|
|
|
*
|
|
|
|
* [ Later note. Even worse! It is buggy for segments _with_ data. RFC
|
|
|
|
* states that events when retransmit arrives after original data are rare.
|
|
|
|
* It is a blatant lie. VJ forgot about fast retransmit! 8)8) It is
|
|
|
|
* the biggest problem on large power networks even with minor reordering.
|
|
|
|
* OK, let's give it small replay window. If peer clock is even 1hz, it is safe
|
|
|
|
* up to bandwidth of 18Gigabit/sec. 8) ]
|
|
|
|
*/
|
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
static int tcp_disordered_ack(const struct sock *sk, const struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2011-10-21 09:22:42 +00:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
const struct tcphdr *th = tcp_hdr(skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
u32 seq = TCP_SKB_CB(skb)->seq;
|
|
|
|
u32 ack = TCP_SKB_CB(skb)->ack_seq;
|
|
|
|
|
|
|
|
return (/* 1. Pure ACK with correct sequence number. */
|
|
|
|
(th->ack && seq == TCP_SKB_CB(skb)->end_seq && seq == tp->rcv_nxt) &&
|
|
|
|
|
|
|
|
/* 2. ... and duplicate ACK. */
|
|
|
|
ack == tp->snd_una &&
|
|
|
|
|
|
|
|
/* 3. ... and does not update window. */
|
|
|
|
!tcp_may_update_window(tp, ack, seq, ntohs(th->window) << tp->rx_opt.snd_wscale) &&
|
|
|
|
|
|
|
|
/* 4. ... and sits in replay window. */
|
2005-08-10 03:10:42 +00:00
|
|
|
(s32)(tp->rx_opt.ts_recent - tp->rx_opt.rcv_tsval) <= (inet_csk(sk)->icsk_rto * 1024) / HZ);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
static inline int tcp_paws_discard(const struct sock *sk,
|
|
|
|
const struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 03:10:42 +00:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2009-03-14 14:23:03 +00:00
|
|
|
|
|
|
|
return !tcp_paws_check(&tp->rx_opt, TCP_PAWS_WINDOW) &&
|
|
|
|
!tcp_disordered_ack(sk, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Check segment sequence number for validity.
|
|
|
|
*
|
|
|
|
* Segment controls are considered valid, if the segment
|
|
|
|
* fits to the window after truncation to the window. Acceptability
|
|
|
|
* of data (and SYN, FIN, of course) is checked separately.
|
|
|
|
* See tcp_data_queue(), for example.
|
|
|
|
*
|
|
|
|
* Also, controls (RST is main one) are accepted using RCV.WUP instead
|
|
|
|
* of RCV.NXT. Peer still did not advance his SND.UNA when we
|
|
|
|
* delayed ACK, so that hisSND.UNA<=ourRCV.WUP.
|
|
|
|
* (borrowed from freebsd)
|
|
|
|
*/
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static inline int tcp_sequence(const struct tcp_sock *tp, u32 seq, u32 end_seq)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
return !before(end_seq, tp->rcv_wup) &&
|
|
|
|
!after(seq, tp->rcv_nxt + tcp_receive_window(tp));
|
|
|
|
}
|
|
|
|
|
|
|
|
/* When we get a reset we do this. */
|
|
|
|
static void tcp_reset(struct sock *sk)
|
|
|
|
{
|
|
|
|
/* We want the right error as BSD sees it (and indeed as we do). */
|
|
|
|
switch (sk->sk_state) {
|
2007-12-31 22:57:14 +00:00
|
|
|
case TCP_SYN_SENT:
|
|
|
|
sk->sk_err = ECONNREFUSED;
|
|
|
|
break;
|
|
|
|
case TCP_CLOSE_WAIT:
|
|
|
|
sk->sk_err = EPIPE;
|
|
|
|
break;
|
|
|
|
case TCP_CLOSE:
|
|
|
|
return;
|
|
|
|
default:
|
|
|
|
sk->sk_err = ECONNRESET;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2010-09-20 22:42:05 +00:00
|
|
|
/* This barrier is coupled with smp_rmb() in tcp_poll() */
|
|
|
|
smp_wmb();
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD))
|
|
|
|
sk->sk_error_report(sk);
|
|
|
|
|
|
|
|
tcp_done(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Process the FIN bit. This now behaves as it is supposed to work
|
|
|
|
* and the FIN takes effect when it is validly part of sequence
|
|
|
|
* space. Not before when we get holes.
|
|
|
|
*
|
|
|
|
* If we are ESTABLISHED, a received fin moves us to CLOSE-WAIT
|
|
|
|
* (and thence onto LAST-ACK and finally, CLOSE, we never enter
|
|
|
|
* TIME-WAIT)
|
|
|
|
*
|
|
|
|
* If we are in FINWAIT-1, a received FIN indicates simultaneous
|
|
|
|
* close and we go into CLOSING (and later onto TIME-WAIT)
|
|
|
|
*
|
|
|
|
* If we are in FINWAIT-2, a received FIN moves us to TIME-WAIT.
|
|
|
|
*/
|
2011-10-20 21:44:03 +00:00
|
|
|
static void tcp_fin(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
inet_csk_schedule_ack(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
sk->sk_shutdown |= RCV_SHUTDOWN;
|
|
|
|
sock_set_flag(sk, SOCK_DONE);
|
|
|
|
|
|
|
|
switch (sk->sk_state) {
|
2007-12-31 22:57:14 +00:00
|
|
|
case TCP_SYN_RECV:
|
|
|
|
case TCP_ESTABLISHED:
|
|
|
|
/* Move to CLOSE_WAIT */
|
|
|
|
tcp_set_state(sk, TCP_CLOSE_WAIT);
|
|
|
|
inet_csk(sk)->icsk_ack.pingpong = 1;
|
|
|
|
break;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
case TCP_CLOSE_WAIT:
|
|
|
|
case TCP_CLOSING:
|
|
|
|
/* Received a retransmission of the FIN, do
|
|
|
|
* nothing.
|
|
|
|
*/
|
|
|
|
break;
|
|
|
|
case TCP_LAST_ACK:
|
|
|
|
/* RFC793: Remain in the LAST-ACK state. */
|
|
|
|
break;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
case TCP_FIN_WAIT1:
|
|
|
|
/* This case occurs when a simultaneous close
|
|
|
|
* happens, we must ack the received FIN and
|
|
|
|
* enter the CLOSING state.
|
|
|
|
*/
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
tcp_set_state(sk, TCP_CLOSING);
|
|
|
|
break;
|
|
|
|
case TCP_FIN_WAIT2:
|
|
|
|
/* Received a FIN -- send ACK and enter TIME_WAIT. */
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
tcp_time_wait(sk, TCP_TIME_WAIT, 0);
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
/* Only TCP_LISTEN and TCP_CLOSE are left, in these
|
|
|
|
* cases we should never reach this piece of code.
|
|
|
|
*/
|
2012-03-11 18:36:11 +00:00
|
|
|
pr_err("%s: Impossible, sk->sk_state=%d\n",
|
2008-03-06 04:47:47 +00:00
|
|
|
__func__, sk->sk_state);
|
2007-12-31 22:57:14 +00:00
|
|
|
break;
|
2007-04-21 00:09:22 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* It _is_ possible, that we have something out-of-order _after_ FIN.
|
|
|
|
* Probably, we should reset in this case. For now drop them.
|
|
|
|
*/
|
|
|
|
__skb_queue_purge(&tp->out_of_order_queue);
|
2007-08-09 12:14:46 +00:00
|
|
|
if (tcp_is_sack(tp))
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_sack_reset(&tp->rx_opt);
|
2007-12-31 08:11:19 +00:00
|
|
|
sk_mem_reclaim(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD)) {
|
|
|
|
sk->sk_state_change(sk);
|
|
|
|
|
|
|
|
/* Do not send POLL_HUP for half duplex close. */
|
|
|
|
if (sk->sk_shutdown == SHUTDOWN_MASK ||
|
|
|
|
sk->sk_state == TCP_CLOSE)
|
2007-11-26 12:10:50 +00:00
|
|
|
sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_HUP);
|
2005-04-16 22:20:36 +00:00
|
|
|
else
|
2007-11-26 12:10:50 +00:00
|
|
|
sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
static inline bool tcp_sack_extend(struct tcp_sack_block *sp, u32 seq,
|
2007-12-31 22:57:14 +00:00
|
|
|
u32 end_seq)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
if (!after(seq, sp->end_seq) && !after(sp->start_seq, end_seq)) {
|
|
|
|
if (before(seq, sp->start_seq))
|
|
|
|
sp->start_seq = seq;
|
|
|
|
if (after(end_seq, sp->end_seq))
|
|
|
|
sp->end_seq = end_seq;
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2008-07-17 03:29:51 +00:00
|
|
|
static void tcp_dsack_set(struct sock *sk, u32 seq, u32 end_seq)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2008-07-17 03:29:51 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 04:56:42 +00:00
|
|
|
if (tcp_is_sack(tp) && sysctl_tcp_dsack) {
|
2008-07-03 08:05:41 +00:00
|
|
|
int mib_idx;
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
if (before(seq, tp->rcv_nxt))
|
2008-07-03 08:05:41 +00:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKOLDSENT;
|
2005-04-16 22:20:36 +00:00
|
|
|
else
|
2008-07-03 08:05:41 +00:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKOFOSENT;
|
|
|
|
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), mib_idx);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
tp->rx_opt.dsack = 1;
|
|
|
|
tp->duplicate_sack[0].start_seq = seq;
|
|
|
|
tp->duplicate_sack[0].end_seq = end_seq;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-07-17 03:29:51 +00:00
|
|
|
static void tcp_dsack_extend(struct sock *sk, u32 seq, u32 end_seq)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2008-07-17 03:29:51 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
if (!tp->rx_opt.dsack)
|
2008-07-17 03:29:51 +00:00
|
|
|
tcp_dsack_set(sk, seq, end_seq);
|
2005-04-16 22:20:36 +00:00
|
|
|
else
|
|
|
|
tcp_sack_extend(tp->duplicate_sack, seq, end_seq);
|
|
|
|
}
|
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static void tcp_send_dupack(struct sock *sk, const struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
|
|
|
|
before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt)) {
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_DELAYEDACKLOST);
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_enter_quickack_mode(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 04:56:42 +00:00
|
|
|
if (tcp_is_sack(tp) && sysctl_tcp_dsack) {
|
2005-04-16 22:20:36 +00:00
|
|
|
u32 end_seq = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
|
|
|
|
if (after(TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt))
|
|
|
|
end_seq = tp->rcv_nxt;
|
2008-07-17 03:29:51 +00:00
|
|
|
tcp_dsack_set(sk, TCP_SKB_CB(skb)->seq, end_seq);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* These routines update the SACK block as out-of-order packets arrive or
|
|
|
|
* in-order packets close up the sequence space.
|
|
|
|
*/
|
|
|
|
static void tcp_sack_maybe_coalesce(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
int this_sack;
|
|
|
|
struct tcp_sack_block *sp = &tp->selective_acks[0];
|
2007-12-31 22:57:14 +00:00
|
|
|
struct tcp_sack_block *swalk = sp + 1;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* See if the recent change to the first SACK eats into
|
|
|
|
* or hits the sequence space of other SACK blocks, if so coalesce.
|
|
|
|
*/
|
2007-12-31 22:57:14 +00:00
|
|
|
for (this_sack = 1; this_sack < tp->rx_opt.num_sacks;) {
|
2005-04-16 22:20:36 +00:00
|
|
|
if (tcp_sack_extend(sp, swalk->start_seq, swalk->end_seq)) {
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/* Zap SWALK, by moving every further SACK up by one slot.
|
|
|
|
* Decrease num_sacks.
|
|
|
|
*/
|
|
|
|
tp->rx_opt.num_sacks--;
|
2007-12-31 22:57:14 +00:00
|
|
|
for (i = this_sack; i < tp->rx_opt.num_sacks; i++)
|
|
|
|
sp[i] = sp[i + 1];
|
2005-04-16 22:20:36 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
this_sack++, swalk++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void tcp_sack_new_ofo_skb(struct sock *sk, u32 seq, u32 end_seq)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct tcp_sack_block *sp = &tp->selective_acks[0];
|
|
|
|
int cur_sacks = tp->rx_opt.num_sacks;
|
|
|
|
int this_sack;
|
|
|
|
|
|
|
|
if (!cur_sacks)
|
|
|
|
goto new_sack;
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
for (this_sack = 0; this_sack < cur_sacks; this_sack++, sp++) {
|
2005-04-16 22:20:36 +00:00
|
|
|
if (tcp_sack_extend(sp, seq, end_seq)) {
|
|
|
|
/* Rotate this_sack to the first one. */
|
2007-12-31 22:57:14 +00:00
|
|
|
for (; this_sack > 0; this_sack--, sp--)
|
2009-03-21 20:36:17 +00:00
|
|
|
swap(*sp, *(sp - 1));
|
2005-04-16 22:20:36 +00:00
|
|
|
if (cur_sacks > 1)
|
|
|
|
tcp_sack_maybe_coalesce(tp);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Could not find an adjacent existing SACK, build a new one,
|
|
|
|
* put it at the front, and shift everyone else down. We
|
|
|
|
* always know there is at least one SACK present already here.
|
|
|
|
*
|
|
|
|
* If the sack array is full, forget about the last one.
|
|
|
|
*/
|
2008-07-19 07:07:02 +00:00
|
|
|
if (this_sack >= TCP_NUM_SACKS) {
|
2005-04-16 22:20:36 +00:00
|
|
|
this_sack--;
|
|
|
|
tp->rx_opt.num_sacks--;
|
|
|
|
sp--;
|
|
|
|
}
|
2007-03-09 04:45:19 +00:00
|
|
|
for (; this_sack > 0; this_sack--, sp--)
|
2007-12-31 22:57:14 +00:00
|
|
|
*sp = *(sp - 1);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
new_sack:
|
|
|
|
/* Build the new head SACK, and we're done. */
|
|
|
|
sp->start_seq = seq;
|
|
|
|
sp->end_seq = end_seq;
|
|
|
|
tp->rx_opt.num_sacks++;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* RCV.NXT advances, some SACKs should be eaten. */
|
|
|
|
|
|
|
|
static void tcp_sack_remove(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
struct tcp_sack_block *sp = &tp->selective_acks[0];
|
|
|
|
int num_sacks = tp->rx_opt.num_sacks;
|
|
|
|
int this_sack;
|
|
|
|
|
|
|
|
/* Empty ofo queue, hence, all the SACKs are eaten. Clear. */
|
2005-07-08 21:57:23 +00:00
|
|
|
if (skb_queue_empty(&tp->out_of_order_queue)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->rx_opt.num_sacks = 0;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
for (this_sack = 0; this_sack < num_sacks;) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Check if the start of the sack is covered by RCV.NXT. */
|
|
|
|
if (!before(tp->rcv_nxt, sp->start_seq)) {
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/* RCV.NXT must cover all the block! */
|
2008-07-26 04:43:18 +00:00
|
|
|
WARN_ON(before(tp->rcv_nxt, sp->end_seq));
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Zap this SACK, by moving forward any other SACKS. */
|
|
|
|
for (i=this_sack+1; i < num_sacks; i++)
|
|
|
|
tp->selective_acks[i-1] = tp->selective_acks[i];
|
|
|
|
num_sacks--;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
this_sack++;
|
|
|
|
sp++;
|
|
|
|
}
|
2009-03-14 14:23:01 +00:00
|
|
|
tp->rx_opt.num_sacks = num_sacks;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* This one checks to see if we can put data from the
|
|
|
|
* out_of_order queue into the receive_queue.
|
|
|
|
*/
|
|
|
|
static void tcp_ofo_queue(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
__u32 dsack_high = tp->rcv_nxt;
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
while ((skb = skb_peek(&tp->out_of_order_queue)) != NULL) {
|
|
|
|
if (after(TCP_SKB_CB(skb)->seq, tp->rcv_nxt))
|
|
|
|
break;
|
|
|
|
|
|
|
|
if (before(TCP_SKB_CB(skb)->seq, dsack_high)) {
|
|
|
|
__u32 dsack = dsack_high;
|
|
|
|
if (before(TCP_SKB_CB(skb)->end_seq, dsack_high))
|
|
|
|
dsack_high = TCP_SKB_CB(skb)->end_seq;
|
2008-07-17 03:29:51 +00:00
|
|
|
tcp_dsack_extend(sk, TCP_SKB_CB(skb)->seq, dsack);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt)) {
|
2010-03-24 07:57:28 +00:00
|
|
|
SOCK_DEBUG(sk, "ofo packet was already received\n");
|
2005-08-10 02:25:21 +00:00
|
|
|
__skb_unlink(skb, &tp->out_of_order_queue);
|
2005-04-16 22:20:36 +00:00
|
|
|
__kfree_skb(skb);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
SOCK_DEBUG(sk, "ofo requeuing : rcv_next %X seq %X - %X\n",
|
|
|
|
tp->rcv_nxt, TCP_SKB_CB(skb)->seq,
|
|
|
|
TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
2005-08-10 02:25:21 +00:00
|
|
|
__skb_unlink(skb, &tp->out_of_order_queue);
|
2005-04-16 22:20:36 +00:00
|
|
|
__skb_queue_tail(&sk->sk_receive_queue, skb);
|
|
|
|
tp->rcv_nxt = TCP_SKB_CB(skb)->end_seq;
|
2007-04-11 04:04:22 +00:00
|
|
|
if (tcp_hdr(skb)->fin)
|
2011-10-20 21:44:03 +00:00
|
|
|
tcp_fin(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_prune_ofo_queue(struct sock *sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
static int tcp_prune_queue(struct sock *sk);
|
|
|
|
|
2012-05-10 01:50:20 +00:00
|
|
|
static int tcp_try_rmem_schedule(struct sock *sk, unsigned int size)
|
2008-04-15 07:33:38 +00:00
|
|
|
{
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf ||
|
|
|
|
!sk_rmem_schedule(sk, size)) {
|
|
|
|
|
|
|
|
if (tcp_prune_queue(sk) < 0)
|
|
|
|
return -1;
|
|
|
|
|
|
|
|
if (!sk_rmem_schedule(sk, size)) {
|
2008-04-16 03:26:34 +00:00
|
|
|
if (!tcp_prune_ofo_queue(sk))
|
|
|
|
return -1;
|
|
|
|
|
2008-04-15 07:33:38 +00:00
|
|
|
if (!sk_rmem_schedule(sk, size))
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2012-04-23 07:11:42 +00:00
|
|
|
/**
|
|
|
|
* tcp_try_coalesce - try to merge skb to prior one
|
|
|
|
* @sk: socket
|
|
|
|
* @to: prior buffer
|
|
|
|
* @from: buffer to add in queue
|
2012-05-02 07:55:58 +00:00
|
|
|
* @fragstolen: pointer to boolean
|
2012-04-23 07:11:42 +00:00
|
|
|
*
|
|
|
|
* Before queueing skb @from after @to, try to merge them
|
|
|
|
* to reduce overall memory use and queue lengths, if cost is small.
|
|
|
|
* Packets in ofo or receive queues can stay a long time.
|
|
|
|
* Better try to coalesce them right now to avoid future collapses.
|
2012-04-23 17:34:36 +00:00
|
|
|
* Returns true if caller should free @from instead of queueing it
|
2012-04-23 07:11:42 +00:00
|
|
|
*/
|
2012-04-23 17:34:36 +00:00
|
|
|
static bool tcp_try_coalesce(struct sock *sk,
|
|
|
|
struct sk_buff *to,
|
2012-04-27 00:38:33 +00:00
|
|
|
struct sk_buff *from,
|
|
|
|
bool *fragstolen)
|
2012-04-23 07:11:42 +00:00
|
|
|
{
|
2012-05-19 03:02:02 +00:00
|
|
|
int delta;
|
2012-04-23 07:11:42 +00:00
|
|
|
|
2012-04-27 00:38:33 +00:00
|
|
|
*fragstolen = false;
|
2012-05-02 21:19:09 +00:00
|
|
|
|
2012-05-19 03:02:02 +00:00
|
|
|
if (tcp_hdr(from)->fin)
|
2012-04-23 17:34:36 +00:00
|
|
|
return false;
|
2012-05-23 17:51:37 +00:00
|
|
|
|
|
|
|
/* Its possible this segment overlaps with prior segment in queue */
|
|
|
|
if (TCP_SKB_CB(from)->seq != TCP_SKB_CB(to)->end_seq)
|
|
|
|
return false;
|
|
|
|
|
2012-05-19 03:02:02 +00:00
|
|
|
if (!skb_try_coalesce(to, from, fragstolen, &delta))
|
2012-04-27 00:38:33 +00:00
|
|
|
return false;
|
|
|
|
|
2012-05-02 21:19:09 +00:00
|
|
|
atomic_add(delta, &sk->sk_rmem_alloc);
|
|
|
|
sk_mem_charge(sk, delta);
|
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPRCVCOALESCE);
|
|
|
|
TCP_SKB_CB(to)->end_seq = TCP_SKB_CB(from)->end_seq;
|
|
|
|
TCP_SKB_CB(to)->ack_seq = TCP_SKB_CB(from)->ack_seq;
|
|
|
|
return true;
|
2012-04-23 07:11:42 +00:00
|
|
|
}
|
|
|
|
|
2012-03-18 11:06:44 +00:00
|
|
|
static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb1;
|
|
|
|
u32 seq, end_seq;
|
|
|
|
|
|
|
|
TCP_ECN_check_ce(tp, skb);
|
|
|
|
|
|
|
|
if (tcp_try_rmem_schedule(sk, skb->truesize)) {
|
|
|
|
/* TODO: should increment a counter */
|
|
|
|
__kfree_skb(skb);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Disable header prediction. */
|
|
|
|
tp->pred_flags = 0;
|
|
|
|
inet_csk_schedule_ack(sk);
|
|
|
|
|
|
|
|
SOCK_DEBUG(sk, "out of order segment: rcv_next %X seq %X - %X\n",
|
|
|
|
tp->rcv_nxt, TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
|
|
|
skb1 = skb_peek_tail(&tp->out_of_order_queue);
|
|
|
|
if (!skb1) {
|
|
|
|
/* Initial out of order segment, build 1 SACK. */
|
|
|
|
if (tcp_is_sack(tp)) {
|
|
|
|
tp->rx_opt.num_sacks = 1;
|
|
|
|
tp->selective_acks[0].start_seq = TCP_SKB_CB(skb)->seq;
|
|
|
|
tp->selective_acks[0].end_seq =
|
|
|
|
TCP_SKB_CB(skb)->end_seq;
|
|
|
|
}
|
|
|
|
__skb_queue_head(&tp->out_of_order_queue, skb);
|
|
|
|
goto end;
|
|
|
|
}
|
|
|
|
|
|
|
|
seq = TCP_SKB_CB(skb)->seq;
|
|
|
|
end_seq = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
|
|
|
|
if (seq == TCP_SKB_CB(skb1)->end_seq) {
|
2012-04-27 00:38:33 +00:00
|
|
|
bool fragstolen;
|
|
|
|
|
|
|
|
if (!tcp_try_coalesce(sk, skb1, skb, &fragstolen)) {
|
2012-04-23 07:11:42 +00:00
|
|
|
__skb_queue_after(&tp->out_of_order_queue, skb1, skb);
|
|
|
|
} else {
|
2012-05-02 07:55:58 +00:00
|
|
|
kfree_skb_partial(skb, fragstolen);
|
2012-03-18 11:07:47 +00:00
|
|
|
skb = NULL;
|
|
|
|
}
|
2012-03-18 11:06:44 +00:00
|
|
|
|
|
|
|
if (!tp->rx_opt.num_sacks ||
|
|
|
|
tp->selective_acks[0].end_seq != seq)
|
|
|
|
goto add_sack;
|
|
|
|
|
|
|
|
/* Common case: data arrive in order after hole. */
|
|
|
|
tp->selective_acks[0].end_seq = end_seq;
|
|
|
|
goto end;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Find place to insert this segment. */
|
|
|
|
while (1) {
|
|
|
|
if (!after(TCP_SKB_CB(skb1)->seq, seq))
|
|
|
|
break;
|
|
|
|
if (skb_queue_is_first(&tp->out_of_order_queue, skb1)) {
|
|
|
|
skb1 = NULL;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
skb1 = skb_queue_prev(&tp->out_of_order_queue, skb1);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Do skb overlap to previous one? */
|
|
|
|
if (skb1 && before(seq, TCP_SKB_CB(skb1)->end_seq)) {
|
|
|
|
if (!after(end_seq, TCP_SKB_CB(skb1)->end_seq)) {
|
|
|
|
/* All the bits are present. Drop. */
|
|
|
|
__kfree_skb(skb);
|
|
|
|
skb = NULL;
|
|
|
|
tcp_dsack_set(sk, seq, end_seq);
|
|
|
|
goto add_sack;
|
|
|
|
}
|
|
|
|
if (after(seq, TCP_SKB_CB(skb1)->seq)) {
|
|
|
|
/* Partial overlap. */
|
|
|
|
tcp_dsack_set(sk, seq,
|
|
|
|
TCP_SKB_CB(skb1)->end_seq);
|
|
|
|
} else {
|
|
|
|
if (skb_queue_is_first(&tp->out_of_order_queue,
|
|
|
|
skb1))
|
|
|
|
skb1 = NULL;
|
|
|
|
else
|
|
|
|
skb1 = skb_queue_prev(
|
|
|
|
&tp->out_of_order_queue,
|
|
|
|
skb1);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!skb1)
|
|
|
|
__skb_queue_head(&tp->out_of_order_queue, skb);
|
|
|
|
else
|
|
|
|
__skb_queue_after(&tp->out_of_order_queue, skb1, skb);
|
|
|
|
|
|
|
|
/* And clean segments covered by new one as whole. */
|
|
|
|
while (!skb_queue_is_last(&tp->out_of_order_queue, skb)) {
|
|
|
|
skb1 = skb_queue_next(&tp->out_of_order_queue, skb);
|
|
|
|
|
|
|
|
if (!after(end_seq, TCP_SKB_CB(skb1)->seq))
|
|
|
|
break;
|
|
|
|
if (before(end_seq, TCP_SKB_CB(skb1)->end_seq)) {
|
|
|
|
tcp_dsack_extend(sk, TCP_SKB_CB(skb1)->seq,
|
|
|
|
end_seq);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
__skb_unlink(skb1, &tp->out_of_order_queue);
|
|
|
|
tcp_dsack_extend(sk, TCP_SKB_CB(skb1)->seq,
|
|
|
|
TCP_SKB_CB(skb1)->end_seq);
|
|
|
|
__kfree_skb(skb1);
|
|
|
|
}
|
|
|
|
|
|
|
|
add_sack:
|
|
|
|
if (tcp_is_sack(tp))
|
|
|
|
tcp_sack_new_ofo_skb(sk, seq, end_seq);
|
|
|
|
end:
|
|
|
|
if (skb)
|
|
|
|
skb_set_owner_r(skb, sk);
|
|
|
|
}
|
|
|
|
|
2012-05-10 01:49:41 +00:00
|
|
|
static int __must_check tcp_queue_rcv(struct sock *sk, struct sk_buff *skb, int hdrlen,
|
2012-05-02 09:58:29 +00:00
|
|
|
bool *fragstolen)
|
|
|
|
{
|
|
|
|
int eaten;
|
|
|
|
struct sk_buff *tail = skb_peek_tail(&sk->sk_receive_queue);
|
|
|
|
|
|
|
|
__skb_pull(skb, hdrlen);
|
|
|
|
eaten = (tail &&
|
|
|
|
tcp_try_coalesce(sk, tail, skb, fragstolen)) ? 1 : 0;
|
|
|
|
tcp_sk(sk)->rcv_nxt = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
if (!eaten) {
|
|
|
|
__skb_queue_tail(&sk->sk_receive_queue, skb);
|
|
|
|
skb_set_owner_r(skb, sk);
|
|
|
|
}
|
|
|
|
return eaten;
|
|
|
|
}
|
2012-03-18 11:06:44 +00:00
|
|
|
|
2012-05-10 01:49:41 +00:00
|
|
|
int tcp_send_rcvq(struct sock *sk, struct msghdr *msg, size_t size)
|
|
|
|
{
|
|
|
|
struct sk_buff *skb;
|
|
|
|
struct tcphdr *th;
|
|
|
|
bool fragstolen;
|
|
|
|
|
2012-05-10 01:50:01 +00:00
|
|
|
if (tcp_try_rmem_schedule(sk, size + sizeof(*th)))
|
|
|
|
goto err;
|
|
|
|
|
2012-05-10 01:49:41 +00:00
|
|
|
skb = alloc_skb(size + sizeof(*th), sk->sk_allocation);
|
|
|
|
if (!skb)
|
|
|
|
goto err;
|
|
|
|
|
|
|
|
th = (struct tcphdr *)skb_put(skb, sizeof(*th));
|
|
|
|
skb_reset_transport_header(skb);
|
|
|
|
memset(th, 0, sizeof(*th));
|
|
|
|
|
|
|
|
if (memcpy_fromiovec(skb_put(skb, size), msg->msg_iov, size))
|
|
|
|
goto err_free;
|
|
|
|
|
|
|
|
TCP_SKB_CB(skb)->seq = tcp_sk(sk)->rcv_nxt;
|
|
|
|
TCP_SKB_CB(skb)->end_seq = TCP_SKB_CB(skb)->seq + size;
|
|
|
|
TCP_SKB_CB(skb)->ack_seq = tcp_sk(sk)->snd_una - 1;
|
|
|
|
|
|
|
|
if (tcp_queue_rcv(sk, skb, sizeof(*th), &fragstolen)) {
|
|
|
|
WARN_ON_ONCE(fragstolen); /* should not happen */
|
|
|
|
__kfree_skb(skb);
|
|
|
|
}
|
|
|
|
return size;
|
|
|
|
|
|
|
|
err_free:
|
|
|
|
kfree_skb(skb);
|
|
|
|
err:
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
static void tcp_data_queue(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
2011-10-21 09:22:42 +00:00
|
|
|
const struct tcphdr *th = tcp_hdr(skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int eaten = -1;
|
2012-04-27 00:38:33 +00:00
|
|
|
bool fragstolen = false;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (TCP_SKB_CB(skb)->seq == TCP_SKB_CB(skb)->end_seq)
|
|
|
|
goto drop;
|
|
|
|
|
2010-04-28 22:31:51 +00:00
|
|
|
skb_dst_drop(skb);
|
2007-12-31 22:57:14 +00:00
|
|
|
__skb_pull(skb, th->doff * 4);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
TCP_ECN_accept_cwr(tp, skb);
|
|
|
|
|
2009-03-14 14:23:01 +00:00
|
|
|
tp->rx_opt.dsack = 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Queue data for delivery to the user.
|
|
|
|
* Packets in sequence go to the receive queue.
|
|
|
|
* Out of sequence packets to the out_of_order_queue.
|
|
|
|
*/
|
|
|
|
if (TCP_SKB_CB(skb)->seq == tp->rcv_nxt) {
|
|
|
|
if (tcp_receive_window(tp) == 0)
|
|
|
|
goto out_of_window;
|
|
|
|
|
|
|
|
/* Ok. In sequence. In window. */
|
|
|
|
if (tp->ucopy.task == current &&
|
|
|
|
tp->copied_seq == tp->rcv_nxt && tp->ucopy.len &&
|
|
|
|
sock_owned_by_user(sk) && !tp->urg_data) {
|
|
|
|
int chunk = min_t(unsigned int, skb->len,
|
2007-12-31 22:57:14 +00:00
|
|
|
tp->ucopy.len);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
__set_current_state(TASK_RUNNING);
|
|
|
|
|
|
|
|
local_bh_enable();
|
|
|
|
if (!skb_copy_datagram_iovec(skb, 0, tp->ucopy.iov, chunk)) {
|
|
|
|
tp->ucopy.len -= chunk;
|
|
|
|
tp->copied_seq += chunk;
|
TCP: fix a bug that triggers large number of TCP RST by mistake
This patch fixes a bug that causes TCP RST packets to be generated
on otherwise correctly behaved applications, e.g., no unread data
on close,..., etc. To trigger the bug, at least two conditions must
be met:
1. The FIN flag is set on the last data packet, i.e., it's not on a
separate, FIN only packet.
2. The size of the last data chunk on the receive side matches
exactly with the size of buffer posted by the receiver, and the
receiver closes the socket without any further read attempt.
This bug was first noticed on our netperf based testbed for our IW10
proposal to IETF where a large number of RST packets were observed.
netperf's read side code meets the condition 2 above 100%.
Before the fix, tcp_data_queue() will queue the last skb that meets
condition 1 to sk_receive_queue even though it has fully copied out
(skb_copy_datagram_iovec()) the data. Then if condition 2 is also met,
tcp_recvmsg() often returns all the copied out data successfully
without actually consuming the skb, due to a check
"if ((chunk = len - tp->ucopy.len) != 0) {"
and
"len -= chunk;"
after tcp_prequeue_process() that causes "len" to become 0 and an
early exit from the big while loop.
I don't see any reason not to free the skb whose data have been fully
consumed in tcp_data_queue(), regardless of the FIN flag. We won't
get there if MSG_PEEK is on. Am I missing some arcane cases related
to urgent data?
Signed-off-by: H.K. Jerry Chu <hkchu@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-01-25 21:46:30 +00:00
|
|
|
eaten = (chunk == skb->len);
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_rcv_space_adjust(sk);
|
|
|
|
}
|
|
|
|
local_bh_disable();
|
|
|
|
}
|
|
|
|
|
|
|
|
if (eaten <= 0) {
|
|
|
|
queue_and_out:
|
|
|
|
if (eaten < 0 &&
|
2008-04-15 07:33:38 +00:00
|
|
|
tcp_try_rmem_schedule(sk, skb->truesize))
|
|
|
|
goto drop;
|
|
|
|
|
2012-05-02 09:58:29 +00:00
|
|
|
eaten = tcp_queue_rcv(sk, skb, 0, &fragstolen);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
tp->rcv_nxt = TCP_SKB_CB(skb)->end_seq;
|
2007-03-09 04:45:19 +00:00
|
|
|
if (skb->len)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_event_data_recv(sk, skb);
|
2007-03-09 04:45:19 +00:00
|
|
|
if (th->fin)
|
2011-10-20 21:44:03 +00:00
|
|
|
tcp_fin(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2005-07-08 21:57:23 +00:00
|
|
|
if (!skb_queue_empty(&tp->out_of_order_queue)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_ofo_queue(sk);
|
|
|
|
|
|
|
|
/* RFC2581. 4.2. SHOULD send immediate ACK, when
|
|
|
|
* gap in queue is filled.
|
|
|
|
*/
|
2005-07-08 21:57:23 +00:00
|
|
|
if (skb_queue_empty(&tp->out_of_order_queue))
|
2005-08-10 03:10:42 +00:00
|
|
|
inet_csk(sk)->icsk_ack.pingpong = 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (tp->rx_opt.num_sacks)
|
|
|
|
tcp_sack_remove(tp);
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_fast_path_check(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2012-05-02 07:55:58 +00:00
|
|
|
if (eaten > 0)
|
|
|
|
kfree_skb_partial(skb, fragstolen);
|
|
|
|
else if (!sock_flag(sk, SOCK_DEAD))
|
2005-04-16 22:20:36 +00:00
|
|
|
sk->sk_data_ready(sk, 0);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt)) {
|
|
|
|
/* A retransmit, 2nd most common case. Force an immediate ack. */
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_DELAYEDACKLOST);
|
2008-07-17 03:29:51 +00:00
|
|
|
tcp_dsack_set(sk, TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
out_of_window:
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_enter_quickack_mode(sk);
|
|
|
|
inet_csk_schedule_ack(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
drop:
|
|
|
|
__kfree_skb(skb);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Out of window. F.e. zero window probe. */
|
|
|
|
if (!before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt + tcp_receive_window(tp)))
|
|
|
|
goto out_of_window;
|
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_enter_quickack_mode(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt)) {
|
|
|
|
/* Partial packet, seq < rcv_next < end_seq */
|
|
|
|
SOCK_DEBUG(sk, "partial packet: rcv_next %X seq %X - %X\n",
|
|
|
|
tp->rcv_nxt, TCP_SKB_CB(skb)->seq,
|
|
|
|
TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
2008-07-17 03:29:51 +00:00
|
|
|
tcp_dsack_set(sk, TCP_SKB_CB(skb)->seq, tp->rcv_nxt);
|
2007-02-09 14:24:47 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* If window is closed, drop tail of packet. But after
|
|
|
|
* remembering D-SACK for its head made in previous line.
|
|
|
|
*/
|
|
|
|
if (!tcp_receive_window(tp))
|
|
|
|
goto out_of_window;
|
|
|
|
goto queue_and_out;
|
|
|
|
}
|
|
|
|
|
2012-03-18 11:06:44 +00:00
|
|
|
tcp_data_queue_ofo(sk, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2008-08-23 12:11:41 +00:00
|
|
|
static struct sk_buff *tcp_collapse_one(struct sock *sk, struct sk_buff *skb,
|
|
|
|
struct sk_buff_head *list)
|
|
|
|
{
|
2009-05-29 04:35:47 +00:00
|
|
|
struct sk_buff *next = NULL;
|
|
|
|
|
|
|
|
if (!skb_queue_is_last(list, skb))
|
|
|
|
next = skb_queue_next(list, skb);
|
2008-08-23 12:11:41 +00:00
|
|
|
|
|
|
|
__skb_unlink(skb, list);
|
|
|
|
__kfree_skb(skb);
|
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPRCVCOLLAPSED);
|
|
|
|
|
|
|
|
return next;
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Collapse contiguous sequence of skbs head..tail with
|
|
|
|
* sequence numbers start..end.
|
2009-05-29 04:35:47 +00:00
|
|
|
*
|
|
|
|
* If tail is NULL, this means until the end of the list.
|
|
|
|
*
|
2005-04-16 22:20:36 +00:00
|
|
|
* Segments with FIN/SYN are not collapsed (only because this
|
|
|
|
* simplifies code)
|
|
|
|
*/
|
|
|
|
static void
|
2005-08-10 02:25:21 +00:00
|
|
|
tcp_collapse(struct sock *sk, struct sk_buff_head *list,
|
|
|
|
struct sk_buff *head, struct sk_buff *tail,
|
|
|
|
u32 start, u32 end)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2009-05-29 04:35:47 +00:00
|
|
|
struct sk_buff *skb, *n;
|
|
|
|
bool end_of_skbs;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2005-11-11 01:13:47 +00:00
|
|
|
/* First, check that queue is collapsible and find
|
2005-04-16 22:20:36 +00:00
|
|
|
* the point where collapsing can be useful. */
|
2009-05-29 04:35:47 +00:00
|
|
|
skb = head;
|
|
|
|
restart:
|
|
|
|
end_of_skbs = true;
|
|
|
|
skb_queue_walk_from_safe(list, skb, n) {
|
|
|
|
if (skb == tail)
|
|
|
|
break;
|
2005-04-16 22:20:36 +00:00
|
|
|
/* No new bits? It is possible on ofo queue. */
|
|
|
|
if (!before(start, TCP_SKB_CB(skb)->end_seq)) {
|
2008-08-23 12:11:41 +00:00
|
|
|
skb = tcp_collapse_one(sk, skb, list);
|
2009-05-29 04:35:47 +00:00
|
|
|
if (!skb)
|
|
|
|
break;
|
|
|
|
goto restart;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* The first skb to collapse is:
|
|
|
|
* - not SYN/FIN and
|
|
|
|
* - bloated or contains data before "start" or
|
|
|
|
* overlaps to the next one.
|
|
|
|
*/
|
2007-04-11 04:04:22 +00:00
|
|
|
if (!tcp_hdr(skb)->syn && !tcp_hdr(skb)->fin &&
|
2005-04-16 22:20:36 +00:00
|
|
|
(tcp_win_from_space(skb->truesize) > skb->len ||
|
2009-05-29 04:35:47 +00:00
|
|
|
before(TCP_SKB_CB(skb)->seq, start))) {
|
|
|
|
end_of_skbs = false;
|
2005-04-16 22:20:36 +00:00
|
|
|
break;
|
2009-05-29 04:35:47 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (!skb_queue_is_last(list, skb)) {
|
|
|
|
struct sk_buff *next = skb_queue_next(list, skb);
|
|
|
|
if (next != tail &&
|
|
|
|
TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(next)->seq) {
|
|
|
|
end_of_skbs = false;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Decided to skip this, advance start seq. */
|
|
|
|
start = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
}
|
2009-05-29 04:35:47 +00:00
|
|
|
if (end_of_skbs || tcp_hdr(skb)->syn || tcp_hdr(skb)->fin)
|
2005-04-16 22:20:36 +00:00
|
|
|
return;
|
|
|
|
|
|
|
|
while (before(start, end)) {
|
|
|
|
struct sk_buff *nskb;
|
2007-10-24 04:07:32 +00:00
|
|
|
unsigned int header = skb_headroom(skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
int copy = SKB_MAX_ORDER(header, 0);
|
|
|
|
|
|
|
|
/* Too big header? This can happen with IPv6. */
|
|
|
|
if (copy < 0)
|
|
|
|
return;
|
2007-12-31 22:57:14 +00:00
|
|
|
if (end - start < copy)
|
|
|
|
copy = end - start;
|
|
|
|
nskb = alloc_skb(copy + header, GFP_ATOMIC);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (!nskb)
|
|
|
|
return;
|
2007-03-10 15:47:22 +00:00
|
|
|
|
2007-03-19 22:33:04 +00:00
|
|
|
skb_set_mac_header(nskb, skb_mac_header(skb) - skb->head);
|
2007-04-26 01:04:18 +00:00
|
|
|
skb_set_network_header(nskb, (skb_network_header(skb) -
|
|
|
|
skb->head));
|
|
|
|
skb_set_transport_header(nskb, (skb_transport_header(skb) -
|
|
|
|
skb->head));
|
2005-04-16 22:20:36 +00:00
|
|
|
skb_reserve(nskb, header);
|
|
|
|
memcpy(nskb->head, skb->head, header);
|
|
|
|
memcpy(nskb->cb, skb->cb, sizeof(skb->cb));
|
|
|
|
TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start;
|
2008-09-22 04:28:51 +00:00
|
|
|
__skb_queue_before(list, skb, nskb);
|
2007-12-31 08:11:19 +00:00
|
|
|
skb_set_owner_r(nskb, sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Copy data, releasing collapsed skbs. */
|
|
|
|
while (copy > 0) {
|
|
|
|
int offset = start - TCP_SKB_CB(skb)->seq;
|
|
|
|
int size = TCP_SKB_CB(skb)->end_seq - start;
|
|
|
|
|
2006-01-09 06:24:28 +00:00
|
|
|
BUG_ON(offset < 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (size > 0) {
|
|
|
|
size = min(copy, size);
|
|
|
|
if (skb_copy_bits(skb, offset, skb_put(nskb, size), size))
|
|
|
|
BUG();
|
|
|
|
TCP_SKB_CB(nskb)->end_seq += size;
|
|
|
|
copy -= size;
|
|
|
|
start += size;
|
|
|
|
}
|
|
|
|
if (!before(start, TCP_SKB_CB(skb)->end_seq)) {
|
2008-08-23 12:11:41 +00:00
|
|
|
skb = tcp_collapse_one(sk, skb, list);
|
2009-05-29 04:35:47 +00:00
|
|
|
if (!skb ||
|
|
|
|
skb == tail ||
|
2007-04-11 04:04:22 +00:00
|
|
|
tcp_hdr(skb)->syn ||
|
|
|
|
tcp_hdr(skb)->fin)
|
2005-04-16 22:20:36 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Collapse ofo queue. Algorithm: select contiguous sequence of skbs
|
|
|
|
* and tcp_collapse() them until all the queue is collapsed.
|
|
|
|
*/
|
|
|
|
static void tcp_collapse_ofo_queue(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb = skb_peek(&tp->out_of_order_queue);
|
|
|
|
struct sk_buff *head;
|
|
|
|
u32 start, end;
|
|
|
|
|
|
|
|
if (skb == NULL)
|
|
|
|
return;
|
|
|
|
|
|
|
|
start = TCP_SKB_CB(skb)->seq;
|
|
|
|
end = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
head = skb;
|
|
|
|
|
|
|
|
for (;;) {
|
2009-05-29 04:35:47 +00:00
|
|
|
struct sk_buff *next = NULL;
|
|
|
|
|
|
|
|
if (!skb_queue_is_last(&tp->out_of_order_queue, skb))
|
|
|
|
next = skb_queue_next(&tp->out_of_order_queue, skb);
|
|
|
|
skb = next;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Segment is terminated when we see gap or when
|
|
|
|
* we are at the end of all the queue. */
|
2009-05-29 04:35:47 +00:00
|
|
|
if (!skb ||
|
2005-04-16 22:20:36 +00:00
|
|
|
after(TCP_SKB_CB(skb)->seq, end) ||
|
|
|
|
before(TCP_SKB_CB(skb)->end_seq, start)) {
|
2005-08-10 02:25:21 +00:00
|
|
|
tcp_collapse(sk, &tp->out_of_order_queue,
|
|
|
|
head, skb, start, end);
|
2005-04-16 22:20:36 +00:00
|
|
|
head = skb;
|
2009-05-29 04:35:47 +00:00
|
|
|
if (!skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
break;
|
|
|
|
/* Start new segment */
|
|
|
|
start = TCP_SKB_CB(skb)->seq;
|
|
|
|
end = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
} else {
|
|
|
|
if (before(TCP_SKB_CB(skb)->seq, start))
|
|
|
|
start = TCP_SKB_CB(skb)->seq;
|
|
|
|
if (after(TCP_SKB_CB(skb)->end_seq, end))
|
|
|
|
end = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-04-15 07:33:38 +00:00
|
|
|
/*
|
|
|
|
* Purge the out-of-order queue.
|
2008-04-16 03:26:34 +00:00
|
|
|
* Return true if queue was pruned.
|
2008-04-15 07:33:38 +00:00
|
|
|
*/
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_prune_ofo_queue(struct sock *sk)
|
2008-04-15 07:33:38 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2012-05-16 23:15:34 +00:00
|
|
|
bool res = false;
|
2008-04-15 07:33:38 +00:00
|
|
|
|
|
|
|
if (!skb_queue_empty(&tp->out_of_order_queue)) {
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_OFOPRUNED);
|
2008-04-15 07:33:38 +00:00
|
|
|
__skb_queue_purge(&tp->out_of_order_queue);
|
|
|
|
|
|
|
|
/* Reset SACK state. A conforming SACK implementation will
|
|
|
|
* do the same at a timeout based retransmit. When a connection
|
|
|
|
* is in a sad state like this, we care only about integrity
|
|
|
|
* of the connection not performance.
|
|
|
|
*/
|
|
|
|
if (tp->rx_opt.sack_ok)
|
|
|
|
tcp_sack_reset(&tp->rx_opt);
|
|
|
|
sk_mem_reclaim(sk);
|
2012-05-16 23:15:34 +00:00
|
|
|
res = true;
|
2008-04-15 07:33:38 +00:00
|
|
|
}
|
2008-04-16 03:26:34 +00:00
|
|
|
return res;
|
2008-04-15 07:33:38 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Reduce allocated memory if we can, trying to get
|
|
|
|
* the socket within its memory limits again.
|
|
|
|
*
|
|
|
|
* Return less than zero if we should start dropping frames
|
|
|
|
* until the socket owning process reads some of the data
|
|
|
|
* to stabilize the situation.
|
|
|
|
*/
|
|
|
|
static int tcp_prune_queue(struct sock *sk)
|
|
|
|
{
|
2007-02-09 14:24:47 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
SOCK_DEBUG(sk, "prune_queue: c=%x\n", tp->copied_seq);
|
|
|
|
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_PRUNECALLED);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) >= sk->sk_rcvbuf)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_clamp_window(sk);
|
2011-12-11 21:47:02 +00:00
|
|
|
else if (sk_under_memory_pressure(sk))
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->rcv_ssthresh = min(tp->rcv_ssthresh, 4U * tp->advmss);
|
|
|
|
|
|
|
|
tcp_collapse_ofo_queue(sk);
|
2009-05-29 04:35:47 +00:00
|
|
|
if (!skb_queue_empty(&sk->sk_receive_queue))
|
|
|
|
tcp_collapse(sk, &sk->sk_receive_queue,
|
|
|
|
skb_peek(&sk->sk_receive_queue),
|
|
|
|
NULL,
|
|
|
|
tp->copied_seq, tp->rcv_nxt);
|
2007-12-31 08:11:19 +00:00
|
|
|
sk_mem_reclaim(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* Collapsing did not help, destructive actions follow.
|
|
|
|
* This must not ever occur. */
|
|
|
|
|
2008-04-15 07:33:38 +00:00
|
|
|
tcp_prune_ofo_queue(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* If we are really being abused, tell the caller to silently
|
|
|
|
* drop receive data on the floor. It will get retransmitted
|
|
|
|
* and hopefully then we'll have sufficient space.
|
|
|
|
*/
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_RCVPRUNED);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Massive buffer overcommit. */
|
|
|
|
tp->pred_flags = 0;
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* RFC2861, slow part. Adjust cwnd, after it was not full during one rto.
|
|
|
|
* As additional protections, we do not touch cwnd in retransmission phases,
|
|
|
|
* and if application hit its sndbuf limit recently.
|
|
|
|
*/
|
|
|
|
void tcp_cwnd_application_limited(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
if (inet_csk(sk)->icsk_ca_state == TCP_CA_Open &&
|
2005-04-16 22:20:36 +00:00
|
|
|
sk->sk_socket && !test_bit(SOCK_NOSPACE, &sk->sk_socket->flags)) {
|
|
|
|
/* Limited by application or receiver window. */
|
2006-08-04 23:57:42 +00:00
|
|
|
u32 init_win = tcp_init_cwnd(tp, __sk_dst_get(sk));
|
|
|
|
u32 win_used = max(tp->snd_cwnd_used, init_win);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (win_used < tp->snd_cwnd) {
|
2005-08-10 07:03:31 +00:00
|
|
|
tp->snd_ssthresh = tcp_current_ssthresh(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
tp->snd_cwnd = (tp->snd_cwnd + win_used) >> 1;
|
|
|
|
}
|
|
|
|
tp->snd_cwnd_used = 0;
|
|
|
|
}
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
}
|
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_should_expand_sndbuf(const struct sock *sk)
|
2005-07-05 22:21:10 +00:00
|
|
|
{
|
2011-10-21 09:22:42 +00:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
|
2005-07-05 22:21:10 +00:00
|
|
|
/* If the user specified a specific send buffer setting, do
|
|
|
|
* not modify it.
|
|
|
|
*/
|
|
|
|
if (sk->sk_userlocks & SOCK_SNDBUF_LOCK)
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2005-07-05 22:21:10 +00:00
|
|
|
|
|
|
|
/* If we are under global TCP memory pressure, do not expand. */
|
2011-12-11 21:47:02 +00:00
|
|
|
if (sk_under_memory_pressure(sk))
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2005-07-05 22:21:10 +00:00
|
|
|
|
|
|
|
/* If we are under soft global TCP memory pressure, do not expand. */
|
2011-12-11 21:47:02 +00:00
|
|
|
if (sk_memory_allocated(sk) >= sk_prot_mem_limits(sk, 0))
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2005-07-05 22:21:10 +00:00
|
|
|
|
|
|
|
/* If we filled the congestion window, do not expand. */
|
|
|
|
if (tp->packets_out >= tp->snd_cwnd)
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2005-07-05 22:21:10 +00:00
|
|
|
|
2012-05-16 23:15:34 +00:00
|
|
|
return true;
|
2005-07-05 22:21:10 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* When incoming ACK allowed to free some skb from write_queue,
|
|
|
|
* we remember this event in flag SOCK_QUEUE_SHRUNK and wake up socket
|
|
|
|
* on the exit from tcp input handler.
|
|
|
|
*
|
|
|
|
* PROBLEM: sndbuf expansion does not work well with largesend.
|
|
|
|
*/
|
|
|
|
static void tcp_new_space(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
if (tcp_should_expand_sndbuf(sk)) {
|
2011-10-13 07:28:54 +00:00
|
|
|
int sndmem = SKB_TRUESIZE(max_t(u32,
|
|
|
|
tp->rx_opt.mss_clamp,
|
|
|
|
tp->mss_cache) +
|
|
|
|
MAX_TCP_HEADER);
|
2008-10-07 21:43:31 +00:00
|
|
|
int demanded = max_t(unsigned int, tp->snd_cwnd,
|
2007-12-31 22:57:14 +00:00
|
|
|
tp->reordering + 1);
|
|
|
|
sndmem *= 2 * demanded;
|
2005-04-16 22:20:36 +00:00
|
|
|
if (sndmem > sk->sk_sndbuf)
|
|
|
|
sk->sk_sndbuf = min(sndmem, sysctl_tcp_wmem[2]);
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
}
|
|
|
|
|
|
|
|
sk->sk_write_space(sk);
|
|
|
|
}
|
|
|
|
|
2006-01-04 00:03:49 +00:00
|
|
|
static void tcp_check_space(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
if (sock_flag(sk, SOCK_QUEUE_SHRUNK)) {
|
|
|
|
sock_reset_flag(sk, SOCK_QUEUE_SHRUNK);
|
|
|
|
if (sk->sk_socket &&
|
|
|
|
test_bit(SOCK_NOSPACE, &sk->sk_socket->flags))
|
|
|
|
tcp_new_space(sk);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
static inline void tcp_data_snd_check(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_push_pending_frames(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_check_space(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Check if sending an ack is needed.
|
|
|
|
*/
|
|
|
|
static void __tcp_ack_snd_check(struct sock *sk, int ofo_possible)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
/* More than one full frame received... */
|
2009-11-23 18:41:23 +00:00
|
|
|
if (((tp->rcv_nxt - tp->rcv_wup) > inet_csk(sk)->icsk_ack.rcv_mss &&
|
2005-04-16 22:20:36 +00:00
|
|
|
/* ... and right edge of window advances far enough.
|
|
|
|
* (tcp_recvmsg() will send ACK otherwise). Or...
|
|
|
|
*/
|
2009-11-23 18:41:23 +00:00
|
|
|
__tcp_select_window(sk) >= tp->rcv_wnd) ||
|
2005-04-16 22:20:36 +00:00
|
|
|
/* We ACK each frame or... */
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_in_quickack_mode(sk) ||
|
2005-04-16 22:20:36 +00:00
|
|
|
/* We have out of order data. */
|
2007-12-31 22:57:14 +00:00
|
|
|
(ofo_possible && skb_peek(&tp->out_of_order_queue))) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Then ack it now */
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
} else {
|
|
|
|
/* Else, send delayed ack. */
|
|
|
|
tcp_send_delayed_ack(sk);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2006-01-04 00:03:49 +00:00
|
|
|
static inline void tcp_ack_snd_check(struct sock *sk)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2005-08-10 03:10:42 +00:00
|
|
|
if (!inet_csk_ack_scheduled(sk)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* We sent a data segment already. */
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
__tcp_ack_snd_check(sk, 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This routine is only called when we have urgent data
|
2005-11-11 01:13:47 +00:00
|
|
|
* signaled. Its the 'slow' part of tcp_urg. It could be
|
2005-04-16 22:20:36 +00:00
|
|
|
* moved inline now as tcp_urg is only called from one
|
|
|
|
* place. We handle URGent data wrong. We have to - as
|
|
|
|
* BSD still doesn't use the correction from RFC961.
|
|
|
|
* For 1003.1g we should support a new option TCP_STDURG to permit
|
|
|
|
* either form (or just set the sysctl tcp_stdurg).
|
|
|
|
*/
|
2007-02-09 14:24:47 +00:00
|
|
|
|
2011-10-21 09:22:42 +00:00
|
|
|
static void tcp_check_urg(struct sock *sk, const struct tcphdr *th)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u32 ptr = ntohs(th->urg_ptr);
|
|
|
|
|
|
|
|
if (ptr && !sysctl_tcp_stdurg)
|
|
|
|
ptr--;
|
|
|
|
ptr += ntohl(th->seq);
|
|
|
|
|
|
|
|
/* Ignore urgent data that we've already seen and read. */
|
|
|
|
if (after(tp->copied_seq, ptr))
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Do not replay urg ptr.
|
|
|
|
*
|
|
|
|
* NOTE: interesting situation not covered by specs.
|
|
|
|
* Misbehaving sender may send urg ptr, pointing to segment,
|
|
|
|
* which we already have in ofo queue. We are not able to fetch
|
|
|
|
* such data and will stay in TCP_URG_NOTYET until will be eaten
|
|
|
|
* by recvmsg(). Seems, we are not obliged to handle such wicked
|
|
|
|
* situations. But it is worth to think about possibility of some
|
|
|
|
* DoSes using some hypothetical application level deadlock.
|
|
|
|
*/
|
|
|
|
if (before(ptr, tp->rcv_nxt))
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Do we already have a newer (or duplicate) urgent pointer? */
|
|
|
|
if (tp->urg_data && !after(ptr, tp->urg_seq))
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Tell the world about our new urgent pointer. */
|
|
|
|
sk_send_sigurg(sk);
|
|
|
|
|
|
|
|
/* We may be adding urgent data when the last byte read was
|
|
|
|
* urgent. To do this requires some care. We cannot just ignore
|
|
|
|
* tp->copied_seq since we would read the last urgent byte again
|
|
|
|
* as data, nor can we alter copied_seq until this data arrives
|
2005-11-11 01:13:47 +00:00
|
|
|
* or we break the semantics of SIOCATMARK (and thus sockatmark())
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
|
|
|
* NOTE. Double Dutch. Rendering to plain English: author of comment
|
|
|
|
* above did something sort of send("A", MSG_OOB); send("B", MSG_OOB);
|
|
|
|
* and expect that both A and B disappear from stream. This is _wrong_.
|
|
|
|
* Though this happens in BSD with high probability, this is occasional.
|
|
|
|
* Any application relying on this is buggy. Note also, that fix "works"
|
|
|
|
* only in this artificial test. Insert some normal data between A and B and we will
|
|
|
|
* decline of BSD again. Verdict: it is better to remove to trap
|
|
|
|
* buggy users.
|
|
|
|
*/
|
|
|
|
if (tp->urg_seq == tp->copied_seq && tp->urg_data &&
|
2007-12-31 22:57:14 +00:00
|
|
|
!sock_flag(sk, SOCK_URGINLINE) && tp->copied_seq != tp->rcv_nxt) {
|
2005-04-16 22:20:36 +00:00
|
|
|
struct sk_buff *skb = skb_peek(&sk->sk_receive_queue);
|
|
|
|
tp->copied_seq++;
|
|
|
|
if (skb && !before(tp->copied_seq, TCP_SKB_CB(skb)->end_seq)) {
|
2005-08-10 02:25:21 +00:00
|
|
|
__skb_unlink(skb, &sk->sk_receive_queue);
|
2005-04-16 22:20:36 +00:00
|
|
|
__kfree_skb(skb);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
tp->urg_data = TCP_URG_NOTYET;
|
|
|
|
tp->urg_seq = ptr;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Disable header prediction. */
|
|
|
|
tp->pred_flags = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* This is the 'fast' part of urgent handling. */
|
2011-10-21 09:22:42 +00:00
|
|
|
static void tcp_urg(struct sock *sk, struct sk_buff *skb, const struct tcphdr *th)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
/* Check if we get a new urgent pointer - normally not. */
|
|
|
|
if (th->urg)
|
2007-12-31 22:57:14 +00:00
|
|
|
tcp_check_urg(sk, th);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Do we wait for any urgent data? - normally not... */
|
|
|
|
if (tp->urg_data == TCP_URG_NOTYET) {
|
|
|
|
u32 ptr = tp->urg_seq - ntohl(th->seq) + (th->doff * 4) -
|
|
|
|
th->syn;
|
|
|
|
|
2007-02-09 14:24:47 +00:00
|
|
|
/* Is the urgent pointer pointing into this packet? */
|
2005-04-16 22:20:36 +00:00
|
|
|
if (ptr < skb->len) {
|
|
|
|
u8 tmp;
|
|
|
|
if (skb_copy_bits(skb, ptr, &tmp, 1))
|
|
|
|
BUG();
|
|
|
|
tp->urg_data = TCP_URG_VALID | tmp;
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD))
|
|
|
|
sk->sk_data_ready(sk, 0);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static int tcp_copy_to_iovec(struct sock *sk, struct sk_buff *skb, int hlen)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int chunk = skb->len - hlen;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
local_bh_enable();
|
2007-04-09 18:59:39 +00:00
|
|
|
if (skb_csum_unnecessary(skb))
|
2005-04-16 22:20:36 +00:00
|
|
|
err = skb_copy_datagram_iovec(skb, hlen, tp->ucopy.iov, chunk);
|
|
|
|
else
|
|
|
|
err = skb_copy_and_csum_datagram_iovec(skb, hlen,
|
|
|
|
tp->ucopy.iov);
|
|
|
|
|
|
|
|
if (!err) {
|
|
|
|
tp->ucopy.len -= chunk;
|
|
|
|
tp->copied_seq += chunk;
|
|
|
|
tcp_rcv_space_adjust(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
local_bh_disable();
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
static __sum16 __tcp_checksum_complete_user(struct sock *sk,
|
|
|
|
struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2006-11-15 05:40:42 +00:00
|
|
|
__sum16 result;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (sock_owned_by_user(sk)) {
|
|
|
|
local_bh_enable();
|
|
|
|
result = __tcp_checksum_complete(skb);
|
|
|
|
local_bh_disable();
|
|
|
|
} else {
|
|
|
|
result = __tcp_checksum_complete(skb);
|
|
|
|
}
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2007-12-31 22:57:14 +00:00
|
|
|
static inline int tcp_checksum_complete_user(struct sock *sk,
|
|
|
|
struct sk_buff *skb)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2007-04-09 18:59:39 +00:00
|
|
|
return !skb_csum_unnecessary(skb) &&
|
2007-12-31 22:57:14 +00:00
|
|
|
__tcp_checksum_complete_user(sk, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2006-05-24 01:05:53 +00:00
|
|
|
#ifdef CONFIG_NET_DMA
|
2012-05-16 23:15:34 +00:00
|
|
|
static bool tcp_dma_try_early_copy(struct sock *sk, struct sk_buff *skb,
|
2007-12-31 22:57:14 +00:00
|
|
|
int hlen)
|
2006-05-24 01:05:53 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int chunk = skb->len - hlen;
|
|
|
|
int dma_cookie;
|
2012-05-16 23:15:34 +00:00
|
|
|
bool copied_early = false;
|
2006-05-24 01:05:53 +00:00
|
|
|
|
|
|
|
if (tp->ucopy.wakeup)
|
2012-05-16 23:15:34 +00:00
|
|
|
return false;
|
2006-05-24 01:05:53 +00:00
|
|
|
|
|
|
|
if (!tp->ucopy.dma_chan && tp->ucopy.pinned_list)
|
2012-04-04 23:10:46 +00:00
|
|
|
tp->ucopy.dma_chan = net_dma_find_channel();
|
2006-05-24 01:05:53 +00:00
|
|
|
|
2007-04-09 18:59:39 +00:00
|
|
|
if (tp->ucopy.dma_chan && skb_csum_unnecessary(skb)) {
|
2006-05-24 01:05:53 +00:00
|
|
|
|
|
|
|
dma_cookie = dma_skb_copy_datagram_iovec(tp->ucopy.dma_chan,
|
2007-12-31 22:57:14 +00:00
|
|
|
skb, hlen,
|
|
|
|
tp->ucopy.iov, chunk,
|
|
|
|
tp->ucopy.pinned_list);
|
2006-05-24 01:05:53 +00:00
|
|
|
|
|
|
|
if (dma_cookie < 0)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
tp->ucopy.dma_cookie = dma_cookie;
|
2012-05-16 23:15:34 +00:00
|
|
|
copied_early = true;
|
2006-05-24 01:05:53 +00:00
|
|
|
|
|
|
|
tp->ucopy.len -= chunk;
|
|
|
|
tp->copied_seq += chunk;
|
|
|
|
tcp_rcv_space_adjust(sk);
|
|
|
|
|
|
|
|
if ((tp->ucopy.len == 0) ||
|
2007-04-11 04:04:22 +00:00
|
|
|
(tcp_flag_word(tcp_hdr(skb)) & TCP_FLAG_PSH) ||
|
2006-05-24 01:05:53 +00:00
|
|
|
(atomic_read(&sk->sk_rmem_alloc) > (sk->sk_rcvbuf >> 1))) {
|
|
|
|
tp->ucopy.wakeup = 1;
|
|
|
|
sk->sk_data_ready(sk, 0);
|
|
|
|
}
|
|
|
|
} else if (chunk > 0) {
|
|
|
|
tp->ucopy.wakeup = 1;
|
|
|
|
sk->sk_data_ready(sk, 0);
|
|
|
|
}
|
|
|
|
out:
|
|
|
|
return copied_early;
|
|
|
|
}
|
|
|
|
#endif /* CONFIG_NET_DMA */
|
|
|
|
|
2008-08-23 12:10:12 +00:00
|
|
|
/* Does PAWS and seqno based validation of an incoming segment, flags will
|
|
|
|
* play significant role here.
|
|
|
|
*/
|
|
|
|
static int tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,
|
2011-10-21 09:22:42 +00:00
|
|
|
const struct tcphdr *th, int syn_inerr)
|
2008-08-23 12:10:12 +00:00
|
|
|
{
|
2011-10-21 09:22:42 +00:00
|
|
|
const u8 *hash_location;
|
2008-08-23 12:10:12 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
/* RFC1323: H1. Apply PAWS check first. */
|
2009-12-02 18:25:27 +00:00
|
|
|
if (tcp_fast_parse_options(skb, th, tp, &hash_location) &&
|
|
|
|
tp->rx_opt.saw_tstamp &&
|
2008-08-23 12:10:12 +00:00
|
|
|
tcp_paws_discard(sk, skb)) {
|
|
|
|
if (!th->rst) {
|
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_PAWSESTABREJECTED);
|
|
|
|
tcp_send_dupack(sk, skb);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
/* Reset is accepted even if it did not pass PAWS. */
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Step 1: check sequence number */
|
|
|
|
if (!tcp_sequence(tp, TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq)) {
|
|
|
|
/* RFC793, page 37: "In all states except SYN-SENT, all reset
|
|
|
|
* (RST) segments are validated by checking their SEQ-fields."
|
|
|
|
* And page 69: "If an incoming segment is not acceptable,
|
|
|
|
* an acknowledgment should be sent in reply (unless the RST
|
|
|
|
* bit is set, if so drop the segment and return)".
|
|
|
|
*/
|
|
|
|
if (!th->rst)
|
|
|
|
tcp_send_dupack(sk, skb);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Step 2: check RST bit */
|
|
|
|
if (th->rst) {
|
|
|
|
tcp_reset(sk);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* ts_recent update must be made after we are sure that the packet
|
|
|
|
* is in window.
|
|
|
|
*/
|
|
|
|
tcp_replace_ts_recent(tp, TCP_SKB_CB(skb)->seq);
|
|
|
|
|
|
|
|
/* step 3: check security and precedence [ignored] */
|
|
|
|
|
|
|
|
/* step 4: Check for a SYN in window. */
|
|
|
|
if (th->syn && !before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt)) {
|
|
|
|
if (syn_inerr)
|
|
|
|
TCP_INC_STATS_BH(sock_net(sk), TCP_MIB_INERRS);
|
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPABORTONSYN);
|
|
|
|
tcp_reset(sk);
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
discard:
|
|
|
|
__kfree_skb(skb);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
2007-02-09 14:24:47 +00:00
|
|
|
* TCP receive function for the ESTABLISHED state.
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
2007-02-09 14:24:47 +00:00
|
|
|
* It is split into a fast path and a slow path. The fast path is
|
2005-04-16 22:20:36 +00:00
|
|
|
* disabled when:
|
|
|
|
* - A zero window was announced from us - zero window probing
|
2007-02-09 14:24:47 +00:00
|
|
|
* is only handled properly in the slow path.
|
2005-04-16 22:20:36 +00:00
|
|
|
* - Out of order segments arrived.
|
|
|
|
* - Urgent data is expected.
|
|
|
|
* - There is no buffer space left
|
|
|
|
* - Unexpected TCP flags/window values/header lengths are received
|
2007-02-09 14:24:47 +00:00
|
|
|
* (detected by checking the TCP header against pred_flags)
|
2005-04-16 22:20:36 +00:00
|
|
|
* - Data is sent in both directions. Fast path only supports pure senders
|
|
|
|
* or pure receivers (this means either the sequence number or the ack
|
|
|
|
* value must stay constant)
|
|
|
|
* - Unexpected TCP option.
|
|
|
|
*
|
2007-02-09 14:24:47 +00:00
|
|
|
* When these conditions are not satisfied it drops into a standard
|
2005-04-16 22:20:36 +00:00
|
|
|
* receive procedure patterned after RFC793 to handle all cases.
|
|
|
|
* The first three cases are guaranteed by proper pred_flags setting,
|
2007-02-09 14:24:47 +00:00
|
|
|
* the rest is checked inline. Fast processing is turned on in
|
2005-04-16 22:20:36 +00:00
|
|
|
* tcp_data_queue when everything is OK.
|
|
|
|
*/
|
|
|
|
int tcp_rcv_established(struct sock *sk, struct sk_buff *skb,
|
2011-10-21 09:22:42 +00:00
|
|
|
const struct tcphdr *th, unsigned int len)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2008-08-23 12:10:12 +00:00
|
|
|
int res;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2012-06-20 04:22:05 +00:00
|
|
|
if (sk->sk_rx_dst) {
|
|
|
|
struct dst_entry *dst = sk->sk_rx_dst;
|
|
|
|
if (unlikely(dst->obsolete)) {
|
|
|
|
if (dst->ops->check(dst, 0) == NULL) {
|
|
|
|
dst_release(dst);
|
|
|
|
sk->sk_rx_dst = NULL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (unlikely(sk->sk_rx_dst == NULL))
|
|
|
|
sk->sk_rx_dst = dst_clone(skb_dst(skb));
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* Header prediction.
|
2007-02-09 14:24:47 +00:00
|
|
|
* The code loosely follows the one in the famous
|
2005-04-16 22:20:36 +00:00
|
|
|
* "30 instruction TCP receive" Van Jacobson mail.
|
2007-02-09 14:24:47 +00:00
|
|
|
*
|
|
|
|
* Van's trick is to deposit buffers into socket queue
|
2005-04-16 22:20:36 +00:00
|
|
|
* on a device interrupt, to call tcp_recv function
|
|
|
|
* on the receive process context and checksum and copy
|
|
|
|
* the buffer to user space. smart...
|
|
|
|
*
|
2007-02-09 14:24:47 +00:00
|
|
|
* Our current scheme is not silly either but we take the
|
2005-04-16 22:20:36 +00:00
|
|
|
* extra cost of the net_bh soft interrupt processing...
|
|
|
|
* We do checksum and copy also but from device to kernel.
|
|
|
|
*/
|
|
|
|
|
|
|
|
tp->rx_opt.saw_tstamp = 0;
|
|
|
|
|
|
|
|
/* pred_flags is 0xS?10 << 16 + snd_wnd
|
2005-11-11 01:13:47 +00:00
|
|
|
* if header_prediction is to be made
|
2005-04-16 22:20:36 +00:00
|
|
|
* 'S' will always be tp->tcp_header_len >> 2
|
|
|
|
* '?' will be 0 for the fast path, otherwise pred_flags is 0 to
|
2007-02-09 14:24:47 +00:00
|
|
|
* turn it off (when there are holes in the receive
|
2005-04-16 22:20:36 +00:00
|
|
|
* space for instance)
|
|
|
|
* PSH flag is ignored.
|
|
|
|
*/
|
|
|
|
|
|
|
|
if ((tcp_flag_word(th) & TCP_HP_BITS) == tp->pred_flags &&
|
2009-03-23 04:49:57 +00:00
|
|
|
TCP_SKB_CB(skb)->seq == tp->rcv_nxt &&
|
|
|
|
!after(TCP_SKB_CB(skb)->ack_seq, tp->snd_nxt)) {
|
2005-04-16 22:20:36 +00:00
|
|
|
int tcp_header_len = tp->tcp_header_len;
|
|
|
|
|
|
|
|
/* Timestamp header prediction: tcp_header_len
|
|
|
|
* is automatically equal to th->doff*4 due to pred_flags
|
|
|
|
* match.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* Check timestamp */
|
|
|
|
if (tcp_header_len == sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED) {
|
|
|
|
/* No? Slow path! */
|
2008-08-23 12:12:29 +00:00
|
|
|
if (!tcp_parse_aligned_timestamp(tp, th))
|
2005-04-16 22:20:36 +00:00
|
|
|
goto slow_path;
|
|
|
|
|
|
|
|
/* If PAWS failed, check it more carefully in slow path */
|
|
|
|
if ((s32)(tp->rx_opt.rcv_tsval - tp->rx_opt.ts_recent) < 0)
|
|
|
|
goto slow_path;
|
|
|
|
|
|
|
|
/* DO NOT update ts_recent here, if checksum fails
|
|
|
|
* and timestamp was corrupted part, it will result
|
|
|
|
* in a hung connection since we will drop all
|
|
|
|
* future packets due to the PAWS test.
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
|
|
|
|
if (len <= tcp_header_len) {
|
|
|
|
/* Bulk data transfer: sender */
|
|
|
|
if (len == tcp_header_len) {
|
|
|
|
/* Predicted packet is in window by definition.
|
|
|
|
* seq == rcv_nxt and rcv_wup <= rcv_nxt.
|
|
|
|
* Hence, check seq<=rcv_wup reduces to:
|
|
|
|
*/
|
|
|
|
if (tcp_header_len ==
|
|
|
|
(sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED) &&
|
|
|
|
tp->rcv_nxt == tp->rcv_wup)
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
|
|
|
|
/* We know that such packets are checksummed
|
|
|
|
* on entry.
|
|
|
|
*/
|
|
|
|
tcp_ack(sk, skb, 0);
|
2007-02-09 14:24:47 +00:00
|
|
|
__kfree_skb(skb);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_data_snd_check(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
return 0;
|
|
|
|
} else { /* Header too small */
|
2008-07-17 03:22:25 +00:00
|
|
|
TCP_INC_STATS_BH(sock_net(sk), TCP_MIB_INERRS);
|
2005-04-16 22:20:36 +00:00
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
int eaten = 0;
|
2006-05-24 01:05:53 +00:00
|
|
|
int copied_early = 0;
|
2012-05-02 09:58:29 +00:00
|
|
|
bool fragstolen = false;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-05-24 01:05:53 +00:00
|
|
|
if (tp->copied_seq == tp->rcv_nxt &&
|
|
|
|
len - tcp_header_len <= tp->ucopy.len) {
|
|
|
|
#ifdef CONFIG_NET_DMA
|
|
|
|
if (tcp_dma_try_early_copy(sk, skb, tcp_header_len)) {
|
|
|
|
copied_early = 1;
|
|
|
|
eaten = 1;
|
|
|
|
}
|
|
|
|
#endif
|
2007-12-31 22:57:14 +00:00
|
|
|
if (tp->ucopy.task == current &&
|
|
|
|
sock_owned_by_user(sk) && !copied_early) {
|
2006-05-24 01:05:53 +00:00
|
|
|
__set_current_state(TASK_RUNNING);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-05-24 01:05:53 +00:00
|
|
|
if (!tcp_copy_to_iovec(sk, skb, tcp_header_len))
|
|
|
|
eaten = 1;
|
|
|
|
}
|
|
|
|
if (eaten) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Predicted packet is in window by definition.
|
|
|
|
* seq == rcv_nxt and rcv_wup <= rcv_nxt.
|
|
|
|
* Hence, check seq<=rcv_wup reduces to:
|
|
|
|
*/
|
|
|
|
if (tcp_header_len ==
|
|
|
|
(sizeof(struct tcphdr) +
|
|
|
|
TCPOLEN_TSTAMP_ALIGNED) &&
|
|
|
|
tp->rcv_nxt == tp->rcv_wup)
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_rcv_rtt_measure_ts(sk, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
__skb_pull(skb, tcp_header_len);
|
|
|
|
tp->rcv_nxt = TCP_SKB_CB(skb)->end_seq;
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPHPHITSTOUSER);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2006-05-24 01:05:53 +00:00
|
|
|
if (copied_early)
|
|
|
|
tcp_cleanup_rbuf(sk, skb->len);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
if (!eaten) {
|
|
|
|
if (tcp_checksum_complete_user(sk, skb))
|
|
|
|
goto csum_error;
|
|
|
|
|
|
|
|
/* Predicted packet is in window by definition.
|
|
|
|
* seq == rcv_nxt and rcv_wup <= rcv_nxt.
|
|
|
|
* Hence, check seq<=rcv_wup reduces to:
|
|
|
|
*/
|
|
|
|
if (tcp_header_len ==
|
|
|
|
(sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED) &&
|
|
|
|
tp->rcv_nxt == tp->rcv_wup)
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_rcv_rtt_measure_ts(sk, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if ((int)skb->truesize > sk->sk_forward_alloc)
|
|
|
|
goto step5;
|
|
|
|
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPHPHITS);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Bulk data transfer: receiver */
|
2012-05-02 09:58:29 +00:00
|
|
|
eaten = tcp_queue_rcv(sk, skb, tcp_header_len,
|
|
|
|
&fragstolen);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_event_data_recv(sk, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (TCP_SKB_CB(skb)->ack_seq != tp->snd_una) {
|
|
|
|
/* Well, only one small jumplet in fast path... */
|
|
|
|
tcp_ack(sk, skb, FLAG_DATA);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_data_snd_check(sk);
|
2005-08-10 03:10:42 +00:00
|
|
|
if (!inet_csk_ack_scheduled(sk))
|
2005-04-16 22:20:36 +00:00
|
|
|
goto no_ack;
|
|
|
|
}
|
|
|
|
|
2008-10-07 22:31:19 +00:00
|
|
|
if (!copied_early || tp->rcv_nxt != tp->rcv_wup)
|
|
|
|
__tcp_ack_snd_check(sk, 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
no_ack:
|
2006-05-24 01:05:53 +00:00
|
|
|
#ifdef CONFIG_NET_DMA
|
|
|
|
if (copied_early)
|
|
|
|
__skb_queue_tail(&sk->sk_async_wait_queue, skb);
|
|
|
|
else
|
|
|
|
#endif
|
2005-04-16 22:20:36 +00:00
|
|
|
if (eaten)
|
2012-05-02 09:58:29 +00:00
|
|
|
kfree_skb_partial(skb, fragstolen);
|
2005-04-16 22:20:36 +00:00
|
|
|
else
|
|
|
|
sk->sk_data_ready(sk, 0);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
slow_path:
|
2007-12-31 22:57:14 +00:00
|
|
|
if (len < (th->doff << 2) || tcp_checksum_complete_user(sk, skb))
|
2005-04-16 22:20:36 +00:00
|
|
|
goto csum_error;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Standard slow path.
|
|
|
|
*/
|
|
|
|
|
2008-08-23 12:10:12 +00:00
|
|
|
res = tcp_validate_incoming(sk, skb, th, 1);
|
|
|
|
if (res <= 0)
|
|
|
|
return -res;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
step5:
|
2009-03-23 04:49:57 +00:00
|
|
|
if (th->ack && tcp_ack(sk, skb, FLAG_SLOWPATH) < 0)
|
|
|
|
goto discard;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_rcv_rtt_measure_ts(sk, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* Process urgent data. */
|
|
|
|
tcp_urg(sk, skb, th);
|
|
|
|
|
|
|
|
/* step 7: process the segment text */
|
|
|
|
tcp_data_queue(sk, skb);
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_data_snd_check(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_ack_snd_check(sk);
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
csum_error:
|
2008-07-17 03:22:25 +00:00
|
|
|
TCP_INC_STATS_BH(sock_net(sk), TCP_MIB_INERRS);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
discard:
|
|
|
|
__kfree_skb(skb);
|
|
|
|
return 0;
|
|
|
|
}
|
2010-07-09 21:22:10 +00:00
|
|
|
EXPORT_SYMBOL(tcp_rcv_established);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2012-04-19 03:40:01 +00:00
|
|
|
void tcp_finish_connect(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
tcp_set_state(sk, TCP_ESTABLISHED);
|
|
|
|
|
2012-06-20 04:22:05 +00:00
|
|
|
if (skb != NULL) {
|
|
|
|
sk->sk_rx_dst = dst_clone(skb_dst(skb));
|
2012-04-19 03:40:01 +00:00
|
|
|
security_inet_conn_established(sk, skb);
|
2012-06-20 04:22:05 +00:00
|
|
|
}
|
2012-04-19 03:40:01 +00:00
|
|
|
|
|
|
|
/* Make sure socket is routed, for correct metrics. */
|
|
|
|
icsk->icsk_af_ops->rebuild_header(sk);
|
|
|
|
|
|
|
|
tcp_init_metrics(sk);
|
|
|
|
|
|
|
|
tcp_init_congestion_control(sk);
|
|
|
|
|
|
|
|
/* Prevent spurious tcp_cwnd_restart() on first data
|
|
|
|
* packet.
|
|
|
|
*/
|
|
|
|
tp->lsndtime = tcp_time_stamp;
|
|
|
|
|
|
|
|
tcp_init_buffer_space(sk);
|
|
|
|
|
|
|
|
if (sock_flag(sk, SOCK_KEEPOPEN))
|
|
|
|
inet_csk_reset_keepalive_timer(sk, keepalive_time_when(tp));
|
|
|
|
|
|
|
|
if (!tp->rx_opt.snd_wscale)
|
|
|
|
__tcp_fast_path_on(tp, tp->snd_wnd);
|
|
|
|
else
|
|
|
|
tp->pred_flags = 0;
|
|
|
|
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD)) {
|
|
|
|
sk->sk_state_change(sk);
|
|
|
|
sk_wake_async(sk, SOCK_WAKE_IO, POLL_OUT);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
|
2011-10-21 09:22:42 +00:00
|
|
|
const struct tcphdr *th, unsigned int len)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2011-10-21 09:22:42 +00:00
|
|
|
const u8 *hash_location;
|
2005-12-14 07:26:10 +00:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2009-12-02 18:25:27 +00:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct tcp_cookie_values *cvp = tp->cookie_values;
|
|
|
|
int saved_clamp = tp->rx_opt.mss_clamp;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 04:56:42 +00:00
|
|
|
tcp_parse_options(skb, &tp->rx_opt, &hash_location, 0);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (th->ack) {
|
|
|
|
/* rfc793:
|
|
|
|
* "If the state is SYN-SENT then
|
|
|
|
* first check the ACK bit
|
|
|
|
* If the ACK bit is set
|
|
|
|
* If SEG.ACK =< ISS, or SEG.ACK > SND.NXT, send
|
|
|
|
* a reset (unless the RST bit is set, if so drop
|
|
|
|
* the segment and return)"
|
|
|
|
*
|
|
|
|
* We do not send data with SYN, so that RFC-correct
|
|
|
|
* test reduces to:
|
|
|
|
*/
|
|
|
|
if (TCP_SKB_CB(skb)->ack_seq != tp->snd_nxt)
|
|
|
|
goto reset_and_undo;
|
|
|
|
|
|
|
|
if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr &&
|
|
|
|
!between(tp->rx_opt.rcv_tsecr, tp->retrans_stamp,
|
|
|
|
tcp_time_stamp)) {
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_PAWSACTIVEREJECTED);
|
2005-04-16 22:20:36 +00:00
|
|
|
goto reset_and_undo;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Now ACK is acceptable.
|
|
|
|
*
|
|
|
|
* "If the RST bit is set
|
|
|
|
* If the ACK was acceptable then signal the user "error:
|
|
|
|
* connection reset", drop the segment, enter CLOSED state,
|
|
|
|
* delete TCB, and return."
|
|
|
|
*/
|
|
|
|
|
|
|
|
if (th->rst) {
|
|
|
|
tcp_reset(sk);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* rfc793:
|
|
|
|
* "fifth, if neither of the SYN or RST bits is set then
|
|
|
|
* drop the segment and return."
|
|
|
|
*
|
|
|
|
* See note below!
|
|
|
|
* --ANK(990513)
|
|
|
|
*/
|
|
|
|
if (!th->syn)
|
|
|
|
goto discard_and_undo;
|
|
|
|
|
|
|
|
/* rfc793:
|
|
|
|
* "If the SYN bit is on ...
|
|
|
|
* are acceptable then ...
|
|
|
|
* (our SYN has been ACKed), change the connection
|
|
|
|
* state to ESTABLISHED..."
|
|
|
|
*/
|
|
|
|
|
|
|
|
TCP_ECN_rcv_synack(tp, th);
|
|
|
|
|
|
|
|
tp->snd_wl1 = TCP_SKB_CB(skb)->seq;
|
|
|
|
tcp_ack(sk, skb, FLAG_SLOWPATH);
|
|
|
|
|
|
|
|
/* Ok.. it's good. Set up sequence numbers and
|
|
|
|
* move to established.
|
|
|
|
*/
|
|
|
|
tp->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;
|
|
|
|
tp->rcv_wup = TCP_SKB_CB(skb)->seq + 1;
|
|
|
|
|
|
|
|
/* RFC1323: The window in SYN & SYN/ACK segments is
|
|
|
|
* never scaled.
|
|
|
|
*/
|
|
|
|
tp->snd_wnd = ntohs(th->window);
|
2009-03-03 06:42:02 +00:00
|
|
|
tcp_init_wl(tp, TCP_SKB_CB(skb)->seq);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (!tp->rx_opt.wscale_ok) {
|
|
|
|
tp->rx_opt.snd_wscale = tp->rx_opt.rcv_wscale = 0;
|
|
|
|
tp->window_clamp = min(tp->window_clamp, 65535U);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (tp->rx_opt.saw_tstamp) {
|
|
|
|
tp->rx_opt.tstamp_ok = 1;
|
|
|
|
tp->tcp_header_len =
|
|
|
|
sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
tp->advmss -= TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
} else {
|
|
|
|
tp->tcp_header_len = sizeof(struct tcphdr);
|
|
|
|
}
|
|
|
|
|
2007-08-09 12:14:46 +00:00
|
|
|
if (tcp_is_sack(tp) && sysctl_tcp_fack)
|
|
|
|
tcp_enable_fack(tp);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-03-21 01:53:41 +00:00
|
|
|
tcp_mtup_init(sk);
|
2005-12-14 07:26:10 +00:00
|
|
|
tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_initialize_rcv_mss(sk);
|
|
|
|
|
|
|
|
/* Remember, tcp_poll() does not lock socket!
|
|
|
|
* Change state from SYN-SENT only after copied_seq
|
|
|
|
* is initialized. */
|
|
|
|
tp->copied_seq = tp->rcv_nxt;
|
2009-12-02 18:25:27 +00:00
|
|
|
|
|
|
|
if (cvp != NULL &&
|
|
|
|
cvp->cookie_pair_size > 0 &&
|
|
|
|
tp->rx_opt.cookie_plus > 0) {
|
|
|
|
int cookie_size = tp->rx_opt.cookie_plus
|
|
|
|
- TCPOLEN_COOKIE_BASE;
|
|
|
|
int cookie_pair_size = cookie_size
|
|
|
|
+ cvp->cookie_desired;
|
|
|
|
|
|
|
|
/* A cookie extension option was sent and returned.
|
|
|
|
* Note that each incoming SYNACK replaces the
|
|
|
|
* Responder cookie. The initial exchange is most
|
|
|
|
* fragile, as protection against spoofing relies
|
|
|
|
* entirely upon the sequence and timestamp (above).
|
|
|
|
* This replacement strategy allows the correct pair to
|
|
|
|
* pass through, while any others will be filtered via
|
|
|
|
* Responder verification later.
|
|
|
|
*/
|
|
|
|
if (sizeof(cvp->cookie_pair) >= cookie_pair_size) {
|
|
|
|
memcpy(&cvp->cookie_pair[cvp->cookie_desired],
|
|
|
|
hash_location, cookie_size);
|
|
|
|
cvp->cookie_pair_size = cookie_pair_size;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2006-12-07 08:11:33 +00:00
|
|
|
smp_mb();
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2012-04-19 03:40:01 +00:00
|
|
|
tcp_finish_connect(sk, skb);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2005-08-10 03:11:56 +00:00
|
|
|
if (sk->sk_write_pending ||
|
|
|
|
icsk->icsk_accept_queue.rskq_defer_accept ||
|
|
|
|
icsk->icsk_ack.pingpong) {
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Save one ACK. Data will be ready after
|
|
|
|
* several ticks, if write_pending is set.
|
|
|
|
*
|
|
|
|
* It may be deleted, but with this feature tcpdumps
|
|
|
|
* look so _wonderfully_ clever, that I was not able
|
|
|
|
* to stand against the temptation 8) --ANK
|
|
|
|
*/
|
2005-08-10 03:10:42 +00:00
|
|
|
inet_csk_schedule_ack(sk);
|
2005-08-10 03:11:56 +00:00
|
|
|
icsk->icsk_ack.lrcvtime = tcp_time_stamp;
|
2005-08-10 03:10:42 +00:00
|
|
|
tcp_enter_quickack_mode(sk);
|
2005-08-10 03:11:08 +00:00
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,
|
|
|
|
TCP_DELACK_MAX, TCP_RTO_MAX);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
discard:
|
|
|
|
__kfree_skb(skb);
|
|
|
|
return 0;
|
|
|
|
} else {
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
}
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* No ACK in the segment */
|
|
|
|
|
|
|
|
if (th->rst) {
|
|
|
|
/* rfc793:
|
|
|
|
* "If the RST bit is set
|
|
|
|
*
|
|
|
|
* Otherwise (no ACK) drop the segment and return."
|
|
|
|
*/
|
|
|
|
|
|
|
|
goto discard_and_undo;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* PAWS check. */
|
2007-12-31 22:57:14 +00:00
|
|
|
if (tp->rx_opt.ts_recent_stamp && tp->rx_opt.saw_tstamp &&
|
2009-03-14 14:23:03 +00:00
|
|
|
tcp_paws_reject(&tp->rx_opt, 0))
|
2005-04-16 22:20:36 +00:00
|
|
|
goto discard_and_undo;
|
|
|
|
|
|
|
|
if (th->syn) {
|
|
|
|
/* We see SYN without ACK. It is attempt of
|
|
|
|
* simultaneous connect with crossed SYNs.
|
|
|
|
* Particularly, it can be connect to self.
|
|
|
|
*/
|
|
|
|
tcp_set_state(sk, TCP_SYN_RECV);
|
|
|
|
|
|
|
|
if (tp->rx_opt.saw_tstamp) {
|
|
|
|
tp->rx_opt.tstamp_ok = 1;
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
tp->tcp_header_len =
|
|
|
|
sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
} else {
|
|
|
|
tp->tcp_header_len = sizeof(struct tcphdr);
|
|
|
|
}
|
|
|
|
|
|
|
|
tp->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;
|
|
|
|
tp->rcv_wup = TCP_SKB_CB(skb)->seq + 1;
|
|
|
|
|
|
|
|
/* RFC1323: The window in SYN & SYN/ACK segments is
|
|
|
|
* never scaled.
|
|
|
|
*/
|
|
|
|
tp->snd_wnd = ntohs(th->window);
|
|
|
|
tp->snd_wl1 = TCP_SKB_CB(skb)->seq;
|
|
|
|
tp->max_window = tp->snd_wnd;
|
|
|
|
|
|
|
|
TCP_ECN_rcv_syn(tp, th);
|
|
|
|
|
2006-03-21 01:53:41 +00:00
|
|
|
tcp_mtup_init(sk);
|
2005-12-14 07:26:10 +00:00
|
|
|
tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_initialize_rcv_mss(sk);
|
|
|
|
|
|
|
|
tcp_send_synack(sk);
|
|
|
|
#if 0
|
|
|
|
/* Note, we could accept data and URG from this segment.
|
|
|
|
* There are no obstacles to make this.
|
|
|
|
*
|
|
|
|
* However, if we ignore data in ACKless segments sometimes,
|
|
|
|
* we have no reasons to accept it sometimes.
|
|
|
|
* Also, seems the code doing it in step6 of tcp_rcv_state_process
|
|
|
|
* is not flawless. So, discard packet for sanity.
|
|
|
|
* Uncomment this return to process the data.
|
|
|
|
*/
|
|
|
|
return -1;
|
|
|
|
#else
|
|
|
|
goto discard;
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
/* "fifth, if neither of the SYN or RST bits is set then
|
|
|
|
* drop the segment and return."
|
|
|
|
*/
|
|
|
|
|
|
|
|
discard_and_undo:
|
|
|
|
tcp_clear_options(&tp->rx_opt);
|
|
|
|
tp->rx_opt.mss_clamp = saved_clamp;
|
|
|
|
goto discard;
|
|
|
|
|
|
|
|
reset_and_undo:
|
|
|
|
tcp_clear_options(&tp->rx_opt);
|
|
|
|
tp->rx_opt.mss_clamp = saved_clamp;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This function implements the receiving procedure of RFC 793 for
|
2007-02-09 14:24:47 +00:00
|
|
|
* all states except ESTABLISHED and TIME_WAIT.
|
2005-04-16 22:20:36 +00:00
|
|
|
* It's called from both tcp_v4_rcv and tcp_v6_rcv and should be
|
|
|
|
* address independent.
|
|
|
|
*/
|
2007-02-09 14:24:47 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb,
|
2011-10-21 09:22:42 +00:00
|
|
|
const struct tcphdr *th, unsigned int len)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-12-14 07:15:52 +00:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
int queued = 0;
|
2008-08-23 12:10:12 +00:00
|
|
|
int res;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
tp->rx_opt.saw_tstamp = 0;
|
|
|
|
|
|
|
|
switch (sk->sk_state) {
|
|
|
|
case TCP_CLOSE:
|
|
|
|
goto discard;
|
|
|
|
|
|
|
|
case TCP_LISTEN:
|
2007-03-09 04:45:19 +00:00
|
|
|
if (th->ack)
|
2005-04-16 22:20:36 +00:00
|
|
|
return 1;
|
|
|
|
|
2007-03-09 04:45:19 +00:00
|
|
|
if (th->rst)
|
2005-04-16 22:20:36 +00:00
|
|
|
goto discard;
|
|
|
|
|
2007-03-09 04:45:19 +00:00
|
|
|
if (th->syn) {
|
2011-12-02 23:41:42 +00:00
|
|
|
if (th->fin)
|
|
|
|
goto discard;
|
2005-12-14 07:15:52 +00:00
|
|
|
if (icsk->icsk_af_ops->conn_request(sk, skb) < 0)
|
2005-04-16 22:20:36 +00:00
|
|
|
return 1;
|
|
|
|
|
2007-02-09 14:24:47 +00:00
|
|
|
/* Now we have several options: In theory there is
|
|
|
|
* nothing else in the frame. KA9Q has an option to
|
2005-04-16 22:20:36 +00:00
|
|
|
* send data with the syn, BSD accepts data with the
|
2007-02-09 14:24:47 +00:00
|
|
|
* syn up to the [to be] advertised window and
|
|
|
|
* Solaris 2.1 gives you a protocol error. For now
|
|
|
|
* we just ignore it, that fits the spec precisely
|
2005-04-16 22:20:36 +00:00
|
|
|
* and avoids incompatibilities. It would be nice in
|
|
|
|
* future to drop through and process the data.
|
|
|
|
*
|
2007-02-09 14:24:47 +00:00
|
|
|
* Now that TTCP is starting to be used we ought to
|
2005-04-16 22:20:36 +00:00
|
|
|
* queue this data.
|
|
|
|
* But, this leaves one open to an easy denial of
|
2007-02-09 14:24:47 +00:00
|
|
|
* service attack, and SYN cookies can't defend
|
2005-04-16 22:20:36 +00:00
|
|
|
* against this problem. So, we drop the data
|
2007-01-24 04:15:06 +00:00
|
|
|
* in the interest of security over speed unless
|
|
|
|
* it's still in use.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2007-01-24 04:15:06 +00:00
|
|
|
kfree_skb(skb);
|
|
|
|
return 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
goto discard;
|
|
|
|
|
|
|
|
case TCP_SYN_SENT:
|
|
|
|
queued = tcp_rcv_synsent_state_process(sk, skb, th, len);
|
|
|
|
if (queued >= 0)
|
|
|
|
return queued;
|
|
|
|
|
|
|
|
/* Do step6 onward by hand. */
|
|
|
|
tcp_urg(sk, skb, th);
|
|
|
|
__kfree_skb(skb);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_data_snd_check(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-08-23 12:10:12 +00:00
|
|
|
res = tcp_validate_incoming(sk, skb, th, 0);
|
|
|
|
if (res <= 0)
|
|
|
|
return -res;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/* step 5: check the ACK field */
|
|
|
|
if (th->ack) {
|
2009-03-23 04:49:57 +00:00
|
|
|
int acceptable = tcp_ack(sk, skb, FLAG_SLOWPATH) > 0;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-03-09 04:45:19 +00:00
|
|
|
switch (sk->sk_state) {
|
2005-04-16 22:20:36 +00:00
|
|
|
case TCP_SYN_RECV:
|
|
|
|
if (acceptable) {
|
|
|
|
tp->copied_seq = tp->rcv_nxt;
|
2006-12-07 08:11:33 +00:00
|
|
|
smp_mb();
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_set_state(sk, TCP_ESTABLISHED);
|
|
|
|
sk->sk_state_change(sk);
|
|
|
|
|
|
|
|
/* Note, that this wakeup is only for marginal
|
|
|
|
* crossed SYN case. Passively open sockets
|
|
|
|
* are not waked up, because sk->sk_sleep ==
|
|
|
|
* NULL and sk->sk_socket == NULL.
|
|
|
|
*/
|
2007-11-26 12:10:50 +00:00
|
|
|
if (sk->sk_socket)
|
|
|
|
sk_wake_async(sk,
|
2007-12-31 22:57:14 +00:00
|
|
|
SOCK_WAKE_IO, POLL_OUT);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
tp->snd_una = TCP_SKB_CB(skb)->ack_seq;
|
|
|
|
tp->snd_wnd = ntohs(th->window) <<
|
|
|
|
tp->rx_opt.snd_wscale;
|
2009-03-03 06:42:02 +00:00
|
|
|
tcp_init_wl(tp, TCP_SKB_CB(skb)->seq);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (tp->rx_opt.tstamp_ok)
|
|
|
|
tp->advmss -= TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
|
|
|
|
/* Make sure socket is routed, for
|
|
|
|
* correct metrics.
|
|
|
|
*/
|
2005-12-14 07:15:52 +00:00
|
|
|
icsk->icsk_af_ops->rebuild_header(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
tcp_init_metrics(sk);
|
|
|
|
|
2005-08-10 07:03:31 +00:00
|
|
|
tcp_init_congestion_control(sk);
|
2005-06-23 19:19:55 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Prevent spurious tcp_cwnd_restart() on
|
|
|
|
* first data packet.
|
|
|
|
*/
|
|
|
|
tp->lsndtime = tcp_time_stamp;
|
|
|
|
|
2006-03-21 01:53:41 +00:00
|
|
|
tcp_mtup_init(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_initialize_rcv_mss(sk);
|
|
|
|
tcp_init_buffer_space(sk);
|
|
|
|
tcp_fast_path_on(tp);
|
|
|
|
} else {
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
|
|
|
|
case TCP_FIN_WAIT1:
|
|
|
|
if (tp->snd_una == tp->write_seq) {
|
|
|
|
tcp_set_state(sk, TCP_FIN_WAIT2);
|
|
|
|
sk->sk_shutdown |= SEND_SHUTDOWN;
|
2010-04-08 23:03:29 +00:00
|
|
|
dst_confirm(__sk_dst_get(sk));
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD))
|
|
|
|
/* Wake up lingering close() */
|
|
|
|
sk->sk_state_change(sk);
|
|
|
|
else {
|
|
|
|
int tmo;
|
|
|
|
|
|
|
|
if (tp->linger2 < 0 ||
|
|
|
|
(TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
|
|
|
|
after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt))) {
|
|
|
|
tcp_done(sk);
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
|
2005-04-16 22:20:36 +00:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2005-08-10 03:10:42 +00:00
|
|
|
tmo = tcp_fin_time(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (tmo > TCP_TIMEWAIT_LEN) {
|
2005-08-10 03:10:42 +00:00
|
|
|
inet_csk_reset_keepalive_timer(sk, tmo - TCP_TIMEWAIT_LEN);
|
2005-04-16 22:20:36 +00:00
|
|
|
} else if (th->fin || sock_owned_by_user(sk)) {
|
|
|
|
/* Bad case. We could lose such FIN otherwise.
|
|
|
|
* It is not a big problem, but it looks confusing
|
|
|
|
* and not so rare event. We still can lose it now,
|
|
|
|
* if it spins in bh_lock_sock(), but it is really
|
|
|
|
* marginal case.
|
|
|
|
*/
|
2005-08-10 03:10:42 +00:00
|
|
|
inet_csk_reset_keepalive_timer(sk, tmo);
|
2005-04-16 22:20:36 +00:00
|
|
|
} else {
|
|
|
|
tcp_time_wait(sk, TCP_FIN_WAIT2, tmo);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
|
|
|
|
case TCP_CLOSING:
|
|
|
|
if (tp->snd_una == tp->write_seq) {
|
|
|
|
tcp_time_wait(sk, TCP_TIME_WAIT, 0);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
|
|
|
|
case TCP_LAST_ACK:
|
|
|
|
if (tp->snd_una == tp->write_seq) {
|
|
|
|
tcp_update_metrics(sk);
|
|
|
|
tcp_done(sk);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
} else
|
|
|
|
goto discard;
|
|
|
|
|
|
|
|
/* step 6: check the URG bit */
|
|
|
|
tcp_urg(sk, skb, th);
|
|
|
|
|
|
|
|
/* step 7: process the segment text */
|
|
|
|
switch (sk->sk_state) {
|
|
|
|
case TCP_CLOSE_WAIT:
|
|
|
|
case TCP_CLOSING:
|
|
|
|
case TCP_LAST_ACK:
|
|
|
|
if (!before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt))
|
|
|
|
break;
|
|
|
|
case TCP_FIN_WAIT1:
|
|
|
|
case TCP_FIN_WAIT2:
|
|
|
|
/* RFC 793 says to queue data in these states,
|
2007-02-09 14:24:47 +00:00
|
|
|
* RFC 1122 says we MUST send a reset.
|
2005-04-16 22:20:36 +00:00
|
|
|
* BSD 4.4 also does reset.
|
|
|
|
*/
|
|
|
|
if (sk->sk_shutdown & RCV_SHUTDOWN) {
|
|
|
|
if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
|
|
|
|
after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt)) {
|
2008-07-17 03:31:16 +00:00
|
|
|
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_reset(sk);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/* Fall through */
|
2007-02-09 14:24:47 +00:00
|
|
|
case TCP_ESTABLISHED:
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_data_queue(sk, skb);
|
|
|
|
queued = 1;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* tcp_data could move socket to TIME-WAIT */
|
|
|
|
if (sk->sk_state != TCP_CLOSE) {
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 05:18:02 +00:00
|
|
|
tcp_data_snd_check(sk);
|
2005-04-16 22:20:36 +00:00
|
|
|
tcp_ack_snd_check(sk);
|
|
|
|
}
|
|
|
|
|
2007-02-09 14:24:47 +00:00
|
|
|
if (!queued) {
|
2005-04-16 22:20:36 +00:00
|
|
|
discard:
|
|
|
|
__kfree_skb(skb);
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(tcp_rcv_state_process);
|