blob: 7c41402718876ef8adc7dc90cd51994e6a4fded9 [file] [log] [blame]
Linus Torvalds1da177e2005-04-16 15:20:36 -07001/*
2 * INET An implementation of the TCP/IP protocol suite for the LINUX
3 * operating system. INET is implemented using the BSD Socket
4 * interface as the means of communication with the user level.
5 *
6 * Implementation of the Transmission Control Protocol(TCP).
7 *
Jesper Juhl02c30a82005-05-05 16:16:16 -07008 * Authors: Ross Biro
Linus Torvalds1da177e2005-04-16 15:20:36 -07009 * Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG>
10 * Mark Evans, <evansmp@uhura.aston.ac.uk>
11 * Corey Minyard <wf-rch!minyard@relay.EU.net>
12 * Florian La Roche, <flla@stud.uni-sb.de>
13 * Charles Hedrick, <hedrick@klinzhai.rutgers.edu>
14 * Linus Torvalds, <torvalds@cs.helsinki.fi>
15 * Alan Cox, <gw4pts@gw4pts.ampr.org>
16 * Matthew Dillon, <dillon@apollo.west.oic.com>
17 * Arnt Gulbrandsen, <agulbra@nvg.unit.no>
18 * Jorge Cwik, <jorge@laser.satlink.net>
19 *
20 * Fixes:
21 * Alan Cox : Numerous verify_area() calls
22 * Alan Cox : Set the ACK bit on a reset
23 * Alan Cox : Stopped it crashing if it closed while
24 * sk->inuse=1 and was trying to connect
25 * (tcp_err()).
26 * Alan Cox : All icmp error handling was broken
27 * pointers passed where wrong and the
28 * socket was looked up backwards. Nobody
29 * tested any icmp error code obviously.
30 * Alan Cox : tcp_err() now handled properly. It
31 * wakes people on errors. poll
32 * behaves and the icmp error race
33 * has gone by moving it into sock.c
34 * Alan Cox : tcp_send_reset() fixed to work for
35 * everything not just packets for
36 * unknown sockets.
37 * Alan Cox : tcp option processing.
38 * Alan Cox : Reset tweaked (still not 100%) [Had
39 * syn rule wrong]
40 * Herp Rosmanith : More reset fixes
41 * Alan Cox : No longer acks invalid rst frames.
42 * Acking any kind of RST is right out.
43 * Alan Cox : Sets an ignore me flag on an rst
44 * receive otherwise odd bits of prattle
45 * escape still
46 * Alan Cox : Fixed another acking RST frame bug.
47 * Should stop LAN workplace lockups.
48 * Alan Cox : Some tidyups using the new skb list
49 * facilities
50 * Alan Cox : sk->keepopen now seems to work
51 * Alan Cox : Pulls options out correctly on accepts
52 * Alan Cox : Fixed assorted sk->rqueue->next errors
53 * Alan Cox : PSH doesn't end a TCP read. Switched a
54 * bit to skb ops.
55 * Alan Cox : Tidied tcp_data to avoid a potential
56 * nasty.
57 * Alan Cox : Added some better commenting, as the
58 * tcp is hard to follow
59 * Alan Cox : Removed incorrect check for 20 * psh
60 * Michael O'Reilly : ack < copied bug fix.
61 * Johannes Stille : Misc tcp fixes (not all in yet).
62 * Alan Cox : FIN with no memory -> CRASH
63 * Alan Cox : Added socket option proto entries.
64 * Also added awareness of them to accept.
65 * Alan Cox : Added TCP options (SOL_TCP)
66 * Alan Cox : Switched wakeup calls to callbacks,
67 * so the kernel can layer network
68 * sockets.
69 * Alan Cox : Use ip_tos/ip_ttl settings.
70 * Alan Cox : Handle FIN (more) properly (we hope).
71 * Alan Cox : RST frames sent on unsynchronised
72 * state ack error.
73 * Alan Cox : Put in missing check for SYN bit.
74 * Alan Cox : Added tcp_select_window() aka NET2E
75 * window non shrink trick.
76 * Alan Cox : Added a couple of small NET2E timer
77 * fixes
78 * Charles Hedrick : TCP fixes
79 * Toomas Tamm : TCP window fixes
80 * Alan Cox : Small URG fix to rlogin ^C ack fight
81 * Charles Hedrick : Rewrote most of it to actually work
82 * Linus : Rewrote tcp_read() and URG handling
83 * completely
84 * Gerhard Koerting: Fixed some missing timer handling
85 * Matthew Dillon : Reworked TCP machine states as per RFC
86 * Gerhard Koerting: PC/TCP workarounds
87 * Adam Caldwell : Assorted timer/timing errors
88 * Matthew Dillon : Fixed another RST bug
89 * Alan Cox : Move to kernel side addressing changes.
90 * Alan Cox : Beginning work on TCP fastpathing
91 * (not yet usable)
92 * Arnt Gulbrandsen: Turbocharged tcp_check() routine.
93 * Alan Cox : TCP fast path debugging
94 * Alan Cox : Window clamping
95 * Michael Riepe : Bug in tcp_check()
96 * Matt Dillon : More TCP improvements and RST bug fixes
97 * Matt Dillon : Yet more small nasties remove from the
98 * TCP code (Be very nice to this man if
99 * tcp finally works 100%) 8)
100 * Alan Cox : BSD accept semantics.
101 * Alan Cox : Reset on closedown bug.
102 * Peter De Schrijver : ENOTCONN check missing in tcp_sendto().
103 * Michael Pall : Handle poll() after URG properly in
104 * all cases.
105 * Michael Pall : Undo the last fix in tcp_read_urg()
106 * (multi URG PUSH broke rlogin).
107 * Michael Pall : Fix the multi URG PUSH problem in
108 * tcp_readable(), poll() after URG
109 * works now.
110 * Michael Pall : recv(...,MSG_OOB) never blocks in the
111 * BSD api.
112 * Alan Cox : Changed the semantics of sk->socket to
113 * fix a race and a signal problem with
114 * accept() and async I/O.
115 * Alan Cox : Relaxed the rules on tcp_sendto().
116 * Yury Shevchuk : Really fixed accept() blocking problem.
117 * Craig I. Hagan : Allow for BSD compatible TIME_WAIT for
118 * clients/servers which listen in on
119 * fixed ports.
120 * Alan Cox : Cleaned the above up and shrank it to
121 * a sensible code size.
122 * Alan Cox : Self connect lockup fix.
123 * Alan Cox : No connect to multicast.
124 * Ross Biro : Close unaccepted children on master
125 * socket close.
126 * Alan Cox : Reset tracing code.
127 * Alan Cox : Spurious resets on shutdown.
128 * Alan Cox : Giant 15 minute/60 second timer error
129 * Alan Cox : Small whoops in polling before an
130 * accept.
131 * Alan Cox : Kept the state trace facility since
132 * it's handy for debugging.
133 * Alan Cox : More reset handler fixes.
134 * Alan Cox : Started rewriting the code based on
135 * the RFC's for other useful protocol
136 * references see: Comer, KA9Q NOS, and
137 * for a reference on the difference
138 * between specifications and how BSD
139 * works see the 4.4lite source.
140 * A.N.Kuznetsov : Don't time wait on completion of tidy
141 * close.
142 * Linus Torvalds : Fin/Shutdown & copied_seq changes.
143 * Linus Torvalds : Fixed BSD port reuse to work first syn
144 * Alan Cox : Reimplemented timers as per the RFC
145 * and using multiple timers for sanity.
146 * Alan Cox : Small bug fixes, and a lot of new
147 * comments.
148 * Alan Cox : Fixed dual reader crash by locking
149 * the buffers (much like datagram.c)
150 * Alan Cox : Fixed stuck sockets in probe. A probe
151 * now gets fed up of retrying without
152 * (even a no space) answer.
153 * Alan Cox : Extracted closing code better
154 * Alan Cox : Fixed the closing state machine to
155 * resemble the RFC.
156 * Alan Cox : More 'per spec' fixes.
157 * Jorge Cwik : Even faster checksumming.
158 * Alan Cox : tcp_data() doesn't ack illegal PSH
159 * only frames. At least one pc tcp stack
160 * generates them.
161 * Alan Cox : Cache last socket.
162 * Alan Cox : Per route irtt.
163 * Matt Day : poll()->select() match BSD precisely on error
164 * Alan Cox : New buffers
165 * Marc Tamsky : Various sk->prot->retransmits and
166 * sk->retransmits misupdating fixed.
167 * Fixed tcp_write_timeout: stuck close,
168 * and TCP syn retries gets used now.
169 * Mark Yarvis : In tcp_read_wakeup(), don't send an
170 * ack if state is TCP_CLOSED.
171 * Alan Cox : Look up device on a retransmit - routes may
172 * change. Doesn't yet cope with MSS shrink right
173 * but it's a start!
174 * Marc Tamsky : Closing in closing fixes.
175 * Mike Shaver : RFC1122 verifications.
176 * Alan Cox : rcv_saddr errors.
177 * Alan Cox : Block double connect().
178 * Alan Cox : Small hooks for enSKIP.
179 * Alexey Kuznetsov: Path MTU discovery.
180 * Alan Cox : Support soft errors.
181 * Alan Cox : Fix MTU discovery pathological case
182 * when the remote claims no mtu!
183 * Marc Tamsky : TCP_CLOSE fix.
184 * Colin (G3TNE) : Send a reset on syn ack replies in
185 * window but wrong (fixes NT lpd problems)
186 * Pedro Roque : Better TCP window handling, delayed ack.
187 * Joerg Reuter : No modification of locked buffers in
188 * tcp_do_retransmit()
189 * Eric Schenk : Changed receiver side silly window
190 * avoidance algorithm to BSD style
191 * algorithm. This doubles throughput
192 * against machines running Solaris,
193 * and seems to result in general
194 * improvement.
195 * Stefan Magdalinski : adjusted tcp_readable() to fix FIONREAD
196 * Willy Konynenberg : Transparent proxying support.
197 * Mike McLagan : Routing by source
198 * Keith Owens : Do proper merging with partial SKB's in
199 * tcp_do_sendmsg to avoid burstiness.
200 * Eric Schenk : Fix fast close down bug with
201 * shutdown() followed by close().
202 * Andi Kleen : Make poll agree with SIGIO
203 * Salvatore Sanfilippo : Support SO_LINGER with linger == 1 and
204 * lingertime == 0 (RFC 793 ABORT Call)
205 * Hirokazu Takahashi : Use copy_from_user() instead of
206 * csum_and_copy_from_user() if possible.
207 *
208 * This program is free software; you can redistribute it and/or
209 * modify it under the terms of the GNU General Public License
210 * as published by the Free Software Foundation; either version
211 * 2 of the License, or(at your option) any later version.
212 *
213 * Description of States:
214 *
215 * TCP_SYN_SENT sent a connection request, waiting for ack
216 *
217 * TCP_SYN_RECV received a connection request, sent ack,
218 * waiting for final ack in three-way handshake.
219 *
220 * TCP_ESTABLISHED connection established
221 *
222 * TCP_FIN_WAIT1 our side has shutdown, waiting to complete
223 * transmission of remaining buffered data
224 *
225 * TCP_FIN_WAIT2 all buffered data sent, waiting for remote
226 * to shutdown
227 *
228 * TCP_CLOSING both sides have shutdown but we still have
229 * data we have to finish sending
230 *
231 * TCP_TIME_WAIT timeout to catch resent junk before entering
232 * closed, can only be entered from FIN_WAIT2
233 * or CLOSING. Required because the other end
234 * may not have gotten our last ACK causing it
235 * to retransmit the data packet (which we ignore)
236 *
237 * TCP_CLOSE_WAIT remote side has shutdown and is waiting for
238 * us to finish writing our data and to shutdown
239 * (we have to close() to move on to LAST_ACK)
240 *
241 * TCP_LAST_ACK out side has shutdown after remote has
242 * shutdown. There may still be data in our
243 * buffer that we have to finish sending
244 *
245 * TCP_CLOSE socket is finished
246 */
247
Joe Perchesafd465032012-03-12 07:03:32 +0000248#define pr_fmt(fmt) "TCP: " fmt
249
Herbert Xucf80e0e2016-01-24 21:20:23 +0800250#include <crypto/hash.h>
Ilpo Järvinen172589c2007-08-28 15:50:33 -0700251#include <linux/kernel.h>
Linus Torvalds1da177e2005-04-16 15:20:36 -0700252#include <linux/module.h>
253#include <linux/types.h>
254#include <linux/fcntl.h>
255#include <linux/poll.h>
Eric Dumazet6e9250f2015-04-28 16:23:49 -0700256#include <linux/inet_diag.h>
Linus Torvalds1da177e2005-04-16 15:20:36 -0700257#include <linux/init.h>
Linus Torvalds1da177e2005-04-16 15:20:36 -0700258#include <linux/fs.h>
Jens Axboe9c55e012007-11-06 23:30:13 -0800259#include <linux/skbuff.h>
Andrew Morton81b23b42008-07-03 03:22:02 -0700260#include <linux/scatterlist.h>
Jens Axboe9c55e012007-11-06 23:30:13 -0800261#include <linux/splice.h>
262#include <linux/net.h>
263#include <linux/socket.h>
Linus Torvalds1da177e2005-04-16 15:20:36 -0700264#include <linux/random.h>
265#include <linux/bootmem.h>
Miquel van Smoorenburg57413eb2008-06-27 17:23:57 -0700266#include <linux/highmem.h>
267#include <linux/swap.h>
David S. Millerb8059ea2006-03-25 01:36:56 -0800268#include <linux/cache.h>
Herbert Xuf4c50d92006-06-22 03:02:40 -0700269#include <linux/err.h>
William Allen Simpsonda5c78c2009-12-02 18:12:09 +0000270#include <linux/time.h>
Tejun Heo5a0e3ad2010-03-24 17:04:11 +0900271#include <linux/slab.h>
Mike Maloney98aaa912017-08-22 17:08:48 -0400272#include <linux/errqueue.h>
Ursula Braun60e2a772017-10-25 11:01:45 +0200273#include <linux/static_key.h>
Linus Torvalds1da177e2005-04-16 15:20:36 -0700274
275#include <net/icmp.h>
Yuchung Chengcf60af02012-07-19 06:43:09 +0000276#include <net/inet_common.h>
Linus Torvalds1da177e2005-04-16 15:20:36 -0700277#include <net/tcp.h>
278#include <net/xfrm.h>
279#include <net/ip.h>
Jens Axboe9c55e012007-11-06 23:30:13 -0800280#include <net/sock.h>
Linus Torvalds1da177e2005-04-16 15:20:36 -0700281
Linus Torvalds7c0f6ba2016-12-24 11:46:01 -0800282#include <linux/uaccess.h>
Linus Torvalds1da177e2005-04-16 15:20:36 -0700283#include <asm/ioctls.h>
Eliezer Tamir076bb0c2013-07-10 17:13:17 +0300284#include <net/busy_poll.h>
Linus Torvalds1da177e2005-04-16 15:20:36 -0700285
Eric Dumazetdd24c002008-11-25 21:17:14 -0800286struct percpu_counter tcp_orphan_count;
Arnaldo Carvalho de Melo0a5578c2005-08-09 20:11:41 -0700287EXPORT_SYMBOL_GPL(tcp_orphan_count);
288
Eric W. Biedermana4fe34b2013-10-19 16:25:36 -0700289long sysctl_tcp_mem[3] __read_mostly;
Eric W. Biedermana4fe34b2013-10-19 16:25:36 -0700290EXPORT_SYMBOL(sysctl_tcp_mem);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700291
Eric Dumazet8d987e52010-11-09 23:24:26 +0000292atomic_long_t tcp_memory_allocated; /* Current allocated memory. */
Linus Torvalds1da177e2005-04-16 15:20:36 -0700293EXPORT_SYMBOL(tcp_memory_allocated);
Eric Dumazet17483762008-11-25 21:16:35 -0800294
Ursula Braun60e2a772017-10-25 11:01:45 +0200295#if IS_ENABLED(CONFIG_SMC)
296DEFINE_STATIC_KEY_FALSE(tcp_have_smc);
297EXPORT_SYMBOL(tcp_have_smc);
298#endif
299
Eric Dumazet17483762008-11-25 21:16:35 -0800300/*
301 * Current number of TCP sockets.
302 */
303struct percpu_counter tcp_sockets_allocated;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700304EXPORT_SYMBOL(tcp_sockets_allocated);
305
306/*
Jens Axboe9c55e012007-11-06 23:30:13 -0800307 * TCP splice context
308 */
309struct tcp_splice_state {
310 struct pipe_inode_info *pipe;
311 size_t len;
312 unsigned int flags;
313};
314
315/*
Linus Torvalds1da177e2005-04-16 15:20:36 -0700316 * Pressure flag: try to collapse.
317 * Technical note: it is used by multiple contexts non atomically.
Hideo Aoki3ab224b2007-12-31 00:11:19 -0800318 * All the __sk_mem_schedule() is of this nature: accounting
Linus Torvalds1da177e2005-04-16 15:20:36 -0700319 * is strict, actions are advisory and have some latency.
320 */
Eric Dumazet06044752017-06-07 13:29:12 -0700321unsigned long tcp_memory_pressure __read_mostly;
322EXPORT_SYMBOL_GPL(tcp_memory_pressure);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700323
Pavel Emelyanov5c52ba12008-07-16 20:28:10 -0700324void tcp_enter_memory_pressure(struct sock *sk)
Linus Torvalds1da177e2005-04-16 15:20:36 -0700325{
Eric Dumazet06044752017-06-07 13:29:12 -0700326 unsigned long val;
327
328 if (tcp_memory_pressure)
329 return;
330 val = jiffies;
331
332 if (!val)
333 val--;
334 if (!cmpxchg(&tcp_memory_pressure, 0, val))
Pavel Emelyanov4e673442008-07-16 20:30:14 -0700335 NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMEMORYPRESSURES);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700336}
Eric Dumazet06044752017-06-07 13:29:12 -0700337EXPORT_SYMBOL_GPL(tcp_enter_memory_pressure);
338
339void tcp_leave_memory_pressure(struct sock *sk)
340{
341 unsigned long val;
342
343 if (!tcp_memory_pressure)
344 return;
345 val = xchg(&tcp_memory_pressure, 0);
346 if (val)
347 NET_ADD_STATS(sock_net(sk), LINUX_MIB_TCPMEMORYPRESSURESCHRONO,
348 jiffies_to_msecs(jiffies - val));
349}
350EXPORT_SYMBOL_GPL(tcp_leave_memory_pressure);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700351
Julian Anastasovb103cf32009-10-19 10:10:40 +0000352/* Convert seconds to retransmits based on initial and max timeout */
353static u8 secs_to_retrans(int seconds, int timeout, int rto_max)
354{
355 u8 res = 0;
356
357 if (seconds > 0) {
358 int period = timeout;
359
360 res = 1;
361 while (seconds > period && res < 255) {
362 res++;
363 timeout <<= 1;
364 if (timeout > rto_max)
365 timeout = rto_max;
366 period += timeout;
367 }
368 }
369 return res;
370}
371
372/* Convert retransmits to seconds based on initial and max timeout */
373static int retrans_to_secs(u8 retrans, int timeout, int rto_max)
374{
375 int period = 0;
376
377 if (retrans > 0) {
378 period = timeout;
379 while (--retrans) {
380 timeout <<= 1;
381 if (timeout > rto_max)
382 timeout = rto_max;
383 period += timeout;
384 }
385 }
386 return period;
387}
388
Wei Wang02635982017-07-28 10:28:20 -0700389static u64 tcp_compute_delivery_rate(const struct tcp_sock *tp)
390{
391 u32 rate = READ_ONCE(tp->rate_delivered);
392 u32 intv = READ_ONCE(tp->rate_interval_us);
393 u64 rate64 = 0;
394
395 if (rate && intv) {
396 rate64 = (u64)rate * tp->mss_cache * USEC_PER_SEC;
397 do_div(rate64, intv);
398 }
399 return rate64;
400}
401
Neal Cardwell900f65d2012-04-19 09:55:21 +0000402/* Address-family independent initialization for a tcp_sock.
403 *
404 * NOTE: A lot of things set to zero explicitly by call to
405 * sk_alloc() so need not be done here.
406 */
407void tcp_init_sock(struct sock *sk)
408{
409 struct inet_connection_sock *icsk = inet_csk(sk);
410 struct tcp_sock *tp = tcp_sk(sk);
411
Yaogong Wang9f5afea2016-09-07 14:49:28 -0700412 tp->out_of_order_queue = RB_ROOT;
Eric Dumazet75c119a2017-10-05 22:21:27 -0700413 sk->tcp_rtx_queue = RB_ROOT;
Neal Cardwell900f65d2012-04-19 09:55:21 +0000414 tcp_init_xmit_timers(sk);
Eric Dumazet46d3cea2012-07-11 05:50:31 +0000415 INIT_LIST_HEAD(&tp->tsq_node);
Eric Dumazete2080072017-10-04 12:59:58 -0700416 INIT_LIST_HEAD(&tp->tsorted_sent_queue);
Neal Cardwell900f65d2012-04-19 09:55:21 +0000417
418 icsk->icsk_rto = TCP_TIMEOUT_INIT;
Eric Dumazet740b0f12014-02-26 14:02:48 -0800419 tp->mdev_us = jiffies_to_usecs(TCP_TIMEOUT_INIT);
Eric Dumazetac9517f2017-05-16 14:00:13 -0700420 minmax_reset(&tp->rtt_min, tcp_jiffies32, ~0U);
Neal Cardwell900f65d2012-04-19 09:55:21 +0000421
422 /* So many TCP implementations out there (incorrectly) count the
423 * initial SYN frame in their delayed-ACK and congestion control
424 * algorithms that we must have the following bandaid to talk
425 * efficiently to them. -DaveM
426 */
427 tp->snd_cwnd = TCP_INIT_CWND;
428
Soheil Hassas Yeganehd7722e82016-09-19 23:39:15 -0400429 /* There's a bubble in the pipe until at least the first ACK. */
430 tp->app_limited = ~0U;
431
Neal Cardwell900f65d2012-04-19 09:55:21 +0000432 /* See draft-stevens-tcpca-spec-01 for discussion of the
433 * initialization of these values.
434 */
435 tp->snd_ssthresh = TCP_INFINITE_SSTHRESH;
436 tp->snd_cwnd_clamp = ~0;
437 tp->mss_cache = TCP_MSS_DEFAULT;
438
Nikolay Borisov1043e252016-02-03 09:46:52 +0200439 tp->reordering = sock_net(sk)->ipv4.sysctl_tcp_reordering;
Florian Westphal55d86942014-09-26 22:37:32 +0200440 tcp_assign_congestion_control(sk);
Neal Cardwell900f65d2012-04-19 09:55:21 +0000441
Andrey Vaginceaa1fe2013-02-11 05:50:17 +0000442 tp->tsoffset = 0;
Priyaranjan Jha1f255692017-11-03 16:38:48 -0700443 tp->rack.reo_wnd_steps = 1;
Andrey Vaginceaa1fe2013-02-11 05:50:17 +0000444
Neal Cardwell900f65d2012-04-19 09:55:21 +0000445 sk->sk_state = TCP_CLOSE;
446
447 sk->sk_write_space = sk_stream_write_space;
448 sock_set_flag(sk, SOCK_USE_WRITE_QUEUE);
449
450 icsk->icsk_sync_mss = tcp_sync_mss;
451
Eric Dumazet356d1832017-11-07 00:29:28 -0800452 sk->sk_sndbuf = sock_net(sk)->ipv4.sysctl_tcp_wmem[1];
453 sk->sk_rcvbuf = sock_net(sk)->ipv4.sysctl_tcp_rmem[1];
Neal Cardwell900f65d2012-04-19 09:55:21 +0000454
Neal Cardwell900f65d2012-04-19 09:55:21 +0000455 sk_sockets_allocated_inc(sk);
Eric Dumazet0a6b2a12018-02-19 11:56:47 -0800456 sk->sk_route_forced_caps = NETIF_F_GSO;
Neal Cardwell900f65d2012-04-19 09:55:21 +0000457}
458EXPORT_SYMBOL(tcp_init_sock);
459
Wei Wang27204aa2017-10-04 10:03:44 -0700460void tcp_init_transfer(struct sock *sk, int bpf_op)
461{
462 struct inet_connection_sock *icsk = inet_csk(sk);
463
464 tcp_mtup_init(sk);
465 icsk->icsk_af_ops->rebuild_header(sk);
466 tcp_init_metrics(sk);
Lawrence Brakmode525be2018-01-25 16:14:09 -0800467 tcp_call_bpf(sk, bpf_op, 0, NULL);
Wei Wang27204aa2017-10-04 10:03:44 -0700468 tcp_init_congestion_control(sk);
469 tcp_init_buffer_space(sk);
470}
471
Eric Dumazet4e8cc222017-10-05 22:21:23 -0700472static void tcp_tx_timestamp(struct sock *sk, u16 tsflags)
Willem de Bruijn4ed2d762014-08-04 22:11:49 -0400473{
Eric Dumazet4e8cc222017-10-05 22:21:23 -0700474 struct sk_buff *skb = tcp_write_queue_tail(sk);
475
Soheil Hassas Yeganehad02c4f2017-01-04 11:19:34 -0500476 if (tsflags && skb) {
Willem de Bruijnf066e2b2014-08-06 15:09:44 -0400477 struct skb_shared_info *shinfo = skb_shinfo(skb);
Soheil Hassas Yeganeh6b084922016-04-02 23:08:08 -0400478 struct tcp_skb_cb *tcb = TCP_SKB_CB(skb);
Willem de Bruijn4ed2d762014-08-04 22:11:49 -0400479
Soheil Hassas Yeganehc14ac942016-04-02 23:08:12 -0400480 sock_tx_timestamp(sk, tsflags, &shinfo->tx_flags);
Soheil Hassas Yeganeh0a2cf202016-04-27 23:39:01 -0400481 if (tsflags & SOF_TIMESTAMPING_TX_ACK)
482 tcb->txstamp_ack = 1;
483 if (tsflags & SOF_TIMESTAMPING_TX_RECORD_MASK)
Willem de Bruijnf066e2b2014-08-06 15:09:44 -0400484 shinfo->tskey = TCP_SKB_CB(skb)->seq + skb->len - 1;
485 }
Willem de Bruijn4ed2d762014-08-04 22:11:49 -0400486}
487
Linus Torvalds1da177e2005-04-16 15:20:36 -0700488/*
Linus Torvalds1da177e2005-04-16 15:20:36 -0700489 * Wait for a TCP event.
490 *
491 * Note that we don't need to lock the socket, as the upper poll layers
492 * take care of normal races (between the test and the event) and we don't
493 * go look at any of the socket buffers directly.
494 */
Al Viroade994f2017-07-03 00:01:49 -0400495__poll_t tcp_poll(struct file *file, struct socket *sock, poll_table *wait)
Linus Torvalds1da177e2005-04-16 15:20:36 -0700496{
Al Viroade994f2017-07-03 00:01:49 -0400497 __poll_t mask;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700498 struct sock *sk = sock->sk;
Eric Dumazetcf533ea2011-10-21 05:22:42 -0400499 const struct tcp_sock *tp = tcp_sk(sk);
Eric Dumazet00fd38d2015-11-12 08:43:18 -0800500 int state;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700501
Eric Dumazetaa395142010-04-20 13:03:51 +0000502 sock_poll_wait(file, sk_sleep(sk), wait);
Eric Dumazet00fd38d2015-11-12 08:43:18 -0800503
Yafang Shao986ffdf2017-12-20 11:12:52 +0800504 state = inet_sk_state_load(sk);
Eric Dumazet00fd38d2015-11-12 08:43:18 -0800505 if (state == TCP_LISTEN)
Arnaldo Carvalho de Melodc40c7b2005-08-23 21:52:58 -0700506 return inet_csk_listen_poll(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700507
508 /* Socket is not locked. We are protected from async events
Will Newton70efce22008-07-16 20:13:43 -0700509 * by poll logic and correct handling of state changes
510 * made by other threads is impossible in any case.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700511 */
512
513 mask = 0;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700514
515 /*
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800516 * EPOLLHUP is certainly not done right. But poll() doesn't
Linus Torvalds1da177e2005-04-16 15:20:36 -0700517 * have a notion of HUP in just one direction, and for a
518 * socket the read side is more interesting.
519 *
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800520 * Some poll() documentation says that EPOLLHUP is incompatible
521 * with the EPOLLOUT/POLLWR flags, so somebody should check this
Linus Torvalds1da177e2005-04-16 15:20:36 -0700522 * all. But careful, it tends to be safer to return too many
523 * bits than too few, and you can easily break real applications
524 * if you don't tell them that something has hung up!
525 *
526 * Check-me.
527 *
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800528 * Check number 1. EPOLLHUP is _UNMASKABLE_ event (see UNIX98 and
Linus Torvalds1da177e2005-04-16 15:20:36 -0700529 * our fs/select.c). It means that after we received EOF,
530 * poll always returns immediately, making impossible poll() on write()
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800531 * in state CLOSE_WAIT. One solution is evident --- to set EPOLLHUP
Linus Torvalds1da177e2005-04-16 15:20:36 -0700532 * if and only if shutdown has been made in both directions.
533 * Actually, it is interesting to look how Solaris and DUX
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800534 * solve this dilemma. I would prefer, if EPOLLHUP were maskable,
Linus Torvalds1da177e2005-04-16 15:20:36 -0700535 * then we could set it on SND_SHUTDOWN. BTW examples given
536 * in Stevens' books assume exactly this behaviour, it explains
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800537 * why EPOLLHUP is incompatible with EPOLLOUT. --ANK
Linus Torvalds1da177e2005-04-16 15:20:36 -0700538 *
539 * NOTE. Check for TCP_CLOSE is added. The goal is to prevent
540 * blocking on fresh not-connected or disconnected socket. --ANK
541 */
Eric Dumazet00fd38d2015-11-12 08:43:18 -0800542 if (sk->sk_shutdown == SHUTDOWN_MASK || state == TCP_CLOSE)
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800543 mask |= EPOLLHUP;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700544 if (sk->sk_shutdown & RCV_SHUTDOWN)
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800545 mask |= EPOLLIN | EPOLLRDNORM | EPOLLRDHUP;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700546
Jerry Chu83368862012-08-31 12:29:12 +0000547 /* Connected or passive Fast Open socket? */
Eric Dumazet00fd38d2015-11-12 08:43:18 -0800548 if (state != TCP_SYN_SENT &&
549 (state != TCP_SYN_RECV || tp->fastopen_rsk)) {
David S. Millerc7004482008-10-06 10:43:54 -0700550 int target = sock_rcvlowat(sk, 0, INT_MAX);
551
552 if (tp->urg_seq == tp->copied_seq &&
553 !sock_flag(sk, SOCK_URGINLINE) &&
554 tp->urg_data)
Alexandra Kossovskyb634f872010-03-18 20:29:24 -0700555 target++;
David S. Millerc7004482008-10-06 10:43:54 -0700556
David S. Millerc7004482008-10-06 10:43:54 -0700557 if (tp->rcv_nxt - tp->copied_seq >= target)
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800558 mask |= EPOLLIN | EPOLLRDNORM;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700559
560 if (!(sk->sk_shutdown & SEND_SHUTDOWN)) {
Eric Dumazet64dc6132013-07-22 20:26:31 -0700561 if (sk_stream_is_writeable(sk)) {
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800562 mask |= EPOLLOUT | EPOLLWRNORM;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700563 } else { /* send SIGIO later */
Eric Dumazet9cd3e072015-11-29 20:03:10 -0800564 sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700565 set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
566
567 /* Race breaker. If space is freed after
568 * wspace test but before the flags are set,
jbaron@akamai.com3c715122015-04-20 20:05:07 +0000569 * IO signal will be lost. Memory barrier
570 * pairs with the input side.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700571 */
jbaron@akamai.com3c715122015-04-20 20:05:07 +0000572 smp_mb__after_atomic();
Eric Dumazet64dc6132013-07-22 20:26:31 -0700573 if (sk_stream_is_writeable(sk))
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800574 mask |= EPOLLOUT | EPOLLWRNORM;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700575 }
KOSAKI Motohirod84ba632010-08-24 16:05:48 +0000576 } else
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800577 mask |= EPOLLOUT | EPOLLWRNORM;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700578
579 if (tp->urg_data & TCP_URG_VALID)
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800580 mask |= EPOLLPRI;
Davide Carattid68be712017-04-26 19:07:35 +0200581 } else if (state == TCP_SYN_SENT && inet_sk(sk)->defer_connect) {
Wei Wang19f6d3f2017-01-23 10:59:22 -0800582 /* Active TCP fastopen socket with defer_connect
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800583 * Return EPOLLOUT so application can call write()
Wei Wang19f6d3f2017-01-23 10:59:22 -0800584 * in order for kernel to generate SYN+data
585 */
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800586 mask |= EPOLLOUT | EPOLLWRNORM;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700587 }
Tom Marshalla4d25802010-09-20 15:42:05 -0700588 /* This barrier is coupled with smp_wmb() in tcp_reset() */
589 smp_rmb();
Willem de Bruijn4ed2d762014-08-04 22:11:49 -0400590 if (sk->sk_err || !skb_queue_empty(&sk->sk_error_queue))
Linus Torvaldsa9a08842018-02-11 14:34:03 -0800591 mask |= EPOLLERR;
Tom Marshalla4d25802010-09-20 15:42:05 -0700592
Linus Torvalds1da177e2005-04-16 15:20:36 -0700593 return mask;
594}
Eric Dumazet4bc2f182010-07-09 21:22:10 +0000595EXPORT_SYMBOL(tcp_poll);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700596
597int tcp_ioctl(struct sock *sk, int cmd, unsigned long arg)
598{
599 struct tcp_sock *tp = tcp_sk(sk);
600 int answ;
Eric Dumazet0e71c552012-10-21 20:06:56 +0000601 bool slow;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700602
603 switch (cmd) {
604 case SIOCINQ:
605 if (sk->sk_state == TCP_LISTEN)
606 return -EINVAL;
607
Eric Dumazet0e71c552012-10-21 20:06:56 +0000608 slow = lock_sock_fast(sk);
Tom Herbert473bd232016-03-07 14:11:05 -0800609 answ = tcp_inq(sk);
Eric Dumazet0e71c552012-10-21 20:06:56 +0000610 unlock_sock_fast(sk, slow);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700611 break;
612 case SIOCATMARK:
613 answ = tp->urg_data && tp->urg_seq == tp->copied_seq;
614 break;
615 case SIOCOUTQ:
616 if (sk->sk_state == TCP_LISTEN)
617 return -EINVAL;
618
619 if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV))
620 answ = 0;
621 else
622 answ = tp->write_seq - tp->snd_una;
623 break;
Mario Schuknecht2f4e1b32011-03-09 14:08:09 -0800624 case SIOCOUTQNSD:
625 if (sk->sk_state == TCP_LISTEN)
626 return -EINVAL;
627
628 if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV))
629 answ = 0;
630 else
631 answ = tp->write_seq - tp->snd_nxt;
632 break;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700633 default:
634 return -ENOIOCTLCMD;
Stephen Hemminger3ff50b72007-04-20 17:09:22 -0700635 }
Linus Torvalds1da177e2005-04-16 15:20:36 -0700636
637 return put_user(answ, (int __user *)arg);
638}
Eric Dumazet4bc2f182010-07-09 21:22:10 +0000639EXPORT_SYMBOL(tcp_ioctl);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700640
Linus Torvalds1da177e2005-04-16 15:20:36 -0700641static inline void tcp_mark_push(struct tcp_sock *tp, struct sk_buff *skb)
642{
Eric Dumazet4de075e2011-09-27 13:25:05 -0400643 TCP_SKB_CB(skb)->tcp_flags |= TCPHDR_PSH;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700644 tp->pushed_seq = tp->write_seq;
645}
646
Eric Dumazeta2a385d2012-05-16 23:15:34 +0000647static inline bool forced_push(const struct tcp_sock *tp)
Linus Torvalds1da177e2005-04-16 15:20:36 -0700648{
649 return after(tp->write_seq, tp->pushed_seq + (tp->max_window >> 1));
650}
651
Eric Dumazetf4a775d2014-09-22 16:29:32 -0700652static void skb_entail(struct sock *sk, struct sk_buff *skb)
Linus Torvalds1da177e2005-04-16 15:20:36 -0700653{
Ilpo Järvinen9e412ba2007-04-20 22:18:02 -0700654 struct tcp_sock *tp = tcp_sk(sk);
Arnaldo Carvalho de Melo352d4802006-11-17 19:59:12 -0200655 struct tcp_skb_cb *tcb = TCP_SKB_CB(skb);
656
657 skb->csum = 0;
658 tcb->seq = tcb->end_seq = tp->write_seq;
Eric Dumazet4de075e2011-09-27 13:25:05 -0400659 tcb->tcp_flags = TCPHDR_ACK;
Arnaldo Carvalho de Melo352d4802006-11-17 19:59:12 -0200660 tcb->sacked = 0;
Eric Dumazetf4a775d2014-09-22 16:29:32 -0700661 __skb_header_release(skb);
David S. Millerfe067e82007-03-07 12:12:44 -0800662 tcp_add_write_queue_tail(sk, skb);
Hideo Aoki3ab224b2007-12-31 00:11:19 -0800663 sk->sk_wmem_queued += skb->truesize;
664 sk_mem_charge(sk, skb->truesize);
David S. Miller89ebd192005-08-23 10:13:06 -0700665 if (tp->nonagle & TCP_NAGLE_PUSH)
YOSHIFUJI Hideakie905a9e2007-02-09 23:24:47 +0900666 tp->nonagle &= ~TCP_NAGLE_PUSH;
Eric Dumazet6f021c62015-08-21 12:30:00 -0700667
668 tcp_slow_start_after_idle_check(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700669}
670
Krishna Kumarafeca342009-12-10 07:16:52 +0000671static inline void tcp_mark_urg(struct tcp_sock *tp, int flags)
Linus Torvalds1da177e2005-04-16 15:20:36 -0700672{
Ilpo Järvinen33f5f572008-10-07 14:43:06 -0700673 if (flags & MSG_OOB)
Linus Torvalds1da177e2005-04-16 15:20:36 -0700674 tp->snd_up = tp->write_seq;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700675}
676
Eric Dumazetf54b3112013-12-05 22:36:05 -0800677/* If a not yet filled skb is pushed, do not send it if
Eric Dumazeta181ceb2013-12-17 09:58:30 -0800678 * we have data packets in Qdisc or NIC queues :
Eric Dumazetf54b3112013-12-05 22:36:05 -0800679 * Because TX completion will happen shortly, it gives a chance
680 * to coalesce future sendmsg() payload into this skb, without
681 * need for a timer, and with no latency trade off.
682 * As packets containing data payload have a bigger truesize
Eric Dumazeta181ceb2013-12-17 09:58:30 -0800683 * than pure acks (dataless) packets, the last checks prevent
684 * autocorking if we only have an ACK in Qdisc/NIC queues,
685 * or if TX completion was delayed after we processed ACK packet.
Eric Dumazetf54b3112013-12-05 22:36:05 -0800686 */
687static bool tcp_should_autocork(struct sock *sk, struct sk_buff *skb,
688 int size_goal)
Linus Torvalds1da177e2005-04-16 15:20:36 -0700689{
Eric Dumazetf54b3112013-12-05 22:36:05 -0800690 return skb->len < size_goal &&
Eric Dumazet790f00e2017-10-27 07:47:29 -0700691 sock_net(sk)->ipv4.sysctl_tcp_autocorking &&
Eric Dumazeta181ceb2013-12-17 09:58:30 -0800692 skb != tcp_write_queue_head(sk) &&
Reshetova, Elena14afee42017-06-30 13:08:00 +0300693 refcount_read(&sk->sk_wmem_alloc) > skb->truesize;
Eric Dumazetf54b3112013-12-05 22:36:05 -0800694}
Krishna Kumarafeca342009-12-10 07:16:52 +0000695
Eric Dumazetf54b3112013-12-05 22:36:05 -0800696static void tcp_push(struct sock *sk, int flags, int mss_now,
697 int nonagle, int size_goal)
698{
699 struct tcp_sock *tp = tcp_sk(sk);
700 struct sk_buff *skb;
Krishna Kumarafeca342009-12-10 07:16:52 +0000701
Eric Dumazetf54b3112013-12-05 22:36:05 -0800702 skb = tcp_write_queue_tail(sk);
Eric Dumazet75c119a2017-10-05 22:21:27 -0700703 if (!skb)
704 return;
Eric Dumazetf54b3112013-12-05 22:36:05 -0800705 if (!(flags & MSG_MORE) || forced_push(tp))
706 tcp_mark_push(tp, skb);
707
708 tcp_mark_urg(tp, flags);
709
710 if (tcp_should_autocork(sk, skb, size_goal)) {
711
712 /* avoid atomic op if TSQ_THROTTLED bit is already set */
Eric Dumazet7aa54702016-12-03 11:14:57 -0800713 if (!test_bit(TSQ_THROTTLED, &sk->sk_tsq_flags)) {
Eric Dumazetf54b3112013-12-05 22:36:05 -0800714 NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPAUTOCORKING);
Eric Dumazet7aa54702016-12-03 11:14:57 -0800715 set_bit(TSQ_THROTTLED, &sk->sk_tsq_flags);
Eric Dumazetf54b3112013-12-05 22:36:05 -0800716 }
Eric Dumazeta181ceb2013-12-17 09:58:30 -0800717 /* It is possible TX completion already happened
718 * before we set TSQ_THROTTLED.
719 */
Reshetova, Elena14afee42017-06-30 13:08:00 +0300720 if (refcount_read(&sk->sk_wmem_alloc) > skb->truesize)
Eric Dumazeta181ceb2013-12-17 09:58:30 -0800721 return;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700722 }
Eric Dumazetf54b3112013-12-05 22:36:05 -0800723
724 if (flags & MSG_MORE)
725 nonagle = TCP_NAGLE_CORK;
726
727 __tcp_push_pending_frames(sk, mss_now, nonagle);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700728}
729
Adrian Bunk6ff77512007-11-06 23:32:26 -0800730static int tcp_splice_data_recv(read_descriptor_t *rd_desc, struct sk_buff *skb,
731 unsigned int offset, size_t len)
Jens Axboe9c55e012007-11-06 23:30:13 -0800732{
733 struct tcp_splice_state *tss = rd_desc->arg.data;
Willy Tarreau33966dd02009-01-13 16:04:36 -0800734 int ret;
Jens Axboe9c55e012007-11-06 23:30:13 -0800735
Hannes Frederic Sowaa60e3cc2015-05-21 17:00:00 +0200736 ret = skb_splice_bits(skb, skb->sk, offset, tss->pipe,
Al Viro25869262016-09-17 21:02:10 -0400737 min(rd_desc->count, len), tss->flags);
Willy Tarreau33966dd02009-01-13 16:04:36 -0800738 if (ret > 0)
739 rd_desc->count -= ret;
740 return ret;
Jens Axboe9c55e012007-11-06 23:30:13 -0800741}
742
743static int __tcp_splice_read(struct sock *sk, struct tcp_splice_state *tss)
744{
745 /* Store TCP splice context information in read_descriptor_t. */
746 read_descriptor_t rd_desc = {
747 .arg.data = tss,
Willy Tarreau33966dd02009-01-13 16:04:36 -0800748 .count = tss->len,
Jens Axboe9c55e012007-11-06 23:30:13 -0800749 };
750
751 return tcp_read_sock(sk, &rd_desc, tcp_splice_data_recv);
752}
753
754/**
755 * tcp_splice_read - splice data from TCP socket to a pipe
756 * @sock: socket to splice from
757 * @ppos: position (not valid)
758 * @pipe: pipe to splice to
759 * @len: number of bytes to splice
760 * @flags: splice modifier flags
761 *
762 * Description:
763 * Will read pages from given socket and fill them into a pipe.
764 *
765 **/
766ssize_t tcp_splice_read(struct socket *sock, loff_t *ppos,
767 struct pipe_inode_info *pipe, size_t len,
768 unsigned int flags)
769{
770 struct sock *sk = sock->sk;
771 struct tcp_splice_state tss = {
772 .pipe = pipe,
773 .len = len,
774 .flags = flags,
775 };
776 long timeo;
777 ssize_t spliced;
778 int ret;
779
Changli Gao3a047bf2010-07-12 21:00:12 +0000780 sock_rps_record_flow(sk);
Jens Axboe9c55e012007-11-06 23:30:13 -0800781 /*
782 * We can't seek on a socket input
783 */
784 if (unlikely(*ppos))
785 return -ESPIPE;
786
787 ret = spliced = 0;
788
789 lock_sock(sk);
790
Eric Dumazet42324c62009-10-01 15:26:00 -0700791 timeo = sock_rcvtimeo(sk, sock->file->f_flags & O_NONBLOCK);
Jens Axboe9c55e012007-11-06 23:30:13 -0800792 while (tss.len) {
793 ret = __tcp_splice_read(sk, &tss);
794 if (ret < 0)
795 break;
796 else if (!ret) {
797 if (spliced)
798 break;
Jens Axboe9c55e012007-11-06 23:30:13 -0800799 if (sock_flag(sk, SOCK_DONE))
800 break;
801 if (sk->sk_err) {
802 ret = sock_error(sk);
803 break;
804 }
805 if (sk->sk_shutdown & RCV_SHUTDOWN)
806 break;
807 if (sk->sk_state == TCP_CLOSE) {
808 /*
809 * This occurs when user tries to read
810 * from never connected socket.
811 */
812 if (!sock_flag(sk, SOCK_DONE))
813 ret = -ENOTCONN;
814 break;
815 }
816 if (!timeo) {
817 ret = -EAGAIN;
818 break;
819 }
Eric Dumazetccf7abb2017-02-03 14:59:38 -0800820 /* if __tcp_splice_read() got nothing while we have
821 * an skb in receive queue, we do not want to loop.
822 * This might happen with URG data.
823 */
824 if (!skb_queue_empty(&sk->sk_receive_queue))
825 break;
Sabrina Dubrocadfbafc92015-07-24 18:19:25 +0200826 sk_wait_data(sk, &timeo, NULL);
Jens Axboe9c55e012007-11-06 23:30:13 -0800827 if (signal_pending(current)) {
828 ret = sock_intr_errno(timeo);
829 break;
830 }
831 continue;
832 }
833 tss.len -= ret;
834 spliced += ret;
835
Willy Tarreau33966dd02009-01-13 16:04:36 -0800836 if (!timeo)
837 break;
Jens Axboe9c55e012007-11-06 23:30:13 -0800838 release_sock(sk);
839 lock_sock(sk);
840
841 if (sk->sk_err || sk->sk_state == TCP_CLOSE ||
Willy Tarreau33966dd02009-01-13 16:04:36 -0800842 (sk->sk_shutdown & RCV_SHUTDOWN) ||
Jens Axboe9c55e012007-11-06 23:30:13 -0800843 signal_pending(current))
844 break;
845 }
846
847 release_sock(sk);
848
849 if (spliced)
850 return spliced;
851
852 return ret;
853}
Eric Dumazet4bc2f182010-07-09 21:22:10 +0000854EXPORT_SYMBOL(tcp_splice_read);
Jens Axboe9c55e012007-11-06 23:30:13 -0800855
Eric Dumazeteb934472015-05-19 13:26:55 -0700856struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp,
857 bool force_schedule)
Pavel Emelyanovf561d0f2007-11-29 20:28:50 +1100858{
859 struct sk_buff *skb;
860
861 /* The TCP header must be at least 32-bit aligned. */
862 size = ALIGN(size, 4);
863
Eric Dumazet8e4d9802015-05-15 12:39:28 -0700864 if (unlikely(tcp_under_memory_pressure(sk)))
865 sk_mem_reclaim_partial(sk);
866
Pavel Emelyanovf561d0f2007-11-29 20:28:50 +1100867 skb = alloc_skb_fclone(size + sk->sk_prot->max_header, gfp);
Eric Dumazet8e4d9802015-05-15 12:39:28 -0700868 if (likely(skb)) {
Eric Dumazeteb934472015-05-19 13:26:55 -0700869 bool mem_scheduled;
Eric Dumazet8e4d9802015-05-15 12:39:28 -0700870
Eric Dumazeteb934472015-05-19 13:26:55 -0700871 if (force_schedule) {
872 mem_scheduled = true;
Eric Dumazet8e4d9802015-05-15 12:39:28 -0700873 sk_forced_mem_schedule(sk, skb->truesize);
874 } else {
Eric Dumazeteb934472015-05-19 13:26:55 -0700875 mem_scheduled = sk_wmem_schedule(sk, skb->truesize);
Eric Dumazet8e4d9802015-05-15 12:39:28 -0700876 }
Eric Dumazeteb934472015-05-19 13:26:55 -0700877 if (likely(mem_scheduled)) {
Eric Dumazeta21d4572012-04-10 20:30:48 +0000878 skb_reserve(skb, sk->sk_prot->max_header);
Pavel Emelyanovf561d0f2007-11-29 20:28:50 +1100879 /*
880 * Make sure that we have exactly size bytes
881 * available to the caller, no more, no less.
882 */
Eric Dumazet16fad692013-03-14 05:40:32 +0000883 skb->reserved_tailroom = skb->end - skb->tail - size;
Eric Dumazete2080072017-10-04 12:59:58 -0700884 INIT_LIST_HEAD(&skb->tcp_tsorted_anchor);
Pavel Emelyanovf561d0f2007-11-29 20:28:50 +1100885 return skb;
886 }
887 __kfree_skb(skb);
888 } else {
Pavel Emelyanov5c52ba12008-07-16 20:28:10 -0700889 sk->sk_prot->enter_memory_pressure(sk);
Pavel Emelyanovf561d0f2007-11-29 20:28:50 +1100890 sk_stream_moderate_sndbuf(sk);
891 }
892 return NULL;
893}
894
Ilpo Järvinen0c54b852009-03-14 14:23:05 +0000895static unsigned int tcp_xmit_size_goal(struct sock *sk, u32 mss_now,
896 int large_allowed)
897{
898 struct tcp_sock *tp = tcp_sk(sk);
Eric Dumazet6c09fa02015-03-05 08:03:06 -0800899 u32 new_size_goal, size_goal;
Ilpo Järvinen0c54b852009-03-14 14:23:05 +0000900
Eric Dumazet74d4a8f2018-02-19 11:56:48 -0800901 if (!large_allowed)
Eric Dumazet605ad7f2014-12-07 12:22:18 -0800902 return mss_now;
Ilpo Järvinen0c54b852009-03-14 14:23:05 +0000903
Eric Dumazet6c09fa02015-03-05 08:03:06 -0800904 /* Note : tcp_tso_autosize() will eventually split this later */
905 new_size_goal = sk->sk_gso_max_size - 1 - MAX_TCP_HEADER;
Eric Dumazet605ad7f2014-12-07 12:22:18 -0800906 new_size_goal = tcp_bound_to_half_wnd(tp, new_size_goal);
Eric Dumazet95bd09e2013-08-27 05:46:32 -0700907
Eric Dumazet605ad7f2014-12-07 12:22:18 -0800908 /* We try hard to avoid divides here */
909 size_goal = tp->gso_segs * mss_now;
910 if (unlikely(new_size_goal < size_goal ||
911 new_size_goal >= size_goal + mss_now)) {
912 tp->gso_segs = min_t(u16, new_size_goal / mss_now,
913 sk->sk_gso_max_segs);
914 size_goal = tp->gso_segs * mss_now;
Ilpo Järvinen0c54b852009-03-14 14:23:05 +0000915 }
916
Eric Dumazet605ad7f2014-12-07 12:22:18 -0800917 return max(size_goal, mss_now);
Ilpo Järvinen0c54b852009-03-14 14:23:05 +0000918}
919
920static int tcp_send_mss(struct sock *sk, int *size_goal, int flags)
921{
922 int mss_now;
923
924 mss_now = tcp_current_mss(sk);
925 *size_goal = tcp_xmit_size_goal(sk, mss_now, !(flags & MSG_OOB));
926
927 return mss_now;
928}
929
Dave Watsone3b56162017-06-14 11:37:26 -0700930ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset,
931 size_t size, int flags)
Linus Torvalds1da177e2005-04-16 15:20:36 -0700932{
933 struct tcp_sock *tp = tcp_sk(sk);
David S. Millerc1b4a7e2005-07-05 15:24:38 -0700934 int mss_now, size_goal;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700935 int err;
936 ssize_t copied;
937 long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT);
938
Jerry Chu83368862012-08-31 12:29:12 +0000939 /* Wait for a connection to finish. One exception is TCP Fast Open
940 * (passive side) where data is allowed to be sent before a connection
941 * is fully established.
942 */
943 if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) &&
944 !tcp_passive_fastopen(sk)) {
Yuvaraja Mariappan686a5622015-10-06 10:53:29 -0700945 err = sk_stream_wait_connect(sk, &timeo);
946 if (err != 0)
Linus Torvalds1da177e2005-04-16 15:20:36 -0700947 goto out_err;
Jerry Chu83368862012-08-31 12:29:12 +0000948 }
Linus Torvalds1da177e2005-04-16 15:20:36 -0700949
Eric Dumazet9cd3e072015-11-29 20:03:10 -0800950 sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700951
Ilpo Järvinen0c54b852009-03-14 14:23:05 +0000952 mss_now = tcp_send_mss(sk, &size_goal, flags);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700953 copied = 0;
954
955 err = -EPIPE;
956 if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN))
Ilpo Järvinen0d6a7752009-02-28 04:44:41 +0000957 goto out_err;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700958
Eric Dumazet64022d02012-12-01 13:07:02 +0000959 while (size > 0) {
David S. Millerfe067e82007-03-07 12:12:44 -0800960 struct sk_buff *skb = tcp_write_queue_tail(sk);
Eric Dumazet38ba0a62012-04-23 17:48:27 +0000961 int copy, i;
Eric Dumazet38ba0a62012-04-23 17:48:27 +0000962 bool can_coalesce;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700963
Eric Dumazet75c119a2017-10-05 22:21:27 -0700964 if (!skb || (copy = size_goal - skb->len) <= 0 ||
Martin KaFai Lauc134ecb2016-04-25 14:44:48 -0700965 !tcp_skb_can_collapse_to(skb)) {
Linus Torvalds1da177e2005-04-16 15:20:36 -0700966new_segment:
967 if (!sk_stream_memory_free(sk))
968 goto wait_for_sndbuf;
969
Eric Dumazeteb934472015-05-19 13:26:55 -0700970 skb = sk_stream_alloc_skb(sk, 0, sk->sk_allocation,
Eric Dumazet75c119a2017-10-05 22:21:27 -0700971 tcp_rtx_and_write_queues_empty(sk));
Linus Torvalds1da177e2005-04-16 15:20:36 -0700972 if (!skb)
973 goto wait_for_memory;
974
Ilpo Järvinen9e412ba2007-04-20 22:18:02 -0700975 skb_entail(sk, skb);
David S. Millerc1b4a7e2005-07-05 15:24:38 -0700976 copy = size_goal;
Linus Torvalds1da177e2005-04-16 15:20:36 -0700977 }
978
979 if (copy > size)
980 copy = size;
981
982 i = skb_shinfo(skb)->nr_frags;
983 can_coalesce = skb_can_coalesce(skb, i, page, offset);
Hans Westgaard Ry5f74f82e2016-02-03 09:26:57 +0100984 if (!can_coalesce && i >= sysctl_max_skb_frags) {
Linus Torvalds1da177e2005-04-16 15:20:36 -0700985 tcp_mark_push(tp, skb);
986 goto new_segment;
987 }
Hideo Aoki3ab224b2007-12-31 00:11:19 -0800988 if (!sk_wmem_schedule(sk, copy))
Linus Torvalds1da177e2005-04-16 15:20:36 -0700989 goto wait_for_memory;
YOSHIFUJI Hideakie905a9e2007-02-09 23:24:47 +0900990
Linus Torvalds1da177e2005-04-16 15:20:36 -0700991 if (can_coalesce) {
Eric Dumazet9e903e02011-10-18 21:00:24 +0000992 skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy);
Linus Torvalds1da177e2005-04-16 15:20:36 -0700993 } else {
994 get_page(page);
995 skb_fill_page_desc(skb, i, page, offset, copy);
996 }
Pravin B Shelarc9af6db2013-02-11 09:27:41 +0000997 skb_shinfo(skb)->tx_flags |= SKBTX_SHARED_FRAG;
Eric Dumazetcef401d2013-01-25 20:34:37 +0000998
Linus Torvalds1da177e2005-04-16 15:20:36 -0700999 skb->len += copy;
1000 skb->data_len += copy;
1001 skb->truesize += copy;
1002 sk->sk_wmem_queued += copy;
Hideo Aoki3ab224b2007-12-31 00:11:19 -08001003 sk_mem_charge(sk, copy);
Patrick McHardy84fa7932006-08-29 16:44:56 -07001004 skb->ip_summed = CHECKSUM_PARTIAL;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001005 tp->write_seq += copy;
1006 TCP_SKB_CB(skb)->end_seq += copy;
Eric Dumazetcd7d8492014-09-24 04:11:22 -07001007 tcp_skb_pcount_set(skb, 0);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001008
1009 if (!copied)
Eric Dumazet4de075e2011-09-27 13:25:05 -04001010 TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001011
1012 copied += copy;
Eric Dumazet64022d02012-12-01 13:07:02 +00001013 offset += copy;
Yuvaraja Mariappan686a5622015-10-06 10:53:29 -07001014 size -= copy;
Soheil Hassas Yeganehad02c4f2017-01-04 11:19:34 -05001015 if (!size)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001016 goto out;
1017
Herbert Xu69d15062008-03-22 15:47:05 -07001018 if (skb->len < size_goal || (flags & MSG_OOB))
Linus Torvalds1da177e2005-04-16 15:20:36 -07001019 continue;
1020
1021 if (forced_push(tp)) {
1022 tcp_mark_push(tp, skb);
Ilpo Järvinen9e412ba2007-04-20 22:18:02 -07001023 __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH);
David S. Millerfe067e82007-03-07 12:12:44 -08001024 } else if (skb == tcp_send_head(sk))
Linus Torvalds1da177e2005-04-16 15:20:36 -07001025 tcp_push_one(sk, mss_now);
1026 continue;
1027
1028wait_for_sndbuf:
1029 set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
1030wait_for_memory:
Eric Dumazetf54b3112013-12-05 22:36:05 -08001031 tcp_push(sk, flags & ~MSG_MORE, mss_now,
1032 TCP_NAGLE_PUSH, size_goal);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001033
Yuvaraja Mariappan686a5622015-10-06 10:53:29 -07001034 err = sk_stream_wait_memory(sk, &timeo);
1035 if (err != 0)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001036 goto do_error;
1037
Ilpo Järvinen0c54b852009-03-14 14:23:05 +00001038 mss_now = tcp_send_mss(sk, &size_goal, flags);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001039 }
1040
1041out:
Soheil Hassas Yeganehad02c4f2017-01-04 11:19:34 -05001042 if (copied) {
Eric Dumazet4e8cc222017-10-05 22:21:23 -07001043 tcp_tx_timestamp(sk, sk->sk_tsflags);
Soheil Hassas Yeganehad02c4f2017-01-04 11:19:34 -05001044 if (!(flags & MSG_SENDPAGE_NOTLAST))
1045 tcp_push(sk, flags, mss_now, tp->nonagle, size_goal);
1046 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07001047 return copied;
1048
1049do_error:
1050 if (copied)
1051 goto out;
1052out_err:
Jason Baronce5ec442015-05-20 15:52:53 +00001053 /* make sure we wake any epoll edge trigger waiter */
Francis Yanb0f71bd2016-11-27 23:07:16 -08001054 if (unlikely(skb_queue_len(&sk->sk_write_queue) == 0 &&
1055 err == -EAGAIN)) {
Jason Baronce5ec442015-05-20 15:52:53 +00001056 sk->sk_write_space(sk);
Francis Yanb0f71bd2016-11-27 23:07:16 -08001057 tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED);
1058 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07001059 return sk_stream_error(sk, flags, err);
1060}
Dave Watsone3b56162017-06-14 11:37:26 -07001061EXPORT_SYMBOL_GPL(do_tcp_sendpages);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001062
Tom Herbert306b13e2017-07-28 16:22:41 -07001063int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset,
1064 size_t size, int flags)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001065{
Eric Dumazetdead7cd2018-02-19 11:56:49 -08001066 if (!(sk->sk_route_caps & NETIF_F_SG))
Eric Dumazetbd9dfc52017-08-25 06:27:05 -07001067 return sock_no_sendpage_locked(sk, page, offset, size, flags);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001068
Soheil Hassas Yeganehd7722e82016-09-19 23:39:15 -04001069 tcp_rate_check_app_limited(sk); /* is sending application-limited? */
1070
Tom Herbert306b13e2017-07-28 16:22:41 -07001071 return do_tcp_sendpages(sk, page, offset, size, flags);
1072}
David S. Miller774c4672017-08-16 15:40:44 -07001073EXPORT_SYMBOL_GPL(tcp_sendpage_locked);
Tom Herbert306b13e2017-07-28 16:22:41 -07001074
1075int tcp_sendpage(struct sock *sk, struct page *page, int offset,
1076 size_t size, int flags)
1077{
1078 int ret;
1079
1080 lock_sock(sk);
1081 ret = tcp_sendpage_locked(sk, page, offset, size, flags);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001082 release_sock(sk);
Tom Herbert306b13e2017-07-28 16:22:41 -07001083
1084 return ret;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001085}
Eric Dumazet4bc2f182010-07-09 21:22:10 +00001086EXPORT_SYMBOL(tcp_sendpage);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001087
Eric Dumazet3613b3d2016-09-15 09:33:02 -07001088/* Do not bother using a page frag for very small frames.
1089 * But use this heuristic only for the first skb in write queue.
1090 *
1091 * Having no payload in skb->head allows better SACK shifting
1092 * in tcp_shift_skb_data(), reducing sack/rack overhead, because
1093 * write queue has less skbs.
1094 * Each skb can hold up to MAX_SKB_FRAGS * 32Kbytes, or ~0.5 MB.
1095 * This also speeds up tso_fragment(), since it wont fallback
1096 * to tcp_fragment().
1097 */
1098static int linear_payload_sz(bool first_skb)
1099{
1100 if (first_skb)
1101 return SKB_WITH_OVERHEAD(2048 - MAX_TCP_HEADER);
1102 return 0;
1103}
1104
Eric Dumazet74d4a8f2018-02-19 11:56:48 -08001105static int select_size(bool first_skb, bool zc)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001106{
Eric Dumazet74d4a8f2018-02-19 11:56:48 -08001107 if (zc)
1108 return 0;
1109 return linear_payload_sz(first_skb);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001110}
1111
Yuchung Chengcf60af02012-07-19 06:43:09 +00001112void tcp_free_fastopen_req(struct tcp_sock *tp)
1113{
Ian Morris00db4122015-04-03 09:17:27 +01001114 if (tp->fastopen_req) {
Yuchung Chengcf60af02012-07-19 06:43:09 +00001115 kfree(tp->fastopen_req);
1116 tp->fastopen_req = NULL;
1117 }
1118}
1119
Eric Dumazetf5ddcbb2014-02-20 10:09:18 -08001120static int tcp_sendmsg_fastopen(struct sock *sk, struct msghdr *msg,
1121 int *copied, size_t size)
Yuchung Chengcf60af02012-07-19 06:43:09 +00001122{
1123 struct tcp_sock *tp = tcp_sk(sk);
Wei Wang19f6d3f2017-01-23 10:59:22 -08001124 struct inet_sock *inet = inet_sk(sk);
Wei Wangba615f62017-05-24 09:59:31 -07001125 struct sockaddr *uaddr = msg->msg_name;
Yuchung Chengcf60af02012-07-19 06:43:09 +00001126 int err, flags;
1127
Haishuang Yane1cfcbe2017-09-27 11:35:40 +08001128 if (!(sock_net(sk)->ipv4.sysctl_tcp_fastopen & TFO_CLIENT_ENABLE) ||
Wei Wangba615f62017-05-24 09:59:31 -07001129 (uaddr && msg->msg_namelen >= sizeof(uaddr->sa_family) &&
1130 uaddr->sa_family == AF_UNSPEC))
Yuchung Chengcf60af02012-07-19 06:43:09 +00001131 return -EOPNOTSUPP;
Ian Morris00db4122015-04-03 09:17:27 +01001132 if (tp->fastopen_req)
Yuchung Chengcf60af02012-07-19 06:43:09 +00001133 return -EALREADY; /* Another Fast Open is in progress */
1134
1135 tp->fastopen_req = kzalloc(sizeof(struct tcp_fastopen_request),
1136 sk->sk_allocation);
Ian Morris51456b22015-04-03 09:17:26 +01001137 if (unlikely(!tp->fastopen_req))
Yuchung Chengcf60af02012-07-19 06:43:09 +00001138 return -ENOBUFS;
1139 tp->fastopen_req->data = msg;
Eric Dumazetf5ddcbb2014-02-20 10:09:18 -08001140 tp->fastopen_req->size = size;
Yuchung Chengcf60af02012-07-19 06:43:09 +00001141
Wei Wang19f6d3f2017-01-23 10:59:22 -08001142 if (inet->defer_connect) {
1143 err = tcp_connect(sk);
1144 /* Same failure procedure as in tcp_v4/6_connect */
1145 if (err) {
1146 tcp_set_state(sk, TCP_CLOSE);
1147 inet->inet_dport = 0;
1148 sk->sk_route_caps = 0;
1149 }
1150 }
Yuchung Chengcf60af02012-07-19 06:43:09 +00001151 flags = (msg->msg_flags & MSG_DONTWAIT) ? O_NONBLOCK : 0;
Wei Wangba615f62017-05-24 09:59:31 -07001152 err = __inet_stream_connect(sk->sk_socket, uaddr,
Willy Tarreau3979ad72017-01-25 14:42:46 +01001153 msg->msg_namelen, flags, 1);
Wei Wang7db92362017-03-01 13:29:48 -08001154 /* fastopen_req could already be freed in __inet_stream_connect
1155 * if the connection times out or gets rst
1156 */
1157 if (tp->fastopen_req) {
1158 *copied = tp->fastopen_req->copied;
1159 tcp_free_fastopen_req(tp);
1160 inet->defer_connect = 0;
1161 }
Yuchung Chengcf60af02012-07-19 06:43:09 +00001162 return err;
1163}
1164
Tom Herbert306b13e2017-07-28 16:22:41 -07001165int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001166{
Linus Torvalds1da177e2005-04-16 15:20:36 -07001167 struct tcp_sock *tp = tcp_sk(sk);
Willem de Bruijnf214f912017-08-03 16:29:44 -04001168 struct ubuf_info *uarg = NULL;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001169 struct sk_buff *skb;
Soheil Hassas Yeganehc14ac942016-04-02 23:08:12 -04001170 struct sockcm_cookie sockc;
Al Viro57be5bd2014-11-28 13:40:20 -05001171 int flags, err, copied = 0;
1172 int mss_now = 0, size_goal, copied_syn = 0;
Eric Dumazetd4011232016-05-02 21:49:25 -07001173 bool process_backlog = false;
Eric Dumazet74d4a8f2018-02-19 11:56:48 -08001174 bool zc = false;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001175 long timeo;
1176
Linus Torvalds1da177e2005-04-16 15:20:36 -07001177 flags = msg->msg_flags;
Willem de Bruijnf214f912017-08-03 16:29:44 -04001178
1179 if (flags & MSG_ZEROCOPY && size) {
1180 if (sk->sk_state != TCP_ESTABLISHED) {
1181 err = -EINVAL;
1182 goto out_err;
1183 }
1184
Eric Dumazet75c119a2017-10-05 22:21:27 -07001185 skb = tcp_write_queue_tail(sk);
Willem de Bruijnf214f912017-08-03 16:29:44 -04001186 uarg = sock_zerocopy_realloc(sk, size, skb_zcopy(skb));
1187 if (!uarg) {
1188 err = -ENOBUFS;
1189 goto out_err;
1190 }
1191
Eric Dumazetdead7cd2018-02-19 11:56:49 -08001192 zc = sk->sk_route_caps & NETIF_F_SG;
Willem de Bruijn02583ad2017-12-22 19:00:19 -05001193 if (!zc)
Willem de Bruijnf214f912017-08-03 16:29:44 -04001194 uarg->zerocopy = 0;
1195 }
1196
Wei Wang19f6d3f2017-01-23 10:59:22 -08001197 if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect)) {
Eric Dumazetf5ddcbb2014-02-20 10:09:18 -08001198 err = tcp_sendmsg_fastopen(sk, msg, &copied_syn, size);
Yuchung Chengcf60af02012-07-19 06:43:09 +00001199 if (err == -EINPROGRESS && copied_syn > 0)
1200 goto out;
1201 else if (err)
1202 goto out_err;
Yuchung Chengcf60af02012-07-19 06:43:09 +00001203 }
1204
Linus Torvalds1da177e2005-04-16 15:20:36 -07001205 timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT);
1206
Soheil Hassas Yeganehd7722e82016-09-19 23:39:15 -04001207 tcp_rate_check_app_limited(sk); /* is sending application-limited? */
1208
Jerry Chu83368862012-08-31 12:29:12 +00001209 /* Wait for a connection to finish. One exception is TCP Fast Open
1210 * (passive side) where data is allowed to be sent before a connection
1211 * is fully established.
1212 */
1213 if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) &&
1214 !tcp_passive_fastopen(sk)) {
Yuvaraja Mariappan686a5622015-10-06 10:53:29 -07001215 err = sk_stream_wait_connect(sk, &timeo);
1216 if (err != 0)
Yuchung Chengcf60af02012-07-19 06:43:09 +00001217 goto do_error;
Jerry Chu83368862012-08-31 12:29:12 +00001218 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07001219
Pavel Emelyanovc0e88ff2012-04-19 03:41:01 +00001220 if (unlikely(tp->repair)) {
1221 if (tp->repair_queue == TCP_RECV_QUEUE) {
1222 copied = tcp_send_rcvq(sk, msg, size);
Christoph Paasch5924f172014-06-28 18:26:37 +02001223 goto out_nopush;
Pavel Emelyanovc0e88ff2012-04-19 03:41:01 +00001224 }
1225
1226 err = -EINVAL;
1227 if (tp->repair_queue == TCP_NO_QUEUE)
1228 goto out_err;
1229
1230 /* 'common' sending to sendq */
1231 }
1232
Soheil Hassas Yeganehc14ac942016-04-02 23:08:12 -04001233 sockc.tsflags = sk->sk_tsflags;
1234 if (msg->msg_controllen) {
1235 err = sock_cmsg_send(sk, msg, &sockc);
1236 if (unlikely(err)) {
1237 err = -EINVAL;
1238 goto out_err;
1239 }
1240 }
1241
Linus Torvalds1da177e2005-04-16 15:20:36 -07001242 /* This should be in poll */
Eric Dumazet9cd3e072015-11-29 20:03:10 -08001243 sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001244
Linus Torvalds1da177e2005-04-16 15:20:36 -07001245 /* Ok commence sending. */
Linus Torvalds1da177e2005-04-16 15:20:36 -07001246 copied = 0;
1247
Eric Dumazetd41a69f2016-04-29 14:16:53 -07001248restart:
1249 mss_now = tcp_send_mss(sk, &size_goal, flags);
1250
Linus Torvalds1da177e2005-04-16 15:20:36 -07001251 err = -EPIPE;
1252 if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN))
Eric Dumazet79d86652016-11-02 14:41:50 -07001253 goto do_error;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001254
Al Viro01e97e62014-12-15 21:39:31 -05001255 while (msg_data_left(msg)) {
Al Viro57be5bd2014-11-28 13:40:20 -05001256 int copy = 0;
1257 int max = size_goal;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001258
Al Viro57be5bd2014-11-28 13:40:20 -05001259 skb = tcp_write_queue_tail(sk);
Eric Dumazet75c119a2017-10-05 22:21:27 -07001260 if (skb) {
Al Viro57be5bd2014-11-28 13:40:20 -05001261 if (skb->ip_summed == CHECKSUM_NONE)
1262 max = mss_now;
1263 copy = max - skb->len;
Yuchung Chengcf60af02012-07-19 06:43:09 +00001264 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07001265
Martin KaFai Lauc134ecb2016-04-25 14:44:48 -07001266 if (copy <= 0 || !tcp_skb_can_collapse_to(skb)) {
Eric Dumazet3613b3d2016-09-15 09:33:02 -07001267 bool first_skb;
Willem de Bruijn8ddab502017-12-22 19:00:20 -05001268 int linear;
Eric Dumazet3613b3d2016-09-15 09:33:02 -07001269
Linus Torvalds1da177e2005-04-16 15:20:36 -07001270new_segment:
Al Viro57be5bd2014-11-28 13:40:20 -05001271 /* Allocate new segment. If the interface is SG,
1272 * allocate skb fitting to single page.
1273 */
1274 if (!sk_stream_memory_free(sk))
1275 goto wait_for_sndbuf;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001276
Eric Dumazetd4011232016-05-02 21:49:25 -07001277 if (process_backlog && sk_flush_backlog(sk)) {
1278 process_backlog = false;
Eric Dumazetd41a69f2016-04-29 14:16:53 -07001279 goto restart;
Eric Dumazetd4011232016-05-02 21:49:25 -07001280 }
Eric Dumazet75c119a2017-10-05 22:21:27 -07001281 first_skb = tcp_rtx_and_write_queues_empty(sk);
Eric Dumazet74d4a8f2018-02-19 11:56:48 -08001282 linear = select_size(first_skb, zc);
Willem de Bruijn8ddab502017-12-22 19:00:20 -05001283 skb = sk_stream_alloc_skb(sk, linear, sk->sk_allocation,
Eric Dumazet3613b3d2016-09-15 09:33:02 -07001284 first_skb);
Al Viro57be5bd2014-11-28 13:40:20 -05001285 if (!skb)
1286 goto wait_for_memory;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001287
Eric Dumazetd4011232016-05-02 21:49:25 -07001288 process_backlog = true;
Eric Dumazetdead7cd2018-02-19 11:56:49 -08001289 skb->ip_summed = CHECKSUM_PARTIAL;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001290
Al Viro57be5bd2014-11-28 13:40:20 -05001291 skb_entail(sk, skb);
1292 copy = size_goal;
1293 max = size_goal;
Andrey Vagin9d186ca2014-08-13 16:03:10 +04001294
Al Viro57be5bd2014-11-28 13:40:20 -05001295 /* All packets are restored as if they have
1296 * already been sent. skb_mstamp isn't set to
1297 * avoid wrong rtt estimation.
1298 */
1299 if (tp->repair)
1300 TCP_SKB_CB(skb)->sacked |= TCPCB_REPAIRED;
1301 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07001302
Al Viro57be5bd2014-11-28 13:40:20 -05001303 /* Try to append data to the end of skb. */
Al Viro01e97e62014-12-15 21:39:31 -05001304 if (copy > msg_data_left(msg))
1305 copy = msg_data_left(msg);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001306
Al Viro57be5bd2014-11-28 13:40:20 -05001307 /* Where to copy to? */
Willem de Bruijn02583ad2017-12-22 19:00:19 -05001308 if (skb_availroom(skb) > 0 && !zc) {
Al Viro57be5bd2014-11-28 13:40:20 -05001309 /* We have some space in skb head. Superb! */
1310 copy = min_t(int, copy, skb_availroom(skb));
1311 err = skb_add_data_nocache(sk, skb, &msg->msg_iter, copy);
1312 if (err)
1313 goto do_fault;
Willem de Bruijn02583ad2017-12-22 19:00:19 -05001314 } else if (!zc) {
Al Viro57be5bd2014-11-28 13:40:20 -05001315 bool merge = true;
1316 int i = skb_shinfo(skb)->nr_frags;
1317 struct page_frag *pfrag = sk_page_frag(sk);
Eric Dumazet761965e2011-12-04 07:05:17 +00001318
Al Viro57be5bd2014-11-28 13:40:20 -05001319 if (!sk_page_frag_refill(sk, pfrag))
1320 goto wait_for_memory;
Eric Dumazet761965e2011-12-04 07:05:17 +00001321
Al Viro57be5bd2014-11-28 13:40:20 -05001322 if (!skb_can_coalesce(skb, i, pfrag->page,
1323 pfrag->offset)) {
Eric Dumazet74d4a8f2018-02-19 11:56:48 -08001324 if (i >= sysctl_max_skb_frags) {
Al Viro57be5bd2014-11-28 13:40:20 -05001325 tcp_mark_push(tp, skb);
1326 goto new_segment;
Eric Dumazet5640f762012-09-23 23:04:42 +00001327 }
Al Viro57be5bd2014-11-28 13:40:20 -05001328 merge = false;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001329 }
1330
Al Viro57be5bd2014-11-28 13:40:20 -05001331 copy = min_t(int, copy, pfrag->size - pfrag->offset);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001332
Al Viro57be5bd2014-11-28 13:40:20 -05001333 if (!sk_wmem_schedule(sk, copy))
1334 goto wait_for_memory;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001335
Al Viro57be5bd2014-11-28 13:40:20 -05001336 err = skb_copy_to_page_nocache(sk, &msg->msg_iter, skb,
1337 pfrag->page,
1338 pfrag->offset,
1339 copy);
1340 if (err)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001341 goto do_error;
1342
Al Viro57be5bd2014-11-28 13:40:20 -05001343 /* Update the skb. */
1344 if (merge) {
1345 skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy);
1346 } else {
1347 skb_fill_page_desc(skb, i, pfrag->page,
1348 pfrag->offset, copy);
Eric Dumazet4e33e342017-02-17 09:11:42 -08001349 page_ref_inc(pfrag->page);
Al Viro57be5bd2014-11-28 13:40:20 -05001350 }
1351 pfrag->offset += copy;
Willem de Bruijnf214f912017-08-03 16:29:44 -04001352 } else {
1353 err = skb_zerocopy_iter_stream(sk, skb, msg, copy, uarg);
Willem de Bruijn111856c2017-12-22 19:00:18 -05001354 if (err == -EMSGSIZE || err == -EEXIST) {
1355 tcp_mark_push(tp, skb);
Willem de Bruijnf214f912017-08-03 16:29:44 -04001356 goto new_segment;
Willem de Bruijn111856c2017-12-22 19:00:18 -05001357 }
Willem de Bruijnf214f912017-08-03 16:29:44 -04001358 if (err < 0)
1359 goto do_error;
1360 copy = err;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001361 }
Al Viro57be5bd2014-11-28 13:40:20 -05001362
1363 if (!copied)
1364 TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH;
1365
1366 tp->write_seq += copy;
1367 TCP_SKB_CB(skb)->end_seq += copy;
1368 tcp_skb_pcount_set(skb, 0);
1369
1370 copied += copy;
Al Viro01e97e62014-12-15 21:39:31 -05001371 if (!msg_data_left(msg)) {
Martin KaFai Lauc134ecb2016-04-25 14:44:48 -07001372 if (unlikely(flags & MSG_EOR))
1373 TCP_SKB_CB(skb)->eor = 1;
Al Viro57be5bd2014-11-28 13:40:20 -05001374 goto out;
1375 }
1376
1377 if (skb->len < max || (flags & MSG_OOB) || unlikely(tp->repair))
1378 continue;
1379
1380 if (forced_push(tp)) {
1381 tcp_mark_push(tp, skb);
1382 __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH);
1383 } else if (skb == tcp_send_head(sk))
1384 tcp_push_one(sk, mss_now);
1385 continue;
1386
1387wait_for_sndbuf:
1388 set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
1389wait_for_memory:
1390 if (copied)
1391 tcp_push(sk, flags & ~MSG_MORE, mss_now,
1392 TCP_NAGLE_PUSH, size_goal);
1393
Yuvaraja Mariappan686a5622015-10-06 10:53:29 -07001394 err = sk_stream_wait_memory(sk, &timeo);
1395 if (err != 0)
Al Viro57be5bd2014-11-28 13:40:20 -05001396 goto do_error;
1397
1398 mss_now = tcp_send_mss(sk, &size_goal, flags);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001399 }
1400
1401out:
Soheil Hassas Yeganehad02c4f2017-01-04 11:19:34 -05001402 if (copied) {
Eric Dumazet4e8cc222017-10-05 22:21:23 -07001403 tcp_tx_timestamp(sk, sockc.tsflags);
Eric Dumazetf54b3112013-12-05 22:36:05 -08001404 tcp_push(sk, flags, mss_now, tp->nonagle, size_goal);
Soheil Hassas Yeganehad02c4f2017-01-04 11:19:34 -05001405 }
Christoph Paasch5924f172014-06-28 18:26:37 +02001406out_nopush:
Willem de Bruijnf214f912017-08-03 16:29:44 -04001407 sock_zerocopy_put(uarg);
Yuchung Chengcf60af02012-07-19 06:43:09 +00001408 return copied + copied_syn;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001409
1410do_fault:
1411 if (!skb->len) {
David S. Millerfe067e82007-03-07 12:12:44 -08001412 tcp_unlink_write_queue(skb, sk);
1413 /* It is the one place in all of TCP, except connection
1414 * reset, where we can be unlinking the send_head.
1415 */
1416 tcp_check_send_head(sk, skb);
Hideo Aoki3ab224b2007-12-31 00:11:19 -08001417 sk_wmem_free_skb(sk, skb);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001418 }
1419
1420do_error:
Yuchung Chengcf60af02012-07-19 06:43:09 +00001421 if (copied + copied_syn)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001422 goto out;
1423out_err:
Willem de Bruijnf214f912017-08-03 16:29:44 -04001424 sock_zerocopy_put_abort(uarg);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001425 err = sk_stream_error(sk, flags, err);
Jason Baronce5ec442015-05-20 15:52:53 +00001426 /* make sure we wake any epoll edge trigger waiter */
Francis Yanb0f71bd2016-11-27 23:07:16 -08001427 if (unlikely(skb_queue_len(&sk->sk_write_queue) == 0 &&
1428 err == -EAGAIN)) {
Jason Baronce5ec442015-05-20 15:52:53 +00001429 sk->sk_write_space(sk);
Francis Yanb0f71bd2016-11-27 23:07:16 -08001430 tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED);
1431 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07001432 return err;
1433}
David S. Miller774c4672017-08-16 15:40:44 -07001434EXPORT_SYMBOL_GPL(tcp_sendmsg_locked);
Tom Herbert306b13e2017-07-28 16:22:41 -07001435
1436int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
1437{
1438 int ret;
1439
1440 lock_sock(sk);
1441 ret = tcp_sendmsg_locked(sk, msg, size);
1442 release_sock(sk);
1443
1444 return ret;
1445}
Eric Dumazet4bc2f182010-07-09 21:22:10 +00001446EXPORT_SYMBOL(tcp_sendmsg);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001447
1448/*
1449 * Handle reading urgent data. BSD has very simple semantics for
1450 * this, no blocking and very strange errors 8)
1451 */
1452
Rami Rosen377f0a02009-03-31 14:43:17 -07001453static int tcp_recv_urg(struct sock *sk, struct msghdr *msg, int len, int flags)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001454{
1455 struct tcp_sock *tp = tcp_sk(sk);
1456
1457 /* No URG data to read. */
1458 if (sock_flag(sk, SOCK_URGINLINE) || !tp->urg_data ||
1459 tp->urg_data == TCP_URG_READ)
1460 return -EINVAL; /* Yes this is right ! */
1461
1462 if (sk->sk_state == TCP_CLOSE && !sock_flag(sk, SOCK_DONE))
1463 return -ENOTCONN;
1464
1465 if (tp->urg_data & TCP_URG_VALID) {
1466 int err = 0;
1467 char c = tp->urg_data;
1468
1469 if (!(flags & MSG_PEEK))
1470 tp->urg_data = TCP_URG_READ;
1471
1472 /* Read urgent data. */
1473 msg->msg_flags |= MSG_OOB;
1474
1475 if (len > 0) {
1476 if (!(flags & MSG_TRUNC))
Al Viro7eab8d92014-04-06 21:51:23 -04001477 err = memcpy_to_msg(msg, &c, 1);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001478 len = 1;
1479 } else
1480 msg->msg_flags |= MSG_TRUNC;
1481
1482 return err ? -EFAULT : len;
1483 }
1484
1485 if (sk->sk_state == TCP_CLOSE || (sk->sk_shutdown & RCV_SHUTDOWN))
1486 return 0;
1487
1488 /* Fixed the recv(..., MSG_OOB) behaviour. BSD docs and
1489 * the available implementations agree in this case:
1490 * this call should never block, independent of the
1491 * blocking state of the socket.
1492 * Mike <pall@rz.uni-karlsruhe.de>
1493 */
1494 return -EAGAIN;
1495}
1496
Pavel Emelyanovc0e88ff2012-04-19 03:41:01 +00001497static int tcp_peek_sndq(struct sock *sk, struct msghdr *msg, int len)
1498{
1499 struct sk_buff *skb;
1500 int copied = 0, err = 0;
1501
1502 /* XXX -- need to support SO_PEEK_OFF */
1503
Eric Dumazet75c119a2017-10-05 22:21:27 -07001504 skb_rbtree_walk(skb, &sk->tcp_rtx_queue) {
1505 err = skb_copy_datagram_msg(skb, 0, msg, skb->len);
1506 if (err)
1507 return err;
1508 copied += skb->len;
1509 }
1510
Pavel Emelyanovc0e88ff2012-04-19 03:41:01 +00001511 skb_queue_walk(&sk->sk_write_queue, skb) {
David S. Miller51f3d022014-11-05 16:46:40 -05001512 err = skb_copy_datagram_msg(skb, 0, msg, skb->len);
Pavel Emelyanovc0e88ff2012-04-19 03:41:01 +00001513 if (err)
1514 break;
1515
1516 copied += skb->len;
1517 }
1518
1519 return err ?: copied;
1520}
1521
Linus Torvalds1da177e2005-04-16 15:20:36 -07001522/* Clean up the receive buffer for full frames taken by the user,
1523 * then send an ACK if necessary. COPIED is the number of bytes
1524 * tcp_recvmsg has given to the user so far, it speeds up the
1525 * calculation of whether or not we must ACK for the sake of
1526 * a window update.
1527 */
Dan Williams3f334072013-12-30 17:12:07 -08001528static void tcp_cleanup_rbuf(struct sock *sk, int copied)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001529{
1530 struct tcp_sock *tp = tcp_sk(sk);
Eric Dumazeta2a385d2012-05-16 23:15:34 +00001531 bool time_to_ack = false;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001532
Linus Torvalds1da177e2005-04-16 15:20:36 -07001533 struct sk_buff *skb = skb_peek(&sk->sk_receive_queue);
1534
Ilpo Järvinend792c102009-11-13 13:56:33 -08001535 WARN(skb && !before(tp->copied_seq, TCP_SKB_CB(skb)->end_seq),
Joe Perches2af6fd82010-10-30 11:08:53 +00001536 "cleanup rbuf bug: copied %X seq %X rcvnxt %X\n",
Ilpo Järvinend792c102009-11-13 13:56:33 -08001537 tp->copied_seq, TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001538
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07001539 if (inet_csk_ack_scheduled(sk)) {
1540 const struct inet_connection_sock *icsk = inet_csk(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001541 /* Delayed ACKs frequently hit locked sockets during bulk
1542 * receive. */
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07001543 if (icsk->icsk_ack.blocked ||
Linus Torvalds1da177e2005-04-16 15:20:36 -07001544 /* Once-per-two-segments ACK was not sent by tcp_input.c */
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07001545 tp->rcv_nxt - tp->rcv_wup > icsk->icsk_ack.rcv_mss ||
Linus Torvalds1da177e2005-04-16 15:20:36 -07001546 /*
1547 * If this read emptied read buffer, we send ACK, if
1548 * connection is not bidirectional, user drained
1549 * receive buffer and there was a small segment
1550 * in queue.
1551 */
Alexey Kuznetsov1ef96962006-09-19 12:52:50 -07001552 (copied > 0 &&
1553 ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED2) ||
1554 ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED) &&
1555 !icsk->icsk_ack.pingpong)) &&
1556 !atomic_read(&sk->sk_rmem_alloc)))
Eric Dumazeta2a385d2012-05-16 23:15:34 +00001557 time_to_ack = true;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001558 }
1559
1560 /* We send an ACK if we can now advertise a non-zero window
1561 * which has been raised "significantly".
1562 *
1563 * Even if window raised up to infinity, do not send window open ACK
1564 * in states, where we will not receive more. It is useless.
1565 */
1566 if (copied > 0 && !time_to_ack && !(sk->sk_shutdown & RCV_SHUTDOWN)) {
1567 __u32 rcv_window_now = tcp_receive_window(tp);
1568
1569 /* Optimize, __tcp_select_window() is not cheap. */
1570 if (2*rcv_window_now <= tp->window_clamp) {
1571 __u32 new_window = __tcp_select_window(sk);
1572
1573 /* Send ACK now, if this read freed lots of space
1574 * in our buffer. Certainly, new_window is new window.
1575 * We can advertise it now, if it is not less than current one.
1576 * "Lots" means "at least twice" here.
1577 */
1578 if (new_window && new_window >= 2 * rcv_window_now)
Eric Dumazeta2a385d2012-05-16 23:15:34 +00001579 time_to_ack = true;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001580 }
1581 }
1582 if (time_to_ack)
1583 tcp_send_ack(sk);
1584}
1585
Eric Dumazetf26845b2013-01-09 20:59:09 +00001586static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001587{
1588 struct sk_buff *skb;
1589 u32 offset;
1590
Eric Dumazetf26845b2013-01-09 20:59:09 +00001591 while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) {
Linus Torvalds1da177e2005-04-16 15:20:36 -07001592 offset = seq - TCP_SKB_CB(skb)->seq;
Eric Dumazet9d691532016-02-01 21:03:08 -08001593 if (unlikely(TCP_SKB_CB(skb)->tcp_flags & TCPHDR_SYN)) {
1594 pr_err_once("%s: found a SYN, please report !\n", __func__);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001595 offset--;
Eric Dumazet9d691532016-02-01 21:03:08 -08001596 }
Eric Dumazete11ecdd2014-09-15 04:19:51 -07001597 if (offset < skb->len || (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)) {
Linus Torvalds1da177e2005-04-16 15:20:36 -07001598 *off = offset;
1599 return skb;
1600 }
Eric Dumazetf26845b2013-01-09 20:59:09 +00001601 /* This looks weird, but this can happen if TCP collapsing
1602 * splitted a fat GRO packet, while we released socket lock
1603 * in skb_splice_bits()
1604 */
Dan Williams7bced392013-12-30 12:37:29 -08001605 sk_eat_skb(sk, skb);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001606 }
1607 return NULL;
1608}
1609
1610/*
1611 * This routine provides an alternative to tcp_recvmsg() for routines
1612 * that would like to handle copying from skbuffs directly in 'sendfile'
1613 * fashion.
1614 * Note:
1615 * - It is assumed that the socket was locked by the caller.
1616 * - The routine does not block.
1617 * - At present, there is no support for reading OOB data
1618 * or for 'peeking' the socket using this routine
1619 * (although both would be easy to implement).
1620 */
1621int tcp_read_sock(struct sock *sk, read_descriptor_t *desc,
1622 sk_read_actor_t recv_actor)
1623{
1624 struct sk_buff *skb;
1625 struct tcp_sock *tp = tcp_sk(sk);
1626 u32 seq = tp->copied_seq;
1627 u32 offset;
1628 int copied = 0;
1629
1630 if (sk->sk_state == TCP_LISTEN)
1631 return -ENOTCONN;
1632 while ((skb = tcp_recv_skb(sk, seq, &offset)) != NULL) {
1633 if (offset < skb->len) {
Octavian Purdila374e7b52008-07-03 03:31:21 -07001634 int used;
1635 size_t len;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001636
1637 len = skb->len - offset;
1638 /* Stop reading if we hit a patch of urgent data */
1639 if (tp->urg_data) {
1640 u32 urg_offset = tp->urg_seq - seq;
1641 if (urg_offset < len)
1642 len = urg_offset;
1643 if (!len)
1644 break;
1645 }
1646 used = recv_actor(desc, skb, offset, len);
Eric Dumazetff905b12013-01-10 07:06:10 +00001647 if (used <= 0) {
Jens Axboeddb61a52007-06-23 23:07:50 -07001648 if (!copied)
1649 copied = used;
1650 break;
1651 } else if (used <= len) {
Linus Torvalds1da177e2005-04-16 15:20:36 -07001652 seq += used;
1653 copied += used;
1654 offset += used;
1655 }
Willy Tarreau02275a22012-12-02 11:49:27 +00001656 /* If recv_actor drops the lock (e.g. TCP splice
Octavian Purdila293ad602008-06-04 15:45:58 -07001657 * receive) the skb pointer might be invalid when
1658 * getting here: tcp_collapse might have deleted it
1659 * while aggregating skbs from the socket queue.
1660 */
Willy Tarreau02275a22012-12-02 11:49:27 +00001661 skb = tcp_recv_skb(sk, seq - 1, &offset);
1662 if (!skb)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001663 break;
Willy Tarreau02275a22012-12-02 11:49:27 +00001664 /* TCP coalescing might have appended data to the skb.
1665 * Try to splice more frags
1666 */
1667 if (offset + 1 != skb->len)
1668 continue;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001669 }
Eric Dumazete11ecdd2014-09-15 04:19:51 -07001670 if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN) {
Dan Williams7bced392013-12-30 12:37:29 -08001671 sk_eat_skb(sk, skb);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001672 ++seq;
1673 break;
1674 }
Dan Williams7bced392013-12-30 12:37:29 -08001675 sk_eat_skb(sk, skb);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001676 if (!desc->count)
1677 break;
Steven J. Magnanibaff42a2010-03-30 13:56:01 -07001678 tp->copied_seq = seq;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001679 }
1680 tp->copied_seq = seq;
1681
1682 tcp_rcv_space_adjust(sk);
1683
1684 /* Clean up data we have read: This will do ACK frames. */
Eric Dumazetf26845b2013-01-09 20:59:09 +00001685 if (copied > 0) {
1686 tcp_recv_skb(sk, seq, &offset);
Chris Leech0e4b4992006-05-23 18:00:16 -07001687 tcp_cleanup_rbuf(sk, copied);
Eric Dumazetf26845b2013-01-09 20:59:09 +00001688 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07001689 return copied;
1690}
Eric Dumazet4bc2f182010-07-09 21:22:10 +00001691EXPORT_SYMBOL(tcp_read_sock);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001692
Tom Herbert32035582016-08-28 14:43:18 -07001693int tcp_peek_len(struct socket *sock)
1694{
1695 return tcp_inq(sock->sk);
1696}
1697EXPORT_SYMBOL(tcp_peek_len);
1698
Mike Maloney98aaa912017-08-22 17:08:48 -04001699static void tcp_update_recv_tstamps(struct sk_buff *skb,
1700 struct scm_timestamping *tss)
1701{
1702 if (skb->tstamp)
1703 tss->ts[0] = ktime_to_timespec(skb->tstamp);
1704 else
1705 tss->ts[0] = (struct timespec) {0};
1706
1707 if (skb_hwtstamps(skb)->hwtstamp)
1708 tss->ts[2] = ktime_to_timespec(skb_hwtstamps(skb)->hwtstamp);
1709 else
1710 tss->ts[2] = (struct timespec) {0};
1711}
1712
1713/* Similar to __sock_recv_timestamp, but does not require an skb */
Wei Yongjun809a79e2018-01-10 07:43:15 +00001714static void tcp_recv_timestamp(struct msghdr *msg, const struct sock *sk,
1715 struct scm_timestamping *tss)
Mike Maloney98aaa912017-08-22 17:08:48 -04001716{
1717 struct timeval tv;
1718 bool has_timestamping = false;
1719
1720 if (tss->ts[0].tv_sec || tss->ts[0].tv_nsec) {
1721 if (sock_flag(sk, SOCK_RCVTSTAMP)) {
1722 if (sock_flag(sk, SOCK_RCVTSTAMPNS)) {
1723 put_cmsg(msg, SOL_SOCKET, SCM_TIMESTAMPNS,
1724 sizeof(tss->ts[0]), &tss->ts[0]);
1725 } else {
1726 tv.tv_sec = tss->ts[0].tv_sec;
1727 tv.tv_usec = tss->ts[0].tv_nsec / 1000;
1728
1729 put_cmsg(msg, SOL_SOCKET, SCM_TIMESTAMP,
1730 sizeof(tv), &tv);
1731 }
1732 }
1733
1734 if (sk->sk_tsflags & SOF_TIMESTAMPING_SOFTWARE)
1735 has_timestamping = true;
1736 else
1737 tss->ts[0] = (struct timespec) {0};
1738 }
1739
1740 if (tss->ts[2].tv_sec || tss->ts[2].tv_nsec) {
1741 if (sk->sk_tsflags & SOF_TIMESTAMPING_RAW_HARDWARE)
1742 has_timestamping = true;
1743 else
1744 tss->ts[2] = (struct timespec) {0};
1745 }
1746
1747 if (has_timestamping) {
1748 tss->ts[1] = (struct timespec) {0};
1749 put_cmsg(msg, SOL_SOCKET, SCM_TIMESTAMPING,
1750 sizeof(*tss), tss);
1751 }
1752}
1753
Linus Torvalds1da177e2005-04-16 15:20:36 -07001754/*
1755 * This routine copies from a sock struct into the user buffer.
1756 *
1757 * Technical note: in 2.3 we work on _locked_ socket, so that
1758 * tricks with *seq access order and skb->users are not required.
1759 * Probably, code can be easily improved even more.
1760 */
1761
Ying Xue1b784142015-03-02 15:37:48 +08001762int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
1763 int flags, int *addr_len)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001764{
1765 struct tcp_sock *tp = tcp_sk(sk);
1766 int copied = 0;
1767 u32 peek_seq;
1768 u32 *seq;
1769 unsigned long used;
1770 int err;
1771 int target; /* Read at least this many bytes */
1772 long timeo;
Sabrina Dubrocadfbafc92015-07-24 18:19:25 +02001773 struct sk_buff *skb, *last;
Ilpo Järvinen77527312009-05-10 20:32:34 +00001774 u32 urg_hole = 0;
Mike Maloney98aaa912017-08-22 17:08:48 -04001775 struct scm_timestamping tss;
1776 bool has_tss = false;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001777
Willem de Bruijn4ed2d762014-08-04 22:11:49 -04001778 if (unlikely(flags & MSG_ERRQUEUE))
Willem de Bruijnf4713a32014-11-26 14:53:02 -05001779 return inet_recv_error(sk, msg, len, addr_len);
Willem de Bruijn4ed2d762014-08-04 22:11:49 -04001780
Eliezer Tamircbf55002013-07-08 16:20:34 +03001781 if (sk_can_busy_loop(sk) && skb_queue_empty(&sk->sk_receive_queue) &&
1782 (sk->sk_state == TCP_ESTABLISHED))
1783 sk_busy_loop(sk, nonblock);
Eliezer Tamird30e3832013-06-10 11:40:10 +03001784
Linus Torvalds1da177e2005-04-16 15:20:36 -07001785 lock_sock(sk);
1786
Linus Torvalds1da177e2005-04-16 15:20:36 -07001787 err = -ENOTCONN;
1788 if (sk->sk_state == TCP_LISTEN)
1789 goto out;
1790
1791 timeo = sock_rcvtimeo(sk, nonblock);
1792
1793 /* Urgent data needs to be handled specially. */
1794 if (flags & MSG_OOB)
1795 goto recv_urg;
1796
Pavel Emelyanovc0e88ff2012-04-19 03:41:01 +00001797 if (unlikely(tp->repair)) {
1798 err = -EPERM;
1799 if (!(flags & MSG_PEEK))
1800 goto out;
1801
1802 if (tp->repair_queue == TCP_SEND_QUEUE)
1803 goto recv_sndq;
1804
1805 err = -EINVAL;
1806 if (tp->repair_queue == TCP_NO_QUEUE)
1807 goto out;
1808
1809 /* 'common' recv queue MSG_PEEK-ing */
1810 }
1811
Linus Torvalds1da177e2005-04-16 15:20:36 -07001812 seq = &tp->copied_seq;
1813 if (flags & MSG_PEEK) {
1814 peek_seq = tp->copied_seq;
1815 seq = &peek_seq;
1816 }
1817
1818 target = sock_rcvlowat(sk, flags & MSG_WAITALL, len);
1819
1820 do {
Linus Torvalds1da177e2005-04-16 15:20:36 -07001821 u32 offset;
1822
1823 /* Are we at urgent data? Stop if we have read anything or have SIGURG pending. */
1824 if (tp->urg_data && tp->urg_seq == *seq) {
1825 if (copied)
1826 break;
1827 if (signal_pending(current)) {
1828 copied = timeo ? sock_intr_errno(timeo) : -EAGAIN;
1829 break;
1830 }
1831 }
1832
1833 /* Next get a buffer. */
1834
Sabrina Dubrocadfbafc92015-07-24 18:19:25 +02001835 last = skb_peek_tail(&sk->sk_receive_queue);
David S. Miller91521942009-05-28 21:35:47 -07001836 skb_queue_walk(&sk->sk_receive_queue, skb) {
Sabrina Dubrocadfbafc92015-07-24 18:19:25 +02001837 last = skb;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001838 /* Now that we have two receive queues this
1839 * shouldn't happen.
1840 */
Ilpo Järvinend792c102009-11-13 13:56:33 -08001841 if (WARN(before(*seq, TCP_SKB_CB(skb)->seq),
Joe Perches2af6fd82010-10-30 11:08:53 +00001842 "recvmsg bug: copied %X seq %X rcvnxt %X fl %X\n",
1843 *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt,
1844 flags))
Linus Torvalds1da177e2005-04-16 15:20:36 -07001845 break;
Ilpo Järvinend792c102009-11-13 13:56:33 -08001846
Linus Torvalds1da177e2005-04-16 15:20:36 -07001847 offset = *seq - TCP_SKB_CB(skb)->seq;
Eric Dumazet9d691532016-02-01 21:03:08 -08001848 if (unlikely(TCP_SKB_CB(skb)->tcp_flags & TCPHDR_SYN)) {
1849 pr_err_once("%s: found a SYN, please report !\n", __func__);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001850 offset--;
Eric Dumazet9d691532016-02-01 21:03:08 -08001851 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07001852 if (offset < skb->len)
1853 goto found_ok_skb;
Eric Dumazete11ecdd2014-09-15 04:19:51 -07001854 if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001855 goto found_fin_ok;
Joe Perches2af6fd82010-10-30 11:08:53 +00001856 WARN(!(flags & MSG_PEEK),
1857 "recvmsg bug 2: copied %X seq %X rcvnxt %X fl %X\n",
1858 *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt, flags);
David S. Miller91521942009-05-28 21:35:47 -07001859 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07001860
1861 /* Well, if we have backlog, try to process it now yet. */
1862
1863 if (copied >= target && !sk->sk_backlog.tail)
1864 break;
1865
1866 if (copied) {
1867 if (sk->sk_err ||
1868 sk->sk_state == TCP_CLOSE ||
1869 (sk->sk_shutdown & RCV_SHUTDOWN) ||
1870 !timeo ||
David S. Miller518a09e2008-11-05 03:36:01 -08001871 signal_pending(current))
Linus Torvalds1da177e2005-04-16 15:20:36 -07001872 break;
1873 } else {
1874 if (sock_flag(sk, SOCK_DONE))
1875 break;
1876
1877 if (sk->sk_err) {
1878 copied = sock_error(sk);
1879 break;
1880 }
1881
1882 if (sk->sk_shutdown & RCV_SHUTDOWN)
1883 break;
1884
1885 if (sk->sk_state == TCP_CLOSE) {
1886 if (!sock_flag(sk, SOCK_DONE)) {
1887 /* This occurs when user tries to read
1888 * from never connected socket.
1889 */
1890 copied = -ENOTCONN;
1891 break;
1892 }
1893 break;
1894 }
1895
1896 if (!timeo) {
1897 copied = -EAGAIN;
1898 break;
1899 }
1900
1901 if (signal_pending(current)) {
1902 copied = sock_intr_errno(timeo);
1903 break;
1904 }
1905 }
1906
Chris Leech0e4b4992006-05-23 18:00:16 -07001907 tcp_cleanup_rbuf(sk, copied);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001908
Linus Torvalds1da177e2005-04-16 15:20:36 -07001909 if (copied >= target) {
1910 /* Do not sleep, just process backlog. */
1911 release_sock(sk);
1912 lock_sock(sk);
Sabrina Dubrocadfbafc92015-07-24 18:19:25 +02001913 } else {
1914 sk_wait_data(sk, &timeo, last);
1915 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07001916
Ilpo Järvinen77527312009-05-10 20:32:34 +00001917 if ((flags & MSG_PEEK) &&
1918 (peek_seq - copied - urg_hole != tp->copied_seq)) {
Joe Perchese87cc472012-05-13 21:56:26 +00001919 net_dbg_ratelimited("TCP(%s:%d): Application bug, race in MSG_PEEK\n",
1920 current->comm,
1921 task_pid_nr(current));
Linus Torvalds1da177e2005-04-16 15:20:36 -07001922 peek_seq = tp->copied_seq;
1923 }
1924 continue;
1925
1926 found_ok_skb:
1927 /* Ok so how much can we use? */
1928 used = skb->len - offset;
1929 if (len < used)
1930 used = len;
1931
1932 /* Do we have urgent data here? */
1933 if (tp->urg_data) {
1934 u32 urg_offset = tp->urg_seq - *seq;
1935 if (urg_offset < used) {
1936 if (!urg_offset) {
1937 if (!sock_flag(sk, SOCK_URGINLINE)) {
1938 ++*seq;
Ilpo Järvinen77527312009-05-10 20:32:34 +00001939 urg_hole++;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001940 offset++;
1941 used--;
1942 if (!used)
1943 goto skip_copy;
1944 }
1945 } else
1946 used = urg_offset;
1947 }
1948 }
1949
1950 if (!(flags & MSG_TRUNC)) {
David S. Miller51f3d022014-11-05 16:46:40 -05001951 err = skb_copy_datagram_msg(skb, offset, msg, used);
Dan Williams7bced392013-12-30 12:37:29 -08001952 if (err) {
1953 /* Exception. Bailout! */
1954 if (!copied)
1955 copied = -EFAULT;
1956 break;
Linus Torvalds1da177e2005-04-16 15:20:36 -07001957 }
1958 }
1959
1960 *seq += used;
1961 copied += used;
1962 len -= used;
1963
1964 tcp_rcv_space_adjust(sk);
1965
1966skip_copy:
Florian Westphal31770e32017-08-30 19:24:58 +02001967 if (tp->urg_data && after(tp->copied_seq, tp->urg_seq)) {
Linus Torvalds1da177e2005-04-16 15:20:36 -07001968 tp->urg_data = 0;
Florian Westphal31770e32017-08-30 19:24:58 +02001969 tcp_fast_path_check(sk);
1970 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07001971 if (used + offset < skb->len)
1972 continue;
1973
Mike Maloney98aaa912017-08-22 17:08:48 -04001974 if (TCP_SKB_CB(skb)->has_rxtstamp) {
1975 tcp_update_recv_tstamps(skb, &tss);
1976 has_tss = true;
1977 }
Eric Dumazete11ecdd2014-09-15 04:19:51 -07001978 if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
Linus Torvalds1da177e2005-04-16 15:20:36 -07001979 goto found_fin_ok;
Dan Williams7bced392013-12-30 12:37:29 -08001980 if (!(flags & MSG_PEEK))
1981 sk_eat_skb(sk, skb);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001982 continue;
1983
1984 found_fin_ok:
1985 /* Process the FIN. */
1986 ++*seq;
Dan Williams7bced392013-12-30 12:37:29 -08001987 if (!(flags & MSG_PEEK))
1988 sk_eat_skb(sk, skb);
Linus Torvalds1da177e2005-04-16 15:20:36 -07001989 break;
1990 } while (len > 0);
1991
Linus Torvalds1da177e2005-04-16 15:20:36 -07001992 /* According to UNIX98, msg_name/msg_namelen are ignored
1993 * on connected socket. I was just happy when found this 8) --ANK
1994 */
1995
Mike Maloney98aaa912017-08-22 17:08:48 -04001996 if (has_tss)
1997 tcp_recv_timestamp(msg, sk, &tss);
1998
Linus Torvalds1da177e2005-04-16 15:20:36 -07001999 /* Clean up data we have read: This will do ACK frames. */
Chris Leech0e4b4992006-05-23 18:00:16 -07002000 tcp_cleanup_rbuf(sk, copied);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002001
Linus Torvalds1da177e2005-04-16 15:20:36 -07002002 release_sock(sk);
2003 return copied;
2004
2005out:
Linus Torvalds1da177e2005-04-16 15:20:36 -07002006 release_sock(sk);
2007 return err;
2008
2009recv_urg:
Rami Rosen377f0a02009-03-31 14:43:17 -07002010 err = tcp_recv_urg(sk, msg, len, flags);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002011 goto out;
Pavel Emelyanovc0e88ff2012-04-19 03:41:01 +00002012
2013recv_sndq:
2014 err = tcp_peek_sndq(sk, msg, len);
2015 goto out;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002016}
Eric Dumazet4bc2f182010-07-09 21:22:10 +00002017EXPORT_SYMBOL(tcp_recvmsg);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002018
Ilpo Järvinen490d5042008-01-12 03:17:20 -08002019void tcp_set_state(struct sock *sk, int state)
2020{
2021 int oldstate = sk->sk_state;
2022
Lawrence Brakmod4487492018-01-25 16:14:15 -08002023 /* We defined a new enum for TCP states that are exported in BPF
2024 * so as not force the internal TCP states to be frozen. The
2025 * following checks will detect if an internal state value ever
2026 * differs from the BPF value. If this ever happens, then we will
2027 * need to remap the internal value to the BPF value before calling
2028 * tcp_call_bpf_2arg.
2029 */
2030 BUILD_BUG_ON((int)BPF_TCP_ESTABLISHED != (int)TCP_ESTABLISHED);
2031 BUILD_BUG_ON((int)BPF_TCP_SYN_SENT != (int)TCP_SYN_SENT);
2032 BUILD_BUG_ON((int)BPF_TCP_SYN_RECV != (int)TCP_SYN_RECV);
2033 BUILD_BUG_ON((int)BPF_TCP_FIN_WAIT1 != (int)TCP_FIN_WAIT1);
2034 BUILD_BUG_ON((int)BPF_TCP_FIN_WAIT2 != (int)TCP_FIN_WAIT2);
2035 BUILD_BUG_ON((int)BPF_TCP_TIME_WAIT != (int)TCP_TIME_WAIT);
2036 BUILD_BUG_ON((int)BPF_TCP_CLOSE != (int)TCP_CLOSE);
2037 BUILD_BUG_ON((int)BPF_TCP_CLOSE_WAIT != (int)TCP_CLOSE_WAIT);
2038 BUILD_BUG_ON((int)BPF_TCP_LAST_ACK != (int)TCP_LAST_ACK);
2039 BUILD_BUG_ON((int)BPF_TCP_LISTEN != (int)TCP_LISTEN);
2040 BUILD_BUG_ON((int)BPF_TCP_CLOSING != (int)TCP_CLOSING);
2041 BUILD_BUG_ON((int)BPF_TCP_NEW_SYN_RECV != (int)TCP_NEW_SYN_RECV);
2042 BUILD_BUG_ON((int)BPF_TCP_MAX_STATES != (int)TCP_MAX_STATES);
2043
2044 if (BPF_SOCK_OPS_TEST_FLAG(tcp_sk(sk), BPF_SOCK_OPS_STATE_CB_FLAG))
2045 tcp_call_bpf_2arg(sk, BPF_SOCK_OPS_STATE_CB, oldstate, state);
Song Liue8fce232017-10-23 09:20:27 -07002046
Ilpo Järvinen490d5042008-01-12 03:17:20 -08002047 switch (state) {
2048 case TCP_ESTABLISHED:
2049 if (oldstate != TCP_ESTABLISHED)
Pavel Emelyanov81cc8a72008-07-16 20:22:04 -07002050 TCP_INC_STATS(sock_net(sk), TCP_MIB_CURRESTAB);
Ilpo Järvinen490d5042008-01-12 03:17:20 -08002051 break;
2052
2053 case TCP_CLOSE:
2054 if (oldstate == TCP_CLOSE_WAIT || oldstate == TCP_ESTABLISHED)
Pavel Emelyanov81cc8a72008-07-16 20:22:04 -07002055 TCP_INC_STATS(sock_net(sk), TCP_MIB_ESTABRESETS);
Ilpo Järvinen490d5042008-01-12 03:17:20 -08002056
2057 sk->sk_prot->unhash(sk);
2058 if (inet_csk(sk)->icsk_bind_hash &&
2059 !(sk->sk_userlocks & SOCK_BINDPORT_LOCK))
Arnaldo Carvalho de Meloab1e0a12008-02-03 04:06:04 -08002060 inet_put_port(sk);
Ilpo Järvinen490d5042008-01-12 03:17:20 -08002061 /* fall through */
2062 default:
Jianjun Kong5a5f3a82008-11-03 00:24:34 -08002063 if (oldstate == TCP_ESTABLISHED)
Pavel Emelyanov74688e42008-07-16 20:22:46 -07002064 TCP_DEC_STATS(sock_net(sk), TCP_MIB_CURRESTAB);
Ilpo Järvinen490d5042008-01-12 03:17:20 -08002065 }
2066
2067 /* Change state AFTER socket is unhashed to avoid closed
2068 * socket sitting in hash tables.
2069 */
Yafang Shao563e0bb2017-12-20 11:12:51 +08002070 inet_sk_state_store(sk, state);
Ilpo Järvinen490d5042008-01-12 03:17:20 -08002071
2072#ifdef STATE_TRACE
Jianjun Kong5a5f3a82008-11-03 00:24:34 -08002073 SOCK_DEBUG(sk, "TCP sk=%p, State %s -> %s\n", sk, statename[oldstate], statename[state]);
Ilpo Järvinen490d5042008-01-12 03:17:20 -08002074#endif
2075}
2076EXPORT_SYMBOL_GPL(tcp_set_state);
2077
Linus Torvalds1da177e2005-04-16 15:20:36 -07002078/*
2079 * State processing on a close. This implements the state shift for
2080 * sending our FIN frame. Note that we only send a FIN for some
2081 * states. A shutdown() may have already sent the FIN, or we may be
2082 * closed.
2083 */
2084
Arjan van de Ven9b5b5cf2005-11-29 16:21:38 -08002085static const unsigned char new_state[16] = {
Linus Torvalds1da177e2005-04-16 15:20:36 -07002086 /* current state: new state: action: */
Eric Dumazet0980c1e2015-03-24 15:58:53 -07002087 [0 /* (Invalid) */] = TCP_CLOSE,
2088 [TCP_ESTABLISHED] = TCP_FIN_WAIT1 | TCP_ACTION_FIN,
2089 [TCP_SYN_SENT] = TCP_CLOSE,
2090 [TCP_SYN_RECV] = TCP_FIN_WAIT1 | TCP_ACTION_FIN,
2091 [TCP_FIN_WAIT1] = TCP_FIN_WAIT1,
2092 [TCP_FIN_WAIT2] = TCP_FIN_WAIT2,
2093 [TCP_TIME_WAIT] = TCP_CLOSE,
2094 [TCP_CLOSE] = TCP_CLOSE,
2095 [TCP_CLOSE_WAIT] = TCP_LAST_ACK | TCP_ACTION_FIN,
2096 [TCP_LAST_ACK] = TCP_LAST_ACK,
2097 [TCP_LISTEN] = TCP_CLOSE,
2098 [TCP_CLOSING] = TCP_CLOSING,
2099 [TCP_NEW_SYN_RECV] = TCP_CLOSE, /* should not happen ! */
Linus Torvalds1da177e2005-04-16 15:20:36 -07002100};
2101
2102static int tcp_close_state(struct sock *sk)
2103{
2104 int next = (int)new_state[sk->sk_state];
2105 int ns = next & TCP_STATE_MASK;
2106
2107 tcp_set_state(sk, ns);
2108
2109 return next & TCP_ACTION_FIN;
2110}
2111
2112/*
2113 * Shutdown the sending side of a connection. Much like close except
Satoru SATOH1f29b052008-04-21 02:27:58 -07002114 * that we don't receive shut down or sock_set_flag(sk, SOCK_DEAD).
Linus Torvalds1da177e2005-04-16 15:20:36 -07002115 */
2116
2117void tcp_shutdown(struct sock *sk, int how)
2118{
2119 /* We need to grab some memory, and put together a FIN,
2120 * and then put it into the queue to be sent.
2121 * Tim MacKenzie(tym@dibbler.cs.monash.edu.au) 4 Dec '92.
2122 */
2123 if (!(how & SEND_SHUTDOWN))
2124 return;
2125
2126 /* If we've already sent a FIN, or it's a closed state, skip this. */
2127 if ((1 << sk->sk_state) &
2128 (TCPF_ESTABLISHED | TCPF_SYN_SENT |
2129 TCPF_SYN_RECV | TCPF_CLOSE_WAIT)) {
2130 /* Clear out any half completed packets. FIN if needed. */
2131 if (tcp_close_state(sk))
2132 tcp_send_fin(sk);
2133 }
2134}
Eric Dumazet4bc2f182010-07-09 21:22:10 +00002135EXPORT_SYMBOL(tcp_shutdown);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002136
Arun Sharmaefcdbf22012-01-30 14:16:06 -08002137bool tcp_check_oom(struct sock *sk, int shift)
2138{
2139 bool too_many_orphans, out_of_socket_memory;
2140
2141 too_many_orphans = tcp_too_many_orphans(sk, shift);
2142 out_of_socket_memory = tcp_out_of_memory(sk);
2143
Joe Perchese87cc472012-05-13 21:56:26 +00002144 if (too_many_orphans)
2145 net_info_ratelimited("too many orphaned sockets\n");
2146 if (out_of_socket_memory)
2147 net_info_ratelimited("out of memory -- consider tuning tcp_mem\n");
Arun Sharmaefcdbf22012-01-30 14:16:06 -08002148 return too_many_orphans || out_of_socket_memory;
2149}
2150
Linus Torvalds1da177e2005-04-16 15:20:36 -07002151void tcp_close(struct sock *sk, long timeout)
2152{
2153 struct sk_buff *skb;
2154 int data_was_unread = 0;
Herbert Xu75c2d9072006-05-03 23:31:35 -07002155 int state;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002156
2157 lock_sock(sk);
2158 sk->sk_shutdown = SHUTDOWN_MASK;
2159
2160 if (sk->sk_state == TCP_LISTEN) {
2161 tcp_set_state(sk, TCP_CLOSE);
2162
2163 /* Special case. */
Arnaldo Carvalho de Melo0a5578c2005-08-09 20:11:41 -07002164 inet_csk_listen_stop(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002165
2166 goto adjudge_to_death;
2167 }
2168
2169 /* We need to flush the recv. buffs. We do this only on the
2170 * descriptor close, not protocol-sourced closes, because the
2171 * reader process may not have drained the data yet!
2172 */
2173 while ((skb = __skb_dequeue(&sk->sk_receive_queue)) != NULL) {
Eric Dumazete11ecdd2014-09-15 04:19:51 -07002174 u32 len = TCP_SKB_CB(skb)->end_seq - TCP_SKB_CB(skb)->seq;
2175
2176 if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
2177 len--;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002178 data_was_unread += len;
2179 __kfree_skb(skb);
2180 }
2181
Hideo Aoki3ab224b2007-12-31 00:11:19 -08002182 sk_mem_reclaim(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002183
Konstantin Khorenko565b7b22010-06-24 21:54:58 -07002184 /* If socket has been already reset (e.g. in tcp_reset()) - kill it. */
2185 if (sk->sk_state == TCP_CLOSE)
2186 goto adjudge_to_death;
2187
Gerrit Renker65bb7232007-04-28 21:21:46 -07002188 /* As outlined in RFC 2525, section 2.17, we send a RST here because
2189 * data was lost. To witness the awful effects of the old behavior of
2190 * always doing a FIN, run an older 2.1.x kernel or 2.0.x, start a bulk
2191 * GET in an FTP client, suspend the process, wait for the client to
2192 * advertise a zero window, then kill -9 the FTP client, wheee...
2193 * Note: timeout is always zero in such a case.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002194 */
Pavel Emelyanovee995282012-04-19 03:40:39 +00002195 if (unlikely(tcp_sk(sk)->repair)) {
2196 sk->sk_prot->disconnect(sk, 0);
2197 } else if (data_was_unread) {
Linus Torvalds1da177e2005-04-16 15:20:36 -07002198 /* Unread data was tossed, zap the connection. */
Eric Dumazet6aef70a2016-04-27 16:44:27 -07002199 NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONCLOSE);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002200 tcp_set_state(sk, TCP_CLOSE);
Wu Fengguangaa133072009-09-02 23:45:45 -07002201 tcp_send_active_reset(sk, sk->sk_allocation);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002202 } else if (sock_flag(sk, SOCK_LINGER) && !sk->sk_lingertime) {
2203 /* Check zero linger _after_ checking for unread data. */
2204 sk->sk_prot->disconnect(sk, 0);
Eric Dumazet6aef70a2016-04-27 16:44:27 -07002205 NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002206 } else if (tcp_close_state(sk)) {
2207 /* We FIN if the application ate all the data before
2208 * zapping the connection.
2209 */
2210
2211 /* RED-PEN. Formally speaking, we have broken TCP state
2212 * machine. State transitions:
2213 *
2214 * TCP_ESTABLISHED -> TCP_FIN_WAIT1
2215 * TCP_SYN_RECV -> TCP_FIN_WAIT1 (forget it, it's impossible)
2216 * TCP_CLOSE_WAIT -> TCP_LAST_ACK
2217 *
2218 * are legal only when FIN has been sent (i.e. in window),
2219 * rather than queued out of window. Purists blame.
2220 *
2221 * F.e. "RFC state" is ESTABLISHED,
2222 * if Linux state is FIN-WAIT-1, but FIN is still not sent.
2223 *
2224 * The visible declinations are that sometimes
2225 * we enter time-wait state, when it is not required really
2226 * (harmless), do not send active resets, when they are
2227 * required by specs (TCP_ESTABLISHED, TCP_CLOSE_WAIT, when
2228 * they look as CLOSING or LAST_ACK for Linux)
2229 * Probably, I missed some more holelets.
2230 * --ANK
Jerry Chu83368862012-08-31 12:29:12 +00002231 * XXX (TFO) - To start off we don't support SYN+ACK+FIN
2232 * in a single packet! (May consider it later but will
2233 * probably need API support or TCP_CORK SYN-ACK until
2234 * data is written and socket is closed.)
Linus Torvalds1da177e2005-04-16 15:20:36 -07002235 */
2236 tcp_send_fin(sk);
2237 }
2238
2239 sk_stream_wait_close(sk, timeout);
2240
2241adjudge_to_death:
Herbert Xu75c2d9072006-05-03 23:31:35 -07002242 state = sk->sk_state;
2243 sock_hold(sk);
2244 sock_orphan(sk);
Herbert Xu75c2d9072006-05-03 23:31:35 -07002245
Linus Torvalds1da177e2005-04-16 15:20:36 -07002246 /* It is the last release_sock in its life. It will remove backlog. */
2247 release_sock(sk);
2248
2249
2250 /* Now socket is owned by kernel and we acquire BH lock
Rohit Chavana777f712017-05-22 11:59:15 +05302251 * to finish close. No need to check for user refs.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002252 */
2253 local_bh_disable();
2254 bh_lock_sock(sk);
Ilpo Järvinen547b7922008-07-25 21:43:18 -07002255 WARN_ON(sock_owned_by_user(sk));
Linus Torvalds1da177e2005-04-16 15:20:36 -07002256
Herbert Xueb4dea52008-12-29 23:04:08 -08002257 percpu_counter_inc(sk->sk_prot->orphan_count);
2258
Herbert Xu75c2d9072006-05-03 23:31:35 -07002259 /* Have we already been destroyed by a softirq or backlog? */
2260 if (state != TCP_CLOSE && sk->sk_state == TCP_CLOSE)
2261 goto out;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002262
2263 /* This is a (useful) BSD violating of the RFC. There is a
2264 * problem with TCP as specified in that the other end could
2265 * keep a socket open forever with no application left this end.
Jesper Juhlb10bd542014-02-09 23:30:32 +01002266 * We use a 1 minute timeout (about the same as BSD) then kill
Linus Torvalds1da177e2005-04-16 15:20:36 -07002267 * our end. If they send after that then tough - BUT: long enough
2268 * that we won't make the old 4*rto = almost no time - whoops
2269 * reset mistake.
2270 *
2271 * Nope, it was not mistake. It is really desired behaviour
2272 * f.e. on http servers, when such sockets are useless, but
2273 * consume significant resources. Let's do it with special
2274 * linger2 option. --ANK
2275 */
2276
2277 if (sk->sk_state == TCP_FIN_WAIT2) {
2278 struct tcp_sock *tp = tcp_sk(sk);
2279 if (tp->linger2 < 0) {
2280 tcp_set_state(sk, TCP_CLOSE);
2281 tcp_send_active_reset(sk, GFP_ATOMIC);
Eric Dumazet02a1d6e2016-04-27 16:44:39 -07002282 __NET_INC_STATS(sock_net(sk),
Pavel Emelyanovde0744a2008-07-16 20:31:16 -07002283 LINUX_MIB_TCPABORTONLINGER);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002284 } else {
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002285 const int tmo = tcp_fin_time(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002286
2287 if (tmo > TCP_TIMEWAIT_LEN) {
David S. Miller52499af2006-07-31 22:32:09 -07002288 inet_csk_reset_keepalive_timer(sk,
2289 tmo - TCP_TIMEWAIT_LEN);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002290 } else {
Linus Torvalds1da177e2005-04-16 15:20:36 -07002291 tcp_time_wait(sk, TCP_FIN_WAIT2, tmo);
2292 goto out;
2293 }
2294 }
2295 }
2296 if (sk->sk_state != TCP_CLOSE) {
Hideo Aoki3ab224b2007-12-31 00:11:19 -08002297 sk_mem_reclaim(sk);
Arun Sharmaefcdbf22012-01-30 14:16:06 -08002298 if (tcp_check_oom(sk, 0)) {
Linus Torvalds1da177e2005-04-16 15:20:36 -07002299 tcp_set_state(sk, TCP_CLOSE);
2300 tcp_send_active_reset(sk, GFP_ATOMIC);
Eric Dumazet02a1d6e2016-04-27 16:44:39 -07002301 __NET_INC_STATS(sock_net(sk),
Pavel Emelyanovde0744a2008-07-16 20:31:16 -07002302 LINUX_MIB_TCPABORTONMEMORY);
Dan Streetman4ee806d2018-01-18 16:14:26 -05002303 } else if (!check_net(sock_net(sk))) {
2304 /* Not possible to send reset; just close */
2305 tcp_set_state(sk, TCP_CLOSE);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002306 }
2307 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07002308
Jerry Chu83368862012-08-31 12:29:12 +00002309 if (sk->sk_state == TCP_CLOSE) {
2310 struct request_sock *req = tcp_sk(sk)->fastopen_rsk;
2311 /* We could get here with a non-NULL req if the socket is
2312 * aborted (e.g., closed with unread data) before 3WHS
2313 * finishes.
2314 */
Ian Morris00db4122015-04-03 09:17:27 +01002315 if (req)
Jerry Chu83368862012-08-31 12:29:12 +00002316 reqsk_fastopen_remove(sk, req, false);
Arnaldo Carvalho de Melo0a5578c2005-08-09 20:11:41 -07002317 inet_csk_destroy_sock(sk);
Jerry Chu83368862012-08-31 12:29:12 +00002318 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07002319 /* Otherwise, socket is reprieved until protocol close. */
2320
2321out:
2322 bh_unlock_sock(sk);
2323 local_bh_enable();
2324 sock_put(sk);
2325}
Eric Dumazet4bc2f182010-07-09 21:22:10 +00002326EXPORT_SYMBOL(tcp_close);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002327
2328/* These states need RST on ABORT according to RFC793 */
2329
Eric Dumazeta2a385d2012-05-16 23:15:34 +00002330static inline bool tcp_need_reset(int state)
Linus Torvalds1da177e2005-04-16 15:20:36 -07002331{
2332 return (1 << state) &
2333 (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT | TCPF_FIN_WAIT1 |
2334 TCPF_FIN_WAIT2 | TCPF_SYN_RECV);
2335}
2336
Eric Dumazet75c119a2017-10-05 22:21:27 -07002337static void tcp_rtx_queue_purge(struct sock *sk)
2338{
2339 struct rb_node *p = rb_first(&sk->tcp_rtx_queue);
2340
2341 while (p) {
2342 struct sk_buff *skb = rb_to_skb(p);
2343
2344 p = rb_next(p);
2345 /* Since we are deleting whole queue, no need to
2346 * list_del(&skb->tcp_tsorted_anchor)
2347 */
2348 tcp_rtx_queue_unlink(skb, sk);
2349 sk_wmem_free_skb(sk, skb);
2350 }
2351}
2352
Eric Dumazetac3f09b2017-10-05 22:21:22 -07002353void tcp_write_queue_purge(struct sock *sk)
2354{
2355 struct sk_buff *skb;
2356
2357 tcp_chrono_stop(sk, TCP_CHRONO_BUSY);
2358 while ((skb = __skb_dequeue(&sk->sk_write_queue)) != NULL) {
2359 tcp_skb_tsorted_anchor_cleanup(skb);
2360 sk_wmem_free_skb(sk, skb);
2361 }
Eric Dumazet75c119a2017-10-05 22:21:27 -07002362 tcp_rtx_queue_purge(sk);
Eric Dumazetac3f09b2017-10-05 22:21:22 -07002363 INIT_LIST_HEAD(&tcp_sk(sk)->tsorted_sent_queue);
2364 sk_mem_reclaim(sk);
2365 tcp_clear_all_retrans_hints(tcp_sk(sk));
2366}
2367
Linus Torvalds1da177e2005-04-16 15:20:36 -07002368int tcp_disconnect(struct sock *sk, int flags)
2369{
2370 struct inet_sock *inet = inet_sk(sk);
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002371 struct inet_connection_sock *icsk = inet_csk(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002372 struct tcp_sock *tp = tcp_sk(sk);
2373 int err = 0;
2374 int old_state = sk->sk_state;
2375
2376 if (old_state != TCP_CLOSE)
2377 tcp_set_state(sk, TCP_CLOSE);
2378
2379 /* ABORT function of RFC793 */
2380 if (old_state == TCP_LISTEN) {
Arnaldo Carvalho de Melo0a5578c2005-08-09 20:11:41 -07002381 inet_csk_listen_stop(sk);
Pavel Emelyanovee995282012-04-19 03:40:39 +00002382 } else if (unlikely(tp->repair)) {
2383 sk->sk_err = ECONNABORTED;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002384 } else if (tcp_need_reset(old_state) ||
2385 (tp->snd_nxt != tp->write_seq &&
2386 (1 << old_state) & (TCPF_CLOSING | TCPF_LAST_ACK))) {
Stephen Hemmingercaa20d9a2005-11-10 17:13:47 -08002387 /* The last check adjusts for discrepancy of Linux wrt. RFC
Linus Torvalds1da177e2005-04-16 15:20:36 -07002388 * states
2389 */
2390 tcp_send_active_reset(sk, gfp_any());
2391 sk->sk_err = ECONNRESET;
2392 } else if (old_state == TCP_SYN_SENT)
2393 sk->sk_err = ECONNRESET;
2394
2395 tcp_clear_xmit_timers(sk);
2396 __skb_queue_purge(&sk->sk_receive_queue);
David S. Millerfe067e82007-03-07 12:12:44 -08002397 tcp_write_queue_purge(sk);
Wei Wangcf1ef3f2017-04-20 14:45:46 -07002398 tcp_fastopen_active_disable_ofo_check(sk);
Yaogong Wang9f5afea2016-09-07 14:49:28 -07002399 skb_rbtree_purge(&tp->out_of_order_queue);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002400
Eric Dumazetc720c7e82009-10-15 06:30:45 +00002401 inet->inet_dport = 0;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002402
2403 if (!(sk->sk_userlocks & SOCK_BINDADDR_LOCK))
2404 inet_reset_saddr(sk);
2405
2406 sk->sk_shutdown = 0;
2407 sock_reset_flag(sk, SOCK_DONE);
Eric Dumazet740b0f12014-02-26 14:02:48 -08002408 tp->srtt_us = 0;
Yuvaraja Mariappan686a5622015-10-06 10:53:29 -07002409 tp->write_seq += tp->max_window + 2;
2410 if (tp->write_seq == 0)
Linus Torvalds1da177e2005-04-16 15:20:36 -07002411 tp->write_seq = 1;
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002412 icsk->icsk_backoff = 0;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002413 tp->snd_cwnd = 2;
Arnaldo Carvalho de Melo6687e982005-08-10 04:03:31 -03002414 icsk->icsk_probes_out = 0;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002415 tp->packets_out = 0;
Ilpo Järvinen0b6a05c2009-09-15 01:30:10 -07002416 tp->snd_ssthresh = TCP_INFINITE_SSTHRESH;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002417 tp->snd_cwnd_cnt = 0;
Eric Dumazet1fdf4752009-11-30 12:53:30 -08002418 tp->window_clamp = 0;
Arnaldo Carvalho de Melo6687e982005-08-10 04:03:31 -03002419 tcp_set_ca_state(sk, TCP_CA_Open);
Yousuk Seungd4761752017-12-07 13:41:34 -08002420 tp->is_sack_reneg = 0;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002421 tcp_clear_retrans(tp);
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002422 inet_csk_delack_init(sk);
Wei Wang499350a2017-05-18 11:22:33 -07002423 /* Initialize rcv_mss to TCP_MIN_MSS to avoid division by 0
2424 * issue in __tcp_select_window()
2425 */
2426 icsk->icsk_ack.rcv_mss = TCP_MIN_MSS;
Srinivas Ajib40b4f72007-05-03 17:32:28 -07002427 memset(&tp->rx_opt, 0, sizeof(tp->rx_opt));
Linus Torvalds1da177e2005-04-16 15:20:36 -07002428 __sk_dst_reset(sk);
WANG Congd747a7a2017-06-24 23:50:30 -07002429 dst_release(sk->sk_rx_dst);
2430 sk->sk_rx_dst = NULL;
Eric Dumazet17c30602017-04-08 08:07:33 -07002431 tcp_saved_syn_free(tp);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002432
Wei Wang7db92362017-03-01 13:29:48 -08002433 /* Clean up fastopen related fields */
2434 tcp_free_fastopen_req(tp);
2435 inet->defer_connect = 0;
2436
Eric Dumazetc720c7e82009-10-15 06:30:45 +00002437 WARN_ON(inet->inet_num && !icsk->icsk_bind_hash);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002438
Li RongQing9b42d552018-01-26 16:40:41 +08002439 if (sk->sk_frag.page) {
2440 put_page(sk->sk_frag.page);
2441 sk->sk_frag.page = NULL;
2442 sk->sk_frag.offset = 0;
2443 }
2444
Linus Torvalds1da177e2005-04-16 15:20:36 -07002445 sk->sk_error_report(sk);
2446 return err;
2447}
Eric Dumazet4bc2f182010-07-09 21:22:10 +00002448EXPORT_SYMBOL(tcp_disconnect);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002449
Eric Dumazeta2a385d2012-05-16 23:15:34 +00002450static inline bool tcp_can_repair_sock(const struct sock *sk)
Pavel Emelyanovee995282012-04-19 03:40:39 +00002451{
Eric W. Biederman52e804c2012-11-16 03:03:05 +00002452 return ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN) &&
Andrey Vagin319b0532016-11-14 18:15:14 -08002453 (sk->sk_state != TCP_LISTEN);
Pavel Emelyanovee995282012-04-19 03:40:39 +00002454}
2455
Andrey Vaginb1ed4c42016-06-27 15:33:56 -07002456static int tcp_repair_set_window(struct tcp_sock *tp, char __user *optbuf, int len)
2457{
2458 struct tcp_repair_window opt;
2459
2460 if (!tp->repair)
2461 return -EPERM;
2462
2463 if (len != sizeof(opt))
2464 return -EINVAL;
2465
2466 if (copy_from_user(&opt, optbuf, sizeof(opt)))
2467 return -EFAULT;
2468
2469 if (opt.max_window < opt.snd_wnd)
2470 return -EINVAL;
2471
2472 if (after(opt.snd_wl1, tp->rcv_nxt + opt.rcv_wnd))
2473 return -EINVAL;
2474
2475 if (after(opt.rcv_wup, tp->rcv_nxt))
2476 return -EINVAL;
2477
2478 tp->snd_wl1 = opt.snd_wl1;
2479 tp->snd_wnd = opt.snd_wnd;
2480 tp->max_window = opt.max_window;
2481
2482 tp->rcv_wnd = opt.rcv_wnd;
2483 tp->rcv_wup = opt.rcv_wup;
2484
2485 return 0;
2486}
2487
Douglas Caetano dos Santos15e56512017-05-26 14:28:00 -03002488static int tcp_repair_options_est(struct sock *sk,
Pavel Emelyanovde248a72012-04-25 23:43:04 +00002489 struct tcp_repair_opt __user *optbuf, unsigned int len)
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002490{
Douglas Caetano dos Santos15e56512017-05-26 14:28:00 -03002491 struct tcp_sock *tp = tcp_sk(sk);
Pavel Emelyanovde248a72012-04-25 23:43:04 +00002492 struct tcp_repair_opt opt;
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002493
Pavel Emelyanovde248a72012-04-25 23:43:04 +00002494 while (len >= sizeof(opt)) {
2495 if (copy_from_user(&opt, optbuf, sizeof(opt)))
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002496 return -EFAULT;
2497
2498 optbuf++;
Pavel Emelyanovde248a72012-04-25 23:43:04 +00002499 len -= sizeof(opt);
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002500
Pavel Emelyanovde248a72012-04-25 23:43:04 +00002501 switch (opt.opt_code) {
2502 case TCPOPT_MSS:
2503 tp->rx_opt.mss_clamp = opt.opt_val;
Douglas Caetano dos Santos15e56512017-05-26 14:28:00 -03002504 tcp_mtup_init(sk);
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002505 break;
Pavel Emelyanovde248a72012-04-25 23:43:04 +00002506 case TCPOPT_WINDOW:
Andrey Vaginbc26ccd2012-09-19 09:40:00 +00002507 {
2508 u16 snd_wscale = opt.opt_val & 0xFFFF;
2509 u16 rcv_wscale = opt.opt_val >> 16;
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002510
Gao Feng589c49c2017-04-04 21:09:48 +08002511 if (snd_wscale > TCP_MAX_WSCALE || rcv_wscale > TCP_MAX_WSCALE)
Andrey Vaginbc26ccd2012-09-19 09:40:00 +00002512 return -EFBIG;
2513
2514 tp->rx_opt.snd_wscale = snd_wscale;
2515 tp->rx_opt.rcv_wscale = rcv_wscale;
2516 tp->rx_opt.wscale_ok = 1;
2517 }
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002518 break;
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002519 case TCPOPT_SACK_PERM:
Pavel Emelyanovde248a72012-04-25 23:43:04 +00002520 if (opt.opt_val != 0)
2521 return -EINVAL;
2522
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002523 tp->rx_opt.sack_ok |= TCP_SACK_SEEN;
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002524 break;
2525 case TCPOPT_TIMESTAMP:
Pavel Emelyanovde248a72012-04-25 23:43:04 +00002526 if (opt.opt_val != 0)
2527 return -EINVAL;
2528
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002529 tp->rx_opt.tstamp_ok = 1;
2530 break;
2531 }
2532 }
2533
2534 return 0;
2535}
2536
Linus Torvalds1da177e2005-04-16 15:20:36 -07002537/*
Linus Torvalds1da177e2005-04-16 15:20:36 -07002538 * Socket option code for TCP.
2539 */
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08002540static int do_tcp_setsockopt(struct sock *sk, int level,
David S. Millerb7058842009-09-30 16:12:20 -07002541 int optname, char __user *optval, unsigned int optlen)
Linus Torvalds1da177e2005-04-16 15:20:36 -07002542{
2543 struct tcp_sock *tp = tcp_sk(sk);
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002544 struct inet_connection_sock *icsk = inet_csk(sk);
Nikolay Borisov1e579ca2016-02-03 09:46:56 +02002545 struct net *net = sock_net(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002546 int val;
2547 int err = 0;
2548
William Allen Simpsone56fb502009-12-02 18:19:30 +00002549 /* These are data/string values, all the others are ints */
2550 switch (optname) {
2551 case TCP_CONGESTION: {
Stephen Hemminger5f8ef482005-06-23 20:37:36 -07002552 char name[TCP_CA_NAME_MAX];
2553
2554 if (optlen < 1)
2555 return -EINVAL;
2556
2557 val = strncpy_from_user(name, optval,
Andrew Morton4fdb78d2009-10-01 15:02:20 -07002558 min_t(long, TCP_CA_NAME_MAX-1, optlen));
Stephen Hemminger5f8ef482005-06-23 20:37:36 -07002559 if (val < 0)
2560 return -EFAULT;
2561 name[val] = 0;
2562
2563 lock_sock(sk);
Sabrina Dubrocaebfa00c2017-08-25 13:10:12 +02002564 err = tcp_set_congestion_control(sk, name, true, true);
Stephen Hemminger5f8ef482005-06-23 20:37:36 -07002565 release_sock(sk);
2566 return err;
2567 }
Dave Watson734942c2017-06-14 11:37:14 -07002568 case TCP_ULP: {
2569 char name[TCP_ULP_NAME_MAX];
2570
2571 if (optlen < 1)
2572 return -EINVAL;
2573
2574 val = strncpy_from_user(name, optval,
2575 min_t(long, TCP_ULP_NAME_MAX - 1,
2576 optlen));
2577 if (val < 0)
2578 return -EFAULT;
2579 name[val] = 0;
2580
2581 lock_sock(sk);
2582 err = tcp_set_ulp(sk, name);
2583 release_sock(sk);
2584 return err;
2585 }
Yuchung Cheng1fba70e2017-10-18 11:22:51 -07002586 case TCP_FASTOPEN_KEY: {
2587 __u8 key[TCP_FASTOPEN_KEY_LENGTH];
2588
2589 if (optlen != sizeof(key))
2590 return -EINVAL;
2591
2592 if (copy_from_user(key, optval, optlen))
2593 return -EFAULT;
2594
2595 return tcp_fastopen_reset_cipher(net, sk, key, sizeof(key));
2596 }
William Allen Simpsone56fb502009-12-02 18:19:30 +00002597 default:
2598 /* fallthru */
2599 break;
Joe Perchesccbd6a52010-05-14 10:58:26 +00002600 }
Stephen Hemminger5f8ef482005-06-23 20:37:36 -07002601
Linus Torvalds1da177e2005-04-16 15:20:36 -07002602 if (optlen < sizeof(int))
2603 return -EINVAL;
2604
2605 if (get_user(val, (int __user *)optval))
2606 return -EFAULT;
2607
2608 lock_sock(sk);
2609
2610 switch (optname) {
2611 case TCP_MAXSEG:
2612 /* Values greater than interface MTU won't take effect. However
2613 * at the point when this call is done we typically don't yet
Rohit Chavana777f712017-05-22 11:59:15 +05302614 * know which interface is going to be used
2615 */
Gao Fengcfc62d82017-03-21 09:28:03 +08002616 if (val && (val < TCP_MIN_MSS || val > MAX_TCP_WINDOW)) {
Linus Torvalds1da177e2005-04-16 15:20:36 -07002617 err = -EINVAL;
2618 break;
2619 }
2620 tp->rx_opt.user_mss = val;
2621 break;
2622
2623 case TCP_NODELAY:
2624 if (val) {
2625 /* TCP_NODELAY is weaker than TCP_CORK, so that
2626 * this option on corked socket is remembered, but
2627 * it is not activated until cork is cleared.
2628 *
2629 * However, when TCP_NODELAY is set we make
2630 * an explicit push, which overrides even TCP_CORK
2631 * for currently queued segments.
2632 */
2633 tp->nonagle |= TCP_NAGLE_OFF|TCP_NAGLE_PUSH;
Ilpo Järvinen9e412ba2007-04-20 22:18:02 -07002634 tcp_push_pending_frames(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002635 } else {
2636 tp->nonagle &= ~TCP_NAGLE_OFF;
2637 }
2638 break;
2639
Andreas Petlund36e31b0a2010-02-18 02:47:01 +00002640 case TCP_THIN_LINEAR_TIMEOUTS:
2641 if (val < 0 || val > 1)
2642 err = -EINVAL;
2643 else
2644 tp->thin_lto = val;
2645 break;
2646
Andreas Petlund7e380172010-02-18 04:48:19 +00002647 case TCP_THIN_DUPACK:
2648 if (val < 0 || val > 1)
2649 err = -EINVAL;
Andreas Petlund7e380172010-02-18 04:48:19 +00002650 break;
2651
Pavel Emelyanovee995282012-04-19 03:40:39 +00002652 case TCP_REPAIR:
2653 if (!tcp_can_repair_sock(sk))
2654 err = -EPERM;
2655 else if (val == 1) {
2656 tp->repair = 1;
2657 sk->sk_reuse = SK_FORCE_REUSE;
2658 tp->repair_queue = TCP_NO_QUEUE;
2659 } else if (val == 0) {
2660 tp->repair = 0;
2661 sk->sk_reuse = SK_NO_REUSE;
2662 tcp_send_window_probe(sk);
2663 } else
2664 err = -EINVAL;
2665
2666 break;
2667
2668 case TCP_REPAIR_QUEUE:
2669 if (!tp->repair)
2670 err = -EPERM;
2671 else if (val < TCP_QUEUES_NR)
2672 tp->repair_queue = val;
2673 else
2674 err = -EINVAL;
2675 break;
2676
2677 case TCP_QUEUE_SEQ:
2678 if (sk->sk_state != TCP_CLOSE)
2679 err = -EPERM;
2680 else if (tp->repair_queue == TCP_SEND_QUEUE)
2681 tp->write_seq = val;
2682 else if (tp->repair_queue == TCP_RECV_QUEUE)
2683 tp->rcv_nxt = val;
2684 else
2685 err = -EINVAL;
2686 break;
2687
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002688 case TCP_REPAIR_OPTIONS:
2689 if (!tp->repair)
2690 err = -EINVAL;
2691 else if (sk->sk_state == TCP_ESTABLISHED)
Douglas Caetano dos Santos15e56512017-05-26 14:28:00 -03002692 err = tcp_repair_options_est(sk,
Pavel Emelyanovde248a72012-04-25 23:43:04 +00002693 (struct tcp_repair_opt __user *)optval,
2694 optlen);
Pavel Emelyanovb139ba42012-04-19 03:41:57 +00002695 else
2696 err = -EPERM;
2697 break;
2698
Linus Torvalds1da177e2005-04-16 15:20:36 -07002699 case TCP_CORK:
2700 /* When set indicates to always queue non-full frames.
2701 * Later the user clears this option and we transmit
2702 * any pending partial frames in the queue. This is
2703 * meant to be used alongside sendfile() to get properly
2704 * filled frames when the user (for example) must write
2705 * out headers with a write() call first and then use
2706 * sendfile to send out the data parts.
2707 *
2708 * TCP_CORK can be set together with TCP_NODELAY and it is
2709 * stronger than TCP_NODELAY.
2710 */
2711 if (val) {
2712 tp->nonagle |= TCP_NAGLE_CORK;
2713 } else {
2714 tp->nonagle &= ~TCP_NAGLE_CORK;
2715 if (tp->nonagle&TCP_NAGLE_OFF)
2716 tp->nonagle |= TCP_NAGLE_PUSH;
Ilpo Järvinen9e412ba2007-04-20 22:18:02 -07002717 tcp_push_pending_frames(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002718 }
2719 break;
2720
2721 case TCP_KEEPIDLE:
2722 if (val < 1 || val > MAX_TCP_KEEPIDLE)
2723 err = -EINVAL;
2724 else {
2725 tp->keepalive_time = val * HZ;
2726 if (sock_flag(sk, SOCK_KEEPOPEN) &&
2727 !((1 << sk->sk_state) &
2728 (TCPF_CLOSE | TCPF_LISTEN))) {
Flavio Leitner6c37e5d2010-04-26 18:33:27 +00002729 u32 elapsed = keepalive_time_elapsed(tp);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002730 if (tp->keepalive_time > elapsed)
2731 elapsed = tp->keepalive_time - elapsed;
2732 else
2733 elapsed = 0;
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002734 inet_csk_reset_keepalive_timer(sk, elapsed);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002735 }
2736 }
2737 break;
2738 case TCP_KEEPINTVL:
2739 if (val < 1 || val > MAX_TCP_KEEPINTVL)
2740 err = -EINVAL;
2741 else
2742 tp->keepalive_intvl = val * HZ;
2743 break;
2744 case TCP_KEEPCNT:
2745 if (val < 1 || val > MAX_TCP_KEEPCNT)
2746 err = -EINVAL;
2747 else
2748 tp->keepalive_probes = val;
2749 break;
2750 case TCP_SYNCNT:
2751 if (val < 1 || val > MAX_TCP_SYNCNT)
2752 err = -EINVAL;
2753 else
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002754 icsk->icsk_syn_retries = val;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002755 break;
2756
Eric Dumazetcd8ae852015-05-03 21:34:46 -07002757 case TCP_SAVE_SYN:
2758 if (val < 0 || val > 1)
2759 err = -EINVAL;
2760 else
2761 tp->save_syn = val;
2762 break;
2763
Linus Torvalds1da177e2005-04-16 15:20:36 -07002764 case TCP_LINGER2:
2765 if (val < 0)
2766 tp->linger2 = -1;
Nikolay Borisov1e579ca2016-02-03 09:46:56 +02002767 else if (val > net->ipv4.sysctl_tcp_fin_timeout / HZ)
Linus Torvalds1da177e2005-04-16 15:20:36 -07002768 tp->linger2 = 0;
2769 else
2770 tp->linger2 = val * HZ;
2771 break;
2772
2773 case TCP_DEFER_ACCEPT:
Julian Anastasovb103cf32009-10-19 10:10:40 +00002774 /* Translate value in seconds to number of retransmits */
2775 icsk->icsk_accept_queue.rskq_defer_accept =
2776 secs_to_retrans(val, TCP_TIMEOUT_INIT / HZ,
2777 TCP_RTO_MAX / HZ);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002778 break;
2779
2780 case TCP_WINDOW_CLAMP:
2781 if (!val) {
2782 if (sk->sk_state != TCP_CLOSE) {
2783 err = -EINVAL;
2784 break;
2785 }
2786 tp->window_clamp = 0;
2787 } else
2788 tp->window_clamp = val < SOCK_MIN_RCVBUF / 2 ?
2789 SOCK_MIN_RCVBUF / 2 : val;
2790 break;
2791
2792 case TCP_QUICKACK:
2793 if (!val) {
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002794 icsk->icsk_ack.pingpong = 1;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002795 } else {
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002796 icsk->icsk_ack.pingpong = 0;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002797 if ((1 << sk->sk_state) &
2798 (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT) &&
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002799 inet_csk_ack_scheduled(sk)) {
2800 icsk->icsk_ack.pending |= ICSK_ACK_PUSHED;
Chris Leech0e4b4992006-05-23 18:00:16 -07002801 tcp_cleanup_rbuf(sk, 1);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002802 if (!(val & 1))
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002803 icsk->icsk_ack.pingpong = 1;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002804 }
2805 }
2806 break;
2807
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08002808#ifdef CONFIG_TCP_MD5SIG
2809 case TCP_MD5SIG:
Ivan Delalande8917a772017-06-15 18:07:07 -07002810 case TCP_MD5SIG_EXT:
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08002811 /* Read the IP->Key mappings from userspace */
Ivan Delalande8917a772017-06-15 18:07:07 -07002812 err = tp->af_specific->md5_parse(sk, optname, optval, optlen);
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08002813 break;
2814#endif
Jerry Chudca43c72010-08-27 19:13:28 +00002815 case TCP_USER_TIMEOUT:
Yuchung Chengb2482302014-09-29 13:20:38 -07002816 /* Cap the max time in ms TCP will retry or probe the window
Jerry Chudca43c72010-08-27 19:13:28 +00002817 * before giving up and aborting (ETIMEDOUT) a connection.
2818 */
Hangbin Liu42493572012-07-26 22:52:21 +00002819 if (val < 0)
2820 err = -EINVAL;
2821 else
2822 icsk->icsk_user_timeout = msecs_to_jiffies(val);
Jerry Chudca43c72010-08-27 19:13:28 +00002823 break;
Jerry Chu83368862012-08-31 12:29:12 +00002824
2825 case TCP_FASTOPEN:
2826 if (val >= 0 && ((1 << sk->sk_state) & (TCPF_CLOSE |
Christoph Paaschdfea2aa2015-06-18 09:15:34 -07002827 TCPF_LISTEN))) {
Haishuang Yan43713842017-09-27 11:35:42 +08002828 tcp_fastopen_init_key_once(net);
Christoph Paaschdfea2aa2015-06-18 09:15:34 -07002829
Eric Dumazet0536fcc2015-09-29 07:42:52 -07002830 fastopen_queue_tune(sk, val);
Christoph Paaschdfea2aa2015-06-18 09:15:34 -07002831 } else {
Jerry Chu83368862012-08-31 12:29:12 +00002832 err = -EINVAL;
Christoph Paaschdfea2aa2015-06-18 09:15:34 -07002833 }
Jerry Chu83368862012-08-31 12:29:12 +00002834 break;
Wei Wang19f6d3f2017-01-23 10:59:22 -08002835 case TCP_FASTOPEN_CONNECT:
2836 if (val > 1 || val < 0) {
2837 err = -EINVAL;
Haishuang Yane1cfcbe2017-09-27 11:35:40 +08002838 } else if (net->ipv4.sysctl_tcp_fastopen & TFO_CLIENT_ENABLE) {
Wei Wang19f6d3f2017-01-23 10:59:22 -08002839 if (sk->sk_state == TCP_CLOSE)
2840 tp->fastopen_connect = val;
2841 else
2842 err = -EINVAL;
2843 } else {
2844 err = -EOPNOTSUPP;
2845 }
2846 break;
Christoph Paasch71c02372017-10-23 13:22:23 -07002847 case TCP_FASTOPEN_NO_COOKIE:
2848 if (val > 1 || val < 0)
2849 err = -EINVAL;
2850 else if (!((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN)))
2851 err = -EINVAL;
2852 else
2853 tp->fastopen_no_cookie = val;
2854 break;
Andrey Vagin93be6ce2013-02-11 05:50:18 +00002855 case TCP_TIMESTAMP:
2856 if (!tp->repair)
2857 err = -EPERM;
2858 else
Eric Dumazet9a568de2017-05-16 14:00:14 -07002859 tp->tsoffset = val - tcp_time_stamp_raw();
Andrey Vagin93be6ce2013-02-11 05:50:18 +00002860 break;
Andrey Vaginb1ed4c42016-06-27 15:33:56 -07002861 case TCP_REPAIR_WINDOW:
2862 err = tcp_repair_set_window(tp, optval, optlen);
2863 break;
Eric Dumazetc9bee3b72013-07-22 20:27:07 -07002864 case TCP_NOTSENT_LOWAT:
2865 tp->notsent_lowat = val;
2866 sk->sk_write_space(sk);
2867 break;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002868 default:
2869 err = -ENOPROTOOPT;
2870 break;
Stephen Hemminger3ff50b72007-04-20 17:09:22 -07002871 }
2872
Linus Torvalds1da177e2005-04-16 15:20:36 -07002873 release_sock(sk);
2874 return err;
2875}
2876
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08002877int tcp_setsockopt(struct sock *sk, int level, int optname, char __user *optval,
David S. Millerb7058842009-09-30 16:12:20 -07002878 unsigned int optlen)
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08002879{
Eric Dumazetcf533ea2011-10-21 05:22:42 -04002880 const struct inet_connection_sock *icsk = inet_csk(sk);
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08002881
2882 if (level != SOL_TCP)
2883 return icsk->icsk_af_ops->setsockopt(sk, level, optname,
2884 optval, optlen);
2885 return do_tcp_setsockopt(sk, level, optname, optval, optlen);
2886}
Eric Dumazet4bc2f182010-07-09 21:22:10 +00002887EXPORT_SYMBOL(tcp_setsockopt);
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08002888
2889#ifdef CONFIG_COMPAT
Arnaldo Carvalho de Melo543d9cf2006-03-20 22:48:35 -08002890int compat_tcp_setsockopt(struct sock *sk, int level, int optname,
David S. Millerb7058842009-09-30 16:12:20 -07002891 char __user *optval, unsigned int optlen)
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08002892{
Arnaldo Carvalho de Melodec73ff2006-03-20 22:46:16 -08002893 if (level != SOL_TCP)
2894 return inet_csk_compat_setsockopt(sk, level, optname,
2895 optval, optlen);
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08002896 return do_tcp_setsockopt(sk, level, optname, optval, optlen);
2897}
Arnaldo Carvalho de Melo543d9cf2006-03-20 22:48:35 -08002898EXPORT_SYMBOL(compat_tcp_setsockopt);
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08002899#endif
2900
Francis Yanefd90172016-11-27 23:07:17 -08002901static void tcp_get_info_chrono_stats(const struct tcp_sock *tp,
2902 struct tcp_info *info)
2903{
2904 u64 stats[__TCP_CHRONO_MAX], total = 0;
2905 enum tcp_chrono i;
2906
2907 for (i = TCP_CHRONO_BUSY; i < __TCP_CHRONO_MAX; ++i) {
2908 stats[i] = tp->chrono_stat[i - 1];
2909 if (i == tp->chrono_type)
Eric Dumazet628174c2017-05-16 14:00:09 -07002910 stats[i] += tcp_jiffies32 - tp->chrono_start;
Francis Yanefd90172016-11-27 23:07:17 -08002911 stats[i] *= USEC_PER_SEC / HZ;
2912 total += stats[i];
2913 }
2914
2915 info->tcpi_busy_time = total;
2916 info->tcpi_rwnd_limited = stats[TCP_CHRONO_RWND_LIMITED];
2917 info->tcpi_sndbuf_limited = stats[TCP_CHRONO_SNDBUF_LIMITED];
2918}
2919
Linus Torvalds1da177e2005-04-16 15:20:36 -07002920/* Return information about state of tcp endpoint in API format. */
Eric Dumazet0df48c22015-04-28 15:28:17 -07002921void tcp_get_info(struct sock *sk, struct tcp_info *info)
Linus Torvalds1da177e2005-04-16 15:20:36 -07002922{
Craig Gallek35ac8382015-06-15 11:26:20 -04002923 const struct tcp_sock *tp = tcp_sk(sk); /* iff sk_type == SOCK_STREAM */
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002924 const struct inet_connection_sock *icsk = inet_csk(sk);
Wei Wang02635982017-07-28 10:28:20 -07002925 u32 now;
Eric Dumazetff5d7492016-01-27 10:52:43 -08002926 u64 rate64;
Eric Dumazet67db3e42016-11-04 11:54:32 -07002927 bool slow;
Eric Dumazetfad9dfe2015-04-16 16:12:28 -07002928 u32 rate;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002929
2930 memset(info, 0, sizeof(*info));
Craig Gallek35ac8382015-06-15 11:26:20 -04002931 if (sk->sk_type != SOCK_STREAM)
2932 return;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002933
Yafang Shao986ffdf2017-12-20 11:12:52 +08002934 info->tcpi_state = inet_sk_state_load(sk);
Eric Dumazet00fd38d2015-11-12 08:43:18 -08002935
Eric Dumazetccbf3bf2016-11-04 11:54:31 -07002936 /* Report meaningful fields for all TCP states, including listeners */
2937 rate = READ_ONCE(sk->sk_pacing_rate);
2938 rate64 = rate != ~0U ? rate : ~0ULL;
Eric Dumazetf522a5f2016-11-09 11:24:22 -08002939 info->tcpi_pacing_rate = rate64;
Eric Dumazetccbf3bf2016-11-04 11:54:31 -07002940
2941 rate = READ_ONCE(sk->sk_max_pacing_rate);
2942 rate64 = rate != ~0U ? rate : ~0ULL;
Eric Dumazetf522a5f2016-11-09 11:24:22 -08002943 info->tcpi_max_pacing_rate = rate64;
Eric Dumazetccbf3bf2016-11-04 11:54:31 -07002944
2945 info->tcpi_reordering = tp->reordering;
2946 info->tcpi_snd_cwnd = tp->snd_cwnd;
2947
2948 if (info->tcpi_state == TCP_LISTEN) {
2949 /* listeners aliased fields :
2950 * tcpi_unacked -> Number of children ready for accept()
2951 * tcpi_sacked -> max backlog
2952 */
2953 info->tcpi_unacked = sk->sk_ack_backlog;
2954 info->tcpi_sacked = sk->sk_max_ack_backlog;
2955 return;
2956 }
Eric Dumazetb369e7f2017-01-09 10:29:27 -08002957
2958 slow = lock_sock_fast(sk);
2959
Arnaldo Carvalho de Melo6687e982005-08-10 04:03:31 -03002960 info->tcpi_ca_state = icsk->icsk_ca_state;
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002961 info->tcpi_retransmits = icsk->icsk_retransmits;
Arnaldo Carvalho de Melo6687e982005-08-10 04:03:31 -03002962 info->tcpi_probes = icsk->icsk_probes_out;
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002963 info->tcpi_backoff = icsk->icsk_backoff;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002964
2965 if (tp->rx_opt.tstamp_ok)
2966 info->tcpi_options |= TCPI_OPT_TIMESTAMPS;
Ilpo Järvinene60402d2007-08-09 15:14:46 +03002967 if (tcp_is_sack(tp))
Linus Torvalds1da177e2005-04-16 15:20:36 -07002968 info->tcpi_options |= TCPI_OPT_SACK;
2969 if (tp->rx_opt.wscale_ok) {
2970 info->tcpi_options |= TCPI_OPT_WSCALE;
2971 info->tcpi_snd_wscale = tp->rx_opt.snd_wscale;
2972 info->tcpi_rcv_wscale = tp->rx_opt.rcv_wscale;
YOSHIFUJI Hideakie905a9e2007-02-09 23:24:47 +09002973 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07002974
Eric Dumazetb5c56932011-10-03 14:01:21 -04002975 if (tp->ecn_flags & TCP_ECN_OK)
Linus Torvalds1da177e2005-04-16 15:20:36 -07002976 info->tcpi_options |= TCPI_OPT_ECN;
Eric Dumazetb5c56932011-10-03 14:01:21 -04002977 if (tp->ecn_flags & TCP_ECN_SEEN)
2978 info->tcpi_options |= TCPI_OPT_ECN_SEEN;
Yuchung Cheng6f736012012-10-19 15:14:44 +00002979 if (tp->syn_data_acked)
2980 info->tcpi_options |= TCPI_OPT_SYN_DATA;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002981
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002982 info->tcpi_rto = jiffies_to_usecs(icsk->icsk_rto);
2983 info->tcpi_ato = jiffies_to_usecs(icsk->icsk_ack.ato);
David S. Millerc1b4a7e2005-07-05 15:24:38 -07002984 info->tcpi_snd_mss = tp->mss_cache;
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002985 info->tcpi_rcv_mss = icsk->icsk_ack.rcv_mss;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002986
Eric Dumazetccbf3bf2016-11-04 11:54:31 -07002987 info->tcpi_unacked = tp->packets_out;
2988 info->tcpi_sacked = tp->sacked_out;
2989
Linus Torvalds1da177e2005-04-16 15:20:36 -07002990 info->tcpi_lost = tp->lost_out;
2991 info->tcpi_retrans = tp->retrans_out;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002992
Eric Dumazetd635fbe2017-05-16 14:00:03 -07002993 now = tcp_jiffies32;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002994 info->tcpi_last_data_sent = jiffies_to_msecs(now - tp->lsndtime);
Arnaldo Carvalho de Melo463c84b2005-08-09 20:10:42 -07002995 info->tcpi_last_data_recv = jiffies_to_msecs(now - icsk->icsk_ack.lrcvtime);
Linus Torvalds1da177e2005-04-16 15:20:36 -07002996 info->tcpi_last_ack_recv = jiffies_to_msecs(now - tp->rcv_tstamp);
2997
Arnaldo Carvalho de Melod83d8462005-12-13 23:26:10 -08002998 info->tcpi_pmtu = icsk->icsk_pmtu_cookie;
Linus Torvalds1da177e2005-04-16 15:20:36 -07002999 info->tcpi_rcv_ssthresh = tp->rcv_ssthresh;
Eric Dumazet740b0f12014-02-26 14:02:48 -08003000 info->tcpi_rtt = tp->srtt_us >> 3;
3001 info->tcpi_rttvar = tp->mdev_us >> 2;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003002 info->tcpi_snd_ssthresh = tp->snd_ssthresh;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003003 info->tcpi_advmss = tp->advmss;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003004
Eric Dumazet645f4c62017-04-25 10:15:41 -07003005 info->tcpi_rcv_rtt = tp->rcv_rtt_est.rtt_us >> 3;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003006 info->tcpi_rcv_space = tp->rcvq_space.space;
3007
3008 info->tcpi_total_retrans = tp->total_retrans;
Eric Dumazet977cb0e2014-02-13 14:27:40 -08003009
Eric Dumazetf522a5f2016-11-09 11:24:22 -08003010 info->tcpi_bytes_acked = tp->bytes_acked;
3011 info->tcpi_bytes_received = tp->bytes_received;
Eric Dumazet67db3e42016-11-04 11:54:32 -07003012 info->tcpi_notsent_bytes = max_t(int, 0, tp->write_seq - tp->snd_nxt);
Francis Yanefd90172016-11-27 23:07:17 -08003013 tcp_get_info_chrono_stats(tp, info);
Eric Dumazet67db3e42016-11-04 11:54:32 -07003014
Marcelo Ricardo Leitner2efd0552015-05-20 16:35:41 -07003015 info->tcpi_segs_out = tp->segs_out;
3016 info->tcpi_segs_in = tp->segs_in;
Eric Dumazetcd9b2662016-02-11 22:02:53 -08003017
Eric Dumazetcd9b2662016-02-11 22:02:53 -08003018 info->tcpi_min_rtt = tcp_min_rtt(tp);
Martin KaFai Laua44d6ea2016-03-14 10:52:15 -07003019 info->tcpi_data_segs_in = tp->data_segs_in;
3020 info->tcpi_data_segs_out = tp->data_segs_out;
Yuchung Chengeb8329e2016-09-19 23:39:16 -04003021
3022 info->tcpi_delivery_rate_app_limited = tp->rate_app_limited ? 1 : 0;
Wei Wang02635982017-07-28 10:28:20 -07003023 rate64 = tcp_compute_delivery_rate(tp);
3024 if (rate64)
Eric Dumazetf522a5f2016-11-09 11:24:22 -08003025 info->tcpi_delivery_rate = rate64;
Eric Dumazetb369e7f2017-01-09 10:29:27 -08003026 unlock_sock_fast(sk, slow);
Linus Torvalds1da177e2005-04-16 15:20:36 -07003027}
Linus Torvalds1da177e2005-04-16 15:20:36 -07003028EXPORT_SYMBOL_GPL(tcp_get_info);
3029
Francis Yan1c885802016-11-27 23:07:18 -08003030struct sk_buff *tcp_get_timestamping_opt_stats(const struct sock *sk)
3031{
3032 const struct tcp_sock *tp = tcp_sk(sk);
3033 struct sk_buff *stats;
3034 struct tcp_info info;
Wei Wangbb7c19f2017-07-28 10:28:21 -07003035 u64 rate64;
3036 u32 rate;
Francis Yan1c885802016-11-27 23:07:18 -08003037
Wei Wangbb7c19f2017-07-28 10:28:21 -07003038 stats = alloc_skb(7 * nla_total_size_64bit(sizeof(u64)) +
3039 3 * nla_total_size(sizeof(u32)) +
3040 2 * nla_total_size(sizeof(u8)), GFP_ATOMIC);
Francis Yan1c885802016-11-27 23:07:18 -08003041 if (!stats)
3042 return NULL;
3043
3044 tcp_get_info_chrono_stats(tp, &info);
3045 nla_put_u64_64bit(stats, TCP_NLA_BUSY,
3046 info.tcpi_busy_time, TCP_NLA_PAD);
3047 nla_put_u64_64bit(stats, TCP_NLA_RWND_LIMITED,
3048 info.tcpi_rwnd_limited, TCP_NLA_PAD);
3049 nla_put_u64_64bit(stats, TCP_NLA_SNDBUF_LIMITED,
3050 info.tcpi_sndbuf_limited, TCP_NLA_PAD);
Yuchung Cheng7e981022017-01-27 16:24:38 -08003051 nla_put_u64_64bit(stats, TCP_NLA_DATA_SEGS_OUT,
3052 tp->data_segs_out, TCP_NLA_PAD);
3053 nla_put_u64_64bit(stats, TCP_NLA_TOTAL_RETRANS,
3054 tp->total_retrans, TCP_NLA_PAD);
Wei Wangbb7c19f2017-07-28 10:28:21 -07003055
3056 rate = READ_ONCE(sk->sk_pacing_rate);
3057 rate64 = rate != ~0U ? rate : ~0ULL;
3058 nla_put_u64_64bit(stats, TCP_NLA_PACING_RATE, rate64, TCP_NLA_PAD);
3059
3060 rate64 = tcp_compute_delivery_rate(tp);
3061 nla_put_u64_64bit(stats, TCP_NLA_DELIVERY_RATE, rate64, TCP_NLA_PAD);
3062
3063 nla_put_u32(stats, TCP_NLA_SND_CWND, tp->snd_cwnd);
3064 nla_put_u32(stats, TCP_NLA_REORDERING, tp->reordering);
3065 nla_put_u32(stats, TCP_NLA_MIN_RTT, tcp_min_rtt(tp));
3066
3067 nla_put_u8(stats, TCP_NLA_RECUR_RETRANS, inet_csk(sk)->icsk_retransmits);
3068 nla_put_u8(stats, TCP_NLA_DELIVERY_RATE_APP_LMT, !!tp->rate_app_limited);
Francis Yan1c885802016-11-27 23:07:18 -08003069 return stats;
3070}
3071
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08003072static int do_tcp_getsockopt(struct sock *sk, int level,
3073 int optname, char __user *optval, int __user *optlen)
Linus Torvalds1da177e2005-04-16 15:20:36 -07003074{
Arnaldo Carvalho de Melo295f7322005-08-09 20:11:56 -07003075 struct inet_connection_sock *icsk = inet_csk(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -07003076 struct tcp_sock *tp = tcp_sk(sk);
Nikolay Borisov6fa25162016-02-03 09:46:49 +02003077 struct net *net = sock_net(sk);
Linus Torvalds1da177e2005-04-16 15:20:36 -07003078 int val, len;
3079
Linus Torvalds1da177e2005-04-16 15:20:36 -07003080 if (get_user(len, optlen))
3081 return -EFAULT;
3082
3083 len = min_t(unsigned int, len, sizeof(int));
3084
3085 if (len < 0)
3086 return -EINVAL;
3087
3088 switch (optname) {
3089 case TCP_MAXSEG:
David S. Millerc1b4a7e2005-07-05 15:24:38 -07003090 val = tp->mss_cache;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003091 if (!val && ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN)))
3092 val = tp->rx_opt.user_mss;
Pavel Emelyanov5e6a3ce2012-04-19 03:41:32 +00003093 if (tp->repair)
3094 val = tp->rx_opt.mss_clamp;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003095 break;
3096 case TCP_NODELAY:
3097 val = !!(tp->nonagle&TCP_NAGLE_OFF);
3098 break;
3099 case TCP_CORK:
3100 val = !!(tp->nonagle&TCP_NAGLE_CORK);
3101 break;
3102 case TCP_KEEPIDLE:
Eric Dumazetdf19a622009-08-28 23:48:54 -07003103 val = keepalive_time_when(tp) / HZ;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003104 break;
3105 case TCP_KEEPINTVL:
Eric Dumazetdf19a622009-08-28 23:48:54 -07003106 val = keepalive_intvl_when(tp) / HZ;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003107 break;
3108 case TCP_KEEPCNT:
Eric Dumazetdf19a622009-08-28 23:48:54 -07003109 val = keepalive_probes(tp);
Linus Torvalds1da177e2005-04-16 15:20:36 -07003110 break;
3111 case TCP_SYNCNT:
Nikolay Borisov6fa25162016-02-03 09:46:49 +02003112 val = icsk->icsk_syn_retries ? : net->ipv4.sysctl_tcp_syn_retries;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003113 break;
3114 case TCP_LINGER2:
3115 val = tp->linger2;
3116 if (val >= 0)
Nikolay Borisov1e579ca2016-02-03 09:46:56 +02003117 val = (val ? : net->ipv4.sysctl_tcp_fin_timeout) / HZ;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003118 break;
3119 case TCP_DEFER_ACCEPT:
Julian Anastasovb103cf32009-10-19 10:10:40 +00003120 val = retrans_to_secs(icsk->icsk_accept_queue.rskq_defer_accept,
3121 TCP_TIMEOUT_INIT / HZ, TCP_RTO_MAX / HZ);
Linus Torvalds1da177e2005-04-16 15:20:36 -07003122 break;
3123 case TCP_WINDOW_CLAMP:
3124 val = tp->window_clamp;
3125 break;
3126 case TCP_INFO: {
3127 struct tcp_info info;
3128
3129 if (get_user(len, optlen))
3130 return -EFAULT;
3131
3132 tcp_get_info(sk, &info);
3133
3134 len = min_t(unsigned int, len, sizeof(info));
3135 if (put_user(len, optlen))
3136 return -EFAULT;
3137 if (copy_to_user(optval, &info, len))
3138 return -EFAULT;
3139 return 0;
3140 }
Eric Dumazet6e9250f2015-04-28 16:23:49 -07003141 case TCP_CC_INFO: {
3142 const struct tcp_congestion_ops *ca_ops;
3143 union tcp_cc_info info;
3144 size_t sz = 0;
3145 int attr;
3146
3147 if (get_user(len, optlen))
3148 return -EFAULT;
3149
3150 ca_ops = icsk->icsk_ca_ops;
3151 if (ca_ops && ca_ops->get_info)
3152 sz = ca_ops->get_info(sk, ~0U, &attr, &info);
3153
3154 len = min_t(unsigned int, len, sz);
3155 if (put_user(len, optlen))
3156 return -EFAULT;
3157 if (copy_to_user(optval, &info, len))
3158 return -EFAULT;
3159 return 0;
3160 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07003161 case TCP_QUICKACK:
Arnaldo Carvalho de Melo295f7322005-08-09 20:11:56 -07003162 val = !icsk->icsk_ack.pingpong;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003163 break;
Stephen Hemminger5f8ef482005-06-23 20:37:36 -07003164
3165 case TCP_CONGESTION:
3166 if (get_user(len, optlen))
3167 return -EFAULT;
3168 len = min_t(unsigned int, len, TCP_CA_NAME_MAX);
3169 if (put_user(len, optlen))
3170 return -EFAULT;
Arnaldo Carvalho de Melo6687e982005-08-10 04:03:31 -03003171 if (copy_to_user(optval, icsk->icsk_ca_ops->name, len))
Stephen Hemminger5f8ef482005-06-23 20:37:36 -07003172 return -EFAULT;
3173 return 0;
William Allen Simpsone56fb502009-12-02 18:19:30 +00003174
Dave Watson734942c2017-06-14 11:37:14 -07003175 case TCP_ULP:
3176 if (get_user(len, optlen))
3177 return -EFAULT;
3178 len = min_t(unsigned int, len, TCP_ULP_NAME_MAX);
Dave Watsond97af302017-06-26 08:36:47 -07003179 if (!icsk->icsk_ulp_ops) {
3180 if (put_user(0, optlen))
3181 return -EFAULT;
3182 return 0;
3183 }
Dave Watson734942c2017-06-14 11:37:14 -07003184 if (put_user(len, optlen))
3185 return -EFAULT;
3186 if (copy_to_user(optval, icsk->icsk_ulp_ops->name, len))
3187 return -EFAULT;
3188 return 0;
3189
Yuchung Cheng1fba70e2017-10-18 11:22:51 -07003190 case TCP_FASTOPEN_KEY: {
3191 __u8 key[TCP_FASTOPEN_KEY_LENGTH];
3192 struct tcp_fastopen_context *ctx;
3193
3194 if (get_user(len, optlen))
3195 return -EFAULT;
3196
3197 rcu_read_lock();
3198 ctx = rcu_dereference(icsk->icsk_accept_queue.fastopenq.ctx);
3199 if (ctx)
3200 memcpy(key, ctx->key, sizeof(key));
3201 else
3202 len = 0;
3203 rcu_read_unlock();
3204
3205 len = min_t(unsigned int, len, sizeof(key));
3206 if (put_user(len, optlen))
3207 return -EFAULT;
3208 if (copy_to_user(optval, key, len))
3209 return -EFAULT;
3210 return 0;
3211 }
Josh Hunt3c0fef02010-07-30 13:49:35 +00003212 case TCP_THIN_LINEAR_TIMEOUTS:
3213 val = tp->thin_lto;
3214 break;
Yuchung Cheng4a7f6002017-01-12 22:11:41 -08003215
Josh Hunt3c0fef02010-07-30 13:49:35 +00003216 case TCP_THIN_DUPACK:
Yuchung Cheng4a7f6002017-01-12 22:11:41 -08003217 val = 0;
Josh Hunt3c0fef02010-07-30 13:49:35 +00003218 break;
Jerry Chudca43c72010-08-27 19:13:28 +00003219
Pavel Emelyanovee995282012-04-19 03:40:39 +00003220 case TCP_REPAIR:
3221 val = tp->repair;
3222 break;
3223
3224 case TCP_REPAIR_QUEUE:
3225 if (tp->repair)
3226 val = tp->repair_queue;
3227 else
3228 return -EINVAL;
3229 break;
3230
Andrey Vaginb1ed4c42016-06-27 15:33:56 -07003231 case TCP_REPAIR_WINDOW: {
3232 struct tcp_repair_window opt;
3233
3234 if (get_user(len, optlen))
3235 return -EFAULT;
3236
3237 if (len != sizeof(opt))
3238 return -EINVAL;
3239
3240 if (!tp->repair)
3241 return -EPERM;
3242
3243 opt.snd_wl1 = tp->snd_wl1;
3244 opt.snd_wnd = tp->snd_wnd;
3245 opt.max_window = tp->max_window;
3246 opt.rcv_wnd = tp->rcv_wnd;
3247 opt.rcv_wup = tp->rcv_wup;
3248
3249 if (copy_to_user(optval, &opt, len))
3250 return -EFAULT;
3251 return 0;
3252 }
Pavel Emelyanovee995282012-04-19 03:40:39 +00003253 case TCP_QUEUE_SEQ:
3254 if (tp->repair_queue == TCP_SEND_QUEUE)
3255 val = tp->write_seq;
3256 else if (tp->repair_queue == TCP_RECV_QUEUE)
3257 val = tp->rcv_nxt;
3258 else
3259 return -EINVAL;
3260 break;
3261
Jerry Chudca43c72010-08-27 19:13:28 +00003262 case TCP_USER_TIMEOUT:
3263 val = jiffies_to_msecs(icsk->icsk_user_timeout);
3264 break;
Kenjiro Nakayama1536e282014-04-17 01:25:01 +09003265
3266 case TCP_FASTOPEN:
Eric Dumazet0536fcc2015-09-29 07:42:52 -07003267 val = icsk->icsk_accept_queue.fastopenq.max_qlen;
Kenjiro Nakayama1536e282014-04-17 01:25:01 +09003268 break;
3269
Wei Wang19f6d3f2017-01-23 10:59:22 -08003270 case TCP_FASTOPEN_CONNECT:
3271 val = tp->fastopen_connect;
3272 break;
3273
Christoph Paasch71c02372017-10-23 13:22:23 -07003274 case TCP_FASTOPEN_NO_COOKIE:
3275 val = tp->fastopen_no_cookie;
3276 break;
3277
Andrey Vagin93be6ce2013-02-11 05:50:18 +00003278 case TCP_TIMESTAMP:
Eric Dumazet9a568de2017-05-16 14:00:14 -07003279 val = tcp_time_stamp_raw() + tp->tsoffset;
Andrey Vagin93be6ce2013-02-11 05:50:18 +00003280 break;
Eric Dumazetc9bee3b72013-07-22 20:27:07 -07003281 case TCP_NOTSENT_LOWAT:
3282 val = tp->notsent_lowat;
3283 break;
Eric Dumazetcd8ae852015-05-03 21:34:46 -07003284 case TCP_SAVE_SYN:
3285 val = tp->save_syn;
3286 break;
3287 case TCP_SAVED_SYN: {
3288 if (get_user(len, optlen))
3289 return -EFAULT;
3290
3291 lock_sock(sk);
3292 if (tp->saved_syn) {
Eric B Munsonaea09292015-05-18 14:35:58 -04003293 if (len < tp->saved_syn[0]) {
3294 if (put_user(tp->saved_syn[0], optlen)) {
3295 release_sock(sk);
3296 return -EFAULT;
3297 }
3298 release_sock(sk);
3299 return -EINVAL;
3300 }
3301 len = tp->saved_syn[0];
Eric Dumazetcd8ae852015-05-03 21:34:46 -07003302 if (put_user(len, optlen)) {
3303 release_sock(sk);
3304 return -EFAULT;
3305 }
3306 if (copy_to_user(optval, tp->saved_syn + 1, len)) {
3307 release_sock(sk);
3308 return -EFAULT;
3309 }
3310 tcp_saved_syn_free(tp);
3311 release_sock(sk);
3312 } else {
3313 release_sock(sk);
3314 len = 0;
3315 if (put_user(len, optlen))
3316 return -EFAULT;
3317 }
3318 return 0;
3319 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07003320 default:
3321 return -ENOPROTOOPT;
Stephen Hemminger3ff50b72007-04-20 17:09:22 -07003322 }
Linus Torvalds1da177e2005-04-16 15:20:36 -07003323
3324 if (put_user(len, optlen))
3325 return -EFAULT;
3326 if (copy_to_user(optval, &val, len))
3327 return -EFAULT;
3328 return 0;
3329}
3330
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08003331int tcp_getsockopt(struct sock *sk, int level, int optname, char __user *optval,
3332 int __user *optlen)
3333{
3334 struct inet_connection_sock *icsk = inet_csk(sk);
3335
3336 if (level != SOL_TCP)
3337 return icsk->icsk_af_ops->getsockopt(sk, level, optname,
3338 optval, optlen);
3339 return do_tcp_getsockopt(sk, level, optname, optval, optlen);
3340}
Eric Dumazet4bc2f182010-07-09 21:22:10 +00003341EXPORT_SYMBOL(tcp_getsockopt);
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08003342
3343#ifdef CONFIG_COMPAT
Arnaldo Carvalho de Melo543d9cf2006-03-20 22:48:35 -08003344int compat_tcp_getsockopt(struct sock *sk, int level, int optname,
3345 char __user *optval, int __user *optlen)
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08003346{
Arnaldo Carvalho de Melodec73ff2006-03-20 22:46:16 -08003347 if (level != SOL_TCP)
3348 return inet_csk_compat_getsockopt(sk, level, optname,
3349 optval, optlen);
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08003350 return do_tcp_getsockopt(sk, level, optname, optval, optlen);
3351}
Arnaldo Carvalho de Melo543d9cf2006-03-20 22:48:35 -08003352EXPORT_SYMBOL(compat_tcp_getsockopt);
Dmitry Mishin3fdadf72006-03-20 22:45:21 -08003353#endif
Linus Torvalds1da177e2005-04-16 15:20:36 -07003354
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003355#ifdef CONFIG_TCP_MD5SIG
Eric Dumazet349ce992014-10-23 12:58:58 -07003356static DEFINE_PER_CPU(struct tcp_md5sig_pool, tcp_md5sig_pool);
Eric Dumazet71cea172013-05-20 06:52:26 +00003357static DEFINE_MUTEX(tcp_md5sig_mutex);
Eric Dumazet349ce992014-10-23 12:58:58 -07003358static bool tcp_md5sig_pool_populated = false;
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003359
Eric Dumazet71cea172013-05-20 06:52:26 +00003360static void __tcp_alloc_md5sig_pool(void)
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003361{
Herbert Xucf80e0e2016-01-24 21:20:23 +08003362 struct crypto_ahash *hash;
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003363 int cpu;
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003364
Herbert Xucf80e0e2016-01-24 21:20:23 +08003365 hash = crypto_alloc_ahash("md5", 0, CRYPTO_ALG_ASYNC);
Linus Torvalds70477372016-03-17 11:22:54 -07003366 if (IS_ERR(hash))
Herbert Xucf80e0e2016-01-24 21:20:23 +08003367 return;
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003368
Herbert Xucf80e0e2016-01-24 21:20:23 +08003369 for_each_possible_cpu(cpu) {
Eric Dumazet19689e32016-06-27 18:51:53 +02003370 void *scratch = per_cpu(tcp_md5sig_pool, cpu).scratch;
Herbert Xucf80e0e2016-01-24 21:20:23 +08003371 struct ahash_request *req;
3372
Eric Dumazet19689e32016-06-27 18:51:53 +02003373 if (!scratch) {
3374 scratch = kmalloc_node(sizeof(union tcp_md5sum_block) +
3375 sizeof(struct tcphdr),
3376 GFP_KERNEL,
3377 cpu_to_node(cpu));
3378 if (!scratch)
3379 return;
3380 per_cpu(tcp_md5sig_pool, cpu).scratch = scratch;
3381 }
Herbert Xucf80e0e2016-01-24 21:20:23 +08003382 if (per_cpu(tcp_md5sig_pool, cpu).md5_req)
3383 continue;
3384
3385 req = ahash_request_alloc(hash, GFP_KERNEL);
3386 if (!req)
3387 return;
3388
3389 ahash_request_set_callback(req, 0, NULL, NULL);
3390
3391 per_cpu(tcp_md5sig_pool, cpu).md5_req = req;
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003392 }
Eric Dumazet349ce992014-10-23 12:58:58 -07003393 /* before setting tcp_md5sig_pool_populated, we must commit all writes
3394 * to memory. See smp_rmb() in tcp_get_md5sig_pool()
Eric Dumazet71cea172013-05-20 06:52:26 +00003395 */
3396 smp_wmb();
Eric Dumazet349ce992014-10-23 12:58:58 -07003397 tcp_md5sig_pool_populated = true;
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003398}
3399
Eric Dumazet71cea172013-05-20 06:52:26 +00003400bool tcp_alloc_md5sig_pool(void)
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003401{
Eric Dumazet349ce992014-10-23 12:58:58 -07003402 if (unlikely(!tcp_md5sig_pool_populated)) {
Eric Dumazet71cea172013-05-20 06:52:26 +00003403 mutex_lock(&tcp_md5sig_mutex);
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003404
Eric Dumazet349ce992014-10-23 12:58:58 -07003405 if (!tcp_md5sig_pool_populated)
Eric Dumazet71cea172013-05-20 06:52:26 +00003406 __tcp_alloc_md5sig_pool();
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003407
Eric Dumazet71cea172013-05-20 06:52:26 +00003408 mutex_unlock(&tcp_md5sig_mutex);
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003409 }
Eric Dumazet349ce992014-10-23 12:58:58 -07003410 return tcp_md5sig_pool_populated;
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003411}
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003412EXPORT_SYMBOL(tcp_alloc_md5sig_pool);
3413
Eric Dumazet35790c02010-05-16 00:34:04 -07003414
3415/**
3416 * tcp_get_md5sig_pool - get md5sig_pool for this user
3417 *
3418 * We use percpu structure, so if we succeed, we exit with preemption
3419 * and BH disabled, to make sure another thread or softirq handling
3420 * wont try to get same context.
3421 */
3422struct tcp_md5sig_pool *tcp_get_md5sig_pool(void)
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003423{
Eric Dumazet35790c02010-05-16 00:34:04 -07003424 local_bh_disable();
Eric Dumazet35790c02010-05-16 00:34:04 -07003425
Eric Dumazet349ce992014-10-23 12:58:58 -07003426 if (tcp_md5sig_pool_populated) {
3427 /* coupled with smp_wmb() in __tcp_alloc_md5sig_pool() */
3428 smp_rmb();
3429 return this_cpu_ptr(&tcp_md5sig_pool);
3430 }
Eric Dumazet35790c02010-05-16 00:34:04 -07003431 local_bh_enable();
3432 return NULL;
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003433}
Eric Dumazet35790c02010-05-16 00:34:04 -07003434EXPORT_SYMBOL(tcp_get_md5sig_pool);
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003435
Adam Langley49a72df2008-07-19 00:01:42 -07003436int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *hp,
Eric Dumazetcf533ea2011-10-21 05:22:42 -04003437 const struct sk_buff *skb, unsigned int header_len)
Adam Langley49a72df2008-07-19 00:01:42 -07003438{
3439 struct scatterlist sg;
3440 const struct tcphdr *tp = tcp_hdr(skb);
Herbert Xucf80e0e2016-01-24 21:20:23 +08003441 struct ahash_request *req = hp->md5_req;
Eric Dumazet95c96172012-04-15 05:58:06 +00003442 unsigned int i;
3443 const unsigned int head_data_len = skb_headlen(skb) > header_len ?
3444 skb_headlen(skb) - header_len : 0;
Adam Langley49a72df2008-07-19 00:01:42 -07003445 const struct skb_shared_info *shi = skb_shinfo(skb);
Eric Dumazetd7fd1b572010-05-17 20:40:51 +00003446 struct sk_buff *frag_iter;
Adam Langley49a72df2008-07-19 00:01:42 -07003447
3448 sg_init_table(&sg, 1);
3449
3450 sg_set_buf(&sg, ((u8 *) tp) + header_len, head_data_len);
Herbert Xucf80e0e2016-01-24 21:20:23 +08003451 ahash_request_set_crypt(req, &sg, NULL, head_data_len);
3452 if (crypto_ahash_update(req))
Adam Langley49a72df2008-07-19 00:01:42 -07003453 return 1;
3454
3455 for (i = 0; i < shi->nr_frags; ++i) {
3456 const struct skb_frag_struct *f = &shi->frags[i];
Eric Dumazet54d27fc2013-05-13 21:25:52 +00003457 unsigned int offset = f->page_offset;
3458 struct page *page = skb_frag_page(f) + (offset >> PAGE_SHIFT);
3459
3460 sg_set_page(&sg, page, skb_frag_size(f),
3461 offset_in_page(offset));
Herbert Xucf80e0e2016-01-24 21:20:23 +08003462 ahash_request_set_crypt(req, &sg, NULL, skb_frag_size(f));
3463 if (crypto_ahash_update(req))
Adam Langley49a72df2008-07-19 00:01:42 -07003464 return 1;
3465 }
3466
Eric Dumazetd7fd1b572010-05-17 20:40:51 +00003467 skb_walk_frags(skb, frag_iter)
3468 if (tcp_md5_hash_skb_data(hp, frag_iter, 0))
3469 return 1;
3470
Adam Langley49a72df2008-07-19 00:01:42 -07003471 return 0;
3472}
Adam Langley49a72df2008-07-19 00:01:42 -07003473EXPORT_SYMBOL(tcp_md5_hash_skb_data);
3474
Eric Dumazetcf533ea2011-10-21 05:22:42 -04003475int tcp_md5_hash_key(struct tcp_md5sig_pool *hp, const struct tcp_md5sig_key *key)
Adam Langley49a72df2008-07-19 00:01:42 -07003476{
3477 struct scatterlist sg;
3478
3479 sg_init_one(&sg, key->key, key->keylen);
Herbert Xucf80e0e2016-01-24 21:20:23 +08003480 ahash_request_set_crypt(hp->md5_req, &sg, NULL, key->keylen);
3481 return crypto_ahash_update(hp->md5_req);
Adam Langley49a72df2008-07-19 00:01:42 -07003482}
Adam Langley49a72df2008-07-19 00:01:42 -07003483EXPORT_SYMBOL(tcp_md5_hash_key);
3484
YOSHIFUJI Hideakicfb6eeb2006-11-14 19:07:45 -08003485#endif
3486
Andi Kleen4ac02ba2007-04-20 17:11:46 -07003487void tcp_done(struct sock *sk)
3488{
Jerry Chu83368862012-08-31 12:29:12 +00003489 struct request_sock *req = tcp_sk(sk)->fastopen_rsk;
3490
Jianjun Kong5a5f3a82008-11-03 00:24:34 -08003491 if (sk->sk_state == TCP_SYN_SENT || sk->sk_state == TCP_SYN_RECV)
Eric Dumazetc10d9312016-04-29 14:16:47 -07003492 TCP_INC_STATS(sock_net(sk), TCP_MIB_ATTEMPTFAILS);
Andi Kleen4ac02ba2007-04-20 17:11:46 -07003493
3494 tcp_set_state(sk, TCP_CLOSE);
3495 tcp_clear_xmit_timers(sk);
Ian Morris00db4122015-04-03 09:17:27 +01003496 if (req)
Jerry Chu83368862012-08-31 12:29:12 +00003497 reqsk_fastopen_remove(sk, req, false);
Andi Kleen4ac02ba2007-04-20 17:11:46 -07003498
3499 sk->sk_shutdown = SHUTDOWN_MASK;
3500
3501 if (!sock_flag(sk, SOCK_DEAD))
3502 sk->sk_state_change(sk);
3503 else
3504 inet_csk_destroy_sock(sk);
3505}
3506EXPORT_SYMBOL_GPL(tcp_done);
3507
Lorenzo Colittic1e64e22015-12-16 12:30:05 +09003508int tcp_abort(struct sock *sk, int err)
3509{
3510 if (!sk_fullsock(sk)) {
Eric Dumazet07f6f4a2015-12-17 16:14:11 -08003511 if (sk->sk_state == TCP_NEW_SYN_RECV) {
3512 struct request_sock *req = inet_reqsk(sk);
3513
3514 local_bh_disable();
3515 inet_csk_reqsk_queue_drop_and_put(req->rsk_listener,
3516 req);
3517 local_bh_enable();
3518 return 0;
3519 }
Lorenzo Colittic1e64e22015-12-16 12:30:05 +09003520 return -EOPNOTSUPP;
3521 }
3522
3523 /* Don't race with userspace socket closes such as tcp_close. */
3524 lock_sock(sk);
3525
Lorenzo Colitti2010b932015-12-22 00:03:44 +09003526 if (sk->sk_state == TCP_LISTEN) {
3527 tcp_set_state(sk, TCP_CLOSE);
3528 inet_csk_listen_stop(sk);
3529 }
3530
Lorenzo Colittic1e64e22015-12-16 12:30:05 +09003531 /* Don't race with BH socket closes such as inet_csk_listen_stop. */
3532 local_bh_disable();
3533 bh_lock_sock(sk);
3534
3535 if (!sock_flag(sk, SOCK_DEAD)) {
3536 sk->sk_err = err;
3537 /* This barrier is coupled with smp_rmb() in tcp_poll() */
3538 smp_wmb();
3539 sk->sk_error_report(sk);
3540 if (tcp_need_reset(sk->sk_state))
3541 tcp_send_active_reset(sk, GFP_ATOMIC);
3542 tcp_done(sk);
3543 }
3544
3545 bh_unlock_sock(sk);
3546 local_bh_enable();
3547 release_sock(sk);
Lorenzo Colittic1e64e22015-12-16 12:30:05 +09003548 return 0;
3549}
3550EXPORT_SYMBOL_GPL(tcp_abort);
3551
Stephen Hemminger5f8ef482005-06-23 20:37:36 -07003552extern struct tcp_congestion_ops tcp_reno;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003553
3554static __initdata unsigned long thash_entries;
3555static int __init set_thash_entries(char *str)
3556{
Eldad Zack413c27d2012-05-19 14:13:18 +00003557 ssize_t ret;
3558
Linus Torvalds1da177e2005-04-16 15:20:36 -07003559 if (!str)
3560 return 0;
Eldad Zack413c27d2012-05-19 14:13:18 +00003561
3562 ret = kstrtoul(str, 0, &thash_entries);
3563 if (ret)
3564 return 0;
3565
Linus Torvalds1da177e2005-04-16 15:20:36 -07003566 return 1;
3567}
3568__setup("thash_entries=", set_thash_entries);
3569
Fabian Frederick47d7a88c2014-10-01 18:27:50 +02003570static void __init tcp_init_mem(void)
Glauber Costa4acb4192012-01-30 01:20:17 +00003571{
Eric Dumazetb66e91c2015-05-15 12:39:30 -07003572 unsigned long limit = nr_free_buffer_pages() / 16;
3573
Glauber Costa4acb4192012-01-30 01:20:17 +00003574 limit = max(limit, 128UL);
Eric Dumazetb66e91c2015-05-15 12:39:30 -07003575 sysctl_tcp_mem[0] = limit / 4 * 3; /* 4.68 % */
3576 sysctl_tcp_mem[1] = limit; /* 6.25 % */
3577 sysctl_tcp_mem[2] = sysctl_tcp_mem[0] * 2; /* 9.37 % */
Glauber Costa4acb4192012-01-30 01:20:17 +00003578}
3579
Linus Torvalds1da177e2005-04-16 15:20:36 -07003580void __init tcp_init(void)
3581{
Eric Dumazetb49960a2012-05-02 02:28:41 +00003582 int max_rshare, max_wshare, cnt;
Eric Dumazetb2d3ea42016-09-19 23:39:12 -04003583 unsigned long limit;
Dimitri Sivanich074b8512012-02-08 12:39:07 -08003584 unsigned int i;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003585
Eric Dumazetb2d3ea42016-09-19 23:39:12 -04003586 BUILD_BUG_ON(sizeof(struct tcp_skb_cb) >
3587 FIELD_SIZEOF(struct sk_buff, cb));
Linus Torvalds1da177e2005-04-16 15:20:36 -07003588
Tejun Heo908c7f12014-09-08 09:51:29 +09003589 percpu_counter_init(&tcp_sockets_allocated, 0, GFP_KERNEL);
3590 percpu_counter_init(&tcp_orphan_count, 0, GFP_KERNEL);
Haishuang Yan1946e672016-12-28 17:52:32 +08003591 inet_hashinfo_init(&tcp_hashinfo);
Martin KaFai Lau27da6d32017-12-01 12:52:32 -08003592 inet_hashinfo2_init(&tcp_hashinfo, "tcp_listen_portaddr_hash",
3593 thash_entries, 21, /* one slot per 2 MB*/
3594 0, 64 * 1024);
Arnaldo Carvalho de Melo6e04e022005-08-09 20:07:35 -07003595 tcp_hashinfo.bind_bucket_cachep =
3596 kmem_cache_create("tcp_bind_bucket",
3597 sizeof(struct inet_bind_bucket), 0,
Paul Mundt20c2df82007-07-20 10:11:58 +09003598 SLAB_HWCACHE_ALIGN|SLAB_PANIC, NULL);
Linus Torvalds1da177e2005-04-16 15:20:36 -07003599
Linus Torvalds1da177e2005-04-16 15:20:36 -07003600 /* Size and allocate the main established and bind bucket
3601 * hash tables.
3602 *
3603 * The methodology is similar to that of the buffer cache.
3604 */
Arnaldo Carvalho de Melo6e04e022005-08-09 20:07:35 -07003605 tcp_hashinfo.ehash =
Linus Torvalds1da177e2005-04-16 15:20:36 -07003606 alloc_large_system_hash("TCP established",
Arnaldo Carvalho de Melo0f7ff922005-08-09 19:59:44 -07003607 sizeof(struct inet_ehash_bucket),
Linus Torvalds1da177e2005-04-16 15:20:36 -07003608 thash_entries,
Eric Dumazetfd90b292012-11-30 10:08:52 +00003609 17, /* one slot per 128 KB of memory */
John Heffner9e950ef2006-11-06 23:10:51 -08003610 0,
Linus Torvalds1da177e2005-04-16 15:20:36 -07003611 NULL,
Eric Dumazetf373b532009-10-09 00:16:19 +00003612 &tcp_hashinfo.ehash_mask,
Tim Bird31fe62b2012-05-23 13:33:35 +00003613 0,
Jean Delvare0ccfe612007-10-30 00:59:25 -07003614 thash_entries ? 0 : 512 * 1024);
Eric Dumazet05dbc7b2013-10-03 00:22:02 -07003615 for (i = 0; i <= tcp_hashinfo.ehash_mask; i++)
Eric Dumazet3ab5aee2008-11-16 19:40:17 -08003616 INIT_HLIST_NULLS_HEAD(&tcp_hashinfo.ehash[i].chain, i);
Eric Dumazet05dbc7b2013-10-03 00:22:02 -07003617
Eric Dumazet230140c2007-11-07 02:40:20 -08003618 if (inet_ehash_locks_alloc(&tcp_hashinfo))
3619 panic("TCP: failed to alloc ehash_locks");
Arnaldo Carvalho de Melo6e04e022005-08-09 20:07:35 -07003620 tcp_hashinfo.bhash =
Linus Torvalds1da177e2005-04-16 15:20:36 -07003621 alloc_large_system_hash("TCP bind",
Arnaldo Carvalho de Melo0f7ff922005-08-09 19:59:44 -07003622 sizeof(struct inet_bind_hashbucket),
Eric Dumazetf373b532009-10-09 00:16:19 +00003623 tcp_hashinfo.ehash_mask + 1,
Eric Dumazetfd90b292012-11-30 10:08:52 +00003624 17, /* one slot per 128 KB of memory */
John Heffner9e950ef2006-11-06 23:10:51 -08003625 0,
Arnaldo Carvalho de Melo6e04e022005-08-09 20:07:35 -07003626 &tcp_hashinfo.bhash_size,
Linus Torvalds1da177e2005-04-16 15:20:36 -07003627 NULL,
Tim Bird31fe62b2012-05-23 13:33:35 +00003628 0,
Linus Torvalds1da177e2005-04-16 15:20:36 -07003629 64 * 1024);
Dimitri Sivanich074b8512012-02-08 12:39:07 -08003630 tcp_hashinfo.bhash_size = 1U << tcp_hashinfo.bhash_size;
Arnaldo Carvalho de Melo6e04e022005-08-09 20:07:35 -07003631 for (i = 0; i < tcp_hashinfo.bhash_size; i++) {
3632 spin_lock_init(&tcp_hashinfo.bhash[i].lock);
3633 INIT_HLIST_HEAD(&tcp_hashinfo.bhash[i].chain);
Linus Torvalds1da177e2005-04-16 15:20:36 -07003634 }
3635
Eric Dumazetc5ed63d2010-08-25 23:02:17 -07003636
3637 cnt = tcp_hashinfo.ehash_mask + 1;
Eric Dumazetc5ed63d2010-08-25 23:02:17 -07003638 sysctl_tcp_max_orphans = cnt / 2;
Linus Torvalds1da177e2005-04-16 15:20:36 -07003639
Eric W. Biedermana4fe34b2013-10-19 16:25:36 -07003640 tcp_init_mem();
Jason Wangc43b8742012-02-02 00:07:00 +00003641 /* Set per-socket limits to no more than 1/128 the pressure threshold */
Eric Dumazet5fb84b12012-04-10 00:56:42 +00003642 limit = nr_free_buffer_pages() << (PAGE_SHIFT - 7);
Eric Dumazetb49960a2012-05-02 02:28:41 +00003643 max_wshare = min(4UL*1024*1024, limit);
3644 max_rshare = min(6UL*1024*1024, limit);
John Heffner7b4f4b52006-03-25 01:34:07 -08003645
Eric Dumazet356d1832017-11-07 00:29:28 -08003646 init_net.ipv4.sysctl_tcp_wmem[0] = SK_MEM_QUANTUM;
3647 init_net.ipv4.sysctl_tcp_wmem[1] = 16*1024;
3648 init_net.ipv4.sysctl_tcp_wmem[2] = max(64*1024, max_wshare);
John Heffner7b4f4b52006-03-25 01:34:07 -08003649
Eric Dumazet356d1832017-11-07 00:29:28 -08003650 init_net.ipv4.sysctl_tcp_rmem[0] = SK_MEM_QUANTUM;
3651 init_net.ipv4.sysctl_tcp_rmem[1] = 87380;
3652 init_net.ipv4.sysctl_tcp_rmem[2] = max(87380, max_rshare);
Linus Torvalds1da177e2005-04-16 15:20:36 -07003653
Joe Perchesafd465032012-03-12 07:03:32 +00003654 pr_info("Hash tables configured (established %u bind %u)\n",
Joe Perches058bd4d2012-03-11 18:36:11 +00003655 tcp_hashinfo.ehash_mask + 1, tcp_hashinfo.bhash_size);
Stephen Hemminger317a76f2005-06-23 12:19:55 -07003656
Haishuang Yan1946e672016-12-28 17:52:32 +08003657 tcp_v4_init();
David S. Miller51c5d0c2012-07-10 00:49:14 -07003658 tcp_metrics_init();
Florian Westphal55d86942014-09-26 22:37:32 +02003659 BUG_ON(tcp_register_congestion_control(&tcp_reno) != 0);
Eric Dumazet46d3cea2012-07-11 05:50:31 +00003660 tcp_tasklet_init();
Linus Torvalds1da177e2005-04-16 15:20:36 -07003661}