linux/net/ipv4
Stanislav Fomichev 9cacf81f81 bpf: Remove extra lock_sock for TCP_ZEROCOPY_RECEIVE
Add custom implementation of getsockopt hook for TCP_ZEROCOPY_RECEIVE.
We skip generic hooks for TCP_ZEROCOPY_RECEIVE and have a custom
call in do_tcp_getsockopt using the on-stack data. This removes
3% overhead for locking/unlocking the socket.

Without this patch:
     3.38%     0.07%  tcp_mmap  [kernel.kallsyms]  [k] __cgroup_bpf_run_filter_getsockopt
            |
             --3.30%--__cgroup_bpf_run_filter_getsockopt
                       |
                        --0.81%--__kmalloc

With the patch applied:
     0.52%     0.12%  tcp_mmap  [kernel.kallsyms]  [k] __cgroup_bpf_run_filter_getsockopt_kern

Note, exporting uapi/tcp.h requires removing netinet/tcp.h
from test_progs.h because those headers have confliciting
definitions.

Signed-off-by: Stanislav Fomichev <sdf@google.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Martin KaFai Lau <kafai@fb.com>
Link: https://lore.kernel.org/bpf/20210115163501.805133-2-sdf@google.com
2021-01-20 14:23:00 -08:00
..
bpfilter net: Revert "net: optimize the sockptr_t for unified kernel/user address spaces" 2020-08-10 12:06:44 -07:00
netfilter netfilter: rpfilter: mask ecn bits before fib lookup 2021-01-19 13:54:30 -08:00
af_inet.c bpf: Allow bpf_{s,g}etsockopt from cgroup bind{4,6} hooks 2020-12-02 13:25:11 -08:00
ah4.c inet: Use fallthrough; 2020-03-12 15:55:00 -07:00
arp.c net: Exempt multicast addresses from five-second neighbor lifetime 2020-11-13 14:24:39 -08:00
bpf_tcp_ca.c bpf: Remove hard-coded btf_vmlinux assumption from BPF verifier 2020-12-03 17:38:21 -08:00
cipso_ipv4.c cipso: fix 'audit_secid' kernel-doc warning in cipso_ipv4.c 2020-09-08 20:03:36 -07:00
datagram.c inet: stop leaking jiffies on the wire 2019-11-01 14:57:52 -07:00
devinet.c treewide: rename nla_strlcpy to nla_strscpy. 2020-11-16 08:08:54 -08:00
esp4_offload.c net: Add MODULE_DESCRIPTION entries to network modules 2020-06-20 21:33:57 -07:00
esp4.c esp: avoid unneeded kmap_atomic call 2021-01-11 18:20:09 -08:00
fib_frontend.c ipv4: Ignore ECN bits for fib lookups in fib_compute_spec_dst() 2020-12-28 14:44:32 -08:00
fib_lookup.h net: add net available in build_state 2020-03-29 22:30:57 -07:00
fib_notifier.c net: fib_notifier: propagate extack down to the notifier block callback 2019-10-04 11:10:56 -07:00
fib_rules.c fib: use indirect call wrappers in the most common fib_rules_ops 2020-07-28 17:42:31 -07:00
fib_semantics.c treewide: rename nla_strlcpy to nla_strscpy. 2020-11-16 08:08:54 -08:00
fib_trie.c nexthop: Remove in-kernel route notifications when nexthop changes 2020-11-06 11:28:50 -08:00
fou.c genetlink: move to smaller ops wherever possible 2020-10-02 19:11:11 -07:00
gre_demux.c erspan: fix version 1 check in gre_parse_header() 2020-12-28 15:00:00 -08:00
gre_offload.c net: gre: recompute gre csum for sctp over gre tunnels 2020-08-03 15:29:44 -07:00
icmp.c lsm,selinux: pass flowi_common instead of flowi to the LSM hooks 2020-11-23 18:36:21 -05:00
igmp.c ip*_mc_gsfget(): lift copyout of struct group_filter into callers 2020-05-20 20:31:27 -04:00
inet_connection_sock.c tcp: fix TCP_USER_TIMEOUT with zero window 2021-01-18 19:59:17 -08:00
inet_diag.c inet_diag: Fix error path to cancel the meseage in inet_req_diag_fill() 2020-11-17 16:08:36 -08:00
inet_fragment.c inet: frags: batch fqdir destroy works 2020-12-12 15:08:54 -08:00
inet_hashtables.c tcp: fix race condition when creating child sockets from syncookies 2020-11-23 16:32:33 -08:00
inet_timewait_sock.c net: Use generic ns_common::count 2020-08-19 14:06:36 +02:00
inetpeer.c inetpeer: fix data-race in inet_putpeer / inet_putpeer 2019-11-07 16:15:56 -08:00
ip_forward.c ipv4: Revert removal of rt_uses_gateway 2019-09-20 18:23:33 -07:00
ip_fragment.c inet: frags: re-introduce skb coalescing for local delivery 2019-08-08 15:55:10 -07:00
ip_gre.c ipv4/ipv6: switch to dev_get_tstats64 2020-11-09 17:50:28 -08:00
ip_input.c bpf: Add socket assign support 2020-03-30 13:45:04 -07:00
ip_options.c net: clean up codestyle for net/ipv4 2020-08-25 06:28:02 -07:00
ip_output.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2021-01-08 13:28:00 -08:00
ip_sockglue.c net: Remove duplicated midx check against 0 2020-08-25 06:23:59 -07:00
ip_tunnel_core.c net: ip_tunnel: clean up endianness conversions 2021-01-08 19:25:35 -08:00
ip_tunnel.c net: fix pmtu check in nopmtudisc mode 2021-01-07 14:42:36 -08:00
ip_vti.c vti: switch to dev_get_tstats64 2020-11-09 17:50:28 -08:00
ipcomp.c ipcomp: assign if_id to child tunnel from parent tunnel 2020-07-09 12:55:37 +02:00
ipconfig.c net: ipconfig: Avoid spurious blank lines in boot log 2020-11-12 14:53:33 -08:00
ipip.c ipv4/ipv6: switch to dev_get_tstats64 2020-11-09 17:50:28 -08:00
ipmr_base.c net: fib_notifier: propagate extack down to the notifier block callback 2019-10-04 11:10:56 -07:00
ipmr.c ipmr: Use full VIF ID in netlink cache reports 2020-09-10 12:25:51 -07:00
Kconfig net: ipv4: remove duplicate "the the" phrase in Kconfig text 2020-08-18 16:02:16 -07:00
Makefile udp_tunnel: add central NIC RX port offload infrastructure 2020-07-10 13:54:00 -07:00
metrics.c treewide: rename nla_strlcpy to nla_strscpy. 2020-11-16 08:08:54 -08:00
netfilter.c netfilter: use actual socket sk rather than skb sk when routing harder 2020-10-30 12:57:39 +01:00
netlink.c
nexthop.c nexthop: Bounce NHA_GATEWAY in FDB nexthop groups 2021-01-07 18:47:18 -08:00
ping.c lsm,selinux: pass flowi_common instead of flowi to the LSM hooks 2020-11-23 18:36:21 -05:00
proc.c net: udp: introduce UDP_MIB_MEMERRORS for udp_mem 2020-11-09 15:34:44 -08:00
protocol.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
raw_diag.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-03-12 22:34:48 -07:00
raw.c lsm,selinux: pass flowi_common instead of flowi to the LSM hooks 2020-11-23 18:36:21 -05:00
route.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-12-03 15:44:09 -08:00
syncookies.c selinux/stable-5.11 PR 20201214 2020-12-16 11:01:04 -08:00
sysctl_net_ipv4.c tcp: reflect tos value received in SYN to the socket 2020-09-10 13:15:40 -07:00
tcp_bbr.c tcp: only postpone PROBE_RTT if RTT is < current min_rtt estimate 2020-11-17 11:03:22 -08:00
tcp_bic.c tcp: fix stretch ACK bugs in BIC 2020-03-16 18:26:54 -07:00
tcp_bpf.c bpf, sockmap: Ensure SO_RCVBUF memory is observed on ingress redirect 2020-11-18 00:12:34 +01:00
tcp_cdg.c
tcp_cong.c tcp: Set INET_ECN_xmit configuration in tcp_reinit_congestion_control 2020-11-20 18:09:47 -08:00
tcp_cubic.c tcp_cubic: use memset and offsetof init 2021-01-15 20:22:16 -08:00
tcp_dctcp.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
tcp_dctcp.h
tcp_diag.c inet_diag: Move the INET_DIAG_REQ_BYTECODE nlattr to cb->data 2020-02-27 18:50:19 -08:00
tcp_fastopen.c bpf: tcp: Add bpf_skops_established() 2020-08-24 14:35:00 -07:00
tcp_highspeed.c Replace HTTP links with HTTPS ones: IPv* 2020-07-06 13:23:03 -07:00
tcp_htcp.c Replace HTTP links with HTTPS ones: IPv* 2020-07-06 13:23:03 -07:00
tcp_hybla.c
tcp_illinois.c
tcp_input.c tcp: fix TCP socket rehash stats mis-accounting 2021-01-19 19:47:20 -08:00
tcp_ipv4.c bpf: Remove extra lock_sock for TCP_ZEROCOPY_RECEIVE 2021-01-20 14:23:00 -08:00
tcp_lp.c net: ipv4: Fix some kerneldoc warnings in TCP Low Priority 2020-10-30 12:10:57 -07:00
tcp_metrics.c fixes-v5.11 2020-12-14 16:40:27 -08:00
tcp_minisocks.c tcp: parse mptcp options contained in reset packets 2020-12-14 17:30:06 -08:00
tcp_nv.c
tcp_offload.c treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
tcp_output.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2021-01-20 12:16:11 -08:00
tcp_rate.c
tcp_recovery.c tcp: avoid slow start during fast recovery on new losses 2020-11-02 17:17:40 -08:00
tcp_scalable.c net: ipv4: delete repeated words 2020-08-24 17:31:20 -07:00
tcp_timer.c tcp: fix TCP socket rehash stats mis-accounting 2021-01-19 19:47:20 -08:00
tcp_ulp.c bpf: sockmap: Only check ULP for TCP sockets 2020-03-09 22:34:58 +01:00
tcp_vegas.c tcp: use semicolons rather than commas to separate statements 2020-10-13 17:11:52 -07:00
tcp_vegas.h
tcp_veno.c Replace HTTP links with HTTPS ones: IPv* 2020-07-06 13:23:03 -07:00
tcp_westwood.c
tcp_yeah.c tcp: fix stretch ACK bugs in Yeah 2020-03-16 18:26:55 -07:00
tcp.c bpf: Remove extra lock_sock for TCP_ZEROCOPY_RECEIVE 2021-01-20 14:23:00 -08:00
tunnel4.c tunnel4: add cb_handler to struct xfrm_tunnel 2020-07-09 12:51:36 +02:00
udp_bpf.c net: sk_msg: Simplify sk_psock initialization 2020-08-21 15:16:11 -07:00
udp_diag.c net: udp: remove redundant initialization in udp_dump_one 2020-11-09 16:42:49 -08:00
udp_impl.h net: pass a sockptr_t into ->setsockopt 2020-07-24 15:41:54 -07:00
udp_offload.c Merge https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-11-12 16:54:48 -08:00
udp_tunnel_core.c udp_tunnel: reshuffle NETIF_F_RX_UDP_TUNNEL_PORT checks 2021-01-07 12:53:29 -08:00
udp_tunnel_nic.c udp_tunnel: add the ability to share port tables 2020-09-28 12:50:12 -07:00
udp_tunnel_stub.c udp_tunnel: add central NIC RX port offload infrastructure 2020-07-10 13:54:00 -07:00
udp.c udp: mask TOS bits in udp_v4_early_demux() 2021-01-19 13:54:30 -08:00
udplite.c net/ipv4: remove compat_ip_{get,set}sockopt 2020-07-19 18:16:41 -07:00
xfrm4_input.c xfrm: state: remove extract_input indirection from xfrm_state_afinfo 2020-05-06 09:40:08 +02:00
xfrm4_output.c xfrm: fix unused variable warning if CONFIG_NETFILTER=n 2020-05-11 15:12:27 +02:00
xfrm4_policy.c net: add bool confirm_neigh parameter for dst_ops.update_pmtu 2019-12-24 22:28:54 -08:00
xfrm4_protocol.c xfrm: add route lookup to xfrm4_rcv_encap 2019-12-09 09:59:07 +01:00
xfrm4_state.c xfrm: remove output_finish indirection from xfrm_state_afinfo 2020-05-06 09:40:08 +02:00
xfrm4_tunnel.c xfrm: interface: fix the priorities for ipip and ipv6 tunnels 2020-10-09 12:29:48 +02:00