Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
Pull networking updates from David Miller:
1) The addition of nftables. No longer will we need protocol aware
firewall filtering modules, it can all live in userspace.
At the core of nftables is a, for lack of a better term, virtual
machine that executes byte codes to inspect packet or metadata
(arriving interface index, etc.) and make verdict decisions.
Besides support for loading packet contents and comparing them, the
interpreter supports lookups in various datastructures as
fundamental operations. For example sets are supports, and
therefore one could create a set of whitelist IP address entries
which have ACCEPT verdicts attached to them, and use the appropriate
byte codes to do such lookups.
Since the interpreted code is composed in userspace, userspace can
do things like optimize things before giving it to the kernel.
Another major improvement is the capability of atomically updating
portions of the ruleset. In the existing netfilter implementation,
one has to update the entire rule set in order to make a change and
this is very expensive.
Userspace tools exist to create nftables rules using existing
netfilter rule sets, but both kernel implementations will need to
co-exist for quite some time as we transition from the old to the
new stuff.
Kudos to Patrick McHardy, Pablo Neira Ayuso, and others who have
worked so hard on this.
2) Daniel Borkmann and Hannes Frederic Sowa made several improvements
to our pseudo-random number generator, mostly used for things like
UDP port randomization and netfitler, amongst other things.
In particular the taus88 generater is updated to taus113, and test
cases are added.
3) Support 64-bit rates in HTB and TBF schedulers, from Eric Dumazet
and Yang Yingliang.
4) Add support for new 577xx tigon3 chips to tg3 driver, from Nithin
Sujir.
5) Fix two fatal flaws in TCP dynamic right sizing, from Eric Dumazet,
Neal Cardwell, and Yuchung Cheng.
6) Allow IP_TOS and IP_TTL to be specified in sendmsg() ancillary
control message data, much like other socket option attributes.
From Francesco Fusco.
7) Allow applications to specify a cap on the rate computed
automatically by the kernel for pacing flows, via a new
SO_MAX_PACING_RATE socket option. From Eric Dumazet.
8) Make the initial autotuned send buffer sizing in TCP more closely
reflect actual needs, from Eric Dumazet.
9) Currently early socket demux only happens for TCP sockets, but we
can do it for connected UDP sockets too. Implementation from Shawn
Bohrer.
10) Refactor inet socket demux with the goal of improving hash demux
performance for listening sockets. With the main goals being able
to use RCU lookups on even request sockets, and eliminating the
listening lock contention. From Eric Dumazet.
11) The bonding layer has many demuxes in it's fast path, and an RCU
conversion was started back in 3.11, several changes here extend the
RCU usage to even more locations. From Ding Tianhong and Wang
Yufen, based upon suggestions by Nikolay Aleksandrov and Veaceslav
Falico.
12) Allow stackability of segmentation offloads to, in particular, allow
segmentation offloading over tunnels. From Eric Dumazet.
13) Significantly improve the handling of secret keys we input into the
various hash functions in the inet hashtables, TCP fast open, as
well as syncookies. From Hannes Frederic Sowa. The key fundamental
operation is "net_get_random_once()" which uses static keys.
Hannes even extended this to ipv4/ipv6 fragmentation handling and
our generic flow dissector.
14) The generic driver layer takes care now to set the driver data to
NULL on device removal, so it's no longer necessary for drivers to
explicitly set it to NULL any more. Many drivers have been cleaned
up in this way, from Jingoo Han.
15) Add a BPF based packet scheduler classifier, from Daniel Borkmann.
16) Improve CRC32 interfaces and generic SKB checksum iterators so that
SCTP's checksumming can more cleanly be handled. Also from Daniel
Borkmann.
17) Add a new PMTU discovery mode, IP_PMTUDISC_INTERFACE, which forces
using the interface MTU value. This helps avoid PMTU attacks,
particularly on DNS servers. From Hannes Frederic Sowa.
18) Use generic XPS for transmit queue steering rather than internal
(re-)implementation in virtio-net. From Jason Wang.
* git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next: (1622 commits)
random32: add test cases for taus113 implementation
random32: upgrade taus88 generator to taus113 from errata paper
random32: move rnd_state to linux/random.h
random32: add prandom_reseed_late() and call when nonblocking pool becomes initialized
random32: add periodic reseeding
random32: fix off-by-one in seeding requirement
PHY: Add RTL8201CP phy_driver to realtek
xtsonic: add missing platform_set_drvdata() in xtsonic_probe()
macmace: add missing platform_set_drvdata() in mace_probe()
ethernet/arc/arc_emac: add missing platform_set_drvdata() in arc_emac_probe()
ipv6: protect for_each_sk_fl_rcu in mem_check with rcu_read_lock_bh
vlan: Implement vlan_dev_get_egress_qos_mask as an inline.
ixgbe: add warning when max_vfs is out of range.
igb: Update link modes display in ethtool
netfilter: push reasm skb through instead of original frag skbs
ip6_output: fragment outgoing reassembled skb properly
MAINTAINERS: mv643xx_eth: take over maintainership from Lennart
net_sched: tbf: support of 64bit rates
ixgbe: deleting dfwd stations out of order can cause null ptr deref
ixgbe: fix build err, num_rx_queues is only available with CONFIG_RPS
...
This commit is contained in:
Linus Torvalds
@@ -577,7 +577,7 @@ EXPORT_SYMBOL(skb_copy_datagram_from_iovec);
|
||||
/**
|
||||
* zerocopy_sg_from_iovec - Build a zerocopy datagram from an iovec
|
||||
* @skb: buffer to copy
|
||||
* @from: io vector to copy to
|
||||
* @from: io vector to copy from
|
||||
* @offset: offset in the io vector to start copying from
|
||||
* @count: amount of vectors to copy to buffer from
|
||||
*
|
||||
|
||||
557
net/core/dev.c
557
net/core/dev.c
File diff suppressed because it is too large
Load Diff
@@ -752,7 +752,7 @@ int dev_mc_del_global(struct net_device *dev, const unsigned char *addr)
|
||||
EXPORT_SYMBOL(dev_mc_del_global);
|
||||
|
||||
/**
|
||||
* dev_mc_sync - Synchronize device's unicast list to another device
|
||||
* dev_mc_sync - Synchronize device's multicast list to another device
|
||||
* @to: destination device
|
||||
* @from: source device
|
||||
*
|
||||
@@ -780,7 +780,7 @@ int dev_mc_sync(struct net_device *to, struct net_device *from)
|
||||
EXPORT_SYMBOL(dev_mc_sync);
|
||||
|
||||
/**
|
||||
* dev_mc_sync_multiple - Synchronize device's unicast list to another
|
||||
* dev_mc_sync_multiple - Synchronize device's multicast list to another
|
||||
* device, but allow for multiple calls to sync to multiple devices.
|
||||
* @to: destination device
|
||||
* @from: source device
|
||||
|
||||
@@ -81,6 +81,8 @@ static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN]
|
||||
[NETIF_F_TSO6_BIT] = "tx-tcp6-segmentation",
|
||||
[NETIF_F_FSO_BIT] = "tx-fcoe-segmentation",
|
||||
[NETIF_F_GSO_GRE_BIT] = "tx-gre-segmentation",
|
||||
[NETIF_F_GSO_IPIP_BIT] = "tx-ipip-segmentation",
|
||||
[NETIF_F_GSO_SIT_BIT] = "tx-sit-segmentation",
|
||||
[NETIF_F_GSO_UDP_TUNNEL_BIT] = "tx-udp_tnl-segmentation",
|
||||
[NETIF_F_GSO_MPLS_BIT] = "tx-mpls-segmentation",
|
||||
|
||||
@@ -94,6 +96,7 @@ static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN]
|
||||
[NETIF_F_LOOPBACK_BIT] = "loopback",
|
||||
[NETIF_F_RXFCS_BIT] = "rx-fcs",
|
||||
[NETIF_F_RXALL_BIT] = "rx-all",
|
||||
[NETIF_F_HW_L2FW_DOFFLOAD_BIT] = "l2-fwd-offload",
|
||||
};
|
||||
|
||||
static int ethtool_get_features(struct net_device *dev, void __user *useraddr)
|
||||
|
||||
@@ -460,7 +460,8 @@ static int fib_nl_delrule(struct sk_buff *skb, struct nlmsghdr* nlh)
|
||||
if (frh->action && (frh->action != rule->action))
|
||||
continue;
|
||||
|
||||
if (frh->table && (frh_get_table(frh, tb) != rule->table))
|
||||
if (frh_get_table(frh, tb) &&
|
||||
(frh_get_table(frh, tb) != rule->table))
|
||||
continue;
|
||||
|
||||
if (tb[FRA_PRIORITY] &&
|
||||
|
||||
@@ -25,9 +25,35 @@ static void iph_to_flow_copy_addrs(struct flow_keys *flow, const struct iphdr *i
|
||||
memcpy(&flow->src, &iph->saddr, sizeof(flow->src) + sizeof(flow->dst));
|
||||
}
|
||||
|
||||
/**
|
||||
* skb_flow_get_ports - extract the upper layer ports and return them
|
||||
* @skb: buffer to extract the ports from
|
||||
* @thoff: transport header offset
|
||||
* @ip_proto: protocol for which to get port offset
|
||||
*
|
||||
* The function will try to retrieve the ports at offset thoff + poff where poff
|
||||
* is the protocol port offset returned from proto_ports_offset
|
||||
*/
|
||||
__be32 skb_flow_get_ports(const struct sk_buff *skb, int thoff, u8 ip_proto)
|
||||
{
|
||||
int poff = proto_ports_offset(ip_proto);
|
||||
|
||||
if (poff >= 0) {
|
||||
__be32 *ports, _ports;
|
||||
|
||||
ports = skb_header_pointer(skb, thoff + poff,
|
||||
sizeof(_ports), &_ports);
|
||||
if (ports)
|
||||
return *ports;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
EXPORT_SYMBOL(skb_flow_get_ports);
|
||||
|
||||
bool skb_flow_dissect(const struct sk_buff *skb, struct flow_keys *flow)
|
||||
{
|
||||
int poff, nhoff = skb_network_offset(skb);
|
||||
int nhoff = skb_network_offset(skb);
|
||||
u8 ip_proto;
|
||||
__be16 proto = skb->protocol;
|
||||
|
||||
@@ -42,13 +68,13 @@ ip:
|
||||
iph = skb_header_pointer(skb, nhoff, sizeof(_iph), &_iph);
|
||||
if (!iph || iph->ihl < 5)
|
||||
return false;
|
||||
nhoff += iph->ihl * 4;
|
||||
|
||||
ip_proto = iph->protocol;
|
||||
if (ip_is_fragment(iph))
|
||||
ip_proto = 0;
|
||||
else
|
||||
ip_proto = iph->protocol;
|
||||
|
||||
iph_to_flow_copy_addrs(flow, iph);
|
||||
nhoff += iph->ihl * 4;
|
||||
break;
|
||||
}
|
||||
case __constant_htons(ETH_P_IPV6): {
|
||||
@@ -150,16 +176,7 @@ ipv6:
|
||||
}
|
||||
|
||||
flow->ip_proto = ip_proto;
|
||||
poff = proto_ports_offset(ip_proto);
|
||||
if (poff >= 0) {
|
||||
__be32 *ports, _ports;
|
||||
|
||||
ports = skb_header_pointer(skb, nhoff + poff,
|
||||
sizeof(_ports), &_ports);
|
||||
if (ports)
|
||||
flow->ports = *ports;
|
||||
}
|
||||
|
||||
flow->ports = skb_flow_get_ports(skb, nhoff, ip_proto);
|
||||
flow->thoff = (u16) nhoff;
|
||||
|
||||
return true;
|
||||
@@ -167,6 +184,22 @@ ipv6:
|
||||
EXPORT_SYMBOL(skb_flow_dissect);
|
||||
|
||||
static u32 hashrnd __read_mostly;
|
||||
static __always_inline void __flow_hash_secret_init(void)
|
||||
{
|
||||
net_get_random_once(&hashrnd, sizeof(hashrnd));
|
||||
}
|
||||
|
||||
static __always_inline u32 __flow_hash_3words(u32 a, u32 b, u32 c)
|
||||
{
|
||||
__flow_hash_secret_init();
|
||||
return jhash_3words(a, b, c, hashrnd);
|
||||
}
|
||||
|
||||
static __always_inline u32 __flow_hash_1word(u32 a)
|
||||
{
|
||||
__flow_hash_secret_init();
|
||||
return jhash_1word(a, hashrnd);
|
||||
}
|
||||
|
||||
/*
|
||||
* __skb_get_rxhash: calculate a flow hash based on src/dst addresses
|
||||
@@ -193,9 +226,9 @@ void __skb_get_rxhash(struct sk_buff *skb)
|
||||
swap(keys.port16[0], keys.port16[1]);
|
||||
}
|
||||
|
||||
hash = jhash_3words((__force u32)keys.dst,
|
||||
(__force u32)keys.src,
|
||||
(__force u32)keys.ports, hashrnd);
|
||||
hash = __flow_hash_3words((__force u32)keys.dst,
|
||||
(__force u32)keys.src,
|
||||
(__force u32)keys.ports);
|
||||
if (!hash)
|
||||
hash = 1;
|
||||
|
||||
@@ -231,7 +264,7 @@ u16 __skb_tx_hash(const struct net_device *dev, const struct sk_buff *skb,
|
||||
hash = skb->sk->sk_hash;
|
||||
else
|
||||
hash = (__force u16) skb->protocol;
|
||||
hash = jhash_1word(hash, hashrnd);
|
||||
hash = __flow_hash_1word(hash);
|
||||
|
||||
return (u16) (((u64) hash * qcount) >> 32) + qoffset;
|
||||
}
|
||||
@@ -323,7 +356,7 @@ static inline int get_xps_queue(struct net_device *dev, struct sk_buff *skb)
|
||||
else
|
||||
hash = (__force u16) skb->protocol ^
|
||||
skb->rxhash;
|
||||
hash = jhash_1word(hash, hashrnd);
|
||||
hash = __flow_hash_1word(hash);
|
||||
queue_index = map->queues[
|
||||
((u64)hash * map->len) >> 32];
|
||||
}
|
||||
@@ -378,11 +411,3 @@ struct netdev_queue *netdev_pick_tx(struct net_device *dev,
|
||||
skb_set_queue_mapping(skb, queue_index);
|
||||
return netdev_get_tx_queue(dev, queue_index);
|
||||
}
|
||||
|
||||
static int __init initialize_hashrnd(void)
|
||||
{
|
||||
get_random_bytes(&hashrnd, sizeof(hashrnd));
|
||||
return 0;
|
||||
}
|
||||
|
||||
late_initcall_sync(initialize_hashrnd);
|
||||
|
||||
@@ -100,7 +100,7 @@ int memcpy_toiovecend(const struct iovec *iov, unsigned char *kdata,
|
||||
EXPORT_SYMBOL(memcpy_toiovecend);
|
||||
|
||||
/*
|
||||
* Copy iovec from kernel. Returns -EFAULT on error.
|
||||
* Copy iovec to kernel. Returns -EFAULT on error.
|
||||
*/
|
||||
|
||||
int memcpy_fromiovecend(unsigned char *kdata, const struct iovec *iov,
|
||||
|
||||
@@ -867,7 +867,7 @@ static void neigh_invalidate(struct neighbour *neigh)
|
||||
static void neigh_probe(struct neighbour *neigh)
|
||||
__releases(neigh->lock)
|
||||
{
|
||||
struct sk_buff *skb = skb_peek(&neigh->arp_queue);
|
||||
struct sk_buff *skb = skb_peek_tail(&neigh->arp_queue);
|
||||
/* keep skb alive even if arp_queue overflows */
|
||||
if (skb)
|
||||
skb = skb_copy(skb, GFP_ATOMIC);
|
||||
|
||||
@@ -1263,7 +1263,7 @@ static void netdev_release(struct device *d)
|
||||
BUG_ON(dev->reg_state != NETREG_RELEASED);
|
||||
|
||||
kfree(dev->ifalias);
|
||||
kfree((char *)dev - dev->padded);
|
||||
netdev_freemem(dev);
|
||||
}
|
||||
|
||||
static const void *net_namespace(struct device *d)
|
||||
|
||||
@@ -222,11 +222,10 @@ static void net_prio_attach(struct cgroup_subsys_state *css,
|
||||
struct cgroup_taskset *tset)
|
||||
{
|
||||
struct task_struct *p;
|
||||
void *v;
|
||||
void *v = (void *)(unsigned long)css->cgroup->id;
|
||||
|
||||
cgroup_taskset_for_each(p, css, tset) {
|
||||
task_lock(p);
|
||||
v = (void *)(unsigned long)task_netprioidx(p);
|
||||
iterate_fd(p->files, 0, update_netprio, v);
|
||||
task_unlock(p);
|
||||
}
|
||||
|
||||
@@ -1647,9 +1647,8 @@ int rtnl_configure_link(struct net_device *dev, const struct ifinfomsg *ifm)
|
||||
}
|
||||
|
||||
dev->rtnl_link_state = RTNL_LINK_INITIALIZED;
|
||||
rtmsg_ifinfo(RTM_NEWLINK, dev, ~0U);
|
||||
|
||||
__dev_notify_flags(dev, old_flags);
|
||||
__dev_notify_flags(dev, old_flags, ~0U);
|
||||
return 0;
|
||||
}
|
||||
EXPORT_SYMBOL(rtnl_configure_link);
|
||||
@@ -1985,14 +1984,15 @@ static int rtnl_dump_all(struct sk_buff *skb, struct netlink_callback *cb)
|
||||
return skb->len;
|
||||
}
|
||||
|
||||
void rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change)
|
||||
void rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change,
|
||||
gfp_t flags)
|
||||
{
|
||||
struct net *net = dev_net(dev);
|
||||
struct sk_buff *skb;
|
||||
int err = -ENOBUFS;
|
||||
size_t if_info_size;
|
||||
|
||||
skb = nlmsg_new((if_info_size = if_nlmsg_size(dev, 0)), GFP_KERNEL);
|
||||
skb = nlmsg_new((if_info_size = if_nlmsg_size(dev, 0)), flags);
|
||||
if (skb == NULL)
|
||||
goto errout;
|
||||
|
||||
@@ -2003,7 +2003,7 @@ void rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change)
|
||||
kfree_skb(skb);
|
||||
goto errout;
|
||||
}
|
||||
rtnl_notify(skb, net, 0, RTNLGRP_LINK, NULL, GFP_KERNEL);
|
||||
rtnl_notify(skb, net, 0, RTNLGRP_LINK, NULL, flags);
|
||||
return;
|
||||
errout:
|
||||
if (err < 0)
|
||||
@@ -2717,7 +2717,7 @@ static int rtnetlink_event(struct notifier_block *this, unsigned long event, voi
|
||||
case NETDEV_JOIN:
|
||||
break;
|
||||
default:
|
||||
rtmsg_ifinfo(RTM_NEWLINK, dev, 0);
|
||||
rtmsg_ifinfo(RTM_NEWLINK, dev, 0, GFP_KERNEL);
|
||||
break;
|
||||
}
|
||||
return NOTIFY_DONE;
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
#include <linux/hrtimer.h>
|
||||
#include <linux/ktime.h>
|
||||
#include <linux/string.h>
|
||||
#include <linux/net.h>
|
||||
|
||||
#include <net/secure_seq.h>
|
||||
|
||||
@@ -15,20 +16,9 @@
|
||||
|
||||
static u32 net_secret[NET_SECRET_SIZE] ____cacheline_aligned;
|
||||
|
||||
static void net_secret_init(void)
|
||||
static __always_inline void net_secret_init(void)
|
||||
{
|
||||
u32 tmp;
|
||||
int i;
|
||||
|
||||
if (likely(net_secret[0]))
|
||||
return;
|
||||
|
||||
for (i = NET_SECRET_SIZE; i > 0;) {
|
||||
do {
|
||||
get_random_bytes(&tmp, sizeof(tmp));
|
||||
} while (!tmp);
|
||||
cmpxchg(&net_secret[--i], 0, tmp);
|
||||
}
|
||||
net_get_random_once(net_secret, sizeof(net_secret));
|
||||
}
|
||||
#endif
|
||||
|
||||
|
||||
@@ -476,6 +476,18 @@ void skb_add_rx_frag(struct sk_buff *skb, int i, struct page *page, int off,
|
||||
}
|
||||
EXPORT_SYMBOL(skb_add_rx_frag);
|
||||
|
||||
void skb_coalesce_rx_frag(struct sk_buff *skb, int i, int size,
|
||||
unsigned int truesize)
|
||||
{
|
||||
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
|
||||
|
||||
skb_frag_size_add(frag, size);
|
||||
skb->len += size;
|
||||
skb->data_len += size;
|
||||
skb->truesize += truesize;
|
||||
}
|
||||
EXPORT_SYMBOL(skb_coalesce_rx_frag);
|
||||
|
||||
static void skb_drop_list(struct sk_buff **listp)
|
||||
{
|
||||
kfree_skb_list(*listp);
|
||||
@@ -580,9 +592,6 @@ static void skb_release_head_state(struct sk_buff *skb)
|
||||
#if IS_ENABLED(CONFIG_NF_CONNTRACK)
|
||||
nf_conntrack_put(skb->nfct);
|
||||
#endif
|
||||
#ifdef NET_SKBUFF_NF_DEFRAG_NEEDED
|
||||
nf_conntrack_put_reasm(skb->nfct_reasm);
|
||||
#endif
|
||||
#ifdef CONFIG_BRIDGE_NETFILTER
|
||||
nf_bridge_put(skb->nf_bridge);
|
||||
#endif
|
||||
@@ -903,6 +912,9 @@ EXPORT_SYMBOL(skb_clone);
|
||||
|
||||
static void skb_headers_offset_update(struct sk_buff *skb, int off)
|
||||
{
|
||||
/* Only adjust this if it actually is csum_start rather than csum */
|
||||
if (skb->ip_summed == CHECKSUM_PARTIAL)
|
||||
skb->csum_start += off;
|
||||
/* {transport,network,mac}_header and tail are relative to skb->head */
|
||||
skb->transport_header += off;
|
||||
skb->network_header += off;
|
||||
@@ -1036,8 +1048,8 @@ EXPORT_SYMBOL(__pskb_copy);
|
||||
* @ntail: room to add at tail
|
||||
* @gfp_mask: allocation priority
|
||||
*
|
||||
* Expands (or creates identical copy, if &nhead and &ntail are zero)
|
||||
* header of skb. &sk_buff itself is not changed. &sk_buff MUST have
|
||||
* Expands (or creates identical copy, if @nhead and @ntail are zero)
|
||||
* header of @skb. &sk_buff itself is not changed. &sk_buff MUST have
|
||||
* reference count of 1. Returns zero in the case of success or error,
|
||||
* if expansion failed. In the last case, &sk_buff is not changed.
|
||||
*
|
||||
@@ -1109,9 +1121,6 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
|
||||
#endif
|
||||
skb->tail += off;
|
||||
skb_headers_offset_update(skb, nhead);
|
||||
/* Only adjust this if it actually is csum_start rather than csum */
|
||||
if (skb->ip_summed == CHECKSUM_PARTIAL)
|
||||
skb->csum_start += nhead;
|
||||
skb->cloned = 0;
|
||||
skb->hdr_len = 0;
|
||||
skb->nohdr = 0;
|
||||
@@ -1176,7 +1185,6 @@ struct sk_buff *skb_copy_expand(const struct sk_buff *skb,
|
||||
NUMA_NO_NODE);
|
||||
int oldheadroom = skb_headroom(skb);
|
||||
int head_copy_len, head_copy_off;
|
||||
int off;
|
||||
|
||||
if (!n)
|
||||
return NULL;
|
||||
@@ -1200,11 +1208,7 @@ struct sk_buff *skb_copy_expand(const struct sk_buff *skb,
|
||||
|
||||
copy_skb_header(n, skb);
|
||||
|
||||
off = newheadroom - oldheadroom;
|
||||
if (n->ip_summed == CHECKSUM_PARTIAL)
|
||||
n->csum_start += off;
|
||||
|
||||
skb_headers_offset_update(n, off);
|
||||
skb_headers_offset_update(n, newheadroom - oldheadroom);
|
||||
|
||||
return n;
|
||||
}
|
||||
@@ -1256,6 +1260,29 @@ free_skb:
|
||||
}
|
||||
EXPORT_SYMBOL(skb_pad);
|
||||
|
||||
/**
|
||||
* pskb_put - add data to the tail of a potentially fragmented buffer
|
||||
* @skb: start of the buffer to use
|
||||
* @tail: tail fragment of the buffer to use
|
||||
* @len: amount of data to add
|
||||
*
|
||||
* This function extends the used data area of the potentially
|
||||
* fragmented buffer. @tail must be the last fragment of @skb -- or
|
||||
* @skb itself. If this would exceed the total buffer size the kernel
|
||||
* will panic. A pointer to the first byte of the extra data is
|
||||
* returned.
|
||||
*/
|
||||
|
||||
unsigned char *pskb_put(struct sk_buff *skb, struct sk_buff *tail, int len)
|
||||
{
|
||||
if (tail != skb) {
|
||||
skb->data_len += len;
|
||||
skb->len += len;
|
||||
}
|
||||
return skb_put(tail, len);
|
||||
}
|
||||
EXPORT_SYMBOL_GPL(pskb_put);
|
||||
|
||||
/**
|
||||
* skb_put - add data to a buffer
|
||||
* @skb: buffer to use
|
||||
@@ -1933,9 +1960,8 @@ fault:
|
||||
EXPORT_SYMBOL(skb_store_bits);
|
||||
|
||||
/* Checksum skb data. */
|
||||
|
||||
__wsum skb_checksum(const struct sk_buff *skb, int offset,
|
||||
int len, __wsum csum)
|
||||
__wsum __skb_checksum(const struct sk_buff *skb, int offset, int len,
|
||||
__wsum csum, const struct skb_checksum_ops *ops)
|
||||
{
|
||||
int start = skb_headlen(skb);
|
||||
int i, copy = start - offset;
|
||||
@@ -1946,7 +1972,7 @@ __wsum skb_checksum(const struct sk_buff *skb, int offset,
|
||||
if (copy > 0) {
|
||||
if (copy > len)
|
||||
copy = len;
|
||||
csum = csum_partial(skb->data + offset, copy, csum);
|
||||
csum = ops->update(skb->data + offset, copy, csum);
|
||||
if ((len -= copy) == 0)
|
||||
return csum;
|
||||
offset += copy;
|
||||
@@ -1967,10 +1993,10 @@ __wsum skb_checksum(const struct sk_buff *skb, int offset,
|
||||
if (copy > len)
|
||||
copy = len;
|
||||
vaddr = kmap_atomic(skb_frag_page(frag));
|
||||
csum2 = csum_partial(vaddr + frag->page_offset +
|
||||
offset - start, copy, 0);
|
||||
csum2 = ops->update(vaddr + frag->page_offset +
|
||||
offset - start, copy, 0);
|
||||
kunmap_atomic(vaddr);
|
||||
csum = csum_block_add(csum, csum2, pos);
|
||||
csum = ops->combine(csum, csum2, pos, copy);
|
||||
if (!(len -= copy))
|
||||
return csum;
|
||||
offset += copy;
|
||||
@@ -1989,9 +2015,9 @@ __wsum skb_checksum(const struct sk_buff *skb, int offset,
|
||||
__wsum csum2;
|
||||
if (copy > len)
|
||||
copy = len;
|
||||
csum2 = skb_checksum(frag_iter, offset - start,
|
||||
copy, 0);
|
||||
csum = csum_block_add(csum, csum2, pos);
|
||||
csum2 = __skb_checksum(frag_iter, offset - start,
|
||||
copy, 0, ops);
|
||||
csum = ops->combine(csum, csum2, pos, copy);
|
||||
if ((len -= copy) == 0)
|
||||
return csum;
|
||||
offset += copy;
|
||||
@@ -2003,6 +2029,18 @@ __wsum skb_checksum(const struct sk_buff *skb, int offset,
|
||||
|
||||
return csum;
|
||||
}
|
||||
EXPORT_SYMBOL(__skb_checksum);
|
||||
|
||||
__wsum skb_checksum(const struct sk_buff *skb, int offset,
|
||||
int len, __wsum csum)
|
||||
{
|
||||
const struct skb_checksum_ops ops = {
|
||||
.update = csum_partial_ext,
|
||||
.combine = csum_block_add_ext,
|
||||
};
|
||||
|
||||
return __skb_checksum(skb, offset, len, csum, &ops);
|
||||
}
|
||||
EXPORT_SYMBOL(skb_checksum);
|
||||
|
||||
/* Both of above in one bottle. */
|
||||
@@ -2522,14 +2560,14 @@ EXPORT_SYMBOL(skb_prepare_seq_read);
|
||||
* @data: destination pointer for data to be returned
|
||||
* @st: state variable
|
||||
*
|
||||
* Reads a block of skb data at &consumed relative to the
|
||||
* Reads a block of skb data at @consumed relative to the
|
||||
* lower offset specified to skb_prepare_seq_read(). Assigns
|
||||
* the head of the data block to &data and returns the length
|
||||
* the head of the data block to @data and returns the length
|
||||
* of the block or 0 if the end of the skb data or the upper
|
||||
* offset has been reached.
|
||||
*
|
||||
* The caller is not required to consume all of the data
|
||||
* returned, i.e. &consumed is typically set to the number
|
||||
* returned, i.e. @consumed is typically set to the number
|
||||
* of bytes already consumed and the next call to
|
||||
* skb_seq_read() will return the remaining part of the block.
|
||||
*
|
||||
@@ -2837,14 +2875,7 @@ struct sk_buff *skb_segment(struct sk_buff *skb, netdev_features_t features)
|
||||
__copy_skb_header(nskb, skb);
|
||||
nskb->mac_len = skb->mac_len;
|
||||
|
||||
/* nskb and skb might have different headroom */
|
||||
if (nskb->ip_summed == CHECKSUM_PARTIAL)
|
||||
nskb->csum_start += skb_headroom(nskb) - headroom;
|
||||
|
||||
skb_reset_mac_header(nskb);
|
||||
skb_set_network_header(nskb, skb->mac_len);
|
||||
nskb->transport_header = (nskb->network_header +
|
||||
skb_network_header_len(skb));
|
||||
skb_headers_offset_update(nskb, skb_headroom(nskb) - headroom);
|
||||
|
||||
skb_copy_from_linear_data_offset(skb, -tnl_hlen,
|
||||
nskb->data - tnl_hlen,
|
||||
@@ -2936,32 +2967,30 @@ EXPORT_SYMBOL_GPL(skb_segment);
|
||||
|
||||
int skb_gro_receive(struct sk_buff **head, struct sk_buff *skb)
|
||||
{
|
||||
struct sk_buff *p = *head;
|
||||
struct sk_buff *nskb;
|
||||
struct skb_shared_info *skbinfo = skb_shinfo(skb);
|
||||
struct skb_shared_info *pinfo = skb_shinfo(p);
|
||||
unsigned int headroom;
|
||||
unsigned int len = skb_gro_len(skb);
|
||||
struct skb_shared_info *pinfo, *skbinfo = skb_shinfo(skb);
|
||||
unsigned int offset = skb_gro_offset(skb);
|
||||
unsigned int headlen = skb_headlen(skb);
|
||||
struct sk_buff *nskb, *lp, *p = *head;
|
||||
unsigned int len = skb_gro_len(skb);
|
||||
unsigned int delta_truesize;
|
||||
unsigned int headroom;
|
||||
|
||||
if (p->len + len >= 65536)
|
||||
if (unlikely(p->len + len >= 65536))
|
||||
return -E2BIG;
|
||||
|
||||
if (pinfo->frag_list)
|
||||
goto merge;
|
||||
else if (headlen <= offset) {
|
||||
lp = NAPI_GRO_CB(p)->last ?: p;
|
||||
pinfo = skb_shinfo(lp);
|
||||
|
||||
if (headlen <= offset) {
|
||||
skb_frag_t *frag;
|
||||
skb_frag_t *frag2;
|
||||
int i = skbinfo->nr_frags;
|
||||
int nr_frags = pinfo->nr_frags + i;
|
||||
|
||||
offset -= headlen;
|
||||
|
||||
if (nr_frags > MAX_SKB_FRAGS)
|
||||
return -E2BIG;
|
||||
goto merge;
|
||||
|
||||
offset -= headlen;
|
||||
pinfo->nr_frags = nr_frags;
|
||||
skbinfo->nr_frags = 0;
|
||||
|
||||
@@ -2992,7 +3021,7 @@ int skb_gro_receive(struct sk_buff **head, struct sk_buff *skb)
|
||||
unsigned int first_offset;
|
||||
|
||||
if (nr_frags + 1 + skbinfo->nr_frags > MAX_SKB_FRAGS)
|
||||
return -E2BIG;
|
||||
goto merge;
|
||||
|
||||
first_offset = skb->data -
|
||||
(unsigned char *)page_address(page) +
|
||||
@@ -3010,7 +3039,10 @@ int skb_gro_receive(struct sk_buff **head, struct sk_buff *skb)
|
||||
delta_truesize = skb->truesize - SKB_DATA_ALIGN(sizeof(struct sk_buff));
|
||||
NAPI_GRO_CB(skb)->free = NAPI_GRO_FREE_STOLEN_HEAD;
|
||||
goto done;
|
||||
} else if (skb_gro_len(p) != pinfo->gso_size)
|
||||
}
|
||||
if (pinfo->frag_list)
|
||||
goto merge;
|
||||
if (skb_gro_len(p) != pinfo->gso_size)
|
||||
return -E2BIG;
|
||||
|
||||
headroom = skb_headroom(p);
|
||||
@@ -3062,16 +3094,24 @@ merge:
|
||||
|
||||
__skb_pull(skb, offset);
|
||||
|
||||
NAPI_GRO_CB(p)->last->next = skb;
|
||||
if (!NAPI_GRO_CB(p)->last)
|
||||
skb_shinfo(p)->frag_list = skb;
|
||||
else
|
||||
NAPI_GRO_CB(p)->last->next = skb;
|
||||
NAPI_GRO_CB(p)->last = skb;
|
||||
skb_header_release(skb);
|
||||
lp = p;
|
||||
|
||||
done:
|
||||
NAPI_GRO_CB(p)->count++;
|
||||
p->data_len += len;
|
||||
p->truesize += delta_truesize;
|
||||
p->len += len;
|
||||
|
||||
if (lp != p) {
|
||||
lp->data_len += len;
|
||||
lp->truesize += delta_truesize;
|
||||
lp->len += len;
|
||||
}
|
||||
NAPI_GRO_CB(skb)->same_flow = 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
@@ -475,12 +475,6 @@ discard_and_relse:
|
||||
}
|
||||
EXPORT_SYMBOL(sk_receive_skb);
|
||||
|
||||
void sk_reset_txq(struct sock *sk)
|
||||
{
|
||||
sk_tx_queue_clear(sk);
|
||||
}
|
||||
EXPORT_SYMBOL(sk_reset_txq);
|
||||
|
||||
struct dst_entry *__sk_dst_check(struct sock *sk, u32 cookie)
|
||||
{
|
||||
struct dst_entry *dst = __sk_dst_get(sk);
|
||||
@@ -914,6 +908,13 @@ set_rcvbuf:
|
||||
}
|
||||
break;
|
||||
#endif
|
||||
|
||||
case SO_MAX_PACING_RATE:
|
||||
sk->sk_max_pacing_rate = val;
|
||||
sk->sk_pacing_rate = min(sk->sk_pacing_rate,
|
||||
sk->sk_max_pacing_rate);
|
||||
break;
|
||||
|
||||
default:
|
||||
ret = -ENOPROTOOPT;
|
||||
break;
|
||||
@@ -1177,6 +1178,10 @@ int sock_getsockopt(struct socket *sock, int level, int optname,
|
||||
break;
|
||||
#endif
|
||||
|
||||
case SO_MAX_PACING_RATE:
|
||||
v.val = sk->sk_max_pacing_rate;
|
||||
break;
|
||||
|
||||
default:
|
||||
return -ENOPROTOOPT;
|
||||
}
|
||||
@@ -1836,7 +1841,17 @@ EXPORT_SYMBOL(sock_alloc_send_skb);
|
||||
/* On 32bit arches, an skb frag is limited to 2^15 */
|
||||
#define SKB_FRAG_PAGE_ORDER get_order(32768)
|
||||
|
||||
bool sk_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
|
||||
/**
|
||||
* skb_page_frag_refill - check that a page_frag contains enough room
|
||||
* @sz: minimum size of the fragment we want to get
|
||||
* @pfrag: pointer to page_frag
|
||||
* @prio: priority for memory allocation
|
||||
*
|
||||
* Note: While this allocator tries to use high order pages, there is
|
||||
* no guarantee that allocations succeed. Therefore, @sz MUST be
|
||||
* less or equal than PAGE_SIZE.
|
||||
*/
|
||||
bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t prio)
|
||||
{
|
||||
int order;
|
||||
|
||||
@@ -1845,16 +1860,16 @@ bool sk_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
|
||||
pfrag->offset = 0;
|
||||
return true;
|
||||
}
|
||||
if (pfrag->offset < pfrag->size)
|
||||
if (pfrag->offset + sz <= pfrag->size)
|
||||
return true;
|
||||
put_page(pfrag->page);
|
||||
}
|
||||
|
||||
/* We restrict high order allocations to users that can afford to wait */
|
||||
order = (sk->sk_allocation & __GFP_WAIT) ? SKB_FRAG_PAGE_ORDER : 0;
|
||||
order = (prio & __GFP_WAIT) ? SKB_FRAG_PAGE_ORDER : 0;
|
||||
|
||||
do {
|
||||
gfp_t gfp = sk->sk_allocation;
|
||||
gfp_t gfp = prio;
|
||||
|
||||
if (order)
|
||||
gfp |= __GFP_COMP | __GFP_NOWARN;
|
||||
@@ -1866,6 +1881,15 @@ bool sk_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
|
||||
}
|
||||
} while (--order >= 0);
|
||||
|
||||
return false;
|
||||
}
|
||||
EXPORT_SYMBOL(skb_page_frag_refill);
|
||||
|
||||
bool sk_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
|
||||
{
|
||||
if (likely(skb_page_frag_refill(32U, pfrag, sk->sk_allocation)))
|
||||
return true;
|
||||
|
||||
sk_enter_memory_pressure(sk);
|
||||
sk_stream_moderate_sndbuf(sk);
|
||||
return false;
|
||||
@@ -2319,6 +2343,7 @@ void sock_init_data(struct socket *sock, struct sock *sk)
|
||||
sk->sk_ll_usec = sysctl_net_busy_read;
|
||||
#endif
|
||||
|
||||
sk->sk_max_pacing_rate = ~0U;
|
||||
sk->sk_pacing_rate = ~0U;
|
||||
/*
|
||||
* Before updating sk_refcnt, we must commit prior changes to memory
|
||||
|
||||
@@ -338,3 +338,52 @@ void inet_proto_csum_replace16(__sum16 *sum, struct sk_buff *skb,
|
||||
csum_unfold(*sum)));
|
||||
}
|
||||
EXPORT_SYMBOL(inet_proto_csum_replace16);
|
||||
|
||||
struct __net_random_once_work {
|
||||
struct work_struct work;
|
||||
struct static_key *key;
|
||||
};
|
||||
|
||||
static void __net_random_once_deferred(struct work_struct *w)
|
||||
{
|
||||
struct __net_random_once_work *work =
|
||||
container_of(w, struct __net_random_once_work, work);
|
||||
if (!static_key_enabled(work->key))
|
||||
static_key_slow_inc(work->key);
|
||||
kfree(work);
|
||||
}
|
||||
|
||||
static void __net_random_once_disable_jump(struct static_key *key)
|
||||
{
|
||||
struct __net_random_once_work *w;
|
||||
|
||||
w = kmalloc(sizeof(*w), GFP_ATOMIC);
|
||||
if (!w)
|
||||
return;
|
||||
|
||||
INIT_WORK(&w->work, __net_random_once_deferred);
|
||||
w->key = key;
|
||||
schedule_work(&w->work);
|
||||
}
|
||||
|
||||
bool __net_get_random_once(void *buf, int nbytes, bool *done,
|
||||
struct static_key *done_key)
|
||||
{
|
||||
static DEFINE_SPINLOCK(lock);
|
||||
unsigned long flags;
|
||||
|
||||
spin_lock_irqsave(&lock, flags);
|
||||
if (*done) {
|
||||
spin_unlock_irqrestore(&lock, flags);
|
||||
return false;
|
||||
}
|
||||
|
||||
get_random_bytes(buf, nbytes);
|
||||
*done = true;
|
||||
spin_unlock_irqrestore(&lock, flags);
|
||||
|
||||
__net_random_once_disable_jump(done_key);
|
||||
|
||||
return true;
|
||||
}
|
||||
EXPORT_SYMBOL(__net_get_random_once);
|
||||
|
||||
Reference in New Issue
Block a user