leandrof/linux
1
0
Fork 0
forked from Minki/linux
Code Issues Pull Requests Packages Projects Releases Wiki Activity
725721a650
linux/net/netfilter
History
NeilBrown 8f0db01800 rhashtable: use bit_spin_locks to protect hash bucket. 
This patch changes rhashtables to use a bit_spin_lock on BIT(1) of the
bucket pointer to lock the hash chain for that bucket.

The benefits of a bit spin_lock are:
 - no need to allocate a separate array of locks.
 - no need to have a configuration option to guide the
   choice of the size of this array
 - locking cost is often a single test-and-set in a cache line
   that will have to be loaded anyway.  When inserting at, or removing
   from, the head of the chain, the unlock is free - writing the new
   address in the bucket head implicitly clears the lock bit.
   For __rhashtable_insert_fast() we ensure this always happens
   when adding a new key.
 - even when lockings costs 2 updates (lock and unlock), they are
   in a cacheline that needs to be read anyway.

The cost of using a bit spin_lock is a little bit of code complexity,
which I think is quite manageable.

Bit spin_locks are sometimes inappropriate because they are not fair -
if multiple CPUs repeatedly contend of the same lock, one CPU can
easily be starved.  This is not a credible situation with rhashtable.
Multiple CPUs may want to repeatedly add or remove objects, but they
will typically do so at different buckets, so they will attempt to
acquire different locks.

As we have more bit-locks than we previously had spinlocks (by at
least a factor of two) we can expect slightly less contention to
go with the slightly better cache behavior and reduced memory
consumption.

To enhance type checking, a new struct is introduced to represent the
  pointer plus lock-bit
that is stored in the bucket-table.  This is "struct rhash_lock_head"
and is empty.  A pointer to this needs to be cast to either an
unsigned lock, or a "struct rhash_head *" to be useful.
Variables of this type are most often called "bkt".

Previously "pprev" would sometimes point to a bucket, and sometimes a
->next pointer in an rhash_head.  As these are now different types,
pprev is NULL when it would have pointed to the bucket. In that case,
'blk' is used, together with correct locking protocol.

Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-04-07 19:12:12 -07:00
..
ipset Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next 2018-12-20 18:20:26 -08:00
ipvs genetlink: make policy common to family 2019-03-22 10:38:23 -04:00
core.c jump_label: move 'asm goto' support test to Kconfig 2019-01-06 09:46:51 +09:00
Kconfig netfilter: fix NETFILTER_XT_TARGET_TEE dependencies 2019-03-18 16:21:54 +01:00
Makefile netfilter: nf_tables: merge ipv4 and ipv6 nat chain types 2019-03-01 14:36:59 +01:00
nf_conncount.c netfilter: nf_conncount: fix argument order to find_next_bit 2018-12-29 02:45:22 +01:00
nf_conntrack_acct.c netfilter: conntrack: remove empty pernet fini stubs 2018-12-21 00:51:54 +01:00
nf_conntrack_amanda.c netfilter: nf_conntrack_amanda: add support for STATE streams 2019-02-27 10:46:39 +01:00
nf_conntrack_broadcast.c netfilter: check if the socket netns is correct. 2018-06-28 22:21:32 +09:00
nf_conntrack_core.c netfilter: nf_conntrack: ensure that CONNTRACK_LOCKS is power of 2 2019-03-01 14:36:46 +01:00
nf_conntrack_ecache.c netfilter: conntrack: remove empty pernet fini stubs 2018-12-21 00:51:54 +01:00
nf_conntrack_expect.c netfilter: conntrack: remove nf_ct_l4proto_find_get 2019-01-18 15:02:34 +01:00
nf_conntrack_extend.c netfilter: conntrack: include kmemleak.h for kmemleak_not_leak() 2018-04-17 10:59:43 +02:00
nf_conntrack_ftp.c netfilter: add __exit mark to helper modules 2018-04-24 10:29:14 +02:00
nf_conntrack_h323_asn1.c netfilter: nf_conntrack_h323: Remove unwanted comments. 2018-01-08 18:01:05 +01:00
nf_conntrack_h323_main.c netfilter: move route indirection to struct nf_ipv6_ops 2018-01-08 18:01:26 +01:00
nf_conntrack_h323_types.c
nf_conntrack_helper.c netfilter: conntrack: remove empty pernet fini stubs 2018-12-21 00:51:54 +01:00
nf_conntrack_irc.c netfilter: add __exit mark to helper modules 2018-04-24 10:29:14 +02:00
nf_conntrack_labels.c netfilter: conntrack: mark extension structs as const 2017-04-26 09:30:22 +02:00
nf_conntrack_netbios_ns.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next 2018-03-30 11:41:18 -04:00
nf_conntrack_netlink.c netfilter: nat: remove nf_nat_l3proto.h and nf_nat_core.h 2019-02-27 10:54:08 +01:00
nf_conntrack_pptp.c netfilter: conntrack: avoid unneeded nf_conntrack_l4proto lookups 2019-01-18 15:02:34 +01:00
nf_conntrack_proto_dccp.c netfilter: conntrack: remove l4proto init and get_net callbacks 2019-01-18 15:02:34 +01:00
nf_conntrack_proto_generic.c netfilter: conntrack: remove l4proto init and get_net callbacks 2019-01-18 15:02:34 +01:00
nf_conntrack_proto_gre.c netfilter: conntrack: remove l4proto destroy hook 2019-01-18 15:02:34 +01:00
nf_conntrack_proto_icmp.c netfilter: conntrack: remove l4proto init and get_net callbacks 2019-01-18 15:02:34 +01:00
nf_conntrack_proto_icmpv6.c netfilter: conntrack: remove l4proto init and get_net callbacks 2019-01-18 15:02:34 +01:00
nf_conntrack_proto_sctp.c netfilter: conntrack: remove l4proto init and get_net callbacks 2019-01-18 15:02:34 +01:00
nf_conntrack_proto_tcp.c netfilter: conntrack: tcp: only close if RST matches exact sequence 2019-03-01 14:19:31 +01:00
nf_conntrack_proto_udp.c netfilter: conntrack: remove l4proto init and get_net callbacks 2019-01-18 15:02:34 +01:00
nf_conntrack_proto.c netfilter: conntrack: fix IPV6=n builds 2019-01-22 23:28:24 +01:00
nf_conntrack_sane.c netfilter: add __exit mark to helper modules 2018-04-24 10:29:14 +02:00
nf_conntrack_seqadj.c netfilter: seqadj: re-load tcp header pointer after possible head reallocation 2018-12-07 10:54:23 +01:00
nf_conntrack_sip.c netfilter: nf_conntrack_sip: remove direct dependency on IPv6 2019-03-18 16:11:54 +01:00
nf_conntrack_snmp.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next 2018-03-30 11:41:18 -04:00
nf_conntrack_standalone.c netfilter: conntrack: fix error path in nf_conntrack_pernet_init() 2019-01-28 11:32:55 +01:00
nf_conntrack_tftp.c netfilter: add __exit mark to helper modules 2018-04-24 10:29:14 +02:00
nf_conntrack_timeout.c netfilter: Export nf_ct_{set,destroy}_timeout() 2019-03-28 16:53:29 -07:00
nf_conntrack_timestamp.c netfilter: conntrack: remove empty pernet fini stubs 2018-12-21 00:51:54 +01:00
nf_dup_netdev.c netfilter: dup: resolve warnings about missing prototypes 2017-05-29 11:32:36 +02:00
nf_flow_table_core.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next 2019-01-28 17:34:38 -08:00
nf_flow_table_inet.c netfilter: nf_flow_table: move init code to nf_flow_table_core.c 2018-04-24 10:28:45 +02:00
nf_flow_table_ip.c netfilter: nf_flow_table: remove unnecessary nat flag check code 2018-09-28 14:28:38 +02:00
nf_internals.h netfilter: core: export raw versions of add/delete hook functions 2018-05-23 09:14:05 +02:00
nf_log_common.c netfilter: avoid using skb->nf_bridge directly 2018-12-19 11:21:37 -08:00
nf_log_netdev.c net: Drop pernet_operations::async 2018-03-27 13:18:09 -04:00
nf_log.c netfilter: nf_log: don't hold nf_log_mutex during user access 2018-06-26 16:48:40 +02:00
nf_nat_amanda.c netfilter: nat: nf_nat_mangle_{udp,tcp}_packet returns boolean 2017-04-06 22:01:38 +02:00
nf_nat_core.c netfilter: remove unneeded switch fall-through 2019-02-27 11:03:59 +01:00
nf_nat_ftp.c netfilter: Replace printk() with pr_*() and define pr_fmt() 2018-03-20 13:44:14 +01:00
nf_nat_helper.c netfilter: nat: remove nf_nat_l3proto.h and nf_nat_core.h 2019-02-27 10:54:08 +01:00
nf_nat_irc.c netfilter: Replace printk() with pr_*() and define pr_fmt() 2018-03-20 13:44:14 +01:00
nf_nat_masquerade.c netfilter: nat: don't register device notifier twice 2019-03-08 16:41:09 +01:00
nf_nat_proto.c netfilter: nat: remove nf_nat_l3proto.h and nf_nat_core.h 2019-02-27 10:54:08 +01:00
nf_nat_redirect.c netfilter: nat: remove unnecessary rcu_read_lock in nf_nat_redirect_ipv{4/6} 2018-09-17 16:11:14 +02:00
nf_nat_sip.c netfilter: nf_nat_sip: fix RTP/RTCP source port translations 2018-12-17 23:43:58 +01:00
nf_nat_tftp.c
nf_queue.c netfilter: avoid using skb->nf_bridge directly 2018-12-19 11:21:37 -08:00
nf_sockopt.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
nf_synproxy_core.c proc: introduce proc_create_net{,_data} 2018-05-16 07:24:30 +02:00
nf_tables_api.c rhashtable: use bit_spin_locks to protect hash bucket. 2019-04-07 19:12:12 -07:00
nf_tables_core.c netfilter: nf_tables: check the result of dereferencing base_chain->stats 2019-03-01 14:34:24 +01:00
nf_tables_set_core.c netfilter: nf_tables: place all set backends in one single module 2018-07-06 19:31:53 +02:00
nf_tables_trace.c netfilter: nf_tables: Allow chain name of up to 255 chars 2017-07-31 20:41:57 +02:00
nfnetlink_acct.c netfilter: fix memory leaks on netlink_dump_start error 2018-08-16 19:37:00 +02:00
nfnetlink_cthelper.c treewide: kzalloc() -> kcalloc() 2018-06-12 16:19:22 -07:00
nfnetlink_cttimeout.c netfilter: conntrack: remove nf_ct_l4proto_find_get 2019-01-18 15:02:34 +01:00
nfnetlink_log.c netfilter: Replace call_rcu_bh(), rcu_barrier_bh(), and synchronize_rcu_bh() 2018-12-01 12:38:23 +01:00
nfnetlink_osf.c netfilter: nfnetlink_osf: add missing fmatch check 2019-01-28 11:09:11 +01:00
nfnetlink_queue.c netfilter: avoid using skb->nf_bridge directly 2018-12-19 11:21:37 -08:00
nfnetlink.c netfilter: nf_tables: use dedicated mutex to guard transactions 2018-07-18 11:26:48 +02:00
nft_bitwise.c netfilter: nf_tables: add direct calls for all builtin expressions 2019-01-18 15:02:33 +01:00
nft_byteorder.c netfilter: nf_tables: add direct calls for all builtin expressions 2019-01-18 15:02:33 +01:00
nft_chain_filter.c netfilter: nf_tables: don't prevent event handler from device cleanup on netns exit 2018-08-16 19:37:03 +02:00
nft_chain_nat.c netfilter: nf_tables: merge ipv4 and ipv6 nat chain types 2019-03-01 14:36:59 +01:00
nft_cmp.c netfilter: nf_tables: add direct calls for all builtin expressions 2019-01-18 15:02:33 +01:00
nft_compat.c netfilter: nft_compat: use .release_ops and remove list of extension 2019-02-27 10:41:24 +01:00
nft_connlimit.c netfilter: nf_conncount: merge lookup and add functions 2018-12-29 02:45:20 +01:00
nft_counter.c netfilter: nft_counter: remove wrong __percpu of nft_counter_resest()'s arg 2019-01-28 11:32:43 +01:00
nft_ct.c netfilter: conntrack: remove nf_ct_l4proto_find_get 2019-01-18 15:02:34 +01:00
nft_dup_netdev.c netfilter: remove two unused variables. 2018-10-19 14:00:33 +02:00
nft_dynset.c netfilter: nf_tables: bogus EBUSY when deleting set after flush 2019-03-11 13:19:24 +01:00
nft_exthdr.c netfilter: nf_tables: merge exthdr expression into nft core 2018-04-27 00:00:56 +02:00
nft_fib_inet.c
nft_fib_netdev.c netfilter: nf_tables: add fib expression to the netdev family 2017-07-31 19:01:40 +02:00
nft_fib.c
nft_flow_offload.c netfilter: nft_flow_offload: fix checking method of conntrack helper 2019-01-14 12:50:59 +01:00
nft_fwd_netdev.c netfilter: remove two unused variables. 2018-10-19 14:00:33 +02:00
nft_hash.c Revert "netfilter: nft_hash: add map lookups for hashing operations" 2019-01-18 15:59:47 +01:00
nft_immediate.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2019-02-08 15:00:17 -08:00
nft_limit.c netfilter: nft_limit: fix packet ratelimiting 2018-05-23 09:50:28 +02:00
nft_log.c netfilter: nf_tables: add NFT_LOGLEVEL_* enumeration and use it 2018-06-07 16:14:00 -04:00
nft_lookup.c netfilter: nf_tables: bogus EBUSY when deleting set after flush 2019-03-11 13:19:24 +01:00
nft_masq.c netfilter: nf_tables: nat: merge nft_masq protocol specific modules 2019-03-01 14:36:59 +01:00
nft_meta.c netfilter: nft_meta: Add NFT_META_I/OIFKIND meta type 2019-01-18 15:58:20 +01:00
nft_nat.c netfilter: nat: remove nf_nat_l3proto.h and nf_nat_core.h 2019-02-27 10:54:08 +01:00
nft_numgen.c Revert "netfilter: nft_numgen: add map lookups for numgen random operations" 2018-10-29 11:11:33 +01:00
nft_objref.c netfilter: nf_tables: bogus EBUSY in helper removal from transaction 2019-03-18 16:22:49 +01:00
nft_osf.c netfilter: nft_osf: check if attribute is present 2018-10-25 10:18:31 +02:00
nft_payload.c netfilter: nf_tables: add direct calls for all builtin expressions 2019-01-18 15:02:33 +01:00
nft_queue.c netfilter: Remove exceptional & on function name 2017-04-07 18:24:47 +02:00
nft_quota.c netfilter: nf_tables: prepare nft_object for lookups via hashtable 2019-01-18 15:02:32 +01:00
nft_range.c netfilter: nf_tables: add direct calls for all builtin expressions 2019-01-18 15:02:33 +01:00
nft_redir.c netfilter: nft_redir: fix module autoload with ip4 2019-03-18 16:22:49 +01:00
nft_reject_inet.c
nft_reject.c netfilter: nf_tables: avoid BUG_ON usage 2018-09-17 16:11:12 +02:00
nft_rt.c netfilter: nf_tables: add direct calls for all builtin expressions 2019-01-18 15:02:33 +01:00
nft_set_bitmap.c netfilter: nft_set: fix allocation size overflow in privsize callback. 2018-08-16 19:36:59 +02:00
nft_set_hash.c netfilter: nft_set_hash: remove nft_hash_key() 2019-02-27 11:08:32 +01:00
nft_set_rbtree.c netfilter: nft_set_rbtree: check for inactive element after flag mismatch 2019-03-18 16:21:09 +01:00
nft_socket.c netfilter: nft_socket: Expose socket mark 2018-07-18 11:26:52 +02:00
nft_tproxy.c netfilter: nft_tproxy: Fix missing-braces warning 2018-08-16 19:37:10 +02:00
nft_tunnel.c netfilter: nft_tunnel: Add dst_cache support 2019-03-01 14:25:06 +01:00
nft_xfrm.c net: use skb_sec_path helper in more places 2018-12-19 11:21:37 -08:00
utils.c netfilter: ipv6: avoid indirect calls for IPV6=y case 2019-02-04 18:21:12 +01:00
x_tables.c netfilter: convert the proto argument from u8 to u16 2019-03-01 14:28:43 +01:00
xt_addrtype.c netfilter: ipv6: avoid indirect calls for IPV6=y case 2019-02-04 18:21:12 +01:00
xt_AUDIT.c audit: eliminate audit_enabled magic number comparison 2018-06-19 10:43:55 -04:00
xt_bpf.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_cgroup.c netfilter: xt_cgroup: shrink size of v2 path 2018-09-17 16:11:03 +02:00
xt_CHECKSUM.c netfilter: xt_checksum: ignore gso skbs 2018-08-24 09:58:16 +02:00
xt_CLASSIFY.c
xt_cluster.c netfilter: xt_cluster: add dependency on conntrack module 2018-08-23 20:26:53 +02:00
xt_comment.c
xt_connbytes.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_connlabel.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_connlimit.c netfilter: use PTR_ERR_OR_ZERO() 2018-07-30 14:07:09 +02:00
xt_connmark.c netfilter: xt_connmark: fix list corruption on rmmod 2018-06-12 19:35:52 +02:00
xt_CONNSECMARK.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_conntrack.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_cpu.c
xt_CT.c netfilter: Export nf_ct_{set,destroy}_timeout() 2019-03-28 16:53:29 -07:00
xt_dccp.c
xt_devgroup.c
xt_dscp.c netfilter: x_tables: remove pr_info where possible 2018-02-14 21:05:33 +01:00
xt_DSCP.c netfilter: x_tables: remove pr_info where possible 2018-02-14 21:05:33 +01:00
xt_ecn.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_esp.c
xt_hashlimit.c mm: convert totalram_pages and totalhigh_pages variables to atomic 2018-12-28 12:11:47 -08:00
xt_helper.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_hl.c
xt_HL.c netfilter: x_tables: remove pr_info where possible 2018-02-14 21:05:33 +01:00
xt_HMARK.c netfilter: x_tables: use pr ratelimiting in matches/targets 2018-02-14 21:05:37 +01:00
xt_IDLETIMER.c netfilter: xt_IDLETIMER: fix sysfs callback function type 2019-03-01 14:36:57 +01:00
xt_ipcomp.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_iprange.c
xt_ipvs.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_l2tp.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_LED.c netfilter: x_tables: fix missing timer initialization in xt_LED 2018-02-14 21:05:39 +01:00
xt_length.c
xt_limit.c netfilter: xt_limit: Spelling s/maxmum/maximum/ 2018-03-05 23:15:50 +01:00
xt_LOG.c
xt_mac.c
xt_mark.c
xt_multiport.c
xt_nat.c netfilter: nat: remove nf_nat_l3proto.h and nf_nat_core.h 2019-02-27 10:54:08 +01:00
xt_NETMAP.c netfilter: add NAT support for shifted portmap ranges 2018-04-24 10:29:12 +02:00
xt_nfacct.c netfilter: nfnetlink_acct: remove useless parameter 2018-03-05 23:15:43 +01:00
xt_NFLOG.c netfilter: xt_NFLOG: use nf_log_packet instead of nfulnl_log_packet. 2018-04-19 13:02:44 +02:00
xt_NFQUEUE.c netfilter: xt_NFQUEUE: use pr ratelimiting 2018-02-14 21:05:35 +01:00
xt_osf.c netfilter: xt_osf: simplify xt_osf_match_packet() 2018-10-16 10:01:50 +02:00
xt_owner.c netfilter: check if the socket netns is correct. 2018-06-28 22:21:32 +09:00
xt_physdev.c netfilter: physdev: relax br_netfilter dependency 2019-01-18 15:02:33 +01:00
xt_pkttype.c
xt_policy.c net: use skb_sec_path helper in more places 2018-12-19 11:21:37 -08:00
xt_quota.c Revert "netfilter: xt_quota: fix the behavior of xt_quota module" 2018-10-19 14:00:34 +02:00
xt_rateest.c netfilter: make xt_rateest hash table per net 2018-03-05 23:15:44 +01:00
xt_RATEEST.c netfilter: xt_RATEEST: remove netns exit routine 2018-11-13 09:57:29 +01:00
xt_realm.c
xt_recent.c netfilter: xt_recent: Use struct_size() in kvzalloc() 2019-02-12 00:39:39 +01:00
xt_REDIRECT.c netfilter: add NAT support for shifted portmap ranges 2018-04-24 10:29:12 +02:00
xt_repldata.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
xt_sctp.c sctp: remove the typedef sctp_chunkhdr_t 2017-07-01 09:08:41 -07:00
xt_SECMARK.c netfilter: xtables: avoid BUG_ON 2018-09-17 16:11:12 +02:00
xt_set.c netfilter: ipset: Limit max timeout value 2018-06-06 14:00:54 +02:00
xt_socket.c netfilter: xt_socket: check sk before checking for netns. 2018-09-28 14:47:41 +02:00
xt_state.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_statistic.c netfilter: x_tables: fix pointer leaks to userspace 2018-01-31 14:59:24 +01:00
xt_string.c netfilter: ebtables: Add string filter 2018-03-30 11:04:12 +02:00
xt_tcpmss.c
xt_TCPMSS.c netfilter: x_tables: use pr ratelimiting in all remaining spots 2018-02-14 21:05:38 +01:00
xt_TCPOPTSTRIP.c
xt_tcpudp.c
xt_TEE.c netfilter: xt_TEE: add missing code to get interface index in checkentry. 2018-10-11 11:29:14 +02:00
xt_time.c netfilter: Replace printk() with pr_*() and define pr_fmt() 2018-03-20 13:44:14 +01:00
xt_TPROXY.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next 2018-07-20 22:28:28 -07:00
xt_TRACE.c
xt_u32.c