Minki/linux
1
0
Fork 1
mirror of https://github.com/torvalds/linux.git synced 2024-11-25 05:32:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

nfs: avoid i_lock contention in nfs_clear_invalid_mapping

Browse Source 
Multi-threaded buffered reads to the same file exposed significant
inode spinlock contention in nfs_clear_invalid_mapping().

Eliminate this spinlock contention by checking flags without locking,
instead using smp_rmb and smp_load_acquire accordingly, but then take
spinlock and double-check these inode flags.

Also refactor nfs_set_cache_invalid() slightly to use
smp_store_release() to pair with nfs_clear_invalid_mapping()'s
smp_load_acquire().

While this fix is beneficial for all multi-threaded buffered reads
issued by an NFS client, this issue was identified in the context of
surprisingly low LOCALIO performance with 4K multi-threaded buffered
read IO.  This fix dramatically speeds up LOCALIO performance:

before: read: IOPS=1583k, BW=6182MiB/s (6482MB/s)(121GiB/20002msec)
after:  read: IOPS=3046k, BW=11.6GiB/s (12.5GB/s)(232GiB/20001msec)

Fixes: 17dfeb9113 ("NFS: Fix races in nfs_revalidate_mapping")
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Anna Schumaker <anna.schumaker@oracle.com>
This commit is contained in:
Mike Snitzer 2024-10-18 17:15:41 -04:00 committed by Anna Schumaker
parent bc29408695
commit 867da60d46
1 changed files with 15 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
fs/nfs/inode.c
View File

@ -205,12 +205,15 @@ void nfs_set_cache_invalid(struct inode *inode, unsigned long flags)
nfs_fscache_invalidate(inode, 0); nfs_fscache_invalidate(inode, 0);
flags &= ~NFS_INO_REVAL_FORCED; flags &= ~NFS_INO_REVAL_FORCED;
nfsi->cache_validity |= flags; flags |= nfsi->cache_validity;
if (inode->i_mapping->nrpages == 0)
flags &= ~NFS_INO_INVALID_DATA;
if (inode->i_mapping->nrpages == 0) { /* pairs with nfs_clear_invalid_mapping()'s smp_load_acquire() */
nfsi->cache_validity &= ~NFS_INO_INVALID_DATA; smp_store_release(&nfsi->cache_validity, flags);
nfs_ooo_clear(nfsi);
} else if (nfsi->cache_validity & NFS_INO_INVALID_DATA) { if (inode->i_mapping->nrpages == 0 ||
nfsi->cache_validity & NFS_INO_INVALID_DATA) {
nfs_ooo_clear(nfsi); nfs_ooo_clear(nfsi);
} }
trace_nfs_set_cache_invalid(inode, 0); trace_nfs_set_cache_invalid(inode, 0);
@ -1421,6 +1424,13 @@ int nfs_clear_invalid_mapping(struct address_space *mapping)
TASK_KILLABLE|TASK_FREEZABLE_UNSAFE); TASK_KILLABLE|TASK_FREEZABLE_UNSAFE);
if (ret) if (ret)
goto out; goto out;
smp_rmb(); /* pairs with smp_wmb() below */
if (test_bit(NFS_INO_INVALIDATING, bitlock))
continue;
/* pairs with nfs_set_cache_invalid()'s smp_store_release() */
if (!(smp_load_acquire(&nfsi->cache_validity) & NFS_INO_INVALID_DATA))
goto out;
/* Slow-path that double-checks with spinlock held */
spin_lock(&inode->i_lock); spin_lock(&inode->i_lock);
if (test_bit(NFS_INO_INVALIDATING, bitlock)) { if (test_bit(NFS_INO_INVALIDATING, bitlock)) {
spin_unlock(&inode->i_lock); spin_unlock(&inode->i_lock);