Johannes Weiner 4dd72b4a47 mm: fadvise: avoid expensive remote LRU cache draining after FADV_DONTNEED ... When FADV_DONTNEED cannot drop all pages in the range, it observes that some pages might still be on per-cpu LRU caches after recent instantiation and so initiates remote calls to all CPUs to flush their local caches. However, in most cases, the fadvise happens from the same context that instantiated the pages, and any pre-LRU pages in the specified range are most likely sitting on the local CPU's LRU cache, and so in many cases this results in unnecessary remote calls, which, in a loaded system, can hold up the fadvise() call significantly. [ I didn't record it in the extreme case we observed at Facebook, unfortunately. We had a slow-to-respond system and noticed it lru_add_drain_all() leading the profile during fadvise calls. This patch came out of thinking about the code and how we commonly call FADV_DONTNEED. FWIW, I wrote a silly directory tree walker/searcher that recurses through /usr to read and FADV_DONTNEED each file it finds. On a 2 socket 40 ht machine, over 1% is spent in lru_add_drain_all(). With the patch, that cost is gone; the local drain cost shows at 0.09%. ] Try to avoid the remote call by flushing the local LRU cache before even attempting to invalidate anything. It's a cheap operation, and the local LRU cache is the most likely to hold any pre-LRU pages in the specified fadvise range. Link: http://lkml.kernel.org/r/20161214210017.GA1465@cmpxchg.org Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Acked-by: Vlastimil Babka <vbabka@suse.cz> Acked-by: Mel Gorman <mgorman@suse.de> Acked-by: Hillf Danton <hillf.zj@alibaba-inc.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>		2016-12-20 09:48:46 -08:00
..
kasan	Power management material for v4.10-rc1	2016-12-13 10:41:53 -08:00
backing-dev.c	writeback: track if we're sleeping on progress in balance_dirty_pages()	2016-11-08 08:28:55 -07:00
balloon_compaction.c	mm: balloon: use general non-lru movable page feature	2016-07-26 16:19:19 -07:00
bootmem.c	mm: kmemleak: avoid using __va() on addresses that don't have a lowmem mapping	2016-10-11 15:06:33 -07:00
cleancache.c	cleancache: constify cleancache_ops structure	2016-01-27 09:09:57 -05:00
cma_debug.c
cma.c	mm/cma.c: check the max limit for cma allocation	2016-11-11 08:12:37 -08:00
cma.h
compaction.c	mm, compaction: allow compaction for GFP_NOFS requests	2016-12-14 16:04:07 -08:00
debug_page_ref.c	mm/page_ref: add tracepoint to track down page reference manipulation	2016-03-17 15:09:34 -07:00
debug.c	mm, debug: print raw struct page data in __dump_page()	2016-12-12 18:55:08 -08:00
dmapool.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
early_ioremap.c	mm/early_ioremap: use offset_in_page macro	2015-11-05 19:34:48 -08:00
fadvise.c	mm: fadvise: avoid expensive remote LRU cache draining after FADV_DONTNEED	2016-12-20 09:48:46 -08:00
failslab.c	mm: fault-inject take over bootstrap kmem_cache check	2016-03-15 16:55:16 -07:00
filemap.c	Merge branch 'akpm' (patches from Andrew)	2016-12-14 17:25:18 -08:00
frame_vector.c	mm: replace get_vaddr_frames() write/force parameters with gup_flags	2016-10-19 08:11:24 -07:00
frontswap.c	mm, frontswap: convert frontswap_enabled to static key	2016-07-26 16:19:19 -07:00
gup.c	mm: unexport __get_user_pages_unlocked()	2016-12-14 16:04:09 -08:00
highmem.c	mm/highmem: make nr_free_highpages() handles all highmem zones by itself	2016-05-19 19:12:14 -07:00
huge_memory.c	mm: join struct fault_env and vm_fault	2016-12-14 16:04:09 -08:00
hugetlb_cgroup.c	mm, hugetlb_cgroup: round limit_in_bytes down to hugepage size	2016-05-20 17:58:30 -07:00
hugetlb.c	mm: add tlb_remove_check_page_size_change to track page size change	2016-12-12 18:55:07 -08:00
hwpoison-inject.c	hwpoison: use page_cgroup_ino for filtering by memcg	2015-09-10 13:29:01 -07:00
init-mm.c	mm: Add a user_ns owner to mm_struct and fix ptrace permission checks	2016-11-22 11:49:48 -06:00
internal.h	mm: add orig_pte field into vm_fault	2016-12-14 16:04:09 -08:00
interval_tree.c
Kconfig	mm: THP page cache support for ppc64	2016-12-12 18:55:08 -08:00
Kconfig.debug	PM / Hibernate: allow hibernation with PAGE_POISONING_ZERO	2016-09-13 02:35:27 +02:00
khugepaged.c	radix-tree: improve multiorder iterators	2016-12-14 16:04:10 -08:00
kmemcheck.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
kmemleak-test.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
kmemleak.c	kmemleak: fix reference to Documentation	2016-12-12 18:55:07 -08:00
ksm.c	mm,ksm: add __GFP_HIGH to the allocation in alloc_stable_node()	2016-10-07 18:46:29 -07:00
list_lru.c	mm/list_lru.c: avoid error-path NULL pointer deref	2016-10-27 18:43:42 -07:00
maccess.c	x86: remove more uaccess_32.h complexity	2016-05-22 17:21:27 -07:00
madvise.c	mm: add tlb_remove_check_page_size_change to track page size change	2016-12-12 18:55:07 -08:00
Makefile	Disable the __builtin_return_address() warning globally after all	2016-10-12 10:23:41 -07:00
memblock.c	mm: kmemleak: avoid using __va() on addresses that don't have a lowmem mapping	2016-10-11 15:06:33 -07:00
memcontrol.c	Merge branch 'akpm' (patches from Andrew)	2016-12-12 20:50:02 -08:00
memory_hotplug.c	mm: remove x86-only restriction of movable_node	2016-12-12 18:55:07 -08:00
memory-failure.c	mm: hwpoison: fix thp split handling in memory_failure()	2016-11-11 08:12:37 -08:00
memory.c	Merge branch 'akpm' (patches from Andrew)	2016-12-14 17:25:18 -08:00
mempolicy.c	mm/mempolicy.c: forbid static or relative flags for local NUMA mode	2016-12-12 18:55:07 -08:00
mempool.c	Revert "mm, mempool: only set __GFP_NOMEMALLOC if there are free elements"	2016-07-28 16:07:41 -07:00
memtest.c	memtest: remove unused header files	2015-09-08 15:35:28 -07:00
migrate.c	lib: radix-tree: check accounting of existing slot replacement users	2016-12-12 18:55:08 -08:00
mincore.c	mm, swap: use offset of swap entry as key of swap cache	2016-10-07 18:46:28 -07:00
mlock.c	thp: fix corner case of munlock() of PTE-mapped THPs	2016-11-30 16:32:52 -08:00
mm_init.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
mmap.c	mm: vma_merge: correct false positive from __vma_unlink->validate_mm_rb	2016-10-07 18:46:29 -07:00
mmu_context.c	mm/mmu_context, sched/core: Fix mmu_context.h assumption	2016-04-28 11:44:19 +02:00
mmu_notifier.c	fix Christoph's email addresses	2016-03-17 15:09:34 -07:00
mmzone.c	mm, page_alloc: inline the fast path of the zonelist iterator	2016-05-19 19:12:14 -07:00
mprotect.c	mm/pkeys: generate pkey system call code only if ARCH_HAS_PKEYS is selected	2016-12-12 18:55:07 -08:00
mremap.c	mremap: move_ptes: check pte dirty after its removal	2016-11-29 08:20:24 -08:00
msync.c	mm/msync: use offset_in_page macro	2015-11-05 19:34:48 -08:00
nobootmem.c	mm: kmemleak: avoid using __va() on addresses that don't have a lowmem mapping	2016-10-11 15:06:33 -07:00
nommu.c	Merge branch 'akpm' (patches from Andrew)	2016-12-14 17:25:18 -08:00
oom_kill.c	oom: print nodemask in the oom report	2016-10-07 18:46:29 -07:00
page_alloc.c	mm: add support for releasing multiple instances of a page	2016-12-14 16:04:08 -08:00
page_counter.c	mm: page_counter: let page_counter_try_charge() return bool	2015-11-05 19:34:48 -08:00
page_ext.c	mm/page_ext: support extra space allocation by page_ext user	2016-10-07 18:46:27 -07:00
page_idle.c	mm, vmscan: move lru_lock to the node	2016-07-28 16:07:41 -07:00
page_io.c	writeback: add wbc_to_write_flags()	2016-11-02 10:24:03 -06:00
page_isolation.c	mm/page_isolation: fix typo: "paes" -> "pages"	2016-10-07 18:46:29 -07:00
page_owner.c	mm/page_owner: don't define fields on struct page_ext by hard-coding	2016-10-07 18:46:27 -07:00
page_poison.c	mm: check the return value of lookup_page_ext for all call sites	2016-06-03 15:06:22 -07:00
page-writeback.c	radix-tree: delete radix_tree_range_tag_if_tagged()	2016-12-14 16:04:10 -08:00
pagewalk.c	thp: rename split_huge_page_pmd() to split_huge_pmd()	2016-01-15 17:56:32 -08:00
percpu-km.c	mm: percpu: use pr_fmt to prefix output	2016-03-17 15:09:34 -07:00
percpu-vm.c
percpu.c	Merge branch 'for-4.10' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu	2016-12-13 12:34:47 -08:00
pgtable-generic.c	mm/thp/migration: switch from flush_tlb_range to flush_pmd_tlb_range	2016-03-17 15:09:34 -07:00
process_vm_access.c	mm: unexport __get_user_pages_unlocked()	2016-12-14 16:04:09 -08:00
quicklist.c	fix Christoph's email addresses	2016-03-17 15:09:34 -07:00
readahead.c	mm: don't cap request size based on read-ahead setting	2016-12-12 18:55:08 -08:00
rmap.c	mm, rmap: handle anon_vma_prepare() common case inline	2016-12-12 18:55:08 -08:00
shmem.c	Merge uncontroversial parts of branch 'readlink' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs	2016-12-17 19:16:12 -08:00
slab_common.c	mm/slab_common.c: check kmem_create_cache flags are common	2016-12-12 18:55:06 -08:00
slab.c	Merge branch 'for-4.10' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq	2016-12-13 12:59:57 -08:00
slab.h	mm, slab: maintain total slab count instead of active count	2016-12-12 18:55:07 -08:00
slob.c	slub: move synchronize_sched out of slab_mutex on shrink	2016-12-12 18:55:06 -08:00
slub.c	slub: avoid false-postive warning	2016-12-12 18:55:06 -08:00
sparse-vmemmap.c	treewide: replace obsolete _refok by __ref	2016-08-02 17:31:41 -04:00
sparse.c	treewide: replace obsolete _refok by __ref	2016-08-02 17:31:41 -04:00
swap_cgroup.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
swap_state.c	mm, swap: use offset of swap entry as key of swap cache	2016-10-07 18:46:28 -07:00
swap.c	thp: reduce usage of huge zero page's atomic counter	2016-10-07 18:46:28 -07:00
swapfile.c	mm: add three more cond_resched() in swapoff	2016-12-12 18:55:08 -08:00
truncate.c	mm: workingset: move shadow entry tracking to radix tree exceptional tracking	2016-12-12 18:55:08 -08:00
usercopy.c	mm: usercopy: Check for module addresses	2016-09-20 16:07:39 -07:00
userfaultfd.c	mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros	2016-04-04 10:41:08 -07:00
util.c	Merge branch 'mm-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip	2016-10-22 09:39:10 -07:00
vmacache.c	mm: unrig VMA cache hit ratio	2016-10-07 18:46:27 -07:00
vmalloc.c	mm: add preempt points into __purge_vmap_area_lazy()	2016-12-12 18:55:08 -08:00
vmpressure.c	mm/vmpressure.c: fix subtree pressure detection	2016-02-03 08:28:43 -08:00
vmscan.c	Merge branch 'akpm' (patches from Andrew)	2016-12-12 20:50:02 -08:00
vmstat.c	mm/vmstat: Convert to hotplug state machine	2016-12-02 00:52:35 +01:00
workingset.c	mm: workingset: update shadow limit to reflect bigger active list	2016-12-12 18:55:08 -08:00
z3fold.c	mm/z3fold.c: avoid modifying HEADLESS page and minor cleanup	2016-06-03 16:02:55 -07:00
zbud.c	mm/zbud.c: use list_last_entry() instead of list_tail_entry()	2016-01-15 11:40:52 -08:00
zpool.c	mm: zsmalloc: constify struct zs_pool name	2015-11-06 17:50:42 -08:00
zsmalloc.c	mm/zsmalloc: Convert to hotplug state machine	2016-12-02 00:52:36 +01:00
zswap.c	mm/zswap: Convert pool to hotplug state machine	2016-12-02 00:52:36 +01:00