linux/arch
Jason Gunthorpe 57efa1fe59 mm/gup: prevent gup_fast from racing with COW during fork
Since commit 70e806e4e6 ("mm: Do early cow for pinned pages during
fork() for ptes") pages under a FOLL_PIN will not be write protected
during COW for fork.  This means that pages returned from
pin_user_pages(FOLL_WRITE) should not become write protected while the pin
is active.

However, there is a small race where get_user_pages_fast(FOLL_PIN) can
establish a FOLL_PIN at the same time copy_present_page() is write
protecting it:

        CPU 0                             CPU 1
   get_user_pages_fast()
    internal_get_user_pages_fast()
                                       copy_page_range()
                                         pte_alloc_map_lock()
                                           copy_present_page()
                                             atomic_read(has_pinned) == 0
					     page_maybe_dma_pinned() == false
     atomic_set(has_pinned, 1);
     gup_pgd_range()
      gup_pte_range()
       pte_t pte = gup_get_pte(ptep)
       pte_access_permitted(pte)
       try_grab_compound_head()
                                             pte = pte_wrprotect(pte)
	                                     set_pte_at();
                                         pte_unmap_unlock()
      // GUP now returns with a write protected page

The first attempt to resolve this by using the write protect caused
problems (and was missing a barrrier), see commit f3c64eda3e ("mm: avoid
early COW write protect games during fork()")

Instead wrap copy_p4d_range() with the write side of a seqcount and check
the read side around gup_pgd_range().  If there is a collision then
get_user_pages_fast() fails and falls back to slow GUP.

Slow GUP is safe against this race because copy_page_range() is only
called while holding the exclusive side of the mmap_lock on the src
mm_struct.

[akpm@linux-foundation.org: coding style fixes]
  Link: https://lore.kernel.org/r/CAHk-=wi=iCnYCARbPGjkVJu9eyYeZ13N64tZYLdOB8CP5Q_PLw@mail.gmail.com

Link: https://lkml.kernel.org/r/2-v4-908497cf359a+4782-gup_fork_jgg@nvidia.com
Fixes: f3c64eda3e ("mm: avoid early COW write protect games during fork()")
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Peter Xu <peterx@redhat.com>
Acked-by: "Ahmed S. Darwish" <a.darwish@linutronix.de>	[seqcount_t parts]
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jann Horn <jannh@google.com>
Cc: Kirill Shutemov <kirill@shutemov.name>
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-12-15 12:13:39 -08:00
..
alpha sched/idle: Fix arch_cpu_idle() vs tracing 2020-11-24 16:47:35 +01:00
arc asm-generic: add correct MAX_POSSIBLE_PHYSMEM_BITS setting 2020-11-27 15:00:35 -08:00
arm ARM: dts: mmp2-olpc-xo-1-75: clear the warnings when make dtbs 2020-12-09 23:00:14 +01:00
arm64 Bugfixes for ARM, x86 and tools. 2020-12-12 10:08:16 -08:00
c6x arch-cleanup-2020-10-22 2020-10-23 10:06:38 -07:00
csky Yet two more places which invoke tracing from RCU disabled regions in the 2020-11-29 11:19:26 -08:00
h8300 sched/idle: Fix arch_cpu_idle() vs tracing 2020-11-24 16:47:35 +01:00
hexagon sched/idle: Fix arch_cpu_idle() vs tracing 2020-11-24 16:47:35 +01:00
ia64 Yet two more places which invoke tracing from RCU disabled regions in the 2020-11-29 11:19:26 -08:00
m68k arch-cleanup-2020-10-22 2020-10-23 10:06:38 -07:00
microblaze sched/idle: Fix arch_cpu_idle() vs tracing 2020-11-24 16:47:35 +01:00
mips Yet two more places which invoke tracing from RCU disabled regions in the 2020-11-29 11:19:26 -08:00
nds32 arch-cleanup-2020-10-22 2020-10-23 10:06:38 -07:00
nios2 sched/idle: Fix arch_cpu_idle() vs tracing 2020-11-24 16:47:35 +01:00
openrisc sched/idle: Fix arch_cpu_idle() vs tracing 2020-11-24 16:47:35 +01:00
parisc sched/idle: Fix arch_cpu_idle() vs tracing 2020-11-24 16:47:35 +01:00
powerpc powerpc fixes for 5.10 #6 2020-12-10 16:36:30 -08:00
riscv RISC-V Fixes for 5.10 (unless there's an rc8) 2020-12-12 09:50:26 -08:00
s390 mm/gup_benchmark: rename to mm/gup_test 2020-12-15 12:13:38 -08:00
sh sched/idle: Fix arch_cpu_idle() vs tracing 2020-11-24 16:47:35 +01:00
sparc Merge branch 'fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs 2020-12-08 15:03:39 -08:00
um Yet two more places which invoke tracing from RCU disabled regions in the 2020-11-29 11:19:26 -08:00
x86 mm/gup: prevent gup_fast from racing with COW during fork 2020-12-15 12:13:39 -08:00
xtensa xtensa: uaccess: Add missing __user to strncpy_from_user() prototype 2020-11-17 05:09:28 -08:00
.gitignore
Kconfig arch/Kconfig: fix spelling mistakes 2020-12-15 12:13:37 -08:00