leandrof/linux
1
0
Fork 0
forked from Minki/linux
Code Issues Pull Requests Packages Projects Releases Wiki Activity
f3f601c1d2
linux/arch/sparc/include/asm
History
David S. Miller f36391d279 sparc64: Fix race in TLB batch processing. 
As reported by Dave Kleikamp, when we emit cross calls to do batched
TLB flush processing we have a race because we do not synchronize on
the sibling cpus completing the cross call.

So meanwhile the TLB batch can be reset (tb->tlb_nr set to zero, etc.)
and either flushes are missed or flushes will flush the wrong
addresses.

Fix this by using generic infrastructure to synchonize on the
completion of the cross call.

This first required getting the flush_tlb_pending() call out from
switch_to() which operates with locks held and interrupts disabled.
The problem is that smp_call_function_many() cannot be invoked with
IRQs disabled and this is explicitly checked for with WARN_ON_ONCE().

We get the batch processing outside of locked IRQ disabled sections by
using some ideas from the powerpc port. Namely, we only batch inside
of arch_{enter,leave}_lazy_mmu_mode() calls.  If we're not in such a
region, we flush TLBs synchronously.

1) Get rid of xcall_flush_tlb_pending and per-cpu type
   implementations.

2) Do TLB batch cross calls instead via:

	smp_call_function_many()
		tlb_pending_func()
			__flush_tlb_pending()

3) Batch only in lazy mmu sequences:

	a) Add 'active' member to struct tlb_batch
	b) Define __HAVE_ARCH_ENTER_LAZY_MMU_MODE
	c) Set 'active' in arch_enter_lazy_mmu_mode()
	d) Run batch and clear 'active' in arch_leave_lazy_mmu_mode()
	e) Check 'active' in tlb_batch_add_one() and do a synchronous
           flush if it's clear.

4) Add infrastructure for synchronous TLB page flushes.

	a) Implement __flush_tlb_page and per-cpu variants, patch
	   as needed.
	b) Likewise for xcall_flush_tlb_page.
	c) Implement smp_flush_tlb_page() to invoke the cross-call.
	d) Wire up global_flush_tlb_page() to the right routine based
           upon CONFIG_SMP

5) It turns out that singleton batches are very common, 2 out of every
   3 batch flushes have only a single entry in them.

   The batch flush waiting is very expensive, both because of the poll
   on sibling cpu completeion, as well as because passing the tlb batch
   pointer to the sibling cpus invokes a shared memory dereference.

   Therefore, in flush_tlb_pending(), if there is only one entry in
   the batch perform a completely asynchronous global_flush_tlb_page()
   instead.

Reported-by: Dave Kleikamp <dave.kleikamp@oracle.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Acked-by: Dave Kleikamp <dave.kleikamp@oracle.com>
2013-04-19 17:26:26 -04:00
..
agp.h agp: kill phys_to_gart() and gart_to_phys() 2009-08-03 09:05:00 +01:00
apb.h
asm-offsets.h kbuild: move asm-offsets.h to include/generated 2009-12-12 13:08:14 +01:00
asm.h sparc: Add asm/asm.h 2008-12-09 00:50:13 -08:00
asmmacro.h sparc32: add support for run-time patching of leon/sun single instructions 2012-05-27 23:52:43 -07:00
atomic_32.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
atomic_64.h sparc: Support atomic64_dec_if_positive properly. 2012-11-09 19:37:59 -08:00
atomic.h
auxio_32.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
auxio_64.h
auxio.h
backoff.h sparc64: Improvde documentation and readability of atomic backoff code. 2012-10-28 13:04:47 -07:00
barrier_32.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
barrier_64.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
barrier.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
bbc.h
bitext.h
bitops_32.h bitops: remove minix bitops from asm/bitops.h 2011-03-23 19:46:22 -07:00
bitops_64.h sparc: Use popc when possible for ffs/__ffs/ffz. 2011-08-02 21:28:53 -07:00
bitops.h
btext.h sparc64: Faster early-boot framebuffer console. 2009-11-27 17:33:43 -08:00
bug.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
bugs.h sparc64: Implement SSTATE purely using notifiers and initcalls. 2008-09-02 00:49:38 -07:00
cache.h sparc32: Remove completely unused code from asm/cache.h 2012-05-19 21:15:37 -07:00
cacheflush_32.h sparc32: Move cache and TLB flushes over to method ops. 2012-05-13 20:49:31 -07:00
cacheflush_64.h sparc32: use flushi when run-time patching in per_cpu_patch 2012-05-20 13:33:36 -07:00
cacheflush.h sparc32: use flushi when run-time patching in per_cpu_patch 2012-05-20 13:33:36 -07:00
cachetlb_32.h sparc32: Move cache and TLB flushes over to method ops. 2012-05-13 20:49:31 -07:00
chafsr.h
checksum_32.h
checksum_64.h
checksum.h
chmctrl.h
clock.h
cmpxchg_32.h sparc32: remove runtime btfix support 2012-05-14 14:05:09 -07:00
cmpxchg_64.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
cmpxchg.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
compat_signal.h sparc: switch to generic sigaltstack 2013-02-03 18:16:17 -05:00
compat.h sparc64: Make montmul/montsqr/mpmul usable in 32-bit threads. 2012-10-26 15:18:37 -07:00
contregs.h sparc32: Remove sun4{,c} control reg definitions from contregs.h. 2012-05-11 21:29:17 -07:00
cpu_type.h sparc32: Remove sun4 and sun4c from enum sparc_cpu. 2012-05-12 00:23:23 -07:00
cpudata_32.h sparc32: generic clockevent support 2012-04-15 10:28:50 -07:00
cpudata_64.h sparc64: Use a seperate counter for timer interrupts and NMI checks, like x86. 2010-04-12 22:37:07 -07:00
cpudata.h
current.h
dcr.h
dcu.h
delay_32.h
delay_64.h
delay.h
device.h of: remove asm/of_device.h 2010-07-24 09:57:52 -06:00
dma-mapping.h sparc: dma-mapping: support debug_dma_mapping_error 2012-11-17 13:20:38 +01:00
dma.h sparc32: Convert mmu_* interfaces from btfixup to method ops. 2012-05-13 13:57:05 -07:00
ebus_dma.h sparc: Move EBUS DMA interfaces into seperate header file. 2008-08-29 23:10:21 -07:00
ecc.h
eeprom.h
elf_32.h default SET_PERSONALITY() in linux/elf.h 2013-02-26 02:46:08 -05:00
elf_64.h sparc64: Add detection for features new in SPARC-T4. 2012-08-18 23:02:36 -07:00
elf.h
estate.h
fb.h sparc: Really fix "console=" for serial consoles. 2010-08-16 12:26:09 -07:00
fbio.h UAPI: (Scripted) Disintegrate arch/sparc/include/asm 2012-10-09 09:47:43 +01:00
fhc.h sparc64: Rewrite central driver. 2008-08-31 20:56:15 -07:00
floppy_32.h sparc: Kill mmu_{un,}lockarea(). 2012-05-13 13:23:16 -07:00
floppy_64.h sparc64: Remove pointless assignment in floppy support. 2012-05-14 20:59:25 -07:00
floppy.h
fpumacro.h
ftrace.h sparc64: Add proper dynamic ftrace support. 2009-06-16 04:56:53 -07:00
futex_32.h
futex_64.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
futex.h
gpio.h gpiolib/arches: Centralise bolierplate asm/gpio.h 2012-05-11 18:00:14 -06:00
hardirq_32.h sparc32: convert to asm-generic/hardirq.h 2009-10-01 13:45:13 -07:00
hardirq_64.h
hardirq.h
head_32.h sparc32: Kill unused defines from asm/head_32.h 2012-05-11 21:43:11 -07:00
head_64.h
head.h
highmem.h sparc32: move kmap_init() to highmem.c 2012-07-26 16:46:17 -07:00
hugetlb.h sparc64: Fix tsb_grow() in atomic context. 2013-02-20 09:46:08 -08:00
hvtramp.h
hw_irq.h
hypervisor.h sparc64: Add hypervisor interfaces for SPARC-T4 perf counter access. 2012-08-18 23:03:53 -07:00
ide.h
idprom.h
intr_queue.h
io_32.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
io_64.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
io-unit.h sparc32: Kill iounit_map_dma_*(). 2008-08-29 02:15:23 -07:00
io.h sparc: add {read,write}*_be routines 2011-05-12 13:44:29 -07:00
ioctls.h UAPI: (Scripted) Disintegrate arch/sparc/include/asm 2012-10-09 09:47:43 +01:00
iommu_32.h sparc: Remove generic SBUS probing layer. 2008-08-29 02:15:21 -07:00
iommu_64.h sparc64: Get rid of pci_controller_info. 2008-09-10 23:07:41 -07:00
iommu.h
irq_32.h sparc32: genirq support 2011-04-19 22:11:40 -07:00
irq_64.h nmi watchdog: do not use cpp symbol in Kconfig 2012-03-23 16:58:31 -07:00
irq.h
irqflags_32.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
irqflags_64.h sparc/irqs: Do not trace arch_local_{*,irq_*} functions 2011-07-06 08:00:29 -07:00
irqflags.h
jump_label.h static keys: Introduce 'struct static_key', static_key_true()/false() and static_key_slow_[inc|dec]() 2012-02-24 10:05:59 +01:00
Kbuild sparc: use asm-generic version of types.h 2013-03-31 19:29:13 -04:00
kdebug_32.h sparc32: kernel/trace/trace.c wants DIE_OOPS 2008-11-01 21:44:01 -07:00
kdebug_64.h sparc64: Implement NMI watchdog on capable cpus. 2009-01-30 00:03:53 -08:00
kdebug.h
kgdb.h
kmap_types.h kmap_types: make most arches use generic header file 2009-06-16 19:47:51 -07:00
kprobes.h
ldc.h
leon_amba.h sparc32,leon: always include leon_kernel in build 2012-05-27 23:52:46 -07:00
leon_pci.h PCI: Make the struct pci_dev * argument of pci_fixup_irqs const. 2011-07-22 08:26:06 -07:00
leon.h sparc32,leon: drop leon_init() 2012-07-26 16:46:16 -07:00
linkage.h
lsu.h
machines.h sparc32: Trivial removal of sun4c references in comments. 2012-05-12 00:35:45 -07:00
mbus.h sparc32: Remove cypress cpu support. 2012-05-15 10:22:00 -07:00
mc146818rtc_32.h
mc146818rtc_64.h sparc64: Use generic CMOS driver. 2008-08-29 14:16:48 -07:00
mc146818rtc.h
mdesc.h sparc64: Probe cpu page size support more portably. 2012-09-06 19:01:25 -07:00
memctrl.h sparc64: Add generic interface for registering a dimm printing handler. 2008-08-24 22:08:34 -07:00
mman.h UAPI: (Scripted) Disintegrate arch/sparc/include/asm 2012-10-09 09:47:43 +01:00
mmu_32.h sparc32: add irq + smp declarations to headers 2011-03-16 18:19:08 -07:00
mmu_64.h sparc64: Support transparent huge pages. 2012-10-09 16:23:06 +09:00
mmu_context_32.h sparc32: centralize all mmu context handling in srmmu.c 2012-07-26 16:46:16 -07:00
mmu_context_64.h sparc64: Support transparent huge pages. 2012-10-09 16:23:06 +09:00
mmu_context.h
mmu.h
mmzone.h Fix node_start/end_pfn() definition for mm/page_cgroup.c 2011-06-27 14:13:09 -07:00
msi.h
mxcc.h
nmi.h sparc64: Initial hw perf counter support. 2009-09-10 06:28:20 -07:00
ns87303.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
obio.h sparc32: move smp ipi to method ops 2012-05-14 14:05:07 -07:00
openprom.h sparc: Pass buffer pointer all the way down to prom_{get,put}char(). 2010-11-30 14:33:29 -08:00
oplib_32.h sparc: fix format string argument for prom_printf() 2012-10-02 23:20:34 -04:00
oplib_64.h sparc: fix format string argument for prom_printf() 2012-10-02 23:20:34 -04:00
oplib.h
page_32.h sparc32: drop sparc_unmapped_base 2012-07-26 16:46:16 -07:00
page_64.h sparc64: Fix tsb_grow() in atomic context. 2013-02-20 09:46:08 -08:00
page.h
parport.h SPARC: drivers: remove __dev* attributes. 2013-01-03 15:57:04 -08:00
pbm.h
pci_32.h PCI: collapse pcibios_resource_to_bus 2012-02-23 20:19:04 -07:00
pci_64.h PCI: collapse pcibios_resource_to_bus 2012-02-23 20:19:04 -07:00
pci.h pci-dma: add linux/pci-dma.h to linux/pci.h 2010-03-12 15:52:42 -08:00
pcic.h sparc32: added CONFIG_PCIC_PCI Kconfig setting 2011-06-02 14:32:37 -07:00
pcr.h sparc64: Add PCR ops for SPARC-T4. 2012-08-18 23:26:19 -07:00
percpu_32.h
percpu_64.h sparc64: Get rid of real_setup_per_cpu_areas(). 2009-06-16 04:56:23 -07:00
percpu.h
perf_event.h perf, arch: Cleanup perf-pmu init vs lockup-detector 2010-11-26 15:14:56 +01:00
pgalloc_32.h sparc32: drop quicklist 2012-07-26 16:46:16 -07:00
pgalloc_64.h sparc64: Support transparent huge pages. 2012-10-09 16:23:06 +09:00
pgalloc.h
pgtable_32.h sparc32: drop unused BAD_PAGE stuff 2012-07-26 16:46:16 -07:00
pgtable_64.h sparc64: Fix race in TLB batch processing. 2013-04-19 17:26:26 -04:00
pgtable.h
pgtsrmmu.h sparc32: srmmu_probe now knows about leon too 2012-05-27 23:52:51 -07:00
pil.h sparc64: Reschedule KGDB capture to a software interrupt. 2009-03-18 23:51:57 -07:00
processor_32.h sparc idle: rename pm_idle to sparc_idle 2013-02-17 23:36:56 -05:00
processor_64.h Merge branch 'arch-microblaze' into no-rebases 2012-11-16 22:28:43 -05:00
processor.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
prom.h of/address: sparc: Declare of_iomap as an extern function for sparc again 2012-11-23 22:01:15 +00:00
psr.h UAPI: (Scripted) Disintegrate arch/sparc/include/asm 2012-10-09 09:47:43 +01:00
ptrace.h Merge commit '517ffce4e1a03aea979fe3a18a3dd1761a24fafb' into arch-sparc 2012-11-16 20:49:06 -05:00
ross.h
rwsem.h rwsem: Move duplicate function prototypes to linux/rwsem.h 2011-01-27 12:30:39 +01:00
sbi.h
scatterlist.h remove needless ISA_DMA_THRESHOLD 2010-08-07 18:15:50 +02:00
scratchpad.h
seccomp.h x86-64: seccomp: fix 32/64 syscall hole 2009-03-02 15:41:30 -08:00
sections.h sparc32: add support for run-time patching of leon/sun single instructions 2012-05-27 23:52:43 -07:00
setup.h UAPI: (Scripted) Disintegrate arch/sparc/include/asm 2012-10-09 09:47:43 +01:00
sfafsr.h
sfp-machine_32.h
sfp-machine_64.h
sfp-machine.h
shmparam_32.h sparc32: Remove sun4 and sun4c from enum sparc_cpu. 2012-05-12 00:23:23 -07:00
shmparam_64.h
shmparam.h
sigcontext.h UAPI: (Scripted) Disintegrate arch/sparc/include/asm 2012-10-09 09:47:43 +01:00
siginfo.h UAPI: (Scripted) Disintegrate arch/sparc/include/asm 2012-10-09 09:47:43 +01:00
signal.h consolidate kernel-side struct sigaction declarations 2013-02-03 15:09:22 -05:00
smp_32.h sparc:cleanup unused code in smp_32.h 2013-03-31 19:29:12 -04:00
smp_64.h sparc64: Add global PMU register dumping via sysrq. 2012-10-16 09:34:01 -07:00
smp.h
sparsemem.h
spinlock_32.h sparc32: unbreak arch_write_unlock() 2011-08-15 14:35:19 -07:00
spinlock_64.h sparc64: remove unnecessary macros from spinlock_64.h 2011-08-15 14:35:18 -07:00
spinlock_types.h locking: Convert raw_rwlock to arch_rwlock 2009-12-14 23:55:32 +01:00
spinlock.h
spitfire.h sparc64: correctly recognize SPARC64-X chips 2013-03-11 05:06:27 -07:00
stacktrace.h
starfire.h sparc64: Delete starfire_cpu_setup(). 2008-08-31 01:40:12 -07:00
string_32.h sparc32: Remove inline strncmp "optimization" for constant counts. 2012-05-11 19:53:29 -07:00
string_64.h sparc: Stop trying to be so fancy and use __builtin_{memcpy,memset}() 2009-12-10 23:32:10 -08:00
string.h
sunbpp.h
swift.h
switch_to_32.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
switch_to_64.h sparc64: Fix race in TLB batch processing. 2013-04-19 17:26:26 -04:00
switch_to.h Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
syscall.h tracing: Unify arch_syscall_addr() implementations 2010-02-17 13:07:21 +01:00
syscalls.h sparc32: switch to generic sys_execve() 2012-10-16 19:03:53 -04:00
termbits.h UAPI: (Scripted) Disintegrate arch/sparc/include/asm 2012-10-09 09:47:43 +01:00
termios.h UAPI: (Scripted) Disintegrate arch/sparc/include/asm 2012-10-09 09:47:43 +01:00
thread_info_32.h sanitize tsk_is_polling() 2012-10-01 09:58:13 -04:00
thread_info_64.h Merge commit '517ffce4e1a03aea979fe3a18a3dd1761a24fafb' into arch-sparc 2012-11-16 20:49:06 -05:00
thread_info.h
timer_32.h sparc32: remove runtime btfix support 2012-05-14 14:05:09 -07:00
timer_64.h SPARC: drivers: remove __dev* attributes. 2013-01-03 15:57:04 -08:00
timer.h
timex_32.h sparc32: generic clockevent support 2012-04-15 10:28:50 -07:00
timex_64.h
timex.h
tlb_32.h
tlb_64.h sparc: mmu_gather rework 2011-05-25 08:39:13 -07:00
tlb.h
tlbflush_32.h sparc32: Move cache and TLB flushes over to method ops. 2012-05-13 20:49:31 -07:00
tlbflush_64.h sparc64: Fix race in TLB batch processing. 2013-04-19 17:26:26 -04:00
tlbflush.h
topology_32.h
topology_64.h sched/numa: Rewrite the CONFIG_NUMA sched domain support 2012-05-09 15:00:55 +02:00
topology.h
trap_block.h sparc64: Store per-cpu offset in trap_block[] 2009-06-16 04:56:11 -07:00
traps.h UAPI: (Scripted) Disintegrate arch/sparc/include/asm 2012-10-09 09:47:43 +01:00
tsb.h sparc64: Fix huge PMD to PTE translation for sun4u in TLB miss handler. 2013-02-20 13:34:41 -08:00
tsunami.h
ttable.h sparc64: Make montmul/montsqr/mpmul usable in 32-bit threads. 2012-10-26 15:18:37 -07:00
turbosparc.h
uaccess_32.h sparc: use the new generic strnlen_user() function 2012-05-26 11:33:54 -07:00
uaccess_64.h sparc64: clear syscall_noerror on the entry to syscall, not on the exit 2012-10-14 19:26:52 -04:00
uaccess.h sparc: Fix user_addr_max() definition. 2012-05-24 13:41:58 -07:00
unaligned.h
unistd.h burying unused conditionals 2013-02-14 09:21:15 -05:00
upa.h
user.h
vaddrs.h sparc32: drop fixmap.h 2012-07-26 16:46:16 -07:00
vga.h BUG: headers with BUG/BUG_ON etc. need linux/bug.h 2012-03-04 17:54:34 -05:00
viking.h
vio.h powerpc+sparc/vio: Modernize driver registration 2012-03-28 11:33:24 +11:00
visasm.h sparc64: Fix sparse warnings in visemul.c 2008-09-11 23:46:40 -07:00
winmacro.h sparc32: Implement hard_smp_processor_id() via instruction patching. 2012-05-14 13:31:38 -07:00
xor_32.h
xor_64.h sparc64: Future proof Niagara cpu detection. 2011-09-16 14:21:33 -07:00
xor.h