Use this_cpu operations to optimize access primitives for highmem.
The main effect is the avoidance of address calculations through the
use of a segment prefix.
V3->V4
- kmap_atomic_idx: Do not return a value.
- Use __this_cpu_dec without HIGHMEM_DEBUG
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
this_cpu_inc_return() saves us a memory access there. Code
size does not change.
V1->V2:
- Fixed the location of the __per_cpu pointer attributes
- Sparse checked
V2->V3:
- Move fixes to __percpu attribute usage to earlier patch
Reviewed-by: Pekka Enberg <penberg@kernel.org>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Supply an implementation for x86 in order to generate more efficient code.
V2->V3:
- Cleanup
- Remove strange type checking from percpu_add_return_op.
tj: - Dropped unused typedef from percpu_add_return_op().
- Renamed ret__ to paro_ret__ in percpu_add_return_op().
- Minor indentation adjustments.
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Introduce generic support for this_cpu_add_return etc.
The fallback is to realize these operations with simpler __this_cpu_ops.
tj: - Reformatted __cpu_size_call_return2() to make it more consistent
with its neighbors.
- Dropped unnecessary temp variable ret__ from
__this_cpu_generic_add_return().
Reviewed-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
__get_cpu_var() can be replaced with this_cpu_read and will then use a
single read instruction with implied address calculation to access the
correct per cpu instance.
However, the address of a per cpu variable passed to __this_cpu_read()
cannot be determined (since it's an implied address conversion through
segment prefixes). Therefore apply this only to uses of __get_cpu_var
where the address of the variable is not used.
Cc: Pekka Enberg <penberg@cs.helsinki.fi>
Cc: Hugh Dickins <hughd@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Use this_cpu_ops to reduce code size and simplify things in various places.
V3->V4:
Move instance of this_cpu_inc_return to a later patchset so that
this patch can be applied without infrastructure changes.
Cc: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Optimize various per cpu area operations through these new percpu
operations. These operations avoid address calculations through the
use of segment prefixes and multiple memory references through RMW
instructions etc.
Reduces code size:
Before:
christoph@linux-2.6$ size fs/buffer.o
text data bss dec hex filename
19169 80 28 19277 4b4d fs/buffer.o
After:
christoph@linux-2.6$ size fs/buffer.o
text data bss dec hex filename
19138 80 28 19246 4b2e fs/buffer.o
V3->V4:
- Move the use of this_cpu_inc_return into a later patch so that
this one can go in without percpu infrastructure changes.
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Christoph Hellwig <hch@lst.de>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
The whole function can be expressed as a simple this_cpu_read() operation.
The function overhead is now likely multiple times that of the single
instruction that is executed in it.
Sedat: fixed compile failure caused by an extra ')'.
Cc: William Hubbs <w.d.hubbs@gmail.com>
Cc: Sedat Dilek <sedat.dilek@gmail.com>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Use this_cpu ops in various places to optimize per cpu data access.
Cc: Jason Baron <jbaron@redhat.com>
Cc: Namhyung Kim <namhyung@gmail.com>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
__get_cpu_var() can be replaced with this_cpu_read and will then use a single
read instruction with implied address calculation to access the correct per cpu
instance.
However, the address of a per cpu variable passed to __this_cpu_read() cannot be
determed (since its an implied address conversion through segment prefixes).
Therefore apply this only to uses of __get_cpu_var where the addres of the
variable is not used.
V3->V4:
- Move one instance of this_cpu_inc_return to a later patch
so that this one can go in without percpu infrastructrure
changes.
Sedat: fixed compile failure caused by an extra ')'.
Cc: Neil Horman <nhorman@tuxdriver.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Sedat Dilek <sedat.dilek@gmail.com>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
this cpu operations can be used to slightly optimize the function. The
changes will avoid some address calculations and replace them with the
use of the percpu segment register.
If one would have this_cpu_inc_return and this_cpu_dec_return then it
would be possible to optimize inc_zone_page_state and
dec_zone_page_state even more.
V1->V2:
- Fix __dec_zone_state overflow handling
- Use s8 variables for temporary storage.
V2->V3:
- Put __percpu annotations in correct places.
Reviewed-by: Pekka Enberg <penberg@kernel.org>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Add me and Christoph Lameter as maintainers for the percpu memory
allocator.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@linux-foundation.org>
Acked-by: Christoph Lameter <cl@linux-foundation.org>
* 'for-linus' of git://git.infradead.org/users/eparis/notify:
fanotify: fill in the metadata_len field on struct fanotify_event_metadata
fanotify: split version into version and metadata_len
fanotify: Dont try to open a file descriptor for the overflow event
fanotify: Introduce FAN_NOFD
fanotify: do not leak user reference on allocation failure
inotify: stop kernel memory leak on file creation failure
fanotify: on group destroy allow all waiters to bypass permission check
fanotify: Dont allow a mask of 0 if setting or removing a mark
fanotify: correct broken ref counting in case adding a mark failed
fanotify: if set by user unset FMODE_NONOTIFY before fsnotify_perm() is called
fanotify: remove packed from access response message
fanotify: deny permissions when no event was sent
* 'upstream' of git://git.linux-mips.org/pub/scm/upstream-linus: (28 commits)
MIPS: Add a CONFIG_FORCE_MAX_ZONEORDER Kconfig option.
MIPS: LD/SD o32 macro GAS fix update
MIPS: Alchemy: fix build with SERIAL_8250=n
MIPS: Rename mips_dma_cache_sync back to dma_cache_sync
MIPS: MT: Fix typo in comment.
SSB: Fix nvram_get on BCM47xx platform
MIPS: BCM47xx: Swap serial console if ttyS1 was specified.
MIPS: BCM47xx: Use sscanf for parsing mac address
MIPS: BCM47xx: Fill values for b43 into SSB sprom
MIPS: BCM47xx: Do not read config from CFE
MIPS: FDT size is a be32
MIPS: Fix CP0 COUNTER clockevent race
MIPS: Fix regression on BCM4710 processor detection
MIPS: JZ4740: Fix pcm device name
MIPS: Separate two consecutive loads in memset.S
MIPS: Send proper signal and siginfo on FP emulator faults.
MIPS: AR7: Fix loops per jiffies on TNETD7200 devices
MIPS: AR7: Fix double ar7_gpio_init declaration
MIPS: Rework GENERIC_HARDIRQS Kconfig.
MIPS: Alchemy: Add return value check for strict_strtoul()
...
For huge page support with base page size of 16K or 32K, we have to
increase the MAX_ORDER so that huge pages can be allocated.
[Ralf: I don't think a user should have to configure obscure constants like
this but for the time being this will have to suffice.]
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1685/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
I am about to commit:
http://sourceware.org/ml/binutils/2010-10/msg00033.html
that fixes a problem with the LD/SD macro currently implemented by GAS for
the o32 ABI in an inconsistent way. This is best illustrated with a
simple program, which I'm copying here from the message above for easier
reference:
$ cat ld.s
ld $5,32767($4)
ld $5,32768($4)
This gets assebled into the following output:
$ mips-linux-as -32 -mips3 -o ld.o ld.s
$ mips-linux-objdump -d ld.o
ld.o: file format elf32-tradbigmips
Disassembly of section .text:
00000000 <.text>:
0: dc857fff ld a1,32767(a0)
4: 3c010001 lui at,0x1
8: 00810821 addu at,a0,at
c: 8c258000 lw a1,-32768(at)
10: 8c268004 lw a2,-32764(at)
...
Oops!
The GAS fix makes the macro behave in a consistent way and pairs of LW/SW
instructions to be output as appropriate regardless of the size of the
offset associated with the address used. The machine instruction is still
available, but to reach it macros have to be disabled first. This has a
side effect of requiring the use of a machine-addressable memory operand.
As some platforms require 64-bit operations for accesses to some I/O
registers LD/SD instructions are used in a couple of places in Linux
regardless of the ABI selected. Here's a fix for some pieces of code
affected I've been able to track down. The fix should be backwards
compatible with all supported binutils releases in existence and can be
used as a reference for any other places or off-tree code. The use of the
"R" constraint guarantees a machine-addressable operand.
Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1680/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
In commit 7d172bfe ("Alchemy: Add UART PM methods") I introduced
platform PM methods which call a function of the 8250 driver;
this patch works around link failures when the kernel is built
without 8250 support.
Signed-off-by: Manuel Lauss <manuel.lauss@googlemail.com>
To: Linux-MIPS <linux-mips@linux-mips.org>
Patchwork: https://patchwork.linux-mips.org/patch/1737/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The nvram_get function was never in the mainline kernel, it only existed in
an external OpenWrt patch. Use nvram_getenv function, which is in mainline
and use an include instead of an extra function declaration. et0macaddr
contains the mac address in text from like 00:11:22:33:44:55. We have to
parse it before adding it into macaddr.
nvram_parse_macaddr will be merged into asm/mach-bcm47xx/nvram.h through
the MIPS git tree and will be available soon. It will not build now without
nvram_parse_macaddr, but it hasn't before either.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
To: linux-mips@linux-mips.org
Cc: mb@bu3sch.de
Cc: netdev@vger.kernel.org
Cc: Hauke Mehrtens <hauke@hauke-m.de>
Acked-by: Michael Buesch <mb@bu3sch.de>
Patchwork: https://patchwork.linux-mips.org/patch/1849/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Some devices like the Netgear WGT634U are using ttyS1 for default console
output. We should switch to that console if it was given in the kernel_args
parameters.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
To: linux-mips@linux-mips.org
Cc: Hauke Mehrtens <hauke@hauke-m.de>
Patchwork: https://patchwork.linux-mips.org/patch/1848/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Instead of writing own function for parsing the mac address we now
use sscanf.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
To: linux-mips@linux-mips.org
Cc: Hauke Mehrtens <hauke@hauke-m.de>
Patchwork: https://patchwork.linux-mips.org/patch/1847/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Fill the sprom with all available values from the nvram. Most of these
new values are needed for the b43 or b43legacy driver.
Parts of this patch have been in OpenWRT for a long time and were written
by Michael Buesch.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
To: linux-mips@linux-mips.org
Cc: Hauke Mehrtens <hauke@hauke-m.de>
Patchwork: https://patchwork.linux-mips.org/patch/1846/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The config options read out here are not stored in CFE but only in NVRAM on
the devices. Remove reading from CFE and only access the NVRAM. Reading out
CFE does not harm but is useless here.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
To: linux-mips@linux-mips.org
Cc: Hauke Mehrtens <hauke@hauke-m.de>
Patchwork: https://patchwork.linux-mips.org/patch/1845/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Consider the following test case:
write_c0_compare(read_c0_count());
Even if the counter doesn't increment during execution, this might not
generate an interrupt until the counter wraps around. The CPU may
perform the comparison each time CP0 COUNT increments, not when CP0
COMPARE is written.
If mips_next_event() is called with a very small delta, and CP0 COUNT
increments during the calculation of "cnt += delta", it is possible
that CP0 COMPARE will be written with the current value of CP0 COUNT.
If this is detected, the function should return -ETIME, to indicate
that the interrupt might not have actually gotten scheduled.
Signed-off-by: Kevin Cernekee <cernekee@gmail.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/1836/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
BCM4710 uses the BMIPS32 core (like BCM6345), not the MIPS 4Kc core as
was previously believed.
Signed-off-by: Kevin Cernekee <cernekee@gmail.com>
Tested-by: Alexandros C. Couloumbis <alex@ozo.com>
Patchwork: https://patchwork.linux-mips.org/patch/1837/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
partial_fixup is used in noreorder block.
Separating two consecutive loads can save one cycle on processors with
GPR intrelock and can fix load-use on processors that need a load delay slot.
Also do so for fwd_fixup.
[Ralf: Only R2000/R3000 class processors are lacking the the load-user
interlock and even some of those got it retrofitted. With R2000/R3000
being fairly uncommon these days the impact of this bug should be minor.]
Signed-off-by: Tony Wu <tung7970@gmail.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1768/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
We were unconditionally sending SIGBUS with an empty siginfo on FP
emulator faults. This differs from what happens when real floating
point hardware would get a fault.
For most faults we need to send SIGSEGV with the faulting address
filled in in the struct siginfo.
Reported-by: Camm Maguire <camm@maguirefamily.org>
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Cc: Camm Maguire <camm@maguirefamily.org>
Patchwork: https://patchwork.linux-mips.org/patch/1727/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
TNETD7200 run their CPU clock faster than the default CPU clock we assume.
In order to have the correct loops per jiffies settings, initialize clocks right
before setting mips_hpt_frequency. As a side effect, we can no longer use
msleep in clocks.c which requires other parts of the kernel to be initialized,
so replace these with mdelay.
Signed-off-by: Florian Fainelli <florian@openwrt.org>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1749/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Recent changes to CONFIG_GENERIC_HARDIRQS have caused us to start getting:
warning: (SMP && SYS_SUPPORTS_SMP) selects IRQ_PER_CPU which has unmet direct dependencies (HAVE_GENERIC_HARDIRQS)
Rearranging our Kconfig quiets the message.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Cc: Thomas Gleixner <tglx@linutronix.de>
Patchwork: https://patchwork.linux-mips.org/patch/1757/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
cc1: warnings being treated as errors
arch/mips/loongson/common/env.c: In function 'prom_init_env':
arch/mips/loongson/common/env.c:49: error: ignoring return value of 'strict_strtol', declared with attribute warn_unused_result
arch/mips/loongson/common/env.c:50: error: ignoring return value of 'strict_strtol', declared with attribute warn_unused_result
arch/mips/loongson/common/env.c:51: error: ignoring return value of 'strict_strtol', declared with attribute warn_unused_result
arch/mips/loongson/common/env.c:52: error: ignoring return value of 'strict_strtol', declared with attribute warn_unused_result
Signed-off-by: Wu Zhangjin <wuzhangjin@gmail.com>
Cc: linux-mips <linux-mips@linux-mips.org>
Patchwork: https://patchwork.linux-mips.org/patch/1762/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The return value of the vmalloc() call in arch/mips/kernel/vpe.c::vpe_open()
is not checked, so we potentially store a null pointer in v->pbuffer. Add
a check for a null return and then return -ENOMEM in that case.
[Ralf: The check added by Jesper's original patch is where it logically
should be. Adding it eleminated the need for the checks in a few other
places, so I removed them. There still is a zillion of other things that
need to be fixed in this file / API.]
Signed-off-by: Jesper Juhl <jj@chaosbits.net>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/1747/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
If PER_LINUX32 has been set on a 32-bit kernel, only twiddle with the
low-order personality bits, let the upper bits pass through.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Cc: Camm Maguire <camm@maguirefamily.org>
Patchwork: https://patchwork.linux-mips.org/patch/1751/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The high bits of current->personality carry settings that we don't want to
clobber on each exec. Only clobber them if the lower bits that indicate
either PER_LINUX or PER_LINUX32 are invalid.
The clobbering prevents us from using useful bits like ADDR_NO_RANDOMIZE.
Reported-by: Camm Maguire <camm@maguirefamily.org>
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Cc: Camm Maguire <camm@maguirefamily.org>
Patchwork: https://patchwork.linux-mips.org/patch/1750/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
This patch fixes the following section mismatch:
WARNING: arch/mips/built-in.o(.text+0xc): Section mismatch in reference from the
function jz4740_init_cmdline() to the variable .init.data:arcs_cmdline
While were at it, make jz4740_init_cmdline static as well.
Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1755/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
We never needed that (->regs[2] is overwritten on return from syscall paths
with return value of syscall, so storing it there early made no sense) and
with new restart logics since d27240bf7e61d2656de18e158ec910a902030847 it
has become really bad - we lose the original syscall number before the
place where we decide that we might need a syscall restart.
Note that for child we do need the assignment to regs[2] - it won't go
through the normal return from syscall path.
[Ralf: Issue found and reported by Lluís; initial investigations by me;
bug finally found and patch by Al; testing by me and Lluís.]
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Tested-by: Lluís Batlle i Rossell <viriketo@gmail.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input:
Input: define separate EVIOCGKEYCODE_V2/EVIOCSKEYCODE_V2
Input: wacom - add another Bamboo Pen ID (0xd4)
Two x86 patches broke lguest:
1) v2.6.35-492-g72d7c3b, which changed x86 to use the memblock allocator.
In lguest, the host places linear page tables at the top of mem, which
used to be enough to get us up to the swapper_pg_dir page tables. With
the first patch, the direct mapping tables used that memory:
Before: kernel direct mapping tables up to 4000000 @ 7000-1a000
After: kernel direct mapping tables up to 4000000 @ 3fed000-4000000
I initially fixed this by lying about the amount of memory we had, so
the kernel wouldn't blatt the lguest boot pagetables (yuk!), but then...
2) v2.6.36-rc8-54-gb40827f, which made x86 boot use initial_page_table.
This was initialized in a part of head_32.S which isn't executed by
lguest; it is then copied into swapper_pg_dir. So we have to initialize
it; and anyway we switch to it before we blatt the old tables, so that
fixes the previous damage as well.
For the moment, I cut & pasted the code into lguest's boot code, but
next merge window I will merge them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Cc: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: x86@kernel.org
lguest is dumb and drops *all* the pagetables for set_pte (which is
only used for kernel mapping manipulation, so it's OK without highmem).
But it's used a lot in boot, too. As a guest optimization, we
suppressed this flushing until the first page switch. Now we have
initial_page_table, that happens much earlier, so extend the heuristic
to wait until we switch to something other than the swapper_pg_dir or
initial_page_table.
As measured on my laptop under kvm, this dropped the time-to-mount-root
from 48 seconds to 4.3 seconds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
fe25c7fc2e "x86: lguest: Convert to new irq chip functions" converted
enable_lguest_irq() to take a struct irq_data *, but didn't fix the one
internal caller.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
To: x86@kernel.org
