linux/arch/sparc/lib/NGcopy_to_user.S

/* NGcopy_to_user.S: Niagara optimized copy to userspace.
 *
 * Copyright (C) 2006, 2007 David S. Miller (davem@davemloft.net)
 */

#define EX_ST(x)		\
98:	x;			\
	.section __ex_table,"a";\
	.align 4;		\
	.word 98b, __ret_one_asi;\
	.text;			\
	.align 4;

#ifndef ASI_AIUS
#define ASI_AIUS	0x11
#endif

#define FUNC_NAME		NGcopy_to_user
#define STORE(type,src,addr)	type##a src, [addr] ASI_AIUS
#define STORE_ASI		ASI_BLK_INIT_QUAD_LDD_AIUS
#define EX_RETVAL(x)		%g0

#ifdef __KERNEL__
	/* Writing to %asi is _expensive_ so we hardcode it.
	 * Reading %asi to check for KERNEL_DS is comparatively
	 * cheap.
	 */
#define PREAMBLE					\
	rd		%asi, %g1;			\
	cmp		%g1, ASI_AIUS;			\
	bne,pn		%icc, ___copy_in_user;		\
	 nop
#endif

#include "NGmemcpy.S"
[SPARC64]: Niagara optimized memcpy() and copy_{to,from}_user(). Signed-off-by: David S. Miller <davem@davemloft.net> 2006-03-06 00:41:56 +00:00			`/* NGcopy_to_user.S: Niagara optimized copy to userspace.`
			`*`
[SPARC64]: Fix missing load-twin usage in Niagara-1 memcpy. For the case where the source is not aligned modulo 8 we don't use load-twins to suck the data in and this kills performance since normal loads allocate in the L1 cache (unlike load-twin) and thus big memcpys swipe the entire L1 D-cache. We need to allocate a register window to implement this properly, but that actually simplifies a lot of things as a nice side-effect. Signed-off-by: David S. Miller <davem@davemloft.net> 2007-10-02 08:03:09 +00:00			`* Copyright (C) 2006, 2007 David S. Miller (davem@davemloft.net)`
[SPARC64]: Niagara optimized memcpy() and copy_{to,from}_user(). Signed-off-by: David S. Miller <davem@davemloft.net> 2006-03-06 00:41:56 +00:00			`*/`

			`#define EX_ST(x) \`
			`98: x; \`
			`.section __ex_table,"a";\`
			`.align 4; \`
sparc64: Kill .fixup section bloat. This is an implementation of a suggestion made by Chris Torek: -------------------- Something else I noticed in passing: the EX and EX_LD/EX_ST macros scattered throughout the various .S files make a fair bit of .fixup code, all of which does the same thing. At the cost of one symbol in copy_in_user.S, you could just have one common two-instruction retl-and-mov-1 fixup that they all share. -------------------- The following is with a defconfig build: text data bss dec hex filename 3972767 344024 584449 4901240 4ac978 vmlinux.orig 3968887 344024 584449 4897360 4aba50 vmlinux Signed-off-by: David S. Miller <davem@davemloft.net> 2009-02-09 06:00:55 +00:00			`.word 98b, __ret_one_asi;\`
[SPARC64]: Niagara optimized memcpy() and copy_{to,from}_user(). Signed-off-by: David S. Miller <davem@davemloft.net> 2006-03-06 00:41:56 +00:00			`.text; \`
			`.align 4;`

			`#ifndef ASI_AIUS`
			`#define ASI_AIUS 0x11`
			`#endif`

			`#define FUNC_NAME NGcopy_to_user`
			`#define STORE(type,src,addr) type##a src, [addr] ASI_AIUS`
			`#define STORE_ASI ASI_BLK_INIT_QUAD_LDD_AIUS`
[SPARC64]: Fix missing load-twin usage in Niagara-1 memcpy. For the case where the source is not aligned modulo 8 we don't use load-twins to suck the data in and this kills performance since normal loads allocate in the L1 cache (unlike load-twin) and thus big memcpys swipe the entire L1 D-cache. We need to allocate a register window to implement this properly, but that actually simplifies a lot of things as a nice side-effect. Signed-off-by: David S. Miller <davem@davemloft.net> 2007-10-02 08:03:09 +00:00			`#define EX_RETVAL(x) %g0`
[SPARC64]: Niagara optimized memcpy() and copy_{to,from}_user(). Signed-off-by: David S. Miller <davem@davemloft.net> 2006-03-06 00:41:56 +00:00
			`#ifdef __KERNEL__`
			`/* Writing to %asi is _expensive_ so we hardcode it.`
			`* Reading %asi to check for KERNEL_DS is comparatively`
			`* cheap.`
			`*/`
			`#define PREAMBLE \`
			`rd %asi, %g1; \`
			`cmp %g1, ASI_AIUS; \`
sparc64: Fix probe_kernel_{read,write}(). This is based upon a report from Chris Torek and his initial patch. From Chris's report: -------------------- This came up in testing kgdb, using the built-in tests -- turn on CONFIG_KGDB_TESTS, then echo V1 > /sys/module/kgdbts/parameters/kgdbts -- but it would affect using kgdb if you were debugging and looking at bad pointers. -------------------- When we get a copy_{from,to}_user() request and the %asi is set to something other than ASI_AIUS (which is userspace) then we branch off to a routine called memcpy_user_stub(). It just does a straight memcpy since we are copying from kernel to kernel in this case. The logic was that since source and destination are both kernel pointers we don't need to have exception checks. But for what probe_kernel_{read,write}() is trying to do, we have to have the checks, otherwise things like kgdb bad kernel pointer accesses don't do the right thing. Signed-off-by: David S. Miller <davem@davemloft.net> 2009-02-09 06:32:31 +00:00			`bne,pn %icc, ___copy_in_user; \`
[SPARC64]: Niagara optimized memcpy() and copy_{to,from}_user(). Signed-off-by: David S. Miller <davem@davemloft.net> 2006-03-06 00:41:56 +00:00			`nop`
			`#endif`

[SPARC64]: Fix some Niagara memcpy() bugs. We need to restore the %asi register properly. For the kernel this means get_fs(), for user this means ASI_PNF. Also, NGcopy_to_user.S was including U3memcpy.S instead of NGmemcpy.S, oops :-) Signed-off-by: David S. Miller <davem@davemloft.net> 2006-02-11 18:30:41 +00:00			`#include "NGmemcpy.S"`