leandrof/linux
1
0
Fork 0
forked from Minki/linux
Code Issues Pull Requests Packages Projects Releases Wiki Activity
afed7d1720
linux/arch/sparc/crypto/aes_asm.S

1545 lines
38 KiB
ArmAsm
Raw Normal View History

License cleanup: add SPDX GPL-2.0 license identifier to files with no license Many source files in the tree are missing licensing information, which makes it harder for compliance tools to determine the correct license. By default all files without license information are under the default license of the kernel, which is GPL version 2. Update the files which contain no license information with the 'GPL-2.0' SPDX license identifier. The SPDX identifier is a legally binding shorthand, which can be used instead of the full boiler plate text. This patch is based on work done by Thomas Gleixner and Kate Stewart and Philippe Ombredanne. How this work was done: Patches were generated and checked against linux-4.14-rc6 for a subset of the use cases: - file had no licensing information it it. - file was a */uapi/* one with no licensing information in it, - file was a */uapi/* one with existing licensing information, Further patches will be generated in subsequent months to fix up cases where non-standard license headers were used, and references to license had to be inferred by heuristics based on keywords. The analysis to determine which SPDX License Identifier to be applied to a file was done in a spreadsheet of side by side results from of the output of two independent scanners (ScanCode & Windriver) producing SPDX tag:value files created by Philippe Ombredanne. Philippe prepared the base worksheet, and did an initial spot review of a few 1000 files. The 4.13 kernel was the starting point of the analysis with 60,537 files assessed. Kate Stewart did a file by file comparison of the scanner results in the spreadsheet to determine which SPDX license identifier(s) to be applied to the file. She confirmed any determination that was not immediately clear with lawyers working with the Linux Foundation. Criteria used to select files for SPDX license identifier tagging was: - Files considered eligible had to be source code files. - Make and config files were included as candidates if they contained >5 lines of source - File already had some variant of a license header in it (even if <5 lines). All documentation files were explicitly excluded. The following heuristics were used to determine which SPDX license identifiers to apply. - when both scanners couldn't find any license traces, file was considered to have no license information in it, and the top level COPYING file license applied. For non */uapi/* files that summary was: SPDX license identifier # files ---------------------------------------------------|------- GPL-2.0 11139 and resulted in the first patch in this series. If that file was a */uapi/* path one, it was "GPL-2.0 WITH Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was: SPDX license identifier # files ---------------------------------------------------|------- GPL-2.0 WITH Linux-syscall-note 930 and resulted in the second patch in this series. - if a file had some form of licensing information in it, and was one of the */uapi/* ones, it was denoted with the Linux-syscall-note if any GPL family license was found in the file or had no licensing in it (per prior point). Results summary: SPDX license identifier # files ---------------------------------------------------|------ GPL-2.0 WITH Linux-syscall-note 270 GPL-2.0+ WITH Linux-syscall-note 169 ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21 ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17 LGPL-2.1+ WITH Linux-syscall-note 15 GPL-1.0+ WITH Linux-syscall-note 14 ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5 LGPL-2.0+ WITH Linux-syscall-note 4 LGPL-2.1 WITH Linux-syscall-note 3 ((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3 ((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1 and that resulted in the third patch in this series. - when the two scanners agreed on the detected license(s), that became the concluded license(s). - when there was disagreement between the two scanners (one detected a license but the other didn't, or they both detected different licenses) a manual inspection of the file occurred. - In most cases a manual inspection of the information in the file resulted in a clear resolution of the license that should apply (and which scanner probably needed to revisit its heuristics). - When it was not immediately clear, the license identifier was confirmed with lawyers working with the Linux Foundation. - If there was any question as to the appropriate license identifier, the file was flagged for further research and to be revisited later in time. In total, over 70 hours of logged manual review was done on the spreadsheet to determine the SPDX license identifiers to apply to the source files by Kate, Philippe, Thomas and, in some cases, confirmation by lawyers working with the Linux Foundation. Kate also obtained a third independent scan of the 4.13 code base from FOSSology, and compared selected files where the other two scanners disagreed against that SPDX file, to see if there was new insights. The Windriver scanner is based on an older version of FOSSology in part, so they are related. Thomas did random spot checks in about 500 files from the spreadsheets for the uapi headers and agreed with SPDX license identifier in the files he inspected. For the non-uapi files Thomas did random spot checks in about 15000 files. In initial set of patches against 4.14-rc6, 3 files were found to have copy/paste license identifier errors, and have been fixed to reflect the correct identifier. Additionally Philippe spent 10 hours this week doing a detailed manual inspection and review of the 12,461 patched files from the initial patch version early this week with: - a full scancode scan run, collecting the matched texts, detected license ids and scores - reviewing anything where there was a license detected (about 500+ files) to ensure that the applied SPDX license was correct - reviewing anything where there was no detection but the patch license was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied SPDX license was correct This produced a worksheet with 20 files needing minor correction. This worksheet was then exported into 3 different .csv files for the different types of files to be modified. These .csv files were then reviewed by Greg. Thomas wrote a script to parse the csv files and add the proper SPDX tag to the file, in the format that the file expected. This script was further refined by Greg based on the output to detect more types of files automatically and to distinguish between header and source .c files (which need different comment types.) Finally Greg ran the script using the .csv files to generate the patches. Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org> Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com> Reviewed-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
/* SPDX-License-Identifier: GPL-2.0 */
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
#include <linux/linkage.h>
#include <asm/visasm.h>
sparc64: Avoid code duplication in crypto assembler. Put the opcode macros in a common header Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-31 19:11:51 +00:00
#include "opcodes.h"
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
#define ENCRYPT_TWO_ROUNDS(KEY_BASE, I0, I1, T0, T1) \
AES_EROUND01(KEY_BASE + 0, I0, I1, T0) \
AES_EROUND23(KEY_BASE + 2, I0, I1, T1) \
AES_EROUND01(KEY_BASE + 4, T0, T1, I0) \
AES_EROUND23(KEY_BASE + 6, T0, T1, I1)
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
#define ENCRYPT_TWO_ROUNDS_2(KEY_BASE, I0, I1, I2, I3, T0, T1, T2, T3) \
AES_EROUND01(KEY_BASE + 0, I0, I1, T0) \
AES_EROUND23(KEY_BASE + 2, I0, I1, T1) \
AES_EROUND01(KEY_BASE + 0, I2, I3, T2) \
AES_EROUND23(KEY_BASE + 2, I2, I3, T3) \
AES_EROUND01(KEY_BASE + 4, T0, T1, I0) \
AES_EROUND23(KEY_BASE + 6, T0, T1, I1) \
AES_EROUND01(KEY_BASE + 4, T2, T3, I2) \
AES_EROUND23(KEY_BASE + 6, T2, T3, I3)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
#define ENCRYPT_TWO_ROUNDS_LAST(KEY_BASE, I0, I1, T0, T1) \
AES_EROUND01(KEY_BASE + 0, I0, I1, T0) \
AES_EROUND23(KEY_BASE + 2, I0, I1, T1) \
AES_EROUND01_L(KEY_BASE + 4, T0, T1, I0) \
AES_EROUND23_L(KEY_BASE + 6, T0, T1, I1)
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
#define ENCRYPT_TWO_ROUNDS_LAST_2(KEY_BASE, I0, I1, I2, I3, T0, T1, T2, T3) \
AES_EROUND01(KEY_BASE + 0, I0, I1, T0) \
AES_EROUND23(KEY_BASE + 2, I0, I1, T1) \
AES_EROUND01(KEY_BASE + 0, I2, I3, T2) \
AES_EROUND23(KEY_BASE + 2, I2, I3, T3) \
AES_EROUND01_L(KEY_BASE + 4, T0, T1, I0) \
AES_EROUND23_L(KEY_BASE + 6, T0, T1, I1) \
AES_EROUND01_L(KEY_BASE + 4, T2, T3, I2) \
AES_EROUND23_L(KEY_BASE + 6, T2, T3, I3)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
/* 10 rounds */
#define ENCRYPT_128(KEY_BASE, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 0, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 8, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 16, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 24, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS_LAST(KEY_BASE + 32, I0, I1, T0, T1)
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
#define ENCRYPT_128_2(KEY_BASE, I0, I1, I2, I3, T0, T1, T2, T3) \
ENCRYPT_TWO_ROUNDS_2(KEY_BASE + 0, I0, I1, I2, I3, T0, T1, T2, T3) \
ENCRYPT_TWO_ROUNDS_2(KEY_BASE + 8, I0, I1, I2, I3, T0, T1, T2, T3) \
ENCRYPT_TWO_ROUNDS_2(KEY_BASE + 16, I0, I1, I2, I3, T0, T1, T2, T3) \
ENCRYPT_TWO_ROUNDS_2(KEY_BASE + 24, I0, I1, I2, I3, T0, T1, T2, T3) \
ENCRYPT_TWO_ROUNDS_LAST_2(KEY_BASE + 32, I0, I1, I2, I3, T0, T1, T2, T3)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
/* 12 rounds */
#define ENCRYPT_192(KEY_BASE, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 0, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 8, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 16, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 24, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 32, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS_LAST(KEY_BASE + 40, I0, I1, T0, T1)
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
#define ENCRYPT_192_2(KEY_BASE, I0, I1, I2, I3, T0, T1, T2, T3) \
ENCRYPT_TWO_ROUNDS_2(KEY_BASE + 0, I0, I1, I2, I3, T0, T1, T2, T3) \
ENCRYPT_TWO_ROUNDS_2(KEY_BASE + 8, I0, I1, I2, I3, T0, T1, T2, T3) \
ENCRYPT_TWO_ROUNDS_2(KEY_BASE + 16, I0, I1, I2, I3, T0, T1, T2, T3) \
ENCRYPT_TWO_ROUNDS_2(KEY_BASE + 24, I0, I1, I2, I3, T0, T1, T2, T3) \
ENCRYPT_TWO_ROUNDS_2(KEY_BASE + 32, I0, I1, I2, I3, T0, T1, T2, T3) \
ENCRYPT_TWO_ROUNDS_LAST_2(KEY_BASE + 40, I0, I1, I2, I3, T0, T1, T2, T3)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
/* 14 rounds */
#define ENCRYPT_256(KEY_BASE, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 0, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 8, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 16, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 24, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 32, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS(KEY_BASE + 40, I0, I1, T0, T1) \
ENCRYPT_TWO_ROUNDS_LAST(KEY_BASE + 48, I0, I1, T0, T1)
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
#define ENCRYPT_256_TWO_ROUNDS_2(KEY_BASE, I0, I1, I2, I3, TMP_BASE) \
ENCRYPT_TWO_ROUNDS_2(KEY_BASE, I0, I1, I2, I3, \
TMP_BASE + 0, TMP_BASE + 2, TMP_BASE + 4, TMP_BASE + 6)
#define ENCRYPT_256_2(KEY_BASE, I0, I1, I2, I3) \
ENCRYPT_256_TWO_ROUNDS_2(KEY_BASE + 0, I0, I1, I2, I3, KEY_BASE + 48) \
ldd [%o0 + 0xd0], %f56; \
ldd [%o0 + 0xd8], %f58; \
ENCRYPT_256_TWO_ROUNDS_2(KEY_BASE + 8, I0, I1, I2, I3, KEY_BASE + 0) \
ldd [%o0 + 0xe0], %f60; \
ldd [%o0 + 0xe8], %f62; \
ENCRYPT_256_TWO_ROUNDS_2(KEY_BASE + 16, I0, I1, I2, I3, KEY_BASE + 0) \
ENCRYPT_256_TWO_ROUNDS_2(KEY_BASE + 24, I0, I1, I2, I3, KEY_BASE + 0) \
ENCRYPT_256_TWO_ROUNDS_2(KEY_BASE + 32, I0, I1, I2, I3, KEY_BASE + 0) \
ENCRYPT_256_TWO_ROUNDS_2(KEY_BASE + 40, I0, I1, I2, I3, KEY_BASE + 0) \
AES_EROUND01(KEY_BASE + 48, I0, I1, KEY_BASE + 0) \
AES_EROUND23(KEY_BASE + 50, I0, I1, KEY_BASE + 2) \
AES_EROUND01(KEY_BASE + 48, I2, I3, KEY_BASE + 4) \
AES_EROUND23(KEY_BASE + 50, I2, I3, KEY_BASE + 6) \
AES_EROUND01_L(KEY_BASE + 52, KEY_BASE + 0, KEY_BASE + 2, I0) \
AES_EROUND23_L(KEY_BASE + 54, KEY_BASE + 0, KEY_BASE + 2, I1) \
ldd [%o0 + 0x10], %f8; \
ldd [%o0 + 0x18], %f10; \
AES_EROUND01_L(KEY_BASE + 52, KEY_BASE + 4, KEY_BASE + 6, I2) \
AES_EROUND23_L(KEY_BASE + 54, KEY_BASE + 4, KEY_BASE + 6, I3) \
ldd [%o0 + 0x20], %f12; \
ldd [%o0 + 0x28], %f14;
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
#define DECRYPT_TWO_ROUNDS(KEY_BASE, I0, I1, T0, T1) \
AES_DROUND23(KEY_BASE + 0, I0, I1, T1) \
AES_DROUND01(KEY_BASE + 2, I0, I1, T0) \
AES_DROUND23(KEY_BASE + 4, T0, T1, I1) \
AES_DROUND01(KEY_BASE + 6, T0, T1, I0)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
#define DECRYPT_TWO_ROUNDS_2(KEY_BASE, I0, I1, I2, I3, T0, T1, T2, T3) \
AES_DROUND23(KEY_BASE + 0, I0, I1, T1) \
AES_DROUND01(KEY_BASE + 2, I0, I1, T0) \
AES_DROUND23(KEY_BASE + 0, I2, I3, T3) \
AES_DROUND01(KEY_BASE + 2, I2, I3, T2) \
AES_DROUND23(KEY_BASE + 4, T0, T1, I1) \
AES_DROUND01(KEY_BASE + 6, T0, T1, I0) \
AES_DROUND23(KEY_BASE + 4, T2, T3, I3) \
AES_DROUND01(KEY_BASE + 6, T2, T3, I2)
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
#define DECRYPT_TWO_ROUNDS_LAST(KEY_BASE, I0, I1, T0, T1) \
AES_DROUND23(KEY_BASE + 0, I0, I1, T1) \
AES_DROUND01(KEY_BASE + 2, I0, I1, T0) \
AES_DROUND23_L(KEY_BASE + 4, T0, T1, I1) \
AES_DROUND01_L(KEY_BASE + 6, T0, T1, I0)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
#define DECRYPT_TWO_ROUNDS_LAST_2(KEY_BASE, I0, I1, I2, I3, T0, T1, T2, T3) \
AES_DROUND23(KEY_BASE + 0, I0, I1, T1) \
AES_DROUND01(KEY_BASE + 2, I0, I1, T0) \
AES_DROUND23(KEY_BASE + 0, I2, I3, T3) \
AES_DROUND01(KEY_BASE + 2, I2, I3, T2) \
AES_DROUND23_L(KEY_BASE + 4, T0, T1, I1) \
AES_DROUND01_L(KEY_BASE + 6, T0, T1, I0) \
AES_DROUND23_L(KEY_BASE + 4, T2, T3, I3) \
AES_DROUND01_L(KEY_BASE + 6, T2, T3, I2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
/* 10 rounds */
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
#define DECRYPT_128(KEY_BASE, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 0, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 8, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 16, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 24, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS_LAST(KEY_BASE + 32, I0, I1, T0, T1)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
#define DECRYPT_128_2(KEY_BASE, I0, I1, I2, I3, T0, T1, T2, T3) \
DECRYPT_TWO_ROUNDS_2(KEY_BASE + 0, I0, I1, I2, I3, T0, T1, T2, T3) \
DECRYPT_TWO_ROUNDS_2(KEY_BASE + 8, I0, I1, I2, I3, T0, T1, T2, T3) \
DECRYPT_TWO_ROUNDS_2(KEY_BASE + 16, I0, I1, I2, I3, T0, T1, T2, T3) \
DECRYPT_TWO_ROUNDS_2(KEY_BASE + 24, I0, I1, I2, I3, T0, T1, T2, T3) \
DECRYPT_TWO_ROUNDS_LAST_2(KEY_BASE + 32, I0, I1, I2, I3, T0, T1, T2, T3)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
/* 12 rounds */
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
#define DECRYPT_192(KEY_BASE, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 0, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 8, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 16, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 24, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 32, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS_LAST(KEY_BASE + 40, I0, I1, T0, T1)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
#define DECRYPT_192_2(KEY_BASE, I0, I1, I2, I3, T0, T1, T2, T3) \
DECRYPT_TWO_ROUNDS_2(KEY_BASE + 0, I0, I1, I2, I3, T0, T1, T2, T3) \
DECRYPT_TWO_ROUNDS_2(KEY_BASE + 8, I0, I1, I2, I3, T0, T1, T2, T3) \
DECRYPT_TWO_ROUNDS_2(KEY_BASE + 16, I0, I1, I2, I3, T0, T1, T2, T3) \
DECRYPT_TWO_ROUNDS_2(KEY_BASE + 24, I0, I1, I2, I3, T0, T1, T2, T3) \
DECRYPT_TWO_ROUNDS_2(KEY_BASE + 32, I0, I1, I2, I3, T0, T1, T2, T3) \
DECRYPT_TWO_ROUNDS_LAST_2(KEY_BASE + 40, I0, I1, I2, I3, T0, T1, T2, T3)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
/* 14 rounds */
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
#define DECRYPT_256(KEY_BASE, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 0, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 8, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 16, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 24, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 32, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS(KEY_BASE + 40, I0, I1, T0, T1) \
DECRYPT_TWO_ROUNDS_LAST(KEY_BASE + 48, I0, I1, T0, T1)
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
#define DECRYPT_256_TWO_ROUNDS_2(KEY_BASE, I0, I1, I2, I3, TMP_BASE) \
DECRYPT_TWO_ROUNDS_2(KEY_BASE, I0, I1, I2, I3, \
TMP_BASE + 0, TMP_BASE + 2, TMP_BASE + 4, TMP_BASE + 6)
#define DECRYPT_256_2(KEY_BASE, I0, I1, I2, I3) \
DECRYPT_256_TWO_ROUNDS_2(KEY_BASE + 0, I0, I1, I2, I3, KEY_BASE + 48) \
ldd [%o0 + 0x18], %f56; \
ldd [%o0 + 0x10], %f58; \
DECRYPT_256_TWO_ROUNDS_2(KEY_BASE + 8, I0, I1, I2, I3, KEY_BASE + 0) \
sparc64: Fix bugs in unrolled 256-bit loops. Some dm-crypt testing revealed several bugs in the 256-bit unrolled loops. The DECRYPT_256_2() macro had two errors: 1) Missing reload of KEY registers %f60 and %f62 2) Missing "\" in penultimate line of definition. In aes_sparc64_ecb_decrypt_256, we were storing the second half of the encryption result from the wrong source registers. In aes_sparc64_ctr_crypt_256 we have to be careful when we fall out of the 32-byte-at-a-time loop and handle a trailing 16-byte chunk. In that case we've clobbered the final key holding registers and have to restore them before executing the ENCRYPT_256() macro. Inside of the 32-byte-at-a-time loop things are OK, because we do this key register restoring during the first few rounds of the ENCRYPT_256_2() macro. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-02 06:05:43 +00:00
ldd [%o0 + 0x08], %f60; \
ldd [%o0 + 0x00], %f62; \
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
DECRYPT_256_TWO_ROUNDS_2(KEY_BASE + 16, I0, I1, I2, I3, KEY_BASE + 0) \
DECRYPT_256_TWO_ROUNDS_2(KEY_BASE + 24, I0, I1, I2, I3, KEY_BASE + 0) \
DECRYPT_256_TWO_ROUNDS_2(KEY_BASE + 32, I0, I1, I2, I3, KEY_BASE + 0) \
DECRYPT_256_TWO_ROUNDS_2(KEY_BASE + 40, I0, I1, I2, I3, KEY_BASE + 0) \
AES_DROUND23(KEY_BASE + 48, I0, I1, KEY_BASE + 2) \
AES_DROUND01(KEY_BASE + 50, I0, I1, KEY_BASE + 0) \
AES_DROUND23(KEY_BASE + 48, I2, I3, KEY_BASE + 6) \
AES_DROUND01(KEY_BASE + 50, I2, I3, KEY_BASE + 4) \
AES_DROUND23_L(KEY_BASE + 52, KEY_BASE + 0, KEY_BASE + 2, I1) \
AES_DROUND01_L(KEY_BASE + 54, KEY_BASE + 0, KEY_BASE + 2, I0) \
ldd [%o0 + 0xd8], %f8; \
ldd [%o0 + 0xd0], %f10; \
AES_DROUND23_L(KEY_BASE + 52, KEY_BASE + 4, KEY_BASE + 6, I3) \
sparc64: Fix bugs in unrolled 256-bit loops. Some dm-crypt testing revealed several bugs in the 256-bit unrolled loops. The DECRYPT_256_2() macro had two errors: 1) Missing reload of KEY registers %f60 and %f62 2) Missing "\" in penultimate line of definition. In aes_sparc64_ecb_decrypt_256, we were storing the second half of the encryption result from the wrong source registers. In aes_sparc64_ctr_crypt_256 we have to be careful when we fall out of the 32-byte-at-a-time loop and handle a trailing 16-byte chunk. In that case we've clobbered the final key holding registers and have to restore them before executing the ENCRYPT_256() macro. Inside of the 32-byte-at-a-time loop things are OK, because we do this key register restoring during the first few rounds of the ENCRYPT_256_2() macro. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-02 06:05:43 +00:00
AES_DROUND01_L(KEY_BASE + 54, KEY_BASE + 4, KEY_BASE + 6, I2) \
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
ldd [%o0 + 0xc8], %f12; \
ldd [%o0 + 0xc0], %f14;
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ENTRY(aes_sparc64_key_expand)
/* %o0=input_key, %o1=output_key, %o2=key_len */
VISEntry
ld [%o0 + 0x00], %f0
ld [%o0 + 0x04], %f1
ld [%o0 + 0x08], %f2
ld [%o0 + 0x0c], %f3
std %f0, [%o1 + 0x00]
std %f2, [%o1 + 0x08]
add %o1, 0x10, %o1
cmp %o2, 24
bl 2f
nop
be 1f
nop
/* 256-bit key expansion */
ld [%o0 + 0x10], %f4
ld [%o0 + 0x14], %f5
ld [%o0 + 0x18], %f6
ld [%o0 + 0x1c], %f7
std %f4, [%o1 + 0x00]
std %f6, [%o1 + 0x08]
add %o1, 0x10, %o1
AES_KEXPAND1(0, 6, 0x0, 8)
AES_KEXPAND2(2, 8, 10)
AES_KEXPAND0(4, 10, 12)
AES_KEXPAND2(6, 12, 14)
AES_KEXPAND1(8, 14, 0x1, 16)
AES_KEXPAND2(10, 16, 18)
AES_KEXPAND0(12, 18, 20)
AES_KEXPAND2(14, 20, 22)
AES_KEXPAND1(16, 22, 0x2, 24)
AES_KEXPAND2(18, 24, 26)
AES_KEXPAND0(20, 26, 28)
AES_KEXPAND2(22, 28, 30)
AES_KEXPAND1(24, 30, 0x3, 32)
AES_KEXPAND2(26, 32, 34)
AES_KEXPAND0(28, 34, 36)
AES_KEXPAND2(30, 36, 38)
AES_KEXPAND1(32, 38, 0x4, 40)
AES_KEXPAND2(34, 40, 42)
AES_KEXPAND0(36, 42, 44)
AES_KEXPAND2(38, 44, 46)
AES_KEXPAND1(40, 46, 0x5, 48)
AES_KEXPAND2(42, 48, 50)
AES_KEXPAND0(44, 50, 52)
AES_KEXPAND2(46, 52, 54)
AES_KEXPAND1(48, 54, 0x6, 56)
AES_KEXPAND2(50, 56, 58)
std %f8, [%o1 + 0x00]
std %f10, [%o1 + 0x08]
std %f12, [%o1 + 0x10]
std %f14, [%o1 + 0x18]
std %f16, [%o1 + 0x20]
std %f18, [%o1 + 0x28]
std %f20, [%o1 + 0x30]
std %f22, [%o1 + 0x38]
std %f24, [%o1 + 0x40]
std %f26, [%o1 + 0x48]
std %f28, [%o1 + 0x50]
std %f30, [%o1 + 0x58]
std %f32, [%o1 + 0x60]
std %f34, [%o1 + 0x68]
std %f36, [%o1 + 0x70]
std %f38, [%o1 + 0x78]
std %f40, [%o1 + 0x80]
std %f42, [%o1 + 0x88]
std %f44, [%o1 + 0x90]
std %f46, [%o1 + 0x98]
std %f48, [%o1 + 0xa0]
std %f50, [%o1 + 0xa8]
std %f52, [%o1 + 0xb0]
std %f54, [%o1 + 0xb8]
std %f56, [%o1 + 0xc0]
ba,pt %xcc, 80f
std %f58, [%o1 + 0xc8]
1:
/* 192-bit key expansion */
ld [%o0 + 0x10], %f4
ld [%o0 + 0x14], %f5
std %f4, [%o1 + 0x00]
add %o1, 0x08, %o1
AES_KEXPAND1(0, 4, 0x0, 6)
AES_KEXPAND2(2, 6, 8)
AES_KEXPAND2(4, 8, 10)
AES_KEXPAND1(6, 10, 0x1, 12)
AES_KEXPAND2(8, 12, 14)
AES_KEXPAND2(10, 14, 16)
AES_KEXPAND1(12, 16, 0x2, 18)
AES_KEXPAND2(14, 18, 20)
AES_KEXPAND2(16, 20, 22)
AES_KEXPAND1(18, 22, 0x3, 24)
AES_KEXPAND2(20, 24, 26)
AES_KEXPAND2(22, 26, 28)
AES_KEXPAND1(24, 28, 0x4, 30)
AES_KEXPAND2(26, 30, 32)
AES_KEXPAND2(28, 32, 34)
AES_KEXPAND1(30, 34, 0x5, 36)
AES_KEXPAND2(32, 36, 38)
AES_KEXPAND2(34, 38, 40)
AES_KEXPAND1(36, 40, 0x6, 42)
AES_KEXPAND2(38, 42, 44)
AES_KEXPAND2(40, 44, 46)
AES_KEXPAND1(42, 46, 0x7, 48)
AES_KEXPAND2(44, 48, 50)
std %f6, [%o1 + 0x00]
std %f8, [%o1 + 0x08]
std %f10, [%o1 + 0x10]
std %f12, [%o1 + 0x18]
std %f14, [%o1 + 0x20]
std %f16, [%o1 + 0x28]
std %f18, [%o1 + 0x30]
std %f20, [%o1 + 0x38]
std %f22, [%o1 + 0x40]
std %f24, [%o1 + 0x48]
std %f26, [%o1 + 0x50]
std %f28, [%o1 + 0x58]
std %f30, [%o1 + 0x60]
std %f32, [%o1 + 0x68]
std %f34, [%o1 + 0x70]
std %f36, [%o1 + 0x78]
std %f38, [%o1 + 0x80]
std %f40, [%o1 + 0x88]
std %f42, [%o1 + 0x90]
std %f44, [%o1 + 0x98]
std %f46, [%o1 + 0xa0]
std %f48, [%o1 + 0xa8]
ba,pt %xcc, 80f
std %f50, [%o1 + 0xb0]
2:
/* 128-bit key expansion */
AES_KEXPAND1(0, 2, 0x0, 4)
AES_KEXPAND2(2, 4, 6)
AES_KEXPAND1(4, 6, 0x1, 8)
AES_KEXPAND2(6, 8, 10)
AES_KEXPAND1(8, 10, 0x2, 12)
AES_KEXPAND2(10, 12, 14)
AES_KEXPAND1(12, 14, 0x3, 16)
AES_KEXPAND2(14, 16, 18)
AES_KEXPAND1(16, 18, 0x4, 20)
AES_KEXPAND2(18, 20, 22)
AES_KEXPAND1(20, 22, 0x5, 24)
AES_KEXPAND2(22, 24, 26)
AES_KEXPAND1(24, 26, 0x6, 28)
AES_KEXPAND2(26, 28, 30)
AES_KEXPAND1(28, 30, 0x7, 32)
AES_KEXPAND2(30, 32, 34)
AES_KEXPAND1(32, 34, 0x8, 36)
AES_KEXPAND2(34, 36, 38)
AES_KEXPAND1(36, 38, 0x9, 40)
AES_KEXPAND2(38, 40, 42)
std %f4, [%o1 + 0x00]
std %f6, [%o1 + 0x08]
std %f8, [%o1 + 0x10]
std %f10, [%o1 + 0x18]
std %f12, [%o1 + 0x20]
std %f14, [%o1 + 0x28]
std %f16, [%o1 + 0x30]
std %f18, [%o1 + 0x38]
std %f20, [%o1 + 0x40]
std %f22, [%o1 + 0x48]
std %f24, [%o1 + 0x50]
std %f26, [%o1 + 0x58]
std %f28, [%o1 + 0x60]
std %f30, [%o1 + 0x68]
std %f32, [%o1 + 0x70]
std %f34, [%o1 + 0x78]
std %f36, [%o1 + 0x80]
std %f38, [%o1 + 0x88]
std %f40, [%o1 + 0x90]
std %f42, [%o1 + 0x98]
80:
retl
VISExit
ENDPROC(aes_sparc64_key_expand)
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_encrypt_128)
/* %o0=key, %o1=input, %o2=output */
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
VISEntry
ld [%o1 + 0x00], %f4
ld [%o1 + 0x04], %f5
ld [%o1 + 0x08], %f6
ld [%o1 + 0x0c], %f7
ldd [%o0 + 0x00], %f8
ldd [%o0 + 0x08], %f10
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ldd [%o0 + 0x10], %f12
ldd [%o0 + 0x18], %f14
ldd [%o0 + 0x20], %f16
ldd [%o0 + 0x28], %f18
ldd [%o0 + 0x30], %f20
ldd [%o0 + 0x38], %f22
ldd [%o0 + 0x40], %f24
ldd [%o0 + 0x48], %f26
ldd [%o0 + 0x50], %f28
ldd [%o0 + 0x58], %f30
ldd [%o0 + 0x60], %f32
ldd [%o0 + 0x68], %f34
ldd [%o0 + 0x70], %f36
ldd [%o0 + 0x78], %f38
ldd [%o0 + 0x80], %f40
ldd [%o0 + 0x88], %f42
ldd [%o0 + 0x90], %f44
ldd [%o0 + 0x98], %f46
ldd [%o0 + 0xa0], %f48
ldd [%o0 + 0xa8], %f50
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
fxor %f8, %f4, %f4
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
fxor %f10, %f6, %f6
ENCRYPT_128(12, 4, 6, 0, 2)
st %f4, [%o2 + 0x00]
st %f5, [%o2 + 0x04]
st %f6, [%o2 + 0x08]
st %f7, [%o2 + 0x0c]
retl
VISExit
ENDPROC(aes_sparc64_encrypt_128)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_encrypt_192)
/* %o0=key, %o1=input, %o2=output */
VISEntry
ld [%o1 + 0x00], %f4
ld [%o1 + 0x04], %f5
ld [%o1 + 0x08], %f6
ld [%o1 + 0x0c], %f7
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ldd [%o0 + 0x00], %f8
ldd [%o0 + 0x08], %f10
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
fxor %f8, %f4, %f4
fxor %f10, %f6, %f6
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldd [%o0 + 0x10], %f8
ldd [%o0 + 0x18], %f10
ldd [%o0 + 0x20], %f12
ldd [%o0 + 0x28], %f14
add %o0, 0x20, %o0
ENCRYPT_TWO_ROUNDS(8, 4, 6, 0, 2)
ldd [%o0 + 0x10], %f12
ldd [%o0 + 0x18], %f14
ldd [%o0 + 0x20], %f16
ldd [%o0 + 0x28], %f18
ldd [%o0 + 0x30], %f20
ldd [%o0 + 0x38], %f22
ldd [%o0 + 0x40], %f24
ldd [%o0 + 0x48], %f26
ldd [%o0 + 0x50], %f28
ldd [%o0 + 0x58], %f30
ldd [%o0 + 0x60], %f32
ldd [%o0 + 0x68], %f34
ldd [%o0 + 0x70], %f36
ldd [%o0 + 0x78], %f38
ldd [%o0 + 0x80], %f40
ldd [%o0 + 0x88], %f42
ldd [%o0 + 0x90], %f44
ldd [%o0 + 0x98], %f46
ldd [%o0 + 0xa0], %f48
ldd [%o0 + 0xa8], %f50
ENCRYPT_128(12, 4, 6, 0, 2)
st %f4, [%o2 + 0x00]
st %f5, [%o2 + 0x04]
st %f6, [%o2 + 0x08]
st %f7, [%o2 + 0x0c]
retl
VISExit
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_encrypt_192)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_encrypt_256)
/* %o0=key, %o1=input, %o2=output */
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
VISEntry
ld [%o1 + 0x00], %f4
ld [%o1 + 0x04], %f5
ld [%o1 + 0x08], %f6
ld [%o1 + 0x0c], %f7
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ldd [%o0 + 0x00], %f8
ldd [%o0 + 0x08], %f10
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
fxor %f8, %f4, %f4
fxor %f10, %f6, %f6
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ldd [%o0 + 0x10], %f8
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ldd [%o0 + 0x18], %f10
ldd [%o0 + 0x20], %f12
ldd [%o0 + 0x28], %f14
add %o0, 0x20, %o0
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENCRYPT_TWO_ROUNDS(8, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ldd [%o0 + 0x10], %f8
ldd [%o0 + 0x18], %f10
ldd [%o0 + 0x20], %f12
ldd [%o0 + 0x28], %f14
add %o0, 0x20, %o0
ENCRYPT_TWO_ROUNDS(8, 4, 6, 0, 2)
ldd [%o0 + 0x10], %f12
ldd [%o0 + 0x18], %f14
ldd [%o0 + 0x20], %f16
ldd [%o0 + 0x28], %f18
ldd [%o0 + 0x30], %f20
ldd [%o0 + 0x38], %f22
ldd [%o0 + 0x40], %f24
ldd [%o0 + 0x48], %f26
ldd [%o0 + 0x50], %f28
ldd [%o0 + 0x58], %f30
ldd [%o0 + 0x60], %f32
ldd [%o0 + 0x68], %f34
ldd [%o0 + 0x70], %f36
ldd [%o0 + 0x78], %f38
ldd [%o0 + 0x80], %f40
ldd [%o0 + 0x88], %f42
ldd [%o0 + 0x90], %f44
ldd [%o0 + 0x98], %f46
ldd [%o0 + 0xa0], %f48
ldd [%o0 + 0xa8], %f50
ENCRYPT_128(12, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
st %f4, [%o2 + 0x00]
st %f5, [%o2 + 0x04]
st %f6, [%o2 + 0x08]
st %f7, [%o2 + 0x0c]
retl
VISExit
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_encrypt_256)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_decrypt_128)
/* %o0=key, %o1=input, %o2=output */
VISEntry
ld [%o1 + 0x00], %f4
ld [%o1 + 0x04], %f5
ld [%o1 + 0x08], %f6
ld [%o1 + 0x0c], %f7
ldd [%o0 + 0xa0], %f8
ldd [%o0 + 0xa8], %f10
ldd [%o0 + 0x98], %f12
ldd [%o0 + 0x90], %f14
ldd [%o0 + 0x88], %f16
ldd [%o0 + 0x80], %f18
ldd [%o0 + 0x78], %f20
ldd [%o0 + 0x70], %f22
ldd [%o0 + 0x68], %f24
ldd [%o0 + 0x60], %f26
ldd [%o0 + 0x58], %f28
ldd [%o0 + 0x50], %f30
ldd [%o0 + 0x48], %f32
ldd [%o0 + 0x40], %f34
ldd [%o0 + 0x38], %f36
ldd [%o0 + 0x30], %f38
ldd [%o0 + 0x28], %f40
ldd [%o0 + 0x20], %f42
ldd [%o0 + 0x18], %f44
ldd [%o0 + 0x10], %f46
ldd [%o0 + 0x08], %f48
ldd [%o0 + 0x00], %f50
fxor %f8, %f4, %f4
fxor %f10, %f6, %f6
DECRYPT_128(12, 4, 6, 0, 2)
st %f4, [%o2 + 0x00]
st %f5, [%o2 + 0x04]
st %f6, [%o2 + 0x08]
st %f7, [%o2 + 0x0c]
retl
VISExit
ENDPROC(aes_sparc64_decrypt_128)
.align 32
ENTRY(aes_sparc64_decrypt_192)
/* %o0=key, %o1=input, %o2=output */
VISEntry
ld [%o1 + 0x00], %f4
ld [%o1 + 0x04], %f5
ld [%o1 + 0x08], %f6
ld [%o1 + 0x0c], %f7
ldd [%o0 + 0xc0], %f8
ldd [%o0 + 0xc8], %f10
ldd [%o0 + 0xb8], %f12
ldd [%o0 + 0xb0], %f14
ldd [%o0 + 0xa8], %f16
ldd [%o0 + 0xa0], %f18
fxor %f8, %f4, %f4
fxor %f10, %f6, %f6
ldd [%o0 + 0x98], %f20
ldd [%o0 + 0x90], %f22
ldd [%o0 + 0x88], %f24
ldd [%o0 + 0x80], %f26
DECRYPT_TWO_ROUNDS(12, 4, 6, 0, 2)
ldd [%o0 + 0x78], %f28
ldd [%o0 + 0x70], %f30
ldd [%o0 + 0x68], %f32
ldd [%o0 + 0x60], %f34
ldd [%o0 + 0x58], %f36
ldd [%o0 + 0x50], %f38
ldd [%o0 + 0x48], %f40
ldd [%o0 + 0x40], %f42
ldd [%o0 + 0x38], %f44
ldd [%o0 + 0x30], %f46
ldd [%o0 + 0x28], %f48
ldd [%o0 + 0x20], %f50
ldd [%o0 + 0x18], %f52
ldd [%o0 + 0x10], %f54
ldd [%o0 + 0x08], %f56
ldd [%o0 + 0x00], %f58
DECRYPT_128(20, 4, 6, 0, 2)
st %f4, [%o2 + 0x00]
st %f5, [%o2 + 0x04]
st %f6, [%o2 + 0x08]
st %f7, [%o2 + 0x0c]
retl
VISExit
ENDPROC(aes_sparc64_decrypt_192)
.align 32
ENTRY(aes_sparc64_decrypt_256)
/* %o0=key, %o1=input, %o2=output */
VISEntry
ld [%o1 + 0x00], %f4
ld [%o1 + 0x04], %f5
ld [%o1 + 0x08], %f6
ld [%o1 + 0x0c], %f7
ldd [%o0 + 0xe0], %f8
ldd [%o0 + 0xe8], %f10
ldd [%o0 + 0xd8], %f12
ldd [%o0 + 0xd0], %f14
ldd [%o0 + 0xc8], %f16
fxor %f8, %f4, %f4
ldd [%o0 + 0xc0], %f18
fxor %f10, %f6, %f6
ldd [%o0 + 0xb8], %f20
AES_DROUND23(12, 4, 6, 2)
ldd [%o0 + 0xb0], %f22
AES_DROUND01(14, 4, 6, 0)
ldd [%o0 + 0xa8], %f24
AES_DROUND23(16, 0, 2, 6)
ldd [%o0 + 0xa0], %f26
AES_DROUND01(18, 0, 2, 4)
ldd [%o0 + 0x98], %f12
AES_DROUND23(20, 4, 6, 2)
ldd [%o0 + 0x90], %f14
AES_DROUND01(22, 4, 6, 0)
ldd [%o0 + 0x88], %f16
AES_DROUND23(24, 0, 2, 6)
ldd [%o0 + 0x80], %f18
AES_DROUND01(26, 0, 2, 4)
ldd [%o0 + 0x78], %f20
AES_DROUND23(12, 4, 6, 2)
ldd [%o0 + 0x70], %f22
AES_DROUND01(14, 4, 6, 0)
ldd [%o0 + 0x68], %f24
AES_DROUND23(16, 0, 2, 6)
ldd [%o0 + 0x60], %f26
AES_DROUND01(18, 0, 2, 4)
ldd [%o0 + 0x58], %f28
AES_DROUND23(20, 4, 6, 2)
ldd [%o0 + 0x50], %f30
AES_DROUND01(22, 4, 6, 0)
ldd [%o0 + 0x48], %f32
AES_DROUND23(24, 0, 2, 6)
ldd [%o0 + 0x40], %f34
AES_DROUND01(26, 0, 2, 4)
ldd [%o0 + 0x38], %f36
AES_DROUND23(28, 4, 6, 2)
ldd [%o0 + 0x30], %f38
AES_DROUND01(30, 4, 6, 0)
ldd [%o0 + 0x28], %f40
AES_DROUND23(32, 0, 2, 6)
ldd [%o0 + 0x20], %f42
AES_DROUND01(34, 0, 2, 4)
ldd [%o0 + 0x18], %f44
AES_DROUND23(36, 4, 6, 2)
ldd [%o0 + 0x10], %f46
AES_DROUND01(38, 4, 6, 0)
ldd [%o0 + 0x08], %f48
AES_DROUND23(40, 0, 2, 6)
ldd [%o0 + 0x00], %f50
AES_DROUND01(42, 0, 2, 4)
AES_DROUND23(44, 4, 6, 2)
AES_DROUND01(46, 4, 6, 0)
AES_DROUND23_L(48, 0, 2, 6)
AES_DROUND01_L(50, 0, 2, 4)
st %f4, [%o2 + 0x00]
st %f5, [%o2 + 0x04]
st %f6, [%o2 + 0x08]
st %f7, [%o2 + 0x0c]
retl
VISExit
ENDPROC(aes_sparc64_decrypt_256)
.align 32
ENTRY(aes_sparc64_load_encrypt_keys_128)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
/* %o0=key */
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
VISEntry
ldd [%o0 + 0x10], %f8
ldd [%o0 + 0x18], %f10
ldd [%o0 + 0x20], %f12
ldd [%o0 + 0x28], %f14
ldd [%o0 + 0x30], %f16
ldd [%o0 + 0x38], %f18
ldd [%o0 + 0x40], %f20
ldd [%o0 + 0x48], %f22
ldd [%o0 + 0x50], %f24
ldd [%o0 + 0x58], %f26
ldd [%o0 + 0x60], %f28
ldd [%o0 + 0x68], %f30
ldd [%o0 + 0x70], %f32
ldd [%o0 + 0x78], %f34
ldd [%o0 + 0x80], %f36
ldd [%o0 + 0x88], %f38
ldd [%o0 + 0x90], %f40
ldd [%o0 + 0x98], %f42
ldd [%o0 + 0xa0], %f44
retl
ldd [%o0 + 0xa8], %f46
ENDPROC(aes_sparc64_load_encrypt_keys_128)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_load_encrypt_keys_192)
/* %o0=key */
VISEntry
ldd [%o0 + 0x10], %f8
ldd [%o0 + 0x18], %f10
ldd [%o0 + 0x20], %f12
ldd [%o0 + 0x28], %f14
ldd [%o0 + 0x30], %f16
ldd [%o0 + 0x38], %f18
ldd [%o0 + 0x40], %f20
ldd [%o0 + 0x48], %f22
ldd [%o0 + 0x50], %f24
ldd [%o0 + 0x58], %f26
ldd [%o0 + 0x60], %f28
ldd [%o0 + 0x68], %f30
ldd [%o0 + 0x70], %f32
ldd [%o0 + 0x78], %f34
ldd [%o0 + 0x80], %f36
ldd [%o0 + 0x88], %f38
ldd [%o0 + 0x90], %f40
ldd [%o0 + 0x98], %f42
ldd [%o0 + 0xa0], %f44
ldd [%o0 + 0xa8], %f46
ldd [%o0 + 0xb0], %f48
ldd [%o0 + 0xb8], %f50
ldd [%o0 + 0xc0], %f52
retl
ldd [%o0 + 0xc8], %f54
ENDPROC(aes_sparc64_load_encrypt_keys_192)
.align 32
ENTRY(aes_sparc64_load_encrypt_keys_256)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
/* %o0=key */
VISEntry
ldd [%o0 + 0x10], %f8
ldd [%o0 + 0x18], %f10
ldd [%o0 + 0x20], %f12
ldd [%o0 + 0x28], %f14
ldd [%o0 + 0x30], %f16
ldd [%o0 + 0x38], %f18
ldd [%o0 + 0x40], %f20
ldd [%o0 + 0x48], %f22
ldd [%o0 + 0x50], %f24
ldd [%o0 + 0x58], %f26
ldd [%o0 + 0x60], %f28
ldd [%o0 + 0x68], %f30
ldd [%o0 + 0x70], %f32
ldd [%o0 + 0x78], %f34
ldd [%o0 + 0x80], %f36
ldd [%o0 + 0x88], %f38
ldd [%o0 + 0x90], %f40
ldd [%o0 + 0x98], %f42
ldd [%o0 + 0xa0], %f44
ldd [%o0 + 0xa8], %f46
ldd [%o0 + 0xb0], %f48
ldd [%o0 + 0xb8], %f50
ldd [%o0 + 0xc0], %f52
ldd [%o0 + 0xc8], %f54
ldd [%o0 + 0xd0], %f56
ldd [%o0 + 0xd8], %f58
ldd [%o0 + 0xe0], %f60
retl
ldd [%o0 + 0xe8], %f62
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_load_encrypt_keys_256)
.align 32
ENTRY(aes_sparc64_load_decrypt_keys_128)
/* %o0=key */
VISEntry
ldd [%o0 + 0x98], %f8
ldd [%o0 + 0x90], %f10
ldd [%o0 + 0x88], %f12
ldd [%o0 + 0x80], %f14
ldd [%o0 + 0x78], %f16
ldd [%o0 + 0x70], %f18
ldd [%o0 + 0x68], %f20
ldd [%o0 + 0x60], %f22
ldd [%o0 + 0x58], %f24
ldd [%o0 + 0x50], %f26
ldd [%o0 + 0x48], %f28
ldd [%o0 + 0x40], %f30
ldd [%o0 + 0x38], %f32
ldd [%o0 + 0x30], %f34
ldd [%o0 + 0x28], %f36
ldd [%o0 + 0x20], %f38
ldd [%o0 + 0x18], %f40
ldd [%o0 + 0x10], %f42
ldd [%o0 + 0x08], %f44
retl
ldd [%o0 + 0x00], %f46
ENDPROC(aes_sparc64_load_decrypt_keys_128)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_load_decrypt_keys_192)
/* %o0=key */
VISEntry
ldd [%o0 + 0xb8], %f8
ldd [%o0 + 0xb0], %f10
ldd [%o0 + 0xa8], %f12
ldd [%o0 + 0xa0], %f14
ldd [%o0 + 0x98], %f16
ldd [%o0 + 0x90], %f18
ldd [%o0 + 0x88], %f20
ldd [%o0 + 0x80], %f22
ldd [%o0 + 0x78], %f24
ldd [%o0 + 0x70], %f26
ldd [%o0 + 0x68], %f28
ldd [%o0 + 0x60], %f30
ldd [%o0 + 0x58], %f32
ldd [%o0 + 0x50], %f34
ldd [%o0 + 0x48], %f36
ldd [%o0 + 0x40], %f38
ldd [%o0 + 0x38], %f40
ldd [%o0 + 0x30], %f42
ldd [%o0 + 0x28], %f44
ldd [%o0 + 0x20], %f46
ldd [%o0 + 0x18], %f48
ldd [%o0 + 0x10], %f50
ldd [%o0 + 0x08], %f52
retl
ldd [%o0 + 0x00], %f54
ENDPROC(aes_sparc64_load_decrypt_keys_192)
.align 32
ENTRY(aes_sparc64_load_decrypt_keys_256)
/* %o0=key */
VISEntry
ldd [%o0 + 0xd8], %f8
ldd [%o0 + 0xd0], %f10
ldd [%o0 + 0xc8], %f12
ldd [%o0 + 0xc0], %f14
ldd [%o0 + 0xb8], %f16
ldd [%o0 + 0xb0], %f18
ldd [%o0 + 0xa8], %f20
ldd [%o0 + 0xa0], %f22
ldd [%o0 + 0x98], %f24
ldd [%o0 + 0x90], %f26
ldd [%o0 + 0x88], %f28
ldd [%o0 + 0x80], %f30
ldd [%o0 + 0x78], %f32
ldd [%o0 + 0x70], %f34
ldd [%o0 + 0x68], %f36
ldd [%o0 + 0x60], %f38
ldd [%o0 + 0x58], %f40
ldd [%o0 + 0x50], %f42
ldd [%o0 + 0x48], %f44
ldd [%o0 + 0x40], %f46
ldd [%o0 + 0x38], %f48
ldd [%o0 + 0x30], %f50
ldd [%o0 + 0x28], %f52
ldd [%o0 + 0x20], %f54
ldd [%o0 + 0x18], %f56
ldd [%o0 + 0x10], %f58
ldd [%o0 + 0x08], %f60
retl
ldd [%o0 + 0x00], %f62
ENDPROC(aes_sparc64_load_decrypt_keys_256)
.align 32
ENTRY(aes_sparc64_ecb_encrypt_128)
/* %o0=key, %o1=input, %o2=output, %o3=len */
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o0 + 0x00], %g1
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
subcc %o3, 0x10, %o3
be 10f
ldx [%o0 + 0x08], %g2
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
1: ldx [%o1 + 0x00], %g3
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o1 + 0x08], %g7
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
ldx [%o1 + 0x10], %o4
ldx [%o1 + 0x18], %o5
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
xor %g1, %o4, %g3
xor %g2, %o5, %g7
MOVXTOD_G3_F60
MOVXTOD_G7_F62
ENCRYPT_128_2(8, 4, 6, 60, 62, 0, 2, 56, 58)
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
std %f60, [%o2 + 0x10]
std %f62, [%o2 + 0x18]
sub %o3, 0x20, %o3
add %o1, 0x20, %o1
brgz %o3, 1b
add %o2, 0x20, %o2
brlz,pt %o3, 11f
nop
10: ldx [%o1 + 0x00], %g3
ldx [%o1 + 0x08], %g7
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENCRYPT_128(8, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
11: retl
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_ecb_encrypt_128)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_ecb_encrypt_192)
/* %o0=key, %o1=input, %o2=output, %o3=len */
ldx [%o0 + 0x00], %g1
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
subcc %o3, 0x10, %o3
be 10f
ldx [%o0 + 0x08], %g2
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
1: ldx [%o1 + 0x00], %g3
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o1 + 0x08], %g7
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
ldx [%o1 + 0x10], %o4
ldx [%o1 + 0x18], %o5
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
xor %g1, %o4, %g3
xor %g2, %o5, %g7
MOVXTOD_G3_F60
MOVXTOD_G7_F62
ENCRYPT_192_2(8, 4, 6, 60, 62, 0, 2, 56, 58)
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
std %f60, [%o2 + 0x10]
std %f62, [%o2 + 0x18]
sub %o3, 0x20, %o3
add %o1, 0x20, %o1
brgz %o3, 1b
add %o2, 0x20, %o2
brlz,pt %o3, 11f
nop
10: ldx [%o1 + 0x00], %g3
ldx [%o1 + 0x08], %g7
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
ENCRYPT_192(8, 4, 6, 0, 2)
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
11: retl
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_ecb_encrypt_192)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_ecb_encrypt_256)
/* %o0=key, %o1=input, %o2=output, %o3=len */
ldx [%o0 + 0x00], %g1
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
subcc %o3, 0x10, %o3
be 10f
ldx [%o0 + 0x08], %g2
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
1: ldx [%o1 + 0x00], %g3
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o1 + 0x08], %g7
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
ldx [%o1 + 0x10], %o4
ldx [%o1 + 0x18], %o5
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
xor %g1, %o4, %g3
xor %g2, %o5, %g7
MOVXTOD_G3_F0
MOVXTOD_G7_F2
ENCRYPT_256_2(8, 4, 6, 0, 2)
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
std %f0, [%o2 + 0x10]
std %f2, [%o2 + 0x18]
sub %o3, 0x20, %o3
add %o1, 0x20, %o1
brgz %o3, 1b
add %o2, 0x20, %o2
brlz,pt %o3, 11f
nop
sparc64: Fix unrolled AES 256-bit key loops. The basic scheme of the block mode assembler is that we start by enabling the FPU, loading the key into the floating point registers, then iterate calling the encrypt/decrypt routine for each block. For the 256-bit key cases, we run short on registers in the unrolled loops. So the {ENCRYPT,DECRYPT}_256_2() macros reload the key registers that get clobbered. The unrolled macros, {ENCRYPT,DECRYPT}_256(), are not mindful of this. So if we have a mix of multi-block and single-block calls, the single-block unrolled 256-bit encrypt/decrypt can run with some of the key registers clobbered. Handle this by always explicitly loading those registers before using the non-unrolled 256-bit macro. This was discovered thanks to all of the new test cases added by Jussi Kivilinna. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 23:19:11 +00:00
10: ldd [%o0 + 0xd0], %f56
ldd [%o0 + 0xd8], %f58
ldd [%o0 + 0xe0], %f60
ldd [%o0 + 0xe8], %f62
ldx [%o1 + 0x00], %g3
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
ldx [%o1 + 0x08], %g7
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENCRYPT_256(8, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Unroll ECB encryption loops in AES driver. The AES opcodes have a 3 cycle latency, so by doing 32-bytes at a time we avoid a pipeline bubble in between every round. For the 256-bit key case, it looks like we're doing more work in order to reload the KEY registers during the loop to make space for scarce temporaries. But the load dual issues with the AES operations so we get the KEY reloads essentially for free. Before: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 264 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 329 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 715 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4248 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 221 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 359 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 803 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5366 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 379 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6041 cycles (8192 bytes) After: testing speed of ecb(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 266 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 256 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 305 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 676 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3981 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 766 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5136 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 268 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 368 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 890 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5718 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 14:51:32 +00:00
11: retl
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_ecb_encrypt_256)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_ecb_decrypt_128)
/* %o0=&key[key_len], %o1=input, %o2=output, %o3=len */
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o0 - 0x10], %g1
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
subcc %o3, 0x10, %o3
be 10f
ldx [%o0 - 0x08], %g2
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
1: ldx [%o1 + 0x00], %g3
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o1 + 0x08], %g7
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
ldx [%o1 + 0x10], %o4
ldx [%o1 + 0x18], %o5
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
xor %g1, %o4, %g3
xor %g2, %o5, %g7
MOVXTOD_G3_F60
MOVXTOD_G7_F62
DECRYPT_128_2(8, 4, 6, 60, 62, 0, 2, 56, 58)
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
std %f60, [%o2 + 0x10]
std %f62, [%o2 + 0x18]
sub %o3, 0x20, %o3
add %o1, 0x20, %o1
brgz,pt %o3, 1b
add %o2, 0x20, %o2
brlz,pt %o3, 11f
nop
10: ldx [%o1 + 0x00], %g3
ldx [%o1 + 0x08], %g7
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
DECRYPT_128(8, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
11: retl
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_ecb_decrypt_128)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_ecb_decrypt_192)
/* %o0=&key[key_len], %o1=input, %o2=output, %o3=len */
ldx [%o0 - 0x10], %g1
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
subcc %o3, 0x10, %o3
be 10f
ldx [%o0 - 0x08], %g2
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
1: ldx [%o1 + 0x00], %g3
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o1 + 0x08], %g7
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
ldx [%o1 + 0x10], %o4
ldx [%o1 + 0x18], %o5
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
xor %g1, %o4, %g3
xor %g2, %o5, %g7
MOVXTOD_G3_F60
MOVXTOD_G7_F62
DECRYPT_192_2(8, 4, 6, 60, 62, 0, 2, 56, 58)
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
std %f60, [%o2 + 0x10]
std %f62, [%o2 + 0x18]
sub %o3, 0x20, %o3
add %o1, 0x20, %o1
brgz,pt %o3, 1b
add %o2, 0x20, %o2
brlz,pt %o3, 11f
nop
10: ldx [%o1 + 0x00], %g3
ldx [%o1 + 0x08], %g7
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
DECRYPT_192(8, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
11: retl
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_ecb_decrypt_192)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_ecb_decrypt_256)
/* %o0=&key[key_len], %o1=input, %o2=output, %o3=len */
ldx [%o0 - 0x10], %g1
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
subcc %o3, 0x10, %o3
sparc64: Fix unrolled AES 256-bit key loops. The basic scheme of the block mode assembler is that we start by enabling the FPU, loading the key into the floating point registers, then iterate calling the encrypt/decrypt routine for each block. For the 256-bit key cases, we run short on registers in the unrolled loops. So the {ENCRYPT,DECRYPT}_256_2() macros reload the key registers that get clobbered. The unrolled macros, {ENCRYPT,DECRYPT}_256(), are not mindful of this. So if we have a mix of multi-block and single-block calls, the single-block unrolled 256-bit encrypt/decrypt can run with some of the key registers clobbered. Handle this by always explicitly loading those registers before using the non-unrolled 256-bit macro. This was discovered thanks to all of the new test cases added by Jussi Kivilinna. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 23:19:11 +00:00
ldx [%o0 - 0x08], %g2
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
be 10f
sparc64: Fix unrolled AES 256-bit key loops. The basic scheme of the block mode assembler is that we start by enabling the FPU, loading the key into the floating point registers, then iterate calling the encrypt/decrypt routine for each block. For the 256-bit key cases, we run short on registers in the unrolled loops. So the {ENCRYPT,DECRYPT}_256_2() macros reload the key registers that get clobbered. The unrolled macros, {ENCRYPT,DECRYPT}_256(), are not mindful of this. So if we have a mix of multi-block and single-block calls, the single-block unrolled 256-bit encrypt/decrypt can run with some of the key registers clobbered. Handle this by always explicitly loading those registers before using the non-unrolled 256-bit macro. This was discovered thanks to all of the new test cases added by Jussi Kivilinna. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 23:19:11 +00:00
sub %o0, 0xf0, %o0
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
1: ldx [%o1 + 0x00], %g3
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o1 + 0x08], %g7
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
ldx [%o1 + 0x10], %o4
ldx [%o1 + 0x18], %o5
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
xor %g1, %o4, %g3
xor %g2, %o5, %g7
MOVXTOD_G3_F0
MOVXTOD_G7_F2
DECRYPT_256_2(8, 4, 6, 0, 2)
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Fix bugs in unrolled 256-bit loops. Some dm-crypt testing revealed several bugs in the 256-bit unrolled loops. The DECRYPT_256_2() macro had two errors: 1) Missing reload of KEY registers %f60 and %f62 2) Missing "\" in penultimate line of definition. In aes_sparc64_ecb_decrypt_256, we were storing the second half of the encryption result from the wrong source registers. In aes_sparc64_ctr_crypt_256 we have to be careful when we fall out of the 32-byte-at-a-time loop and handle a trailing 16-byte chunk. In that case we've clobbered the final key holding registers and have to restore them before executing the ENCRYPT_256() macro. Inside of the 32-byte-at-a-time loop things are OK, because we do this key register restoring during the first few rounds of the ENCRYPT_256_2() macro. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-02 06:05:43 +00:00
std %f0, [%o2 + 0x10]
std %f2, [%o2 + 0x18]
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
sub %o3, 0x20, %o3
add %o1, 0x20, %o1
brgz,pt %o3, 1b
add %o2, 0x20, %o2
brlz,pt %o3, 11f
nop
sparc64: Fix unrolled AES 256-bit key loops. The basic scheme of the block mode assembler is that we start by enabling the FPU, loading the key into the floating point registers, then iterate calling the encrypt/decrypt routine for each block. For the 256-bit key cases, we run short on registers in the unrolled loops. So the {ENCRYPT,DECRYPT}_256_2() macros reload the key registers that get clobbered. The unrolled macros, {ENCRYPT,DECRYPT}_256(), are not mindful of this. So if we have a mix of multi-block and single-block calls, the single-block unrolled 256-bit encrypt/decrypt can run with some of the key registers clobbered. Handle this by always explicitly loading those registers before using the non-unrolled 256-bit macro. This was discovered thanks to all of the new test cases added by Jussi Kivilinna. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 23:19:11 +00:00
10: ldd [%o0 + 0x18], %f56
ldd [%o0 + 0x10], %f58
ldd [%o0 + 0x08], %f60
ldd [%o0 + 0x00], %f62
ldx [%o1 + 0x00], %g3
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
ldx [%o1 + 0x08], %g7
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
DECRYPT_256(8, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Unroll ECB decryption loops in AES driver. Before: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 223 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 230 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 325 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 719 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4266 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 234 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 353 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 808 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5344 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 243 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 393 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6039 cycles (8192 bytes) After: testing speed of ecb(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 226 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 231 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 313 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 681 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 3964 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 205 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 341 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 770 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5050 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 216 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 250 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 371 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 869 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 5494 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:11:01 +00:00
11: retl
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_ecb_decrypt_256)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_cbc_encrypt_128)
/* %o0=key, %o1=input, %o2=output, %o3=len, %o4=IV */
ldd [%o4 + 0x00], %f4
ldd [%o4 + 0x08], %f6
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o0 + 0x00], %g1
ldx [%o0 + 0x08], %g2
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
1: ldx [%o1 + 0x00], %g3
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o1 + 0x08], %g7
add %o1, 0x10, %o1
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F0
MOVXTOD_G7_F2
fxor %f4, %f0, %f4
fxor %f6, %f2, %f6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENCRYPT_128(8, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
subcc %o3, 0x10, %o3
bne,pt %xcc, 1b
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
add %o2, 0x10, %o2
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
std %f4, [%o4 + 0x00]
std %f6, [%o4 + 0x08]
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
retl
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_cbc_encrypt_128)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_cbc_encrypt_192)
/* %o0=key, %o1=input, %o2=output, %o3=len, %o4=IV */
ldd [%o4 + 0x00], %f4
ldd [%o4 + 0x08], %f6
ldx [%o0 + 0x00], %g1
ldx [%o0 + 0x08], %g2
1: ldx [%o1 + 0x00], %g3
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o1 + 0x08], %g7
add %o1, 0x10, %o1
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F0
MOVXTOD_G7_F2
fxor %f4, %f0, %f4
fxor %f6, %f2, %f6
ENCRYPT_192(8, 4, 6, 0, 2)
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
subcc %o3, 0x10, %o3
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
bne,pt %xcc, 1b
add %o2, 0x10, %o2
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
std %f4, [%o4 + 0x00]
std %f6, [%o4 + 0x08]
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
retl
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_cbc_encrypt_192)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_cbc_encrypt_256)
/* %o0=key, %o1=input, %o2=output, %o3=len, %o4=IV */
ldd [%o4 + 0x00], %f4
ldd [%o4 + 0x08], %f6
ldx [%o0 + 0x00], %g1
ldx [%o0 + 0x08], %g2
1: ldx [%o1 + 0x00], %g3
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o1 + 0x08], %g7
add %o1, 0x10, %o1
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F0
MOVXTOD_G7_F2
fxor %f4, %f0, %f4
fxor %f6, %f2, %f6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENCRYPT_256(8, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
subcc %o3, 0x10, %o3
bne,pt %xcc, 1b
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
add %o2, 0x10, %o2
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
std %f4, [%o4 + 0x00]
std %f6, [%o4 + 0x08]
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
retl
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_cbc_encrypt_256)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_cbc_decrypt_128)
/* %o0=&key[key_len], %o1=input, %o2=output, %o3=len, %o4=iv */
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
ldx [%o0 - 0x10], %g1
ldx [%o0 - 0x08], %g2
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ldx [%o4 + 0x00], %o0
ldx [%o4 + 0x08], %o5
1: ldx [%o1 + 0x00], %g3
ldx [%o1 + 0x08], %g7
add %o1, 0x10, %o1
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
DECRYPT_128(8, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
MOVXTOD_O0_F0
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
MOVXTOD_O5_F2
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %o0
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
xor %g2, %g7, %o5
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
fxor %f4, %f0, %f4
fxor %f6, %f2, %f6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
subcc %o3, 0x10, %o3
bne,pt %xcc, 1b
add %o2, 0x10, %o2
stx %o0, [%o4 + 0x00]
stx %o5, [%o4 + 0x08]
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
retl
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_cbc_decrypt_128)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_cbc_decrypt_192)
/* %o0=&key[key_len], %o1=input, %o2=output, %o3=len, %o4=iv */
ldx [%o0 - 0x10], %g1
ldx [%o0 - 0x08], %g2
ldx [%o4 + 0x00], %o0
ldx [%o4 + 0x08], %o5
1: ldx [%o1 + 0x00], %g3
ldx [%o1 + 0x08], %g7
add %o1, 0x10, %o1
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
DECRYPT_192(8, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
MOVXTOD_O0_F0
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
MOVXTOD_O5_F2
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %o0
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
xor %g2, %g7, %o5
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
fxor %f4, %f0, %f4
fxor %f6, %f2, %f6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
subcc %o3, 0x10, %o3
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
bne,pt %xcc, 1b
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
add %o2, 0x10, %o2
stx %o0, [%o4 + 0x00]
stx %o5, [%o4 + 0x08]
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
retl
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_cbc_decrypt_192)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
.align 32
ENTRY(aes_sparc64_cbc_decrypt_256)
/* %o0=&key[key_len], %o1=input, %o2=output, %o3=len, %o4=iv */
ldx [%o0 - 0x10], %g1
ldx [%o0 - 0x08], %g2
ldx [%o4 + 0x00], %o0
ldx [%o4 + 0x08], %o5
1: ldx [%o1 + 0x00], %g3
ldx [%o1 + 0x08], %g7
add %o1, 0x10, %o1
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %g3
xor %g2, %g7, %g7
MOVXTOD_G3_F4
MOVXTOD_G7_F6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
DECRYPT_256(8, 4, 6, 0, 2)
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
MOVXTOD_O0_F0
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
MOVXTOD_O5_F2
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
xor %g1, %g3, %o0
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
xor %g2, %g7, %o5
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
fxor %f4, %f0, %f4
fxor %f6, %f2, %f6
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
subcc %o3, 0x10, %o3
bne,pt %xcc, 1b
add %o2, 0x10, %o2
stx %o0, [%o4 + 0x00]
stx %o5, [%o4 + 0x08]
sparc64: Add AES driver making use of the new aes opcodes. Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
2012-08-21 10:58:13 +00:00
retl
nop
sparc64: Move AES driver over to a methods based implementation. Instead of testing and branching off of the key size on every encrypt/decrypt call, use method ops assigned at key set time. Reverse the order of float registers used for decryption to make future changes easier. Align all assembler routines on a 32-byte boundary. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 19:50:16 +00:00
ENDPROC(aes_sparc64_cbc_decrypt_256)
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
.align 32
ENTRY(aes_sparc64_ctr_crypt_128)
/* %o0=key, %o1=input, %o2=output, %o3=len, %o4=IV */
ldx [%o4 + 0x00], %g3
ldx [%o4 + 0x08], %g7
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
subcc %o3, 0x10, %o3
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
ldx [%o0 + 0x00], %g1
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
be 10f
ldx [%o0 + 0x08], %g2
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
1: xor %g1, %g3, %o5
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
MOVXTOD_O5_F0
xor %g2, %g7, %o5
MOVXTOD_O5_F2
add %g7, 1, %g7
add %g3, 1, %o5
movrz %g7, %o5, %g3
xor %g1, %g3, %o5
MOVXTOD_O5_F4
xor %g2, %g7, %o5
MOVXTOD_O5_F6
add %g7, 1, %g7
add %g3, 1, %o5
movrz %g7, %o5, %g3
ENCRYPT_128_2(8, 0, 2, 4, 6, 56, 58, 60, 62)
ldd [%o1 + 0x00], %f56
ldd [%o1 + 0x08], %f58
ldd [%o1 + 0x10], %f60
ldd [%o1 + 0x18], %f62
fxor %f56, %f0, %f56
fxor %f58, %f2, %f58
fxor %f60, %f4, %f60
fxor %f62, %f6, %f62
std %f56, [%o2 + 0x00]
std %f58, [%o2 + 0x08]
std %f60, [%o2 + 0x10]
std %f62, [%o2 + 0x18]
subcc %o3, 0x20, %o3
add %o1, 0x20, %o1
brgz %o3, 1b
add %o2, 0x20, %o2
brlz,pt %o3, 11f
nop
10: xor %g1, %g3, %o5
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
MOVXTOD_O5_F0
xor %g2, %g7, %o5
MOVXTOD_O5_F2
add %g7, 1, %g7
add %g3, 1, %o5
movrz %g7, %o5, %g3
ENCRYPT_128(8, 0, 2, 4, 6)
ldd [%o1 + 0x00], %f4
ldd [%o1 + 0x08], %f6
fxor %f4, %f0, %f4
fxor %f6, %f2, %f6
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
11: stx %g3, [%o4 + 0x00]
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
retl
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
stx %g7, [%o4 + 0x08]
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
ENDPROC(aes_sparc64_ctr_crypt_128)
.align 32
ENTRY(aes_sparc64_ctr_crypt_192)
/* %o0=key, %o1=input, %o2=output, %o3=len, %o4=IV */
ldx [%o4 + 0x00], %g3
ldx [%o4 + 0x08], %g7
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
subcc %o3, 0x10, %o3
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
ldx [%o0 + 0x00], %g1
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
be 10f
ldx [%o0 + 0x08], %g2
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
1: xor %g1, %g3, %o5
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
MOVXTOD_O5_F0
xor %g2, %g7, %o5
MOVXTOD_O5_F2
add %g7, 1, %g7
add %g3, 1, %o5
movrz %g7, %o5, %g3
xor %g1, %g3, %o5
MOVXTOD_O5_F4
xor %g2, %g7, %o5
MOVXTOD_O5_F6
add %g7, 1, %g7
add %g3, 1, %o5
movrz %g7, %o5, %g3
ENCRYPT_192_2(8, 0, 2, 4, 6, 56, 58, 60, 62)
ldd [%o1 + 0x00], %f56
ldd [%o1 + 0x08], %f58
ldd [%o1 + 0x10], %f60
ldd [%o1 + 0x18], %f62
fxor %f56, %f0, %f56
fxor %f58, %f2, %f58
fxor %f60, %f4, %f60
fxor %f62, %f6, %f62
std %f56, [%o2 + 0x00]
std %f58, [%o2 + 0x08]
std %f60, [%o2 + 0x10]
std %f62, [%o2 + 0x18]
subcc %o3, 0x20, %o3
add %o1, 0x20, %o1
brgz %o3, 1b
add %o2, 0x20, %o2
brlz,pt %o3, 11f
nop
10: xor %g1, %g3, %o5
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
MOVXTOD_O5_F0
xor %g2, %g7, %o5
MOVXTOD_O5_F2
add %g7, 1, %g7
add %g3, 1, %o5
movrz %g7, %o5, %g3
ENCRYPT_192(8, 0, 2, 4, 6)
ldd [%o1 + 0x00], %f4
ldd [%o1 + 0x08], %f6
fxor %f4, %f0, %f4
fxor %f6, %f2, %f6
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
11: stx %g3, [%o4 + 0x00]
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
retl
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
stx %g7, [%o4 + 0x08]
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
ENDPROC(aes_sparc64_ctr_crypt_192)
.align 32
ENTRY(aes_sparc64_ctr_crypt_256)
/* %o0=key, %o1=input, %o2=output, %o3=len, %o4=IV */
ldx [%o4 + 0x00], %g3
ldx [%o4 + 0x08], %g7
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
subcc %o3, 0x10, %o3
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
ldx [%o0 + 0x00], %g1
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
be 10f
ldx [%o0 + 0x08], %g2
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
1: xor %g1, %g3, %o5
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
MOVXTOD_O5_F0
xor %g2, %g7, %o5
MOVXTOD_O5_F2
add %g7, 1, %g7
add %g3, 1, %o5
movrz %g7, %o5, %g3
xor %g1, %g3, %o5
MOVXTOD_O5_F4
xor %g2, %g7, %o5
MOVXTOD_O5_F6
add %g7, 1, %g7
add %g3, 1, %o5
movrz %g7, %o5, %g3
ENCRYPT_256_2(8, 0, 2, 4, 6)
ldd [%o1 + 0x00], %f56
ldd [%o1 + 0x08], %f58
ldd [%o1 + 0x10], %f60
ldd [%o1 + 0x18], %f62
fxor %f56, %f0, %f56
fxor %f58, %f2, %f58
fxor %f60, %f4, %f60
fxor %f62, %f6, %f62
std %f56, [%o2 + 0x00]
std %f58, [%o2 + 0x08]
std %f60, [%o2 + 0x10]
std %f62, [%o2 + 0x18]
subcc %o3, 0x20, %o3
add %o1, 0x20, %o1
brgz %o3, 1b
add %o2, 0x20, %o2
brlz,pt %o3, 11f
nop
sparc64: Fix unrolled AES 256-bit key loops. The basic scheme of the block mode assembler is that we start by enabling the FPU, loading the key into the floating point registers, then iterate calling the encrypt/decrypt routine for each block. For the 256-bit key cases, we run short on registers in the unrolled loops. So the {ENCRYPT,DECRYPT}_256_2() macros reload the key registers that get clobbered. The unrolled macros, {ENCRYPT,DECRYPT}_256(), are not mindful of this. So if we have a mix of multi-block and single-block calls, the single-block unrolled 256-bit encrypt/decrypt can run with some of the key registers clobbered. Handle this by always explicitly loading those registers before using the non-unrolled 256-bit macro. This was discovered thanks to all of the new test cases added by Jussi Kivilinna. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 23:19:11 +00:00
10: ldd [%o0 + 0xd0], %f56
sparc64: Fix bugs in unrolled 256-bit loops. Some dm-crypt testing revealed several bugs in the 256-bit unrolled loops. The DECRYPT_256_2() macro had two errors: 1) Missing reload of KEY registers %f60 and %f62 2) Missing "\" in penultimate line of definition. In aes_sparc64_ecb_decrypt_256, we were storing the second half of the encryption result from the wrong source registers. In aes_sparc64_ctr_crypt_256 we have to be careful when we fall out of the 32-byte-at-a-time loop and handle a trailing 16-byte chunk. In that case we've clobbered the final key holding registers and have to restore them before executing the ENCRYPT_256() macro. Inside of the 32-byte-at-a-time loop things are OK, because we do this key register restoring during the first few rounds of the ENCRYPT_256_2() macro. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-02 06:05:43 +00:00
ldd [%o0 + 0xd8], %f58
ldd [%o0 + 0xe0], %f60
ldd [%o0 + 0xe8], %f62
sparc64: Fix unrolled AES 256-bit key loops. The basic scheme of the block mode assembler is that we start by enabling the FPU, loading the key into the floating point registers, then iterate calling the encrypt/decrypt routine for each block. For the 256-bit key cases, we run short on registers in the unrolled loops. So the {ENCRYPT,DECRYPT}_256_2() macros reload the key registers that get clobbered. The unrolled macros, {ENCRYPT,DECRYPT}_256(), are not mindful of this. So if we have a mix of multi-block and single-block calls, the single-block unrolled 256-bit encrypt/decrypt can run with some of the key registers clobbered. Handle this by always explicitly loading those registers before using the non-unrolled 256-bit macro. This was discovered thanks to all of the new test cases added by Jussi Kivilinna. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-12-19 23:19:11 +00:00
xor %g1, %g3, %o5
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
MOVXTOD_O5_F0
xor %g2, %g7, %o5
MOVXTOD_O5_F2
add %g7, 1, %g7
add %g3, 1, %o5
movrz %g7, %o5, %g3
ENCRYPT_256(8, 0, 2, 4, 6)
ldd [%o1 + 0x00], %f4
ldd [%o1 + 0x08], %f6
fxor %f4, %f0, %f4
fxor %f6, %f2, %f6
std %f4, [%o2 + 0x00]
std %f6, [%o2 + 0x08]
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
11: stx %g3, [%o4 + 0x00]
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
retl
sparc64: Unroll CTR crypt loops in AES driver. Before: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 244 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 360 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 814 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 378 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 939 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6395 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 249 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 414 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1073 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7110 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 225 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 810 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 5021 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 240 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 376 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 938 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6380 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 251 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 411 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1070 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 7114 cycles (8192 bytes) After: testing speed of ctr(aes) encryption test 0 (128 bit key, 16 byte blocks): 1 operation in 211 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 246 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 344 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 799 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4975 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 236 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 365 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 6055 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 209 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 255 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 404 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6669 cycles (8192 bytes) testing speed of ctr(aes) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 210 cycles (16 bytes) test 1 (128 bit key, 64 byte blocks): 1 operation in 233 cycles (64 bytes) test 2 (128 bit key, 256 byte blocks): 1 operation in 340 cycles (256 bytes) test 3 (128 bit key, 1024 byte blocks): 1 operation in 818 cycles (1024 bytes) test 4 (128 bit key, 8192 byte blocks): 1 operation in 4956 cycles (8192 bytes) test 5 (192 bit key, 16 byte blocks): 1 operation in 206 cycles (16 bytes) test 6 (192 bit key, 64 byte blocks): 1 operation in 239 cycles (64 bytes) test 7 (192 bit key, 256 byte blocks): 1 operation in 361 cycles (256 bytes) test 8 (192 bit key, 1024 byte blocks): 1 operation in 888 cycles (1024 bytes) test 9 (192 bit key, 8192 byte blocks): 1 operation in 5996 cycles (8192 bytes) test 10 (256 bit key, 16 byte blocks): 1 operation in 214 cycles (16 bytes) test 11 (256 bit key, 64 byte blocks): 1 operation in 248 cycles (64 bytes) test 12 (256 bit key, 256 byte blocks): 1 operation in 395 cycles (256 bytes) test 13 (256 bit key, 1024 byte blocks): 1 operation in 1010 cycles (1024 bytes) test 14 (256 bit key, 8192 byte blocks): 1 operation in 6664 cycles (8192 bytes) Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-30 15:40:44 +00:00
stx %g7, [%o4 + 0x08]
sparc64: Add ctr mode support to AES driver. Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-29 21:49:23 +00:00
ENDPROC(aes_sparc64_ctr_crypt_256)
Reference in New Issue Copy Permalink