2 Commits

Author	SHA1	Message	Date
Christophe Leroy	b38460bc46	kunit: Fix checksum tests on big endian CPUs ... On powerpc64le checksum kunit tests work: [ 2.011457][ T1] KTAP version 1 [ 2.011662][ T1] # Subtest: checksum [ 2.011848][ T1] 1..3 [ 2.034710][ T1] ok 1 test_csum_fixed_random_inputs [ 2.079325][ T1] ok 2 test_csum_all_carry_inputs [ 2.127102][ T1] ok 3 test_csum_no_carry_inputs [ 2.127202][ T1] # checksum: pass:3 fail:0 skip:0 total:3 [ 2.127533][ T1] # Totals: pass:3 fail:0 skip:0 total:3 [ 2.127956][ T1] ok 1 checksum But on powerpc64 and powerpc32 they fail: [ 1.859890][ T1] KTAP version 1 [ 1.860041][ T1] # Subtest: checksum [ 1.860201][ T1] 1..3 [ 1.861927][ T58] # test_csum_fixed_random_inputs: ASSERTION FAILED at lib/checksum_kunit.c:243 [ 1.861927][ T58] Expected result == expec, but [ 1.861927][ T58] result == 54991 (0xd6cf) [ 1.861927][ T58] expec == 33316 (0x8224) [ 1.863742][ T1] not ok 1 test_csum_fixed_random_inputs [ 1.864520][ T60] # test_csum_all_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:267 [ 1.864520][ T60] Expected result == expec, but [ 1.864520][ T60] result == 255 (0xff) [ 1.864520][ T60] expec == 65280 (0xff00) [ 1.868820][ T1] not ok 2 test_csum_all_carry_inputs [ 1.869977][ T62] # test_csum_no_carry_inputs: ASSERTION FAILED at lib/checksum_kunit.c:306 [ 1.869977][ T62] Expected result == expec, but [ 1.869977][ T62] result == 64515 (0xfc03) [ 1.869977][ T62] expec == 0 (0x0) [ 1.872060][ T1] not ok 3 test_csum_no_carry_inputs [ 1.872102][ T1] # checksum: pass:0 fail:3 skip:0 total:3 [ 1.872458][ T1] # Totals: pass:0 fail:3 skip:0 total:3 [ 1.872791][ T1] not ok 3 checksum This is because all expected values were calculated for X86 which is little endian. On big endian systems all precalculated 16 bits halves must be byte swapped. And this is confirmed by a huge amount of sparse errors when building with C=2 So fix all sparse errors and it will naturally work on all endianness. Fixes: 688eb8191b ("x86/csum: Improve performance of `csum_partial`") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: David S. Miller <davem@davemloft.net>	2023-08-25 10:14:34 +01:00
Noah Goldstein	688eb8191b	x86/csum: Improve performance of csum_partial ... 1) Add special case for len == 40 as that is the hottest value. The nets a ~8-9% latency improvement and a ~30% throughput improvement in the len == 40 case. 2) Use multiple accumulators in the 64-byte loop. This dramatically improves ILP and results in up to a 40% latency/throughput improvement (better for more iterations). Results from benchmarking on Icelake. Times measured with rdtsc() len lat_new lat_old r tput_new tput_old r 8 3.58 3.47 1.032 3.58 3.51 1.021 16 4.14 4.02 1.028 3.96 3.78 1.046 24 4.99 5.03 0.992 4.23 4.03 1.050 32 5.09 5.08 1.001 4.68 4.47 1.048 40 5.57 6.08 0.916 3.05 4.43 0.690 48 6.65 6.63 1.003 4.97 4.69 1.059 56 7.74 7.72 1.003 5.22 4.95 1.055 64 6.65 7.22 0.921 6.38 6.42 0.994 96 9.43 9.96 0.946 7.46 7.54 0.990 128 9.39 12.15 0.773 8.90 8.79 1.012 200 12.65 18.08 0.699 11.63 11.60 1.002 272 15.82 23.37 0.677 14.43 14.35 1.005 440 24.12 36.43 0.662 21.57 22.69 0.951 952 46.20 74.01 0.624 42.98 53.12 0.809 1024 47.12 78.24 0.602 46.36 58.83 0.788 1552 72.01 117.30 0.614 71.92 96.78 0.743 2048 93.07 153.25 0.607 93.28 137.20 0.680 2600 114.73 194.30 0.590 114.28 179.32 0.637 3608 156.34 268.41 0.582 154.97 254.02 0.610 4096 175.01 304.03 0.576 175.89 292.08 0.602 There is no such thing as a free lunch, however, and the special case for len == 40 does add overhead to the len != 40 cases. This seems to amount to be ~5% throughput and slightly less in terms of latency. Testing: Part of this change is a new kunit test. The tests check all alignment X length pairs in [0, 64) X [0, 512). There are three cases. 1) Precomputed random inputs/seed. The expected results where generated use the generic implementation (which is assumed to be non-buggy). 2) An input of all 1s. The goal of this test is to catch any case a carry is missing. 3) An input that never carries. The goal of this test si to catch any case of incorrectly carrying. More exhaustive tests that test all alignment X length pairs in [0, 8192) X [0, 8192] on random data are also available here: https://github.com/goldsteinn/csum-reproduction The reposity also has the code for reproducing the above benchmark numbers. Signed-off-by: Noah Goldstein <goldstein.w.n@gmail.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/all/20230511011002.935690-1-goldstein.w.n%40gmail.com	2023-05-25 10:55:18 -07:00