linux/scripts/xz_wrap.sh

163 lines
3.4 KiB
Bash
Raw Permalink Normal View History

#!/bin/sh
# SPDX-License-Identifier: 0BSD
#
# This is a wrapper for xz to compress the kernel image using appropriate
# compression options depending on the architecture.
#
# Author: Lasse Collin <lasse.collin@tukaani.org>
xz: adjust arch-specific options for better kernel compression Use LZMA2 options that match the arch-specific alignment of instructions. This change reduces compressed kernel size 0-2 % depending on the arch. On 1-byte-aligned x86 it makes no difference and on 4-byte-aligned archs it helps the most. Use the ARM-Thumb filter for ARM-Thumb2 kernels. This reduces compressed kernel size about 5 %.[1] Previously such kernels were compressed using the ARM filter which didn't do anything useful with ARM-Thumb2 code. Add BCJ filter support for ARM64 and RISC-V. Compared to unfiltered XZ or plain LZMA, the compressed kernel size is reduced about 5 % on ARM64 and 7 % on RISC-V. A new enough version of the xz tool is required: 5.4.0 for ARM64 and 5.6.0 for RISC-V. With an old xz version, a message is printed to standard error and the kernel is compressed without the filter. Update lib/decompress_unxz.c to match the changes to xz_wrap.sh. Update the CONFIG_KERNEL_XZ help text in init/Kconfig: - Add the RISC-V and ARM64 filters. - Clarify that the PowerPC filter is for big endian only. - Omit IA-64. Link: https://lore.kernel.org/lkml/1637379771-39449-1-git-send-email-zhongjubin@huawei.com/ [1] Link: https://lkml.kernel.org/r/20240721133633.47721-15-lasse.collin@tukaani.org Signed-off-by: Lasse Collin <lasse.collin@tukaani.org> Reviewed-by: Sam James <sam@gentoo.org> Cc: Simon Glass <sjg@chromium.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Jubin Zhong <zhongjubin@huawei.com> Cc: Jules Maselbas <jmaselbas@zdiv.net> Cc: Emil Renner Berthing <emil.renner.berthing@canonical.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: Joel Stanley <joel@jms.id.au> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Krzysztof Kozlowski <krzk@kernel.org> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Randy Dunlap <rdunlap@infradead.org> Cc: Rui Li <me@lirui.org> Cc: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-07-21 13:36:29 +00:00
# This has specialized settings for the following archs. However,
# XZ-compressed kernel isn't currently supported on every listed arch.
#
# Arch Align Notes
# arm 2/4 ARM and ARM-Thumb2
# arm64 4
# csky 2
# loongarch 4
# mips 2/4 MicroMIPS is 2-byte aligned
# parisc 4
# powerpc 4 Uses its own wrapper for compressors instead of this.
# riscv 2/4
# s390 2
# sh 2
# sparc 4
# x86 1
# A few archs use 2-byte or 4-byte aligned instructions depending on
# the kernel config. This function is used to check if the relevant
# config option is set to "y".
is_enabled()
{
grep -q "^$1=y$" include/config/auto.conf
}
# XZ_VERSION is needed to disable features that aren't available in
# old XZ Utils versions.
XZ_VERSION=$($XZ --robot --version) || exit
XZ_VERSION=$(printf '%s\n' "$XZ_VERSION" | sed -n 's/^XZ_VERSION=//p')
# Assume that no BCJ filter is available.
BCJ=
xz: adjust arch-specific options for better kernel compression Use LZMA2 options that match the arch-specific alignment of instructions. This change reduces compressed kernel size 0-2 % depending on the arch. On 1-byte-aligned x86 it makes no difference and on 4-byte-aligned archs it helps the most. Use the ARM-Thumb filter for ARM-Thumb2 kernels. This reduces compressed kernel size about 5 %.[1] Previously such kernels were compressed using the ARM filter which didn't do anything useful with ARM-Thumb2 code. Add BCJ filter support for ARM64 and RISC-V. Compared to unfiltered XZ or plain LZMA, the compressed kernel size is reduced about 5 % on ARM64 and 7 % on RISC-V. A new enough version of the xz tool is required: 5.4.0 for ARM64 and 5.6.0 for RISC-V. With an old xz version, a message is printed to standard error and the kernel is compressed without the filter. Update lib/decompress_unxz.c to match the changes to xz_wrap.sh. Update the CONFIG_KERNEL_XZ help text in init/Kconfig: - Add the RISC-V and ARM64 filters. - Clarify that the PowerPC filter is for big endian only. - Omit IA-64. Link: https://lore.kernel.org/lkml/1637379771-39449-1-git-send-email-zhongjubin@huawei.com/ [1] Link: https://lkml.kernel.org/r/20240721133633.47721-15-lasse.collin@tukaani.org Signed-off-by: Lasse Collin <lasse.collin@tukaani.org> Reviewed-by: Sam James <sam@gentoo.org> Cc: Simon Glass <sjg@chromium.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Jubin Zhong <zhongjubin@huawei.com> Cc: Jules Maselbas <jmaselbas@zdiv.net> Cc: Emil Renner Berthing <emil.renner.berthing@canonical.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: Joel Stanley <joel@jms.id.au> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Krzysztof Kozlowski <krzk@kernel.org> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Randy Dunlap <rdunlap@infradead.org> Cc: Rui Li <me@lirui.org> Cc: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-07-21 13:36:29 +00:00
# Set the instruction alignment to 1, 2, or 4 bytes.
#
# Set the BCJ filter if one is available.
# It must match the #ifdef usage in lib/decompress_unxz.c.
case $SRCARCH in
xz: adjust arch-specific options for better kernel compression Use LZMA2 options that match the arch-specific alignment of instructions. This change reduces compressed kernel size 0-2 % depending on the arch. On 1-byte-aligned x86 it makes no difference and on 4-byte-aligned archs it helps the most. Use the ARM-Thumb filter for ARM-Thumb2 kernels. This reduces compressed kernel size about 5 %.[1] Previously such kernels were compressed using the ARM filter which didn't do anything useful with ARM-Thumb2 code. Add BCJ filter support for ARM64 and RISC-V. Compared to unfiltered XZ or plain LZMA, the compressed kernel size is reduced about 5 % on ARM64 and 7 % on RISC-V. A new enough version of the xz tool is required: 5.4.0 for ARM64 and 5.6.0 for RISC-V. With an old xz version, a message is printed to standard error and the kernel is compressed without the filter. Update lib/decompress_unxz.c to match the changes to xz_wrap.sh. Update the CONFIG_KERNEL_XZ help text in init/Kconfig: - Add the RISC-V and ARM64 filters. - Clarify that the PowerPC filter is for big endian only. - Omit IA-64. Link: https://lore.kernel.org/lkml/1637379771-39449-1-git-send-email-zhongjubin@huawei.com/ [1] Link: https://lkml.kernel.org/r/20240721133633.47721-15-lasse.collin@tukaani.org Signed-off-by: Lasse Collin <lasse.collin@tukaani.org> Reviewed-by: Sam James <sam@gentoo.org> Cc: Simon Glass <sjg@chromium.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Jubin Zhong <zhongjubin@huawei.com> Cc: Jules Maselbas <jmaselbas@zdiv.net> Cc: Emil Renner Berthing <emil.renner.berthing@canonical.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: Joel Stanley <joel@jms.id.au> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Krzysztof Kozlowski <krzk@kernel.org> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Randy Dunlap <rdunlap@infradead.org> Cc: Rui Li <me@lirui.org> Cc: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-07-21 13:36:29 +00:00
arm)
if is_enabled CONFIG_THUMB2_KERNEL; then
ALIGN=2
BCJ=--armthumb
else
ALIGN=4
BCJ=--arm
fi
;;
arm64)
ALIGN=4
# ARM64 filter was added in XZ Utils 5.4.0.
if [ "$XZ_VERSION" -ge 50040002 ]; then
BCJ=--arm64
else
echo "$0: Upgrading to xz >= 5.4.0" \
"would enable the ARM64 filter" \
"for better compression" >&2
fi
;;
csky)
ALIGN=2
;;
loongarch)
ALIGN=4
;;
mips)
if is_enabled CONFIG_CPU_MICROMIPS; then
ALIGN=2
else
ALIGN=4
fi
;;
parisc)
ALIGN=4
;;
powerpc)
ALIGN=4
# The filter is only for big endian instruction encoding.
if is_enabled CONFIG_CPU_BIG_ENDIAN; then
BCJ=--powerpc
fi
;;
riscv)
if is_enabled CONFIG_RISCV_ISA_C; then
ALIGN=2
else
ALIGN=4
fi
# RISC-V filter was added in XZ Utils 5.6.0.
if [ "$XZ_VERSION" -ge 50060002 ]; then
BCJ=--riscv
else
echo "$0: Upgrading to xz >= 5.6.0" \
"would enable the RISC-V filter" \
"for better compression" >&2
fi
;;
s390)
ALIGN=2
;;
sh)
ALIGN=2
;;
sparc)
ALIGN=4
BCJ=--sparc
;;
x86)
ALIGN=1
BCJ=--x86
;;
*)
echo "$0: Arch-specific tuning is missing for '$SRCARCH'" >&2
# Guess 2-byte-aligned instructions. Guessing too low
# should hurt less than guessing too high.
ALIGN=2
;;
esac
# Select the LZMA2 options matching the instruction alignment.
case $ALIGN in
1) LZMA2OPTS= ;;
2) LZMA2OPTS=lp=1 ;;
4) LZMA2OPTS=lp=2,lc=2 ;;
*) echo "$0: ALIGN wrong or missing" >&2; exit 1 ;;
esac
xz: use 128 MiB dictionary and force single-threaded mode This only affects kernel image compression, not any other xz usage. Desktop kernels on x86-64 are already around 60 MiB. Using a dictionary larger than 32 MiB should have no downsides nowadays as anyone building the kernel should have plenty of RAM. 128 MiB dictionary needs 1346 MiB of RAM with xz versions 5.0.x - 5.6.x in single-threaded mode. On archs that use xz_wrap.sh, kernel decompression is done in single-call mode so a larger dictionary doesn't affect boot-time memory requirements. xz >= 5.6.0 uses multithreaded mode by default which compresses slightly worse than single-threaded mode. Kernel compression rarely used more than one thread anyway because with 32 MiB dictionary size the default block size was 96 MiB in multithreaded mode. So only a single thread was used anyway unless the kernel was over 96 MiB. Comparison to CONFIG_KERNEL_LZMA: It uses "lzma -9" which mapped to 32 MiB dictionary in LZMA Utils 4.32.7 (the final release in 2008). Nowadays the lzma tool on most systems is from XZ Utils where -9 maps to 64 MiB dictionary. So using a 32 MiB dictionary with CONFIG_KERNEL_XZ may have compressed big kernels slightly worse than the old LZMA option. Comparison to CONFIG_KERNEL_ZSTD: zstd uses 128 MiB dictionary. Link: https://lkml.kernel.org/r/20240721133633.47721-14-lasse.collin@tukaani.org Signed-off-by: Lasse Collin <lasse.collin@tukaani.org> Reviewed-by: Sam James <sam@gentoo.org> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Emil Renner Berthing <emil.renner.berthing@canonical.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: Joel Stanley <joel@jms.id.au> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Jubin Zhong <zhongjubin@huawei.com> Cc: Jules Maselbas <jmaselbas@zdiv.net> Cc: Krzysztof Kozlowski <krzk@kernel.org> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Randy Dunlap <rdunlap@infradead.org> Cc: Rui Li <me@lirui.org> Cc: Simon Glass <sjg@chromium.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-07-21 13:36:28 +00:00
# Use single-threaded mode because it compresses a little better
# (and uses less RAM) than multithreaded mode.
#
# For the best compression, the dictionary size shouldn't be
# smaller than the uncompressed kernel. 128 MiB dictionary
# needs less than 1400 MiB of RAM in single-threaded mode.
#
# On the archs that use this script to compress the kernel,
# decompression in the preboot code is done in single-call mode.
# Thus the dictionary size doesn't affect the memory requirements
# of the preboot decompressor at all.
exec $XZ --check=crc32 --threads=1 $BCJ --lzma2=$LZMA2OPTS,dict=128MiB