mirror of
https://github.com/torvalds/linux.git
synced 2024-11-21 19:41:42 +00:00
57e6700145
The patch titled ("powerpc: make fadump resilient with memory add/remove
events") has made significant changes to the implementation of fadump,
particularly on elfcorehdr creation and fadump crash info header
structure. Therefore, updating the fadump implementation documentation
to reflect those changes.
Following updates are done to firmware assisted dump documentation:
1. The elfcorehdr is no longer stored after fadump HDR in the reserved
dump area. Instead, the second kernel dynamically allocates memory
for the elfcorehdr within the address range from 0 to the boot memory
size. Therefore, update figures 1 and 2 of Memory Reservation during
the first and second kernels to reflect this change.
2. A version field has been added to the fadump header to manage the
future changes to fadump crash info header structure without changing
the fadump header magic number in the future. Therefore, remove the
corresponding TODO from the document.
Signed-off-by: Sourabh Jain <sourabhjain@linux.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://msgid.link/20240422195932.1583833-4-sourabhjain@linux.ibm.com
375 lines
16 KiB
ReStructuredText
375 lines
16 KiB
ReStructuredText
======================
|
|
Firmware-Assisted Dump
|
|
======================
|
|
|
|
July 2011
|
|
|
|
The goal of firmware-assisted dump is to enable the dump of
|
|
a crashed system, and to do so from a fully-reset system, and
|
|
to minimize the total elapsed time until the system is back
|
|
in production use.
|
|
|
|
- Firmware-Assisted Dump (FADump) infrastructure is intended to replace
|
|
the existing phyp assisted dump.
|
|
- Fadump uses the same firmware interfaces and memory reservation model
|
|
as phyp assisted dump.
|
|
- Unlike phyp dump, FADump exports the memory dump through /proc/vmcore
|
|
in the ELF format in the same way as kdump. This helps us reuse the
|
|
kdump infrastructure for dump capture and filtering.
|
|
- Unlike phyp dump, userspace tool does not need to refer any sysfs
|
|
interface while reading /proc/vmcore.
|
|
- Unlike phyp dump, FADump allows user to release all the memory reserved
|
|
for dump, with a single operation of echo 1 > /sys/kernel/fadump_release_mem.
|
|
- Once enabled through kernel boot parameter, FADump can be
|
|
started/stopped through /sys/kernel/fadump_registered interface (see
|
|
sysfs files section below) and can be easily integrated with kdump
|
|
service start/stop init scripts.
|
|
|
|
Comparing with kdump or other strategies, firmware-assisted
|
|
dump offers several strong, practical advantages:
|
|
|
|
- Unlike kdump, the system has been reset, and loaded
|
|
with a fresh copy of the kernel. In particular,
|
|
PCI and I/O devices have been reinitialized and are
|
|
in a clean, consistent state.
|
|
- Once the dump is copied out, the memory that held the dump
|
|
is immediately available to the running kernel. And therefore,
|
|
unlike kdump, FADump doesn't need a 2nd reboot to get back
|
|
the system to the production configuration.
|
|
|
|
The above can only be accomplished by coordination with,
|
|
and assistance from the Power firmware. The procedure is
|
|
as follows:
|
|
|
|
- The first kernel registers the sections of memory with the
|
|
Power firmware for dump preservation during OS initialization.
|
|
These registered sections of memory are reserved by the first
|
|
kernel during early boot.
|
|
|
|
- When system crashes, the Power firmware will copy the registered
|
|
low memory regions (boot memory) from source to destination area.
|
|
It will also save hardware PTE's.
|
|
|
|
NOTE:
|
|
The term 'boot memory' means size of the low memory chunk
|
|
that is required for a kernel to boot successfully when
|
|
booted with restricted memory. By default, the boot memory
|
|
size will be the larger of 5% of system RAM or 256MB.
|
|
Alternatively, user can also specify boot memory size
|
|
through boot parameter 'crashkernel=' which will override
|
|
the default calculated size. Use this option if default
|
|
boot memory size is not sufficient for second kernel to
|
|
boot successfully. For syntax of crashkernel= parameter,
|
|
refer to Documentation/admin-guide/kdump/kdump.rst. If any
|
|
offset is provided in crashkernel= parameter, it will be
|
|
ignored as FADump uses a predefined offset to reserve memory
|
|
for boot memory dump preservation in case of a crash.
|
|
|
|
- After the low memory (boot memory) area has been saved, the
|
|
firmware will reset PCI and other hardware state. It will
|
|
*not* clear the RAM. It will then launch the bootloader, as
|
|
normal.
|
|
|
|
- The freshly booted kernel will notice that there is a new node
|
|
(rtas/ibm,kernel-dump on pSeries or ibm,opal/dump/mpipl-boot
|
|
on OPAL platform) in the device tree, indicating that
|
|
there is crash data available from a previous boot. During
|
|
the early boot OS will reserve rest of the memory above
|
|
boot memory size effectively booting with restricted memory
|
|
size. This will make sure that this kernel (also, referred
|
|
to as second kernel or capture kernel) will not touch any
|
|
of the dump memory area.
|
|
|
|
- User-space tools will read /proc/vmcore to obtain the contents
|
|
of memory, which holds the previous crashed kernel dump in ELF
|
|
format. The userspace tools may copy this info to disk, or
|
|
network, nas, san, iscsi, etc. as desired.
|
|
|
|
- Once the userspace tool is done saving dump, it will echo
|
|
'1' to /sys/kernel/fadump_release_mem to release the reserved
|
|
memory back to general use, except the memory required for
|
|
next firmware-assisted dump registration.
|
|
|
|
e.g.::
|
|
|
|
# echo 1 > /sys/kernel/fadump_release_mem
|
|
|
|
Please note that the firmware-assisted dump feature
|
|
is only available on POWER6 and above systems on pSeries
|
|
(PowerVM) platform and POWER9 and above systems with OP940
|
|
or later firmware versions on PowerNV (OPAL) platform.
|
|
Note that, OPAL firmware exports ibm,opal/dump node when
|
|
FADump is supported on PowerNV platform.
|
|
|
|
On OPAL based machines, system first boots into an intermittent
|
|
kernel (referred to as petitboot kernel) before booting into the
|
|
capture kernel. This kernel would have minimal kernel and/or
|
|
userspace support to process crash data. Such kernel needs to
|
|
preserve previously crash'ed kernel's memory for the subsequent
|
|
capture kernel boot to process this crash data. Kernel config
|
|
option CONFIG_PRESERVE_FA_DUMP has to be enabled on such kernel
|
|
to ensure that crash data is preserved to process later.
|
|
|
|
-- On OPAL based machines (PowerNV), if the kernel is build with
|
|
CONFIG_OPAL_CORE=y, OPAL memory at the time of crash is also
|
|
exported as /sys/firmware/opal/mpipl/core file. This procfs file is
|
|
helpful in debugging OPAL crashes with GDB. The kernel memory
|
|
used for exporting this procfs file can be released by echo'ing
|
|
'1' to /sys/firmware/opal/mpipl/release_core node.
|
|
|
|
e.g.
|
|
# echo 1 > /sys/firmware/opal/mpipl/release_core
|
|
|
|
Implementation details:
|
|
-----------------------
|
|
|
|
During boot, a check is made to see if firmware supports
|
|
this feature on that particular machine. If it does, then
|
|
we check to see if an active dump is waiting for us. If yes
|
|
then everything but boot memory size of RAM is reserved during
|
|
early boot (See Fig. 2). This area is released once we finish
|
|
collecting the dump from user land scripts (e.g. kdump scripts)
|
|
that are run. If there is dump data, then the
|
|
/sys/kernel/fadump_release_mem file is created, and the reserved
|
|
memory is held.
|
|
|
|
If there is no waiting dump data, then only the memory required to
|
|
hold CPU state, HPTE region, boot memory dump, and FADump header is
|
|
usually reserved at an offset greater than boot memory size (see Fig. 1).
|
|
This area is *not* released: this region will be kept permanently
|
|
reserved, so that it can act as a receptacle for a copy of the boot
|
|
memory content in addition to CPU state and HPTE region, in the case
|
|
a crash does occur.
|
|
|
|
Since this reserved memory area is used only after the system crash,
|
|
there is no point in blocking this significant chunk of memory from
|
|
production kernel. Hence, the implementation uses the Linux kernel's
|
|
Contiguous Memory Allocator (CMA) for memory reservation if CMA is
|
|
configured for kernel. With CMA reservation this memory will be
|
|
available for applications to use it, while kernel is prevented from
|
|
using it. With this FADump will still be able to capture all of the
|
|
kernel memory and most of the user space memory except the user pages
|
|
that were present in CMA region::
|
|
|
|
o Memory Reservation during first kernel
|
|
|
|
Low memory Top of memory
|
|
0 boot memory size |<------ Reserved dump area ----->| |
|
|
| | | Permanent Reservation | |
|
|
V V | | V
|
|
+-----------+-----/ /---+---+----+-----------+-------+----+-----+
|
|
| | |///|////| DUMP | HDR |////| |
|
|
+-----------+-----/ /---+---+----+-----------+-------+----+-----+
|
|
| ^ ^ ^ ^ ^
|
|
| | | | | |
|
|
\ CPU HPTE / | |
|
|
-------------------------------- | |
|
|
Boot memory content gets transferred | |
|
|
to reserved area by firmware at the | |
|
|
time of crash. | |
|
|
FADump Header |
|
|
(meta area) |
|
|
|
|
|
|
|
|
Metadata: This area holds a metadata structure whose
|
|
address is registered with f/w and retrieved in the
|
|
second kernel after crash, on platforms that support
|
|
tags (OPAL). Having such structure with info needed
|
|
to process the crashdump eases dump capture process.
|
|
|
|
Fig. 1
|
|
|
|
|
|
o Memory Reservation during second kernel after crash
|
|
|
|
Low memory Top of memory
|
|
0 boot memory size |
|
|
| |<------------ Crash preserved area ------------>|
|
|
V V |<--- Reserved dump area --->| |
|
|
+----+---+--+-----/ /---+---+----+-------+-----+-----+-------+
|
|
| |ELF| | |///|////| DUMP | HDR |/////| |
|
|
+----+---+--+-----/ /---+---+----+-------+-----+-----+-------+
|
|
| | | | | |
|
|
----- ------------------------------ ---------------
|
|
\ | |
|
|
\ | |
|
|
\ | |
|
|
\ | ----------------------------
|
|
\ | /
|
|
\ | /
|
|
\ | /
|
|
/proc/vmcore
|
|
|
|
|
|
+---+
|
|
|///| -> Regions (CPU, HPTE & Metadata) marked like this in the above
|
|
+---+ figures are not always present. For example, OPAL platform
|
|
does not have CPU & HPTE regions while Metadata region is
|
|
not supported on pSeries currently.
|
|
|
|
+---+
|
|
|ELF| -> elfcorehdr, it is created in second kernel after crash.
|
|
+---+
|
|
|
|
Note: Memory from 0 to the boot memory size is used by second kernel
|
|
|
|
Fig. 2
|
|
|
|
|
|
Currently the dump will be copied from /proc/vmcore to a new file upon
|
|
user intervention. The dump data available through /proc/vmcore will be
|
|
in ELF format. Hence the existing kdump infrastructure (kdump scripts)
|
|
to save the dump works fine with minor modifications. KDump scripts on
|
|
major Distro releases have already been modified to work seamlessly (no
|
|
user intervention in saving the dump) when FADump is used, instead of
|
|
KDump, as dump mechanism.
|
|
|
|
The tools to examine the dump will be same as the ones
|
|
used for kdump.
|
|
|
|
How to enable firmware-assisted dump (FADump):
|
|
----------------------------------------------
|
|
|
|
1. Set config option CONFIG_FA_DUMP=y and build kernel.
|
|
2. Boot into linux kernel with 'fadump=on' kernel cmdline option.
|
|
By default, FADump reserved memory will be initialized as CMA area.
|
|
Alternatively, user can boot linux kernel with 'fadump=nocma' to
|
|
prevent FADump to use CMA.
|
|
3. Optionally, user can also set 'crashkernel=' kernel cmdline
|
|
to specify size of the memory to reserve for boot memory dump
|
|
preservation.
|
|
|
|
NOTE:
|
|
1. 'fadump_reserve_mem=' parameter has been deprecated. Instead
|
|
use 'crashkernel=' to specify size of the memory to reserve
|
|
for boot memory dump preservation.
|
|
2. If firmware-assisted dump fails to reserve memory then it
|
|
will fallback to existing kdump mechanism if 'crashkernel='
|
|
option is set at kernel cmdline.
|
|
3. if user wants to capture all of user space memory and ok with
|
|
reserved memory not available to production system, then
|
|
'fadump=nocma' kernel parameter can be used to fallback to
|
|
old behaviour.
|
|
|
|
Sysfs/debugfs files:
|
|
--------------------
|
|
|
|
Firmware-assisted dump feature uses sysfs file system to hold
|
|
the control files and debugfs file to display memory reserved region.
|
|
|
|
Here is the list of files under kernel sysfs:
|
|
|
|
/sys/kernel/fadump_enabled
|
|
This is used to display the FADump status.
|
|
|
|
- 0 = FADump is disabled
|
|
- 1 = FADump is enabled
|
|
|
|
This interface can be used by kdump init scripts to identify if
|
|
FADump is enabled in the kernel and act accordingly.
|
|
|
|
/sys/kernel/fadump_registered
|
|
This is used to display the FADump registration status as well
|
|
as to control (start/stop) the FADump registration.
|
|
|
|
- 0 = FADump is not registered.
|
|
- 1 = FADump is registered and ready to handle system crash.
|
|
|
|
To register FADump echo 1 > /sys/kernel/fadump_registered and
|
|
echo 0 > /sys/kernel/fadump_registered for un-register and stop the
|
|
FADump. Once the FADump is un-registered, the system crash will not
|
|
be handled and vmcore will not be captured. This interface can be
|
|
easily integrated with kdump service start/stop.
|
|
|
|
/sys/kernel/fadump/mem_reserved
|
|
|
|
This is used to display the memory reserved by FADump for saving the
|
|
crash dump.
|
|
|
|
/sys/kernel/fadump_release_mem
|
|
This file is available only when FADump is active during
|
|
second kernel. This is used to release the reserved memory
|
|
region that are held for saving crash dump. To release the
|
|
reserved memory echo 1 to it::
|
|
|
|
echo 1 > /sys/kernel/fadump_release_mem
|
|
|
|
After echo 1, the content of the /sys/kernel/debug/powerpc/fadump_region
|
|
file will change to reflect the new memory reservations.
|
|
|
|
The existing userspace tools (kdump infrastructure) can be easily
|
|
enhanced to use this interface to release the memory reserved for
|
|
dump and continue without 2nd reboot.
|
|
|
|
Note: /sys/kernel/fadump_release_opalcore sysfs has moved to
|
|
/sys/firmware/opal/mpipl/release_core
|
|
|
|
/sys/firmware/opal/mpipl/release_core
|
|
|
|
This file is available only on OPAL based machines when FADump is
|
|
active during capture kernel. This is used to release the memory
|
|
used by the kernel to export /sys/firmware/opal/mpipl/core file. To
|
|
release this memory, echo '1' to it:
|
|
|
|
echo 1 > /sys/firmware/opal/mpipl/release_core
|
|
|
|
Note: The following FADump sysfs files are deprecated.
|
|
|
|
+----------------------------------+--------------------------------+
|
|
| Deprecated | Alternative |
|
|
+----------------------------------+--------------------------------+
|
|
| /sys/kernel/fadump_enabled | /sys/kernel/fadump/enabled |
|
|
+----------------------------------+--------------------------------+
|
|
| /sys/kernel/fadump_registered | /sys/kernel/fadump/registered |
|
|
+----------------------------------+--------------------------------+
|
|
| /sys/kernel/fadump_release_mem | /sys/kernel/fadump/release_mem |
|
|
+----------------------------------+--------------------------------+
|
|
|
|
Here is the list of files under powerpc debugfs:
|
|
(Assuming debugfs is mounted on /sys/kernel/debug directory.)
|
|
|
|
/sys/kernel/debug/powerpc/fadump_region
|
|
This file shows the reserved memory regions if FADump is
|
|
enabled otherwise this file is empty. The output format
|
|
is::
|
|
|
|
<region>: [<start>-<end>] <reserved-size> bytes, Dumped: <dump-size>
|
|
|
|
and for kernel DUMP region is:
|
|
|
|
DUMP: Src: <src-addr>, Dest: <dest-addr>, Size: <size>, Dumped: # bytes
|
|
|
|
e.g.
|
|
Contents when FADump is registered during first kernel::
|
|
|
|
# cat /sys/kernel/debug/powerpc/fadump_region
|
|
CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x0
|
|
HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x0
|
|
DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x0
|
|
|
|
Contents when FADump is active during second kernel::
|
|
|
|
# cat /sys/kernel/debug/powerpc/fadump_region
|
|
CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x40020
|
|
HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x1000
|
|
DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x10000000
|
|
: [0x00000010000000-0x0000006ffaffff] 0x5ffb0000 bytes, Dumped: 0x5ffb0000
|
|
|
|
|
|
NOTE:
|
|
Please refer to Documentation/filesystems/debugfs.rst on
|
|
how to mount the debugfs filesystem.
|
|
|
|
|
|
TODO:
|
|
-----
|
|
- Need to come up with the better approach to find out more
|
|
accurate boot memory size that is required for a kernel to
|
|
boot successfully when booted with restricted memory.
|
|
|
|
Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
|
|
|
|
This document is based on the original documentation written for phyp
|
|
|
|
assisted dump by Linas Vepstas and Manish Ahuja.
|