APEI tables testing

Resources


Overview


ACPI platform error interface (APEI) convey error information to OS. APEI extends existing hardware error reporting mechanisms and brings them together as components of a coherent hardware error infrastructure. APEI takes advantage of the additional hardware error information available in today’s hardware devices and integrates much more closely with the system firmware. As a result, APEI provides the following benefits:

  • Allows for more extensive error data to be made available in a standard error record format for determining the root cause of hardware errors.
  • Is extensible, so that as hardware vendors add new and better hardware error reporting mechanisms to their devices, APEI allows the platform and the OSPM to gracefully accommodate the new mechanisms.

This provides information to help system designers understand basic issues about hardware errors, the relationship between the firmware and OS, and information about error handling and the APEI architecture components. APEI consists of four separate tables:

  • Error Record Serialization Table (ERST)
  • BOOT Error Record Table (BERT)
  • Hardware Error Source Table (HEST)
  • Error Injection Table (EINJ)

See this for more details about each of listed tables.

APEI requirements


APEI needs:

  • Hardware which are able to:
    • discover its own errors
    • report errors as irqs
  • Firmware support, handle with h/w conveyed error occurence:
    • preparation of error specific informations for OS
    • passing control back to OS

While waiting for ARM/Aarch64 boards to become easily available to developers, emulation can be used:

  • h/w registers - blobs loaded to set aside physical space
  • basic firmware features - SCI type of notification

Required modifications


According to APEI requirements some modifications need to be done regarding usual steps. You will need several things:

  1. ARMv8 - FVP Base Fast Models,

  2. UEFI which provides APEI tables.

  3. APEI blob for a given APEI tables,

  4. GRUB2 which can load given blob,
  5. Kernel with applied patches,

Please refer to the following sections: ACPI kernel via UEFI and GRUB2 to setup environment. Keep in mind modifications below during SD card image creation. Good approach is to setup normal ACPI kerne via UEFI and GRUB2 and then apply below modifications.

UEFI with APEI tables

  • UEFI repo which support APEI tables: git clone git://git.linaro.org/people/tomasz.nowicki/uefi-next-acpi.git

Kernel building

  • Additional patches: git clone ssh://git.linaro.org/people/tomasz.nowicki/acpi.git -b arm64-apei-test

  • Additional kernel config requirements: ACPI, ACPI_APEI, ACPI_APEI_EINJ, ACPI_APEI_GHES, ACPI_HED, ACDEBUG_FS, ACPI_SCI_EMULATE, EDAC_GHES
  • Copy arch/arm64/boot/Image to /acpi/ SD card directory.
  • Do not forget to rebuild fvp-base-gicv2-psci.dtb and put it to /acpi/ SD card directory

ACPI blob

  • Clone tool: git clone git.linaro.org:/arm/acpi/acpi-asl.git
  • Command line depends on APEI table you want to test:

make PADDR=0x81e0000 rtsm_ve-aemv8a.acpi APEI=<bert | erst | einj>
  • Line above produce blob: apei.acpi that can be copied to the same directory as kernel image.

GRUB2

set debug="linux,loader,mm"
set term="vt100"

menuentry 'ACPI FVP Linux' {
        linux /acpi/Image console=ttyAMA0 earlyprintk=pl011,0x1c090000  root=/dev/mmcblk0p2 ext4 rootwait rw mmci.fmax=12000000 
        devicetree /acpi/fvp-base-gicv2-psci.dtb
        apei /acpi/apei.acpi
}

How to test APEI tables

BERT table


Once blob is loaded kernel check list of errors (boot process) pointed from BERT, any errors with status flag set is printed out:

...
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
sdhci-pltfm: SDHCI platform and OF driver helper
TCP: cubic registered
NET: Registered protocol family 17
[Hardware Error]: Error record from previous boot:
[Hardware Error]: APEI generic hardware error status
[Hardware Error]: severity: 1, fatal
[Hardware Error]: section: 0, severity: 0, recoverable
[Hardware Error]: flags: 0x00
[Hardware Error]: section_type: memory error
EXT4-fs (vda2): mounted filesystem with ordered data mode. Opts: (null)
VFS: Mounted root (ext4 filesystem) on device 254:2.
devtmpfs: mounted
Freeing unused kernel memory: 188K (ffffffc000567000 - ffffffc000596000)
INIT: version 2.88 booting
Starting udev
...

ERST table


When there is kernel panic ERST driver dump log to region reserved within the blob. Once kernel back to life it can mount pstore file system and read logs (files) for more details.

...
panic!
[reboot]
# mount -t pstore - /sys/fs/pstore
# ls -al /sys/fs/pstore/
total 0
drwxr-xr-x 2 root root     0 Sep  5 15:19 ./
drwxr-xr-x 4 root root     0 Jan  1  1970 ../
-r--r--r-- 1 root root 29374 Sep  5 14:01 dmesg-erst-0
# cat /sys/fs/pstore/dmesg-erst-0 
Panic#1 Part1
<6>Booting Linux on physical CPU 0x0
<5>Linux version 3.11.0-rc5-01911-g1c30624-dirty (tn@tn-HP3-PC) (gcc version 4.7.3 20130226 (prerelease) (crosstool-NG linaro-1.13.1-4.7-2013.03-20130313 - Linaro GCC 2013.03) ) #178 SMP PREEMPT Fri Sep 6 10:32:38 CEST 2013
<1>CPU: ARMv7 Processor [410fc0f4] revision 4 (ARMv7), cr=10c5387d
<1>CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
<6>Machine: SAMSUNG EXYNOS5 (Flattened Device Tree), model: Insignal Arndale evaluation board based on EXYNOS5250
<2>NR_BANKS too low, ignoring high memory
<1>Memory policy: ECC disabled, Data cache writealloc
<1>CPU EXYNOS5250 (id 0x43520010)
<7>On node 0 totalpages: 518144
<7>  Normal zone: 1520 pages used for memmap
<7>  Normal zone: 0 pages reserved
<7>  Normal zone: 190464 pages, LIFO batch:31
<7>  HighMem zone: 2576 pages used for memmap
<7>  HighMem zone: 327680 pages, LIFO batch:31
<1>ACPI: RSDP 42010008 00024 (v02 LINARO)
<1>ACPI: XSDT 4201002c 000C4 (v01 LINARO ARNDALE  00000014 INTL 20130214)
<1>ACPI: FACP 420100f0 0010C (v05 LINARO ARNDALE  00000000 INTL 20130214)
<1>ACPI: DSDT 420101fc 002A7 (v02 LINARO ARNDALE  00000002 INTL 20130214)
<1>ACPI: MSCT 420104e3 00090 (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: MCHI 42010573 00045 (v01 LINARO ARNDALE  02000715 INTL 20130214)
<1>ACPI: FPDT 420105b8 00064 (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: ERST 4201061c 00230 (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: EINJ 4201084c 00130 (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: CPEP 4201097c 00034 (v01 LINARO ARNDALE  00000000 INTL 20130214)
<1>ACPI: UEFI 420109b0 00036 (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: SRAT 420109e6 00080 (v03 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: SPMI 42010a66 00041 (v04 LINARO ARNDALE  00000000 INTL 20130214)
<1>ACPI: SLIT 42010aa7 001BC (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: OEM0 42010c63 00024 (v01 LINARO ARNDALE  0000000A INTL 20130214)
<1>ACPI: MPST 42010c87 000B6 (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: MCFG 42010d3d 0003C (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: APIC 42010d79 000F6 (v03 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: HEST 42010e6f 00068 (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: GTDT 42010ed7 00050 (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: ECDT 42010f27 00042 (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: BERT 42010f69 00030 (v01 LINARO ARNDALE  00000001 INTL 20130214)
<1>ACPI: SSDT 42010f99 00024 (v02 LINARO ARNDALE  00000001 INTL 20130214)
<6>ACPI: GIC (acpi_id[0x0000] gic_id[0x0000] enabled)
<6>1 CPUs available, 1 CPUs total
<6>ACPI: GIC Distributor (id[0x0000] address[0x00000000] gsi_base[0])
<6>Using ACPI (MADT) for SMP configuration information
<6>PERCPU: Embedded 9 pages/cpu @c1730000 s12928 r8192 d15744 u36864
<7>pcpu-alloc: s12928 r8192 d15744 u36864 alloc=9*4096
<7>pcpu-alloc: [0] 0 [0] 1 
<1>Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 516624
<5>Kernel command line: root=/dev/mmcblk1p3 rw rootwait console=ttySAC2,115200n8 init --no-log debug
<6>PID hash table entries: 4096 (order: 2, 16384 bytes)
<6>Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
<6>Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
<1>Memory: 2047952K/2072576K available (4674K kernel code, 275K rwdata, 1440K rodata, 384K init, 409K bss, 24624K reserved, 1310720K highmem)
<5>Virtual kernel memory layout:
<5>    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
<5>    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
<5>    vmalloc : 0xf0000000 - 0xff000000   ( 240 MB)
<5>    lowmem  : 0xc0000000 - 0xef800000   ( 760 MB)
<5>    pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
<5>    modules : 0xbf000000 - 0xbfe00000   (  14 MB)
<5>      .text : 0xc0008000 - 0xc0600acc   (6115 kB)
<5>      .init : 0xc0601000 - 0xc0661280   ( 385 kB)
<5>      .data : 0xc0662000 - 0xc06a6f48   ( 276 kB)
<5>       .bss : 0xc06a6f48 - 0xc070d49c   ( 410 kB)
<6>SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
<6>Preemptible hierarchical RCU implementation.
<6>NR_IRQS:16 nr_irqs:16 16
<6>Exynos5250: clock setup completed, armclk=1000000000
<6>Exynos: Audss: clock setup completed
<6>Architected local timer running at 24.00MHz (phys).
<3>arch_timer: can't register interrupt 29 (-22)
...

EINJ table


See EINJ for more details about ACPI injection mechanism.

ECC error are used to demonstrate EINJ functionality.

$ cat /sys/kernel/debug/apei/einj/available_error_type
0x00000004    Processor Uncorrectable fatal
0x00000008    Memory Correctable
0x00000200    Platform Correctable

$ echo 0x00000008 > /sys/kernel/debug/apei/einj/error_type
$ cat /sys/kernel/debug/apei/einj/error_type
0x00000008
$ echo 0x00000008 > /sys/kernel/debug/apei/einj/error_inject
{1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
{1}[Hardware Error]: It has been corrected by h/w and requires no further action
{1}[Hardware Error]: event severity: corrected
{1}[Hardware Error]:  Error 0, type: recoverable
{1}[Hardware Error]:   section_type: memory error
{1}[Hardware Error]:   error_status: 0x0000000000000400
{1}[Hardware Error]:   physical_address: 0xffffffffffffffff
{1}[Hardware Error]:   error_type: 2, single-bit ECC
EDAC MC0: 1 CE Single-bit ECC on unknown label (card:1 module:2 bank:3 row:1 col:2 bit_pos:5 page:0xfffffffffffff offset:0xfff grain:0 syndrome:0x0 - status(0x0000000000000400): Storage error in DRAM memory)

SCI emulate According to error reporting flow firmware layer is responsible for SCI line triggering. Systems that do not have necessary firmware support need to emulate SCI behaviour. This require additional step (NOTE: kernel config requirements: ACPI_SCI_EMULATE):

$ echo "HED 128" > /sys/kernel/debug/acpi/sci_notify

Tracepoint

You can also observe ECC error using tracepoints as shown below.

NOTE: make sure that the kernel tracing infrastructure is enabled: CONFIG_FTRACE (-> Kernel hacking)

  • Enable event before you trigger ECC error.

$ echo 1 > /sys/kernel/debug/tracing/events/ras/mc_event/enable
  • Trigger ECC error:

$ echo 0x00000008 > /sys/kernel/debug/apei/einj/error_type
$ echo 0x00000008 > /sys/kernel/debug/apei/einj/error_inject
$ echo "HED 128" > /sys/kernel/debug/acpi/sci_notify
  • Check if kernel hunted error down:

$ cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 1/1   #P:4
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
     kworker/0:1-330   [000] ....  1962.718664: mc_event: 1 Corrected error: Single-bit ECC on unknown label (mc:0 location:-1:-1:-1 address:0xfffffffffff grain:1 syndrome:0x00000000 APEI location: card:1 module:2 bank:3 row:1 col:2 bit_pos:5 status(0x0000000000000400): Storage error in DRAM memory)

PERF tool

perf tool is another way of catching error. Native compilation is the best way to have perf tool on ARM64 OE filesystem and avoid lib mismatch.

  • Run perf targeting ECC error event (NOTE: -a allows to collect event from all available CPUs):

$ perf record -a -e ras:mc_event sleep 60
  • Trigger ECC error in separate terminal:

$ echo 0x00000008 > /sys/kernel/debug/apei/einj/error_type
$ echo 0x00000008 > /sys/kernel/debug/apei/einj/error_inject
$ echo "HED 128" > /sys/kernel/debug/acpi/sci_notify
  • Go back to perf record terminal, wait for command completion and then read perf.data (created by perf record) and display the profile:

$ ./perf report 
# ========
# captured on: Tue Apr  1 08:56:35 2014
# hostname : genericarmv8
# os release : 3.14.0-rc2+
# perf version : 3.12.0-rc1
# arch : aarch64
# nrcpus online : 4
# nrcpus avail : 4
# total memory : 4057104 kB
# cmdline : /home/root/acpi/tools/perf/perf record -e ras:mc_event -a sleep 60 
# event : name = ras:mc_event, type = 2, config = 0x191, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, exc
# HEADER_CPU_TOPOLOGY info available, use -I to display
# pmu mappings: cpu = 4, software = 1, tracepoint = 2, breakpoint = 5
# ========
#
# Samples: 1  of event 'ras:mc_event'
# Event count (approx.): 1
#
# Overhead      Command     Shared Object                  Symbol
# ........  ...........  ................  ......................
#
   100.00%  kworker/0:1  [unknown]         [.] 00000000000

LEG/ServerArchitecture/RAS/APEITests (last modified 2017-08-17 12:13:02)