GDB Linux Kernel Awareness

Introduction

This page describes the work in the ST landing team to create a GDB Linux kernel threading layer called linux-kthread which supports the Linux kernel thread runtime. It allows anything that the Linux kernel has created a struct task_struct for to be represented as a GDB thread object. This allows a user using GDB to debug the Linux kernel to see all the sleeping threads in the system rather than just physical CPU's (as is the case currently upstream) and then use the GDB contextual commands such as thread <num>, frame <num> to easily switch between all the threads in the system, inspect data structures and get backtraces etc.

This work is based on patches by STMicroelectronics which were used with their STMC2 JTAG debugger.

Code Repositories

Branch linux-kthread-v1 contains the first submission of the re-factored kernel threading layer which was submitted to gdb-patches mailing list December 2016 [1].

This followed a previous minimal C version submitted in March 2016 which was still closely aligned to the ST code structure [2].

Linux Kernel Awareness QEMU Test environment

The easiest way to test linux-kthread and GDB python extensions without hardware is to use QEMU visualization to emulate ARM hardware. You can then boot a mainline Linux ARM kernel, and debug it with a ARM GDB which contains the linux-kthread threading layer.

The following steps assume that the reader is already experienced in cross compiling, and building kernels. An ubuntu/debian build host is assumed.

Prerequisites

You will need the following packages installed on your system:

sudo apt-get update
sudo apt-get install -y build-essential git-core gcc-arm-linux-gnueabi qemu-system-arm
sudo apt-get install -y texinfo libncurses5-dev python-dev bison flex
sudo apt-get install -y libfdt-dev device-tree-compiler libpixman-1-dev libglib2.0-dev zlib1g-dev

Building GDB, QEMU, and Linux

Currently only an ARM target is supported in the architecture layer for linux-kthread. To quickly get started using GDB with linux-kthread to debug a ARM Linux kernel with no hardware it is easiest to use the build system developed by the ST landing team. This allows you to easily build GDB, QEMU and Linux for different architectures and targets, and could form the basis of some automated regression testing in the future.

1. Clone the build system

git clone ssh://git@git.linaro.org/people/peter.griffin/lkd/lkd-build.git

2. Change to qemu-arm target directory (to build Linux, QEMU and GDB for ARM).

cd lkd-build/targets/qemu-arm

3. Build GDB linux-kthread-v1 branch

make BINUTILS_GIT=ssh://git@git.linaro.org/people/peter.griffin/lkd/binutils-gdb.git BINUTILS_BRANCH=linux-kthread-v1 binutils-gdb

4. Build Linux kernel (CROSS_ COMPILE= prefix can be passed if using another ARM toolchain). To enable GDB to have knowledge of C pre processor information, it is recommended to update 'linux/Makefile' and update the CONFIG_DEBUG_INFO section to enable -g3 (this requires significantly more disk space for debug symbols).

make linux dtbs

5. Build QEMU for ARM

make qemu

Running Linux in QEMU and debugging with GDB

Once the build process has completed, you will need to have two terminals open. I prefer to have these side-by-side for visibility

Terminal 1: Running Kernel in QEMU

One terminal will be required to boot a kernel, and observe it's console output. These scripts disable graphics console capability of QEMU, simplifying the process to a text based view.

Bot ARM Linux SMP kernel (make qemu_arma9 boots a single core ARM machine).

make qemu_arma9_smp

Example Terminal 1 output

make qemu_arma9_smp 
make -C /home/griffinp/Software/gdb/lkd/lkd/sources/linux O=/home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- INSTALL_MOD_PATH=/opt/debian/wheezy-armel-rootfs -j8 vexpress-v2p-ca15-tc1.dtb
make[1]: Entering directory '/home/griffinp/Software/gdb/lkd/lkd/sources/linux'
make[2]: Entering directory '/home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux'
make[2]: Leaving directory '/home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux'
make[1]: Leaving directory '/home/griffinp/Software/gdb/lkd/lkd/sources/linux'
echo 32768 > qemu_gdbserver_port
qemu-system-arm \
        -kernel /home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux/arch/arm/boot/zImage \
        -dtb /home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux/arch/arm/boot/dts//vexpress-v2p-ca9.dtb \
        -M vexpress-a9 \
        -smp 2 \
        -m 1024 \
        -append 'root=/dev/nfs nfsroot=10.0.2.2:/opt/debian/wheezy-armel-rootfs,tcp,v3 rw ip=dhcp mem=1024M raid=noautodetect rootwait console=ttyAMA0,38400n8 devtmpfs.mount=0' \
        -nographic \
         \
        -gdb tcp::32768 \
        -redir tcp:2222::22
qemu-system-arm: -redir tcp:2222::22: The -redir option is deprecated. Please use '-netdev user,hostfwd=...' instead.
audio: Could not init `oss' audio driver
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.0-rc1-00001-g0167f47-dirty (griffinp@griffinp-ThinkPad-X1-Carbon-2nd) (gcc version 4.9.3 20141031 (prerelease) (Linaro GCC 2014.11) ) #9 SMP Mon Dec 19 16:28:59 GMT 2016
[    0.000000] CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[    0.000000] OF: fdt:Machine model: V2P-CA9
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 64 MiB at 0x9c000000
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] percpu: Embedded 14 pages/cpu @ef7ac000 s27072 r8192 d22080 u57344
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 260608
[    0.000000] Kernel command line: root=/dev/nfs nfsroot=10.0.2.2:/opt/debian/wheezy-armel-rootfs,tcp,v3 rw ip=dhcp mem=1024M raid=noautodetect rootwait console=ttyAMA0,38400n8 devtmpfs.mount=0
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
<snip>

Terminal 2: Connecting to the QEMU kernel with GDB

In the other terminal, you will establish the connection to the running kernel:

make qemu-gdb

Example Terminal 2 output

make qemu-gdb
# Link the Kernel VMLinux locally for python lx-symbols
ln -f -s /home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux/vmlinux
\
/home/griffinp/Software/gdb/lkd/lkd/host/usr/bin//arm-linux-gdb \
        /home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux/vmlinux \
        -iex 'add-auto-load-safe-path /home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux' \
        -ex 'target remote localhost:32768' \
#               -iex 'lx-symbols'
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=arm-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...

warning: /home/griffinp/Software/gdb/lkd/gdb-helpers/load-helpers.py: No such file or directory
Reading symbols from /home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux/vmlinux...done.
Remote debugging using localhost:32768
[New init]
[New [kthreadd]]
[New [ksoftirqd/0]]
[New [kworker/0:0]]
[New [kworker/0:0H]]
<snip>

Using GDB linux-kthread thread layer

When you have a running kernel, and GDB session, if GDB detects the Linux kernel runtime the linux-kthread layer will automatically activate, enumerating all the kernel threads in the system and you will be provided with a GDB prompt, see example output below:

make qemu-gdb
# Link the Kernel VMLinux locally for python lx-symbols
ln -f -s /home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux/vmlinux
\
/home/griffinp/Software/gdb/lkd/lkd/host/usr/bin//arm-linux-gdb \
        /home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux/vmlinux \
        -iex 'add-auto-load-safe-path /home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux' \
        -ex 'target remote localhost:32768' \
#               -iex 'lx-symbols'
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=arm-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...

warning: /home/griffinp/Software/gdb/lkd/gdb-helpers/load-helpers.py: No such file or directory
Reading symbols from /home/griffinp/Software/gdb/lkd/lkd/objects/arm/qemu-arm/linux/vmlinux...done.
Remote debugging using localhost:32768
[New init]
[New [kthreadd]]
[New [ksoftirqd/0]]
[New [kworker/0:0]]
[New [kworker/0:0H]]
[New [kworker/u8:0]]
[New [rcu_sched]]
<snip>
[Switching to udevd]
generic_handle_irq (irq=<optimized out>) at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/irq/irqdesc.c:597
597             return 0;
(gdb)

At this point, you can inspect the thread list to see the new tasks integrated into the GDB thread list:

(gdb) info threads
  Id   Target Id         Frame 
  1    [swapper/0] (pid: 0 tgid: 0) __raw_spin_lock (lock=<optimized out>)
    at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/include/linux/spinlock_api_smp.h:145
  2    [swapper/1] (pid: 0 tgid: 0) context_switch (cookie=..., next=<optimized out>, prev=<optimized out>, 
    rq=<optimized out>) at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/sched/core.c:2902
  3    init (pid: 1 tgid: 1) context_switch (cookie=..., next=<optimized out>, prev=<optimized out>, 
    rq=<optimized out>) at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/sched/core.c:2902
  4    [kthreadd] (pid: 2 tgid: 2) context_switch (cookie=..., next=<optimized out>, prev=<optimized out>, 
    rq=<optimized out>) at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/sched/core.c:2902
  5    [ksoftirqd/0] (pid: 3 tgid: 3) context_switch (cookie=..., next=<optimized out>, prev=<optimized out>, 
    rq=<optimized out>) at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/sched/core.c:2902

And you can select a thread, and backtrace it in the expected way:

(gdb) thread 79
[Switching to thread 79 ([kworker/0:1H])]
#0  context_switch (cookie=..., next=<optimized out>, prev=<optimized out>, rq=<optimized out>)
    at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/sched/core.c:2902
2902            return finish_task_switch(prev);

(gdb) bt
#0  context_switch (cookie=..., next=<optimized out>, prev=<optimized out>, rq=<optimized out>)
    at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/sched/core.c:2902
#1  __schedule (preempt=<unavailable>)
    at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/sched/core.c:3402
#2  0xc0bb9e8c in schedule () at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/sched/core.c:3457
#3  0xc03588a4 in worker_thread (__worker=0xeee39d00)
    at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/workqueue.c:2251
#4  0xc035d9ac in kthread (_create=0xeed98b00)
    at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/kthread.c:209
#5  0xc0307ef8 in ret_from_fork ()
    at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/arch/arm/kernel/entry-common.S:118

The build system also enables the GDB python extensions in the Linux kernel configuration so these can be used in conjunction with the linux-kthread layer. On v4.10-rc2 kernel the following GDB python commands are available.

[Switching to udevd]
__raw_spin_lock (lock=<optimized out>)
    at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/include/linux/spinlock_api_smp.h:145
145             LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
(gdb) lx-
lx-cmdline     lx-dmesg       lx-iomem       lx-list-check  lx-mounts      lx-ps          lx-version
lx-cpus        lx-fdtdump     lx-ioports     lx-lsmod       lx-pageoffset  lx-symbols

If the Linux kernel is halted in userspace the linux-kthread layer will deactivate and revert to showing the physical CPU from the layer beneath (current upstream GDB default behaviour).

^Clinux_kthread_wait() target stopped in user space. Disabling linux-kthread

Thread 1 received signal SIGINT, Interrupt.
[Switching to Thread 1]
0x00024c44 in ?? ()
(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 1 (CPU#0 [running]) 0x00024c44 in ?? ()
  2    Thread 2 (CPU#1 [running]) __do_softirq ()
    at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/kernel/softirq.c:270
(gdb) 

If the target is later subsequently halted when executing in the kernel, the linux-kthread layer can be reactivated with a new GDB command set linuxkthread loaded. It is easy to determine you have halted the target within the kernel as GDB is able to determine the symbol where the physical CPU has stopped. See example trace below

^C
Thread 1 received signal SIGINT, Interrupt.
cpu_v7_do_idle () at /home/griffinp/Software/gdb/lkd/lkd/sources/linux/arch/arm/mm/proc-v7.S:75
75              ret     lr
(gdb) set linuxkthread loaded
[New init]
[New [kthreadd]]
[New [ksoftirqd/0]]
[New [kworker/0:0]]
[New [kworker/0:0H]]
[New [kworker/u8:0]]
[New [rcu_sched]]
[New [rcu_bh]]
[New [migration/0]]
[New [lru-add-drain]]
[New [watchdog/0]]
[New [cpuhp/0]]
[New [cpuhp/1]]
[New [watchdog/1]]
[New [migration/1]]
[New [ksoftirqd/1]]
[New [kworker/1:0]]
[New [kworker/1:0H]]
[New [cpuhp/2]]
<snip>
(gdb)

Other resources

The team members have spoken about this at ELC-E in the past. See videos below.

[1] https://cygwin.com/ml/gdb-patches/2016-12/msg00383.html. [2] https://sourceware.org/ml/gdb-patches/2016-03/msg00020.html

LandingTeams/ST/GDB (last modified 2017-02-21 08:04:13)