emdbg.debug.crashdebug

CrashDebug Post-Mortem Analysis

CrashDebug is a post-mortem debugging tool for Cortex-M microcontrollers. You can create core dumps from GDB using the px4_coredump command (see emdbg.debug.gdb).

python3 -m emdbg.debug.gdb -py --elf path/to/firmware.elf crashdebug --dump coredump.txt

Analyzing Hardfault Logs

PX4 generated log files in case a hardfault is detected, which can be passed directly to this backend to be converted on the fly by emdbg.analyze.hardfault.

$ python3.8 -m emdbg.debug.gdb --elf px4_fmu-v5x_default.elf -py crashdebug --dump hardfault.log
memset (s=0x0, c=0, n=80) at string/lib_memset.c:126
(gdb) backtrace
#0  memset (s=0x0, c=0, n=80) at string/lib_memset.c:126
#1  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Note
The hardfault log only contains a copy of the stack and registers. The converter fills unknown memory with the 0xfeedc0de value, which may then display wrong or incomplete results in some of the GDB plugins. Keep this in mind when analyzing a hardfault.

In this example, the backtrace is broken, however, by inspecting the registers you can see a valid location in the link register (LR) and manually inspect it:

(gdb) info registers
r0             0x0                 0
r1             0x0                 0
r2             0x50                80
r3             0x0                 0
r4             0x70                112
r5             0x0                 0
r6             0x0                 0
r7             0x18                24
r8             0x2001a660          536979040
r9             0x20044434          537150516
r10            0x0                 0
r11            0x0                 0
r12            0x70                112
sp             0x2001a600          0x2001a600
lr             0x817f17f           135786879
pc             0x8019f30           0x8019f30 <memset+120>
xpsr           0x21000200          553648640
fpscr          0xfeedc0de          -17973026
msp            0x2001a600          536978944
psp            0x2001a600          536978944
(gdb) info symbol 0x817f17f
uORB::DeviceNode::write(file*, char const*, unsigned int) + 435 in section .text

To understand the actual hardfault reason, use arm scb /h to show the descriptions of the bit fields:

(gdb) arm scb /h
CFSR             = 00000400          // Configurable Fault Status Register
    MMFSR          ......00 - 00     // MemManage Fault Status Register
    BFSR           ....04.. - 04     // BusFault Status Register
    IMPRECISERR    .....4.. - 1      // Indicates if imprecise data access error has occurred.
    UFSR           0000.... - 0000   // UsageFault Status Register
HFSR             = 40000000          // HardFault Status Register
    FORCED         4....... - 1      // Indicates that a fault with configurable priority has been escalated to a HardFault exception.
DFSR             = 00000000          // Debug Fault Status Register
MMFAR            = 00000000          // MemManage Fault Address Register
BFAR             = 00000000          // BusFault Address Register
AFSR             = 00000000          // Auxiliary Fault Status Register
ABFSR - M7       = 00000308          // Auxiliary Bus Fault Status - Cortex M7
        AXIMTYPE   .....3.. - DECERR // Indicates the type of fault on the AXIM interface
    AXIM           .......8 - 1      // Asynchronous fault on AXIM interface

In this example, an imprecise, asynchronous bus fault has occurred on the AXIM interface. Together with the backtrace and register information we can interpret this error as a write to address zero inside the memset function, which was passed a NULL pointer from inside uORB::DeviceNode::write.

Installation

You need to have the platform-specific CrashDebug binary available in your path.

Ubuntu

sudo curl -L https://github.com/adamgreen/CrashDebug/raw/master/bins/lin64/CrashDebug \
          -o /usr/bin/CrashDebug
sudo chmod +x /usr/bin/CrashDebug

macOS

curl -L https://github.com/adamgreen/CrashDebug/raw/master/bins/osx64/CrashDebug \
     -o $HOMEBREW_PREFIX/bin/CrashDebug
# Clear the quarantine flag
sudo xattr -r -d com.apple.quarantine $HOMEBREW_PREFIX/bin/CrashDebug

Alternatively you can specify the binary path in your environment:

export PX4_CRASHDEBUG_BINARY=path/to/CrashDebug

Note You need to have the Rosetta emulation layer installed on ARM64 macOS:
softwareupdate --install-rosetta --agree-to-license

View Source

 1# Copyright (c) 2020-2022, Niklas Hauser
 2# Copyright (c) 2023, Auterion AG
 3# SPDX-License-Identifier: BSD-3-Clause
 4
 5"""
 6.. include:: crashdebug.md
 7"""
 8
 9from __future__ import annotations
10import os, platform, tempfile
11from pathlib import Path
12from .backend import ProbeBackend
13
14
15class CrashProbeBackend(ProbeBackend):
16    """
17    CrashDebug specific debug backend implementation. Note that this
18    implementation only pretends to connect to the microcontroller so that GDB
19    can access the device's memory.
20
21    .. warning::  You cannot execute code using this backend.
22        You can only look around the device's memories at least as far as they
23        have been saved by the coredump command.
24    """
25    def __init__(self, coredump: Path):
26        super().__init__()
27        coredump = Path(coredump)
28        if coredump.name.startswith("coredump") and coredump.suffix.lower() == ".txt":
29            self.coredump = coredump
30            self._tmpfile = None
31        else:
32            contents = coredump.read_text()
33            if "arm_hardfault" in contents:
34                from ..analyze import convert_hardfault
35                tmpfile = tempfile.NamedTemporaryFile(mode="w+t", delete=False)
36                tmpfile.writelines(convert_hardfault(contents))
37                tmpfile.flush()
38                self.coredump = tmpfile.name
39                self._tmpfile = tmpfile
40            else:
41                raise NotImplementedError("Unknown coredump format! "
42                        "Only coredump_{datetime}.txt or hardfault*.log is supported!")
43
44        self.binary = "CrashDebug"
45        if "Windows" in platform.platform():
46            self.binary = "CrashDebug.exe"
47        self.binary = os.environ.get("PX4_CRASHDEBUG_BINARY", self.binary)
48        self.name = "crashdebug"
49
50    def init(self, elf: Path):
51        return ["set target-charset ASCII",
52                "target remote | {} --elf {} --dump {}"
53                .format(self.binary, elf, self.coredump)]
54
55    def stop(self):
56        if self._tmpfile is not None:
57            self._tmpfile.close()
58            os.unlink(self._tmpfile.name)
59
60
61def _add_subparser(subparser):
62    parser = subparser.add_parser("crashdebug", help="Use CrashDebug as Backend.")
63    parser.add_argument(
64            "--dump",
65            dest="coredump",
66            default="coredump.txt",
67            type=Path,
68            help="Path to coredump file.")
69    parser.set_defaults(backend=lambda args: CrashProbeBackend(args.coredump))
70    return parser

class CrashProbeBackend(emdbg.debug.backend.ProbeBackend): View Source

16class CrashProbeBackend(ProbeBackend):
17    """
18    CrashDebug specific debug backend implementation. Note that this
19    implementation only pretends to connect to the microcontroller so that GDB
20    can access the device's memory.
21
22    .. warning::  You cannot execute code using this backend.
23        You can only look around the device's memories at least as far as they
24        have been saved by the coredump command.
25    """
26    def __init__(self, coredump: Path):
27        super().__init__()
28        coredump = Path(coredump)
29        if coredump.name.startswith("coredump") and coredump.suffix.lower() == ".txt":
30            self.coredump = coredump
31            self._tmpfile = None
32        else:
33            contents = coredump.read_text()
34            if "arm_hardfault" in contents:
35                from ..analyze import convert_hardfault
36                tmpfile = tempfile.NamedTemporaryFile(mode="w+t", delete=False)
37                tmpfile.writelines(convert_hardfault(contents))
38                tmpfile.flush()
39                self.coredump = tmpfile.name
40                self._tmpfile = tmpfile
41            else:
42                raise NotImplementedError("Unknown coredump format! "
43                        "Only coredump_{datetime}.txt or hardfault*.log is supported!")
44
45        self.binary = "CrashDebug"
46        if "Windows" in platform.platform():
47            self.binary = "CrashDebug.exe"
48        self.binary = os.environ.get("PX4_CRASHDEBUG_BINARY", self.binary)
49        self.name = "crashdebug"
50
51    def init(self, elf: Path):
52        return ["set target-charset ASCII",
53                "target remote | {} --elf {} --dump {}"
54                .format(self.binary, elf, self.coredump)]
55
56    def stop(self):
57        if self._tmpfile is not None:
58            self._tmpfile.close()
59            os.unlink(self._tmpfile.name)

CrashDebug specific debug backend implementation. Note that this implementation only pretends to connect to the microcontroller so that GDB can access the device's memory.

You cannot execute code using this backend.

You can only look around the device's memories at least as far as they have been saved by the coredump command.

CrashProbeBackend(coredump: pathlib.Path) View Source

26    def __init__(self, coredump: Path):
27        super().__init__()
28        coredump = Path(coredump)
29        if coredump.name.startswith("coredump") and coredump.suffix.lower() == ".txt":
30            self.coredump = coredump
31            self._tmpfile = None
32        else:
33            contents = coredump.read_text()
34            if "arm_hardfault" in contents:
35                from ..analyze import convert_hardfault
36                tmpfile = tempfile.NamedTemporaryFile(mode="w+t", delete=False)
37                tmpfile.writelines(convert_hardfault(contents))
38                tmpfile.flush()
39                self.coredump = tmpfile.name
40                self._tmpfile = tmpfile
41            else:
42                raise NotImplementedError("Unknown coredump format! "
43                        "Only coredump_{datetime}.txt or hardfault*.log is supported!")
44
45        self.binary = "CrashDebug"
46        if "Windows" in platform.platform():
47            self.binary = "CrashDebug.exe"
48        self.binary = os.environ.get("PX4_CRASHDEBUG_BINARY", self.binary)
49        self.name = "crashdebug"

Parameters

remote: Extended remote location.

binary

name

def init(self, elf: pathlib.Path): View Source

51    def init(self, elf: Path):
52        return ["set target-charset ASCII",
53                "target remote | {} --elf {} --dump {}"
54                .format(self.binary, elf, self.coredump)]

Returns a list of GDB commands that connect GDB to the debug probe. The default implementation returns target extended-remote {self.remote}.

def stop(self): View Source

56    def stop(self):
57        if self._tmpfile is not None:
58            self._tmpfile.close()
59            os.unlink(self._tmpfile.name)

Halts the debug probe process.

Inherited Members

emdbg.debug.backend.ProbeBackend: remote; start; scope