Crash utility exits with error code 1 when analyzing kernel crash

Support and Help
,
1

24.04 LTS,
Linux 6.14.0-29-generic #29~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Aug 14 16:52:50 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Problem Description:
crash utility is crashing (error code 1) when attempting to analyze kernel crash dumps.

Setup kdump & generated kernel panic using “echo 1 > /proc/sys/kernel/sysrq” but, crash cannot access it:

# crash /usr/lib/debug/boot/vmlinux-6.14.0-29-generic     dump.202509161821 

crash 8.0.4
Copyright (C) 2002-2022  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 10.2                             
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

# echo $?
1

running as root user and file is readable fine:

$ :/var/crash/202509161821# ls -l
total 299144
-rw------- 1 root whoopsie    119627 Sep 16 18:21 dmesg.202509161821
-rw-r--r-- 1 root whoopsie 306200163 Sep 16 18:21 dump.202509161821

What I’ve Tried:
running with ‘strace’ noticed after it opens the debug file it complains not finding this:

14:06:20.564658 newfstatat(AT_FDCWD, "/usr/lib/debug/boot/vmlinux-6.14.0-29-generic.dwp", 0x7ffd0d815350, 0) = -1 ENOENT (No such file or directory) <0.000021>
14:06:20.565346 newfstatat(AT_FDCWD, "./usr/lib/debug/boot/vmlinux-6.14.0-29-generic.dwp", 0x7ffd0d815300, 0) = -1 ENOENT (No such file or directory) <0.000019>
14:06:20.565437 newfstatat(AT_FDCWD, "/usr/lib/debug/usr/lib/debug/boot/vmlinux-6.14.0-29-generic.dwp", 0x7ffd0d815300, 0) = -1 ENOENT (No such file or directory) <0.000013>
14:06:20.565482 newfstatat(AT_FDCWD, "/usr/lib/debug/vmlinux-6.14.0-29-generic.dwp", 0x7ffd0d815310, 0) = -1 ENOENT (No such file or directory) <0.000010>

shortly after also complains not finding .gdb file:

14:06:20.654983 openat(AT_FDCWD, "/usr/lib/debug/boot/vmlinux-6.14.0-29-generic-gdb.gdb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000014>
14:06:20.655042 openat(AT_FDCWD, "/usr/lib/debug/usr/lib/debug/boot/vmlinux-6.14.0-29-generic-gdb.gdb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000010>
14:06:20.655086 openat(AT_FDCWD, "/usr/share/gdb/auto-load/usr/lib/debug/boot/vmlinux-6.14.0-29-generic-gdb.gdb", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000012>

I think both are normally not there, we only have this one:

# ls -l /usr/lib/debug/boot/vmlinux-6.14.0-29-generic*
-rw-r--r-- 1 root root 450705920 Aug 14 18:02 /usr/lib/debug/boot/vmlinux-6.14.0-29-generic

tail of strace is:

14:06:20.661240 rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER|SA_NODEFER, sa_restorer=0x7b0841845330}, NULL, 8) = 0 <0.000008>
14:06:20.661281 rt_sigaction(SIGINT, {sa_handler=0x5ec383cbceb0, sa_mask=[], sa_flags=SA_RESTORER|SA_NODEFER, sa_restorer=0x7b0841845330}, NULL, 8) = 0 <0.000008>
14:06:20.661322 rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER|SA_NODEFER, sa_restorer=0x7b0841845330}, NULL, 8) = 0 <0.000008>
14:06:20.661360 write(1, "\n", 1
)       = 1 <0.000119>
14:06:20.661579 lseek(3, 10312, SEEK_SET) = 10312 <0.000010>
14:06:20.661617 read(3, "OSRELEASE=6.14.0-29-generic\nBUIL"..., 3276) = 3276 <0.000011>
14:06:20.661748 unlink("/var/tmp/ramdump_elf_XXXXXX") = -1 ENOENT (No such file or directory) <0.002921>
14:06:20.664817 exit_group(1)           = ?
14:06:20.690105 +++ exited with 1 +++

for those interested, full crash strace https://filebin.net/custom-bin/crash.strace.1

Fixing this will help anaylze the real kernel crash… this one is simulated, and still ‘crash’ program fails for some reason.

2

Hello and welcome!

I didn’t use crash myself yet. Just skimmed over the manpage. From section notes: “If crash does not work, look for a newer version: kernel evolution frequently makes crash updates necessary.” Is crash up to date? Is your system up to date?

Maybe you’re lacking kernel debug information?

Did you try using gdb?

3

well, that’s why all output is provided, crash version used: 8.0.4

crash/noble,now 8.0.4-1ubuntu2 amd64 [installed]

possibly there is newer, but system is updated/upgraded subject to the mirrors used :slight_smile:

as for gdb, tried but not sure exactly how on ubuntu, this reads the kernel symbol file:

# gdb /usr/lib/debug/boot/vmlinux-6.14.0-29-generic

however, ‘target kvm’ is not available, and target core fails:

(gdb) target kvm /var/crash/202509171542/dump.202509171542
Undefined target command: "kvm /var/crash/202509171542/dump.202509171542".  Try "help target".
(gdb) target /var/crash/202509171542/dump.202509171542 
Undefined target command: "/var/crash/202509171542/dump.202509171542 ".  Try "help target".
(gdb) target core /var/crash/202509171542/dump.202509171542 
"/var/crash/202509171542/dump.202509171542" is not a core dump: file format not recognized
4

Please file a bug by running the following on that system:

ubuntu-bug crash
5

Thanks, done that! thanks for suggestion… I guess it works for others :slight_smile:

1 Like
6

You’re likely just the first to run this against a 6.14 dump :neutral_face: I see there are few 6.14-specific patches that we don’t have…

7

FTR: Bug #2125145 “Crash utility exits with error code 1 when analyzi...” : Bugs : crash package : Ubuntu

8

This topic was automatically closed after 30 days. New replies are no longer allowed.