最近两天做实验中，使用sysrq-c触发系统崩溃后，生成转储文件vmcore。但是使用crash分析的时候却提示vmcore与vmLinux不匹配

# crash /var/crash/127.0.0.1-2014.07.13-18\:23\:21/vmcore /usr/lib/debug/lib/modules/3.9.5-301.fc19.i686/vmlinux

crash 6.1.4-1.fc19
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
GNU gdb (GDB) 7.3.1
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
crash: invalid kernel virtual address: c7c0  type: "possible"
WARNING: cannot read cpu_possible_map
crash: invalid kernel virtual address: 364c0cf  type: "present"
WARNING: cannot read cpu_present_map
crash: page excluded: kernel virtual address: d0005405  type: "online"
WARNING: cannot read cpu_online_map
crash: /usr/lib/debug/lib/modules/3.9.5-301.fc19.i686/vmlinux and /var/crash/127.0.0.1-2014.07.13-18:23:21/vmcore do not match!
Usage:
  crash [OPTION]... NAMELIST MEMORY-IMAGE  (dumpfile form)
  crash [OPTION]... [NAMELIST]             (live system form)
Enter "crash -h" for details.

在网上也没有找到什么解决办法，只是说vmlinux与vmcore版本不匹配。

后来仔细观察，发现系统版本与安装的软件版本还是有一点小的区别

# uname -a

Linux localhost.localdomain 3.9.5-301.fc19.i686.PAE #1 SMP Tue Jun 11 19:46:44 UTC 2013 i686 i686 i386 GNU/Linux

重点在PAE三个字，物理地址扩展。

原来还是安装的系统架构不同。粗心了。

在软件源中寻找，找到这样的一个软件包

kernel-PAE-debuginfo-3.9.5-301.fc19.i686.rpm

于是，安装之

之后果然发现多出一个目录，而原来只有一个，即

/usr/lib/debug/lib/modules/3.9.5-301.fc19.i686

现在多出一个

/usr/lib/debug/lib/modules/3.9.5-301.fc19.i686.PAE

再次crash分析，这次vmlinux需要在后来这个目录中找

# crash /var/crash/127.0.0.1-2014.07.13-18\:23\:21/vmcore  /usr/lib/debug/lib/modules/3.9.5-301.fc19.i686.PAE/vmlinux

之后果然进去了

crash 6.1.4-1.fc19
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
GNU gdb (GDB) 7.3.1
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
      KERNEL: /usr/lib/debug/lib/modules/3.9.5-301.fc19.i686.PAE/vmlinux
    DUMPFILE: /var/crash/127.0.0.1-2014.07.13-18:23:21/vmcore  [PARTIAL DUMP]
        CPUS: 2
        DATE: Mon Jul 14 22:22:37 2014
      UPTIME: 07:04:31
LOAD AVERAGE: 0.00, 0.04, 0.08
       TASKS: 92
    NODENAME: localhost.localdomain
     RELEASE: 3.9.5-301.fc19.i686.PAE
     VERSION: #1 SMP Tue Jun 11 19:46:44 UTC 2013
     MACHINE: i686  (2600 Mhz)
      MEMORY: 1 GB
       PANIC: "Oops: 0002 [#1] SMP " (check log for details)
         PID: 1384
     COMMAND: "bash"
        TASK: f47fb300  [THREAD_INFO: f4640000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)
crash> 

Linux 内核发送崩溃时，kdump会生成一个内核转储文件 vmcore 。 可以通过 分析 vmcore 分析 出内核崩溃的原因。 crash 是一个被广泛应用的内核奔溃转储文件 分析 工具。使用 crash 调试内核转储文件，需要安装 crash 工具 和 内核调试工具kernel-debuginfo。 安装需要的软件 1、查看系统内核 [root@qd01-stop-f re e015 ~]# uname -r 3.10.0-1160.15.2.el7.x86_64 2、安装kdump, crash yum install Linux 内核（以下简称内核）是一个 不 与特定进程相关的功能集合，内核的代码很难轻易的在调试器中执行 和 跟踪。开发者认为，内核如果发生了错误，就 不 应该继续运行。因此内核发生错误时，它的行为通常被设定为系统崩溃，机器重启。基于动态存储器的电气特性，机器重启后，上次错误发生时的现场会遭到破坏，这使得查找内核的错误变得异常困难。 内核社区 和 一些商业公司为此开发了很多种调试技术 和 工具，希望... crash 7.1.3 Co pyright (C) 2002-2014 Re d Hat, Inc. Co pyright (C) 2004, 2005, 2006, 2010 IBM Cor po ra tion Co pyright (C) 1999-200 ra mdump是 分析 kernel crash 的资料，利用工具想gdb debug core dump一样直接debug ra mdump。 该工具在gdb上开发，使它可以支持kernel，因此使用 方法 和 gdb一样，只 不 过增加了支持kernel的一些命令。 crash 官方网站：http://people. re dhat. co m/anderson/如何编译 crash ，然后用它... wget http://ddebs.ubuntu. co m/pool/main/l/ linux -lts- ra ring/ linux -image-3.8.0-29-generic-dbgsym_3.8.0-29.42~p re cise1_amd64.ddeb 2. crash /usr/ li b/debug/boot/ vmlinux -3.8.0-29-generic dump.201409