Linux 應用程式除錯技術/核心分析

基礎

為了獲得良好的呼叫堆疊，重要的是 gdb 載入與生成核心轉儲的程式載入的相同庫。如果我們分析核心的機器與核心轉儲的機器具有不同的庫（或庫位於不同的位置），則將庫複製到分析機器上，以映象轉儲機器。例如

$ tree .
.
|-- juggler-29964.core
|-- lib64
|   |-- ld-linux-x86-64.so.2
|   |-- libc.so.6
|   |-- libm.so.6
|   |-- libpthread.so.0
|   `-- librt.so.1
...

在 gdb 提示符下

(gdb) set solib-absolute-prefix ./
(gdb) set solib-search-path .
(gdb) file ../../../../../threadpool/bin.v2/libs/threadpool/example/juggler/gcc-4.1.2/debug/link-static/threading-multi/juggler
Reading symbols from /home/aurelian_melinte/threadpool/threadpool-0_2_5-src/threadpool/bin.v2/libs/threadpool/example/juggler/gcc-4.1.2/debug/link-static/threading-multi/juggler...done.
(gdb) core-file juggler-29964.core 
Reading symbols from ./lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for ./lib64/librt.so.1
Reading symbols from ./lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for ./lib64/libm.so.6
Reading symbols from ./lib64/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for ./lib64/libpthread.so.0
Reading symbols from ./lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for ./lib64/libc.so.6
Reading symbols from ./lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for ./lib64/ld-linux-x86-64.so.2
Core was generated by `../../../../bin.v2/libs/threadpool/example/juggler/gcc-4.1.2/debug/link-static/'.
Program terminated with signal 6, Aborted.
#0  0x0000003684030265 in raise () from ./lib64/libc.so.6
(gdb) frame 2
#2  0x0000000000404ae1 in dump_core_and_terminate () at juggler.cpp:30

analyze-cores

這是一個指令碼，它將為每個核心檔案生成一個基本報告。當核心向你傾盆而下時，這個指令碼很有用

#!/bin/bash

#
# A script to extract core-file informations
#


if [ $# -ne 1 ]
then
  echo "Usage: `basename $0` <for-binary-image>"
  exit -1
else
  binimg=$1
fi


# Today and yesterdays cores
cores=`find . -name '*.core' -mtime -1`

#cores=`find . -name '*.core'`


for core in $cores 
do 
  gdblogfile="$core-gdb.log"
  rm $gdblogfile
  
  bininfo=`ls -l $binimg`
  coreinfo=`ls -l $core`
  
  gdb -batch \
      -ex "set logging file $gdblogfile" \
      -ex "set logging on" \
      -ex "set pagination off" \
      -ex "printf \"**\n** Process info for $binimg - $core \n** Generated `date`\n\"" \
      -ex "printf \"**\n** $bininfo \n** $coreinfo\n**\n\"" \
      -ex "file $binimg" \
      -ex "core-file $core" \
      -ex "bt" \
      -ex "info proc" \
      -ex "printf \"*\n* Libraries \n*\n\"" \
      -ex "info sharedlib" \
      -ex "printf \"*\n* Memory map \n*\n\"" \
      -ex "info target" \
      -ex "printf \"*\n* Registers \n*\n\"" \
      -ex "info registers" \
      -ex "printf \"*\n* Current instructions \n*\n\"" -ex "x/16i \$pc" \
      -ex "printf \"*\n* Threads (full) \n*\n\"" \
      -ex "info threads" \
      -ex "bt" \
      -ex "thread apply all bt full" \
      -ex "printf \"*\n* Threads (basic) \n*\n\"" \
      -ex "info threads" \
      -ex "thread apply all bt" \
      -ex "printf \"*\n* Done \n*\n\"" \
      -ex "quit" 
done

預定義命令

相同的報告功能可以被預定義到 gdb 中

define procinfo
    printf "**\n** Process Info: \n**\n"
    info proc
    
    printf "*\n* Libraries \n*\n"
    info sharedlib
    
    printf "*\n* Memory Map \n*\n"
    info target
    
    printf "*\n* Registers \n*\n"
    info registers
    
    printf "*\n* Current Instructions \n*\n" 
    x/16i $pc
    
    printf "*\n* Threads (basic) \n*\n"
    info threads
    thread apply all bt
end
document procinfo
Infos about the debugee. 
end

define analyze
    procinfo
    
    printf "*\n* Threads (full) \n*\n"
    info threads
    bt
    thread apply all bt full
end

分析程序

一個指令碼，它將為執行的程序生成一個基本報告和一個核心檔案

#!/bin/bash

#
# A script to generate a core and a status report for a running process. 
#


if [ $# -ne 1 ]
then
  echo "Usage: `basename $0` <PID>"
  exit -1
else
  pid=$1
fi


gdblogfile="analyze-$pid.log"
rm $gdblogfile

corefile="core-$pid.core"
  
gdb -batch \
      -ex "set logging file $gdblogfile" \
      -ex "set logging on" \
      -ex "set pagination off" \
      -ex "printf \"**\n** Process info for PID=$pid \n** Generated `date`\n\"" \
      -ex "printf \"**\n** Core: $corefile \n**\n\"" \
      -ex "attach $pid" \
      -ex "bt" \
      -ex "info proc" \
      -ex "printf \"*\n* Libraries \n*\n\"" \
      -ex "info sharedlib" \
      -ex "printf \"*\n* Memory map \n*\n\"" \
      -ex "info target" \
      -ex "printf \"*\n* Registers \n*\n\"" \
      -ex "info registers" \
      -ex "printf \"*\n* Current instructions \n*\n\"" -ex "x/16i \$pc" \
      -ex "printf \"*\n* Threads (full) \n*\n\"" \
      -ex "info threads" \
      -ex "bt" \
      -ex "thread apply all bt full" \
      -ex "printf \"*\n* Threads (basic) \n*\n\"" \
      -ex "info threads" \
      -ex "thread apply all bt" \
      -ex "printf \"*\n* Done \n*\n\"" \
      -ex "generate-core-file $corefile" \
      -ex "detach" \
      -ex "quit"

執行緒區域性儲存

使用 gdb 在核心檔案中訪問 TLS 資料相當困難。

參考資料

• __thread 變數