linuxhookwrite,关于Linux的core dump-skykiker

core dump简介

core dump就是在进程crash时把包括内存在内的现场保留下来,以备故障分析。 但有时候,进程crash了却没有输出core,因为有一些因素会影响输出还是不输出core文件。 常见的一个coredump开关是ulimit -c,它限制允许输出的coredump文件的最大size,如果要输出的core文件大小超过这个值将不输出core文件。

ulimit -c的输出为0,代表关闭core dump输出。

[root@srdsdevapp69 ~]# ulimit -c0

设置ulimit -c unlimited,将不对core文件大小做限制

[root@srdsdevapp69 ~]# ulimit -c unlimited[root@srdsdevapp69 ~]# ulimit -cunlimited

这样设置的ulimit值只在当前会话中有效,重开一个终端起进程是不受影响的。

ulimit -c只是众多影响core输出因素中的一个,其它因素可以参考man。

$ man core...There are various circumstances in which a core dump file is not produced:*Theprocessdoesnot have permission to write the core file.(By default the core file is called core,and is created in the current working directory.See below for details on naming.)Writing the core filewillfailif the directory in which it is to be created is non-writable, or if a file with the same nameexists and is not writable or is not a regular file (e.g., it is a directory or a symbolic link).*A (writable, regular) file with the same name as would be used for the core dump already exists, but thereis more than one hard link to that file.*The file system where the core dump file would be created is full; or has run out of inodes; or is mountedread-only; or the user has reached their quota for the file system.*The directory in which the core dump file is to be created does not exist.*The RLIMIT_CORE (core file size) or RLIMIT_FSIZE (file size) resource limits for the processaresettozero; see getrlimit(2) and the documentation of the shell’s ulimit command (limit in csh(1)).*The binary being executed by the process does not have read permission enabled.*Theprocessis executing a set-user-ID (set-group-ID) program that is owned by a user (group) other thanthe real user (group) ID of the process.(However, see the description oftheprctl(2)PR_SET_DUMPABLEoperation, and the description of the /proc/sys/fs/suid_dumpable file in proc(5).)

其实还漏了一个,进程可以捕获那些本来会出core的信号,然后自己来处理,比如MySQL就是这么干的。

abrtd

RHEL/CentOS下默认开启abrtd进行故障现场记录(包括生成coredump)和故障报告

此时abrtd进程是启动的,

[root@srdsdevapp69 ~]# service abrtd statusabrtd (pid8711) is running...

core文件的生成位置被重定向到了abrt-hook-ccpp

[root@srdsdevapp69 ~]# cat /proc/sys/kernel/core_pattern|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e 测试coredump

生成以下产生coredump的程序,并执行。

testcoredump.c:

int main(){return 1/0;}

编译并执行

$gcc testcoredump.c -o testcoredump$./testcoredump

查看系统日志,中途临时产生了core文件,但最后又被删掉了。

$tail -f/var/log/messages...Dec8 09:54:44 srdsdevapp69 kernel: testcoredump[4028] trap divide error ip:400489 sp:7fff5a54b200 error:0 in testcoredump[400000+1000]Dec8 09:54:44 srdsdevapp69 abrtd: Directory 'ccpp-2016-12-08-09:54:44-4028' creation detectedDec8 09:54:44 srdsdevapp69 abrt[4029]: Saved core dump of pid 4028 (/root/testcoredump) to /var/spool/abrt/ccpp-2016-12-08-09:54:44-4028 (184320 bytes)Dec8 09:54:44 srdsdevapp69 abrtd: Executable '/root/testcoredump' doesn't belong to any packageDec8 09:54:44 srdsdevapp69 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-12-08-09:54:44-4028' exited with 1Dec8 09:54:44 srdsdevapp69 abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2016-12-08-09:54:44-4028, deleting

abrtd默认只保留软件包里的程序产生的core文件,修改下面的参数可以让其记录所有程序的core文件。

$vi /etc/abrt/abrt-action-save-package-data.conf...ProcessUnpackaged = yes

再执行一次测试程序就好生成core文件了

Dec8 10:04:30 srdsdevapp69 kernel: testcoredump[9189] trap divide error ip:400489 sp:7fff99973b30 error:0 in testcoredump[400000+1000]Dec8 10:04:30 srdsdevapp69 abrtd: Directory 'ccpp-2016-12-08-10:04:30-9189' creation detectedDec8 10:04:30 srdsdevapp69 abrt[9190]: Saved core dump of pid 9189 (/root/testcoredump) to /var/spool/abrt/ccpp-2016-12-08-10:04:30-9189 (184320 bytes)Dec8 10:04:31 srdsdevapp69 kernel: Bridge firewalling registeredDec8 10:04:44 srdsdevapp69 abrtd: Sending an email...Dec8 10:04:44 srdsdevapp69 abrtd: Email was sent to: root@localhostDec8 10:04:44 srdsdevapp69 abrtd: New problem directory /var/spool/abrt/ccpp-2016-12-08-10:04:30-9189, processingDec8 10:04:44 srdsdevapp69 abrtd: No actions are found for event 'notify'

abrtd可以识别出是重复问题,并能够去重,这可以防止core文件生成的过多把磁盘用光。

Dec8 10:18:35 srdsdevapp69 kernel: testcoredump[16598] trap divide error ip:400489 sp:7fff26cc9f50 error:0 in testcoredump[400000+1000]Dec8 10:18:35 srdsdevapp69 abrtd: Directory 'ccpp-2016-12-08-10:18:35-16598' creation detectedDec8 10:18:35 srdsdevapp69 abrt[16599]: Saved core dump of pid 16598 (/root/testcoredump) to /var/spool/abrt/ccpp-2016-12-08-10:18:35-16598 (184320 bytes)Dec8 10:18:45 srdsdevapp69 abrtd: Sending an email...Dec8 10:18:45 srdsdevapp69 abrtd: Email was sent to: root@localhostDec8 10:18:45 srdsdevapp69 abrtd: Duplicate: UUIDDec8 10:18:45 srdsdevapp69 abrtd: DUP_OF_DIR: /var/spool/abrt/ccpp-2016-12-08-10:04:30-9189Dec8 10:18:45 srdsdevapp69 abrtd: Problem directory is a duplicate of /var/spool/abrt/ccpp-2016-12-08-10:04:30-9189Dec8 10:18:45 srdsdevapp69 abrtd: Deleting problem directory ccpp-2016-12-08-10:18:35-16598 (dup of ccpp-2016-12-08-10:04:30-9189)Dec8 10:18:45 srdsdevapp69 abrtd: No actions are found for event 'notify_dup'

abrtd对crash报告的大小(主要是core文件)有限制(参数MaxCrashReportsSize设置),超过了也不会生成core文件,相应的日志如下。

Dec8 14:10:32 srdsdevapp69 abrt[10548]: Saved core dump of pid 10527 (/usr/local/Percona-Server-5.6.29-rel76.2-Linux.x86_64.ssl101/bin/mysqld) to /var/spool/abrt/ccpp-2016-12-08-14:10:00-10527 (10513362944 bytes)Dec8 14:10:32 srdsdevapp69 abrtd: Directory 'ccpp-2016-12-08-14:10:00-10527' creation detectedDec8 14:10:32 srdsdevapp69 abrtd: Size of '/var/spool/abrt' >= 1000 MB, deleting 'ccpp-2016-12-08-14:05:43-8080'Dec8 14:10:32 srdsdevapp69 abrt[10548]: /var/spool/abrt is 25854515653 bytes (more than 1279MiB), deleting 'ccpp-2016-12-08-14:05:43-8080'Dec8 14:10:32 srdsdevapp69 abrt[10548]: Lock file '/var/spool/abrt/ccpp-2016-12-08-14:05:43-8080/.lock' is locked by process 7893Dec8 14:10:32 srdsdevapp69 abrt[10548]: '/var/spool/abrt/ccpp-2016-12-08-14:05:43-8080' does not existDec8 14:10:41 srdsdevapp69 abrtd: Sending an email...Dec8 14:10:41 srdsdevapp69 abrtd: Email was sent to: root@localhostDec8 14:10:41 srdsdevapp69 abrtd: New problem directory /var/spool/abrt/ccpp-2016-12-08-14:10:00-10527, processingDec8 14:10:41 srdsdevapp69 abrtd: No actions are found for event 'notify' abrtd如何工作

abrtd是监控/var/spool/abrt/目录触发的,做个copy操作也会触发abrtd。

[root@srdsdevapp69 abrt]# cp -rf ccpp-2016-12-08-10:04:30-9189 ccpp-2016-12-08-10:04:30-91891

下面是产生的系统日志:

Dec8 10:35:33 srdsdevapp69 abrtd: Directory 'ccpp-2016-12-08-10:04:30-91891' creation detectedDec8 10:35:33 srdsdevapp69 abrtd: Duplicate: UUIDDec8 10:35:33 srdsdevapp69 abrtd: DUP_OF_DIR: /var/spool/abrt/ccpp-2016-12-08-10:04:30-9189Dec8 10:35:33 srdsdevapp69 abrtd: Problem directory is a duplicate of /var/spool/abrt/ccpp-2016-12-08-10:04:30-9189Dec8 10:35:33 srdsdevapp69 abrtd: Deleting problem directory ccpp-2016-12-08-10:04:30-91891 (dup of ccpp-2016-12-08-10:04:30-9189)Dec8 10:35:33 srdsdevapp69 abrtd: No actions are found for event 'notify_dup'

如果修改core生成目录,不使用abrt-hook-ccpp回调程序等于禁用了abrtd

echo "/data/core-%e-%p-%t">/proc/sys/kernel/core_pattern

再发生coredump时/var/log/messages中没有abrtd相关的记录

Dec8 10:30:24 srdsdevapp69 kernel: testcoredump[23050] trap divide error ip:400489 sp:7fff9f01dfb0 error:0 in testcoredump[400000+1000]

此时core文件会被直接生成到/proc/sys/kernel/core_pattern指定的位置

/data/core-testcoredump-23050-1481164224

由于/proc/sys/kernel/core_pattern中未使用abrt-hook-ccpp回调程序,检查abrt-ccpp服务状态也会相应的返回服务未启动。

[root@srdsdevapp69 ~]# service abrt-ccpp status[root@srdsdevapp69 ~]# echo $?3

恢复/proc/sys/kernel/core_pattern之后,abrt-ccpp服务变回正常

[root@srdsdevapp69 ~]# echo "|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e">/proc/sys/kernel/core_pattern[root@srdsdevapp69 ~]# service abrt-ccpp status[root@srdsdevapp69 ~]# echo $?0

如果停止abrtd

/proc/sys/kernel/core_pattern为"|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e"

会在生成当前目录生成core文件

Dec8 10:46:21 srdsdevapp69 kernel: testcoredump[31364] trap divide error ip:400489 sp:7fff15d6f450 error:0 in testcoredump[400000+1000]Dec8 10:46:21 srdsdevapp69 abrt[31365]: abrtd is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'Dec8 10:46:21 srdsdevapp69 abrt[31365]: Saved core dump of pid 31364 to /root/core.31364 (184320 bytes) 开启MySQL的coredump

MySQL的服务进程mysqld会自己捕获可能引起crash的信号,默认会输出调用栈后异常退出不会生成core文件。

2016-12-08 11:14:51 14034 [Note] /usr/local/mysql/bin/mysqld: ready for connections.Version: '5.6.29-76.2-debug-log'socket: '/mysqlrds/data/mysql.sock'port: 3306Source distribution03:18:43 UTC - mysqld got signal 8 ;This could be because you hit a bug. It is also possible that this binaryor one of the libraries it was linked against is corrupt, improperly built,or misconfigured. This error can also be caused by malfunctioning hardware.We will try our best to scrape up some info that will hopefully helpdiagnose the problem, but since we have already crashed,something is definitely wrong and this may fail.Please help us make Percona Server better by reporting anybugs at key_buffer_size=33554432read_buffer_size=2097152max_used_connections=2max_threads=100001thread_count=1connection_count=1It is possible that mysqld could use up tokey_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 307242932 Kbytes of memoryHope that's ok; if not, decrease some variables in the equation.Thread pointer: 0x2427ca20Attempting backtrace. You can use the following information to find outwhere mysqld died. If you see no messages after this, something wentterribly wrong...stack_bottom = 7fd53066bca8 thread_stack 0x40000/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0xaf23c9]/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x42e)[0x74d42a]/lib64/libpthread.so.0[0x3805a0f7e0]/usr/local/mysql/bin/mysqld(_Z19mysql_rename_tablesP3THDP10TABLE_LISTb+0x6c)[0x82fa64]/usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x2aab)[0x8079e9]/usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x588)[0x810ce3]/usr/local/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xd8b)[0x80228a]/usr/local/mysql/bin/mysqld(_Z10do_commandP3THD+0x3bd)[0x801087]/usr/local/mysql/bin/mysqld(_Z26threadpool_process_requestP3THD+0x71)[0x8ec721]/usr/local/mysql/bin/mysqld[0x8ef363]/usr/local/mysql/bin/mysqld[0x8ef5a0]/usr/local/mysql/bin/mysqld(pfs_spawn_thread+0x159)[0xe14049]/lib64/libpthread.so.0[0x3805a07aa1]/lib64/libc.so.6(clone+0x6d)[0x32286e893d]Trying to get some variables.Some pointers may be invalid and cause the dump to abort.Query (7fd508004d80): is an invalid pointerConnection ID (thread ID): 1Status: NOT_KILLEDYou may download the Percona Server operations manual by visiting You may find informationin the manual which will help you identify the cause of the crash.

要使其产生core文件必须打开--core-file开关

mysqld --defaults-file=/home/mysql/etc/my.cnf --core-file &

也可以将这个参数加入到my.cnf文件中

core_file core文件的大小

关于core文件的大小有个奇怪的现象,其实际占用的磁盘空间可能远小于文件大小。

比如下面的core文件,文件大小10GB,但实际占用磁盘只有2GB(1940984 * 512B)。

[root@srdsdevapp69 ccpp-2016-12-08-14:10:00-10527]# stat coredumpFile: `coredump'Size: 10513362944 Blocks: 1940984IO Block: 4096 regular fileDevice: fd03h/64771dInode: 14990 Links: 1Access: (0640/-rw-r-----)Uid: (173/abrt) Gid: (512/ mysql)Access: 2016-12-08 14:10:41.886280668 +0800Modify: 2016-12-08 14:10:27.704523443 +0800Change: 2016-12-08 14:10:27.704523443 +0800

这是由于系统在生成core文件时,skip了部分全零的块,即文件中有hole(用dd的seek可以模拟这个现象)。不管是在/proc/sys/kernel/core_pattern中设置abrt-hook-ccpp程序还是直接设置文件目录,都是这个现象。这其实是一个不错的优化,节省了磁盘空间也加快了core文件生成速度。

阅读(18027) | 评论(0) | 转发(0) |0

上一篇:基于Pacemaker的PostgreSQL一主多从读负载均衡集群搭建

下一篇:GitLab误删 300G数据的一点启示

白落梅散文读后感docx白落梅散文集读后感安源煤业600397财务指标辛巴回归首秀破快手纪录怎样选购黄金手镯吴曼思相关电影电视剧今日水印相机老版本下载韩国男子服装古代catia怎么不见返回草图工具catia中草图工具怎么用com3d2实时修改器如何安装友谊短小说青春故事短篇小说感人简单齐白石的画的特点77第76章第七十六章小丑竟是我自己医院之小说带系统的医生文偷窥日比比亲眼看到爸妈那个啥咋办裸婚时代简介裸婚时代内容简介转生恶役千金类小说转身恶役只好拔除旗标转生恶役类小说爱情电影安娜卡列尼娜普通话版浙江众安建设集团有限公司中标项目