coredump

[sprite]

Добрый день, коллеги.
Стал у меня в последнее время вываливаться сервер в coredump и довольно часто, только выключение кнопкой питания приводит его в чувства. Ни как не могу отследить проблему, на экране ошибка kernel trap 12 with interrupts disabled и current process могут быть разные то syncer , то pagedaemon, то еще чего нибудь. Пытаюсь получить данны из самого дампа, для этого в rc.conf прописал

Код: Выделить всё

dumpdev="AUTO"
dumpdir="/var/crash"
savecore_flags="-v"

ядро собранно с makeoptions DEBUG=-g

но vmcore.N в /var/crash все пустые, пробую

Код: Выделить всё

[root@nas6 /var/crash]# kgdb /boot/kernel/kernel vmcore.3 
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Failed to open vmcore: wrong minidump version. expected 1 got 2
(kgdb) 

результат ноль Подскажите, что я делаю не так
В /var/crash имеются файлы info.N в них что-то записано, например

Код: Выделить всё

[root@nas6 /var/crash]# cat info.3 
Dump header from device /dev/da0s1b
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 827527168B (789 MB)
  Blocksize: 512
  Dumptime: Mon Oct 10 00:55:13 2011
  Hostname: nas6
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 7.4-RELEASE-p4 #2: Sun Oct  9 20:56:47 MSD 2011
    ivan@nas6:/usr/obj/usr/src/sys/NAS
  Panic String: sbflush_internal: cc 0 || mb 0xffffff00080cde00 || mbcnt 4608
  Dump Parity: 54328185
  Bounds: 3
  Dump Status: good

Ядро 7.4-RELEASE и пробовал FreeBSD 7.4-RELEASE-p4, железка HP DL160G6. Подскажите куда копать, чтобы разобраться с этой проблемой.

[Хостинг HostFood.ru]

Тарифы на хостинг в России, от 12 рублей: https://www.host-food.ru/tariffs/hosting/
Тарифы на виртуальные сервера (VPS/VDS/KVM) в РФ, от 189 руб.: https://www.host-food.ru/tariffs/virtualny-server-vps/
Выделенные сервера, Россия, Москва, от 2000 рублей (HP Proliant G5, Intel Xeon E5430 (2.66GHz, Quad-Core, 12Mb), 8Gb RAM, 2x300Gb SAS HDD, P400i, 512Mb, BBU):
https://www.host-food.ru/tariffs/vydelennyi-server-ds/
Недорогие домены в популярных зонах: https://www.host-food.ru/domains/

[hizel]

Failed to open vmcore: wrong minidump version. expected 1 got 2

такое осчусчение что у вас userland и kernel разных версий

[hizel]

посмотрите /usr/src/UPDATING про minidump и amd64

[sprite]

понятно, спасибо за совет. попробую поковырять в этом направление.

[Bayerische]

Когда случается хронический крах, разумно проверить состояние носителей. Далее, памяти и общую стабильность железа.

[sprite]

Поставил для чистоты опыта чистую систему
FreeBSD nas7 7.4-STABLE FreeBSD 7.4-STABLE #0: Wed Oct 12 13:22:42 UTC 2011 ivan@nas7:/usr/obj/usr/src/sys/NAS amd64
с таким же конфигом ядра. От GENERIC отличается только добавленными

Код: Выделить всё

options         MROUTING                # Multicast routing
options         IPFIREWALL              #firewall
options         IPFIREWALL_VERBOSE      #enable logging to syslogd(8)
options         IPFIREWALL_VERBOSE_LIMIT=100    #limit verbosity
options         IPFIREWALL_DEFAULT_TO_ACCEPT    #allow everything by default
options         IPFIREWALL_FORWARD      #packet destination changes
options         IPFIREWALL_NAT          #ipfw kernel nat support
options         LIBALIAS
options         ROUTETABLES=3

валится стало чаще и более разнообразно чаще других в таком виде (файл в атаче.)
Получилось содрать vmcore, правда не от случая из картики
результат kgdb такой

Код: Выделить всё

nas7# kgdb kernel.debug /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: sbdrop
cpuid = 4
Uptime: 1h25m13s
Physical memory: 12266 MB
Dumping 1074 MB: 1059 1043 1027 1011 995 979 963 947 931 915 899 883 867 851 835 819 803 787 771 755 739 723 707 691 675 659 643 627 611 595 579 563 547 531 515 499 483 467 451 435 419 403 387 371 355 339 323 307 291 275 259 243 227 211 195 179 163 147 131 115 99 83 67 51 35 19 3

Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nullfs.ko
Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_socket.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/ng_mppc.ko...Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_mppc.ko
Reading symbols from /boot/kernel/rc4.ko...Reading symbols from /boot/kernel/rc4.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/rc4.ko
Reading symbols from /boot/kernel/ng_tee.ko...Reading symbols from /boot/kernel/ng_tee.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_tee.ko
Reading symbols from /boot/kernel/ng_pptpgre.ko...Reading symbols from /boot/kernel/ng_pptpgre.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_pptpgre.ko
Reading symbols from /boot/kernel/ng_ksocket.ko...Reading symbols from /boot/kernel/ng_ksocket.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ksocket.ko
Reading symbols from /boot/kernel/ng_iface.ko...Reading symbols from /boot/kernel/ng_iface.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_iface.ko
Reading symbols from /boot/kernel/ng_ppp.ko...Reading symbols from /boot/kernel/ng_ppp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ppp.ko
Reading symbols from /boot/kernel/ng_vjc.ko...Reading symbols from /boot/kernel/ng_vjc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_vjc.ko
Reading symbols from /boot/kernel/ng_netflow.ko...Reading symbols from /boot/kernel/ng_netflow.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_netflow.ko
Reading symbols from /boot/kernel/ng_tcpmss.ko...Reading symbols from /boot/kernel/ng_tcpmss.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_tcpmss.ko
Reading symbols from /boot/kernel/ng_bpf.ko...Reading symbols from /boot/kernel/ng_bpf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_bpf.ko
Reading symbols from /boot/kernel/ng_car.ko...Reading symbols from /boot/kernel/ng_car.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_car.ko
#0  doadump () at pcpu.h:196
196             __asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xffffff001f95dae0 in ?? ()
#2  0xffffffff8054ee6a in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:421
#3  0xffffffff8054f265 in panic (fmt=0x104 <Address 0x104 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:576
#4  0xffffffff805ab606 in sbdrop_internal (sb=Variable "sb" is not available.
) at /usr/src/sys/kern/uipc_sockbuf.c:843
#5  0xffffffff805ab705 in sbflush_internal (sb=0xffffff0305991bc8) at /usr/src/sys/kern/uipc_sockbuf.c:806
#6  0xffffffff805ab819 in sbrelease_internal (sb=0xffffff0305991bc8, so=0xffffff0305991b40) at /usr/src/sys/kern/uipc_sockbuf.c:328
#7  0xffffffff805ad520 in sofree (so=0xffffff0305991b40) at /usr/src/sys/kern/uipc_socket.c:621
#8  0xffffffff805aeef4 in soclose (so=0xffffff0305991b40) at /usr/src/sys/kern/uipc_socket.c:692
#9  0xffffffff80e3595f in ng_ksocket_shutdown (node=0xffffff0305d0fb00) at /usr/src/sys/modules/netgraph/ksocket/../../../netgraph/ng_ksocket.c:950
#10 0xffffffff80e2879b in ng_rmnode (node=0xffffff0305d0fb00, dummy1=Variable "dummy1" is not available.
) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:735
#11 0xffffffff80e28b8c in ng_apply_item (node=0xffffff0305d0fb00, item=0xffffff001fcb78c0, rw=1)
    at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2407
#12 0xffffffff80e29c7e in ngthread (arg=Variable "arg" is not available.
) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3304
#13 0xffffffff805291e3 in fork_exit (callout=0xffffffff80e29b20 <ngthread>, arg=0x0, frame=0xffffff8166f8ac80) at /usr/src/sys/kern/kern_fork.c:798
#14 0xffffffff80818a5e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:554
#15 0x0000000000000000 in ?? ()
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000001 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000d8e000 in ?? ()
#40 0xffffffff80bb7bc0 in tdq_cpu ()
#41 0xffffffff80bae5c0 in tdq_cpu ()
#42 0xffffffff80bb7cc0 in tdq_groups ()
#43 0x0000000000000000 in ?? ()
#44 0xffffff001f95dae0 in ?? ()
#45 0xffffff8166f8a7f8 in ?? ()
#46 0x0000000000000000 in ?? ()
#47 0xffffffff80572cf9 in sched_switch (td=0x0, newtd=0x8005331d0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1948
#48 0x0000000000000000 in ?? ()
#49 0x0000000000000000 in ?? ()
#50 0x0000000000000000 in ?? ()
#51 0x0000000000000000 in ?? ()
#52 0x0000000000000000 in ?? ()
#53 0x0000000000000000 in ?? ()
#54 0x0000000000000000 in ?? ()
#55 0x0000000000000000 in ?? ()
#56 0x0000000000000000 in ?? ()
#57 0x0000000000000000 in ?? ()
#58 0x0000000000000000 in ?? ()
#59 0x0000000000000000 in ?? ()
#60 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit--- 
#61 0x0000000000000000 in ?? ()
#62 0x0000000000000000 in ?? ()
#63 0x0000000000000000 in ?? ()
#64 0x0000000000000000 in ?? ()
#65 0x0000000000000000 in ?? ()
#66 0x0000000000000000 in ?? ()
#67 0x0000000000000000 in ?? ()
#68 0x0000000000000000 in ?? ()
#69 0x0000000000000000 in ?? ()
#70 0x0000000000000000 in ?? ()
#71 0x0000000000000000 in ?? ()
#72 0x0000000000000000 in ?? ()
#73 0x0000000000000000 in ?? ()
#74 0x0000000000000000 in ?? ()
#75 0x0000000000000000 in ?? ()
#76 0x0000000000000000 in ?? ()
#77 0x0000000000000000 in ?? ()
#78 0x0000000000000000 in ?? ()
#79 0x0000000000000000 in ?? ()
#80 0x0000000000000000 in ?? ()
#81 0x0000000000000000 in ?? ()
#82 0x0000000000000000 in ?? ()
#83 0x0000000000000000 in ?? ()
#84 0x0000000000000000 in ?? ()
#85 0x0000000000000000 in ?? ()
#86 0x0000000000000000 in ?? ()
#87 0x0000000000000000 in ?? ()
#88 0x0000000000000000 in ?? ()
#89 0x0000000000000000 in ?? ()
#90 0x0000000000000000 in ?? ()
#91 0x0000000000000000 in ?? ()
#92 0x0000000000000000 in ?? ()
#93 0x0000000000000000 in ?? ()
#94 0x0000000000000000 in ?? ()
#95 0x0000000000000000 in ?? ()
#96 0x0000000000000000 in ?? ()
#97 0x0000000000000000 in ?? ()
#98 0x0000000000000000 in ?? ()
#99 0x0000000000000000 in ?? ()
#100 0x0000000000000000 in ?? ()
#101 0x0000000000000000 in ?? ()
#102 0x0000000000000000 in ?? ()
#103 0x0000000000000000 in ?? ()
#104 0x0000000000000000 in ?? ()
#105 0x0000000000000000 in ?? ()
#106 0x0000000000000000 in ?? ()
#107 0x0000000000000000 in ?? ()
#108 0x0000000000000000 in ?? ()
#109 0x0000000000000000 in ?? ()
#110 0x0000000000000000 in ?? ()
#111 0x0000000000000000 in ?? ()
#112 0x0000000000000000 in ?? ()
#113 0x0000000000000000 in ?? ()
#114 0x0000000000000000 in ?? ()
#115 0x0000000000000000 in ?? ()
Cannot access memory at address 0xffffff8166f8b000

насколько я понял, что-то с ng_car, что имено может кто подскажет?