На всех стоит FreeBSD 9.2-RELEASE #0
Железо:
CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3200.18-MHz K8-class CPU)
memory : 64 GB ddr3
HDD: 2x240 GB SSD Intel
Network: 1 Gbit Intel card
На всех трех живет портал domen.com
Соответственно в днс - стоят три айпишника
На сервере живет апач 2.2.24, php 5.3, mysql 5.1 PerconaDB
Месяц назад начались проблемы
На серверах, то на одном, то на втором, то на третьем рандомно, начал ложится апач
Приблизительно в одно и тоже время.. с 9.00 до 12.00
А именно - error лог апача начинает мгновенно наполнятся воплями что
Код: Выделить всё
[Tue Apr 15 17:05:48 2014] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 0 idle, and 151 total children
[Tue Apr 15 17:05:49 2014] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 16 children, there are 2 idle, and 159 total children
[Tue Apr 15 17:13:53 2014] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 6 idle, and 165 total children
[Tue Apr 15 17:46:26 2014] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 6 idle, and 130 total children
[Tue Apr 15 17:56:38 2014] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 4 idle, and 138 total children
[Tue Apr 15 18:08:29 2014] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 3 idle, and 125 total children
Код: Выделить всё
server reached MaxClients setting, consider raising the MaxClients settingКод: Выделить всё
Timeout 30
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 1
ServerLimit 100000
<IfModule prefork.c>
StartServers 150
MinSpareServers 10
MaxSpareServers 160
MaxClients 100000
MaxRequestsPerChild 20000
</IfModule>
ListenBackLog 10000- netstat -an | grep -i esta |wc -l
4464
netstat -an | grep SYN - пусто
Код: Выделить всё
netstat -Lan
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen Local Address
tcp46 4336/0/10000 *.80
tcp46 0/0/2500 *.3306
tcp4 0/0/128 *.22
tcp6 0/0/128 *.22
tcp4 0/0/128 *.199
unix 0/0/2500 /tmp/mysql.sock
unix 0/0/4 /var/run/devd.pipe
Код: Выделить всё
cat /etc/sysctl.conf
net.inet.carp.preempt=1
net.inet.ip.fw.dyn_max=16384
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.icmplim=400
net.inet.ip.intr_queue_maxlen=4096
net.inet.tcp.drop_synfin=1
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.finwait2_timeout=3000
net.inet.tcp.maxtcptw=200000
net.inet.tcp.msl=5000
net.inet.tcp.syncookies_only=1
net.inet.icmp.log_redirect=1
net.inet.icmp.drop_redirect=1
net.inet.ip.redirect=0
net.inet6.ip6.redirect=0
kern.maxfilesperproc=200000
kern.maxvnodes=256000
kern.maxfiles=204800
kern.maxfilesperproc=200000
kern.ipc.nmbclusters=262144
kern.ipc.maxsockets=204800
kern.ipc.somaxconn=10000Код: Выделить всё
cat /boot/loader.conf
# Accept filters for data, http and DNS requests
# Useful when your software uses select() instead of kevent/kqueue or when you under DDoS
# DNS accf available on 8.0+
accf_data_load="YES"
accf_http_load="YES"
accf_dns_load="YES"
# If you have really busy webserver with apache13 you may run out of processes
kern.maxproc=15000
# Async IO system calls
aio_load="YES"
kern.ipc.nmbclusters=0правилами фаервола ограничиваю соединения к апачу до 32 с одного адреса :
Код: Выделить всё
#Global section >>>
ipfw add check-state
# Pass our Server
ipfw add pass ip from 'table(1)' to 'table(0)' via ${oif}
# Pass WEB from/to Internet
ipfw add pass tcp from any to 'table(0)' 80 via ${oif} setup limit src-addr 32
# Pass WEB from/to Internet
ipfw add pass tcp from any to 'table(0)' 80 via ${oif} established
# Pass Mail from/to Internet
ipfw add pass tcp from any to 'table(0)' 25 via ${oif} setup limit src-addr 32
# Pass Mail from/to Internet
ipfw add pass tcp from any to 'table(0)' 25 via ${oif} established
# Pass SSH, MySQL from/to Internet (Our HOSTS)
ipfw add pass ip from 'table(2)' to 'table(0)' 22,443 via ${oif}
# Pass PING from/to Internet (Our HOSTS)
ipfw add pass icmp from 'table(2)' to 'table(0)' via ${oif} icmptypes 8
# Allow Fragments
ipfw add pass all from any to any frag
# Paranoya!!!
ipfw add deny ip from any to 'table(0)' 0-1024,3306,55610
Не понимаю почему и где затыкается апач.
