Тюнинг по нагрузку и странные ребуты

Alexander A. Gnatyna gnatyna at gmail.com
Sat Mar 22 13:11:36 MSK 2008


Здравствуйте. 
у меня была похожая проблема, сервер читал, писал и отдавал кучу мелких 
файлов. Решилось добавлением памяти ядру, 
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
и 
vfs.ufs.dirhash_maxmem=134217728
kern.maxvnodes=300000

uname -rps
FreeBSD 6.2-RELEASE i386

Гнатына Александр.

В сообщении от Saturday 22 March 2008 12:40:46 Михаил Монашёв написал(а):
> Здравствуйте,
>
> Осенью я настроил сервер на обслуживание большого числа соединений по
> презентации Игоря:
>
> FreeBSD 6.3-PRERELEASE
>
> >cat /boot/loader.conf
>
> autoboot_delay="1"
> userconfig_script_load="YES"
> beastie_disable="YES"
>
> net.inet.tcp.syncache.hashsize="1024"
> net.inet.tcp.syncache.bucketlimit="100"
> net.inet.tcp.tcbhashsize="4096"
>
> vm.kmem_size="1G"
> kern.ipc.nmbclusters="262144"
>
> net.inet.tcp.hostcache.hashsize="4096"
> net.inet.tcp.hostcache.bucketlimit="100"
>
> >cat /etc/sysctl.conf
>
> net.inet.tcp.msl=7500
> net.inet.tcp.delayed_ack=0
>
> net.inet.ip.random_id=1
>
> net.inet.ip.fw.dyn_buckets=32768
> net.inet.ip.fw.dyn_max=32768
> net.inet.ip.fw.dyn_ack_lifetime=120
> net.inet.ip.fw.dyn_syn_lifetime=10
> net.inet.ip.fw.dyn_fin_lifetime=2
> net.inet.ip.fw.dyn_short_lifetime=10
>
> net.inet.tcp.keepidle=40000
> net.inet.tcp.keepintvl=40000
> net.inet.tcp.keepinit=40000
>
> net.inet.tcp.sendspace=65536
> net.inet.tcp.recvspace=16384
>
> net.inet.tcp.blackhole=2
> net.inet.udp.blackhole=1
>
> net.inet.tcp.maxtcptw=40960
>
> net.inet.ip.portrange.randomized=0
>
> # bugly
> # net.inet.tcp.nolocaltimewait=1
>
> net.inet.tcp.syncookies=1
>
> kern.ipc.maxsockets=204800
>
> kern.maxfiles=204800
> kern.maxfilesperproc=32768
>
> kern.ipc.somaxconn=4096
>
>
> Сейчас  периодически  начались самостоятельные ребуты сервера. В логах
> FreeBSD пусто. В логах жизнедеятельности железа тоже всё нормально (на
> интеловской  матери  есть одна хрень, которая собирает данные с разных
> компонентов сервера и пишет её в свою флэ-память.) Такое ощущение, что
> питание  просто  вытаскивают  из  сервера на секунду. Происходит такой
> ребут  раз  в  1-2 дня. Мучаюсь уже неделю. Сейчас FreeBSD поднимем до
> нормальной версии 6.3, но не уверен, что поможет.
>
> Так  вот  вопрос:  могут  ли  эти  ребуты  быть причиной моего кривого
> тюнинга  FreeBSD.  Или  же  грешить на железо, которое вдруг почему-то
> начало глючить...
>
> >vmstat -z
>
> ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS 
> FAILURES
>
> UMA Kegs:                 240,        0,       67,        8,       67,     
>   0 UMA Zones:                280,        0,       67,       11,       67, 
>       0 UMA Slabs:                128,        0,     1760,      763,  
> 147183,        0 UMA RCntSlabs:            128,        0,     2328,     
> 340,   130792,        0 UMA Hash:                 256,        0,        3, 
>      12,        6,        0 16 Bucket:                152,        0,      
> 69,       56,      118,        0 32 Bucket:                280,        0,  
>     27,       57,       91,        0 64 Bucket:                536,       
> 0,       38,       74,      127,       65 128 Bucket:              1048,   
>     0,      755,      103,     2924,   617837 VM OBJECT:               
> 224,        0,    28720,    42714, 10236642,        0 MAP:                 
>     352,        0,        7,       15,        7,        0 KMAP ENTRY:      
>         112,    98406,       21,      276,   164359,        0 MAP ENTRY:   
>             112,        0,     9491,     2356, 36019655,        0 PV ENTRY:
>                  48,  5706864,  2002569,   162039, 2086499412,        0 DP
> fakepg:                120,        0,        0,        0,        0,       
> 0 mt_zone:                 1024,        0,      134,       10,      134,   
>     0 16:                        16,        0,     2291,      901,
> 53841173,        0 32:                        32,        0,     5163,    
> 1604, 14835282,        0 64:                        64,        0,     2717,
>     2099, 48217270,        0 128:                      128,        0,    
> 4994,      922, 31667804,        0 256:                      256,        0,
>      882,      603,  9527058,        0 512:                      512,      
>  0,     1384,      660,  1634008,        0 1024:                    1024,  
>      0,      587,      413,   346743,        0 2048:                   
> 2048,        0,       57,      329,   404452,        0 4096:               
>     4096,        0,      262,      305,  2182521,        0 Files:          
>          120,        0,     8245,     1458, 37445260,        0 TURNSTILE:  
>              104,        0,      961,      128,     1237,        0 PROC:   
>                  856,        0,      149,      331,   371712,        0
> THREAD:                   608,        0,      676,      284,     9502,     
>   0 KSEGRP:                   136,        0,      676,      260,     3008, 
>       0 UPCALL:                    88,        0,        0,        0,       
> 5,        0 SLEEPQUEUE:                64,        0,      961,      159,   
>  1237,        0 VMSPACE:                  544,        0,      111,     
> 428,   371674,        0 mbuf_packet:              256,        0,     3787, 
>     301, 415797233,        0 mbuf:                     256,        0,     
> 556,     1296, 593199194,        0 mbuf_cluster:            2048,   262144,
>     4088,      568, 148587055,        0 mbuf_jumbo_pagesize:     4096,     
>   0,        0,        0,        0,        0 mbuf_jumbo_9k:           9216, 
>       0,        0,        0,        0,        0 mbuf_jumbo_16k:        
> 16384,        0,        0,        0,        0,        0 ACL UMA zone:      
>       388,        0,        0,        0,        0,        0 g_bio:         
>           216,        0,        4,     9860, 50455273,        0 VNODE:     
>               496,        0,    45795,    44749,  1905792,        0
> VNODEPOLL:                152,        0,        0,        0,        0,     
>   0 NAMEI:                   1024,        0,        0,      580, 43832293, 
>       0 S VFS Cache:              104,        0,    45784,    26684, 
> 1416847,        0 L VFS Cache:              327,        0,     2908,    
> 4052,   302194,        0 DIRHASH:                 1024,        0,     1856,
>      180,    12081,        0 PIPE:                     768,        0,      
>  8,      457,   244287,        0 KNOTE:                    120,        0,  
>   7335,      911, 156090348,        0 socket:                   616,  
> 204804,     7357,     3611, 17920375,        0 unpcb:                   
> 200,   204801,       95,      665,    39511,        0 ipq:                 
>      56,     8253,        0,       63,       46,        0 udpcb:           
>         304,   204804,        6,      366,    80825,        0 inpcb:       
>             304,   204804,    11690,     4450, 17799979,        0 tcpcb:   
>                 752,   204800,     7246,     3644, 17799979,        0
> tcptw:                     80,    40995,     4444,     5726, 11187304,     
>   0 syncache:                 128,    15370,       51,      355,  6486053, 
>       0 hostcache:                136,    15372,    11163,      289,   
> 60523,        0 tcpreass:                  40,    16464,        2,     
> 670,   349949,        0 sackhole:                  32,        0,       39, 
>     668,  1124926,        0 ripcb:                    304,   204804,       
> 2,       58,       53,        0 rtentry:                  264,        0,   
>    38,       32,       41,        0 divcb:                    304,  
> 204804,        1,       23,        1,        0 IPFW dynamic rule:       
> 120,        0,    11231,      425,  5436107,        0 SWAPMETA:            
>     288,   116519,      898,      129,     1861,        0 Mountpoints:     
>         792,        0,       12,       18,       12,        0 FFS inode:   
>             192,        0,    45745,    28435,  1905620,        0 FFS1
> dinode:              128,        0,        0,        0,        0,        0
> FFS2 dinode:              256,        0,    45745,    23480,  1905620,     
>   0
>
> >netstat -s -p tcp
>
> tcp:
>         388830540 packets sent
>                 127026230 data packets (130426602571 bytes)
>                 2547241 data packets (3184142700 bytes) retransmitted
>                 90608 data packets unnecessarily retransmitted
>                 208 resends initiated by MTU discovery
>                 212916641 ack-only packets (0 delayed)
>                 0 URG only packets
>                 19541 window probe packets
>                 19085752 window update packets
>                 27279475 control packets
>         347853681 packets received
>                 137363681 acks (for 129899106763 bytes)
>                 16041798 duplicate acks
>                 1610 acks for unsent data
>                 192313042 packets (179744150693 bytes) received in-sequence
>                 1131583 completely duplicate packets (97511822 bytes)
>                 23860 old duplicate packets
>                 7144 packets with some dup. data (4380337 bytes duped)
>                 350145 out-of-order packets (433368103 bytes)
>                 2878 packets (717212 bytes) of data after window
>                 359 window probes
>                 8980292 window update packets
>                 6014 packets received after close
>                 2241 discarded for bad checksums
>                 0 discarded for bad header offset fields
>                 0 discarded because packet too short
>         11350776 connection requests
>         6474009 connection accepts
>         541002 bad connection attempts
>         0 listen queue overflows
>         1925 ignored RSTs in the windows
>         17757687 connections established (including accepts)
>         17813031 connections closed (including 565736 drops)
>                 2445514 connections updated cached RTT on close
>                 2467782 connections updated cached RTT variance on close
>                 2058318 connections updated cached ssthresh on close
>         55688 embryonic connections dropped
>         121980133 segments updated rtt (of 93433055 attempts)
>         1939824 retransmit timeouts
>                 15977 connections dropped by rexmit timeout
>         31239 persist timeouts
>                 15 connections dropped by persist timeout
>         2817074 keepalive timeouts
>                 958555 keepalive probes sent
>                 67906 connections dropped by keepalive
>         3602212 correct ACK header predictions
>         140392129 correct data packet header predictions
>         6494799 syncache entries added
>                 73149 retransmitted
>                 73443 dupsyn
>                 18 dropped
>                 6474009 completed
>                 0 bucket overflow
>                 0 cache overflow
>                 9429 reset
>                 11181 stale
>                 0 aborted
>                 0 badack
>                 87 unreach
>                 0 zone failures
>         6494817 cookies sent
>         0 cookies received
>         389309 SACK recovery episodes
>         611350 segment rexmits in SACK recovery episodes
>         861610061 byte rexmits in SACK recovery episodes
>         2877534 SACK options (SACK blocks) received
>         303214 SACK options (SACK blocks) sent
>         0 SACK scoreboard overflow
>
> >netstat -m
>
> 4147/1688/5835 mbufs in use (current/cache/total)
> 3597/1301/4898/262144 mbuf clusters in use (current/cache/total/max)
> 3597/415 mbuf+clusters out of packet secondary zone in use (current/cache)
> 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
> 0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
> 8230K/3024K/11254K bytes allocated to network (current/cache/total)
> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
> 0/0/0 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 48518 requests for I/O initiated by sendfile
> 626 calls to protocol drain routines
>
>
> P.S.
>
> Как исправить вот это:
> >vmstat -z
>
> ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS 
> FAILURES 64 Bucket:                536,        0,       38,       74,     
> 127,       65 ^^ 128 Bucket:              1048,        0,      755,     
> 103,     2924,   617837 ^^^^^^ и надо ли это исправлять?
>
> --
>
> С уважением,
> Михаил Монашёв, SoftSearch.ru
> mailto:postmaster at softsearch.ru
> ICQ# 166233339
> http://michael.mindmix.ru/
> Без бэкапа по жизни.







More information about the nginx-ru mailing list