Пролагивание коннектов при проверке синтаксиса

Вт Ноя 13 17:23:03 UTC 2018

Добрый день.

Есть сервер (Dual Xeon E5620, средняя нагрузка проца в праймтайм 20%) с
nginx в главной и по сути единственной роли, задача которого проксирование
и больше ничего.

В конфиге 101 listen на уникальные ip:port, nginx без сторонних модулей.
1.15.4, но думаю что версия тут не при чем.
Debian Linux Jessy с ядром 3.16.

В среднем одновременных коннектов открыто 150-200 тыс. 70% из них по 443
порту.
в логе собирается время отвремя апстрима и было замечено, то раз в 30
секунд оно пролагивает у части коннектов на лишнюю секунду, т.е. обычно
0.01, а тут 1.01 сек.
Выяснилось что раз в 30 секунд срабатывает проверка конфига заббиксом, т.е.
вызывается
nginx -t

Ручной вызов nginx -t привел к появлению в логах лагающих  запросов в это
время, выключение аббикс агента такие запросы убирает вообще.
команда
time nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
real    0m0.601s
user    0m0.044s
sys     0m0.432s
показывает 0.4 с лишним секунды в режиме ядра.
а попытка разобраться чем же занято ядра выдало ниже следующее

strace -Ttt nginx -t 2>&1 | grep bind

т.е. bind на 443 порты занимает 15 тысячных секунды, против стотысячных
долей у прочих биндов.

Есть ли идеи, как решить проблему пролагиваний при проверке конфига?
Вариант убрать её из заббикса считаю академически неправильным, просьба его
не рассматривать.

reuseport не используется, может он помочь?

20:06:08.293194 bind(57, {sa_family=AF_INET, sin_port=htons(515),
sin_addr=inet_addr("192.168.72.55")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000013>
20:06:08.293372 bind(57, {sa_family=AF_INET, sin_port=htons(8186),
sin_addr=inet_addr("192.168.72.149")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000012>
20:06:08.293560 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.208")}, 16) = -1 EADDRINUSE (Address already
in use) <0.016715>
20:06:08.310458 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.149")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015673>
20:06:08.326346 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.148")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015727>
20:06:08.342292 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.214")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015751>
20:06:08.358260 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.235")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015733>
20:06:08.374236 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.230")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015803>
20:06:08.390575 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.217")}, 16) = -1 EADDRINUSE (Address already
in use) <0.016433>
20:06:08.407377 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.213")}, 16) = -1 EADDRINUSE (Address already
in use) <0.016389>
20:06:08.423989 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.229")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015756>
20:06:08.440013 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.239")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015815>
20:06:08.456204 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.219")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014685>
20:06:08.471140 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.212")}, 16) = -1 EADDRINUSE (Address already
in use) <0.016018>
20:06:08.487397 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.228")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014647>
20:06:08.502250 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.222")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014635>
20:06:08.517105 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.238")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014834>
20:06:08.532162 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.220")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014635>
20:06:08.547049 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.236")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014566>
20:06:08.561946 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.221")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015155>
20:06:08.577406 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.237")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014706>
20:06:08.592334 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.210")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014753>
20:06:08.607351 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.226")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014676>
20:06:08.622244 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.209")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014591>
20:06:08.637050 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.225")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014654>
20:06:08.651926 bind(57, {sa_family=AF_INET, sin_port=htons(2843),
sin_addr=inet_addr("192.168.72.149")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000015>
20:06:08.652115 bind(57, {sa_family=AF_INET, sin_port=htons(2080),
sin_addr=inet_addr("192.168.72.149")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000010>
20:06:08.652284 bind(57, {sa_family=AF_INET, sin_port=htons(2080),
sin_addr=inet_addr("192.168.72.148")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000009>
20:06:08.652444 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.214")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000044>
20:06:08.652645 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.235")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000016>
20:06:08.652834 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.230")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000026>
20:06:08.653012 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.217")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000018>
20:06:08.653175 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.213")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000015>
20:06:08.653336 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.229")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000014>
20:06:08.653522 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.212")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000026>
20:06:08.653762 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.228")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000070>
20:06:08.654010 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.220")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000022>
20:06:08.654203 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.236")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000033>
20:06:08.654403 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.221")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000021>
20:06:08.654604 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.237")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000016>
20:06:08.654766 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.210")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000014>
20:06:08.654915 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.226")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000013>
20:06:08.655060 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.209")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000013>
20:06:08.655223 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.225")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000027>
20:06:08.655405 bind(57, {sa_family=AF_INET, sin_port=htons(62002),
sin_addr=inet_addr("192.168.72.214")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000011>
20:06:08.655574 bind(57, {sa_family=AF_INET, sin_port=htons(62003),
sin_addr=inet_addr("192.168.72.214")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000009>
20:06:08.655726 bind(57, {sa_family=AF_INET, sin_port=htons(843),
sin_addr=inet_addr("192.168.72.224")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000394>
20:06:08.656262 bind(57, {sa_family=AF_INET, sin_port=htons(843),
sin_addr=inet_addr("192.168.72.211")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000100>
20:06:08.656506 bind(57, {sa_family=AF_INET, sin_port=htons(843),
sin_addr=inet_addr("192.168.72.208")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000100>
20:06:08.656748 bind(57, {sa_family=AF_INET, sin_port=htons(843),
sin_addr=inet_addr("192.168.72.234")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000100>
20:06:08.656988 bind(57, {sa_family=AF_INET, sin_port=htons(843),
sin_addr=inet_addr("192.168.72.231")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000105>
20:06:08.657239 bind(57, {sa_family=AF_INET, sin_port=htons(843),
sin_addr=inet_addr("192.168.72.227")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000101>
20:06:08.657483 bind(57, {sa_family=AF_INET, sin_port=htons(843),
sin_addr=inet_addr("192.168.72.223")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000116>
20:06:08.657810 bind(57, {sa_family=AF_INET, sin_port=htons(843),
sin_addr=inet_addr("192.168.72.218")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000132>
20:06:08.658155 bind(57, {sa_family=AF_INET, sin_port=htons(843),
sin_addr=inet_addr("192.168.72.219")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000131>
20:06:08.658494 bind(57, {sa_family=AF_INET, sin_port=htons(843),
sin_addr=inet_addr("192.168.72.221")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000134>
20:06:08.658903 bind(57, {sa_family=AF_INET, sin_port=htons(843),
sin_addr=inet_addr("192.168.72.237")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000184>
20:06:08.659339 bind(57, {sa_family=AF_INET, sin_port=htons(8161),
sin_addr=inet_addr("192.168.72.230")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000018>
20:06:08.659580 bind(57, {sa_family=AF_INET, sin_port=htons(8162),
sin_addr=inet_addr("192.168.72.230")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000013>
20:06:08.659815 bind(57, {sa_family=AF_INET, sin_port=htons(8188),
sin_addr=inet_addr("192.168.72.212")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000029>
20:06:08.660066 bind(57, {sa_family=AF_INET, sin_port=htons(8188),
sin_addr=inet_addr("192.168.72.228")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000014>
20:06:08.660253 bind(57, {sa_family=AF_INET, sin_port=htons(8189),
sin_addr=inet_addr("192.168.72.212")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000012>
20:06:08.660435 bind(57, {sa_family=AF_INET, sin_port=htons(8189),
sin_addr=inet_addr("192.168.72.228")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000010>
20:06:08.660622 bind(57, {sa_family=AF_INET, sin_port=htons(2003),
sin_addr=inet_addr("192.168.72.220")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000013>
20:06:08.660799 bind(57, {sa_family=AF_INET, sin_port=htons(2003),
sin_addr=inet_addr("192.168.72.236")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000010>
20:06:08.660980 bind(57, {sa_family=AF_INET, sin_port=htons(2003),
sin_addr=inet_addr("192.168.72.221")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000015>
20:06:08.661200 bind(57, {sa_family=AF_INET, sin_port=htons(2003),
sin_addr=inet_addr("192.168.72.237")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000011>
20:06:08.661399 bind(57, {sa_family=AF_INET, sin_port=htons(2003),
sin_addr=inet_addr("192.168.72.210")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000014>
20:06:08.661606 bind(57, {sa_family=AF_INET, sin_port=htons(2003),
sin_addr=inet_addr("192.168.72.226")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000009>
20:06:08.661773 bind(57, {sa_family=AF_INET, sin_port=htons(2003),
sin_addr=inet_addr("192.168.72.209")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000009>
20:06:08.661938 bind(57, {sa_family=AF_INET, sin_port=htons(2003),
sin_addr=inet_addr("192.168.72.225")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000010>
20:06:08.662102 bind(57, {sa_family=AF_INET, sin_port=htons(22443),
sin_addr=inet_addr("192.168.72.221")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000010>
20:06:08.662330 bind(57, {sa_family=AF_INET, sin_port=htons(22443),
sin_addr=inet_addr("192.168.72.237")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000009>
20:06:08.662497 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.133")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000027>
20:06:08.662681 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.146")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000018>
20:06:08.662871 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.211")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000026>
20:06:08.663061 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.55")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000024>
20:06:08.663243 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.216")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000018>
20:06:08.663479 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.143")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000017>
20:06:08.663665 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.233")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000018>
20:06:08.663839 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.142")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000017>
20:06:08.664053 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.13.177.64")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000017>
20:06:08.664230 bind(57, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("192.168.72.135")}, 16) = -1 EADDRINUSE (Address already
in use) <0.000016>
20:06:08.664403 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.133")}, 16) = -1 EADDRINUSE (Address already
in use) <0.016182>
20:06:08.680876 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.146")}, 16) = -1 EADDRINUSE (Address already
in use) <0.016322>
20:06:08.697446 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.211")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014816>
20:06:08.712585 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.55")}, 16) = -1 EADDRINUSE (Address already
in use) <0.016207>
20:06:08.729253 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.216")}, 16) = -1 EADDRINUSE (Address already
in use) <0.014857>
20:06:08.744430 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.143")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015272>
20:06:08.760014 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.233")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015102>
20:06:08.775590 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.142")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015129>
20:06:08.791125 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.57")}, 16) = -1 EADDRINUSE (Address already
in use) <0.015431>
20:06:08.806845 bind(57, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("192.168.72.135")}, 16) = -1 EADDRINUSE (Address already
in use) <0.016197>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-ru/attachments/20181113/05ac0379/attachment-0001.html>