<div id="__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: #000000;text-align: left" dir="ltr"><div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;" id="docs-internal-guid-5f650991-7fff-a7fd-173d-0059483ad3b5"><span style="font-size: 11pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">We have quite a high volume site, we have 4 front end nginx servers, each:</span></p><ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;"><li dir="ltr" style="list-style-type: disc;font-size: 11pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre" aria-level="1"><p dir="ltr" style="line-height:1.38;margin-top:12pt;margin-bottom:0pt;" role="presentation"><span style="font-size: 11pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">AMD EPYC 7402P 24-Core Processor</span></p></li><li dir="ltr" style="list-style-type: disc;font-size: 11pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre" aria-level="1"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;" role="presentation"><span style="font-size: 11pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">INTEL SSDPELKX020T8 ( 2TB NVMe )</span></p></li><li dir="ltr" style="list-style-type: disc;font-size: 11pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre" aria-level="1"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;" role="presentation"><span style="font-size: 11pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">Dual  Broadcom BCM57416 NetXtreme-E 10GBase-T</span></p></li><li dir="ltr" style="list-style-type: disc;font-size: 11pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre" aria-level="1"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:12pt;" role="presentation"><span style="font-size: 11pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">512GB of RAM</span></p></li></ul><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">We have a fairly complex nginx config with sharded caches as explained in https://www.nginx.com/blog/shared-caches-nginx-plus-cache-clusters-part-1/</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">We see this problem on :</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">nginx version: nginx/1.19.6</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">built by gcc 8.3.0 (Debian 8.3.0-6)</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">built with OpenSSL 1.1.1d  10 Sep 2019</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">TLS SNI support enabled</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">configure arguments: --add-module=/root/incubator-pagespeed-ngx-latest-stable --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_ssl_module --with-http_stub_status_module --with-pcre-jit --with-http_secure_link_module --with-http_v2_module --with-http_realip_module --with-stream_geoip_module --http-scgi-temp-path=/tmp --http-uwsgi-temp-path=/tmp --http-fastcgi-temp-path=/tmp --http-proxy-temp-path=/tmp --http-log-path=/var/log/nginx/access --error-log-path=/var/log/nginx/error --pid-path=/var/run/nginx.pid --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/sbin --prefix=/usr --with-threads</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">Pagespeed is our only third party module and it is version 1.13.35.2-0</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">Some nginx process start to spin in a tight loop, strace shows:</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">write(168, "H\0\0\0\0\0\0\0 W|\244\230U\0\0@y\20\244\230U\0\0", 24) = -1 EAGAIN (Resource temporarily unavailable)</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">write(168, "H\0\0\0\0\0\0\0 W|\244\230U\0\0@y\20\244\230U\0\0", 24) = -1 EAGAIN (Resource temporarily unavailable)</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">write(168, "H\0\0\0\0\0\0\0 W|\244\230U\0\0@y\20\244\230U\0\0", 24) = -1 EAGAIN (Resource temporarily unavailable)</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">write(168, "H\0\0\0\0\0\0\0 W|\244\230U\0\0@y\20\244\230U\0\0", 24) = -1 EAGAIN (Resource temporarily unavailable)</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">write(168, "H\0\0\0\0\0\0\0 W|\244\230U\0\0@y\20\244\230U\0\0", 24) = -1 EAGAIN (Resource temporarily unavailable)</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">write(168, "H\0\0\0\0\0\0\0 W|\244\230U\0\0@y\20\244\230U\0\0", 24) = -1 EAGAIN (Resource temporarily unavailable)</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">looking in /proc </span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">root@ao3-front08:/proc/799697/fd# ls -l 168</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">l-wx------ 1 nginx nginx 64 Jan 18 22:05 168 -> 'pipe:[2914414548]'</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">root@ao3-front08:/proc# grep 2914414548 /tmp/fds</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">lr-x------ 1 nginx nginx 64 Jan 18 22:05 799697/fd/167 -> pipe:[2914414548]</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;font-family: "Courier New";background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">l-wx------ 1 nginx nginx 64 Jan 18 22:05 799697/fd/168 -> pipe:[2914414548]</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 10pt;background-color: transparent;font-variant-numeric: normal;font-variant-east-asian: normal;vertical-align: baseline;white-space: pre-wrap">The issue happens more when load is higher. Has anyone some advice as my current hack of killing processes that have used more than 1800 seconds of cpu is wrong.</span></p><br><br><br><br></div></div><img class="mailbird" width="1" height="1" style="border:0; width:1; height:1;" src="https://tracking.getmailbird.com/opentrackingpixel/?messageId=Mailbird-c68bc6af-8915-46b9-b617-7ecec7a556ea@catbus.co.uk&senderHash=F1787C035D24ABEC777BECB23A36785AF368C4F8FB78032263BDCDCCD4B79D98&recipientHash=D837831948A591CC70C565B04A45189CE16B0A1E9E43440A61E101C22B7912C1&internalId=d3c42f48-5ce6-40d5-950d-b42824bb059d">