<div dir="ltr"><div dir="ltr">Hey all,<div> <br>We have a new set up running large amounts of data through a container nginx. This is crashing, without error. Forcing a reboot to recover daily at the moment. We are getting nothing from nginx logs or the docker logs of any use. Any suggestions to debugging this?</div><div><br></div><div><div style="display:inline-block;padding:4px"><span style="font-family:Consolas,monospace;font-size:12pt;white-space:pre;background-color:rgb(12,12,12);color:rgb(204,204,204)">CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                    PORTS                          NAMES<br>9bc827d6ccd7        nginx:stable</span></div><br></div>Container is running on ubuntu 18.04.3 LTS OS.<br>The last thing we see in the logs is<div><font color="#000000" face="Consolas, monospace"><span style="font-size:16px;white-space:pre;background-color:rgb(243,243,243)"><br></span></font></div><div><div style="display:inline-block;white-space:pre;background-color:rgb(12,12,12);font-family:Consolas,monospace;font-size:12pt;padding:4px"><span style="color:rgb(204,204,204)"> 18:15:36 [alert] 7#7: ignore long locked inactive cache entry 9f78089258be73e98f58abed986ddb8b, count:1<br>2019/11/13 18:25:36 [alert] 7#7: ignore long locked inactive cache entry 9f78089258be73e98f58abed986ddb8b, count:1<br>2019/11/13 18:35:36 [alert] 7#7: ignore long locked inactive cache entry 9f78089258be73e98f58abed986ddb8b, count:1</span></div></div><br>We're tried a lot of things such as down grading the nginx version as we were using latest but this just prolonged the crash time. Small improvement by a day<br>Any suggestions welcome</div><div dir="ltr"><br></div><div>Nginx conf is below<br><br>Windows Terminal<div style="display:inline-block;white-space:pre;background-color:rgb(12,12,12);font-family:Consolas,monospace;font-size:12pt;padding:4px"><span style="color:rgb(204,204,204)">worker_rlimit_nofile 30000;<br>events {}<br>http {<br>    log_format compression '$remote_addr - $remote_user [$time_local] '<br>                           '"$request" $status $body_bytes_sent '<br>                           '"$http_referer" "$http_user_agent" "$gzip_ratio"';<br>     error_log /etc/nginx/error_log.log warn;<br>    client_max_body_size 20m;<br>    server_names_hash_bucket_size 512;<br>    proxy_headers_hash_bucket_size 128;<br>    proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=content_cache:10m max_size=10g use_temp_path=off;<br>    upstream hub_node {<br>        server hub-node:3000;<br>        keepalive 16;<br>    }<br>    upstream hub_cms {<br>        server hub-be:80;<br>        keepalive 16;<br>    }<br>    upstream hub_analytics {<br>        server hub-matomo:80;<br>        keepalive 16;<br>    }<br></span><span style="color:rgb(0,0,0);background-color:rgb(0,0,0)"><br></span><span style="color:rgb(204,204,204)">    server {<br>        listen       443  default_server;<br>        server_name  _;<br>        return       418;<br>    }<br></span><span style="color:rgb(0,0,0);background-color:rgb(0,0,0)"><br></span><span style="color:rgb(204,204,204)">    server {<br>        listen 443 ssl http2;<br>        server_name <a href="http://digital-hub.bwi.dpn.gov.uk">digital-hub.bwi.dpn.gov.uk</a>;<br>        add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";<br>        location /sites/default/files/ {<br>                proxy_set_header   Host              $host;<br>                proxy_set_header   X-Real-IP         $remote_addr;<br>                proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;<br>                proxy_set_header   X-Forwarded-Proto $scheme;<br>                proxy_set_header X-Forwarded-For $remote_addr;<br>                proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;<br>                proxy_cache_valid 200 302 10m;<br>                proxy_cache content_cache;<br>                proxy_pass <a href="http://hub_cms/sites/default/files/">http://hub_cms/sites/default/files/</a>;<br></span><span style="color:rgb(0,0,0);background-color:rgb(0,0,0)"><br></span><span style="color:rgb(204,204,204)">        }<br>        location / {<br>                access_log /var/log/nginx/access.log compression buffer=32k;<br>                proxy_pass <a href="http://hub_node/">http://hub_node/</a>;<br>        }<br>    }<br></span><span style="color:rgb(0,0,0);background-color:rgb(0,0,0)"><br></span><span style="color:rgb(204,204,204)">    server {<br>        listen 443 ssl http2;<br>        server_name <a href="http://analytics.digital-hub.bwi.dpn.gov.uk">analytics.digital-hub.bwi.dpn.gov.uk</a>;<br>        add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";<br>        location / {<br>            proxy_set_header Host              $host;<br>            proxy_set_header X-Real-IP         $remote_addr;<br>            proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;<br>            proxy_set_header X-Forwarded-Proto $scheme;<br>            proxy_pass <a href="http://hub_analytics/">http://hub_analytics/</a>;<br>        }<br>    }<br></span><span style="color:rgb(0,0,0);background-color:rgb(0,0,0)"><br></span><span style="color:rgb(204,204,204)">    server {<br>        listen 443 ssl http2;<br>        server_name <a href="http://content.digital-hub.bwi.dpn.gov.uk">content.digital-hub.bwi.dpn.gov.uk</a>;<br>        add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";<br>        location / {<br>            proxy_set_header Host              $host;<br>            proxy_set_header X-Real-IP         $remote_addr;<br>            proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;<br>            proxy_set_header X-Forwarded-Proto $scheme;<br>            proxy_set_header X-Forwarded-For $remote_addr;<br>            proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;<br>            proxy_cache_valid 200 302 10m;<br>            proxy_cache content_cache;<br>            proxy_pass <a href="http://hub_cms/">http://hub_cms/</a>;<br>        }<br>    }<br></span><span style="color:rgb(0,0,0);background-color:rgb(0,0,0)"><br></span><span style="color:rgb(204,204,204)">    ssl_certificate /etc/letsencrypt/live/localhost/san.digital-hub.crt;<br>    ssl_certificate_key /etc/letsencrypt/live/localhost/san.digital-hub.rsa;<br>    ssl_session_timeout 1d;<br>    ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions<br>    ssl_session_tickets off;<br>    ssl_protocols TLSv1.2 TLSv1.3;<br>    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-<br>SHA256:DHE-RSA-AES256-GCM-SHA384;<br>    ssl_prefer_server_ciphers off;<br>    # HSTS (ngx_http_headers_module is required) (63072000 seconds)<br>    add_header Strict-Transport-Security "max-age=63072000" always;</span></div><br></div></div>