<div dir="ltr">That is exactly the issue. Seeing what the "reload" did to the memory (starting a new worker process) was the culprit. I was thinking the configuration reload should just refresh what's in memory, but it clearly doubles the requirement of memory and must wait until the previous child can stop gracefully.<div><br></div><div>Thank you for the help!</div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jul 27, 2018 at 10:56 AM Shaun Tarves <<a href="mailto:shaun.tarves@jackpinetech.com">shaun.tarves@jackpinetech.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Here are the relevant parts of our configuration:</div>
<span></span><div><br></div>
worker_processes 1;<br>pid /var/run/nginx.pid;<br>events {<br> worker_connections 512;<br>}<br>http {<br> server {<br> listen xx.xx.xx.xx:443 default_server ssl;<br> ssl on;<br> ssl_certificate /opt/xxx.pem;<br> ssl_certificate_key /opt/xxx.key<br> ssl_ciphers 'AES128+EECDH:AES128+EDH:!aNULL';<br> ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br> ssl_session_cache shared:SSL:10m;<br> ssl_prefer_server_ciphers on;<br> ssl_verify_client optional;<br> ssl_client_certificate /opt/ca.crt.pem<br> ssl_crl /opt/ca.crl/.pem;<br> }<br>}<div><br></div><div>During a "reload" command, here is how our ps looks:</div><div><br></div><div>
<span></span>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">[root@www nginx]# service nginx reload</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">Reloading nginx: <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>[</span><span class="m_-9159489464844070864gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(52,188,38)"><span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>OK<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span></span><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">]</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">[root@www nginx]# ps -ef | grep nginx</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">root<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9605 <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>1<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9 15:06 ?<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:17 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">cons3rt <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9606<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9605<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>0 15:06 ?<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:00 nginx: worker process<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span></span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">root <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>11009 27847<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>0 15:09 pts/2<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:00 grep nginx</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">[root@www nginx]# ps -ef | grep nginx</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">root<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9605 <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>1 10 15:06 ?<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:24 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">cons3rt <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9606<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9605<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>0 15:06 ?<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:00 nginx: worker process is shutting down <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span></span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">root <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>11091 27847<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>0 15:10 pts/2<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:00 grep nginx</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">[root@www nginx]# ps -ef | grep nginx</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">root<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9605 <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>1 10 15:06 ?<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:24 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">cons3rt <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9606<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9605<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>0 15:06 ?<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:00 nginx: worker process is shutting down <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span></span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">root <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>11362 27847<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>0 15:10 pts/2<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:00 grep nginx</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">[root@www nginx]# ps -ef | grep nginx</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">root<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9605 <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>1<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9 15:06 ?<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:24 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">cons3rt <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9606<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9605<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>1 15:06 ?<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:02 nginx: worker process is shutting down <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span></span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">root <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>11395 27847<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>0 15:10 pts/2<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:00 grep nginx</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">[root@www nginx]# vi /var/log/nginx/error.log</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">[root@www nginx]# ps -ef | grep nginx</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">root<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9605 <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>1<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>7 15:06 ?<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:24 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">cons3rt <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9606<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>9605<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>5 15:06 ?<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:19 nginx: worker process is shutting down <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span></span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">root <span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>11771 27847<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>0 15:12 pts/2<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>00:00:00 grep nginx</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">[root@www nginx]# service nginx stop</span></p>
<p class="m_-9159489464844070864gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">Stopping nginx:<span class="m_-9159489464844070864gmail-Apple-converted-space"> </span>[</span><span class="m_-9159489464844070864gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">FAILED</span><span class="m_-9159489464844070864gmail-s1" style="font-variant-ligatures:no-common-ligatures">]</span></p>
<br></div><div><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Thu, Jul 26, 2018 at 4:16 PM Shaun Tarves <<a href="mailto:shaun.tarves@jackpinetech.com" target="_blank">shaun.tarves@jackpinetech.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span style="font-size:small;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Hi,</span><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">We are trying to use nginx to support the DoD PKI infrastructure, which includes many DoD and contractor CRLs. The combined CRL file is over 350MB in size, which seems to crash nginx during a reload (at least on Red Hat 6). Our cert/key/crl set up is valid and working, and when only including a subset of the CRL files we have, reloads work fine.</div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">When we concatenate all the CRLs we need to support, the config reload request causes worker threads to become defunct and messages in the error log indicate the following:</div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial"><span></span><p class="m_-9159489464844070864m_-3091859800804134833gmail-m_2419373206062243596gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864m_-3091859800804134833gmail-m_2419373206062243596gmail-s1" style="font-variant-ligatures:no-common-ligatures">2018/07/26 16:05:25 [alert] 30624#30624: fork() failed while spawning "worker process" (12: Cannot allocate memory)</span></p><p class="m_-9159489464844070864m_-3091859800804134833gmail-m_2419373206062243596gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864m_-3091859800804134833gmail-m_2419373206062243596gmail-s1" style="font-variant-ligatures:no-common-ligatures">2018/07/26 16:05:25 [alert] 30624#30624: sendmsg() failed (9: Bad file descriptor)</span></p><p class="m_-9159489464844070864m_-3091859800804134833gmail-m_2419373206062243596gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span class="m_-9159489464844070864m_-3091859800804134833gmail-m_2419373206062243596gmail-s1" style="font-variant-ligatures:no-common-ligatures">2018/07/26 16:08:42 [alert] 30624#30624: worker process 1611 exited on signal 9</span></p><br></div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">Is there any way we can get nginx to support such a large volume of CRLs?</div><br></div>
</blockquote></div></div></div>
</blockquote></div>