Large CRL file crashing nginx on reload

[Shaun Tarves]

That is exactly the issue. Seeing what the "reload" did to the memory
(starting a new worker process) was the culprit. I was thinking the
configuration reload should just refresh what's in memory, but it clearly
doubles the requirement of memory and must wait until the previous child
can stop gracefully.

Thank you for the help!

On Fri, Jul 27, 2018 at 10:56 AM Shaun Tarves <shaun.tarves at jackpinetech.com>
wrote:

> Here are the relevant parts of our configuration:
>
> worker_processes  1;
> pid        /var/run/nginx.pid;
> events {
>     worker_connections  512;
> }
> http {
>   server {
>     listen xx.xx.xx.xx:443 default_server ssl;
>     ssl on;
>     ssl_certificate /opt/xxx.pem;
>     ssl_certificate_key /opt/xxx.key
>     ssl_ciphers 'AES128+EECDH:AES128+EDH:!aNULL';
>     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
>     ssl_session_cache shared:SSL:10m;
>     ssl_prefer_server_ciphers on;
>     ssl_verify_client optional;
>     ssl_client_certificate /opt/ca.crt.pem
>     ssl_crl /opt/ca.crl/.pem;
>   }
> }
>
> During a "reload" command, here is how our ps looks:
>
> [root at www nginx]# service nginx reload
>
> Reloading nginx:                                           [  OK  ]
>
> [root at www nginx]# ps -ef | grep nginx
>
> root      9605     1  9 15:06 ?        00:00:17 nginx: master process
> /usr/sbin/nginx -c /etc/nginx/nginx.conf
>
> cons3rt   9606  9605  0 15:06 ?        00:00:00 nginx: worker process
>
>
> root     11009 27847  0 15:09 pts/2    00:00:00 grep nginx
>
> [root at www nginx]# ps -ef | grep nginx
>
> root      9605     1 10 15:06 ?        00:00:24 nginx: master process
> /usr/sbin/nginx -c /etc/nginx/nginx.conf
>
> cons3rt   9606  9605  0 15:06 ?        00:00:00 nginx: worker process is
> shutting down
>
> root     11091 27847  0 15:10 pts/2    00:00:00 grep nginx
>
> [root at www nginx]# ps -ef | grep nginx
>
> root      9605     1 10 15:06 ?        00:00:24 nginx: master process
> /usr/sbin/nginx -c /etc/nginx/nginx.conf
>
> cons3rt   9606  9605  0 15:06 ?        00:00:00 nginx: worker process is
> shutting down
>
> root     11362 27847  0 15:10 pts/2    00:00:00 grep nginx
>
> [root at www nginx]# ps -ef | grep nginx
>
> root      9605     1  9 15:06 ?        00:00:24 nginx: master process
> /usr/sbin/nginx -c /etc/nginx/nginx.conf
>
> cons3rt   9606  9605  1 15:06 ?        00:00:02 nginx: worker process is
> shutting down
>
> root     11395 27847  0 15:10 pts/2    00:00:00 grep nginx
>
> [root at www nginx]# vi /var/log/nginx/error.log
>
> [root at www nginx]# ps -ef | grep nginx
>
> root      9605     1  7 15:06 ?        00:00:24 nginx: master process
> /usr/sbin/nginx -c /etc/nginx/nginx.conf
>
> cons3rt   9606  9605  5 15:06 ?        00:00:19 nginx: worker process is
> shutting down
>
> root     11771 27847  0 15:12 pts/2    00:00:00 grep nginx
>
> [root at www nginx]# service nginx stop
>
> Stopping nginx:                                            [FAILED]
>
>
>
> On Thu, Jul 26, 2018 at 4:16 PM Shaun Tarves <
> shaun.tarves at jackpinetech.com> wrote:
>
>> Hi,
>>
>> We are trying to use nginx to support the DoD PKI infrastructure, which
>> includes many DoD and contractor CRLs. The combined CRL file is over 350MB
>> in size, which seems to crash nginx during a reload (at least on Red Hat
>> 6). Our cert/key/crl set up is valid and working, and when only including a
>> subset of the CRL files we have, reloads work fine.
>>
>> When we concatenate all the CRLs we need to support, the config reload
>> request causes worker threads to become defunct and messages in the error
>> log indicate the following:
>>
>> 2018/07/26 16:05:25 [alert] 30624#30624: fork() failed while spawning
>> "worker process" (12: Cannot allocate memory)
>>
>> 2018/07/26 16:05:25 [alert] 30624#30624: sendmsg() failed (9: Bad file
>> descriptor)
>>
>> 2018/07/26 16:08:42 [alert] 30624#30624: worker process 1611 exited on
>> signal 9
>>
>> Is there any way we can get nginx to support such a large volume of CRLs?
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20180727/208dc076/attachment-0001.html>