NGINX Worker process stuck, CPU usage at 100%

[Ruslan Ermilov]

On Fri, Mar 06, 2015 at 09:59:47PM +0000, Kazlman, David wrote:
> I've migrated my server over from lighttpd to NGINX(memory leaks were causing cache issues which invoked OOM Killer in Linux). It seems that after a while of running(about 30 minutes) with NGINX  and processing requests just fine the NGINX worker process gets stuck in a loop maxing out the CPU and not responding to any other requests.  I am running NGINX version 1.6.2,  OpenSSL 1.2.0,
> PHP-FPM.
> PHP 5.5.19 (fpm-fcgi) (built: Mar  5 2015 10:12:12)
> Copyright (c) 1997-2014 The PHP Group
> Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
> 
> This is a very minimal use of the webserver (1 user, at most 2 requests from the browser every 2 seconds).
> 
> I turned on debugging in nginx and captured a log.  The worker threads seems to be hung near the bottom of the log, but I can't pick anything out of the log that indicates a problem.  I don't see any errors.  I attached the debug log, can someone please help look it over and see if anything stands out?
> 
> I've noticed sometimes when this occurs the log is reporting:
> 
> 2015/03/05 20:13:34 [info] 698#0: *1348 peer closed connection in SSL handshake while SSL handshaking, client: 192.168.0.126, server: 0.0.0.0:443
> 
> But this isn't always the case.  I can repeat this hang after about 30 minutes pretty consistently.  I've tried 3 different versions of openSSL (1.0.0, 1.0.1, 1.0.2) and 2 versions of NGINX (1.6.2, 1.7.10) and nothing seems to resolve the issue so it may be pointing to a configuration problem.  I have also tried unix sockets vs tcp sockets as the fastcgi transfer mechanism (listen = /var/run/php-fpm.sock and listen = 127.0.0.1:9000) and both seem to act similarly
> 
> 
> If I kill the webserver and restart it, the webserver starts acting on requests just fine again, and after some amount of time gets back into this state. My configuration is as follows:
> 
> nginx.conf:
> 
> #user  nobody;
> worker_processes  1;
> 
> #error_log  logs/error.log;
> #error_log  logs/error.log  notice;
> error_log   /var/log/nginx/error.log  debug;
> 
> #pid        logs/nginx.pid;
> 
> 
> events {
>     worker_connections  1024;
> }
> 
> 
> http {
>     include       mime.types;
>     default_type  application/octet-stream;
> 
>     sendfile        on;
>     keepalive_timeout  65;
> 
>     client_max_body_size 30M;
> 
>     # Redirect HTTP Requests to HTTPS
>     #
>     server {
>        listen         80;
>        server_name    localhost;
>        return         301 https://$host;
>                 }
> 
>     # HTTPS server
>     #
>     server {
>         listen       443 ssl;
>         server_name  $host;
> 
>         ssl on;
>         ssl_certificate      /mnt/emmc/ssl/nginx.crt;
>         ssl_certificate_key  /mnt/emmc/ssl/nginx.key;
> 
>                   root /var/www/htdocs/;
> 
>                                   location / {
>                                                 try_files $uri $uri/ /index.php;
>                                                 index index.php;
>                                   }
> 
>         location ~ \.php$ {
>                                 fastcgi_split_path_info ^(.+\.php)(/.+)$;
>             fastcgi_pass  unix:/var/run/php-fpm/sock;
>             fastcgi_index index.php;
>             include fastcgi_params;
>             fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
>             include /etc/nginx/fastcgi.conf;
>         }
>     }
> 
> }

[...]

http://wiki.nginx.org/Debugging#Asking_for_help