nginx high i/o

Stefanita Rares Dumitrescu katmai at
Sat Nov 21 16:19:16 MSK 2009

         fastcgi_max_temp_file_size 0;
This directive turns off fastcgi buffering according to the source code.

i did this, and the buffer errors are gone. however, the pages load very 
very slow. take about 5-10 seconds to load the page.

could it be my php-fpm.conf ?

the thing is why i am going nuts about this is because the same exact 
config (same nginx/php-fpm), i am running on a centos server, and i am 
able to push over 200 mb/s and everything works perfectly fine.

i switched over to freebsd to squeeze some more performance out of the 
server though.

also i have noticed that when more and more connections are established 
to the server, my php-cgi processes are frozen in "sbwait" state:

32514 nobody      1   4    0 60816K 14252K sbwait 4   0:12  0.00% php-cgi
32562 nobody      1   4    0 60816K 14212K sbwait 2   0:11  0.00% php-cgi
32577 nobody      1   4    0 60816K 14240K sbwait 6   0:11  0.00% php-cgi
32479 nobody      1   4    0 60816K 14244K sbwait 0   0:11  0.00% php-cgi
32475 nobody      1   4    0 60816K 14256K sbwait 0   0:11  0.00% php-cgi
32497 nobody      1   4    0 60816K 14228K sbwait 0   0:11  0.00% php-cgi
32527 nobody      1   4    0 60816K 14208K sbwait 1   0:11  0.00% php-cgi
32564 nobody      1   4    0 60816K 14192K sbwait 4   0:10  0.00% php-cgi

posted php-fpm.conf below.

<?xml version="1.0" ?>

         All relative paths in this config are relative to php's install 

         <section name="global_options">

                 Pid file

                 Error log file

                 Log level
                 <value name="log_level">notice</value>

                 When this amount of php processes exited with SIGSEGV 
or SIGBUS ...
                 <value name="emergency_restart_threshold">10</value>

                 ... in a less than this interval of time, a graceful 
restart will be initiated.
                 Useful to work around accidental curruptions in 
accelerator's shared memory.
                 <value name="emergency_restart_interval">1m</value>

                 Time limit on waiting child's reaction on signals from 
                 <value name="process_control_timeout">5s</value>

                 Set to 'no' to debug fpm
                 <value name="daemonize">yes</value>



                 <section name="pool">

                         Name of pool. Used in logs and stats.
                         <value name="name">nginx</value>

                         Address to accept fastcgi requests on.
                         Valid syntax is '' or just 
'port' or '/path/to/unix/socket'
                         <value name="listen_address"></value>

                         <value name="listen_options">

                                 Set listen(2) backlog
                                 <value name="backlog">-1</value>

                                 Set permissions for unix socket, if one 
                                 In Linux read/write permissions must be 
set in order to allow connections from web server.
                                 Many BSD-derrived systems allow 
connections regardless of permissions.
                                 <value name="owner"></value>
                                 <value name="group"></value>
                                 <value name="mode">0666</value>

                         Additional php.ini defines, specific to this 
pool of workers.
                         <value name="php_defines">
                 <!--            <value 
name="sendmail_path">/usr/sbin/sendmail -t -i</value>            -->
                 <!--            <value name="display_errors">0</value> 

                         Unix user of processes
                         <value name="user">nobody</value>

                         Unix group of processes
                         <value name="group">nobody</value>

                         Process manager settings
                         <value name="pm">

                                 Sets style of controling worker process 
                                 Valid values are 'static' and 'apache-like'
                                 <value name="style">static</value>

                                 Sets the limit on the number of 
simultaneous requests that will be served.
                                 Equivalent to Apache MaxClients directive.
                                 Equivalent to PHP_FCGI_CHILDREN 
environment in original php.fcgi
                                 Used with any pm_style.
                                 <value name="max_children">128</value>

                                 Settings group for 'apache-like' pm style
                                 <value name="apache_like">

                                         Sets the number of server 
processes created on startup.
                                         Used only when 'apache-like' 
pm_style is selected

                                         Sets the desired minimum number 
of idle server processes.
                                         Used only when 'apache-like' 
pm_style is selected

                                         Sets the desired maximum number 
of idle server processes.
                                         Used only when 'apache-like' 
pm_style is selected



                         The timeout (in seconds) for serving a single 
request after which the worker process will be terminated
                         Should be used when 'max_execution_time' ini 
option does not stop script execution for some reason
                         '0s' means 'off'
                         <value name="request_terminate_timeout">0s</value>

                         The timeout (in seconds) for serving of single 
request after which a php backtrace will be dumped to slow.log file
                         '0s' means 'off'
                         <value name="request_slowlog_timeout">0s</value>

                         The log file for slow requests
                         <value name="slowlog">logs/slow.log</value>

                         Set open file desc rlimit
                         <value name="rlimit_files">2048</value>

                         Set max core size rlimit
                         <value name="rlimit_core">0</value>

                         Chroot to this directory at the start, absolute 
                         <value name="chroot"></value>

                         Chdir to this directory at the start, absolute path
                         <value name="chdir"></value>

                         Redirect workers' stdout and stderr into main 
error log.
                         If not set, they will be redirected to 
/dev/null, according to FastCGI specs
                         <value name="catch_workers_output">yes</value>

                         How much requests each process should execute 
before respawn.
                         Useful to work around memory leaks in 3rd party 
                         For endless request processing please specify 0
                         Equivalent to PHP_FCGI_MAX_REQUESTS
                         <value name="max_requests">500</value>

                         Comma separated list of ipv4 addresses of 
FastCGI clients that allowed to connect.
                         Equivalent to FCGI_WEB_SERVER_ADDRS environment 
in original php.fcgi (5.2.2+)
                         Makes sense only with AF_INET listening socket.
                         <value name="allowed_clients"></value>

                         Pass environment variables like LD_LIBRARY_PATH
                         All $VARIABLEs are taken from current environment
                         <value name="environment">
                                 <value name="HOSTNAME">$HOSTNAME</value>
                                 <value name="TMP">/tmp</value>
                                 <value name="TMPDIR">/tmp</value>
                                 <value name="TEMP">/tmp</value>
                                 <value name="OSTYPE">$OSTYPE</value>
                                 <value name="MACHTYPE">$MACHTYPE</value>
                                 <value name="MALLOC_CHECK_">2</value>




Maxim Dounin wrote:
> Hello!
> On Sat, Nov 21, 2009 at 01:39:38AM +0100, Stefanita Rares Dumitrescu wrote:
>> actually i got a bunch of errors, in error log, as i enabled it to
>> get some idea on what's going on:
>> 2009/11/21 08:47:44 [warn] 24393#0: *57036 an upstream response is
>> buffered to a temporary file
>> /var/tmp/nginx/fastcgi_temp/1/56/0024741561 while reading upstream,
>> client:
>> i am so sleepy right now, could you please tell me where do i get
>> the fastcgi buffers optimizations ? i searched on the wiki but i
>> can't think straight.
> Start from tuning fastcgi_buffers:
> If you haven't had enough memory to deligate it for buffers you 
> may consider using fastcgi_max_temp_file_size 0; - this will 
> disable disk buffering at cost of not reading data from backend 
> connection (i.e. keeping backend busy).
> You may also consider investigating why your backend replies are 
> so big (fastcgi_buffers should be 64k in total on your platform, 
> this is much more than typical generated html...) and moving them 
> to nginx itself if possible (e.g. by using X-Accel-Redirect).
> Maxim Dounin
>> Maxim Dounin wrote:
>>> Hello!
>>> On Sat, Nov 21, 2009 at 12:21:52AM +0100, Stefanita Rares Dumitrescu wrote:
>>>> hi guys,
>>>> i have posted a question a while ago, and now i am coming back once more.
>>>> i have now a dual quad core xeon system, running freebsd 7.2 latest
>>>> patch level, 64 bit, and 16 gb ram. using the latest nginx-devel.
>>>> the OS runs on a ssd, the db is set on a normal drive, and the site
>>>> files reside on another ssd.
>>>> i had a pretty similar config just instead i was using centos.
>>>> the i/o on the main drive (os drive) is constantly at 22 mb/s.
>>>> i have disabled nginx logging, and everything has been setup from
>>>> the ground. i have no idea why do i get such high i/o, because i
>>>> have no other apps that could cause this load.
>>> Do you use proxy for big replies and/or have big client uploads?
>>> If yes, it's probably caused by disk buffering (you should have
>>> warnings in logs, but you have no logs...) and you probably want
>>> to tune your buffer sizes and/or proxy_temp_file_max_size.
>>> Maxim Dounin
>>>> could you please point me to some direction ? i don't know what to do.
>>>>             ad4              ad5              ad6
>>>>  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s
>>>> 54.56 428 22.79   0.00   0  0.00  32.89   4  0.14
>>>> 55.05 492 26.46   0.00   0  0.00  26.38   8  0.21
>>>> 55.24 452 24.40   0.00   0  0.00  36.36   5  0.20
>>>> 53.20 457 23.76  22.31  19  0.42  19.88   8  0.16
>>>> 53.33 464 24.15   0.00   0  0.00  48.47  23  1.11
>>>> 54.14 403 21.32   0.00   0  0.00  16.18 217  3.43
>>>> 52.75 406 20.90   0.00   0  0.00  17.49 178  3.04
>>>> 55.75 402 21.90   0.00   0  0.00  13.42  12  0.16

More information about the nginx mailing list