Uneven High Load on the Nginx Server

[Maxim Dounin]

Hello!

On Tue, Sep 27, 2016 at 01:09:41PM -0400, anish10dec wrote:

> We are having two Nginx Server acting as Caching Server behind haproxy
> loadbalancer. We are observing a high load on one of the server though we
> see equal number of requests coming on the server from application per sec.
> 
> We see that out of two server on which load is high i.e around 5 , response
> time /latency is high in delivering the content . On same server attached
> stats module screenshot shows more number of requests in "Writing" as
> comapred to other one on which load is 0.5 and response time /latency is
> also low .
> 
> Please help what might be causing high load and high number of writing on
> one of server.
> 
> Active connections: 8619 
> server accepts handled requests
> 33204889 33204889 38066647 
> Reading: 0 Writing: 755 Waiting: 7863
> 
> 
> Active connections: 10959 
> server accepts handled requests
> 34625312 34625312 39974933 
> Reading: 0 Writing: 3700 Waiting: 7259

When nginx is reading request headers from a connection, it will 
be counted as "reading".  Once the request header is completely 
read, the connection will be counted as "writing" till the request 
is complete.

That is, there are lots of various factors which can affect number 
of writing connections, in particular:

- a cache is not working for some reason, causing more requests 
  (compared to your 2nd server) to be passed to upstream;

- the upstream server configured is slow;

- the disk subsystem is slow/overloaded and can't cope with load, 
  causing nginx to spend more time reading cached responses from 
  disk.

Given that you also see load 5 and assuming you are using Linux, 
I would suggests that the problem is disk subsystem, as processes 
waiting for disk are counted by Linux in the load average.  You 
may want to take a closer look at your disk subsystem.

-- 
Maxim Dounin
http://nginx.org/