High number connections-writing stuck

[Marcelo MD]

=)

That module passed all our tests, both in dev and staging environments. We
were deploying in batches and got it on the first one. We also rolled back
before any incidents.

Not perfect but pretty safe, I think.

Maybe we need to be more 'creative' with our testing.

On Tue, Jan 19, 2016 at 6:47 AM, B.R. <reallfqq-nginx at yahoo.fr> wrote:

> Testingin production?
> Well you are not the only one... far from that.
>
> There are other means to create 'realistic' traffic on benches, provided
> you are thinking those tests well, though.
> ---
> *B. R.*
>
> On Mon, Jan 18, 2016 at 7:33 PM, Marcelo MD <lists.md at gmail.com> wrote:
>
>> Hi,
>>
>> Just getting back to you. Sorry about the delay.
>>
>> This behaviour was caused by a custom patch in a module. The patch was
>> ending the requests without finalizing them, leaking connections.
>>
>> Eventually Nginx just exploded =) Nothing like production workload to
>> stress things out.
>>
>> Thanks a lot.
>>
>>
>>
>> On Sun, Dec 20, 2015 at 11:04 AM, Valentin V. Bartenev <vbart at nginx.com>
>> wrote:
>>
>>> On Friday 18 December 2015 15:55:47 Marcelo MD wrote:
>>> > Hi,
>>> >
>>> > Recently we added a 'thread_pool' directive to our main configuration.
>>> A
>>> > few hours later we saw a huge increase in the connections_writing stat
>>> as
>>> > reported by stub_status module. This number reached +- 3800 and is
>>> stuck
>>> > there since. The server in question is operating normally, but this is
>>> very
>>> > strange.
>>> >
>>> > Any hints on what this could be?
>>> >
>>> >
>>> > Some info:
>>> >
>>> > - Here is a graph of the stats reported, for a server with thread_pool
>>> and
>>> > another without: http://imgur.com/a/lF2EL
>>> >
>>> > - I don`t have older data anymore, but the jump from <100 to +- 3800
>>> > connections_writing happened in two sharp jumps. The first one
>>> following a
>>> > reload;
>>> >
>>> > - The machines' hardware and software are identical except for the
>>> > thread_pool directive in their nginx.conf. They live in two different
>>> data
>>> > centers;
>>> >
>>> > - Both machines are performing normally. Nothing unusual in CPU or RAM
>>> > usage. Nginx performance is about the same.
>>> >
>>> > - Reloading Nginx with 'nginx -s reload' does nothing. Restarting the
>>> > process brings connections_writing down.
>>> [..]
>>>
>>> As I understand from your message everything works well.  You should also
>>> check the error_log, if it doesn't have anything suspicious then there is
>>> nothing to worry about.
>>>
>>> The increased number of writing connections can be explained by increased
>>> concurrency.  Now nginx processing cycle doesn't block on disk and can
>>> accept more connections at the same time.  All the connections that were
>>> waiting in listen socket before are waiting now in thread pool.
>>>
>>>   wbr, Valentin V. Bartenev
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx at nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>>
>>
>>
>>
>> --
>> Marcelo Mallmann Dias
>>
>> _______________________________________________
>> nginx mailing list
>> nginx at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>



-- 
Marcelo Mallmann Dias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20160119/5154258e/attachment.html>