Unexpected behaviour "aio threads" option

[Bart Warmerdam]

A test with debug logging is scheduled for upcoming Monday. I hope to be able to deliver the logs by then. 

Thanks, 

B.


On November 24, 2015 4:32:11 PM GMT+01:00, "Valentin V. Bartenev" <vbart at nginx.com> wrote:
>On Tuesday 24 November 2015 15:10:12 Bart Warmerdam wrote:
>> 
>> Hello,
>> 
>> On a system with a load of about 500-600 URI/sec I see some
>unexpected
>> behaviour when using "aio threads" option in the configuration.
>> 
>> System setup:
>> The system runs on RHEL6.6 with 3 workers running nginx 1.9.6 with
>> thread support. Content is cached and populated by a
>proxied-upstream.
>> The cache location is a tmpfs file system with more then enough space
>> at all times. Proxy buffer size 8k. The output buffer is default (no
>> config item, so 2 32k). Keepalive timeout 75s. Sendfile is enabled.
>> 
>> Seen behaviour:
>> On the WAF in front of this system I see occasional hangs on
>resources
>> (mainly larger files like js, jpeg, ..). Seen in the WAF log is that
>> this WAF waits for the transfer to be completed until nginx closes
>the
>> connection at the keepalive time of 75s. In the nginx access.log I
>see
>> the entry served from cache (upstream server '-') with the correct
>> content length. In the tcp dump I see the response of this call to
>> contain a content-length header with the correct length, a server
>time
>> header over 1 minute older then the tcpdump timestamp (all servers
>are
>> ntp-connected). The served jpeg is half-way in its cache lifetime at
>> that time and there are previous served entries from cache without
>> incomplete transfers. In the tcp dump the jpeg file starts to differ
>> from the original after 32168 bytes and misses 8192 bytes after which
>> the remaining content is served (which is identical to original).
>From
>> the tcpdump I can extract the file which is missing 8192 bytes.
>> 
>> We have also a dump when during the proxied call this same behaviour
>> was seen. The upstream call is started to get a jpeg from the origin.
>> After a few packets the data is sent to the WAF. The complete
>upstream
>> file is retrieved (can be validated in the tcpdump that the jpeg is
>> complete and correctly retrieved), but not all the data is sent to
>the
>> listening socket to the WAF.
>> 
>> 
>> If I change the setup to "aio on" or "aio off" this behaviour is not
>> seen. This is the only change in the configuration between the tests.
>> It looks like this behaviour only affects bigger files. I have not
>seen
>> this effect on small files or proxied responses.
>> 
>> 
>> Does anyone have the same experience with this option. And what is
>the
>> best way to proceed in tracing this?
>> 
>[..]
>
>Could you provide the debug log?
>http://nginx.org/en/docs/debugging_log.html
>
>  wbr, Valentin V. Bartenev
>
>_______________________________________________
>nginx-devel mailing list
>nginx-devel at nginx.org
>http://mailman.nginx.org/mailman/listinfo/nginx-devel

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151127/2e7306ec/attachment.html>