Possible cached file corruption with aio_write enabled

[vedranf]

Hello,

I've recently upgraded one of the nginx servers within a caching
(proxy_cache module) cluster from 1.8.1 to 1.10 and soon after I noticed
unusually high number of various errors only on that server which I
eventually pin pointed to a mismatch between the actual cached file size on
disk and size reported in file metadata (either content-length or something
else). Apparently, cached files on 1.10 are cut short and miss up to few
hundred kilobytes at their ends (for files over 100 MB in total size). For a
few files I checked, the larger file was, more content it missed at the end.
Eventually I found the culprit to be aio_write which I enabled the same time
I upgraded nginx. Disabling it and removing all already cached files
resolved the problem.

Relevant directives:

    thread_pool default threads=4 max_queue=65536;
    sendfile on;
    aio            threads=default;
    aio_write   on;
    # output_buffers are used if sendfile is not used
    # output_buffers 8 512k;
    read_ahead  1;

    proxy_cache_path  /cache  ...  use_temp_path=on;
    proxy_buffering                 on;
    proxy_buffers           32 64k;
    proxy_busy_buffers_size         128k;
    proxy_temp_file_write_size      128k;

OS is Linux 4.1.

Thanks,
Vedran

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,267142,267142#msg-267142