Hi Maxim,

Sorry for my late reply.

We use Nginx as CDN Cache. And it is normal case that the responses are 
different from different origin servers. For instance: the origin server could be 
the customer server or some other CDN provider Cache server. The response 
header could be different.

In race condition, I found the inconsistency between disk file and metadata in
shared memory. 
When (c->uniq != of.uniq) happens, I saw " too long header" in error log.

To fix this, I picked the ngx_pagesize to initiate c->body_start again. then the "too long header" 
is gone from error log.

I will try your patches next week. Thanks!


On Wed, Nov 28, 2018 at 4:45 AM Maxim Dounin <mdounin@mdounin.ru> wrote:
Hello!

On Mon, Nov 26, 2018 at 06:11:33PM -0800, Yongguang Bai wrote:

> # HG changeset patch
> # User Yongguang Bai <yongguang.bai@quantil.com>
> # Date 1543276278 28800
> #      Mon Nov 26 15:51:18 2018 -0800
> # Node ID 4d48e1f3ebf6f0634e8e2cbd057d9fcfbea9d988
> # Parent  a7ff19afbb14795fef14f599a304d0ad21052b70
> Mail: nginx_error:cache file has too long header
>
> This error is printed when cached file is expired and the
> response is changed in proxied server, and Nginx is under
> heavy load.
>
> diff -r a7ff19afbb14 -r 4d48e1f3ebf6 src/http/ngx_http_file_cache.c
> --- a/src/http/ngx_http_file_cache.c  Mon Nov 26 18:29:56 2018 +0300
> +++ b/src/http/ngx_http_file_cache.c  Mon Nov 26 15:51:18 2018 -0800
> @@ -376,6 +376,10 @@
>      ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
>                     "http file cache fd: %d", of.fd);

> +    if(c->uniq != of.uniq){
> +        c->body_start = ngx_pagesize;
> +    }
> +
>      c->file.fd = of.fd;
>      c->file.log = r->connection->log;
>      c->uniq = of.uniq;

This change looks wrong.  In particular, ngx_pagesize, which is
usually 4k, can be smaller than existing c->body_start, and this
change will make things worse instead of fixing anything.

Also, could you please elaborate what exactly you are trying to
fix?

There is a known race which may lead to "cache file has too long
header" error - and, actually the error was added to ensure that
this race is properly handled if happens, see here:

http://hg.nginx.org/nginx/rev/6f97afc238de
http://mailman.nginx.org/pipermail/nginx-devel/2011-September/001287.html

Note that this race can only result in problems if for some reason
the backend returns responses with different headers nearly at the
same time.  This may indicate that there is something wrong with
the backend.

Also, there is a known bug with multiple variants which also can
lead to the same error.  The patch can be found here:

http://mailman.nginx.org/pipermail/nginx-devel/2018-January/010774.html

If you are hitting this bug, please test if the patch works for
you.

--
Maxim Dounin
http://mdounin.ru/


--

Best Regards,

Yongguang Bai

Engineering Manager

Quantil

Connecting users with content...it's that simple.

Direct: +1-408-421-6265
Office: +1-888-847-9851
 

Tweeter  Google Plus  Linked In