[PATCH] Mail: nginx_error:cache file has too long header

[Yongguang Bai]

Hi Maxim,

Sorry for my late reply.

We use Nginx as CDN Cache. And it is normal case that the responses are
different from different origin servers. For instance: the origin server
could be
the customer server or some other CDN provider Cache server. The response
header could be different.

In race condition, I found the inconsistency between disk file and metadata
in
shared memory.
When (c->uniq != of.uniq) happens, I saw " too long header" in error log.

To fix this, I picked the ngx_pagesize to initiate c->body_start again.
then the "too long header"
is gone from error log.

I will try your patches next week. Thanks!


On Wed, Nov 28, 2018 at 4:45 AM Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
>
> On Mon, Nov 26, 2018 at 06:11:33PM -0800, Yongguang Bai wrote:
>
> > # HG changeset patch
> > # User Yongguang Bai <yongguang.bai at quantil.com>
> > # Date 1543276278 28800
> > #      Mon Nov 26 15:51:18 2018 -0800
> > # Node ID 4d48e1f3ebf6f0634e8e2cbd057d9fcfbea9d988
> > # Parent  a7ff19afbb14795fef14f599a304d0ad21052b70
> > Mail: nginx_error:cache file has too long header
> >
> > This error is printed when cached file is expired and the
> > response is changed in proxied server, and Nginx is under
> > heavy load.
> >
> > diff -r a7ff19afbb14 -r 4d48e1f3ebf6 src/http/ngx_http_file_cache.c
> > --- a/src/http/ngx_http_file_cache.c  Mon Nov 26 18:29:56 2018 +0300
> > +++ b/src/http/ngx_http_file_cache.c  Mon Nov 26 15:51:18 2018 -0800
> > @@ -376,6 +376,10 @@
> >      ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
> >                     "http file cache fd: %d", of.fd);
> >
> > +    if(c->uniq != of.uniq){
> > +        c->body_start = ngx_pagesize;
> > +    }
> > +
> >      c->file.fd = of.fd;
> >      c->file.log = r->connection->log;
> >      c->uniq = of.uniq;
>
> This change looks wrong.  In particular, ngx_pagesize, which is
> usually 4k, can be smaller than existing c->body_start, and this
> change will make things worse instead of fixing anything.
>
> Also, could you please elaborate what exactly you are trying to
> fix?
>
> There is a known race which may lead to "cache file has too long
> header" error - and, actually the error was added to ensure that
> this race is properly handled if happens, see here:
>
> http://hg.nginx.org/nginx/rev/6f97afc238de
> http://mailman.nginx.org/pipermail/nginx-devel/2011-September/001287.html
>
> Note that this race can only result in problems if for some reason
> the backend returns responses with different headers nearly at the
> same time.  This may indicate that there is something wrong with
> the backend.
>
> Also, there is a known bug with multiple variants which also can
> lead to the same error.  The patch can be found here:
>
> http://mailman.nginx.org/pipermail/nginx-devel/2018-January/010774.html
>
> If you are hitting this bug, please test if the patch works for
> you.
>
> --
> Maxim Dounin
> http://mdounin.ru/
>


-- 

Best Regards,

*Yongguang Bai*

Engineering Manager
[image: Quantil]

Connecting users with content...it's that simple.

Direct: +1-408-421-6265
Office: +1-888-847-9851

[image: Tweeter] <https://twitter.com/Team_Quantil>  [image: Google Plus]
<https://plus.google.com/+Quantil_team/>  [image: Linked In]
<https://www.linkedin.com/company/quantil>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20181130/2b6ee3a1/attachment.html>