nginx crash

[Maxim Dounin]

Hello!

On Wed, Nov 23, 2011 at 08:21:35PM +0000, António P. P. Almeida wrote:

> On 23 Nov 2011 12h29 WET, mdounin at mdounin.ru wrote:
> 
> > Hello!
> >
> 
> >>
> >> I'm having a similar issue. The php-fpm workers stop responding and
> >> since the FCGI cache lifetime is 15s I get a bunch of SEGVs on the
> >> cache manager.
> >
> > By "lifetime" you mean "fastcgi_cache_path ... inactive=15s"?
> 
> Nope. I mean this:
> 
> fastcgi_cache_path /var/cache/nginx/microcache levels=1:2 keys_zone=microcache:10M max_size=1G inactive=2h loader_threshold=2592000000 loader_sleep=1 loader_files=100000;
> 
> ## For 200 and 301 make the cache valid for 30s.
> fastcgi_cache_valid 200 301 30s;
> 
> Sorry it's 30s not 15.

OK, this looks sane.  Cache valid time shouldn't be related to the 
problem (it may be, but likely indirectly).

> > If yes, than you may want to change your configuration: this isn't 
> > going to work well as nginx uses "inactive" as a guard against 
> > stale cache entry locks (left by crashed workers, if any).  It 
> > was never expected to be lower than time required to fetch 
> > a resource from an upstream, and setting it to lower expected to 
> > cause problems.
> >
> > If no, you may want to provide more details.  (Ideally I would 
> > like to see full debug log showing the crash, the "ignore long 
> > locked inactive cache entry" alert and some time before it, but I 
> > understand it would be hard to obtain unless you are able to 
> > reproduce the problem at will.)
> 
> I'll try to debug it, but it's not reproducible at will.

Could you please provide some more details about use pattern?  
E.g. how many requests per second?  How many unique cache keys per 
second?

With 10M cache zone it should be able to hold about 80k cache 
nodes.  How long it will take to consume them all?

It may be also good idea to show bracktraces from crashes you've 
observed.

> > Recently posted patch series[1] addresses major part of the 
> > problem: it resolves deadlock after such crashes.  It would be 
> > fine to resolve the crash itself too, but I wasn't yet able to 
> > trace the problem cause (unless it's caused by too low inactive 
> > time, which is mostly a configuration problem, see above).
> 
> Are you guys planning a new dev release or perhaps it's better for me
> to apply the patches and rebuild me debian package, since it will take
> a few days for a new dev release that incorporates those patches?

Next dev release is expected somewhere on next week.

Maxim Dounin