Cache doesn't expire as should

[Руслан Закиров]

On Tue, Dec 22, 2015 at 4:49 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
>
> On Tue, Dec 22, 2015 at 02:43:11PM +0300, Руслан Закиров wrote:
>
> [...]
>
> > Our error log has "ignore long locked inactive cache entry" alerts, but I
> > really couldn't match it to "defreeze" event. Access log has
> STALE/UPDATING
> > requests between the alert and EXPIRED (cache updating) request.
>
> The "ignore long locked inactive cache entry" alerts indicate that
> a cache entry was locked by some request, and wasn't unlocked for
> a long time.  The alert is expected to appear if a cache node is
> locked for cache inactive time (as set by proxy_cache_path
> inactive=, 10 minutes by default).


Inactive is defined in the config, but it's set to default 10m.

What happens with requests after this time? Do they hit backend and update
cache? Do they use stale version?

I'm going to check "long locked" messages in the log to see how many was
for "/" location.
The hash should be the same if we didn't change cache key, right?

Most likely reason is that a worker died or was killed
> while holding a lock on a cache node (i.e., while a request was
> waiting for a new response from a backend).
>

Shouldn't be there a record in error log? Error log level at warn.


>
> Trivial things to consider:
>
> - check logs for segmentation faults;
>

no seg faults in logs

- if you are using 3rd party modules / patches, try without them;
>

from freebsd port, updated gist [1] with `nginx -V` output


> - make sure you don't kill worker processes yourself or using some
>   automation scripts (in particular, don't try to terminate old
>   worker processes after a configuration reload).
>

One recent appearance of the problem was at 1:30AM and I checked
logs for crazy midnight deploys - nothing.

Also, we don't use anything custom to restart nginx, just regular services
management tools.


[1] https://gist.github.com/ruz/05456767750715f6b54e


>
> --
> Maxim Dounin
> http://nginx.org/
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>



-- 
Руслан Закиров
Руководитель отдела разработки веб-сервисов
+7(916) 597-92-69, ruz @  <http://www.sports.ru/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20151223/fbaaf97d/attachment.html>