<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 22, 2015 at 4:49 PM, Maxim Dounin <span dir="ltr"><<a href="mailto:mdounin@mdounin.ru" target="_blank">mdounin@mdounin.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hello!<br>
<br>
On Tue, Dec 22, 2015 at 02:43:11PM +0300, Руслан Закиров wrote:<br>
<br>
[...]<br>
<span class=""><br>
> Our error log has "ignore long locked inactive cache entry" alerts, but I<br>
> really couldn't match it to "defreeze" event. Access log has STALE/UPDATING<br>
> requests between the alert and EXPIRED (cache updating) request.<br>
<br>
</span>The "ignore long locked inactive cache entry" alerts indicate that<br>
a cache entry was locked by some request, and wasn't unlocked for<br>
a long time. The alert is expected to appear if a cache node is<br>
locked for cache inactive time (as set by proxy_cache_path<br>
inactive=, 10 minutes by default). </blockquote><div><br></div><div>Inactive is defined in the config, but it's set to default 10m.</div><div><br></div><div>What happens with requests after this time? Do they hit backend and update cache? Do they use stale version?</div><div><br></div><div>I'm going to check "long locked" messages in the log to see how many was for "/" location.</div><div>The hash should be the same if we didn't change cache key, right?</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Most likely reason is that a worker died or was killed<br>
while holding a lock on a cache node (i.e., while a request was<br>
waiting for a new response from a backend).<br></blockquote><div><br></div><div>Shouldn't be there a record in error log? Error log level at warn.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Trivial things to consider:<br>
<br>
- check logs for segmentation faults;<br></blockquote><div><br></div><div>no seg faults in logs</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
- if you are using 3rd party modules / patches, try without them;<br></blockquote><div><br></div><div>from freebsd port, updated gist [1] with `nginx -V` output</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
- make sure you don't kill worker processes yourself or using some<br>
automation scripts (in particular, don't try to terminate old<br>
worker processes after a configuration reload).<br></blockquote><div><br></div><div>One recent appearance of the problem was at 1:30AM and I checked</div><div>logs for crazy midnight deploys - nothing.</div><div><br></div><div>Also, we don't use anything custom to restart nginx, just regular services</div><div>management tools.</div><div><br></div><div><br></div><div>[1] <a href="https://gist.github.com/ruz/05456767750715f6b54e">https://gist.github.com/ruz/05456767750715f6b54e</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span class=""><font color="#888888"><br>
--<br>
Maxim Dounin<br>
<a href="http://nginx.org/" rel="noreferrer" target="_blank">http://nginx.org/</a><br>
</font></span><div class=""><div class="h5"><br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div>Руслан Закиров</div><div>Руководитель отдела разработки веб-сервисов</div><div><span>+7(916) 597-92-69</span>, <span>ruz @ <a href="http://www.sports.ru/" target="_blank"><img src="http://farm7.static.flickr.com/6235/6210250811_19a888dbba_o.jpg" width="43" height="14" style="vertical-align: bottom; margin-right: 0px;"></a></span></div></div></div>
</div></div>