<div dir="ltr"><img width="0" height="0" class="mailtrack-img" alt="" style="display:flex" src="https://mailtrack.io/trace/mail/c1238648c32f8ff2880b64e1af0c79c0a29ef1b8.png?u=74734"><div>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">Thanks for this Michael. <span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">This is so surprising. If someone decides to Dos and crawls
the website with a rogue header, this will essentially bypass the cache and put
a strain on the website. In fact, I was hit by a dos attack that’s when I started
looking at logs and realized the large number of MISSes. <span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">Can someone please help? <span></span></p>
</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:#000000"><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><table cellspacing="0" cellpadding="0" width="400" style="font-size:12px;font-family:Helvetica,Arial,sans-serif"><tbody><tr><td colspan="2">- Cherian</td></tr></tbody></table></div></div></div>
<br><div class="gmail_quote">On Sat, May 12, 2018 at 12:01 PM, Friscia, Michael <span dir="ltr"><<a href="mailto:michael.friscia@yale.edu" target="_blank">michael.friscia@yale.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_-267008425796858038divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p style="margin-top:0;margin-bottom:0">I'm not sure if this will help, but I ignore/hide a lot, this is in my config</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"></p>
<div>proxy_ignore_headers X-Accel-Expires Expires Cache-Control Set-Cookie;<br>
proxy_hide_header X-Accel-Expires;</div>
<div>proxy_hide_header Pragma;<br>
proxy_hide_header Server;<br>
proxy_hide_header Request-Context;<br>
proxy_hide_header X-Powered-By;<br>
proxy_hide_header X-AspNet-Version;<br>
proxy_hide_header X-AspNetMvc-Version;</div>
<p></p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">I have not experienced the problem you mention, I just thought I would offer my config.<br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<div id="m_-267008425796858038Signature">
<div id="m_-267008425796858038divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div class="m_-267008425796858038BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="m_-267008425796858038PlainText">
<p class="m_-267008425796858038x_MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt;color:#44546a">______________________________<wbr>_____________</span></p>
<p class="m_-267008425796858038x_MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt;color:#44546a">Michael Friscia</span></p>
<p class="m_-267008425796858038x_MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt;color:#44546a">Office of Communications</span></p>
<p class="m_-267008425796858038x_MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt;color:#44546a">Yale School of Medicine</span></p>
<p class="m_-267008425796858038x_MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt;color:#44546a">(203) 737-7932 – office</span></p>
<p class="m_-267008425796858038x_MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt;color:#44546a">(203) 931-5381 – mobile</span></p>
<p class="m_-267008425796858038x_MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt;color:#44546a"><a href="https://mailtrack.io/trace/link/8357a0bdd8c40c2ff5b7d91c7797cbc7a8535ffb?url=http%3A%2F%2Fweb.yale.edu%2F&userId=74734&signature=d652edf1f4f21323" rel="noopener noreferrer" id="m_-267008425796858038LPNoLP" target="_blank"><span style="color:#44546a">http://web.yale.edu</span></a></span></p>
</div>
</span></font></div>
</div>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block;width:98%">
<div id="m_-267008425796858038divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><span class=""><b>From:</b> nginx <<a href="mailto:nginx-bounces@nginx.org" target="_blank">nginx-bounces@nginx.org</a>> on behalf of Quintin Par <<a href="mailto:quintinpar@gmail.com" target="_blank">quintinpar@gmail.com</a>><br>
</span><b>Sent:</b> Saturday, May 12, 2018 1:32 PM<br>
<b>To:</b> <a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<b>Subject:</b> Re: Debugging Nginx Cache Misses: Hitting high number of MISS despite high proxy valid</font>
<div> </div>
</div><div><div class="h5">
<div>
<div dir="ltr"><img class="m_-267008425796858038x_mailtrack-img" alt="" style="display:flex" width="0" height="0" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7">
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
That’s the tricky part. These MISSes are intermittent. Whenever I run curl I get HITs but I end up seeing a lot of MISS in the logs.
<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
How do I log these MiSSes with the reason? I want to know what headers ended up bypassing the cache.<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
Here’s my caching config<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span><span> </span>proxy_pass <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8000&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=F-qGMOyS74uE8JM-dOLmNH92bQ1xQ-7Rj1d6k-_WST4&s=NHvlb1WColNwTWBF36P1whJdu5iWHK9_50IDHugaEdQ&e=" id="m_-267008425796858038LPlnk106503" class="m_-267008425796858038OWAAutoLink" target="_blank">
http://127.0.0.1:8000</a>;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_set_header X-Real-IP<span> </span>$remote_addr;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span><span> </span>proxy_set_header X-Forwarded-Proto https;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_set_header X-Forwarded-Port 443;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span># If logged in, don't cache.<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>if ($http_cookie ~* "comment_author_|wordpress_(?!<wbr>test_cookie)|wp-postpass_" ) {<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span><span> </span>set $do_not_cache 1;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>}<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_cache_key "$scheme://$host$request_uri$<wbr>do_not_cache";<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_cache staticfilecache;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>add_header Cache-Control public;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_cache_valid<span> </span>200 120d;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_hide_header "Set-Cookie";<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_ignore_headers<span> </span>"Set-Cookie";<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_ignore_headers<span> </span>"Cache-Control";<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_hide_header "Cache-Control";<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_pass_header X-Accel-Expires;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_set_header Accept-Encoding "";<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_ignore_headers Expires;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span><span> </span>add_header X-Cache-Status $upstream_cache_status;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_cache_use_stale<span> </span>timeout;<span></span></p>
<p class="m_-267008425796858038x_MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif">
<span> </span>proxy_cache_bypass $arg_nocache $do_not_cache;<br clear="all">
</p>
<div>
<div dir="ltr" class="m_-267008425796858038x_gmail_signature">
<div class="m_-267008425796858038x_gmail_signature">- Quintin</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="m_-267008425796858038x_gmail_quote">
<div dir="ltr">On Sat, May 12, 2018 at 10:29 AM Lucas Rolff <<a href="mailto:lucas@lucasrolff.com" id="m_-267008425796858038LPlnk26217" class="m_-267008425796858038OWAAutoLink" target="_blank">lucas@lucasrolff.com</a>> wrote:<br>
</div>
<blockquote class="m_-267008425796858038x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="DA">
<div class="m_-267008425796858038x_m_5039261820505189399WordSection1">
<p class="m_-267008425796858038x_MsoNormal"><span lang="EN-GB">It can be as simple as doing a curl to your “origin” url (the one you proxy_pass to) for the files you see that gets a lot of MISS’s – if there’s odd headers such as cookies etc, then you’ll most likely experience a
bad cache if your nginx is configured to not ignore those headers.<u></u><u></u></span></p>
<p class="m_-267008425796858038x_MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">nginx <<a href="mailto:nginx-bounces@nginx.org" id="m_-267008425796858038LPlnk804477" class="m_-267008425796858038OWAAutoLink" target="_blank">nginx-bounces@nginx.org</a>> on behalf of Quintin Par <<a href="mailto:quintinpar@gmail.com" id="m_-267008425796858038LPlnk19630" class="m_-267008425796858038OWAAutoLink" target="_blank">quintinpar@gmail.com</a>><br>
<b>Reply-To: </b>"<a href="mailto:nginx@nginx.org" id="m_-267008425796858038LPlnk651746" class="m_-267008425796858038OWAAutoLink" target="_blank">nginx@nginx.org</a>" <<a href="mailto:nginx@nginx.org" id="m_-267008425796858038LPlnk352994" class="m_-267008425796858038OWAAutoLink" target="_blank">nginx@nginx.org</a>><br>
<b>Date: </b>Saturday, 12 May 2018 at 18.26<br>
<b>To: </b>"<a href="mailto:nginx@nginx.org" id="m_-267008425796858038LPlnk715093" class="m_-267008425796858038OWAAutoLink" target="_blank">nginx@nginx.org</a>" <<a href="mailto:nginx@nginx.org" id="m_-267008425796858038LPlnk340378" class="m_-267008425796858038OWAAutoLink" target="_blank">nginx@nginx.org</a>><br>
<b>Subject: </b>Debugging Nginx Cache Misses: Hitting high number of MISS despite high proxy valid<u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><a name="m_-267008425796858038_x_m_5039261820505189399__MailOriginalBody" id="m_-267008425796858038LPlnk841575" class="m_-267008425796858038OWAAutoLink"><img id="m_-267008425796858038x_m_5039261820505189399_x0000_i1025" alt="https://mailtrack.io/trace/mail/86a613eb1ce46a4e7fa6f9eb96989cddae639800.png?u=74734" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"><u></u><u></u></a></p>
<div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">My proxy cache path is set to a very high size</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><u></u> <u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">proxy_cache_path /var/lib/nginx/cache levels=1:2 keys_zone=staticfilecache:<wbr>180m max_size=700m;</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">and the size used is only</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><u></u> <u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">sudo du -sh *</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">14M cache</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">4.0K proxy</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">Proxy cache valid is set to</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><u></u> <u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">proxy_cache_valid 200 120d;</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">I track HIT and MISS via</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><u></u> <u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">add_header X-Cache-Status $upstream_cache_status;</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">Despite these settings I am seeing a lot of MISSes. And this is for pages I intentionally ran a cache warmer an hour ago.</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><u></u> <u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black">How do I debug why these MISSes are happening? How do I find out if the miss was due to eviction, expiration, some rogue header etc? Does Nginx
provide commands for this?</span></span><span><u></u><u></u></span></p>
</div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span><span style="font-family:"Arial",sans-serif;color:black"><u></u> <u></u></span></span></p>
</div>
</div>
<div>
<div>
<div>
<p class="m_-267008425796858038x_MsoNormal" style="margin-left:36.0pt"><span>- Quintin</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
______________________________<wbr>_________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" id="m_-267008425796858038LPlnk579149" class="m_-267008425796858038OWAAutoLink" target="_blank">nginx@nginx.org</a><br>
<a href="https://mailtrack.io/trace/link/122c3dbd333c388f47f5c2776af9ebc3fc75ae10?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__mailman.nginx.org_mailman_listinfo_nginx%26d%3DDwMFaQ%26c%3DcjytLXgP8ixuoHflwc-poQ%26r%3DwvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs%26m%3DF-qGMOyS74uE8JM-dOLmNH92bQ1xQ-7Rj1d6k-_WST4%26s%3DD3LnZhfobOtlEStCvCDrcwmHydEHaGRFC4gnWvRT5Uk%26e%3D&userId=74734&signature=56c7a7ad18b2c057" rel="noreferrer" id="m_-267008425796858038LPlnk537643" class="m_-267008425796858038OWAAutoLink" target="_blank">http://mailman.nginx.org/<wbr>mailman/listinfo/nginx</a></blockquote>
</div>
</div>
</div></div></div>
</div>
</div>
<br>______________________________<wbr>_________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="https://mailtrack.io/trace/link/92c2700d67bd6891ca1606e2df4e0f11c6d82260?url=http%3A%2F%2Fmailman.nginx.org%2Fmailman%2Flistinfo%2Fnginx&userId=74734&signature=3763121afa828bb7" rel="noreferrer" target="_blank">http://mailman.nginx.org/<wbr>mailman/listinfo/nginx</a><br></blockquote></div><br></div>