Debugging Nginx Cache Misses: Hitting high number of MISS despite high proxy valid
Friscia, Michael
michael.friscia at yale.edu
Mon May 14 11:33:30 UTC 2018
I wish I had a lead for you. I’ve never seen that behavoir
___________________________________________
Michael Friscia
Office of Communications
Yale School of Medicine
(203) 737-7932 - office
(203) 931-5381 - mobile
http://web.yale.edu<http://web.yale.edu/>
From: nginx <nginx-bounces at nginx.org> on behalf of Quintin Par <quintinpar at gmail.com>
Reply-To: "nginx at nginx.org" <nginx at nginx.org>
Date: Monday, May 14, 2018 at 12:07 AM
To: "nginx at nginx.org" <nginx at nginx.org>
Subject: Re: Debugging Nginx Cache Misses: Hitting high number of MISS despite high proxy valid
[https://mailtrack.io/trace/mail/830e676b314f1b30986adfc1c7df5f967b9aa282.png?u=74734]
Thanks all for the response. Michael, I am going to add those header ignores.
Still puzzled by the large number of MISSEs and I’ve no clue why they are happening. Leads appreciated.
- Quintin
On Sun, May 13, 2018 at 6:12 PM, c0nw0nk <nginx-forum at forum.nginx.org<mailto:nginx-forum at forum.nginx.org>> wrote:
You know you can DoS sites with Cache MISS via switching up URL params and
arguements.
Examples :
HIT :
index.php?var1=one&var2=two
MISS :
index.php?var2=two&var1=one
MISS :
index.php?random=1
index.php?random=2
index.php?random=3
etc etc
Inserting random arguements to URL's will cause cache misses and changing
the order of existing valid URL arguements will also cause misses.
Cherian Thomas Wrote:
-------------------------------------------------------
> Thanks for this Michael.
>
>
>
> This is so surprising. If someone decides to Dos and crawls the
> website
> with a rogue header, this will essentially bypass the cache and put a
> strain on the website. In fact, I was hit by a dos attack that’s when
> I
> started looking at logs and realized the large number of MISSes.
>
>
>
> Can someone please help?
>
>
> - Cherian
>
> On Sat, May 12, 2018 at 12:01 PM, Friscia, Michael
> <michael.friscia at yale.edu<mailto:michael.friscia at yale.edu>
> > wrote:
>
> > I'm not sure if this will help, but I ignore/hide a lot, this is in
> my
> > config
> >
> >
> > proxy_ignore_headers X-Accel-Expires Expires Cache-Control
> Set-Cookie;
> > proxy_hide_header X-Accel-Expires;
> > proxy_hide_header Pragma;
> > proxy_hide_header Server;
> > proxy_hide_header Request-Context;
> > proxy_hide_header X-Powered-By;
> > proxy_hide_header X-AspNet-Version;
> > proxy_hide_header X-AspNetMvc-Version;
> >
> >
> > I have not experienced the problem you mention, I just thought I
> would
> > offer my config.
> >
> >
> > ___________________________________________
> >
> > Michael Friscia
> >
> > Office of Communications
> >
> > Yale School of Medicine
> >
> > (203) 737-7932 – office
> >
> > (203) 931-5381 – mobile
> >
> > http://web.yale.edu<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_a61adbc81bbb4743e50220408108f7e1b8f3af40-3Furl-3Dhttp-253A-252F-252Fweb.yale.edu-26userId-3D74734-26signature-3D0767ce63378dc575&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=O1tCcwZQL21v9zwRURBVCpRvRUqzfABxnBSBBf0u_tk&e=>
> >
> <https://mailtrack.io/trace/link/8357a0bdd8c40c2ff5b7d91c7797cbc7a8535<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_661443b9951f60c19cd0ed2ec67ca9c38485a127-3Furl-3Dhttps-253A-252F-252Fmailtrack.io-252Ftrace-252Flink-252F8357a0bdd8c40c2ff5b7d91c7797cbc7a8535-26userId-3D74734-26signature-3Dfd94611bb5198158&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=TfPgIoe6C1z5XdZOtXynwFbMrT7U6VPz-wLAp1xA4iM&e=>
> ffb?url=http%3A%2F%2Fweb.yale.edu<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_8d2b22d027b9e7af0a2468545c2e35529237af19-3Furl-3Dhttp-253A-252F-252F2Fweb.yale.edu-26userId-3D74734-26signature-3D5ab2d28a496b50f6&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=UCopn1kMzqaPgemTq0LMNiijc-lSc23NgceWu6zq1-w&e=>%2F&userId=74734&signature=d652edf1f4
> f21323>
> >
> >
> > ------------------------------
> > *From:* nginx <nginx-bounces at nginx.org<mailto:nginx-bounces at nginx.org>> on behalf of Quintin Par <
> > quintinpar at gmail.com<mailto:quintinpar at gmail.com>>
> > *Sent:* Saturday, May 12, 2018 1:32 PM
> > *To:* nginx at nginx.org<mailto:nginx at nginx.org>
> > *Subject:* Re: Debugging Nginx Cache Misses: Hitting high number of
> MISS
> > despite high proxy valid
> >
> >
> > That’s the tricky part. These MISSes are intermittent. Whenever I
> run curl
> > I get HITs but I end up seeing a lot of MISS in the logs.
> >
> >
> >
> > How do I log these MiSSes with the reason? I want to know what
> headers
> > ended up bypassing the cache.
> >
> >
> >
> > Here’s my caching config
> >
> >
> >
> > proxy_pass http://127.0.0.1:8000<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_071291057b0a07a97c3170df6ceb9706ad0e553d-3Furl-3Dhttp-253A-252F-252F127.0.0.1-253A8000-26userId-3D74734-26signature-3D21d883fe1973c407&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=Fo8waNeI0IbTqTFlCvBd5KR8BkK3vPM4Zfe6evBaEJk&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8000&<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_6864e1b6645eae9d83bd78154bd244cbd3132407-3Furl-3Dhttps-253A-252F-252Furldefense.proofpoint.com-252Fv2-252Furl-253Fu-253Dhttp-2D3A-5F-5F127.0.0.1-2D3A8000-2526-26userId-3D74734-26signature-3D05baa72c55f6e580&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=JDBRefk34RZouWT_M-rzY8GQHTq8COy62hvMgxAtEmQ&e=>
> d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_
> lS023SJrs&m=F-qGMOyS74uE8JM-dOLmNH92bQ1xQ-7Rj1d6k-_WST4&s=NHvlb1WColNw
> TWBF36P1whJdu5iWHK9_50IDHugaEdQ&e=>
> > ;
> >
> > proxy_set_header X-Real-IP $remote_addr;
> >
> > proxy_set_header X-Forwarded-For
> > $proxy_add_x_forwarded_for;
> >
> > proxy_set_header X-Forwarded-Proto https;
> >
> > proxy_set_header X-Forwarded-Port 443;
> >
> >
> >
> > # If logged in, don't cache.
> >
> > if ($http_cookie ~*
> "comment_author_|wordpress_(?!test_cookie)|wp-postpass_"
> > ) {
> >
> > set $do_not_cache 1;
> >
> > }
> >
> > proxy_cache_key "$scheme://$host$request_uri$
> > do_not_cache";
> >
> > proxy_cache staticfilecache;
> >
> > add_header Cache-Control public;
> >
> > proxy_cache_valid 200 120d;
> >
> > proxy_hide_header "Set-Cookie";
> >
> > proxy_ignore_headers "Set-Cookie";
> >
> > proxy_ignore_headers "Cache-Control";
> >
> > proxy_hide_header "Cache-Control";
> >
> > proxy_pass_header X-Accel-Expires;
> >
> >
> >
> > proxy_set_header Accept-Encoding "";
> >
> > proxy_ignore_headers Expires;
> >
> > add_header X-Cache-Status $upstream_cache_status;
> >
> > proxy_cache_use_stale timeout;
> >
> > proxy_cache_bypass $arg_nocache $do_not_cache;
> > - Quintin
> >
> >
> > On Sat, May 12, 2018 at 10:29 AM Lucas Rolff <lucas at lucasrolff.com<mailto:lucas at lucasrolff.com>>
> wrote:
> >
> > It can be as simple as doing a curl to your “origin” url (the one
> you
> > proxy_pass to) for the files you see that gets a lot of MISS’s – if
> there’s
> > odd headers such as cookies etc, then you’ll most likely experience
> a bad
> > cache if your nginx is configured to not ignore those headers.
> >
> >
> >
> > *From: *nginx <nginx-bounces at nginx.org<mailto:nginx-bounces at nginx.org>> on behalf of Quintin Par <
> > quintinpar at gmail.com<mailto:quintinpar at gmail.com>>
> > *Reply-To: *"nginx at nginx.org<mailto:nginx at nginx.org>" <nginx at nginx.org<mailto:nginx at nginx.org>>
> > *Date: *Saturday, 12 May 2018 at 18.26
> > *To: *"nginx at nginx.org<mailto:nginx at nginx.org>" <nginx at nginx.org<mailto:nginx at nginx.org>>
> > *Subject: *Debugging Nginx Cache Misses: Hitting high number of MISS
> > despite high proxy valid
> >
> >
> >
> > [image:
> >
> https://mailtrack.io/trace/mail/86a613eb1ce46a4e7fa6f9eb96989cddae6398<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_mail_86a613eb1ce46a4e7fa6f9eb96989cddae6398&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=t6tQa_F2yg_hzSVp55rmbf0M7gyW7TwdbPQrVPlL_QU&e=>
> 00.png?u=74734]
> >
> > My proxy cache path is set to a very high size
> >
> >
> >
> > proxy_cache_path /var/lib/nginx/cache levels=1:2
> > keys_zone=staticfilecache:180m max_size=700m;
> >
> > and the size used is only
> >
> >
> >
> > sudo du -sh *
> >
> > 14M cache
> >
> > 4.0K proxy
> >
> > Proxy cache valid is set to
> >
> >
> >
> > proxy_cache_valid 200 120d;
> >
> > I track HIT and MISS via
> >
> >
> >
> > add_header X-Cache-Status $upstream_cache_status;
> >
> > Despite these settings I am seeing a lot of MISSes. And this is for
> pages
> > I intentionally ran a cache warmer an hour ago.
> >
> >
> >
> > How do I debug why these MISSes are happening? How do I find out if
> the
> > miss was due to eviction, expiration, some rogue header etc? Does
> Nginx
> > provide commands for this?
> >
> >
> >
> > - Quintin
> > _______________________________________________
> > nginx mailing list
> > nginx at nginx.org<mailto:nginx at nginx.org>
> > http://mailman.nginx.org/mailman/listinfo/nginx<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_956685bf1c269e5b5e505d57769f24a31e3e2442-3Furl-3Dhttp-253A-252F-252Fmailman.nginx.org-252Fmailman-252Flistinfo-252Fnginx-26userId-3D74734-26signature-3D61a29f8655dde16e&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=YorFnQbIkHqdoF-fJvWjdmyC0o-HIxp572ydWwa9dIE&e=>
> >
> <https://mailtrack.io/trace/link/122c3dbd333c388f47f5c2776af9ebc3fc75a<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_0f96ef0fff2b29b47c79cd24c346157878aaf2e5-3Furl-3Dhttps-253A-252F-252Fmailtrack.io-252Ftrace-252Flink-252F122c3dbd333c388f47f5c2776af9ebc3fc75a-26userId-3D74734-26signature-3D0b1e1864a472eee2&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=6Xo5m5oLMU1rkxxxid3nY6k6qFEq5wIJJQ2tS68Fa74&e=>
> e10?url=https%3A%2F%2Furldefense.proofpoint.com<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_5a068de37a59a883da6fd59fdd4026a152a7fc91-3Furl-3Dhttp-253A-252F-252F2Furldefense.proofpoint.com-26userId-3D74734-26signature-3Dca8f6ddc8276a370&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=vCAHT0l6ftZIJwxWTMO43L_gQWuFc9Tit3XWD2PX0AU&e=>%2Fv2%2Furl%3Fu%3Dhttp-
> 3A__mailman.nginx.org_mailman_listinfo_nginx%26d%3DDwMFaQ%26c%3DcjytLX
> gP8ixuoHflwc-poQ%26r%3DwvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs%26m
> %3DF-qGMOyS74uE8JM-dOLmNH92bQ1xQ-7Rj1d6k-_WST4%26s%3DD3LnZhfobOtlEStCv
> CDrcwmHydEHaGRFC4gnWvRT5Uk%26e%3D&userId=74734&signature=56c7a7ad18b2c
> 057>
> >
> >
> > _______________________________________________
> > nginx mailing list
> > nginx at nginx.org<mailto:nginx at nginx.org>
> > http://mailman.nginx.org/mailman/listinfo/nginx<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_f500ef35fc0275c82402a7af89180ae2c67cea6a-3Furl-3Dhttp-253A-252F-252Fmailman.nginx.org-252Fmailman-252Flistinfo-252Fnginx-26userId-3D74734-26signature-3Daa7675f47e061eec&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=9eB5NNsgiaK6OikBqpCpk0jRbUVRh6KO9jP5oPvAUPc&e=>
> >
> <https://mailtrack.io/trace/link/92c2700d67bd6891ca1606e2df4e0f11c6d82<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_d6afed06499ad18204cf041056d4781772869d72-3Furl-3Dhttps-253A-252F-252Fmailtrack.io-252Ftrace-252Flink-252F92c2700d67bd6891ca1606e2df4e0f11c6d82-26userId-3D74734-26signature-3D59dcf4fe89ac3c3c&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=iEjpU-y-FYVBYbXZEB2XFcitUAbNDW5k_QboCKMKOYU&e=>
> 260?url=http%3A%2F%2Fmailman.nginx.org<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_3ec600220aa90db4d165256c22910f3c97fa118d-3Furl-3Dhttp-253A-252F-252F2Fmailman.nginx.org-26userId-3D74734-26signature-3Dc116773b55639f01&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=25kpBZscjcQ5uT8rtttwWxSFzvAiFDBMonE4n8NqmIc&e=>%2Fmailman%2Flistinfo%2Fnginx&us
> erId=74734&signature=3763121afa828bb7>
> >
> _______________________________________________
> nginx mailing list
> nginx at nginx.org<mailto:nginx at nginx.org>
> http://mailman.nginx.org/mailman/listinfo/nginx<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_8e6777181b5012ff78b980aafec44306b2954bae-3Furl-3Dhttp-253A-252F-252Fmailman.nginx.org-252Fmailman-252Flistinfo-252Fnginx-26userId-3D74734-26signature-3D2adebca7901eccce&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=bWwHj7Y30LZeaKzvZv1OqIDyTYd43JCvp913RlN1Sas&e=>
Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279764,279771#msg-279771<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_89e8f350a5c632ccafaadd90a9a8114ecac2e688-3Furl-3Dhttps-253A-252F-252Fforum.nginx.org-252Fread.php-253F2-252C279764-252C279771-2523msg-2D279771-26userId-3D74734-26signature-3D3a01022d1b56bd07&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=NK0qMChGLygem8v511Kj1qNHl-EfH7pUgumMAy-fvVY&e=>
_______________________________________________
nginx mailing list
nginx at nginx.org<mailto:nginx at nginx.org>
http://mailman.nginx.org/mailman/listinfo/nginx<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailtrack.io_trace_link_8e6777181b5012ff78b980aafec44306b2954bae-3Furl-3Dhttp-253A-252F-252Fmailman.nginx.org-252Fmailman-252Flistinfo-252Fnginx-26userId-3D74734-26signature-3D2adebca7901eccce&d=DwMFaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=VAiNmOShRG3O2ssbHNaws7RgYVuLfij3x1kR7OgxKos&s=bWwHj7Y30LZeaKzvZv1OqIDyTYd43JCvp913RlN1Sas&e=>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20180514/e359c987/attachment-0001.html>
More information about the nginx
mailing list