<div dir="ltr">We currently have ~30k req/s but our network is growing very fast so i need to make sure our architecture is scalable . <div><br><div>After some researching i've decided to go with individual nginx nodes for now . If we encounter too much request to our upstream, i'm gonna set up the multi layer architecture you mentioned probably .</div></div><div><br></div><div>Thank you for your help.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Sep 23, 2017 at 2:37 PM, Lucas Rolff <span dir="ltr"><<a href="mailto:lucas@lucasrolff.com" target="_blank">lucas@lucasrolff.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-GB" link="blue" vlink="purple">
<div class="m_-2124491589366470784WordSection1"><span class="">
<p class="MsoNormal">> if one node had the storage capacity to satisfy my needs it couldn't handle all the requests<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</span><p class="MsoNormal">What amount of requests / traffic are we talking about, and which kind of hardware do you use?<br>
You can make nginx serve 20+ gigabit of traffic from a single machine if the content is right, or 50k+ req/s<u></u><u></u></p><span class="">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">> But at this point i'm beginning to think if it's even worth it . Should i settle for having multiple nginx nodes requesting the same item to our upstream server ?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</span><p class="MsoNormal">If you’re offloading 99.xx% of the content to nginx anyway, a few extra requests to the upstream shouldn’t really matter much.<u></u><u></u></p>
<p class="MsoNormal">You could even have multiple layers of nginx to lower the amount of upstream connections going to the server – so on your let’s say 10 nginx instances, you could use 1-2 nginx instances as upstream, and on those 1-2 nginx instances use
the actual upstream.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Generally speaking you’ll have downsides with sharing storage or cache between multiple servers, and it just adds a lot of complexity to minimize the cost and then it might turn out you actually do not save anything anyway.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Best Regards,<u></u><u></u></p>
<p class="MsoNormal">Lucas<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">nginx <<a href="mailto:nginx-bounces@nginx.org" target="_blank">nginx-bounces@nginx.org</a>> on behalf of Amir Keshavarz <<a href="mailto:amirkekh@gmail.com" target="_blank">amirkekh@gmail.com</a>><br>
<b>Reply-To: </b>"<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>" <<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>><br>
<b>Date: </b>Saturday, 23 September 2017 at 11.48<br>
<b>To: </b>"<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>" <<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>><br>
<b>Subject: </b>Re: Scaling nginx caching storage<u></u><u></u></span></p>
</div><div><div class="h5">
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Sorry for the confusion . <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">My problem is that i need to cache items as much as possible so even if one node had the storage capacity to satisfy my needs it couldn't handle all the requests and we can't afford multiple nginx nodes request
to our main server each time an item is requested on a different nginx node .<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">For that problem i have afew scenarios but they either have huge overhead on our servers and our network or are not suitable for <span style="font-size:10.0pt;color:black">sensitive production env because it causes
weird problems ( sharing storage ) .</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;color:black">But at this point i'm beginning to think if it's even worth it . Should i settle for having multiple nginx nodes requesting the same item to our upstream server ?</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">On Sat, Sep 23, 2017 at 1:48 PM, Lucas Rolff <<a href="mailto:lucas@lucasrolff.com" target="_blank">lucas@lucasrolff.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
> is there any way to share a cache directory between two nginx instances ?<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
> If it can't be done what do you think is the best way to go when we need to scale the nginx caching storage ?<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
One is about using same storage for two nginx instances, the other one is scaling the nginx cache storage.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
I believe it’s two different things.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
There’s nothing that prevents you from having two nginx instances reading from the same cache storage – however you will get into scenarios where if you try to write from both machines (Let’s say it tries to cache the same file on both nginx instances), you
might have some issues.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
Why exactly would you need two instances to share the same storage?<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
And what scale do you mean by scaling the nginx caching storage?<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
Currently there’s really only a limit to your disk size and the size of your keys_zone – if you have 50 terabytes of storage, just set the keys_zone size to be big enough to contain the amount of files you wanna manage (you can store about 8000 files per 1
megabyte).<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:72.0pt">
<b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">nginx <<a href="mailto:nginx-bounces@nginx.org" target="_blank">nginx-bounces@nginx.org</a>> on behalf of Amir Keshavarz <<a href="mailto:amirkekh@gmail.com" target="_blank">amirkekh@gmail.com</a>><br>
<b>Reply-To: </b>"<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>" <<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>><br>
<b>Date: </b>Saturday, 23 September 2017 at 10.58<br>
<b>To: </b>"<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>" <<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>><br>
<b>Subject: </b>Scaling nginx caching storage</span><u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:72.0pt">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:72.0pt">
Hello, <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:72.0pt">
Since nginx stores some cache metadata in memory , is there any way to share a cache directory between two nginx instances ?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:72.0pt">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:72.0pt">
If it can't be done what do you think is the best way to go when we need to scale the nginx caching storage ?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:72.0pt">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:72.0pt">
Thanks<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><br>
______________________________<wbr>_________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/<wbr>mailman/listinfo/nginx</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
<br>______________________________<wbr>_________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/<wbr>mailman/listinfo/nginx</a><br></blockquote></div><br></div>