<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="text-align: left; direction: ltr; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" bgcolor="#ffffff" text="#403c3a" link="#c26c00" vlink="#403c3a">
<div>Hi Jeff,</div>
<div><br>
</div>
<div>This is pretty much what I'm now looking at doing, with Some HA proxy servers behind the External Google LB and then their backend being an internal Google LB which then balances across our Varnish caching layer and eventually the Nginx app servers.</div>
<div><br>
</div>
<div>Thank you for sharing your config it'll be a good base for us to start from. We moved from AWS for cost/performance reasons but also because Google LBs allow for us to have a static public facing IP address. Currently our customers point the base of their
domain at a server which just redirects requests to the www subdomain and the www subdomain is pointed at a friendly CNAME which used to then be pointed at an AWS ELB CNAME. It now points at an IP address and we can slowly get our customers to update their
DNS to point the root of the domain at the Google LB IP before this work is ready.</div>
<div><br>
</div>
<div>Once again many thanks Jeff and for everyone else for their replies,</div>
<div><br>
</div>
<div>Kind regards,</div>
<div>Richard </div>
<div><br>
</div>
<div>On Tue, 2019-02-12 at 10:37 -0500, Jeff Dyke wrote:</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Hi Richard. HAProxy defaults to reading all certs in a directory and matching hosts names via SNI. Here is the top of my haproxy config, you can see how i redirect LE requests to another server, which solely services up responses to acme-challenges:
<div><br>
</div>
<div>
<div>frontend http</div>
<div> mode http</div>
<div> bind <a href="http://0.0.0.0:80">0.0.0.0:80</a></div>
<div><br>
</div>
<div> #if this is a LE Request send it to a server on this host for renewals</div>
<div> acl letsencrypt-request path_beg -i /.well-known/acme-challenge/</div>
<div> redirect scheme https code 301 unless letsencrypt-request</div>
<div> use_backend letsencrypt-backend if letsencrypt-request</div>
<div><br>
</div>
<div>frontend https</div>
<div> mode tcp</div>
<div> bind <a href="http://0.0.0.0:443">0.0.0.0:443</a> ssl crt /etc/haproxy/certs alpn h2,http/1.1 ecdhe secp384r1</div>
<div> timeout http-request 10s</div>
<div> log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts \ %ac/%fc/%bc/%sc/%rc %sq/%bq SSL_version:%sslv SSL_cypher:%sslc SNI:%[ssl_fc_has_sni]"</div>
<div> #send all HTTP/2 traffic to a specific backend</div>
<div> use_backend http2-nodes if { ssl_fc_alpn -i h2 }</div>
<div> #send HTTP/1.1 and HTTP/1.0 to default, which don't speak HTTP/2</div>
<div> default_backend http1-nodes</div>
</div>
<div><br>
</div>
<div>I'm not sure exactly how this would work with GCP, but if you use AWS ELB's they will give you certs (you have to prove you own the domain), but you have to be able to use an ELB, which could change ips at any time. Unfortunately this didn't work for
us b/c a few of our larger customers whitelist ips and not domain names. which is why i have stayed with HAProxy. </div>
<div><br>
</div>
<div>Jeff</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Feb 12, 2019 at 4:04 AM Richard Paul <<a href="mailto:Richard@primarysite.net">Richard@primarysite.net</a>> wrote:<br>
</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div style="text-align:left;direction:ltr" bgcolor="#ffffff">
<div>Hi Jeff</div>
<div><br>
</div>
<div>That's interesting, how do you manage the progamming to load the right certificate for the right domain coming in as the server name? We need to load the right certificate for the incoming domain and the 12000 figure is the number of unique vanity domains
without the www. subdomains.</div>
<div><br>
</div>
<div>We're planning to follow the same path as you though, we're essentially putting these Nginx TLS terminators (fronted by GCP load balancers) in front of our existing Varnish caching and Nginx backend infrastructure which currently only listen on port 80.</div>
<div><br>
</div>
<div>I couldn't work out what the limits are at LE as it's not clear with regards to adding new unique domains limits. I'm going to have to ask in the forums at some point so that I can work out what our daily batches are going to be.</div>
<div><br>
</div>
<div>Kind regards,</div>
<div>Richard</div>
<div><br>
</div>
<div>On Mon, 2019-02-11 at 14:33 -0500, Jeff Dyke wrote:</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div dir="ltr">
<div dir="ltr">I use haproxy in a similar way as stated by Rainer, rather than having hundreds and hundreds of config files (yes there are other ways), i have 1 for haproxy and 2(on multiple machines defined in HAProxy). One for my main domain that listens
to an "real" server_name and another that listens to `server_name _;` All of the nginx servers simply listen on 80 and 81 to handle non H2 clients and the application does the correct thing with the domain. Which is where YMMV as all applications differ. </div>
<div dir="ltr"><br>
</div>
<div dir="ltr">I found this much simpler and easier to maintain over time. I got around the LE limits by a staggered migration, so i was only requesting what was in the limit each day, then have a custom script that calls LE (which is also on the same machine
as HAProxy) when certs are about 10 days out, so the staggering stays within the limits. When i was using custom configuration, i was build them via python using a yaml file and nginx would effectively be a jinja2 template. But even that became onerous.
When going down the nginx path ensure you pay attention to the variables that control domain hash sizes. <a href="http://nginx.org/en/docs/hash.html" target="_blank">http://nginx.org/en/docs/hash.html</a>
<div><br>
</div>
<div>HTH, good luck!<br>
Jeff</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Feb 11, 2019 at 1:58 PM Rainer Duffner <<a href="mailto:rainer@ultra-secure.de" target="_blank">rainer@ultra-secure.de</a>> wrote:<br>
</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div><br>
<div><br>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div>Am 11.02.2019 um 16:16 schrieb rick_pri <<a href="mailto:nginx-forum@forum.nginx.org" target="_blank">nginx-forum@forum.nginx.org</a>>:</div>
<br class="gmail-m_-805991365783229451gmail-m_-6597626323511961655Apple-interchange-newline">
<div><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">However,
our customers, with about 12000 domain names at present have</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
</div>
</blockquote>
</div>
<br>
<div><br>
</div>
<div>Let’s Encrypt rate limits will likely make these very difficult to obtain and also to renew.</div>
<div><br>
</div>
<div>If you own the DNS, maybe using Wildcard DNS entries is more practical.</div>
<div><br>
</div>
<div>Then, HAProxy allows to just drop all the certificates in a directory and let itself figure out the domain-names it has to answer.</div>
<div>At least, that’s what my co-worker told me.</div>
<div><br>
</div>
<div>Also, there’s the fabio LB with similar goal-posts.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
</div>
<pre>_______________________________________________</pre>
<pre>nginx mailing list</pre>
<a href="mailto:nginx@nginx.org" target="_blank">
<pre>nginx@nginx.org</pre>
</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">
<pre>http://mailman.nginx.org/mailman/listinfo/nginx</pre>
</a></blockquote>
</div>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></blockquote>
</div>
<pre>_______________________________________________</pre>
<pre>nginx mailing list</pre>
<a href="mailto:nginx@nginx.org">
<pre>nginx@nginx.org</pre>
</a>
<pre><br></pre>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx">
<pre>http://mailman.nginx.org/mailman/listinfo/nginx</pre>
</a></blockquote>
</body>
</html>