<div dir="ltr">On Tue, Dec 17, 2013 at 7:59 PM, Alexš<span dir="ltr"><<a href="mailto:alex@zeitgeist.se" target="_blank">alex@zeitgeist.se</a>></span>šwrote:<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
I did some empirical testing and with my configuration (given cipher<br>size, padding, and all), I came to 1370 bytes as being the optimal size<br>for avoid fragmenting of TLS record fragmentation.<br></blockquote><div><br>
</div><div>Ah, right, we're not setting the "total" record size... Rather, we're setting the maximum payload size within the record. On top of that there is the extra 5 bytes for the record header, plus MAC and padding (if block cipher is used) -- so that's 5 bytes + up to 32 extra bytes per record. Add IP (40 bytes for IPv6), TCP header (20), and some room for TCP options (40), and we're looking at ~1360 bytes.. Which is close to what you're seeing in your testing.</div>
<div>š</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">> If you only distinguish pre and post TLS handshake then you'll still (likely) incur the extra RTT on first app-data record -- that's what we're trying to avoid by reducing the default record size. For HTTP traffic, I think you want 1400 bytes records. Once we're out of slow-start, you can switch back to larger record size.<br>
<br></div>Maybe I am wrong but I was of the belief that you should always try to<br>fit TLS records into individual TCP segments. Hence you should always<br>try to keep TLS record ~1400 bytes (or 1370 in my case), no matter the<br>
TCP Window.<br></blockquote><div><br></div><div>For interactive traffic I think that's generally true as it eliminates the edge case of CWND overflows (extra RTT of buffering) and minimizes impact of packet reordering and packet loss. FWIW, for these exact reasons the Google frontend servers have been using TLS record = TCP segment for a few years now... So there is good precedent to using this as a default.</div>
<div><br></div><div>That said, small records do incur overhead due to extra framing, plus more CPU cycles (more MACs and framing processing). So, in some instances, if you're delivering large streams (e.g. video), you may want to use larger records... Exposing record size as a configurable option would address this.</div>
<div>š</div><div class="gmail_extra"><div class="gmail_quote">On Wed, Dec 18, 2013 at 8:38 AM, Maxim Dounin <span dir="ltr"><<a href="mailto:mdounin@mdounin.ru" target="_blank">mdounin@mdounin.ru</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br><div class="im">> Although now on closer inspection there seems to be another gotcha in there<br>
> that I overlooked: it's emitting two packets, one is 1389 bytes, and second<br>
> is ~31 extra bytes, which means the actual record is 1429 bytes. Obviously,<br>
> this should be a single packet... and 1400 bytes.<br>
<br>
</div>We've discussed this alot here a while ago, and it turns<br>
out that it's very non-trivial task to fill exactly one packet -<br>
as space in packets may vary depending on TCP options used, MTU,<br>
tunnels used on a way to a client, etc.<br></blockquote><div><br></div><div>Yes, that's a good point.š</div><div>š</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
On the other hand, it looks good enough to have records up to<br>
initial CWND in size without any significant latency changes. šAnd<br>
with IW10 this basically means that anything up to about 14k<br>
should be fine (with RFC3390, something like 4k should be ok).<br>
It also reduces bandwidth costs associated with using multiple<br>
records.<br></blockquote><div><br></div><div>In theory, I agree with you, but in practice even while trying to play with this on my own server it appears to be more tricky than that: to ~reliably avoid the CWND overflow I have to set the record size <10k.. There are also differences in how the CWND is increased (byte based vs packet based) across different platforms, and other edge cases I'm surely overlooking. Also, while this addresses the CWND overflow during slowstart, smaller records offer additional benefits as they help minimize impact of reordering and packet loss (not eliminate, but reduce its negative impact in some cases).š</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Just in case, below is a patch to play with SSL buffer size:<br>
<br>
# HG changeset patch<br>
# User Maxim Dounin <<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>><br>
# Date 1387302972 -14400<br>
# š š šTue Dec 17 21:56:12 2013 +0400<br>
# Node ID 090a57a2a599049152e87693369b6921efcd6bca<br>
# Parent še7d1a00f06731d7508ec120c1ac91c337d15c669<br>
SSL: ssl_buffer_size directive.<br></blockquote><div><br></div><div>Just tried it on my local server, works as advertised. :-)</div><div><br></div><div>Defaults matter and we should optimize for best performance out of the box... Can we update NGX_SSL_BUFSIZE size as part of this patch? My current suggestion is 1360 bytes as this guarantees best possible case for helping the browser start processing data as soon as possible: minimal impact of reordering / packet loss / no CWND overflows.</div>
</div></div></div>