[PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS

[Maxim Dounin]

Hello!

On Mon, Oct 18, 2021 at 06:26:47PM +0300, Sergey Kandaurov wrote:

> > On 27 Sep 2021, at 16:18, Maxim Dounin <mdounin at mdounin.ru> wrote:
> > 
> > # HG changeset patch
> > # User Maxim Dounin <mdounin at mdounin.ru>
> > # Date 1632717779 -10800
> > #      Mon Sep 27 07:42:59 2021 +0300
> > # Node ID ff514bf17f7f2257dcf036c5c973b74672cefa9a
> > # Parent  8f0fd60c33c106fba5f1ce3cafe990f15fcccc0c
> > SSL: SSL_sendfile() support with kernel TLS.
> > 
> > Requires OpenSSL 3.0 compiled with "enable-ktls" option.  Further, KTLS
> > needs to be enabled in kernel, and in OpenSSL, either via OpenSSL
> > configuration file or with "ssl_conf_command Options KTLS;" in nginx
> > configuration.
> > 
> > On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
> > can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf".
> 
> I am not sure about mentioning ktls_ocf.ko in the commit message.
> The module is only present in FreeBSD 13.0, it was removed post 13.0,
> and the functionality is now always present in kernels with KERN_TLS:
> https://cgit.freebsd.org/src/commit/?id=21e3c1fbe246
> Further, it is one of many options to enable KTLS.
> It could be better to refer to man ktls(4), instead:
> 
> : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
> : can be enabled with "sysctl kern.ipc.tls.enable=1", see man ktls(4).
> 
> (but I don't insist)

I would rather keep it explicitly mentioned, since it is a 
required step on FreeBSD 13, and this is the only FreeBSD release 
with KTLS so far.  I don't object adding ktls(4) reference though, 
updated with the following:

: On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and             
: can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf"       
: to load a software backend, see man ktls(4) for details.                        

> > On Linux, kernel TLS is available starting with kernel 4.13 (at least 5.2
> > is recommended), and needs kernel compiled with CONFIG_TLS=y (with
> > CONFIG_TLS=m, which is used at least on Ubuntu 21.04 by default,
> > the tls module needs to be loaded with "modprobe tls").
> 
> On Linux I observe a problem sending data with short socket buffer space.
> It is Ubuntu 20.04 (5.4.0) and 21.04 (5.11.0), with epoll and select
> event methods.  As per tcpdump traces, it looks like the buffer cannot
> be pushed to the network, although it is reported as if it was sent.
> The simplest I could grab (see below) with ssl_buffer_size 4k and sndbuf 8k
> (note that unlike SSL_write(), buffers aren't limited with ssl_buffer_size).

You mean records?  SSL buffer size limits buffering, and as a side 
effect it limits maximum size of SSL records generated, since 
nginx always uses the buffer to call SSL_write().  With 
SSL_sendfile(), it does not limit records generated by sendfile, 
since the buffer is not used for SSL_sendfile().

(Just for the record, as of now there is no way to limit maximum 
record size with SSL_sendfile() (except may be by calling 
SSL_sendfile() many times with small file fragments, but this 
approach looks awful), as there are no kernel interfaces to 
control maximum record size.  Further, OpenSSL disables KTLS if 
SSL_CTX_set_max_send_fragment() is used with anything other than 
16384, see tls1_change_cipher_state(), "ktls supports only the 
maximum fragment size".  I don't think this is a major problem 
though.)

> It doesn't stuck starting with sndbuf 16k, so it might have something
> with how KTLS send buffers correspond with TCP send buffers.
> (In contrast, the FreeBSD sendfile is strictly constrained by the available
> send buffer space and hence shouldn't have this problem.)
> So it doesn't look like a major issue.

I was able to reproduce this with sndbuf=32k over localhost on 
Ubuntu 21.04 (5.11.0-18-generic).  Does not seem to happen with 
larger buffers, but might be I'm just not testing it hard enough.

Over (emulated) network I was able to reproduce this with 
sndbuf=24k, but not with larger buffers.

[...]

> I've added additional debugging to SSL_sendfile()
> to see that sendfile() returns EBUSY (11).

Nitpicking: EAGAIN, not EBUSY.  EBUSY on Linux is 16, and 
sendfile() on Linux shouldn't return EBUSY.

Overall, this looks like an issue in Linux KTLS implementation, 
probably related to the socket buffer size.  While it would be 
good to mitigate this on our side if possible, I don't see 
anything obvious (I've tried tcp_nodelay, but it doesn't help).

[...]

-- 
Maxim Dounin
http://mdounin.ru/