[PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS

[Sergey Kandaurov]

> On 19 Oct 2021, at 04:54, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> Hello!
> 
> On Mon, Oct 18, 2021 at 06:26:47PM +0300, Sergey Kandaurov wrote:
> 
>>> On 27 Sep 2021, at 16:18, Maxim Dounin <mdounin at mdounin.ru> wrote:
>>> 
>>> # HG changeset patch
>>> # User Maxim Dounin <mdounin at mdounin.ru>
>>> # Date 1632717779 -10800
>>> #      Mon Sep 27 07:42:59 2021 +0300
>>> # Node ID ff514bf17f7f2257dcf036c5c973b74672cefa9a
>>> # Parent  8f0fd60c33c106fba5f1ce3cafe990f15fcccc0c
>>> SSL: SSL_sendfile() support with kernel TLS.
>>> 
>>> Requires OpenSSL 3.0 compiled with "enable-ktls" option.  Further, KTLS
>>> needs to be enabled in kernel, and in OpenSSL, either via OpenSSL
>>> configuration file or with "ssl_conf_command Options KTLS;" in nginx
>>> configuration.
>>> 
>>> On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
>>> can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf".
>> 
>> I am not sure about mentioning ktls_ocf.ko in the commit message.
>> The module is only present in FreeBSD 13.0, it was removed post 13.0,
>> and the functionality is now always present in kernels with KERN_TLS:
>> https://cgit.freebsd.org/src/commit/?id=21e3c1fbe246
>> Further, it is one of many options to enable KTLS.
>> It could be better to refer to man ktls(4), instead:
>> 
>> : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
>> : can be enabled with "sysctl kern.ipc.tls.enable=1", see man ktls(4).
>> 
>> (but I don't insist)
> 
> I would rather keep it explicitly mentioned, since it is a 
> required step on FreeBSD 13, and this is the only FreeBSD release 
> with KTLS so far.  I don't object adding ktls(4) reference though, 
> updated with the following:
> 
> : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and             
> : can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf"       
> : to load a software backend, see man ktls(4) for details.                        
> 

It looks good, thanks.

>>> On Linux, kernel TLS is available starting with kernel 4.13 (at least 5.2
>>> is recommended), and needs kernel compiled with CONFIG_TLS=y (with
>>> CONFIG_TLS=m, which is used at least on Ubuntu 21.04 by default,
>>> the tls module needs to be loaded with "modprobe tls").
>> 
>> On Linux I observe a problem sending data with short socket buffer space.
>> It is Ubuntu 20.04 (5.4.0) and 21.04 (5.11.0), with epoll and select
>> event methods.  As per tcpdump traces, it looks like the buffer cannot
>> be pushed to the network, although it is reported as if it was sent.
>> The simplest I could grab (see below) with ssl_buffer_size 4k and sndbuf 8k
>> (note that unlike SSL_write(), buffers aren't limited with ssl_buffer_size).
> 
> You mean records?  SSL buffer size limits buffering, and as a side 
> effect it limits maximum size of SSL records generated, since 
> nginx always uses the buffer to call SSL_write().  With 
> SSL_sendfile(), it does not limit records generated by sendfile, 
> since the buffer is not used for SSL_sendfile().
> 
> (Just for the record, as of now there is no way to limit maximum 
> record size with SSL_sendfile() (except may be by calling 
> SSL_sendfile() many times with small file fragments, but this 
> approach looks awful), as there are no kernel interfaces to 
> control maximum record size.  Further, OpenSSL disables KTLS if 
> SSL_CTX_set_max_send_fragment() is used with anything other than 
> 16384, see tls1_change_cipher_state(), "ktls supports only the 
> maximum fragment size".  I don't think this is a major problem 
> though.)

Ok, it was useful to know.

> 
>> It doesn't stuck starting with sndbuf 16k, so it might have something
>> with how KTLS send buffers correspond with TCP send buffers.
>> (In contrast, the FreeBSD sendfile is strictly constrained by the available
>> send buffer space and hence shouldn't have this problem.)
>> So it doesn't look like a major issue.
> 
> I was able to reproduce this with sndbuf=32k over localhost on 
> Ubuntu 21.04 (5.11.0-18-generic).  Does not seem to happen with 
> larger buffers, but might be I'm just not testing it hard enough.
> 
> Over (emulated) network I was able to reproduce this with 
> sndbuf=24k, but not with larger buffers.
> 
> [...]
> 
>> I've added additional debugging to SSL_sendfile()
>> to see that sendfile() returns EBUSY (11).
> 
> Nitpicking: EAGAIN, not EBUSY.  EBUSY on Linux is 16, and 
> sendfile() on Linux shouldn't return EBUSY.

Yes, surely EAGAIN.  Thanks for noticing.

> 
> Overall, this looks like an issue in Linux KTLS implementation, 
> probably related to the socket buffer size.  While it would be 
> good to mitigate this on our side if possible, I don't see 
> anything obvious (I've tried tcp_nodelay, but it doesn't help).
> 
> [...]

I think so, too.

-- 
Sergey Kandaurov