[PATCH 2 of 2] SSL: SSL_sendfile() support with kernel TLS
pluknet at nginx.com
Tue Oct 19 10:49:48 UTC 2021
> On 19 Oct 2021, at 04:54, Maxim Dounin <mdounin at mdounin.ru> wrote:
> On Mon, Oct 18, 2021 at 06:26:47PM +0300, Sergey Kandaurov wrote:
>>> On 27 Sep 2021, at 16:18, Maxim Dounin <mdounin at mdounin.ru> wrote:
>>> # HG changeset patch
>>> # User Maxim Dounin <mdounin at mdounin.ru>
>>> # Date 1632717779 -10800
>>> # Mon Sep 27 07:42:59 2021 +0300
>>> # Node ID ff514bf17f7f2257dcf036c5c973b74672cefa9a
>>> # Parent 8f0fd60c33c106fba5f1ce3cafe990f15fcccc0c
>>> SSL: SSL_sendfile() support with kernel TLS.
>>> Requires OpenSSL 3.0 compiled with "enable-ktls" option. Further, KTLS
>>> needs to be enabled in kernel, and in OpenSSL, either via OpenSSL
>>> configuration file or with "ssl_conf_command Options KTLS;" in nginx
>>> On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
>>> can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf".
>> I am not sure about mentioning ktls_ocf.ko in the commit message.
>> The module is only present in FreeBSD 13.0, it was removed post 13.0,
>> and the functionality is now always present in kernels with KERN_TLS:
>> Further, it is one of many options to enable KTLS.
>> It could be better to refer to man ktls(4), instead:
>> : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
>> : can be enabled with "sysctl kern.ipc.tls.enable=1", see man ktls(4).
>> (but I don't insist)
> I would rather keep it explicitly mentioned, since it is a
> required step on FreeBSD 13, and this is the only FreeBSD release
> with KTLS so far. I don't object adding ktls(4) reference though,
> updated with the following:
> : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and
> : can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf"
> : to load a software backend, see man ktls(4) for details.
It looks good, thanks.
>>> On Linux, kernel TLS is available starting with kernel 4.13 (at least 5.2
>>> is recommended), and needs kernel compiled with CONFIG_TLS=y (with
>>> CONFIG_TLS=m, which is used at least on Ubuntu 21.04 by default,
>>> the tls module needs to be loaded with "modprobe tls").
>> On Linux I observe a problem sending data with short socket buffer space.
>> It is Ubuntu 20.04 (5.4.0) and 21.04 (5.11.0), with epoll and select
>> event methods. As per tcpdump traces, it looks like the buffer cannot
>> be pushed to the network, although it is reported as if it was sent.
>> The simplest I could grab (see below) with ssl_buffer_size 4k and sndbuf 8k
>> (note that unlike SSL_write(), buffers aren't limited with ssl_buffer_size).
> You mean records? SSL buffer size limits buffering, and as a side
> effect it limits maximum size of SSL records generated, since
> nginx always uses the buffer to call SSL_write(). With
> SSL_sendfile(), it does not limit records generated by sendfile,
> since the buffer is not used for SSL_sendfile().
> (Just for the record, as of now there is no way to limit maximum
> record size with SSL_sendfile() (except may be by calling
> SSL_sendfile() many times with small file fragments, but this
> approach looks awful), as there are no kernel interfaces to
> control maximum record size. Further, OpenSSL disables KTLS if
> SSL_CTX_set_max_send_fragment() is used with anything other than
> 16384, see tls1_change_cipher_state(), "ktls supports only the
> maximum fragment size". I don't think this is a major problem
Ok, it was useful to know.
>> It doesn't stuck starting with sndbuf 16k, so it might have something
>> with how KTLS send buffers correspond with TCP send buffers.
>> (In contrast, the FreeBSD sendfile is strictly constrained by the available
>> send buffer space and hence shouldn't have this problem.)
>> So it doesn't look like a major issue.
> I was able to reproduce this with sndbuf=32k over localhost on
> Ubuntu 21.04 (5.11.0-18-generic). Does not seem to happen with
> larger buffers, but might be I'm just not testing it hard enough.
> Over (emulated) network I was able to reproduce this with
> sndbuf=24k, but not with larger buffers.
>> I've added additional debugging to SSL_sendfile()
>> to see that sendfile() returns EBUSY (11).
> Nitpicking: EAGAIN, not EBUSY. EBUSY on Linux is 16, and
> sendfile() on Linux shouldn't return EBUSY.
Yes, surely EAGAIN. Thanks for noticing.
> Overall, this looks like an issue in Linux KTLS implementation,
> probably related to the socket buffer size. While it would be
> good to mitigate this on our side if possible, I don't see
> anything obvious (I've tried tcp_nodelay, but it doesn't help).
I think so, too.
More information about the nginx-devel