nginx KTLS and HTTP/2 performance degradation

[Maxim Dounin]

Hello!

On Thu, Dec 02, 2021 at 02:05:52PM +0200, Lyuben Stoev wrote:

> Hello,
>      I have tested the nginx with the patch 
> https://hg.nginx.org/nginx/rev/65946a191197 (SSL: SSL_sendfile() support 
> with kernel TLS.) following the nginx blog article 
> https://www.nginx.com/blog/improving-nginx-performance-with-kernel-tls/ 
> And it sort of works, but I have bad performance when making HTTP/2 
> requests. If I made a HTTP/1.1 request there is 30-35% increase in 
> performance as the Nginx blog article stated, but when I changed the 
> request to use HTTP/2 the request was 40% slower than an ordinary nginx 
> without KTLS enabled. Does anyone have such perfomance degradation with 
> nginx KTLS and HTTP/2? I am using generic setup - Ubuntu 20.04.3 LTS and 
> kernels 5.8.0-63-generic (the same results are with 5.4.0-91-generic). 
> The nginx vritual host is the same as in the Nginx blog article with 
> exception of adding http2 to the listen! OpenSSL 3.0.0 and nginx 1.21.4 
> are used.
> The KTLS seems to work, because the strace and debug logs show it. Just 
> the sstrange thing is when using HTTP2, the sendfile syscalls look:
>      write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
>      sendfile(39, 131, [1418218] => [1426410], 8192) = 8192
>      write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
>      sendfile(39, 131, [1426410] => [1434602], 8192) = 8192
>      write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
>      sendfile(39, 131, [1434602] => [1442794], 8192) = 8192
>      write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
>      sendfile(39, 131, [1442794] => [1450986], 8192) = 8192
>      write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
>      sendfile(39, 131, [1450986] => [1459178], 8192) = 8192
>      write(39, "\0 \0\0\0\0\0\0\1", 9)       = 9
>      sendfile(39, 131, [1459178] => [1467370], 8192) = 8192
> 
> It is always 8K and there are thousands of sendfile syscalls....

That's expected, because of HTTP/2 framing.  Unfortunately, HTTP/2 
isn't designed to work with sendfile(), and sending large files 
over HTTP/2 require a lot of sendfile() syscalls.  In general, for 
HTTP/2 it is better to keep sendfile() disabled.

-- 
Maxim Dounin
http://mdounin.ru/