Hello!
This patch series add kernel TLS / SSL_sendfile() support.
Works on FreeBSD 13.0+ and Linux with kernel 4.13+ (at least 5.2
is recommended, tested with 5.11).
The following questions need additional testing/attention:
- What about EINTR? Looks like it simply results in SSL_ERROR_WANT_WRITE,
so might need extra checking to make sure there will be another write
event.
- What about SSL_sendfile(), early data and write blocking?
Ref. c->ssl->write_blocked, 7431:294162223c7c by pluknet@.
Looks like it is not a problem with SSL_sendfile(), but needs
further checking.
- What about FreeBSD aio sendfile (aka SF_NODISKIO)? Might be
easy enough to support.
Review and testing appreciated.
--
Maxim Dounin
Hello,
I have tested the nginx with the patch
https://hg.nginx.org/nginx/rev/65946a191197 (SSL: SSL_sendfile() support
with kernel TLS.) following the nginx blog article
https://www.nginx.com/blog/improving-nginx-performance-with-kernel-tls/
And it sort of works, but I have bad performance when making HTTP/2
requests. If I made a HTTP/1.1 request there is 30-35% increase in
performance as the Nginx blog article stated, but when I changed the
request to use HTTP/2 the request was 40% slower than an ordinary nginx
without KTLS enabled. Does anyone have such perfomance degradation with
nginx KTLS and HTTP/2? I am using generic setup - Ubuntu 20.04.3 LTS and
kernels 5.8.0-63-generic (the same results are with 5.4.0-91-generic).
The nginx vritual host is the same as in the Nginx blog article with
exception of adding http2 to the listen! OpenSSL 3.0.0 and nginx 1.21.4
are used.
The KTLS seems to work, because the strace and debug logs show it. Just
the sstrange thing is when using HTTP2, the sendfile syscalls look:
write(39, "\0 \0\0\0\0\0\0\1", 9) = 9
sendfile(39, 131, [1418218] => [1426410], 8192) = 8192
write(39, "\0 \0\0\0\0\0\0\1", 9) = 9
sendfile(39, 131, [1426410] => [1434602], 8192) = 8192
write(39, "\0 \0\0\0\0\0\0\1", 9) = 9
sendfile(39, 131, [1434602] => [1442794], 8192) = 8192
write(39, "\0 \0\0\0\0\0\0\1", 9) = 9
sendfile(39, 131, [1442794] => [1450986], 8192) = 8192
write(39, "\0 \0\0\0\0\0\0\1", 9) = 9
sendfile(39, 131, [1450986] => [1459178], 8192) = 8192
write(39, "\0 \0\0\0\0\0\0\1", 9) = 9
sendfile(39, 131, [1459178] => [1467370], 8192) = 8192
It is always 8K and there are thousands of sendfile syscalls....
And when the HTTP/1.1 is used the sendfile syscall changes the window:
sendfile(32, 33, [586170368] => [586694656], 487571456) = 524288
sendfile(32, 33, [586694656], 487047168) = -1 EAGAIN (Resource
temporarily unavailable)
epoll_wait(8, [{EPOLLOUT, {u32=1602391248, u64=94426458393808}}],
512, 59711) = 1
sendfile(32, 33, [586694656] => [594673664], 487047168) = 7979008
sendfile(32, 33, [594673664] => [595001344], 479068160) = 327680
sendfile(32, 33, [595001344], 478740480) = -1 EAGAIN (Resource
temporarily unavailable)
epoll_wait(8, [{EPOLLOUT, {u32=1602391248, u64=94426458393808}}],
512, 60000) = 1
sendfile(32, 33, [595001344] => [604028928], 478740480) = 9027584
sendfile(32, 33, [604028928] => [604356608], 469712896) = 327680
sendfile(32, 33, [604356608] => [604749824], 469385216) = 393216
sendfile(32, 33, [604749824] => [605192192], 468992000) = 442368
Does anyone receive similar results?
root@srv-tests:~# curl --http1.1 --tls13-ciphers TLS_AES_256_GCM_SHA384
https://mytests.local/1G --output /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1024M 100 1024M 0 0 1170M 0 --:--:-- --:--:-- --:--:--
1168M
root@srv-tests:~# curl --http1.1 --tls13-ciphers TLS_AES_256_GCM_SHA384
https://mytests.local/1G --output /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1024M 100 1024M 0 0 1197M 0 --:--:-- --:--:-- --:--:--
1196M
root@srv-tests:~# curl --http2 --tls13-ciphers TLS_AES_256_GCM_SHA384
https://mytests.local/1G --output /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1024M 100 1024M 0 0 340M 0 0:00:03 0:00:03
--:--:-- 340M
root@srv-tests:~# curl --http2 --tls13-ciphers TLS_AES_256_GCM_SHA384
https://mytests.local/1G --output /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1024M 100 1024M 0 0 338M 0 0:00:03 0:00:03
--:--:-- 338M
srv-prod ~ # h2load -n 1 -c 1 -t 1 https://mytests.local/1G
starting benchmark...
spawning thread #0: 1 total client(s). 1 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 100% done
finished in 7.64s, 0.13 req/s, 134.11MB/s
requests: 1 total, 1 started, 1 done, 1 succeeded, 0 failed, 0 errored,
0 timeout
status codes: 1 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 1.00GB (1074922031) total, 492B (492) headers (space savings
21.66%), 1.00GB (1073741824) data
min max mean sd +/- sd
time for request: 7.63s 7.63s 7.63s 0us 100.00%
time for connect: 13.46ms 13.46ms 13.46ms 0us 100.00%
time to 1st byte: 15.05ms 15.05ms 15.05ms 0us 100.00%
req/s : 0.13 0.13 0.13 0.00 100.00%
srv-prod ~ # h2load --h1 -n 1 -c 1 -t 1 https://mytests.local/1G
starting benchmark...
spawning thread #0: 1 total client(s). 1 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: http/1.1
progress: 100% done
finished in 2.65s, 0.38 req/s, 386.41MB/s
requests: 1 total, 1 started, 1 done, 1 succeeded, 0 failed, 0 errored,
0 timeout
status codes: 1 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 1.00GB (1073742546) total, 631B (631) headers (space savings
0.00%), 1.00GB (1073741824) data
min max mean sd +/- sd
time for request: 2.64s 2.64s 2.64s 0us 100.00%
time for connect: 13.72ms 13.72ms 13.72ms 0us 100.00%
time to 1st byte: 14.86ms 14.86ms 14.86ms 0us 100.00%
req/s : 0.38 0.38 0.38 0.00 100.00%
With regards,
Lyuben Stoev
# HG changeset patch
# User Tracey Jaquith <tracey(a)archive.org>
# Date 1623797180 0
# Tue Jun 15 22:46:20 2021 +0000
# Node ID 1879d49fe0cf739f48287b5a38a83d3a1adab939
# Parent 5f765427c17ac8cf753967387562201cf4f78dc4
Add optional "mp4_exact_start" nginx config off/on to show video between keyframes.
archive.org has been using mod_h264_streaming with a similar "exact start" patch from me since 2013.
We just moved to nginx mp4 module and are using this patch.
The technique is to find the video keyframe just before the desired "start" time, and send
that down the wire so video playback can start immediately.
Next calculate how many video samples are between the keyframe and desired "start" time
and update the STTS atom where those samples move the duration from (typically) 1001 to 1.
This way, initial unwanted video frames play at ~1/30,000s -- so visually the
video & audio start playing immediately.
You can see an example before/after here (nginx binary built with mp4 module + patch):
https://pi.archive.org/0/items/CSPAN_20160425_022500_2011_White_House_Corre…https://pi.archive.org/0/items/CSPAN_20160425_022500_2011_White_House_Corre…
Tested on linux and macosx.
(this is me: https://github.com/traceypooh )
diff -r 5f765427c17a -r 1879d49fe0cf src/http/modules/ngx_http_mp4_module.c
--- a/src/http/modules/ngx_http_mp4_module.c Tue Jun 01 17:37:51 2021 +0300
+++ b/src/http/modules/ngx_http_mp4_module.c Tue Jun 15 22:46:20 2021 +0000
@@ -43,6 +43,7 @@
typedef struct {
size_t buffer_size;
size_t max_buffer_size;
+ ngx_flag_t exact_start;
} ngx_http_mp4_conf_t;
@@ -340,6 +341,13 @@
offsetof(ngx_http_mp4_conf_t, max_buffer_size),
NULL },
+ { ngx_string("mp4_exact_start"),
+ NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
+ ngx_conf_set_flag_slot,
+ NGX_HTTP_LOC_CONF_OFFSET,
+ offsetof(ngx_http_mp4_conf_t, exact_start),
+ NULL },
+
ngx_null_command
};
@@ -2156,6 +2164,83 @@
static ngx_int_t
+ngx_http_mp4_exact_start_video(ngx_http_mp4_file_t *mp4, ngx_http_mp4_trak_t *trak)
+{
+ uint32_t n, speedup_samples, current_count;
+ ngx_uint_t sample_keyframe, start_sample_exact;
+ ngx_mp4_stts_entry_t *entry, *entries_array;
+ ngx_buf_t *data;
+
+ data = trak->out[NGX_HTTP_MP4_STTS_DATA].buf;
+
+ // Find the keyframe just before the desired start time - so that we can emit an mp4
+ // where the first frame is a keyframe. We'll "speed up" the first frames to 1000x
+ // normal speed (typically), so they won't be noticed. But this way, perceptively,
+ // playback of the _video_ track can start immediately
+ // (and not have to wait until the keyframe _after_ the desired starting time frame).
+ start_sample_exact = trak->start_sample;
+ for (n = 0; n < trak->sync_samples_entries; n++) {
+ // each element of array is the sample number of a keyframe
+ // sync samples starts from 1 -- so subtract 1
+ sample_keyframe = ngx_mp4_get_32value(trak->stss_data_buf.pos + (n * 4)) - 1;
+ if (sample_keyframe <= trak->start_sample) {
+ start_sample_exact = sample_keyframe;
+ }
+ if (sample_keyframe >= trak->start_sample) {
+ break;
+ }
+ }
+
+ if (start_sample_exact < trak->start_sample) {
+ // We're going to prepend an entry with duration=1 for the frames we want to "not see".
+ // MOST of the time (eg: constant video framerate),
+ // we're taking a single element entry array and making it two.
+ speedup_samples = trak->start_sample - start_sample_exact;
+
+ ngx_log_debug3(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0,
+ "exact trak start_sample move %l to %l (speed up %d samples)\n",
+ trak->start_sample, start_sample_exact, speedup_samples);
+
+ entries_array = ngx_palloc(mp4->request->pool,
+ (1 + trak->time_to_sample_entries) * sizeof(ngx_mp4_stts_entry_t));
+ if (entries_array == NULL) {
+ return NGX_ERROR;
+ }
+ entry = &(entries_array[1]);
+ ngx_memcpy(entry, (ngx_mp4_stts_entry_t *)data->pos,
+ trak->time_to_sample_entries * sizeof(ngx_mp4_stts_entry_t));
+
+ current_count = ngx_mp4_get_32value(entry->count);
+ ngx_log_debug1(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0,
+ "exact split in 2 video STTS entry from count:%d", current_count);
+
+ if (current_count <= speedup_samples) {
+ return NGX_ERROR;
+ }
+
+ ngx_mp4_set_32value(entry->count, current_count - speedup_samples);
+ ngx_log_debug2(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0,
+ "exact split new[1]: count:%d duration:%d",
+ ngx_mp4_get_32value(entry->count),
+ ngx_mp4_get_32value(entry->duration));
+ entry--;
+ ngx_mp4_set_32value(entry->count, speedup_samples);
+ ngx_mp4_set_32value(entry->duration, 1);
+ ngx_log_debug1(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0,
+ "exact split new[0]: count:%d duration:1",
+ ngx_mp4_get_32value(entry->count));
+
+ data->pos = (u_char *) entry;
+ trak->time_to_sample_entries++;
+ trak->start_sample = start_sample_exact;
+ data->last = (u_char *) (entry + trak->time_to_sample_entries);
+ }
+
+ return NGX_OK;
+}
+
+
+static ngx_int_t
ngx_http_mp4_crop_stts_data(ngx_http_mp4_file_t *mp4,
ngx_http_mp4_trak_t *trak, ngx_uint_t start)
{
@@ -2164,6 +2249,8 @@
ngx_buf_t *data;
ngx_uint_t start_sample, entries, start_sec;
ngx_mp4_stts_entry_t *entry, *end;
+ ngx_http_mp4_conf_t *conf;
+
if (start) {
start_sec = mp4->start;
@@ -2238,6 +2325,10 @@
"start_sample:%ui, new count:%uD",
trak->start_sample, count - rest);
+ conf = ngx_http_get_module_loc_conf(mp4->request, ngx_http_mp4_module);
+ if (conf->exact_start) {
+ ngx_http_mp4_exact_start_video(mp4, trak);
+ }
} else {
ngx_mp4_set_32value(entry->count, rest);
data->last = (u_char *) (entry + 1);
@@ -3590,6 +3681,7 @@
conf->buffer_size = NGX_CONF_UNSET_SIZE;
conf->max_buffer_size = NGX_CONF_UNSET_SIZE;
+ conf->exact_start = NGX_CONF_UNSET;
return conf;
}
Hello!
The following patch series is expected to resolve issues with multiple
headers with identical names. In particular, the following Trac tickets
are fixed:
https://trac.nginx.org/nginx/ticket/1724
Nginx doesn't sanitize and is inconsistent with multiple, repeated input headers
https://trac.nginx.org/nginx/ticket/1316
$http_ variables only contain the first field-value
https://trac.nginx.org/nginx/ticket/1843
$upstream_http_set_cookie includes only first cookie
https://trac.nginx.org/nginx/ticket/1423
response vary headers not used in the cache key
https://trac.nginx.org/nginx/ticket/485
Multiple WWW-Authenticate headers
The basic idea is that all headers are linked lists now, which makes
it possible to easily concatenate multiple headers without introducing
arrays, as it was previously done with Cookie and X-Forwarded-For request
headers and Cache-Control, Link, and Set-Cookie response headers.
Futher, such approach makes it possible to easily concatenate unknown
headers at runtime, so it makes it possible to properly merge headers
passed to CGI-like backends.
Review and testing appreciated.
--
Maxim Dounin
Hello!
On Tue, Apr 19, 2022 at 10:59:12PM +0100, Vadim Fedorenko wrote:
> On 19.04.2022 16:01, Maxim Dounin wrote:
> >
> > On Sun, Apr 17, 2022 at 08:50:25PM +0100, Vadim Fedorenko via nginx-devel wrote:
> >
> >> On 17.04.2022 02:55, Maxim Dounin wrote:
> >>
> >>> I'm quite sceptical about attempts to fix this by introducing
> >>> various flags and reverting cacheable status back to 1. This is
> >>> not how it should be fixed.
> >>
> >> Yeah, the solution might be a bit complicated, but I couldn't find
> >> another way without breaking concept of independent header parsing.
> >> Could you please suggest something if you think that current approach
> >> is wrong? The ticket that this patch tries to fix is 6 years old and
> >> still has discussions going on without any solution.
> >
> > That's basically why the ticket is still not fixed: there is no
> > easy fix, and the issue is rather minor, especially compared to
> > the complexity of the changes required to fix it.
> >
> Hmm.. I would argue on the minority of the issue, especially when we talk about
> absence of compatibility with RFC and actual discussions in the ticket. Anyway.
While current behaviour might be suboptimal in some use cases, it
does not violate RFC, as caching is not something required to
happen. So we are talking about optimizing some use cases here,
and the same optimization can be easily obtained by adjusting
headers in the backend response.
> > IMHO, the most promising approach would be to move
> > Cache-Control/Exires/X-Accel-Expires handling elsewhere, and
> > handle them somewhere in ngx_http_upstream_process_headers(),
> > probably along with ngx_http_file_cache_valid() as well.
> >
> Ok, cool, I updated my patch, now it's based on the work from Yugo that was left
> without comments and is following the way you suggest. Take a look at v2, please.
Apart multiple style issues, the patch fails to address the
X-Accel-Expires vs. Cache-Control order issue as outlined in the
ticket, that is, the response with:
: X-Accel-Expires: 10
: Cache-Control: max-age=100, stale-while-revalidate=1000
results in different behaviour than the one with:
: Cache-Control: max-age=100, stale-while-revalidate=1000
: X-Accel-Expires: 10
It is also seems to be completely unreadable. (Yugo's patch tried
to address this, though had multiple style and readability issues
as well.)
--
Maxim Dounin
http://mdounin.ru/
it is desirable to avoid pool allocations at early stages of quic connection
processing. Currently, code in protection.c and tokens.c allocates memory
dynamically, while this is not strictly necessary, as allocated objects have
fixed size and sometimes short lifetime. The patchset revises this cases
and removes pool usage.
This patchset prepares base to more lightweight early packet processing
(parsing, retry and rejection with error without creating connection object
and memory allocations)
# HG changeset patch
# User Sergey Kandaurov <pluknet(a)nginx.com>
# Date 1651239481 -14400
# Fri Apr 29 17:38:01 2022 +0400
# Node ID 35afae4b3dffff6718c0cab3ceb16b9de207c20a
# Parent a736a7a613ea6e182ff86fbadcb98bb0f8891c0b
Configure: recognize arm64 machine name as a synonym for aarch64.
In particular, this sets a reasonable cacheline size on FreeBSD and macOS,
which prefer to use this name and both lack _SC_LEVEL1_DCACHE_LINESIZE.
diff --git a/auto/os/conf b/auto/os/conf
--- a/auto/os/conf
+++ b/auto/os/conf
@@ -110,7 +110,7 @@ case "$NGX_MACHINE" in
NGX_MACH_CACHE_LINE=64
;;
- aarch64 )
+ aarch64 | arm64)
have=NGX_ALIGNMENT value=16 . auto/define
NGX_MACH_CACHE_LINE=64
;;
# HG changeset patch
# User stdanley <shanlei(a)asiainfo.com>
# Date 1650788278 -28800
# Sun Apr 24 16:17:58 2022 +0800
# Node ID 522acbe88486d027383075c8208edd6fcc0a3aa6
# Parent a736a7a613ea6e182ff86fbadcb98bb0f8891c0b
patch: call modules' exit_process in reverse order to solve module
dependency.
We once have developed a module which depends on ngx_thread_pool module.
but nginx hungs when began to shutdown. trace shows that ngx_thread_pool
cant finish "exit_process" because our module is using a thread, our
module must
call "exit_process" before ngx_thread_pool.
so we present this patch to solve modules' dependency issue.
diff -r a736a7a613ea -r 522acbe88486 src/os/unix/ngx_process_cycle.c
--- a/src/os/unix/ngx_process_cycle.c Tue Feb 08 17:35:27 2022 +0300
+++ b/src/os/unix/ngx_process_cycle.c Sun Apr 24 16:17:58 2022 +0800
@@ -300,10 +300,13 @@
ngx_process_events_and_timers(cycle);
if (ngx_terminate || ngx_quit) {
-
- for (i = 0; cycle->modules[i]; i++) {
- if (cycle->modules[i]->exit_process) {
- cycle->modules[i]->exit_process(cycle);
+ ngx_int_t j, k = 0;
+ //for (i = 0; cycle->modules[i]; i++) {
+ for (j = 0; cycle->modules[j]; j++, k++)
+ ;
+ for (j = k - 1; j >= 0; j--) {
+ if (cycle->modules[j]->exit_process) {
+ cycle->modules[j]->exit_process(cycle);
}
}
@@ -950,10 +953,14 @@
{
ngx_uint_t i;
ngx_connection_t *c;
+ ngx_int_t j, k=0;
- for (i = 0; cycle->modules[i]; i++) {
- if (cycle->modules[i]->exit_process) {
- cycle->modules[i]->exit_process(cycle);
+ //for (i = 0; cycle->modules[i]; i++) {
+ for (j = 0; cycle->modules[j]; j++, k++)
+ ;
+ for (j = k - 1; j >= 0; j--) {
+ if (cycle->modules[j]->exit_process) {
+ cycle->modules[j]->exit_process(cycle);
}
}