SSL Reuse not happening in s3 presigned urls

[Vijay Kumar Kamannavar]

Hello.

I am using nginx reverse proxy for s3 presigned urls.
I am running nginx as a container using nginx:1.25.2 debian image. My host
has 16 Core and 32GB.

Below is the nginx configuration.

user  nginx;
worker_processes 14;
pid /run/nginx.pid;
worker_rlimit_nofile 40000;
events {
  worker_connections 1024;
}
http {
 upstream s3_backend {
        server <mybucket>.s3.amazonaws.com:443;
         keepalive 10;
    }

 log_format combined_ssl '$remote_addr - $remote_user [$time_local] '
            '"$request" $status $body_bytes_sent '
            '"$http_referer" "$http_user_agent" '
            '$ssl_protocol/$ssl_cipher '
            '$ssl_session_reused';
  proxy_ssl_session_reuse on;
  proxy_ssl_server_name on;

    # HTTPS server block with SSL certificate and S3 reverse proxy
    server {
        listen 443 ssl;
        ssl_protocols         SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers           HIGH:!aNULL:!MD5;
        ssl_session_cache    shared:SSL:10m;
        ssl_session_timeout  10m;
        ssl_session_tickets off;
        server_name <mydomain>.company.com;

        # Path to your SSL certificate and private key
        ssl_certificate  /etc/nginx/domain.crt;
        ssl_certificate_key /etc/nginx/domain.key;

       location /s3proxy/<mybucket> {
            rewrite /s3proxy/<mybucket>/(.*) /$1  break;
            proxy_pass https://s3_backend;
            proxy_redirect     off;
            proxy_http_version 1.1;
            proxy_set_header   "Connection" '';
            proxy_set_header Host <mybucket>.s3.amazonaws.com;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            error_log /var/log/nginx/ssl_debug.log debug;
    }
  }
}


But in the log /var/log/nginx/ssl_debug.log, I see SSL Handshake every time
when I request an S3 object via proxy using S3presigned URLs.

Below is the log I see every time for every request.

2023/09/30 18:07:19 [debug] 36#36: *9 event timer add: 22: 60000:721858477
2023/09/30 18:07:19 [debug] 36#36: *9 http finalize request: -4,
"/blob/zte3odk1ymnl at CIBC-2mb
/singleurl0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIASQOYQRM4CTRY6I54%2F20230930>
2023/09/30 18:07:19 [debug] 36#36: *9 http request count:2 blk:0
2023/09/30 18:07:19 [debug] 36#36: *9 http run request:
"/blob/zte3odk1ymnl at CIBC-2mb
/singleurl0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIASQOYQRM4CTRY6I54%2F20230930%2Fus-eas>
2023/09/30 18:07:19 [debug] 36#36: *9 http upstream check client, write
event:1, "/blob/zte3odk1ymnl at CIBC-2mb/singleurl0"
2023/09/30 18:07:19 [debug] 36#36: *9 http upstream request:
"/blob/zte3odk1ymnl at CIBC-2mb
/singleurl0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIASQOYQRM4CTRY6I54%2F20230930%2Fu>
2023/09/30 18:07:19 [debug] 36#36: *9 http upstream send request handler
2023/09/30 18:07:19 [debug] 36#36: *9 malloc: 000055ED330A1DD0:96
2023/09/30 18:07:19 [debug] 36#36: *9 upstream SSL server name: "s3_backend"
2023/09/30 18:07:19 [debug] 36#36: *9 set session: 0000000000000000
2023/09/30 18:07:19 [debug] 36#36: *9 tcp_nodelay
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 0
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 1
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 0
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 1
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 1
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: -1
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_get_error: 2
2023/09/30 18:07:19 [debug] 36#36: *9 SSL handshake handler: 0
2023/09/30 18:07:19 [debug] 36#36: *9 save session: 000055ED330FBAC0
2023/09/30 18:07:19 [debug] 36#36: *9 SSL_do_handshake: 1
2023/09/30 18:07:19 [debug] 36#36: *9 SSL: TLSv1.2, cipher:
"ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128)
Mac=AEAD"
2023/09/30 18:07:19 [debug] 36#36: *9 *http upstream ssl handshake*:
"/blob/zte3odk1ymnl at CIBC-2mb
/singleurl0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIASQOYQRM4CTRY6I54%2F202309>
2023/09/30 18:07:19 [debug] 36#36: *9 http upstream send request
2023/09/30 18:07:19 [debug] 36#36: *9 http upstream send request body

If I run 4K clients using a simulator,I will see 100% CPU in the nginx
container.I believe if we cache SSL sessions then SSL handshake for every
request will be avoided hence we may not have high CPU at nginx container.

Can you please help how to achieve SSL Cache? how to make sure the CPU is
not high? Is there any reason why the CPU is high other than SSL Handshake.
Thanks,
Vijay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20231001/d24cbc60/attachment.htm>