slow connection on SSL port (TTFB)

[neomaq]

Hello
there is a problem:
slow connection to nginx server

telnet server 443
1-8 random sec before TTFB

all possible network stack tunings are applied, similar problems are not
observed on other(non nginx) ports

32 vCPU   Intel(R) Xeon(R) CPU E5-2630 v4 
96 GB RAM
avg CPU load -20%
1 GB network (tested on local internal network)

there are over 1400 virtual hosts with SSL
the problem is observed during busy hours

nginx:
user www-data;
worker_processes 64;
pid /run/nginx.pid;
worker_rlimit_nofile 16384;
events {
        use epoll;
        worker_connections 16384;
        multi_accept on;}
http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_names_hash_max_size 524280;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
}
----------------------------------------
there are 5-15K  ESTANLISHED connections and over 17K open/TIME_WAIT ports

What can be done to reduce the connection time to the server?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285142,285142#msg-285142