Issues building Nginx using boringssl

[Jeffrey Walton]

On Tue, Feb 20, 2024 at 10:19 PM Jinze YANG <rttwyjz at gmail.com> wrote:
>
> After I built libssl as a shared library, the compilation could be completed normally, but I encountered some problems after compilation. The details are as follows:
> root at VM-8-12-debian /www/server/nginx/sbin # ./nginx -t
> ./nginx: symbol lookup error: ./nginx: undefined symbol: SSL_library_init
> root at VM-8-12-debian /www/server/nginx/sbin # ./nginx -V
> nginx version: nginx/1.25.4
> built by gcc 12.2.0 (Debian 12.2.0-14)
> built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with OpenSSL 3.0.11 19 Sep 2023)
> TLS SNI support enabled
> configure arguments: --user=www --group=www --prefix=/www/server/nginx --with-pcre --add-module=/root/ngx_brotli --with-http_v2_module --with-stream --with-stream_ssl_module --with-http_ssl_module --with-http_gzip_static_module --with-http_gunzip_module --with-http_sub_module --with-http_flv_module --with-http_addition_module --with-http_realip_module --with-http_mp4_module --with-ld-opt=-Wl,-E --with-cc-opt=-Wno-error --with-ld-opt=-ljemalloc --with-http_dav_module --with-http_v3_module --with-cc-opt=-I/root/boringssl/include --with-ld-opt='-L/root/boringssl/build/ssl -L/root/boringssl/build/crypto -Wl,-rpath=/root/boringssl/build/ssl -Wl,-rpath=/root/boringssl/build/crypto -Wl,--enable-new-dtags'

This is kind of interesting in a morbid sort of way:

    undefined symbol: SSL_library_init

That's the old way to initialize OpenSSL. It is available in OpenSSL
1.0.2 and below. Does BoringSSL also use it? Also see
<https://wiki.openssl.org/index.php/Library_Initialization>.

Nowadays you should be initializing OpenSSL with OPENSSL_init_ssl()
and possibly OPENSSL_init_crypto(). Does BoringSSL also do it that way
nowadays? Also see
<https://www.openssl.org/docs/manmaster/man3/OPENSSL_init_ssl.html>

To see which libraries nginx is loading, issue the following. You
should see the output detail the libraries you expect from
/root/boringssl/build/ssl/libssl.so and
/root/boringssl/build/crypto/libcrypto.so (my output is from a distro
provided installation):

   $ ldd $(command -v nginx)
       linux-vdso.so.1 (0x00007ffc94bf8000)
       libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f05d0e33000)
       libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f05d0d98000)
       libssl.so.3 => /lib64/libssl.so.3 (0x00007f05d0cf5000)
       libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f05d0800000)
       libz.so.1 => /lib64/libz.so.1 (0x00007f05d0cdb000)
       libprofiler.so.0 => /lib64/libprofiler.so.0 (0x00007f05d07e8000)
       libc.so.6 => /lib64/libc.so.6 (0x00007f05d0606000)
       libunwind.so.8 => /lib64/libunwind.so.8 (0x00007f05d05ec000)
       libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f05d0200000)
       libm.so.6 => /lib64/libm.so.6 (0x00007f05d050b000)
       libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f05d04e7000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f05d0fda000)

I believe OPENSSL_init_ssl is part of libssl.so. You should be able to
verify the symbol is exported:

   $ nm -D /lib64/libssl.so.3 | grep ' T ' | grep OPENSSL_init
   00000000000309d0 T OPENSSL_init_ssl@@OPENSSL_3.0.0

Grepping for the capital ' T ' is important. It means you are grepping
for symbols that are defined, and not including undefined symbols:

    $ nm -D /lib64/libssl.so.3 | grep OPENSSL_init
                    U OPENSSL_init_crypto at OPENSSL_3.0.0
    00000000000309d0 T OPENSSL_init_ssl@@OPENSSL_3.0.0

And SSL_library_init is not present because my distro provides OpenSSL 3.0:

    $ nm -D /lib64/libssl.so.3 | grep SSL_library_init
    $

So it sounds like BoringSSL is doing something different than modern
OpenSSL. Or you are compiling and then runtime linking against
different versions of the libraries.

Jeff