SSL3_CTX_CTRL:called a function you should not call

[Maxim Dounin]

Hello!

On Tue, Mar 17, 2015 at 06:25:51AM -0400, rbqdg9 wrote:

> Maxim Dounin Wrote:
> -------------------------------------------------------
> > If you see problems with nginx 1.7.9, consider following hints 
> > at http://wiki.nginx.org/Debugging.
> I think it will not help (at least if not did by anyone who really knows
> both openssl and nginx internals).
> the problem is quickly traceable to 
> 
> long
> ssl3_ctx_callback_ctrl(SSL_CTX *ctx, int cmd, void (*fp)(void))
> {
>         CERT    *cert;
> 
>         cert = ctx->cert;
> 
>         switch (cmd) {
>         case SSL_CTRL_SET_TMP_RSA_CB:
>                 SSLerr(SSL_F_SSL3_CTX_CTRL,
> ERR_R_SHOULD_NOT_HAVE_BEEN_CALLED);
> (yes, this occurence, exactly)
> 
> inside libressl-2.1.3/ssl/s3_lib.c, and this function seems newer called by
> nginx code directly and not supposed to be externally-called at all.
> The pure openssl  have some pointer-magic in this place, dropped by libressl
> developers (with the data structure itself, so no easy way to bring it
> back)

I see no magic in the OpenSSL here.  It looks like the alert is 
due to LibreSSL dropped the support for export ciphers, while 
nginx calls SSL_CTX_set_tmp_rsa_callback() to be able to support 
them if configured to do so.  So, the alert is harmless and can be 
safely ignored.  It's just a result of LibreSSL dropping support 
for parts of the OpenSSL API nginx uses.

> I think the only thing developers may do (if not willing to really
> investigate and fix this issue) - just stop declaring nginx compatibility
> with libressl. It not only  nonworking, but worse - it cleanly execute some
> garbage instead of code.

The only thing we declaring is that nginx can be built with 
LibreSSL.  And it is going to work as long as LibreSSL does the 
right thing and don't deverge from the OpenSSL API too much.  We 
consider both LibreSSL and BoringSSL to be interesting 
experimental libraries, and plan to preserve at least minimal 
support as long as it doesn't require too much effort.

> (I have full system log of stack-protection mechanics trying to prevent
> this)
> 
> and yes, 1.7.10 still does the same. The problem itself does not appear on
> any connection, just in some special cases, but easely reproduceable.

So again:
http://wiki.nginx.org/Debugging

-- 
Maxim Dounin
http://nginx.org/