SSL3_CTX_CTRL:called a function you should not call

[rbqdg9]

Maxim Dounin Wrote:
-------------------------------------------------------
> If you see problems with nginx 1.7.9, consider following hints 
> at http://wiki.nginx.org/Debugging.
I think it will not help (at least if not did by anyone who really knows
both openssl and nginx internals).
the problem is quickly traceable to 

long
ssl3_ctx_callback_ctrl(SSL_CTX *ctx, int cmd, void (*fp)(void))
{
        CERT    *cert;

        cert = ctx->cert;

        switch (cmd) {
        case SSL_CTRL_SET_TMP_RSA_CB:
                SSLerr(SSL_F_SSL3_CTX_CTRL,
ERR_R_SHOULD_NOT_HAVE_BEEN_CALLED);
(yes, this occurence, exactly)

inside libressl-2.1.3/ssl/s3_lib.c, and this function seems newer called by
nginx code directly and not supposed to be externally-called at all.
The pure openssl  have some pointer-magic in this place, dropped by libressl
developers (with the data structure itself, so no easy way to bring it
back)

I think the only thing developers may do (if not willing to really
investigate and fix this issue) - just stop declaring nginx compatibility
with libressl. It not only  nonworking, but worse - it cleanly execute some
garbage instead of code.
(I have full system log of stack-protection mechanics trying to prevent
this)

and yes, 1.7.10 still does the same. The problem itself does not appear on
any connection, just in some special cases, but easely reproduceable.

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,256381,257313#msg-257313