<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi,<br></div>Maybe this help.<br><a href="http://www.doublersolutions.com/docs/dce/osfdocs/htmls/develop/appdev/Appde193.htm">http://www.doublersolutions.com/docs/dce/osfdocs/htmls/develop/appdev/Appde193.htm</a><br><br>"One solution to the problem of calling <b>fork( )</b> in a multithreaded
environment exists. (Note that this method will not work for server
application code or any other application code that is
invoked by a callback from a library.) Before an application performs a <b>fork( )</b> followed by something other than <b>exec( )</b>, it must cancel all of the other threads. After it joins
the canceled threads, it can safely <b>fork( )</b> because it is the
only thread in existence. This means that libraries that create threads
must establish cancel handlers that propagate the
cancel to the created threads and join them. The application should
save enough state so that the threads can be recreated and restarted
after the <b>fork( )</b> processing completes. "<br><br></div><div>Best regards,<br></div><div>Ranier Vilela<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em qui, 13 de jun de 2019 às 22:09, Sinan Kaya <<a href="mailto:Okaya@kernel.org">Okaya@kernel.org</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I wanted to hear opinions on this surprising observation while<br>
developing an nginx module.<br>
<br>
We hit this issue while developing an nginx module. During nginx<br>
module's startup process, it starts up nginx but also does a lot of<br>
other work, some of which involves using glibc's implementation of<br>
timer_create, a posix function.<br>
<br>
<a href="https://code.woboq.org/userspace/glibc/sysdeps/unix/sysv/linux/timer_create.c.html" rel="noreferrer" target="_blank">https://code.woboq.org/userspace/glibc/sysdeps/unix/sysv/linux/timer_create.c.html</a><br>
<a href="https://code.woboq.org/userspace/glibc/sysdeps/unix/sysv/linux/timer_routines.c.html#__active_timer_sigev_thread_lock" rel="noreferrer" target="_blank">https://code.woboq.org/userspace/glibc/sysdeps/unix/sysv/linux/timer_routines.c.html#__active_timer_sigev_thread_lock</a><br>
<br>
<br>
Looking at the glibc source code for timer_create, we can see it has a<br>
global mutex named __active_timer_sigev_thread_lock. If nginx happens<br>
to call fork() while the rest of the nginx module code is calling<br>
timer_create, the fork() will make a copy of global memory which<br>
includes the global mutex as being owned by some other thread. In the<br>
newly-forked process, that thread will not exist, and the mutex will<br>
always be owned by a non-existent thread. I'm pasting in a GDB trace<br>
showing this below the rest of my explanatory text.<br>
<br>
<br>
This thread is blocked on timer_create. It is trying to enter a<br>
pthread_mutex_t which is a global named __active_timer_sigev_thread_lock.<br>
<br>
(gdb) bt<br>
<br>
#0 0x0000ff02aab1634c in __lll_lock_wait (<br>
futex=futex@entry=0xff02aaaee2c8 <__active_timer_sigev_thread_lock>,<br>
private=0) at /usr/src/debug/glibc/2.27-r0/git/nptl/lowlevellock.c:46<br>
<br>
#1 0x0000ff02aab0f648 in __GI___pthread_mutex_lock (<br>
mutex=mutex@entry=0xff02aaaee2c8 <__active_timer_sigev_thread_lock>)<br>
at /usr/src/debug/glibc/2.27-r0/git/nptl/pthread_mutex_lock.c:78<br>
<br>
#2 0x0000ff02aaadbb20 in timer_create (clock_id=<optimized out>,<br>
evp=0xff02a9e181a8, timerid=0xff02a41e43a0)<br>
at<br>
/usr/src/debug/glibc/2.27-r0/git/sysdeps/unix/sysv/linux/timer_create.c:159<br>
<br>
...<br>
<br>
#16 0x0000000000695db0 in ngx_thread_pool_cycle (data=0xff02a0027700)<br>
at<br>
/usr/src/debug/nginx-staticdev/1.14.2-r0/nginx-1.14.2/src/core/ngx_thread_pool.c:342<br>
#17 0x0000ff02aab0cf78 in start_thread (arg=0xff02a8e160d6)<br>
at /usr/src/debug/glibc/2.27-r0/git/nptl/pthread_create.c:463<br>
<br>
#18 0x0000ff02aa2efe2c in thread_start ()<br>
at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78<br>
<br>
<br>
By looking at this link below, almost majority of the C library<br>
functions are unsafe when called inside the nginx callback.<br>
<br>
One way to work around the issue is use using exec() functions so that<br>
each child process gets a fresh copy of the parent rather than getting<br>
a copy.<br>
<br>
<a href="https://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them" rel="noreferrer" target="_blank">https://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them</a><br>
<br>
"Library functions<br>
<br>
Problem with mutexes and critical code sections implies another<br>
non-obvious issue. It's theoretically possible to write your code<br>
executed in threads so that you are sure it's safe to call fork when<br>
such threads run but in practice there is one big problem: library<br>
functions. You're never sure if a library function you are using doesn't<br>
use global data. Even if it is thread safe, it may be achieved using<br>
mutexes internally. You are never sure. Even system library functions<br>
that are thread-safe may use locks internally."<br>
_______________________________________________<br>
nginx-devel mailing list<br>
<a href="mailto:nginx-devel@nginx.org" target="_blank">nginx-devel@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx-devel" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx-devel</a><br>
</blockquote></div>