Test the OOM situation

[Maxim Dounin]

Hello!

On Wed, May 19, 2010 at 04:01:07PM +0400, Peter Leonov wrote:

> On 18.05.2010, at 23:26, Maxim Dounin wrote:
> 
> > Hello!
> > 
> > On Tue, May 18, 2010 at 08:25:46PM +0400, Peter Leonov wrote:
> > 
> >> Hello devels,
> >> 
> >> There is a module that requires a heavy testing in environment with
> >> low memory. Especialy it could be good to test how it goes if no
> >> more memory is available in the system.
> >> 
> >> So the questions are: is there a way to test such a module without a
> >> virtual machines? and are there some tools which might be of help?
> > 
> > You may tune memory limit for a given process instead of using 
> > virtual machine.  Though this won't really help with testing nginx 
> > modules as pool allocator will protect most of allocations from 
> > seeing errors (while they still can happen with sligtly different 
> > config).
> 
> You mean that the module will be even left untouched as far
> as the main request pool fail to be allocated. Understood.

Not really.  I mean that most allocations will be satisfied from 
already allocated pool's block and hence will never fail.  Only 
allocations happen to require additional pool block may fail.

> > Injecting random errors into allocation routines proven to be much 
> > more usable.  Just for completeness: attached trivial patch which 
> > I used while searching for allocation error handling bugs in nginx 
> > itself.
> 
> That is near what I was looking for :)
> 
> But it gives a very interesting results. For example a crash within the core:
> 
> 40740 Bus error               (core dumped)
> 
> #0  0x0000484b in ngx_destroy_pool (pool=0x3262b0) at src/core/ngx_palloc.c:74
> 74	    for (p = pool, n = pool->d.next; /* void */; p = n, n = n->d.next) {
> 
> #0  0x0000484b in ngx_destroy_pool (pool=0x3262b0) at src/core/ngx_palloc.c:74
> #1  0x0002cd16 in ngx_http_close_connection (c=0x879e58) at src/http/ngx_http_request.c:3020
> #2  0x0002f56a in ngx_http_set_keepalive [inlined] () at /Users/peter/pro/nginx/src/http/ngx_http_request.c:2559
> #3  0x0002f56a in ngx_http_finalize_connection (r=<value temporarily unavailable, due to optimizations>) at src/http/ngx_http_request.c:2121
> #4  0x0002ff66 in ngx_http_finalize_request (r=0x807c00, rc=0) at src/http/ngx_http_request.c:2029
> #5  0x0003068a in ngx_http_writer (r=0x807c00) at src/http/ngx_http_request.c:2247
> #6  0x0002ca70 in ngx_http_run_posted_requests (c=0x879e58) at src/http/ngx_http_request.c:1812
> #7  0x0003eb06 in ngx_http_upstream_handler (ev=0x88fe78) at src/http/ngx_http_upstream.c:879
> #8  0x00022ebe in ngx_kqueue_process_events (cycle=0x813628, timer=2990, flags=<value temporarily unavailable, due to optimizations>) at src/event/modules/ngx_kqueue_module.c:683
> #9  0x00018d85 in ngx_process_events_and_timers (cycle=0x813628) at src/event/ngx_event.c:245
> #10 0x0001fb44 in ngx_single_process_cycle (cycle=0x813628) at src/os/unix/ngx_process_cycle.c:306
> #11 0x00003c5c in main (argc=7, argv=0xbffff558) at src/core/nginx.c:393

This may happen due to bugs in a module as well.  If you are able 
to reproduce the problem without non-official modules/patches - 
this probably worth further investigation.  You may want to try to 
reproduce the problem without optimization and/or share a way to 
reproduce it.

Note well: last time I've tested nginx with these patches heavily 
was before 0.8.0.  But I don't see any immediate segfaults/bus 
errors while running test suite against 0.8.37 with these patches.

Maxim Dounin