On test-suite coverage

[Maxim Konovalov]

Hi Hugo,

This is great to see your interest to nginx tests.

On 14.01.2022 18:45, Hugo Lefeuvre wrote:
> Hello!
> 
> As part of a research project, I measured the coverage achieved by Nginx's
> test-suite using gcov. Taking a look at the results, my colleagues and
> myself were somewhat surprised to realize that the coverage capped at about
> 70% line coverage and 81% function coverage; we expected something closer
> to 90% line coverage.  core/nginx.c for example only gets 268 lines / 584

This is more or less in line with our numbers.  Just from today report 
for mainline: 77.8% for lines coverage and 93.1% for functions.  These 
figures include njs code though.

> covered. Similarly, more than half of core/ngx_resover.c (1042 lines /
> 2138) is not covered.
>  > I was wondering if I did something wrong in my measurements, if this is a
> known weakness of the test-suite, and in the latter case, if this is
> something that the Nginx project is open to receiving contributions on.
> 
Probably not a weakness but unjustified expectations? :-)

I think the biggest non-covered part comes from various errors paths.
Some of them could be hard to trigger.  For example, for memory
allocation failures we use separate nodes with modified ngx_palloc.c
which enables random memory allocation errors.  This is not a part of
the standard test suite though.

Anyway, any meaningful contributions in this area will be highly 
appreciated.

http://nginx.org/en/docs/contributing_changes.html

There are several things to keep in mind: we want to keep the suite 
compact as it is now and be able to have it integrated into CI/CD 
systems easily.

Thanks,

Maxim

-- 
Maxim Konovalov