Unsafe AIO under FreeBSD?

[Maxim Dounin]

Hello!

On Tue, Jan 17, 2023 at 12:04:31PM -0600, Mahmoud Al-Qudsi wrote:

> Hello all,
> 
> By default, FreeBSD restricts potentially unsafe AIO operations (as
> determined by the target fd type) and operations like aio_read(2) will
> “falsely” return EOPNOTSUPP to avoid a potentially dangerous operation
> that can result in blocking the aio threadpool hanging the system or the
> process, per aio(4).
> 
> I’ve observed in production with an nginx/1.23.3 instance (compiled with
> --with-file-aio) running on FreeBSD 13.1-RELEASE-p5, configured with
> `aio on;` (and `use kqueue;` though I suspect that is not relevant),
> the following syslog entry:
> 
>     pid 1125 (nginx) is attempting to use unsafe AIO requests - not
>     logging anymore
> 
> My curiosity got the best of me and I decided to allow unsafe aio
> requests to see what would happen (`sysctl vfs.aio.enable_unsafe=1`).
> It’s been about 24 hours and I haven’t noticed any ill or adverse
> effects, at least judging by my scrutiny of the logs, though I intend to
> continue to closely monitor this server and see what happens.
> 
> My question is whether or not nginx does anything “advanced” with aio
> under FreeBSD, beyond using aio for operations on “sockets, raw disk
> devices, and regular files on local filesystems,” which is the “safe”
> list, again per aio(4), while other types of fds are blocked unless
> unsafe aio is enabled.
> 
> On this server, nginx is serving static files from various zfs datasets
> and is functioning as a reverse proxy to http and fastcgi upstreams. I
> do have a few 3rd party modules statically compiled into nginx, so I'm
> naturally limiting my question to core/stock nginx behavior to the best
> of its developers’ knowledge :)
> 
> I don't have all the system logs but in a sample of the logs preserved
> going back to November 2022 the "unsafe AIO" is not repeated anywhere,
> leading me to _suspect_ that this isn't "normal" nginx behavior and that
> I probably should *not* be enabling unsafe AIO - but curiosity is a hell
> of a drug!

The only aio operation nginx uses is aio_read(), and it does 
nothing "advanced" - just reads normal files which are being 
served by nginx.

Further, nginx explicitly checks files being served, and rejects 
non-regular files.  As such, the "unsafe AIO" checks shouldn't be 
triggered unless you are trying to serve something from non-local 
file systems (well, you indeed shouldn't).

In general, if an aio_read() error happens, you should be able to 
see corresponding error in nginx error log at the "crit" level.  
The error will look like "[crit] ... aio_read("/path/to/file") 
failed (45: Operation not supported)".  It should make it possible 
to find out what actually causes the error.

-- 
Maxim Dounin
http://mdounin.ru/