Unsafe AIO under FreeBSD?

[Mahmoud Al-Qudsi]

Hello all,

By default, FreeBSD restricts potentially unsafe AIO operations (as
determined by the target fd type) and operations like aio_read(2) will
“falsely” return EOPNOTSUPP to avoid a potentially dangerous operation
that can result in blocking the aio threadpool hanging the system or the
process, per aio(4).

I’ve observed in production with an nginx/1.23.3 instance (compiled with
--with-file-aio) running on FreeBSD 13.1-RELEASE-p5, configured with
`aio on;` (and `use kqueue;` though I suspect that is not relevant),
the following syslog entry:

    pid 1125 (nginx) is attempting to use unsafe AIO requests - not
    logging anymore

My curiosity got the best of me and I decided to allow unsafe aio
requests to see what would happen (`sysctl vfs.aio.enable_unsafe=1`).
It’s been about 24 hours and I haven’t noticed any ill or adverse
effects, at least judging by my scrutiny of the logs, though I intend to
continue to closely monitor this server and see what happens.

My question is whether or not nginx does anything “advanced” with aio
under FreeBSD, beyond using aio for operations on “sockets, raw disk
devices, and regular files on local filesystems,” which is the “safe”
list, again per aio(4), while other types of fds are blocked unless
unsafe aio is enabled.

On this server, nginx is serving static files from various zfs datasets
and is functioning as a reverse proxy to http and fastcgi upstreams. I
do have a few 3rd party modules statically compiled into nginx, so I'm
naturally limiting my question to core/stock nginx behavior to the best
of its developers’ knowledge :)

I don't have all the system logs but in a sample of the logs preserved
going back to November 2022 the "unsafe AIO" is not repeated anywhere,
leading me to _suspect_ that this isn't "normal" nginx behavior and that
I probably should *not* be enabling unsafe AIO - but curiosity is a hell
of a drug!

Thanks,

Mahmoud Al-Qudsi
NeoSmart Technologies