fsync()-in webdav PUT

[Aziz Rozyev]

Valery, 

may you please suggest how you came to the conclusion that 

“fsync simply instructs OS to ensure consistency of a file"?

As far as understand simply instructing OS staff come at no cost, right?

> Without fsyncing file's data and metadata a client will receive a positive reply before data has reached the storage, thus leaving non-zero probability that states of two systems involved into a web transaction end up inconsistent.


I understand why one may need consistency, but doing so with fsyncing is non-sense.

Here is what man page says in that regard:


fsync()  transfers  ("flushes")  all  modified  in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent
       storage device) so that all changed information can be retrieved even after the system crashed or was rebooted.  This includes writing through or flushing a disk cache if present.  The call
       blocks until the device reports that the transfer has completed.  It also flushes metadata information associated with the file (see stat(2)).




br,
Aziz.





> On 28 Feb 2018, at 21:24, Valery Kholodkov <valery+nginxen at grid.net.ru> wrote:
> 
> It's completely clear why someone would need to flush file's data and metadata upon a WebDAV PUT operation. That is because many architectures expect a PUT operation to be completely settled before a reply is returned.
> 
> Without fsyncing file's data and metadata a client will receive a positive reply before data has reached the storage, thus leaving non-zero probability that states of two systems involved into a web transaction end up inconsistent.
> 
> Further, the exact moment when the data of certain specific file reaches the storage depends on numerous factors, for example, I/O contention. Consequently, the exact moment when the data of a file being uploaded reaches the storage can be only determined by executing fsync.
> 
> val
> 
> On 28-02-18 11:04, Aziz Rozyev wrote:
>> While it’s not clear why one may need to flush the data on each http operation,
>> I can imagine to what performance degradation that may lead of.
>> if it’s not a some kind of funny clustering among nodes, I wouldn't care much
>> where actual data is, RAM still should be much faster, than disk I/O.
>> br,
>> Aziz.
>>> On 28 Feb 2018, at 12:30, Nagy, Attila <bra at fsn.hu> wrote:
>>> 
>>> On 02/27/2018 02:24 PM, Maxim Dounin wrote:
>>>> 
>>>>> Now, that nginx supports running threads, are there plans to convert at
>>>>> least DAV PUTs into it's own thread(pool), so make it possible to do
>>>>> non-blocking (from nginx's event loop PoV) fsync on the uploaded file?
>>>> No, there are no such plans.
>>>> 
>>>> (Also, trying to do fsync() might not be the best idea even in
>>>> threads.  A reliable server might be a better option.)
>>>> 
>>> What do you mean by a reliable server?
>>> I want to make sure when the HTTP operation returns, the file is on the disk, not just in a buffer waiting for an indefinite amount of time to be flushed.
>>> This is what fsync is for.
>>> 
>>> Why doing this in a thread is not a good idea? It would'nt block nginx that way.
> 
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx