Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You're quoting a sentence fragment of a non-normative section of the spec, in other words you're half quoting a part of the spec that is not intended to be interpreted formally, literally, or even required to be adhered to. The rationale section of the POSIX spec is strictly informative.

The fact that you left out the part of the sentence that says "The fsync() function is intended to..." suggests that you may be intentionally hiding this fact to support your point.

Even if we go by the non-normative text, that section explicitly states that the spec allows for, and I quote "It is explicitly intended that a null implementation is permitted."

At any rate, the normative section of the spec states only the following:

"The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes. The nature of the transfer is implementation-defined."

Emphasis mine.

It would be unwise to dismiss the wealth of resources that have been provided to you in other comments to better understand this issue, all of them quite extensive and written by experts on this subject, just because of an informal section of the standard



Setting aside the appeal to authority and the outright condescension, POLA still can't be dispelled using a "nasal demons" argument.

In practice the intentional vagueness is indistinguishable from obfuscation, and doing so to please every vendor for conformance's sake is straightforward intellectual dishonesty. In this regard, the informational section is tantamount to an extended reprimand from the editors.

If you really want to trade language-lawyer broadsides, then leaning on the phrase "implementation-defined" was a tragic mistake, because this makes the Linux documentation normative for the circumstances, yet the behaviour occurring is not described in the Linux fsync(2) manpage, nor are there cross-references to anything FS-specific. It gets worse: the engineering solution to this schmozzle is switch to direct IO, but the documentation of Linux's open(2) then describes the behaviour of O_SYNC|O_DIRECT in terms of fsync(2) (and similarly for O_DSYNC). All this simply highlights that the whole line of thinking is an unproductive pissing contest; the root cause of the postgres issue is vendor politics.


The nature of the transfer is implementation-defined.

Does "nature of the transfer" cover "the extent of the data set"? To me, the nature of the transfer refers to the technical implementation of the writeout mechanism. Especially given that the rationale explicitly mentions "since the concepts of "buffer cache", "system crash", "physical write", and "non-volatile storage" are not defined here, the wording has to be more abstract" it seems to me that "nature of the transfer" should not be interpreted as "the data to be written out".

But even going by the formal Description only, it does not follow to me that an implementation can choose to selectively ignore part of the data. These are the phrases used in that section:

- The fsync() function shall request that all data for the open file descriptor [..] to be transferred to the storage device

- [It] shall force all currently queued I/O operations [..] to the synchronized I/O completion state

- All I/O operations shall be completed

Each sentence contains the word "all".

The return value description reads "Upon successful completion, fsync() shall return 0". Based on this thread, the Linux kernel should never return 0 if a previous fsync failed, because the kernel has no intention of honouring the request.


Leaving aside that the normative part of the spec is still very clear that fsync() applies to all data for the open file descriptor, your line of reasoning fails for another reason. If we accept that fsync() as implemented in Linux is correct, what is its use? Is there any way to use fsync() "correctly" in the presence of IO errors?

As far as I understand, the answer is "definitely not" on Linux before 4.13, since failed async writes to the file could clear the dirty bit long before you called fsync(). There are some improvements since then, so arguably fsync() can now be used for something meaningful, though you have to take great care, and the only recourse on error is probably to start rewriting the file from scratch, even for transient errors (since the buffers may or may not be lost after the error, but future calls to fsync() will report success regardless).

Whether this behavior is a good idea or not is the only question that matters, since POSIX compliance hasn't mattered to Linux anyway for who knows how many years.


> The fact that you left out the part of the sentence that says "The fsync() function is intended to..." suggests that you may be intentionally hiding this fact to support your point.

No, that was not my intention. I just started the sentence "the job of fsync() is" and then went and looked for the portion of the quote that fit. I don't see a semantic difference, but I edited that back in, since you think I'm pulling a fast one on you.

> You're quoting a sentence fragment of a non-normative section of the spec, in other words you're half quoting a part of the spec that is not intended to be interpreted formally, literally, or even required to be adhered to. The rationale section of the POSIX spec is strictly informative.

Would be nice if you could include which part of the standard claims the rationale is non-normative. Moreover, the justifications against it are themselves based on "history" rather than anything in the text, so even if we assume it's non-normative, you can't really use this as an excuse to dismiss it as if it's somehow less valid than arbitrary tidbits from history. At that point you could try to agree that the (purportedly) non-normative text in fact does contradict the current behavior, but argue that you should have the latitude to ignore it. That's quite a bit different from denying its meaning in the first place.

> "The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes. The nature of the transfer is implementation-defined."

The "nature of the transfer" is implementation-defined, yes. They don't care if you do it all at once, in parallel or sequentially, in bursts or uniformly, by boat or by rocket, or whatever. The question of which data is to be transferred is completely unrelated to that. If I say "please take my family to the airport, the nature of the transportation is up to you", you don't get to leave half my family behind and then justify it by claiming I left the choice of family members up to you. I did not leave that up to you. I just said you can choose the nature of the transport. Not the whether or whom of the transport!

> It would be unwise to dismiss the wealth of resources that have been provided to you in other comments to better understand this issue, all of them quite extensive and written by experts on this subject, just because of an informal section of the standard

If experts claimed the standard was clear but perhaps inconsistent or self-contradictory, that would be very different than what's going on right now. Or if they argued "we agree this behavior is actually inconsistent with the rationale, but we don't believe the rationale is normative", then that would also be a different scenario. But when people are just saying the standard says something other than what it actually says, what am I supposed to do? Agree with them because they're "experts", despite the fact that the standard says the opposite in plain English?


> The fact that you left out the part of the sentence that says "The fsync() function is intended to..." suggests that you may be intentionally hiding this fact to support your point.

It's very difficult to read this accusation as having been made in good faith, since the person you're accusing of intentionally omitting part of a quote already quoted that very text in their previous comment:

>>>> The tl;dr (and blunt truth) is that Linux doesn't implement POSIX's semantics of fsync() correctly. The standard says "the fsync() function is intended to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk".


>...the person you're accusing of intentionally omitting part of a quote already quoted that very text in their previous comment:

That was edited back in after I made my comment (this is admitted to by OP themselves).

To the extent that I make an accusation it's that it's absurd to think you can dismiss the wealth of information on this subject and come to a remotely comprehensive understanding of how to implement reliable IO functions by simply reading a couple of sentences of the POSIX spec and dismiss the numerous other sources that go into very fine detail on this subject.

This is not a subject you just read a few sentences on and then make bold claims about how Linux doesn't implement fsync properly, implying that the experts who have gone to great lengths to implement fsync on Linux just don't understand it; that all the papers on this subject, the numerous mailing list posts, the very submission itself, can all readily be dismissed as a simple failure to read what is as clear as day as if none of the people discussing this issue have ever bothered to read the spec.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: