"...Next, the reason why fsync() has the behaviour that it does is one
ofhe the most common cases of I/O storage errors in buffered use
cases, certainly as seen by the community distros, is the user who
pulls out USB stick while it is in use. In that case, if there are
dirtied pages in the page cache, the question is what can you do?
Sooner or later the writes will time out, and if you leave the pages
dirty, then it effectively becomes a permanent memory leak. You can't
unmount the file system --- that requires writing out all of the pages
such that the dirty bit is turned off. And if you don't clear the
dirty bit on an I/O error, then they can never be cleaned. You can't
even re-insert the USB stick; the re-inserted USB stick will get a new
block device. Worse, when the USB stick was pulled, it will have
suffered a power drop, and see above about what could happen after a
power drop for non-power fail certified flash devices --- it goes
double for the cheap sh*t USB sticks found in the checkout aisle of
Micro Center.
So this is the explanation for why Linux handles I/O errors by
clearing the dirty bit after reporting the error up to user space.
And why there is not eagerness to solve the problem simply by "don't
clear the dirty bit". For every one Postgres installation that might
have a better recover after an I/O error, there's probably a thousand
clueless Fedora and Ubuntu users who will have a much worse user
experience after a USB stick pull happens.
I can think of things that could be done --- for example, it could be
switchable on a per-block device basis (or maybe a per-mount basis)
whether or not the dirty bit gets cleared after the error is reported
to userspace. And perhaps there could be a new unmount flag that
causes all dirty pages to be wiped out, which could be used to recover
after a permanent loss of the block device. But the question is who
is going to invest the time to make these changes? If there is a
company who is willing to pay to comission this work, it's almost
certainly soluble..."
and this part of the paragraph:
"...But again, of the companies who have client code where we
care about robustness and proper handling of failed disk drives, and
which have a kernel team on staff, pretty much all of the ones I can
think of (e.g., Oracle, Google, etc.) use O_DIRECT and they don't try
to make buffered writes and error reporting via fsync(2) work well..."
I conclude that:
1) Fixing fsync() would cause other issues and might be technically challenging,
and some within Linux community dont have it as their highest priority
2) The onus falls mostly mostly on Postgres team, to re-implement their code
so as to use the "better" technical solution, as implemented for Linux
for similar scenarios by Oracle, Google and others...
> In that case, if there are dirtied pages in the page cache, the question is what can you do? Sooner or later the writes will time out, and if you leave the pages dirty, then it effectively becomes a permanent memory leak.
FreeBSD keeps the dirty buffer (always marked as dirty) in memory as long as the device exists:
Honestly even if a device disappears it's probably worth keeping around those pending writes for several seconds. It might come back, and if it comes back fast enough it probably hasn't been altered by any other machine.
Super interesting (I hadn't seen it before), thanks for sharing! I feel like there might be lots of potential solutions they're ignoring though.
> Sooner or later the writes will time out, and if you leave the pages dirty, then it effectively becomes a permanent memory leak.
Why not just get rid of the pages after the underlying medium is removed? Especially since, after that point, you wouldn't be able to guarantee much about the device's contents anyway, even if it were plugged back in.
Also, even if I buy that this is a trade-off with a permanent memory leak (which I don't, for lots of reasons like the above), isn't that better than outright corruption...? At least you can reboot to get rid of a leak. There's no guarantee you can get back corrupted data!
Looking at it from another side...The whole Direct I/O path looks like a worst can of worms...maybe it's easier to review again fsync :-)
"...The exact semantics of Direct I/O (O_DIRECT) are not well specified. It is not a part of POSIX, or SUS, or any other formal standards specification. The exact meaning of O_DIRECT has historically been negotiated in non-public discussions between powerful enterprise database companies and proprietary Unix systems, and its behaviour has generally been passed down as oral lore rather than as a formal set of requirements and specifications..."
"The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances." -- Linus
Or (3) companies that make $$$ or save $$$ from using Postgres could hire kernel developers to implement a better solution for buffered I/O. The problem is that the companies who are serious about I/O recovery use Direct I/O, so engineers that are employed by those companies have plenty of other improvements such as io_uring, improving general storage performance, adding support for inline encryption engines to improve performance on mobile devices, etc., etc., etc.
People seem to forget that Open Source does not mean that users get to demand that unpaid volunteers will magically do work for their pet feature requests. It just means that the source code is available and people are free to improve the code to make it better fit their use case. A proprietary OS is like a car whose hood is welded shut, and only the dealer is allowed to service it. An open source OS means that you can take the car to whomever you like, or even service the car, or improve the car, yourself. It does not mean that you get to have service or improvements to your car engine for free.
The other thing to note here is that Postgres was issuing fsync(2) calls from different processes, and some of those processes were ignoring the error return from fsync(2). If there is an I/O error, fsync(2) will tell userspace about it. However, there is nothing in POSIX which states that once a file has an I/O error associated with it, the fsync(2) system call will return errors forever and ever, Amen. So Postgres was being a bit dodgy with error returns as well, and was demanding that something that POSIX clearly never promised.
1) Based on this https://wiki.postgresql.org/wiki/Fsync_Errors and this https://lwn.net/Articles/752105/
particularly this justification:
"...Next, the reason why fsync() has the behaviour that it does is one ofhe the most common cases of I/O storage errors in buffered use cases, certainly as seen by the community distros, is the user who pulls out USB stick while it is in use. In that case, if there are dirtied pages in the page cache, the question is what can you do? Sooner or later the writes will time out, and if you leave the pages dirty, then it effectively becomes a permanent memory leak. You can't unmount the file system --- that requires writing out all of the pages such that the dirty bit is turned off. And if you don't clear the dirty bit on an I/O error, then they can never be cleaned. You can't even re-insert the USB stick; the re-inserted USB stick will get a new block device. Worse, when the USB stick was pulled, it will have suffered a power drop, and see above about what could happen after a power drop for non-power fail certified flash devices --- it goes double for the cheap sh*t USB sticks found in the checkout aisle of Micro Center.
So this is the explanation for why Linux handles I/O errors by clearing the dirty bit after reporting the error up to user space. And why there is not eagerness to solve the problem simply by "don't clear the dirty bit". For every one Postgres installation that might have a better recover after an I/O error, there's probably a thousand clueless Fedora and Ubuntu users who will have a much worse user experience after a USB stick pull happens.
I can think of things that could be done --- for example, it could be switchable on a per-block device basis (or maybe a per-mount basis) whether or not the dirty bit gets cleared after the error is reported to userspace. And perhaps there could be a new unmount flag that causes all dirty pages to be wiped out, which could be used to recover after a permanent loss of the block device. But the question is who is going to invest the time to make these changes? If there is a company who is willing to pay to comission this work, it's almost certainly soluble..."
and this part of the paragraph:
"...But again, of the companies who have client code where we care about robustness and proper handling of failed disk drives, and which have a kernel team on staff, pretty much all of the ones I can think of (e.g., Oracle, Google, etc.) use O_DIRECT and they don't try to make buffered writes and error reporting via fsync(2) work well..."
I conclude that:
1) Fixing fsync() would cause other issues and might be technically challenging, and some within Linux community dont have it as their highest priority
2) The onus falls mostly mostly on Postgres team, to re-implement their code so as to use the "better" technical solution, as implemented for Linux for similar scenarios by Oracle, Google and others...