The short of it is that fsync on Linux used to be really bad at reporting errors (POSIX APIs are still bad, but we've now sorted out some ways to work around this particular source of pain)

IIRC, after an IO error you could get into a situation where on the next fsync postgres thought things were back on track, but Linux had actually just thrown away some data.