Yes, you can mmap() and munmap() as needed, but that makes things a lot less convenient - you'll still need to keep track of which data is on the disk and which data isn't, etc. At this point, you've already solved half of the problem of using malloc(), read() and free() instead of mmap() and munmap().
POSIX mandates a 32-bit int - UNIX hasn't ever run on a 16-bit machine (yes, I know there are some abominations.) Au contraire, 32-bit machines are still common; download one of the 'whole archive' files from http://popcon.debian.org/ and run:
$ awk '$2 ~ /^linux-image-/ && $2 ~ /86/ { SUM += $6; }; END {print SUM;}' by_recent
$ awk '$2 ~ /^linux-image-/ && $2 ~ /amd64/ { SUM += $6; }; END {print SUM;}' by_recent
I got 8174 for /86/ and 4420 for /amd64/. Admittedly, this is in no way representative of anything, but clearly people are still using i386 (note that I summed the 'recent' column!)
In short, i386 is not dead. And suggesting that it cannot handle "gigabytes of data" is absurd - Postfix/Dovecot, or PostgreSQL, or SAMBA, or pretty much whatever, can handle many, many gigabytes of data without issue on a i386 platform. Yes, you'll need to upgrade to 64-bit for the truly demanding tasks, but a database with a couple of (tens of) GB of data in it can work perfectly well on a scavenged i386, which is one of the good things of a unix-ish system.
(That said, you can trade off ease of implementation for being crippled on 32-bit platforms. But you are sacrificing something.)
> POSIX mandates a 32-bit int - UNIX hasn't ever run on a 16-bit machine (yes, I know there are some abominations.)
That's afaik absolutely not true, Unix was actually developed on 16 bit machines, back in the stone age of computing, roughly 1969 (that's why the unix 'epoch' starts in 1970).
After a while 32 bit machine became available and Unix was ported to them.
Unless you wish to call the earliest versions of Unix 'abominations, but I'm thinking you have Xenix and such in mind when you write that.
> you'll still need to keep track of which data is on the
disk and which data isn't
I'm not following this. As I see it, all you have to do is keep insure that no more than 3GB are mapped for your process at a given time. You still let the kernel decide what's faulted in and when things are written out.
Maybe my perspective is skewed: I'm viewing it from the perspective of real-time search, with large posting lists being frequently read and updated, and lazily written out to disk. Is there a different situation that is harder?
> In short, i386 is not dead. And suggesting that it cannot handle "gigabytes of data" is absurd
I didn't mean to imply it couldn't, or that it wasn't in common use. Rather I was wondering why I as a open source software author (http://wiki.apache.org/incubator/LucyProposal) would choose to make my life more complicated by targeting new software at a 32-bit platform at the expense of code simplicity.
Edit to add: I'm not trying to claim I'm definitively right. Rather, my point is that I care about the answers to these questions in practice as well as in theory. I worry about taking the wrong approach.
POSIX mandates a 32-bit int - UNIX hasn't ever run on a 16-bit machine (yes, I know there are some abominations.) Au contraire, 32-bit machines are still common; download one of the 'whole archive' files from http://popcon.debian.org/ and run:
I got 8174 for /86/ and 4420 for /amd64/. Admittedly, this is in no way representative of anything, but clearly people are still using i386 (note that I summed the 'recent' column!)In short, i386 is not dead. And suggesting that it cannot handle "gigabytes of data" is absurd - Postfix/Dovecot, or PostgreSQL, or SAMBA, or pretty much whatever, can handle many, many gigabytes of data without issue on a i386 platform. Yes, you'll need to upgrade to 64-bit for the truly demanding tasks, but a database with a couple of (tens of) GB of data in it can work perfectly well on a scavenged i386, which is one of the good things of a unix-ish system.
(That said, you can trade off ease of implementation for being crippled on 32-bit platforms. But you are sacrificing something.)