Lots of those people complaining about population decline aren't really talking about the absolute number. They're dog-whistling about demographic replacement - the idea that minorities are reproducing too fast and whites are reproducing too slow.