In Adam Smith's time, the biggest rentier of them all was the Imperial era British government along with its patrons, titled members of the nobility and the landed gentry. Even in Smith's time, free markets did mean freedom from government interference.
Land is big deal. Value of land comes mostly from it's surrounding infrastructure. Taxing it is one of the least harmful taxes.