Yes. I'd also recommend Inversions only after you've read some of the others - and start with Consider Phelbas (which for non-readers, is told from the perspective of a non-Culture citizen).
"Consider Phlebas" has its merits, especially the outsider's POV of the Culture. But I'm part of the faction that feels The Player of Games is a better introduction, as it's more polished and shows off the Culture itself more.
Hmm - I like TPOG, but I can't say I enjoyed it or had it give me as much perspective on the Culture as Phlebas did - but I know its a favorite for a lot of people. Partially because I was more interested in the Changer as a character than the gameplayer.
Strictly speaking, so is most of Use of Weapons and The Hydrogen Sonata, as well as significant portions of Look to Windward, Matter, and Surface Detail.
In fact, the more I think about it, the more I'm realizing that one of his more common means of telling us about the Culture was through the eyes of non-Culture citizens.
You're right - but Phelbus stands out to me because you get almost no internal culture perspective - you are almost entirely embedded with a person who does not simply dislike the culture, but is actively working towards its destruction. Its a bold choice given the Culture was (in Banks' own words) the closest thing to a Utopian ideal.
I'd recommend reading the Culture series (3 of the above mentioned books are part of it) in order if possible [0], they all have something to offer.
By the way, I'd recommend against reading any reviews for his books prior, they're easily spoiled.
0: http://en.wikipedia.org/wiki/Culture_series