The concept of "privilege" as used in social justice speak is merely a tool used to discredit people who are in the wrong demographic/social/cultural categories without having to address the content of their arguments. And pointing this out is labeled "derailing", to avoid having to address that line of argument either.
I'm sure someone, somewhere has used the concept of privilege to unfairly discredit another person's opinion, but I think you're trying to apply cold, hard reasoning to something that requires deeper thought. Privilege, when used in social justice, refers to an area where one person has an advantage and another person belonging to a different group does not. If you need help understanding the concept, take a few minutes to read this excellent post on the subject: https://sindeloke.wordpress.com/2010/01/13/37/
This doesn't mean a white person can never talk about race. It means they need to understand they're coming from an entirely different perspective— a perspective that has the upper hand. Social justice covers topics that are inherently complex and might not conform to strict models and theories. This means where an argument comes from has a lot of bearing on its validity. That white person from earlier can participate, but they have to understand they've been living in a world where they get a head start, and often times, they have no idea when that world is giving it to them.
Anyway, that was a bit of a rant, so please read the blog post I linked to get a better idea.
