I'm sure that depends on how you define "performance" -- for me, the greatest benefit is in hand comfort, not speed.

Switching between Qwerty and Colemak for any of the reasons you mention has never been a problem for me either. Obviously, going from Colemak to Qwerty is going to be inconvenient for the sole reason that Qwerty is inconvenient, but that's only because you become aware of how much you contort your fingers unnecessarily once you've been typing for a while not doing that.

I agree keybindings can be an issue, though. For me, personally, it's minor compared to everything else, though.