Not that I have used any of these lenses, but I'll try
TS-E is heavier
Using filters is more difficult (and slower to set up?)
16-35 is better for walk around and opportunistic shots while the lens is already on the camera (even if that is only 15% of your shots - that's still some shots)
You already have a TS-E - would the 17mm open up that many more opportunities?