I would so love to have the 17 TS-E, it's a fantastic lens even without the mechanics behind it. Between it and the 24, I have a hard time deciding, especially since I'm not sure they would replace my regular architectural lens. (!?)
Since I shoot interiors all the time, you'd think it would be the perfect lens, but it's really not. I'm extremely time limited when shooting virtual tours, so by the time I'd have myself set up for one shot, I'd be halfway through the house with my 17-40. I can't say this about the 16-35/4L IS just yet as there's no Adobe Lens Profile for it!
Anyway, I've found I've gotten so good at shooting with the UWA I barely need to correct it for verticals - I even amaze myself sometimes. I hand hold 90% of my shots, and 90% of those are as wide as I can possibly get. Your typical realtor wants it to look like you can roller skate in the bathroom. I also have the Rokinon 14/2.8, but haven't started using it yet although I should. It's just sort of an interruption to my workflow if I plop in a different lens.
These pics were using the 17-40. Although I have no problem (mostly) keeping my verticals straight, eliminating the last little bit of distortion can be important (less on the verticals than horizontal), so I'm chomping at the bit for the Lens Profile!
The shots below were on a tripod - one of the few times I do this, but I was doing different exposures so we could see the bay - and the place was gigantic so I wanted consistent height in the virtual tour.
In this really cool Eichler shot you can see my reflection (oops!) as I squat and hand-hold. There was no HDR or anything else here, I never do that for tours.
This was using the Rokinon - I was testing the available 5D profile, which I modified to work in RAW instead of just JPEG. You can see the impressive verticals all the way up to the very top, where it starts to lose it. I don't have this problem with the 17-40. Can't speak for the 16-35/4 just yet! I think with something this wide that's not too bad. So yes, I shoot as wide as I can all the time, where you say you don't. Hmmmm.
So, see how I didn't answer your question at all?
Why would I want to talk you out of the TS-E? Are you nuts? If you can afford it, I see no reason not to have it. However, I would certainly never be without something like the 16-35/4L IS. You can whip your camera right out of the bag and start shooting, with no thought as to initial setup or anything else. The TS-E is a different sort of animal entirely. The 16-35 is like a tasty sandwich at your favorite beach-side grill. The TS-E is like a 4 course meal at an expensive restaurant with an astounding panoramic view. You might have the sandwich more often and enjoy it every time, where the 4 course meal won't happen as often but is something to be savored and remembered. You plan for it and dress accordingly.
Let us know how it works out for you.