I just got back from Japan. Obviously it depends on your style, but what I found myself using the most often was 35-40mm equivalent (for street photography) and "as wide as possible" (for cityscape and landscape). Unfortunately for me that was 29mm equivalent, but I'm assuming since you have all that L glass you have a full frame body as well. In that case, I recommend the 17-40 and the 35 for day and night, respectively, if you want to save carry weight all day. If you do a lot of telephoto stuff (not my style), add the 135. Could be useful if you go places that you can't get up close, of which there are a decent number.

For what it's worth, packing and daily carrying are a totally different issue. I packed my nifty fifty but ended up using the 18-55 90+% of the time and only swapping to the 50mm at night, and never carrying both lenses with me. It was great to feel unburdened and I don't really feel like I "missed" any shots I would have gotten by carrying both at once.

Have fun in Japan!