I used to own both, however I have sold the f/4L IS.

The f/4 is super sharp, I really loved that lens. However I started shooting quite a few weddings and f/4 just wasn't wide enough for low lit receptions etc. The weight of the f/2.8 is significant compared to the f/4 however it's as sharp as the f/4 wide open, maybe even a bit sharper.

The f/2.8 is on my camera a lot, as was the f/4. I do really like the bokeh at 200mm and f/2.8 though, it's spectacular for portraits.

If you're using it a lot for hiking then you'd definitely appreciate the lighter weight of the f/4. Unless you've got a really good back pack such as one of the f-stop bags, in which case the weight isn't as big an issue.