I guess it comes down to personal preference. The 24-70mm f/2.8L is a nice lens, but I haven't used it very much at all for nearly a year now because I am a huge fan of really narrow depth of field, and for me the 24-70mm (even wide open at f/2.8) didn't deliver the depth of field I wanted. Don't get me wrong, you can still get good subject isolation from it (particularly with the 5D2), just not to the level that I like. Whereas at 200mm and f/2.8 aperture you can really separate a subject from its background.
For general walkaround style photography then the 24-70mm is most certainly a good lens, and would also lean itself toward landscape photography more so than the 70-200mm. 24mm on full frame is wide enough for most landscape work.
So yes, the 24-70mm would potentially suit you very well but it just doesn't quite suit the style of photography that I like to work towards any more.
Here's a couple of older examples of shots I've taken with the 5D2 + 24-70mm f/2.8 combo. Since then I've changed my style a little. It can definitely produce some great results. These probably don't illustrate my point about depth of field very well but it might give you an idea of what the lens is capable of.
27mm @ f/11:
Nay and Jono by Ben__Taylor, on Flickr
24mm @ f/9.0
Simpson Desert Series # 3 by Ben__Taylor, on Flickr
24mm @ f/8.0 (cropped)
Moo Juice! by Ben__Taylor, on Flickr
24mm @ f/2.8
Cleveland Pt by Ben__Taylor, on Flickr
50mm @ f/3.5
Cuteness... by Ben__Taylor, on Flickr
Here's a couple with the 5D2 and 70-200mm f/2.8L IS II
200mm @ f/2.8
Buff Sided Robin by Ben__Taylor, on Flickr
200mm @ f/2.8
Eastern Brown Snake by Ben__Taylor, on Flickr
I hope this helps you anyway. It's hard to describe sometimes why I chose one lens over the other, some are just my "favourites" and others are there because at some time they might come in handy.
Ben