Personally, I would prefer the lens you already have (17-55 f/2.8 IS) as a general purpose lens to the 24-105 f/4 IS. Okay, the 24-105 has more reach. But I would think the wider angle and smaller f/number of the 17-55 would make it a better walk around lens. Can you remind me why you want the 24-105.
For photos of toddlers, I like the 70-200 range. I think f/4 would be good enough most of the time for this purpose if you like tight protraits, because children are small (so you need to stop down more to get them all in focus than you would for a similar shot of an adult... most people don't take this into account. On the other hand, if you want to do full body shots, though, f/2.8 will be handy at least some of the time. Also, since you mentioned that you don't shoot one type of subject in particular, I think you might want to err on the side of versatility and go for the f/2.8.
Some will say the fast prime is ideal for indoor pics of kids, and sometimes I do use them. But lately I've been leaning toward pics in the f/4 (which would be sort of like f/2.8 on a crop) range with IS (which would favor the 70-200 over the 35mm f/1.4). Your subject must be still for this to work, but at least you have enough DOF to get more than one eye in focus. All depends on what style picture you want to take, I guess.
I think I just babbled instead of giving you an answer.