Well, partly it is aperture - f/2.8 and faster lenses let in enough light to activate the higher-sensitivity central autofocus point, meaning both more accurate and faster determination of proper focus.


But, mostly the autofocus speed is due to the lens design - and again, aperture plays a role here. Longer focal length lenses that have wide apertures mean more glass to move than a shorter, equally wide-aperture lens. That's because the f number is a ratio of focal length to aperture diameter (and the glass elements must be as wide as the aperture). A 200mm f/2 lens must have a 100mm diameter aperture (200/100=2) and thus glass elements that are also 100mm in diameter, whereas a 200mm f/2.8 lens only requires a 72mm diameter. More glass means slower to move. Zoom lenses have additional elements that must be moved to maintain parfocality (the lens remains in focus as it's zoomed), again meaning slower focusing.


Also, not all USMs are created equal. Older lenses (based on year of release) generally have slower USM motors, because technological advances have led to faster motors.


Finally, there are special cases, like the EF 85mm f/1.2L that you mention. Although that lens has a USM motor, it's also a front-focusing lens (the front of the lens extends during focusing). Internal or rear focusing lenses generally focus faster.