Several factors influence AF speed.
  1. Optical design of the lens. A lens whose focusing group is large/heavy (e.g. EF 85/1.2L II), or whose optical formula is such that focusing requires a longer travel (EF 100/2.8L macro IS), will focus more slowly.
  2. Maximum aperture of the lens. An f/2.8 or wider-aperture lens will generally focus faster than an f/4 or slower lens, primarily due to the AF sensor's added sensitivity at f/2.8 or faster; but this is not always the case.
  3. USM design. AFD motors are slower and noisier than USM motors. The larger the USM motor, the more torque it generates, and the faster it can move the focusing group.
  4. Ambient light level. AF performance is impaired at very low light levels in the absence of any AF assist beam.
  5. Subject contrast. Low subject contrast, or contrast in a direction opposite to the AF sensor's sensitivity, inhibits AF acquisition.
  6. Subject distance. Close focusing is generally harder than distant focusing, due to shallower DOF.
  7. Optical stability. The stability of the image presented to the AF sensor affects its performance.



I may have missed a few other factors as well, but these tend to be the main ones. Of the ones that are relevant to the lens itself, you can see how it really comes down to a combination of factors, and not just one. It depends on how Canon builds the lens. The 85/1.2L II is a good example of a relatively slow-focusing lens, mainly because its focusing group is HEAVY--a large, thick aspherical element needs to be moved.