It's probably just two additional factors: 1) the difference between x feet and infinity, and 2) the difference in the center of perspective. If you setup a tripod and just change lenses, then you may not realize that you're also changing the location where light enters the lens-- the entrance pupil. For example, imagine you have a 50mm lens that is 5 feet long, but through the magic of optics gives you an angle of view that is truly 50mm. Then you have a 50mm lens that is only 5 inches long. If you setup the tripod and compare them by just swapping lenses, the 5-foot-long 50mm lens will give a different field of view. But if you instead align them based on the center of perspective, which is the entrance pupil (or virtual image of the lens' aperture), for both lenses, then compare them, they would yield the exact same field of view (and angle of view).