Solution to your problem: put a pair of handwarmers (the disposable air-activated ones) around the lens barrel, and secure them in place with a rubber band.