The deer do return on their natural schedule. Much has to do with sunrise and sunset in addition to the rut, breeding and fawning.
The deer in our suburb can always find food and rarely resort to getting most shrubs and plantings. Being in an area with some slightly higher ground with trees and brush separated by very wide low marshy/wet areas, the routes do not change, but the time when you are up and around to notice them does change radically, especially the farther north you go. - It must tough for a deer much farther north when the sun comes up very early and sets very late (like 10:00-11:00 PM).
The deer do not really leave, but they change their patterns and timing.