This is normal. No problem really.
You need an expandable caulk like ISO. It is paintable or even comes in 360 colors. You can order it online, or go to your local Lowes DIY store. The caulk will work in freezing weather and takes about 7 days to cure, so don't play with it once you get it set with a putty knife into the hole and scraped down.
I use it all the time, works just fine.