Hey Allthumbs, Welcome to the Froum:
I feel like I'm talking to myself (All Thumbs).
Perhaps you could open the damper, put a piece of 1" angle up in the cavity and lay it down over the gap. If you could get a good fit and get it to lay flat on the gap, it would cut out the draft and would be permanent. Almost any kind of grout or mortar will crack, break up and fall through the gap; whereas the steel will be there for years.