Replacing shutters that were screwed into wood slats in my brick walls?

[soparklion11]

My shutters are screwed into wooden slats (circa 1945) that are the width of a mortar joint and 2-3" wide.

I suppose that the right answer is to hang new shutters via spikes into anchors that I place in the brick.

What should I do with the pieces of wood? Just pull them out and patch with mortar?