It could be anything. Holes drilled for electrical, even the holes for baseboard pipes through the floor show up on my infrared.
Check your attic insulation for windwashing at the eves. This happens when winds blow the insulation away from the most important are of the attic, right where you get ice damming at the outside wall and roof area.
Seal any holes you find in walls in the attic, insulate around outlets on outside walls and use the outlet gaskets, they help to slow down the breeze.
Check your window and door insulation and weatherstripping.
And in my opinion, you do not need a blower door test, it's a waste of $$.
Air leaks are easy to find this time of year, open an upstairs window or the attic hatch that may not be insulated, and you will feel the makeup cold air comming through in any downstairs leaking areas.
Could be any one of these or all combined to make a few rooms cold.