It depends on the type of brick. I have clay bricks, and the two sides of my house don't have much overhang. As a result, the sides of the house get wet a lot, and when clay bricks get wet and then the water freezes, they get "spalling". Progressively larger pieces of the brick flake off -- every spring I have flakes of brick on my driveway, and some bricks have lost as much as an inch of their surface.
I'm replacing about a dozen bricks on one side of the house right now, and when it's done I'm definitely applying a sealer to the two side walls that get wet.