Prime it if the brick is a contrasting color or has different colors from brick to brick.
Latex is fine, block filler isn't necessary on smooth brick.
Never use an oil base paint,
always use an oil base stain..
Stain and paint have such different properties that Oil will work better for stain but oil basedpaint would harden to a flakey mess.