First, I'm assuming this is the little pushbutton switch that's activitated by the door jam pushing on it. I've seen some of those switches become faulty and actually can be pushed too far in. In other words, with the switch pushbutton all the way out, the switch is OPEN. When you push the button in, the switch is CLOSED. But, if you push the button in even farther, the swich goes to OPEN again and therefore the alarm system can't be set. This may be happening because there is so little clearance in most door jams that the switch is pushed in too far. Personally, I would just replace this switch with a new one since they are not expensive. Trying to repair it will just be aggravating because it may not work consistantly.
P.S. I hate it when people resurrect old posts!