THAT is unusual ... Normally, these rails are attached with screws because they will corrode at some point and need replacing. Drilling will be hard because you cannot hold the drill exactly on the nail head. You will slip off to the side every time.
A small Dremel grinder with a metal cutting wheel (as Neal suggested) is probably the best. Gently cut the head off and do not penetrate the aluminum rail. Once the rail is free, you might have enough of a stub to grip the nail and remove it. If not, grind the nail flush to the tile so you can install the new shower rail and door. You might hit the hole with some silicone to prevent an possiblity of water getting in there ... I always apply silicone to the side and top of the new rails.