Thank you for the prompt reply however I think I should have either given a better description of the problem or posted a picture.
I do know how to remove a standard towel bar that uses brackets with a set screw, just as you described. Unfortunately, this is/was an old towel bar, not a modern one. All that remains attached to the wall at this time are the brackets that held the bar. They are firmly attached to the wall with four screws each, and the entire bracket is completely painted over. The surface of the wall is some sort of plastic, on top of drywall. I'd like to remove those brackets with minimal damage to the wall, and then I'll just spackle over the area and re-paint.