That little hole is not a push hole, it is an allen wrench hole.
Find the right size allen wrench and turn that little screw a couple of turns.
Do not try to remove it completely....Just back it off the shaft and you
will release the thumbturn. After you take that off just pry gently with
a clean putty knife under the decorative plate on the lock.
It will come off and expose the two screws that you seek!
Best of luck!