Or, for a more controlled break. you can use a 20 oz. hammer. It takes a lot of licks but the block will begin to pulverize soon and you can control what you want to break out.
I don't understand why you wanted a square hole; dryer vents are usually round 4" pipe.
Craig mentioned a legitimate consideration; is your block filled with concrete? I know that many of the houses in Florida are, and even have reinforcing bars in them too.
It is possible to break out half a block, place a sleeve of PVC pipe and concrete around it, then you can run your 4" aluminum pipe through the sleeve. It makes a neat opening. (make sure your aluminum pipe will slip easily through the PVC sleeve before you start, you may want to use SDR21 PVC pipe for the sleeve. SDR21 is thinner, allowing more inside diameter.