Welcome to the site. A 4" duct should be enough but you also need to look at the length of the duct and the number of bends in it. One ninety degree bend is like adding 5 ft to the run. I took ojne bathroom apart to find a three inch duct with five elbows in three feet and one joint had fallen apart and the end of the duct was stuck into fiberglass insulation for twenty years. When the fan is running put a kleenex over the screen, it should hold it there. If that works try it again with the bathroom door closed, sometimes there just isn't enough repacement air.
For water getting around the shower curtain, I've seen success with the new curved curtain rods and the trick is to move the rod into the shower area about 5 inches. With the curve you still have more room than with a straight rod.