Experienced sheet metal guys will tell you that you have a "junk air" problem. A natural, relatively smooth flow of air molecules is being compromised by multiple changes in direction and small (but significant) in-line restrictions. I strongly suspect that changing the run to vent vertically through the roof would be significantly more advantageious than going down through the soffit. I don't ever recall seeing a bath fan run increased from 4" to 6" to improve flow in the (40) years I've been looking at them. And a marginal fan pushing into a larger run and transition will definitely create more junk air.