Dryer vents are subject to local building codes. You should check with your local building inspector's office.
I thought 4" was mandatory but maybe not in your area.
If you were required to have a home inspection before you bought the home
and 4" is the requirement, then you may have a claim against the inspector if you incur costs to upgrade the vent.