The vent should be back a lot closer to the other wall. I can't imagine that there is a lot of framing in the way preventing it from being 2-3" off of the other wall. Be that as is may, moving the vent will involve cutting another hole in the side of the house and trying to patch the existing hole.
If you can post the make and model number of the dryer we can look it up online to see if it has the capability of venting out that side and what it would take to make it happen. You might need some help, but I think that would be the easiest solution if your dryer can do it. Otherwise, I think you need to relocate the vent further back as bud mentions. A decent handyman could probably handle either task. If you're not handy and don't have tools a good handyman can be invaluable.