If the wall studs are straight and if your comfortable (temp wise) in the basement then I would leave well enough alone. Don't add additional vapor barriers, don't remove any that exist. Just replace paneling with drywall.
A vapor barrier can actually trap moisture and contribute to or cause issues. I'm not saying one is never warranted, it just needs to be right.
Since you are dry now why tempt fate?
If you do rip it all out then do the rigid foam glued to the wall, taped seams and then build a new 2x4 wall in front of it. Don't use furring strips. If you need additional insulation beyond what the rigid foam gives you then add UNFACED fiberglass to the 2x4 wall.
See
Info-511: Basement Insulation — Building Science Information for some good recommendations.