One thing is sure. If there is mold there is moisture.
Mold needs two things to grow:
1- organic matter in which it feeds (in this case the drywall and probably the latex in the paint as well)
2- Moisture levels at of above 60%.
In a home you can't avoid using organic materials. You patch those walls without taking care of the moisture problem, mold will only come back.
So you are left with the task of controlling the moisture.You will probably need to remove all that drywall and look at the insulation behind it. My bet is that it is wet as well.
You will need to locate the source of the moisture. Are those internal or external walls? Look for pipe leaks, hole in the siding, leaks in the roof.
Buy an inexpensive hygrometer at any hardware store and check the relative humidity in the air. If it is above 60%, you might need to run a dehumidifier.
Be very careful however dealing with the moldy walls. You don't want to disturb it and stir more spores in the air to cause a broader infestation. Although only a portion of mold species are actually toxic, even the non toxic species are know to trigger allergy symptoms in sensitive people.
(In addition to staining your surfaces and smelling very bad)
If you suspect it might me toxic mold, have it tested. After that, just take the proper measures to keep the humidity below 60% and any mold in the air will dry out and lay dormant, no longer releasing spores and growing on surfaces.