Do you have a water softener, and can you check that it's working properly?
"Soap scum" is soap that has lost it's solubility in water. The higher the hardness of your water, the more soap scum will be formed by it.
Maybe use a phosphoric acid based toilet bowl cleaner to check to see if it's soap scum. Since soap is soluble in water, hand washing in clean water would remove a soap film from the clothes, but not a soap scum film. If applying phosphoric acid to the clothes removes the white film, it was probably soap scum.