Yes it does rub off on your hands.... Thank you.Is the paint on the siding now chalky? If you rub your hand across it do you get paint on your hand? I'd probably use a garden sprayer with a solution of TSP (available in the paint section of your local home improvement store, or paint store) and then powerwash it. Then use a paint recommended at the paint store for aluminum siding.