I'd be inclined to seal the door first and see what happens. Thin 3/8" foam applied to the door stops. Try testing a big towel for that gap under the door. If the reduction is noticeable you can look at more thorough and permanent seals. If the door is hollow core, change out for solid core.
If the door sealing helps, the next level would be to apply drywall inside the closet. The added mass will likely be all that's needed.
If the door sealing helps, the next level would be to apply drywall inside the closet. The added mass will likely be all that's needed.