Projects are learning opportunities. If you want a door to behave, the jamb must be "plumb" (meaning perfectly vertical). Since the floor is "level" (side to side) you don't have to worry about it.
Temporarily removing the jack stud on the hinge side is no big deal. The header above the door can carry the load while you realign and/or shim the new jack stud to make it plumb.
Honestly, the drywall issue is minimal. If you don't know how to do it, here is your opportunity to learn how. As for the casing ... you might have to rip it down. Take what you can get and do your best!