No way to tell by the picture just how bad it is.
If it's partical board, #1 it should never have been used in a bathroom, linoulium should never have been directly attached to it, there should have been under layment first.
If it's just the underlayment that's bad and not the subflooring then you can just cut that area out, (underlament is only 1/4" thick) fill in the seams with floor leveler and install new flooring.
That linolium was suppost to be all the way under the 1/4 round, not just up to it. I alway use a thin bead of siilicone between the flooring and the base board then the 1/4 round.
Resilicone caulk all around the sliding door frame on the inside of the frame after all old caulking has been removed and that area cleaned.
Make sure the drain holes are clear in the bottom rail.
At the tub to floor seal use 100% silicone then install a piece of vinyl cap moulding that gets set in a bed of silicone. It will help deflect the water away from that seam.