Sounds like the panels themselves are starting to give out. The frames carry all that glass weight for long periods of time in weather and start to sag.
That and the framing around the door itself may also be moving slightly and shrinkage of the floor joists at the sagging threshold.....basically this all adds up to door removal or adjustment in place.
The issue is how the weather seal works if you adjust in place.
And ...Sometimes you need a new door.
All this movement in a door opening with only an eighth of an inch gap on all sides does not leave much room for door to end up not working properly.
Hope this helps...or at least makes sense.