You answered your own question. If you don't want it to last long, go with the cover it up route. Options are limited to plastic and duct tape, or a siliconed piece of plexiglass.
But I think it sounds like you need to get ready for a project, I would take it all apart and re-tile the shower. ANY drywall behind the tile IS going to fail, and usually in a big way. The soap dish is just the begining.
Look on the bright side, at least you found it without a ton of mold and rotten wood like most of my customers who always wait to fix it later.
Let us know what you get into. We will be here to help.