Wow, sounds like you have a bit of a mess.
Your basic instinct is right - put a complete coat of material over the whole mess. This is known as "skim coat". You can use all-purpose joint compound and and cover it all, but it usually takes two or more coats, plus some sanding at the end to get it smooth.
But ... before you do that, you need to deal with the areas where the face paper of the drywall has been damaged. If you just skim coat over it without proper preparation, you may get some bubbling of the paper under your mud, and you don't want that.
You could always test a little area with a thin layer of mud to see if it will bubble. If it does, just scrape off the mud right away and let dry before proceeding.
Rather than go into the whole process of treating this kind of situation, it is easier if you would read a webpage I have devoted to this very topic. I deal with this kind of situation all the time with my clients, so I have found out what works and what doesn't.
How To Repair Damaged Drywall After Stripping Wallpaper
Regarding the rest of the wall, you can skim coat over wallpaper glue with no problem. I do it routinely. But you have to have each coat dry thoroughly before you touch it for a second time. Otherwise, it turns to "peanut butter" under your tool and makes a mess.
Try to remove any remaining wallpaper. The fuzzy stuff you might be able to skim coat over. Test and see.
There is a certain art to doing skim coat well. I have a page about that also.
Good luck.