If the paint is coming off, the paper drywall covering may be coming off. If so, I would suggest trying to scrape off any of the loose, fuzzy fibers without damaging the drywall. Possibly sandpaper.
Lay down a nice skim coat of drywall mud (top coat), whether only the spots that need it and feathering out when it’s dry, or skim coat the entire surface.
Primer is a must imho.
If you plan to re-apply wallpaper, be sure to use "sizing", which can be found at any big box store, Sherwin Williams etc.