If there's any adhesive left behind, I'd use mineral spirits to remove it. Any mineral spirits left behind will evaporate completely without leaving a residue.
Baby oil is mineral oil with some perfume in it. (Manufacturers refer to the perfumes they add to products as "fragrance", but what you're rubbing the baby down with is almost pure mineral oil.)
Mineral oil, in turn, is a product of petroleum distillation. Petroleum contains all kinds of hydrocarbon based chemicals, but those that consist of a non-branching chain of carbon atoms that only have single bonds between each pair of carbon atoms are collectively referred to as "paraffins". (You can also have cyclic paraffins, where the carbon atoms are arranged in a loop, like cyclo butane. What's important to earning the title of "paraffin" is that there only be single bonds between the carbon atoms; double and triple bonds are verbotten.) Ethane, propane, butane and pentane are the simplest "paraffins", (and methane is also considered a "paraffin") even though you couldn't make a candle out of any of these gasses. The paraffin that is sold to seal canning jars and from which cheap candles are made consists of much heavier paraffins, with carbon numbers from C20 to C40. Mineral oil consists of paraffins from about C15 to C40, and it's those heavy paraffins in mineral oil that I'd be concerned about. They won't evaporate. They might not cause any harm being there, but if you ended up wanting to paint that surface in the future, you might find the paint won't stick to it until you cleaned those heavy paraffins off.
I'm sure something like baby oil would work, but by using baby oil you make removing the residual adhesive a two step process; cuz you then have to remove the residual baby oil.
Mineral oil - Wikipedia, the free encyclopedia
Paraffin - Wikipedia, the free encyclopedia