Should I chip away at the remnants of the grout before I seal the gap, or should I just seal over it?
Anything loose, take it out with a flat screwdriver, razor knife, pull scraper, putty knife, etc.
Clean the area and borders well with straight rubbing alcohol and a white plastic Scotchbrite pad.
A colored pad can leak some color while cleaning.
If using a colored pad, typically green, do a second wipe down with alcohol and a clean white cloth like an old Tee shirt or white athletic sock.
Let dry for at least half hour, then use blue painters’ tape to carefully mask off your intended new caulk profile.
Apply caulk carefully, don’t leave any bubbles or gaps.
Don’t overfill, add only slightly more than you want as a finished layer.
With fingers cleaned with alcohol, immediately smooth and tool the caulk bead.
In very long runs, do this in sections of about four or five feet, so the caulk can’t skim over before tooling.
Peel off the blue tape immediately after tooling any finished straight section.
Pull the tape back towards itself, and slightly off axis away from the caulk, this gives a cleaner separation.