I used wood glue on a test corner and checked it this morning and it is still damp so I will take your advice just go with what slowandsteady recommended, the Foamboard VOC Adhesive.
The other reason I went with the foam was to damper the sound. It is a plastic 4" and unlike the old cast-iron type, you can hear stink water every time a toilet is flushed or sink drains. This is kind of a quiet space so wanted to get rid of as much of that sound as possible. As an aside, I would've had to go even further into the room if covered with anything, 1/4" ply or sheetrock which would also be a lot more work.
Again, this is tucked in a corner with no potential fire hazard. I suppose I could put fireproof paint over it...