Luan would be a nightmare to fasten down in that sitiation.
Sheet vinyl would be a problem over bare tile too..
Leveling compound would shrink over the deep joints between the tile.
I've not had to deal with this problem because I always removed the tile down to the bare wood. That is exactly what I'm recommending to you also. When you want a job done right, you need to actually do it right. Remove the tile add a layer of plywood to level and sturdy the floor, sand the joints if you can detect them by touch, then use a floor leveler to cover the joints and cover the nailheads. You will come out with a good quality floor job and no worries about the tile releasing the Luan or the leveler.
Check with your supplier, the vinyl may require a certain type of underlayment to keep your warranty valid.
Welcome to the forum,
Tom