permits are the only way to go if you have to ask what we think. Make a drawing of what you want to do. Go get a permit and find out at what stage they want to see it.
When he says you should have removed the drywall behind the ledger, tell him you wanted to keep the it for the fire stop. He will want lag bolts thru the ledger to something solid. If your span is more than 7 ft you will need bridging 2x2 diangle bracing
between joists. and solid block where bridging does't fit. you buy bidging pre cut.