Well getting the fence up was half the battle.
When I install my fence posts I pack the dirt around the post solid by using a sledgehammer or axe on the soil around it AS it gets filled in ...about every six inches .
If you do this firmly you should have minimal movement.
Some folks use stones or concret unmixed but I just go with the soil. When it is firmly compacted it does a great job.
Depending on how loose a post remains I may run some watter at the post to compact and settle the dirt further..
Happy tamping.