Eggs in one basket is a hard lesson to learn.

I have been in your shoes and walked all those construction miles in one house.
You need to see it comming before it happens and set your company up for the slow periods, they happen every ten years in the cycle up here.
Diversification for when you see the change comming also helps.
I started to look into home inspections after 20 years of building, and remodeling. Then that lead to training because of all the info from my past work. Which leads to other oportunities to be hired for speaking engagements.
I also have started energy audits which will help folks with decisions on the homes that are old and new.
Now I am interested in building again because of all the junk on the market which is very inefficient.
So don't despair and wait for other people in slow times, you need to grab your business by the horns and learn as much as you can about where you are going, sometimes it is hard to change direction, but if you do not, you will end up doing a job working for someone else you will not like.
Good luck.