Is the aggregate actually crumbling or is it just being released from the comcrete?
You may have "burned" the cement matrix around the aggregate with too much acid or did not wash at the right time for the concentration. This can reduce the resistance to freezing & thawing cyles or ever wetting & drying cylces.
I have no idea of what your climate is like, but you obviously have freezing. Around here, you also need air entrained concrete to resist the freezing & thawing cycles. It is required by code for exposed concrete. Because of this the concrete suppliers end up supplying most of the exposed concrete, since they are familiar with the air entraining admixtures.