Never seen one "fall apart". A sealer will help preserve the look and the integrity of the concrete. I like the sealers that leave a little gloss on the surface and make the agg look wet. The problem is, that makes it slick when it's wet. Concrete exposed to the elements will erode and crack. Usually it will not crumble. The cracking is usually caused by the ground freezing and swelling, then thawing. Sidewalks are terrible about cracking because of their typical long and narrow structure. Control joints are usually the answer to the cracking problem but not always.
Exposed aggregate is about looks. Sealers add to the appeal. If I were you, I'd seal it anyway.