On our house we had the roofers to tear the chimney down out of sight and cover it with decking and shingles. We saved the fireplace inside with the mantle and hearth, closed the damper, put some of the bricks in the chimney to hold the damper down. Then, just to keep the fireplace authentic without all the roof leaks, we put a set of ventless gas logs in the fireplace. We have the amenity with out the leaky problems (best of both worlds).
To remove the whole chimney and all remnants of it will involve a lot more work than mine did. I would think you should get it done for half the amount you mentioned, $2,000 to $2500.
Please post back and let us know how it worked out.