I looked at one house about five years ago. They added a room addition that, for some reason, enclosed the back of the stone chimney/fireplace. So, the stone chimney was on the inside of the house and was over half of that particular interior wall. I thought that it looked ok though.
So, I see no reason why you couldn't do it as far as looks go. I will admit that I know next to nothing about the insulating properties of just having brick for the wall. I would think it is ok, but someone will hopefully address that.