I've been out of Caer Scima since October in my own establishment, and there's been no reprisal. [For now, at least. It'll be a while longer to see if there's really repercussion.]

And plenty of people who don't want to fight head to the Station, even temporarily. I think if you duck out for even a little, you'll be quite okay.