Originally referred to bits of Ireland eg far SW beyond the jurisdiction of england, subsequerntly fell into metaphorical use as unacceptable bad/bad behaviour/ bad condition. I think it comes from some derivation of fence or at least has same origin as "pailsade" which also indicates a boundary.