A wall that blocks people out also blocks people in.
From Frost's "Mending Wall:"

"Before I built a wall I’d ask to know
What I was walling in or walling out,
And to whom I was like to give offense.
Something there is that doesn't love a wall,
That wants it down."