it's not like they COULD do anything to us if we left
Ok, Al Queda is so poorly run that they can only do one attack every decade.
Other organizations are intelligent. The Muslim Brotherhood is an umbrella organization, acting as a religious group. It has over 75% of all Islamic men in it. The ones that are not "radical" are members of the Muslim Brotherhood solely.
They are very, very smart. As an umbrella organization, they are only a cover up. Many members of the Muslim Brotherhood join organizations that "aren't connected," but in reality are the partners of the leaders of the Muslim Brotherhood. Those organizations attack us, then get taken down by us, without scathing the Muslim Brotherhood, at all.
In doing this, we can't attack nor demand that the Muslim Brotherhood be shut down, because it is "a peaceful religious group."