If they want a certain atmosphere, they can't have any kids acting up, and can't be asking people to calm them down or leave.
Restaurants are not public services. As private businesses, they have the right to be exclusive.
of course this isn't directly related to the exact issue of whether they should. but I'm not sure what to think of it.