I kinda like that. People would only do it if the person really needed to be banned, not just vote kicking because they have a vendetta against them.
It would have to fail pretty miserable to ban the original voter though. Like 80% no's. (If it just fails naturally because of a lack of yes's, then the caller is ok)