The problem is that this could be used to further justify actions. For example if you don't work, you don't pay taxes. If you lose your job and protest some tax that costed your job at a failing company, the government could silence you because you don't pay taxes just because they can silence tax exempt organizations.