space technology development is directly linked to conventional military technology development and we can't trust a neutral nation to secure national secrets. plus the ESA is more or less a NATO thing since the ESA is collaborating with NASA. Hell, we might as well rename the ESA to the North Atlantic Space Agency, which is still NASA and nothing would really change besides the acronym being more accurate, the U.S. is driving the space program.
tl;dr, switzerland is not allowed into the european/nato space agency because of national security