that is very true but its nice to have a lot of colors. so you dont have to keep pressing E rapidly to get the color you want, you can just simply scroll and get what you need. but i do agree with what you say, it does make it kinda ugly looking, but it looks doesn't really matter for a color set.
It takes up space on the screen and it's unpractical.
It's better to have not many rows but organized columns.
Let's take for instance LeetLegacy Colorset:

First
4 columns are infact the same color with lightness changed, the fourth column being transparent. Thus' you can only press E to get to the same color in a different lightning.
The 5th column contains special color.
The 6th column contains grayscale, last 3 rows are transparent grayscale.
The 7th column (last) contains browns, because wood need different colors it's useful to have a whole column for it.
Truenos idea much?