One way to tell if we all see the same (or at least very similar) colors are the colorblind tests doctors give their patients.

With these images, we can conclude that most people see red and green as opposite colors, be it the same colors or not. For those that do not know how these images work, those who are not colorblind will see a "7" in the middle of the circle, and those who are red/green colorblind will see some other number.
If the colors on this image were not always red and green and could be seen as red and orange or green and yellow or some other similar combination for different viewers, it would probably be much more difficult for some people to see the number in the center.
Arguably, one could say that "the different colors are all in the same positions on the color wheel for all people, it's just rotated is all" but still, this shows that people at least have similar vision.