Color is determined by sight, which is what we use to see the color that an object does not absorb.
If an object is "Black", then it absorbs all color, and reflect none of it, therefore not being represented by a color. Black, being a shade on the scale of any color, is not considered "All the colors at once". It is the absolute absorption of light, and it isn't any color at all.