Wrong, scientifically speaking, it is every colour at once. Remember, the colours we see, are the result of an object absorbing every colour but that so it reflects. Which in turn lets us see that colour.
You're referencing pigmentation (which would give you a weird greenish brown, anyway). In reference to the actual black, there are no photons of light, and therefore there cannot be color. White is all of the primary colors combined, using light.
In pigmentation, black is a color, while white can be (debatable). Black isn't the absence of color, it is an actual color -- a creation. It's dark, but it isn't the perfect absence of light. This would be like saying we use paints which act as a malleable, consistent-like black hole when using straight black. In reality, the easiest way to deal with this misunderstanding is to separate the perception of black and the absence of color.
In reference to OP, it is not impossible. The concept of photons bouncing from object to object is difficult to explain, sure, but it is not impossible.