As long as you use $Pref::Server::AllowColorLoading, there won't be any problem.
There's only 3 forms of color loading.
One where it just uses the colorID, whether it's even the same color or not.
One where it checks for the closest match to the colors, and assigns colorIDs based on that.
And one very broken one that attempts to append the save's colorset to the current colorset, whether or not it exceeds 64 or any matching color exists, then tries to assign the bricks the new colorIDs.