When you have a bigger player the engine is having to perform many more collision checks for each time they move (especially if they're walking on lots of bricks or something) plus the simple collision mesh inaccuracies are exaggerated by scale. It just results in a decrease of overall gameplay quality.