My friend asked his school priest.
Answer: No. Doing it too much is.
The main reason is that not getting off for extended periods of time can cause medical issues. So, were it a sin, all men would develop some sort of bad medical issue.
(Yes, this problem is caused by a buildup of excess sperm.)
Albeit true that this issue would require a pretty damn long time of not fapping. (around 3 years? correct me if i'm wrong.)