in reality there is basically nothing you can do. you couldn't ever get him to pay attorney's fees on this sort of matter, and a legal case with a lawyer would probably cost between 0.5 and 5 thousand dollars.
you'd lose like 50x more money than you'd gain if you attempted to go to court on this issue.
and the reality is, without any evidence, nothing could happen anyway,