The admission to drugging women was just that; an admission to drugging women. There was no admission to raping women, and there was nothing that said that he did it without the women's consent.
If you look at his statements, he's said that the accusers and he would have consensual love and the pair would decide together to take drugs in order to heighten the experience. It's entirely possible and considering that this was during the 70's, when everyone was trying everything in order to get high, highly likely that this was the case. Even the situation in 2004 was likely to have followed this pattern.
The most Cosby did was get loveual with women off-stage, straying from the father-figure personality he outwardly displayed. He's not a rapist. There's just no definitive proof.