Some people will say yes and some people will say no. The same argument you made could be used to outlaw a LOT of human behavior, though.
For me personally, I universally don't think it's fair that I could be stripped of some of my rights without due process - that bit is important to me regardless of whether that is used for wrongdoing by others or not. A better solution would be to make due process happen faster, imo...or for the state to take a more proactive role in protecting the accuser until that due process runs its course.
I agree with that...but that is not how it's currently implemented.