Hate figure known as Roosh V said he could not guarantee safety of those who attended