Some will vote Tory solely to oust Labour leader from his job