To politically threaten violence against self organized groups will threaten and weaken the sanctity of freedom other self organized groups feel. Religious organizations can be crime free and voluntary. Banning them would endanger everyone's freedom.
Religious institutions are often corrupted. They promoted/promotes war (every religion), covered child rapes (Christian specialty), instigated religious purges... No religion can claim having clean institutions.
most major religions have developed strong political undercurrents (ie Catholicism/anti-abortion). To support a religion is to allow these undercurrents to continue, even if you don't agree with them yourself. Abolishing religion would move society forwards and away from these prejudices.