Charities and academics condemn minister's comments on large families