Why, according to Rawls, must both slave and slaveholder be supposed to agree that slavery is unjust?

Now the conception of justice as fairness, when applied to the practice of slavery with its offices of slaveholder and slave, would not allow one to consider the advantages of the slaveholder in the first place. as that office is not in accordance with principles which could be mutually acknowledged, the gains accruing to the slaveholder, assuming them to exist, cannot be counted as in any way mitigating the injustice of the practice. the question whether these gains outweigh the disadvantages to the slave and to society cannot arise, since in considering the justice of slavery these gains have no weight at all which requires that they be overridden