Actually I read a subtext here. Brutus wants to seem humble by saying he is ugly. He has (stupidly) set himself up for a backhanded comment by Caesar: Caesar basically says, yes Brutus is ugly, but that's OK because that kind of ugliness is suitable for someone Brutus' age. Just blame the age. :P