It means that Brutus looks better (e.g., more handsome) as he gets older.

Brutus is ugly so he is age appropriate. lol

Actually I read a subtext here. Brutus wants to seem humble by saying he is ugly. He has (stupidly) set himself up for a backhanded comment by Caesar: Caesar basically says, yes Brutus is ugly, but that's OK because that kind of ugliness is suitable for someone Brutus' age. Just blame the age. :P