The placement determines which word your are modifying.
You speak perfect German. In this case, 'perfect' modifies 'German', and is therefore an adjective.
You speak German perfectly. Here, 'perfectly' modifies 'speak', a verb, which makes it an adverb.
There is no logical reason for this. It is just one of the rules that we go by. Maybe someone can disagree - I know the rule for this placement - I'm just not sure of the reason.