There is no grammatical reason that you cannot use "mostly" to modify "prefer" since prefer is a verb and "mostly" is an adverb.
However, somehow, it sounds more natural for me to say:
Most people prefer to take their holidays in (the) summer. (Most Americans would use "the" with seasons, whereas I understand that the British do not)