The reason is because "few" (and also "little") has dual meaning of a sort.
When it's used by itself (i.e. "few"), it means "almost none", but if you say "A few", it means "some (small number)" as opposed to none . The former stresses the negative aspect of "none" (non-existent), while the latter stresses the positiveness of "some"(existing).
"quite" acts as an intensifier, so if you intensify the positive meaning of "a few", you get "many". That is why you should always say "quite a few" and never "quite few" to mean "many".
"very few" similarly intensifies the negative "few" (no "a"), to mean almost nothing.
The opposites, "quite few" and "very a few" are not used because they can be confusing.
The above also means that you should be mindful of whether to include "a" or not before "few" or "little", as it can change the meaning (there are a lot of sloppy uses out there).
For example, compare:
1. I have a few friends = I do have friends, even though the number is small.
2. I have few friends = I have practically no friends.
The two sentences mean nearly the opposite things, so "a" is not optional.