I used to think that doves were just pigeons that have albinism, but it seems that, though they have many similarities, they differ in size (and color, obviously). But yeah, I have wondered this as well, for a long time. LOL

An excellent question, and one that most native English speakers do not know, lol.  Basically, there is a family of birds (meaning a group of species) that is made up of all species of pigeons and doves, of which there are hundreds.  Whether a specific species is considered a pigeon or a dove is completely arbitrary and based on nothing.  But the most common bird referred to as a pigeon is a rock dove, which is the dirty street pigeon you see eating garbage and harassing people eating McDonalds.  The birds most commonly referred to as doves include white doves (of any species), and mourning doves, but mostly people just call any filthy dove or pigeon living in the city a pigeon, and any wild pigeon or dove in the countryside that is clean and not eating french fries or bothering people will usually be called a dove.
There is a family of mourning doves nesting outside my house now... :)