English grammar enthusiasts don't want to end a sentence with a preposition (words such as of, to, or by), so "Who are you speaking of?" is properly written as "Of whom are you speaking?"
"Who" becomes "whom" because it's the object of the preposition "of".
You are also correct in saying "To whom are you speaking?"