"Sorry for taking your time" does seem quite right to me because "taking your time" means to proceed slowly and carefully. It's much safer (if you want to be understood) to say "Sorry for taking up your time" or "Sorry for taking up so much of your time."
But, even better, is "thank you for your patience". You hear that a lot. It's better (psychologically) to give the listener a compliment than an apology.