You can say the sentence you said as you are not calling anyone man here. Oh man, is an exclamation not a form of address, however if you were to call a woman, man, then the answer is most definitely no! Roborto is right, generally women like to be addressed by their name, if you don't know it, use Miss or ma'am, don't call us things like,chick, bird etc either! oh also we dont call women 'lady' unless it actually part of an official title.