In English, we just use certain family relationships as names or terms of address while others we don't. Mother, Father, Grandmother, Grandfather (and related words) are all universally used as names / terms of address. Some relationships, such as aunt and uncle, may be in certain dialects, but are not commonly. Brother, sister, niece, nephew, etc. are just relationships. We don't call the people who have that relationship to us by that name. We use their given names. Only the words for older generations are used as names / terms of address.
Actually, in your example, it would actually be much more common for both mother and brother to have my. It is very common to call your mother 'Mother' when addressing her. It is less common to do so when talking about her.