when ボール is used by itself, i think the first image that might appear in one's mind is like a sports ball such as a soccer ball or a tennis ball.
but i think 玉 is more or less used to indicate a ball-shaped something.
if you were to use 玉 in the context which ボール is appropriate, people might understand but could think it sounds strange.
think of, for instance, when you want your friend to throw you a basketball but you ask to be thrown a sphere... i think it's the same kind of deal for 玉 and ボール.
but that is just what i have accumulated from my experience.
please get a second opinion from a native just to be certain :)