"orange" is 橙子, I think. It's the size of a large tennis ball, the skin is usually thicker and a lighter or "yellower" orange. 橘子 is "tangerine" -- size of a billiard ball, usually thinner skin, a darker or "redder" orange. For me as an American, "Mandarin" by itself sounds like "普通话" or "高官". For "橘子", *I* would say "mandarin orange", not just "mandarin", but online dictionaries seem to disagree with me. Maybe it's a British thing (as wywu suggests)? Also for American *me*, "tangerine" implies fresh fruit while "mandarin oranges" suggests canned fruit slices.
So:
橙子: orange
橘子: tangerine, mandarin orange
tangerine: 新鲜水果
mandarin orange: 平常指罐头食品?
I find it as confusing as you do. Maybe think of it this way: All poodles are dogs; not all dogs are poodles. If you've heard someone call a "橘子" "an orange", it's not because we don't have them or can't tell the difference, it's because you use the "bigger" class word first (I think of a "tangerine" as a *kind* of "orange"), then get more specific if you need/want to. If I pointed to a four-legged animal on the street and said, "What's that?", you would probably first say "That's a dog." before "That's a poodle."