Generally speaking: Write in English. Write on a paper (or on a hand). The song, however, chose 'in' to evoke "being part of" and play with the idea of "permanence." Are we written in the stars? = are we written among the stars/as part of the stars *You can think of "in" here as "included" or "inside" or "among" *Write on a star = writing on the surface of a star, which isn't the image the song wants to evoke Or are we written in the sand? = amidst the sand *When you write in the sand, you have to etched or dig a part of it, so it's IN the sand. It's not on the surface of the sand, it's within it. See Elsa's comment above. *Write on the sand = write on the surface. Maybe you used water or paint or sticks or something you can put on the surface of the sand, which again isn't what the song is trying to make you imagine Are we written in the stars, baby, or are we written in the sand?' = Is this permanent or just a passing thing?

With sand, there might be some logic, in that if you write on it, you make lines in it. It's similar to chiselling words into a stone tablet. And in the case of a tattoo, it is also injected into the skin. There is some flexibility with those. Even if they are wrong, we usually understand what is meant. And of course songwriters are famous for taking liberties with language...

I would say "in" because it is engraved into the sand and not on top of it. In general if there is any sort of 3D element I would use "in". I count sand as 3D because when draw in the sand you are leaving a dent inside the sand not on top of it. With names in a tattoo I see the names as being inside a bigger tattoo then with number on a hand, it's like writing on paper. It is technically on top of the skin not inside it. I hope that makes some sense :)