In my experience you can use all three with 'build' But, you tend to hear the sizes connected with the word build more in the field of police work. I'm referring to suspect descriptions.
In America we tend to say someone is either heavy or medium weight or thin. At least that's what I've generally always used. Of course you could be crude and call someone fat. Skinny is the same as thin and isn't rude at all (especially if you're telling a woman that she's skinny).