Usually with synonyms, the words have different levels of formality (words that come from French are usually more formal, and words from Old English are informal). For example, 'maternity' and 'motherhood' are synonyms, but 'maternity' (from French, maternité) sounds more formal and is used in scientific situations (maternal mortality rate) and fashion (maternity clothing), while motherhood (from Old English, moderhode) sounds less formal and is used in regular conversation.
So, 'annoy' is not a very formal word, but it is more formal than 'to bug' which is slang. ('Annoy' comes from French, ennuyer; and 'bug' comes from Old English, bugge). You could use 'annoy' in a formal essay or in conversation, but 'bug' is only for informal conversation.
This comes naturally to native English speakers, but I think you can learn with practice and by listening to when native speakers choose certain words over others.