I would say that "Nouvel An" will generally refer to the celebration (you will use "Nouvel An" as you would use "Noël" ) , so to the New year's Eve or to the 1st of January . If you tell someone "Viens passer le Nouvel An avec nous !" you don't expect he will be your guest during the whole year :o) , you just invite him to celebrate New Year's Eve  with you . Note it's almost always written with capitals : Nouvel An.

"nouvelle année" just refers to the year to come . When you send season's greetings you can write "A l'occasion de la nouvelle année, je t'adresse mes meilleurs voeux de santé, bonheur, et réussite" . Or you can ask a friend "Quelles résolutions as-tu prises pour la nouvelle année ?" "Je vais essayer d'arrêter de fumer !" 

Hope it helps ... :o)