It's an unusual wording, so I Googled and found that it is a line from a song entitled "Forbidden Colours," described in a Wikipedia article,
http://en.wikipedia.org/wiki/Forbidden_Colours
"My love wears forbidden colours" sounds to me like a poetic way to say "forbidden love," much as a song by Billy Joel uses the words "when I wore a younger man's clothes" to mean simply "when I was a young man."
Wikipedia says "The title of the song is taken from Japanese writer Yukio Mishima's 1953 novel Forbidden Colors; although not directly related to the film, both works include exploration of homosexual themes." If this is correct, then my first guess is correct.
In 2014 in the United States people openly and casually discuss homosexuality, using words such as "gay." Fifty years ago, even the subject was close to taboo in the United States, and people like to use various euphemisms to describe it. For some reason, the color "lavender" was sometimes used to suggest homosexuality, Carl Sandburg, referring to the relationship between president Abraham Lincoln and his friend Joshua Speed, said that it had "a streak of lavender, and spots soft as May violets." Other colors and euphemisms have been used. A Noel Coward song suggests than in the 1940s in England, homosexuals might have worn green carnations as a signal. Thus, lavender and green might be "forbidden colors" signaling forbidden love.
Reading the lyrics myself, like much poetry it is all about feeling and suggestion and it doesn't have any clear meaning. I don't think it's possible to say for sure what "forbidden colours" means here. But I think I can say that homosexuality it one of the _possibilities_ that the phrase suggests to me.