It's a little unusual. We wouldn't normally use 'get down' in this way. One of the possible ways to use this phrasal verb is to use it to replace 'feel sad'. For example, "Whenever I think about all the problems in the world I get down." It is informal. Like most phrasal verbs. However, generally this is a feeling which is produced by yourself, not something you do to someone else. One exception is that you do often hear people saying "I don't want to get you down, but...".
You might have been thinking of the phrasal verb "let down" which means to disappoint/not fulfil a promise?
If not, then using 'make sad/unhappy' is best