Hi Becoh -- I'm a native English speaker and have begun serious study of my "first" second language: Japanese. I did a bit of study on this awhile back, and I have learned a couple of things.
Since so much of Japan's writing is based on Chinese symbols, learning Japanese would be much easier if you were to learn Mandarin first. However, since the Japanese system of symbols is a simplified version, it's easier to learn Japanese initially than it is to learn Mandarin. From a conversational dialogue standpoint, I believe that it is also easier to learn Japanese. Why? The main reason is dialects. In Japan, you speak Japanese, and you are spoken to in Japanese. In China it isn't so simple. Depending on where you are and who you are speaking with, it will vary and can not only be very confusing, but frustrating as well. Additionally, just from my own personal study of Japanese, I find that the sounds are very, very easy and simple, and flow more easily. Just listen to the Japanese alphabet on youtube if you don't believe me! ;) Lastly, conversational Japanese really isn't that large: less than 20,000 words versus almost 300,000 in English! From what I have read, Japanese is about equal in difficulty to learn for English speakers as English is for Japanese speakers. However, Chinese is considered about one and a half times more difficult.
Final note: learn the main one that you have a desire to learn. By doing so, you will work harder, do better, and enjoy it more!
Ganbatte!