Indeed. Chinese and English are totally different families of language altogether.
1. Pronunciation (the 4 tones in Chinese, stresses in English etc)
2. Writing (combinations of "logo" or radicals to form a Chinese characters, a finite number of 26 alphabets in English etc)
3. Grammar (no notion of plural/singular, tenses in Chinese, word order etc)
and many other aspects.
All these differences cause a change in learning context and there is a need to learn a new set of rules that are so alien to a new learner.
It is a challenge for the Chinese to pick up English language and the non-Chinese to pick up Chinese language, _but_ it is definitely not impossible.