Hi Emar,
Personally, rather than say "difficult for me to get the grasp of", I would say, "difficult for me to grasp". (And it should be "difficult for me" not "difficult to me".)
But people do sometimes use the phrase "get a grasp of something". But in my experience, it's mainly "get a grasp" not "get the grasp". There is another similar phrase with "the": "Phrasal verbs are difficult for me to get the hang of." "Get the hang of something" is an informal way to say "learn how to do something".
So, to be clear, I for your sentence, I would write:
"Phrasal verbs are difficult for me to grasp." or "It's difficult for me to grasp phrasal verbs."