For me, speed of speaking. You can learn vocabulary and idioms and in spite of knowledge you don't understand what speaker says.I would say nationality makes a difference. It's easier to listen people from Canada/US than from the UK.

For me it's sometimes rather difficult to understand native speakers talking to each other. It's much easier to understand BBC documentaries.I think, it isn't because of a lack of vocabulary.

I think I am getting more and more used to listening to American speakers but sometimes I find it hard to understand British speakers, especially when they talk really fast. It's mainly the accent and pronunciation of native speakers that makes it difficult or easy to understand. Also, unknown words or phrases trips me up sometimes. When I hear something I don't understand, I try to guess what the speaker said and then I miss what he/she says next because I am too distracted trying to figure out the word/phrase I didn't understand. The easiest to understand is someone I am talking to face to face, mainly because there is no distortion. Next is listening to someone speak, maybe in a movie or a TV show or podcast, with headphones on. Third in the list is watching movies, TV shows without headphones. The TV or computer speakers tend to distort the sound a bit and sometimes it can get difficult to understand. The most difficult to understand is on the phone, especially when the transmission isn't very clear.

I think it's the speed, I find much difficult understand the pronunciation when the people are talking quickly. I lack even many words in my vocabulary of course, but when I'm reading I can understand the meaning of the sentences even if I don't know some words, while when I'm listening to someone and I don't understand a word, I can not understand the rest of the sentence because my mind is focused to deciphering the unknown word.

I feel If I'm listen and I can see the speaker, I feel it will be easier, but I can't understand them immediatly if they say a long sentence, grammar is OK, I think some practical vocabulary I don't know.that's all