Yes, it does sort of contain the idea that he didn't know at all about something.
But this expression is used in a particular context. It is used to remark on the fact that a person committed some action while entirely unaware of what was going to happen next, or what he was getting himself into.
The "little" that the person knows really implies that they could not anticipate the outcome or the consequences.
"Little did he know, when he started his journey, how difficult it was going to be."
"When he asked the beautiful woman for directions, little did he know that this simple question was going to connect him with the love of his life."
"Little did he know that he was about to embark on the most thrilling experience of his life."
"Little did I know that John was going to become such a great employee."