"I don't know what I am doing here." is correct. For the other to be correct, it would have to be two sentences. So I don't know. What am I doing here? But this means something completely different. For instance, you would say it if someone asked you "What are you doing here?" "I don't know what I am doing here." is just stating that you don't know the reason why you are here.

‘’I don't know what I am doing here‘’ is correct.

I don't know what I am doing here.