In my experience, I will speak the language that is the easiest in terms of keeping the conversation going. I remember from when I was trying to learn Spanish in Spain, that I would always switch back to English whenever I would get stuck in a conversation or if there would be awkward silences.
I guess the same goes for when you speak Dutch in the Netherlands. I reckon a lot of people will find it really interesting that you learn Dutch, but if you wouldn't speak it well enough to keep a conversation going, yes, then I would definitely switch back to English.
Nobody means to be rude, but you know, people very much dislike awkward conversations :)