"down" has many different uses - it can be an adverb, adjective, preposition, verb, or noun depending on the situation
In your example, "down" is being used as a preposition meaning: moving further along the course of something
examples with the same use as yours:
"he lived down the street"
"I walked down the street"
You can substitute "on" for "down" in "wandering down the street" but it does imply a bit of a different meaning - even so, most English speakers would know what you are trying to say
Hope this made sense!