grammatically both are ok. but "go to the downtown" just sounds so rare and unnatural.
same with "home":
"i'm going home." i guess you've never heard people saying "i'm going to home"
i'd say...it's a matter of habbit
FYI, "go downtown" is also a slang